- How do you copy adjust the split size

1.	- How do you copy adjust the split size
Answer» Step-by-step explanation: I have a bunch of small files in an HDFS directory. Although the volume of the files is relatively small, the AMOUNT of processing time per file is huge. That is, a 64mb file, which is the default split size for TextInputFormat, would take even several hours to be processed. What I need to do, is to reduce the split size, so that I can UTILIZE even more nodes for a job. So the question is, how is it possible to split the files by let's say 10kb? Do I need to IMPLEMENT my own INPUTFORMAT and RECORDREADER for this, or is there any parameter to set? Thanks

Answer»

Step-by-step explanation:

I have a bunch of small files in an HDFS directory. Although the volume of the files is relatively small, the AMOUNT of processing time per file is huge. That is, a 64mb file, which is the default split size for TextInputFormat, would take even several hours to be processed.

What I need to do, is to reduce the split size, so that I can UTILIZE even more nodes for a job.

So the question is, how is it possible to split the files by let's say 10kb? Do I need to IMPLEMENT my own INPUTFORMAT and RECORDREADER for this, or is there any parameter to set? Thanks

Discussion

No Comment Found

Related InterviewSolutions

1. ਜੇਕਰ ਕੋਈ ਵਕਰ ਰੇਖਾ ਆਪਣੇ ਆਪ ਨੂੰ ਨਾਂ ਕੱਟੇ ਤਾਂ ਉਹਵਕਰ ਹੁੰਦੀIf any curve does not intersect itself then the curve is called
Tanusec²u
Check whether the following equations are consistent or inconsistent. Solve them graphically.a) 3x+2y=5 2x - 3y = 7b) 2x - 3y = 8 4x -6y = 9c) 3/2 x + 5/3 y = 7 9x - 10y=14d) 5x-3y = 11 -10x+6y = -22e) x +2y = 8 2x+3y = 12f) x + y = 5 2x+2y = 10g) x - y = 8 3x-3y = 16h) 2x + y-6 = 0 4x-2y- 4 = 0i) 2x-2y - 2 = 0 4x-4y- 5 = 0
Can a perfect square be a term of the sequence 2,5,8,11... How can you realize this?
U wGENERAL SIMPLIFICATIONS273.A platform is to be built for a machine and the size would be 12 metreslong, 6 metres high and 3 decimetres thickness. How many bricks arerequired, if the size of a brick is 24 x 12 x 5 cm ?[I.T.I. July 1984, Motor Mech. Preliminary]
The map shows the location of the airport in a warehouse in a city. Though not displayed on the map, there is also a factory 88 miles due North of the warehouse. A truck travelled from the warehouse to the airport and then to the factory will is the total number of miles to truck traveled?A. 176 miles B. 198 milesC. 220 milesD. 250 milesPls only serious answers
Find the number of terms in the series 2, 5, 8, ... if the last term is 95.
Find the cube root of the following number A:5832 B:85.184 C:42.875with easy process explanation
One is asked to say a two digit number.a) How many two digit numbers are there ? b) What is the probability of getting a number in which one of the digits is 1 ? c) What is the probability of getting a number in which the product of the digits is a prime number ?
5. A cubical tank with edge 2 m is filled withwater. How many buckets of capacity 10 litrecan be filled from the tank?