· 8 years ago · Jun 28, 2018, 01:14 PM
1#### ENV
2```
3HDP263
4```
5#### Exception
6##### Client
7```
8ERROR [Thread-123]: compactor.Worker (Worker.java:run(191)) - Caught exception while trying to compact id:123,dbname:hive_acid,tableName:hive_acid_table,partName:hive_acid_part=part_name,state:^@,type:MAJOR,properties:null,runAs:null,tooManyAborts:false,highestTxnId:0. Marking failed to avoid repeated failures, java.io.IOException: Minor compactor job failed for Hadoop JobId:job_XXXXXX_XXXX at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.launchCompactionJob(CompactorMR.java:314)
9 at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:269)
10 at org.apache.hadoop.hive.ql.txn.compactor.Worker$1.run(Worker.java:175)
11 at java.security.AccessController.doPrivileged(Native Method)
12 at javax.security.auth.Subject.doAs(Subject.java:422)
13 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
14 at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:172)
15
16 ```
17 #### Exception in mapreduce job
18 ```
19 FATAL [IPC Server handler 11 on 12345] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_XXXXX - exited : org.apache.hadoop.fs.FileAlreadyExistsException: XXXXXXXXXXXXXXXXXX/base_00000XX/bucket_00000 for client already exists
20 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2811)
21 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2698)
22 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2582)
23 at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:736)
24 at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:409)
25 at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
26 at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
27 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
28 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
29 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
30 at java.security.AccessController.doPrivileged(Native Method)
31 at javax.security.auth.Subject.doAs(Subject.java:422)
32 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
33 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)
34
35 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
36 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
37 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
38 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
39 at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
40```
41
42#### Reason
43```
44during compaction hive trigger mapreduce job which create a base or delta file at TMP_LOCATION depending on the what compaction it is running.
45ORC writing is memory intensive operation and situation become worse if you writing wide ORC table(too many columns). writing wide ORC require more
46memory and small yarn container size are not good enough. In above scenario user is running mapreduce job with 2G container size which was not enough, as
47memory start growing and reached beyond 2G of container, yarn physical memory checker kills the container which does not give the chance to container to
48clean up TMP_LOCATION and subsequent task start failing with FileAlreadyExistsException.
49```
50
51#### Resoultion
52```
53try running the compaction with big yarn container size.
54ALTER TABLE TABLENAME partition (PART_NAME='PART_VALUE') COMPACT 'MINOR' WITH OVERWRITE TBLPROPERTIES ('compactor.mapreduce.map.memory.mb'='4096')
55```
56
57#### Bonus
58```
59// run compaction job in debug mode
60ALTER TABLE TABLENAME partition (PART_NAME='PART_VALUE') COMPACT 'MINOR' WITH OVERWRITE TBLPROPERTIES ("compactor.mapreduce.map.log.level"="DEBUG","compactor.yarn.app.mapreduce.am.log.level"="DEBUG");
61```