· 9 years ago · Dec 08, 2016, 09:42 PM
1$ gcloud dataproc jobs submit hadoop --cluster hadoop1 --jar target/wordcount-mapreduce-0-SNAPSHOT-jar-with-dependencies.jar wordcount-hbase /input blah
2WARNING: The '--' argument must be specified between gcloud specific args on the left and JOB_ARGS on the right. IMPORTANT: previously, commands allowed the omission of the --, and unparsed arguments were treated as implementation args. This usage is being deprecated and will be removed in March 2017.
3This will be strictly enforced in March 2017. Use 'gcloud beta dataproc jobs submit hadoop' to see new behavior.
4Using 'wordcount-hbase /input blah' for JOB_ARGS.
5Copying file://target/wordcount-mapreduce-0-SNAPSHOT-jar-with-dependencies.jar [Content-Type=application/java-archive]...
6| [1 files][ 60.8 MiB/ 60.8 MiB] 315.2 KiB/s
7Operation completed over 1 objects/60.8 MiB.
8Job [11b3a31a-4af9-415b-bcbc-fecf66ae68b6] submitted.
9Waiting for job output...
10SLF4J: Class path contains multiple SLF4J bindings.
11SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
12SLF4J: Found binding in [jar:file:/tmp/11b3a31a-4af9-415b-bcbc-fecf66ae68b6/wordcount-mapreduce-0-SNAPSHOT-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
13SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
14SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
1516/12/08 19:24:18 INFO gcs.GoogleHadoopFileSystemBase: GHFS version: 1.5.5-hadoop2
1616/12/08 19:24:18 INFO sample.CreateTable: Creating Table blah
1716/12/08 19:24:18 INFO grpc.BigtableSession: Opening connection for projectId dabble-151614, instanceId hadoop1, on data host bigtable.googleapis.com, table admin host bigtableadmin.googleapis.com.
1816/12/08 19:24:18 INFO grpc.BigtableSession: Bigtable options: BigtableOptions{dataHost=bigtable.googleapis.com, tableAdminHost=bigtableadmin.googleapis.com, instanceAdminHost=bigtableadmin.googleapis.com, projectId=dabble-151614, instanceId=hadoop1, userAgent=hbase-1.2.1, credentialType=DefaultCredentials, port=443, dataChannelCount=10, retryOptions=RetryOptions{retriesEnabled=true, allowRetriesWithoutTimestamp=false, statusToRetryOn=[DEADLINE_EXCEEDED, INTERNAL, ABORTED, UNAVAILABLE], initialBackoffMillis=5, maxElaspedBackoffMillis=60000, backoffMultiplier=2.0, streamingBufferSize=60, readPartialRowTimeoutMillis=60000, maxScanTimeoutRetries=3}, bulkOptions=BulkOptions{asyncMutatorCount=2, useBulkApi=true, bulkMaxKeyCount=25, bulkMaxRequestSize=1048576, maxInflightRpcs=500, maxMemory=351220531}, callOptionsConfig=CallOptionsConfig{useTimeout=false, shortRpcTimeoutMs=60000, longRpcTimeoutMs=60000}, usePlaintextNegotiation=false}.
1916/12/08 19:24:18 INFO grpc.BigtableSession: SslContext: gRPC is using the OpenSSL provider (tcnactive jar - Open Ssl version: BoringSSL)
2016/12/08 19:24:19 INFO io.RefreshingOAuth2CredentialsInterceptor: Refreshing the OAuth token
21Dec 08, 2016 7:24:19 PM com.google.bigtable.repackaged.io.grpc.internal.ManagedChannelImpl <init>
22INFO: [ManagedChannelImpl@24d4a409] Created with target directaddress:///bigtable.googleapis.com/74.125.202.95:443
23Dec 08, 2016 7:24:19 PM com.google.bigtable.repackaged.io.grpc.internal.ManagedChannelImpl <init>
24INFO: [ManagedChannelImpl@3bb5b4a0] Created with target directaddress:///bigtable.googleapis.com/74.125.202.95:443
25Dec 08, 2016 7:24:19 PM com.google.bigtable.repackaged.io.grpc.internal.ManagedChannelImpl <init>
26INFO: [ManagedChannelImpl@4099eb8a] Created with target directaddress:///bigtable.googleapis.com/74.125.202.95:443
27Dec 08, 2016 7:24:19 PM com.google.bigtable.repackaged.io.grpc.internal.ManagedChannelImpl <init>
28INFO: [ManagedChannelImpl@26ce4cb] Created with target directaddress:///bigtable.googleapis.com/74.125.202.95:443
29Dec 08, 2016 7:24:19 PM com.google.bigtable.repackaged.io.grpc.internal.ManagedChannelImpl <init>
30INFO: [ManagedChannelImpl@57b414f0] Created with target directaddress:///bigtable.googleapis.com/74.125.202.95:443
31Dec 08, 2016 7:24:19 PM com.google.bigtable.repackaged.io.grpc.internal.ManagedChannelImpl <init>
32INFO: [ManagedChannelImpl@4132ba12] Created with target directaddress:///bigtable.googleapis.com/74.125.202.95:443
33Dec 08, 2016 7:24:19 PM com.google.bigtable.repackaged.io.grpc.internal.ManagedChannelImpl <init>
34INFO: [ManagedChannelImpl@4ef8dc05] Created with target directaddress:///bigtable.googleapis.com/74.125.202.95:443
35Dec 08, 2016 7:24:19 PM com.google.bigtable.repackaged.io.grpc.internal.ManagedChannelImpl <init>
36INFO: [ManagedChannelImpl@2bd733ad] Created with target directaddress:///bigtable.googleapis.com/74.125.202.95:443
37Dec 08, 2016 7:24:19 PM com.google.bigtable.repackaged.io.grpc.internal.ManagedChannelImpl <init>
38INFO: [ManagedChannelImpl@5c33a1b3] Created with target directaddress:///bigtable.googleapis.com/74.125.202.95:443
39Dec 08, 2016 7:24:19 PM com.google.bigtable.repackaged.io.grpc.internal.ManagedChannelImpl <init>
40INFO: [ManagedChannelImpl@885620d] Created with target directaddress:///bigtable.googleapis.com/74.125.202.95:443
41Dec 08, 2016 7:24:19 PM com.google.bigtable.repackaged.io.grpc.internal.ManagedChannelImpl <init>
42INFO: [ManagedChannelImpl@7812acea] Created with target directaddress:///bigtableadmin.googleapis.com/74.125.201.95:443
4316/12/08 19:24:19 ERROR sample.CreateTable: Could not figure out if table blah exists.
44java.io.IOException: Failed to listTables
45 at org.apache.hadoop.hbase.client.AbstractBigtableAdmin.requestTableList(AbstractBigtableAdmin.java:258)
46 at org.apache.hadoop.hbase.client.AbstractBigtableAdmin.listTableNames(AbstractBigtableAdmin.java:245)
47 at org.apache.hadoop.hbase.client.AbstractBigtableAdmin.listTableNames(AbstractBigtableAdmin.java:206)
48 at org.apache.hadoop.hbase.client.AbstractBigtableAdmin.listTableNames(AbstractBigtableAdmin.java:198)
49 at org.apache.hadoop.hbase.client.AbstractBigtableAdmin.tableExists(AbstractBigtableAdmin.java:140)
50 at com.example.bigtable.sample.CreateTable.tableExists(CreateTable.java:89)
51 at com.example.bigtable.sample.CreateTable.createTable(CreateTable.java:64)
52 at com.example.bigtable.sample.WordCountHBase.main(WordCountHBase.java:107)
53 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
54 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
55 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
56 at java.lang.reflect.Method.invoke(Method.java:498)
57 at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
58 at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
59 at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:152)
60 at com.example.bigtable.sample.WordCountDriver.main(WordCountDriver.java:36)
61 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
62 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
63 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
64 at java.lang.reflect.Method.invoke(Method.java:498)
65 at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
66 at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
67 at com.google.cloud.hadoop.services.agent.job.shim.HadoopRunJarShim.main(HadoopRunJarShim.java:12)
68Caused by: com.google.bigtable.repackaged.io.grpc.StatusRuntimeException: NOT_FOUND: Failed to read: (475580530846,hadoop1)
69 at com.google.bigtable.repackaged.io.grpc.Status.asRuntimeException(Status.java:503)
70 at com.google.bigtable.repackaged.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:207)
71 at com.google.bigtable.repackaged.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:140)
72 at com.google.bigtable.admin.v2.BigtableTableAdminGrpc$BigtableTableAdminBlockingStub.listTables(BigtableTableAdminGrpc.java:425)
73 at com.google.cloud.bigtable.grpc.BigtableTableAdminGrpcClient.listTables(BigtableTableAdminGrpcClient.java:52)
74 at org.apache.hadoop.hbase.client.AbstractBigtableAdmin.requestTableList(AbstractBigtableAdmin.java:256)
75 ... 22 more
7616/12/08 19:24:20 ERROR sample.CreateTable: Could not create table blah
77java.io.IOException: Failed to create table 'blah'
78 at org.apache.hadoop.hbase.client.AbstractBigtableAdmin.createTable(AbstractBigtableAdmin.java:365)
79 at org.apache.hadoop.hbase.client.AbstractBigtableAdmin.createTable(AbstractBigtableAdmin.java:322)
80 at com.example.bigtable.sample.CreateTable.createTable(CreateTable.java:73)
81 at com.example.bigtable.sample.WordCountHBase.main(WordCountHBase.java:107)
82 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
83 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
84 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
85 at java.lang.reflect.Method.invoke(Method.java:498)
86 at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
87 at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
88 at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:152)
89 at com.example.bigtable.sample.WordCountDriver.main(WordCountDriver.java:36)
90 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
91 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
92 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
93 at java.lang.reflect.Method.invoke(Method.java:498)
94 at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
95 at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
96 at com.google.cloud.hadoop.services.agent.job.shim.HadoopRunJarShim.main(HadoopRunJarShim.java:12)
97Caused by: com.google.bigtable.repackaged.io.grpc.StatusRuntimeException: NOT_FOUND: Failed to read: (475580530846,hadoop1)
98 at com.google.bigtable.repackaged.io.grpc.Status.asRuntimeException(Status.java:503)
99 at com.google.bigtable.repackaged.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:207)
100 at com.google.bigtable.repackaged.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:140)
101 at com.google.bigtable.admin.v2.BigtableTableAdminGrpc$BigtableTableAdminBlockingStub.createTable(BigtableTableAdminGrpc.java:419)
102 at com.google.cloud.bigtable.grpc.BigtableTableAdminGrpcClient.createTable(BigtableTableAdminGrpcClient.java:64)
103 at org.apache.hadoop.hbase.client.AbstractBigtableAdmin.createTable(AbstractBigtableAdmin.java:358)
104 ... 18 more
105Dec 08, 2016 7:24:20 PM com.google.bigtable.repackaged.io.grpc.internal.ManagedChannelImpl maybeTerminateChannel
106INFO: [ManagedChannelImpl@24d4a409] Terminated
107Dec 08, 2016 7:24:20 PM com.google.bigtable.repackaged.io.grpc.internal.ManagedChannelImpl maybeTerminateChannel
108INFO: [ManagedChannelImpl@3bb5b4a0] Terminated
109Dec 08, 2016 7:24:20 PM com.google.bigtable.repackaged.io.grpc.internal.ManagedChannelImpl maybeTerminateChannel
110INFO: [ManagedChannelImpl@4099eb8a] Terminated
111Dec 08, 2016 7:24:20 PM com.google.bigtable.repackaged.io.grpc.internal.ManagedChannelImpl maybeTerminateChannel
112INFO: [ManagedChannelImpl@26ce4cb] Terminated
113Dec 08, 2016 7:24:20 PM com.google.bigtable.repackaged.io.grpc.internal.ManagedChannelImpl maybeTerminateChannel
114INFO: [ManagedChannelImpl@57b414f0] Terminated
115Dec 08, 2016 7:24:20 PM com.google.bigtable.repackaged.io.grpc.internal.ManagedChannelImpl maybeTerminateChannel
116INFO: [ManagedChannelImpl@4132ba12] Terminated
117Dec 08, 2016 7:24:20 PM com.google.bigtable.repackaged.io.grpc.internal.ManagedChannelImpl maybeTerminateChannel
118INFO: [ManagedChannelImpl@4ef8dc05] Terminated
119Dec 08, 2016 7:24:20 PM com.google.bigtable.repackaged.io.grpc.internal.ManagedChannelImpl maybeTerminateChannel
120INFO: [ManagedChannelImpl@2bd733ad] Terminated
121Dec 08, 2016 7:24:20 PM com.google.bigtable.repackaged.io.grpc.internal.ManagedChannelImpl maybeTerminateChannel
122INFO: [ManagedChannelImpl@5c33a1b3] Terminated
123Dec 08, 2016 7:24:20 PM com.google.bigtable.repackaged.io.grpc.internal.ManagedChannelImpl maybeTerminateChannel
124INFO: [ManagedChannelImpl@885620d] Terminated
125Dec 08, 2016 7:24:20 PM com.google.bigtable.repackaged.io.grpc.internal.ManagedChannelImpl maybeTerminateChannel
126INFO: [ManagedChannelImpl@7812acea] Terminated
12716/12/08 19:24:20 INFO client.RMProxy: Connecting to ResourceManager at hadoop1-m/10.128.0.2:8032
12816/12/08 19:24:21 INFO input.FileInputFormat: Total input paths to process : 1
12916/12/08 19:24:21 INFO mapreduce.JobSubmitter: number of splits:1
13016/12/08 19:24:21 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1481220311366_0006
13116/12/08 19:24:22 INFO impl.YarnClientImpl: Submitted application application_1481220311366_0006
13216/12/08 19:24:22 INFO mapreduce.Job: The url to track the job: http://hadoop1-m:8088/proxy/application_1481220311366_0006/
13316/12/08 19:24:22 INFO mapreduce.Job: Running job: job_1481220311366_0006
13416/12/08 19:24:29 INFO mapreduce.Job: Job job_1481220311366_0006 running in uber mode : false
13516/12/08 19:24:29 INFO mapreduce.Job: map 0% reduce 0%
13616/12/08 19:24:33 INFO mapreduce.Job: map 100% reduce 0%
13716/12/08 19:24:44 INFO mapreduce.Job: map 100% reduce 25%
13816/12/08 19:24:45 INFO mapreduce.Job: map 100% reduce 38%
13916/12/08 19:25:49 INFO mapreduce.Job: Task Id : attempt_1481220311366_0006_r_000000_0, Status : FAILED
140Error: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 17 actions: BigtableRetriesExhaustedException: 17 times,
141 at com.google.cloud.bigtable.hbase.BigtableBufferedMutator.handleExceptions(BigtableBufferedMutator.java:415)
142 at com.google.cloud.bigtable.hbase.BigtableBufferedMutator.flush(BigtableBufferedMutator.java:236)
143 at com.google.cloud.bigtable.hbase.BigtableBufferedMutator.close(BigtableBufferedMutator.java:212)
144 at org.apache.hadoop.hbase.client.AbstractBigtableConnection$2.close(AbstractBigtableConnection.java:181)
145 at org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.close(TableOutputFormat.java:111)
146 at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.close(ReduceTask.java:550)
147 at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:629)
148 at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
149 at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
150 at java.security.AccessController.doPrivileged(Native Method)
151 at javax.security.auth.Subject.doAs(Subject.java:422)
152 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
153 at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
154
155Container killed by the ApplicationMaster.
156Container killed on request. Exit code is 143
157Container exited with a non-zero exit code 143
158
15916/12/08 19:25:50 INFO mapreduce.Job: map 100% reduce 25%
16016/12/08 19:25:58 INFO mapreduce.Job: map 100% reduce 38%
16116/12/08 19:26:04 INFO mapreduce.Job: Task Id : attempt_1481220311366_0006_r_000002_0, Status : FAILED
162Error: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 13 actions: BigtableRetriesExhaustedException: 13 times,
163 at com.google.cloud.bigtable.hbase.BigtableBufferedMutator.handleExceptions(BigtableBufferedMutator.java:415)
164 at com.google.cloud.bigtable.hbase.BigtableBufferedMutator.flush(BigtableBufferedMutator.java:236)
165 at com.google.cloud.bigtable.hbase.BigtableBufferedMutator.close(BigtableBufferedMutator.java:212)
166 at org.apache.hadoop.hbase.client.AbstractBigtableConnection$2.close(AbstractBigtableConnection.java:181)
167 at org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.close(TableOutputFormat.java:111)
168 at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.close(ReduceTask.java:550)
169 at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:629)
170 at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
171 at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
172 at java.security.AccessController.doPrivileged(Native Method)
173 at javax.security.auth.Subject.doAs(Subject.java:422)
174 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
175 at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
176
177Container killed by the ApplicationMaster.
178Container killed on request. Exit code is 143
179Container exited with a non-zero exit code 143
180
18116/12/08 19:26:05 INFO mapreduce.Job: map 100% reduce 25%
18216/12/08 19:26:13 INFO mapreduce.Job: map 100% reduce 38%
18316/12/08 19:26:14 INFO mapreduce.Job: Task Id : attempt_1481220311366_0006_r_000001_0, Status : FAILED
184Error: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 22 actions: BigtableRetriesExhaustedException: 22 times,
185 at com.google.cloud.bigtable.hbase.BigtableBufferedMutator.handleExceptions(BigtableBufferedMutator.java:415)
186 at com.google.cloud.bigtable.hbase.BigtableBufferedMutator.flush(BigtableBufferedMutator.java:236)
187 at com.google.cloud.bigtable.hbase.BigtableBufferedMutator.close(BigtableBufferedMutator.java:212)
188 at org.apache.hadoop.hbase.client.AbstractBigtableConnection$2.close(AbstractBigtableConnection.java:181)
189 at org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.close(TableOutputFormat.java:111)
190 at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.close(ReduceTask.java:550)
191 at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:629)
192 at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
193 at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
194 at java.security.AccessController.doPrivileged(Native Method)
195 at javax.security.auth.Subject.doAs(Subject.java:422)
196 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
197 at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
198
199Container killed by the ApplicationMaster.
200Container killed on request. Exit code is 143
201Container exited with a non-zero exit code 143
202
20316/12/08 19:26:15 INFO mapreduce.Job: map 100% reduce 25%
20416/12/08 19:26:23 INFO mapreduce.Job: map 100% reduce 38%