· 9 years ago · Sep 29, 2016, 09:24 AM
1Oozie YAML Unit test
2
3Whole YAML configuration is didvied into four secion
4Declaration
5Import
6setup
7Initialization
8Job Defination
9HDFS operation
10HCat operation
11Job stubbing
12Usage Guideline
13validation
14output validation
15job validation
16Usage Guideline
17tearUp
18Maven configuration
19Examples
20Bundle Test
21Coordinator test
22Coordinator test with HCat
23Workflow Test
24Workflow Test with skip Job submission
25Whole YAML configuration is didvied into four secion
26
27Declaration
28Import
29Setup
30Initialization
31Job Definition
32Stubbing
33HDFS operation
34HCat operation
35Validation
36tearUp
37Declaration
38
39This section can be used to defined test case properties like test case name, package name, etc.
40
41name: String, required
42Name of the test cases. Java file will be created with the same name.
43package: String, Optional
44The Java package of the generated test class.
45skip_test : boolean, Optional
46A boolean that indicates whether or not to execute this test. This parameter defaults to false.
47copy_dependencies_files_hdfs: List of Strings, Optional
48List of HDFS of paths where all maven dependent jar will be copied. This is useful if for pig/hive jobs.
49Examples:
50
51 package: org.apache.oozie.yamltest
52 name: TestWorkflowActionFromBundle
53 copy_dependencies_files_hdfs:
54 - /tmp/workflow-test/lib
55Import
56
57imports: List of Strings, Optional
58A list of Java classes to import in the generated Oozie unit Java file. * Examples: *
59 import:
60 - org.apache.oozie.util.DateUtils
61 - java.util.Date
62setup
63
64setup: List of Strings, Optional
65List of java statement that can be used for junit setup method.
66setup can also include HDFS and HCat operation
67Initialization
68
69initialization: List of Strings, Optional
70List of java statement that can be used for initialization. This statement will be added first in test case.
71Examples:
72
73initialization:
74 - "Date date = new Date()"
75 - "String currentTime = DateUtils.formatDateOozieTZ(date)"
76Job Defination
77
78This section can be used to define job properties
79
80job_property: String, Optional
81job_property can be a Java Properties file (.properties) or a Hadoop XML Configuration file (.xml).
82properties: String, Optional
83Overridden key/value configuration use in job submission. This value overrides the value of job_property * Examples: *
84job:
85 properties:
86 raw_logs_path: "/tmp/test"
87 aggregated_logs_path: "/user/test"
88 workflowPath: "/testDirPath"
89 start: "2010-02-01T10:00Z"
90job:
91 job_property : src/test/resources/job-1.properties
92 properties:
93 oozie.wf.application.path: "/tmp/workflow-test/"
94 jobTracker: !func getJobTrackerUri()
95 nameNode: !func getNameNodeUri()
96HDFS operation
97
98Set of HDFS operation. Can be used to create dummy input dataset for coordinator or input file for workflow. Supported operation are
99
100mkdir: List, Optional
101The mkdir command creates the specified directory. If the directory already exists, it does a no-op.
102touchz: List, Optional
103Touchz creates a zero length file in the specified path if none exists. If one already exists, then touchz will perform a touch operation.
104delete: List, Optional
105List of file/dir to be deleted.
106delete: List, Optional
107Deletes the specified path.
108chmod: List, Optional
109The chmod command changes the permissions for the specified path.
110permission: String, Optional
111Permission of path. Permissions can be specified using the Unix Symbolic representation (e.g. -rwxrw-rw-) or an octal representation (755)
112path: String, Optional
113HDF path
114copyFromLocal: List, Optional
115List of file to be copied to HDFS
116source: String, mandatory
117destination: String, mandatory
118move: List, Optional
119source: String, mandatory
120destination: String, mandatory
121Examples:
122
123hdfs:
124 touchz:
125 - /tmp/test/2010/02/01/09/_SUCCESS
126 copyFromLocal:
127 - source: coordinator-test-3.xml
128 destination: /tmp/workflow-test/coordinator-test-3.xml
129HCat operation
130
131Set of HCat operation. Can be used to create HCat partition for coordinator/workflow. HCat operation can be configured in nested form. At parent level, at has two operation delete and create
132
133create: Can be used to create HCat partition
134delete: Can be used to delete HCat partition
135Both delete and create have following structure
136 databases:
137 - database: db_test
138 tables:
139 - table: table_test
140 columns: "dt,country"
141 partitions:
142 - dt=20100201;country=usa
143database: List, Optional
144List of databases to be created/deleted
145table
146List of tables to create/delete for the database.
147columns
148Table columns definition
149partitions
150partition to be created.
151Examples:
152
153hcat:
154 create:
155 databases:
156 - database: db_test
157 tables:
158 - table: table_test
159 columns: "dt,country"
160 partitions:
161 - dt=20100201;country=usa
162 - database: db_test_1
163Job stubbing
164
165List of coordinator job, workflow, workflow action, pig script or hive script to be skipped while running test It's a nested configuration.
166
167skip_all_job_submission: boolean, Optional
168All job submission will be skipped
169skip_pig_job_submission: boolean, Optional
170All pig job submission will be skipped
171skip_hive_job_submission: boolean, Optional
172All hive job submission will be skipped
173skip_spark_job_submission: boolean, Optional
174All spark job submission will be skipped
175skip_java_job_submission: boolean, Optional
176All java job submission will be skipped
177skip_shell_job_submission: boolean, Optional
178All shell job submission will be skipped
179skip_distcp_job_submission: boolean, Optional
180All shell job submission will be skipped
181coordinator: List, mandatory
182List of coordinator to be skipped.
183workflow_action:List, Optional
184List of workflow action to be skipped.
185skip_job_submission:boolean, Optional
186Indicate that action should be executed, but only Hadoop job submission is ignored.
187coordinators: List, Optional
188List of coordinators to be skipped
189coordinator: String, Optional
190Name of the coordinator which has to be skipped
191workflow:
192just a marked that workflow of a coordinator will be skipped.
193workflow_actions:
194List of workflow action that will be skipped.
195workflow_action: String, Optional
196name of the workflow action
197skip_job_submission:boolean, Optional
198If true, job submission will be skipped.
199status: String, Optional
200Status of skipped action. It will be useful if somebody wants to test different execution path of workflow. The default value is OK. Supported status are OK and ERROR.
201output: List, Optional
202Copy output to for skipped action
203source: String, Optional
204destination: String, Optional
205Usage Guideline
206
207Skip is a nested configuration, if submitted job is bundle, to skip workflow action one need to configure it in nested way
208skip:
209 coordinators:
210 - coordinator: coord-1
211 - coordinator: coord-2
212 workflow:
213 - coordinator: coord-3
214 workflow:
215 workflow_actions:
216 - workflow_action : pig
217 skip_job_submission: true
218 status: OK
219 output :
220 - path:
221 source: /tmp/resources/action-1-output
222 destination: /tmp/test/output/
223 - path:
224 source: src/test/resources/action-2-output
225 destination: /tmp/test/output/
226Above example will skip coord-1 submission. Coord-2 will be submitted, but there won't be any workflow submission. For coord-3, job submission of action "pig" will be skipped. File /tmp/resources/action-1-output and /tmp/resources/action-2-output will copied to path /tmp/test/output/.
227If submitted job is coordinator below configuration can be used for skipping jobs
228Examples:
229skip:
230 workflow:
231 workflow_actions:
232 - status: OK
233 output:
234 - source: src/test/resources/action-1-output
235 destination: "/tmp/test/output/"
236 - source: src/test/resources/action-2-output
237 destination: "/tmp/test/output/"
238 workflow_action: pig
239 skip_job_submission: true
240 - status: FAILED
241 workflow_action: hive-action
242validation
243
244Validation is dived into two section.
245
246output validation
247job validation
248output validation
249
250TODO
251
252job validation
253
254This section can be used to validate job status and attributes. Job validation syntax is very similar to skip syntax. One need to use nested configuration to validate job.
255
256sleep : integer, Optional
257Thread sleep.
258wait_for_status: integer, Optional
259Testcase will wait for status.
260status: String, Optional
261Assert job status
262not_status: String, Optional
263Assert job status as false
264nominal_time: String, Optional
265Assert coord action nominal time
266coordinators: List, Optional
267List of coordinators for validation
268coordinator: String, Optional
269Name of the coordinator which will be used for validation
270workflow:
271Just a marker that workflow of a coordinator will be skipped.
272workflow_actions:
273List of workflow action that will be skipped.
274workflow_action: String, Optional
275Name of the workflow action which will be used for validation
276status: String, Optional
277Validate job status
278not_status:String, Optional
279Assert job status as false
280job_conf: List, Optional
281key: String, Optional
282expected_value: String, Optional
283Validate job conf.
284Usage Guideline
285
286Like stubbing {{validate_job}} can have nested configuration. If the submitted job is bundle then to validate workflow action, one need to use nested configuration.
287
288validate_job:
289 sleep: 60000
290 status: RUNNING
291 coordinators:
292 - coordinator: coord-3
293 not_status: PREP
294 coordinator_actions:
295 - coordinator_action : "@2"
296 sleep: 6000
297 wait_for_status: RUNNING
298 nominal_time: 2010-02-01T11:00Z
299 workflow:
300 workflow_actions:
301 - workflow_action : pig
302 status: OK
303tearUp
304
305tearUp: List of Strings, Optional
306List of java statement that can be used for junit tearUp method.
307tearUp can also include HDFS and HCat operation.
308Maven configuration
309
310To enable oozie unit yaml, one need add oozie-yaml plugin to maven.
311
312<plugin>
313 <groupId>org.apache.oozie</groupId>
314 <artifactId>oozie-yaml</artifactId>
315 <version>0.1-SNAPSHOT</version>
316 <executions>
317 <execution>
318 <goals>
319 <goal>oozieyaml</goal>
320 </goals>
321 </execution>
322 </executions>
323</plugin>
324oozie-yaml plugin to maven will search for any oozie yaml file and if it's there it will convert it into respective jnit testcase and will exeute. There will be surefire report.
325
326Examples
327
328Bundle Test
329
330name: TestBundleDisableCoord
331output_path: ./target/oozieyaml-sourcesjob:
332 properties:
333 raw_logs_path: "/tmp/test"
334 aggregated_logs_path: "/user/test"
335 workflowPath: "/testDirPath"
336 start: "2010-02-01T10:00Z"
337 end: "2012-02-03T23:59Z"
338 oozie.bundle.application.path: "/tmp/workflow-test/bundle-multiple-coord.xml"
339 queueName: "defult"
340 coord_path_1: "/tmp/workflow-test/coordinator-test-1.xml"
341 coord_path_2: "/tmp/workflow-test/coordinator-test-2.xml"
342 jobTracker: !func getJobTrackerUri()
343 nameNode: !func getNameNodeUri()
344hdfs:
345 touchz:
346 - /tmp/test/2010/02/01/09/_SUCCESS
347 - /tmp/test/2010/02/01/10/_SUCCESS
348 - /tmp/test/2010/02/01/11/_SUCCESS
349 - /tmp/test/2010/02/01/12/_SUCCESS
350 copyFromLocal:
351 - source: bundle-multiple-coord.xml
352 destination: "/tmp/workflow-test/bundle-multiple-coord.xml"
353 - source: coordinator-test-1.xml
354 destination: /tmp/workflow-test/coordinator-test-1.xml
355 - source: coordinator-test-2.xml
356skip:
357 coordinators:
358 - coordinator: coord_1
359validations:
360 validate_job:
361 sleep: 6000
362 status: RUNNING
363 coordinators:
364 - coordinator: coord_2
365 not_status: PREP
366 coordinator_actions:
367 - coordinator_action : "@2"
368 not_status: WAITING
369 nominal_time: 2010-02-01T11:00Z
370Coordinator test
371
372name: TestCoordinator
373output_path: ./target/oozieyaml-sourcesjob:
374 properties:
375 raw_logs_path: "/tmp/test"
376 aggregated_logs_path: "/user/test"
377 workflowPath: "/testDirPath"
378 start: "2010-02-01T10:00Z"
379 end: "2012-02-03T23:59Z"
380 oozie.coord.application.path: "/tmp/workflow-test/coordinator.xml"
381 queueName: "defult"
382 jobTracker: !func getJobTrackerUri()
383 nameNode: !func getNameNodeUri()
384hdfs:
385 touchz:
386 - /tmp/test/2010/02/01/09/_SUCCESS
387 - /tmp/test/2010/02/01/10/_SUCCESS
388 - /tmp/test/2010/02/01/11/_SUCCESS
389 - /tmp/test/2010/02/01/12/_SUCCESS
390 copyFromLocal:
391 - source: coordinator-test.xml
392 destination: "/tmp/workflow-test/coordinator.xml"
393validations:
394 validate_job:
395 sleep: 6000
396 job_conf:
397 - key: queueName
398 expected_value: defult
399 coordinator_actions:
400 - coordinator_action : "@2"
401 not_status: WAITING
402 nominal_time: 2010-02-01T11:00Z
403Coordinator test with HCat
404
405name: TestCoordinatorWithHcat
406output_path: ./target/oozieyaml-sourcesjob:
407 properties:
408 raw_logs_path: "/tmp/test"
409 aggregated_logs_path: "/user/test"
410 workflowAppUri: "/testDirPath"
411 start: "2010-02-01T10:00Z"
412 end: "2012-02-03T23:59Z"
413 oozie.coord.application.path: "/tmp/workflow-test/coordinator.xml"
414 queueName: "defult"
415 db: db_test
416 table: table_test
417 dataOut: dataOut
418 jobTracker: !func getJobTrackerUri()
419 nameNode: !func getNameNodeUri()
420 hcatNode: !func \"hcat://\" + getMetastoreAuthority()
421hdfs:
422 mkdir:
423 - "/tmp/workflow-test/"
424 copyFromLocal:
425 - source: coordinator-hcat-test.xml
426 destination: "/tmp/workflow-test/coordinator.xml"
427hcat:
428 create:
429 databases:
430 - database: db_test
431 tables:
432 - table: table_test
433 columns: "dt,country"
434 partitions:
435 - dt=20100201;country=usa
436validations:
437 validate_job:
438 sleep: 6000
439 coordinator_actions:
440 - coordinator_action : "@1"
441 not_status: WAITING
442 nominal_time: 2010-02-01T10:00Z
443Workflow Test
444
445name: TestWorkflowWithPig
446output_path: ./target/oozieyaml-sources
447copy_dependencies_files_hdfs:
448 - /tmp/workflow-test/libjob:
449 properties:
450 INPUT: "/tmp/input/dummp.txt"
451 oozie.wf.application.path: "/tmp/workflow-test/"
452 queueName: "defult"
453 OUTPUT: "/tmp/output"
454 jobTracker: !func getJobTrackerUri()
455 nameNode: !func getNameNodeUri()
456hdfs:
457 mkdir:
458 - "/tmp/workflow-test/"
459 touchz:
460 - /tmp/input/dummp.txt
461 copyFromLocal:
462 - source: echo.pig
463 destination: "/tmp/workflow-test/echo.pig"
464 - source: workflow-test-with-pig.xml
465 destination: "/tmp/workflow-test/workflow.xml"
466validations:
467 validate_job:
468 sleep: 60000
469 status: SUCCEEDED
470 workflow_actions:
471 - workflow_action : pig
472 status: OK
473Workflow Test with skip Job submission
474
475name: TestWorkflowJobStatus
476output_path: ./target/oozieyaml-sources
477copy_dependencies_files_hdfs:
478 - /tmp/workflow-test/libjob:
479 properties:
480 INPUT: "/tmp/input/dummp.txt"
481 oozie.wf.application.path: "/tmp/workflow-test/"
482 queueName: "defult"
483 OUTPUT: "/tmp/output"
484 jobTracker: !func getJobTrackerUri()
485 nameNode: !func getNameNodeUri()
486hdfs:
487 mkdir:
488 - "/tmp/workflow-test/"
489 touchz:
490 - /tmp/input/dummp.txt
491 copyFromLocal:
492 - source: echo.pig
493 destination: "/tmp/workflow-test/echo.pig"
494 - source: workflow-test-with-output.xml
495 destination: "/tmp/workflow-test/workflow.xml"
496skip:
497 workflow_actions:
498 - workflow_action : pig
499 skip_job_submission: true
500 status: OK
501 output:
502 - source: /tmp/workflow-test/echo.pig
503 destination: "/tmp/workflow-test/dummy.pig"
504 - workflow_action : pig_2
505 skip_job_submission: true
506 status: ERROR
507validations:
508 validate_job:
509 wait_for_status: KILLED
510 workflow_actions:
511 - workflow_action : pig
512 status: OK
513 - workflow_action : pig_1
514 status: OK
515 - workflow_action : pig_2
516 status: ERROR