-
[OOZIE] workflow의 job의 status 확인 본문
서비스에서 사용하고 있는 데이터 중 일부는 sqoop을 oozie의 workflow에 등록해서 데이터를 생성하고 있었다.
그러던 중 해당 job이 정상으로 처리되었는지 확인 할 수 있는 flag를 넣어 줄수 있느냐는 요청을 받았다.
물론 각 job이 끝난 뒤에 fs를 통해서 flag를 만들수는 있었으나, 이미 사용하고 있는 job들이 너무 많아서 그렇게 하기는 좀 비효율적이여서,
다른 방법을 찾아보기로 했다.
1> sqoop이 완료되면 자동으로 flag를 남기는 기능이 있을까?
2> oozie에서 해당 job이 끝나면 flag를 남기게 할수 있지 않을까?
이러저리 방법을 찾아봤는데, 1>의 방법은 지원이 되는게 없었고, 2>번의 경우엔 해당 job이 끝나고, 다음 job이 시작될때,
준비에서 mkdir로 폴더를 만들어 줄수는 있지만,, 몬가,, 아닌듯 해서 다른 방법을 찾아보기로 했다.
이러저리 찾아보다 보니, oozie cli를 통해서 workflow의 status를 확인 할 수 있었다.
문제는 실행시 마다 변경되는 job id를 확인하는게 문제였다.
그래서 먼저 변하지 않는 workflow명으로 검색을 한 후에 해당 id를 획득하고 해당 id로 status를 확인하는 것으로 하였다.
1> workflow명으로 job id 확인
$oozie jobs -oozie http://myoozie:11000/oozie -filter name="myworkflow"
Job ID App Name Status User Group Started Ended
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W myworkflow SUCCEEDED xxx - 2019-10-16 15:05 GMT 2019-10-16 15:49 GMT
------------------------------------------------------------------------------------------------------------------------------------
0005183-180828161051398-oozie-oozi-W myworkflow SUCCEEDED xxx - 2019-10-15 15:05 GMT 2019-10-15 15:48 GMT
------------------------------------------------------------------------------------------------------------------------------------
0005173-180828161051398-oozie-oozi-W myworkflow SUCCEEDED xxx - 2019-10-14 15:05 GMT 2019-10-14 15:54 GMT
------------------------------------------------------------------------------------------------------------------------------------
0005163-180828161051398-oozie-oozi-W myworkflow SUCCEEDED xxx - 2019-10-13 15:05 GMT 2019-10-13 15:48 GMT
------------------------------------------------------------------------------------------------------------------------------------
0005153-180828161051398-oozie-oozi-W myworkflow SUCCEEDED xxx - 2019-10-12 15:05 GMT 2019-10-12 15:48 GMT
------------------------------------------------------------------------------------------------------------------------------------
0005143-180828161051398-oozie-oozi-W myworkflow SUCCEEDED xxx - 2019-10-11 15:05 GMT 2019-10-11 15:48 GMT
------------------------------------------------------------------------------------------------------------------------------------
0005133-180828161051398-oozie-oozi-W myworkflow SUCCEEDED xxx - 2019-10-10 15:05 GMT 2019-10-10 15:49 GMT
------------------------------------------------------------------------------------------------------------------------------------
0005123-180828161051398-oozie-oozi-W myworkflow SUCCEEDED xxx - 2019-10-09 15:05 GMT 2019-10-09 15:49 GMT
------------------------------------------------------------------------------------------------------------------------------------2> 확인한 job id로 상세 정보 확인
oozie job -oozie http://myoozie:11000/oozie -info 0005193-180828161051398-oozie-oozi-W
Job ID : 0005193-180828161051398-oozie-oozi-W
------------------------------------------------------------------------------------------------------------------------------------
Workflow Name : myworkflow
App Path : hdfs://myhadoop:8020/user/hue/oozie/workspaces/hue-oozie-1524706449.21
Status : SUCCEEDED
Run : 0
User : hdfs
Group : -
Created : 2019-10-16 15:05 GMT
Started : 2019-10-16 15:05 GMT
Last Modified : 2019-10-16 15:49 GMT
Ended : 2019-10-16 15:49 GMT
CoordAction ID: 0004873-180828161051398-oozie-oozi-C@31
Actions
------------------------------------------------------------------------------------------------------------------------------------
ID Status Ext ID Ext Status Err Code
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@:start: OK - OK -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@shell-70a3 OK job_1535440204589_412898SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@fork-ba27 OK - OK -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@fork-0fb2 OK - OK -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@fork-3e16 OK - OK -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@fork-3729 OK - OK -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-9572 OK job_1535440204589_412902SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-b99e OK job_1535440204589_412903SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-9684 OK job_1535440204589_412904SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-f31d OK job_1535440204589_412905SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-5510 OK job_1535440204589_412907SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-3ed2 OK job_1535440204589_412906SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-a734 OK job_1535440204589_412919SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-bf22 OK job_1535440204589_412927SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-023b OK job_1535440204589_412929SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-f00a OK job_1535440204589_412930SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-d7b9 OK job_1535440204589_412933SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-8578 OK job_1535440204589_412934SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-a7b5 OK job_1535440204589_412937SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@join-b0a3 OK - OK -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@fork-15ca OK - OK -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-0d8c OK job_1535440204589_412938SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-f5e2 OK job_1535440204589_412939SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-f38f OK job_1535440204589_412941SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-78a8 OK job_1535440204589_412945SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@join-4b4d OK - OK -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-965d OK job_1535440204589_412946SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-4f04 OK job_1535440204589_412947SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@fork-d638 OK - OK -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-379a OK job_1535440204589_412952SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-a5b5 OK job_1535440204589_412951SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-99eb OK job_1535440204589_412955SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-6f19 OK job_1535440204589_412956SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@join-1197 OK - OK -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@join-b656 OK - OK -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-7aa3 OK job_1535440204589_412962SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-1bb4 OK job_1535440204589_412979SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-620d OK job_1535440204589_412985SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-265b OK job_1535440204589_413005SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-fdc0 OK job_1535440204589_413007SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@join-4062 OK - OK -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@fork-3912 OK - OK -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-425c OK job_1535440204589_413009SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@sqoop-b3e7 OK job_1535440204589_413010SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@join-b673 OK - OK -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@subworkflow-512a OK 0005195-180828161051398-oozie-oozi-WSUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@join-4d49 OK - OK -
------------------------------------------------------------------------------------------------------------------------------------
0005193-180828161051398-oozie-oozi-W@End OK - OK -
------------------------------------------------------------------------------------------------------------------------------------
위와 같이 해당 workflow의 각각의 처리된 정보를 얻을 수 있다.
oozie cli
https://oozie.apache.org/docs/3.3.2/DG_CommandLineTool.html#Common_CLI_Options
'HADOOP' 카테고리의 다른 글
| hadoop client 설치 (0) | 2019.12.18 |
|---|---|
| [impala-shell]python3에서 SyntaxError 처리 (0) | 2019.10.01 |
| [sqoop] CLOB/BLOB column data import (0) | 2019.09.09 |
| [CDH] "Java 프로세스에서 검사한 이 호스트의 호스트 이름과 정식 이름이 동일하지 않습니다." (0) | 2019.09.09 |