Learn With Me

Issue:

2020-01-10 07:44:36,595 - call returned (0, '')
2020-01-10 07:44:36,597 - NameNode HA states: active_namenodes = [(u'nn1', 'localhost1:50470')], standby_namenodes = [(u'nn2', 'localhost2:50470')], unknown_namenodes = []
2020-01-10 07:44:36,599 - call['ambari-sudo.sh su oozie -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET -k '"'"'https://localhost1:
50470/webhdfs/v1/user/oozie/share?op=GETFILESTATUS&user.name=oozie'"'"' 1>/tmp/tmpJMhRbc 2>/tmp/tmpih8zQr''] {'logoutput': None, 'quiet': False}
2020-01-10 07:44:36,801 - call returned (0, '')
2020-01-10 07:44:36,802 - Execute['cd /data/var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozie-start.sh'] {'environment': {'OOZIE_CONFIG': u'/usr/hdp/current/o
ozie-server/conf'}, 'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/
null 2>&1'", 'user': 'oozie'}
2020-01-10 07:44:38,652 - call['source /usr/hdp/current/oozie-server/conf/oozie-env.sh ; oozie admin -oozie http://localhost1:11000/oozie -shareliblist hive | grep "\[Available ShareLib\]" -A 5'] {'logoutput': True, 'tries': 10, 'user': 'oozie', 'try_sleep': 5}
Connection exception has occurred [ java.net.ConnectException Connection refused (Connection refused) ]. Trying after 1 sec. Retry count = 1
Connection exception has occurred [ java.net.ConnectException Connection refused (Connection refused) ]. Trying after 2 sec. Retry count = 2
Connection exception has occurred [ java.net.ConnectException Connection refused (Connection refused) ]. Trying after 4 sec. Retry count = 3
Connection exception has occurred [ java.net.ConnectException Connection refused (Connection refused) ]. Trying after 8 sec. Retry count = 4
Connection exception has occurred [ java.net.ConnectException Connection refused (Connection refused) ]. Trying after 16 sec. Retry count = 5
Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 5. Exception = Connection refused (Connection refused)

Findings in logs

1. Oozie logs:

++
2020-01-10 07:44:41,091 FATAL Services:514 - SERVER[localhost1] java.io.IOException: No such file or directory
java.io.IOException: No such file or directory
++

2. Oozie-error.log:

++
2020-01-10 07:44:41,091 FATAL Services:514 - SERVER[localhost1] java.io.IOException: No such file or directory
java.io.IOException: No such file or directory
at java.io.UnixFileSystem.createFileExclusively(Native Method)
++

3. Cataline.log

++
ERROR: Oozie could not be started

REASON: org.apache.oozie.service.ServiceException: E0001: Could not create runtime directory,

Stacktrace:
--------------------------------------------------------------
org.apache.oozie.service.ServiceException: E0001: Could not create runtime directory,
at org.apache.oozie.service.Services.createRuntimeDir(Services.java:142)
at org.apache.oozie.service.Services.<init>(Services.java:126)
++

Solution

Check /var/tmp/oozie

#SSH to OOZIE HOST
ls -lrt /var/tmp/oozie

#If directory doesnot exists
mkdir -p /var/tmp/oozie
chmod 777 /var/tmp/oozie
export CATALINA_TMPDIR=${CATALINA_TMPDIR:-/var/tmp/oozie}

Start Oozie Service now.