The system has encountered an error when servicing the request, Please try again. If the error persists, please contact the system administrator.
Just i wanted to share the issue i encountered today. We are unable to login to application due to the error above. We got the error instead of login page.
What i did?
- As soon as i get the error in login page, i checked the apache logs. I can see some chain failed error but it was tripping all the time (Not only today), So i decided it should nt be cause of the today error.
- I logged into admin console and checked all managed servers since we are in 12.2 (Everything was runnig fine).
-I bounced apache once but no use
- I went to db tier and connected to sqlplus where i found the actual issue. When i connected to sqlplus i got the error below :
ERROR:
ORA-00020: maximum number of processes (200) exceeded
I found the cause of the issue. Application was unable to create db connection since limit is overloaded.
How i narrowed down the issue?
I did how many processed connected to db.
ps -ef|grep oratest (Make sure you have only one instance running under OS user, I had only one instance running under oratest)
I checked how many connections coming from outside,
ps -ef|grep -i local=no|grep oratest
I found out totally 170 sessions were created from outside.
How i fixed?
I decided to kill all local=no processes to reduce number of processes. My environment was test and i knew the impact.
****Please double check and check with your manager or your peers before you kill any such process in your environment.****
ps -ef|grep -i local=no|grep -i oratest|grep -v grep|awk '{print $2}'|xargs kill -9
Above command will get all the process id's of local=no processes and kill them.
Within a minute, limit is freed up and i was able to connect to sqlplus. As soon as application were able to connect to db, we got the login page.
Thanks!
Sikky
No comments:
Post a Comment