Saturday, 8 February 2014

Trouble shoot Apache segmentation fault issues

Server restarted gracefully.

Graceful restart requested, doing restart

[Thu Apr 26 02:17:26 2012] [notice] seg fault or similar nasty error detected in the parent process

We found the problem by using strace to watch httpd activity during failure. I had found that a /etc/init.d/httpd stop and /etc/init.d/httpd start worked fine, but /etc/init.d/httpd graceful or apachectl graceful brought the system down.

we find the parent process to all other httpd processes, the one with ‘1’ as the parent process, and strace it to watch activity using ps,

ps -aef | grep httpd

Go to screen & run,

strace -aef -p 8021 2>&1 | tee /tmp/strace.out

issue the command apachectl graceful

-Watch the file strace.out,
We see access for ioncube_loader_lin_5.3.so, which succeeds with a file handle (20), and access for an ffmpeg.so module, which fails with ENOENT (-1). The process then logs to the error log, and a segfault occurs.

Commenting out the ffmpeg.so line (insert a semicolon at the beginning of the line) and re-running the test gave another seg fault, but this time on rar.so. So, out that goes too, comment that out and start httpd and gracefully restart it and… no error! The server stays up, strace output continues and operation is uninterrupted.

No comments:

Post a Comment