[Resolved] APC & litespeed restarts because of unknown reason

[semprot]

After using lsphp 6.3, and use "suEXEC Daemon" on "Run On Start Up", APC is still restarted (my litespeed php5 config is attached below).
When i checked the litespeed log, i see some messages that could be the reason of the litespeed to be restarted?

No request delivery notification has been received from LSAPI process group [-1], possible run away process.
...
Retry with new process group.
...
connection to [uds://tmp/lshttpd/lsphp5.sock.29027] on request #0, confirmed, 0, associated process: -1, running: 0, error: Connection timed out!
...
Suspicious request timeout, associated process: -1, running: 0.
Max retries has been reached, 503!
oops! 503 Service Unavailable

Does my litespeed config is not optimized?

Runtime Settings
apc.cache_by_default	1
apc.canonicalize	1
apc.coredump_unmap	0
apc.enable_cli	1
apc.enabled	1
apc.file_md5	0
apc.file_update_protection	2
apc.filters	
apc.gc_ttl	3600
apc.include_once_override	0
apc.lazy_classes	0
apc.lazy_functions	0
apc.max_file_size	20M
apc.mmap_file_mask	/home/apc/apc.GKhZuh
apc.num_files_hint	1000
apc.preload_path	
apc.report_autofilter	0
apc.rfc1867	0
apc.rfc1867_freq	0
apc.rfc1867_name	APC_UPLOAD_PROGRESS
apc.rfc1867_prefix	upload_
apc.rfc1867_ttl	3600
apc.serializer	default
apc.shm_segments	1
apc.shm_size	7144M
apc.slam_defense	1
apc.stat	0
apc.stat_ctime	0
apc.ttl	0
apc.use_request_time	1
apc.user_entries_hint	400000
apc.user_ttl	7200
apc.write_lock	1

 

[NiteWave]

the lsphp5 config is normal and ok.

use cloudlinux or not ?

please paste the "pstree -p" output under litespeed root process

 

[semprot]

the lsphp5 config is normal and ok.

use cloudlinux or not ?

please paste the "pstree -p" output under litespeed root process

1. i think i use "CENTOS 5.9" (from WHM).
2. here is the result. (i used 4 cpu license just 15 mins ago, it was 2 cpu before, i was upgrading the license)

|-litespeed(9359)-+-httpd(9360)
|                 |-litespeed(9363)-+-admin_php(9429)
|                 |                 |-splitlogs(9430)
|                 |                 |-splitlogs(9431)
|                 |                 |-{litespeed}(9365)
|                 |                 `-{litespeed}(9366)
|                 |-litespeed(9364)-+-splitlogs(9375)
|                 |                 |-splitlogs(9376)
|                 |                 |-{litespeed}(9368)
|                 |                 `-{litespeed}(9369)
|                 |-litespeed(9367)-+-splitlogs(9508)
|                 |                 |-splitlogs(9509)
|                 |                 |-{litespeed}(9372)
|                 |                 `-{litespeed}(9374)
|                 |-litespeed(9370)-+-splitlogs(9435)
|                 |                 |-splitlogs(9436)
|                 |                 |-{litespeed}(9371)
|                 |                 `-{litespeed}(9373)
|                 `-lsphp5(9362)-+-lsphp5(9377)
|                                |-lsphp5(9378)
|                                |-lsphp5(9379)
|                                |-lsphp5(9380)
|                                |-lsphp5(9381)
|                                |-lsphp5(9382)
|                                |-lsphp5(9383)
|                                |-lsphp5(9397)
|                                |-lsphp5(9398)
|                                |-lsphp5(9399)
|                                |-lsphp5(9416)
|                                |-lsphp5(9425)
|                                |-lsphp5(9426)
|                                |-lsphp5(9427)
|                                |-lsphp5(9428)
|                                |-lsphp5(9433)
|                                |-lsphp5(9493)
|                                |-lsphp5(9506)
|                                `-lsphp5(9507)

 

[NiteWave]

from the pstree output, php run in SuEXEC daemon mode as expected.

>APC is still restarted
because of webserver restart which because of

>Max retries has been reached, 503!
>oops! 503 Service Unavailable

now should trouble-shoot the 503 error issue. when it's resolved, APC issue will be fixed too.

regarding php 503 error, it's because of php process crash. there are many reasons a php process could crash. a simple php script won't cause 503 error.

1st, check the php extensions you've installed. see if able to disable some or all of them to reduce 503 error. please paste here the list of your current extensions(.so)

 

[semprot]

from the pstree output, php run in SuEXEC daemon mode as expected.

>APC is still restarted
because of webserver restart which because of

>Max retries has been reached, 503!
>oops! 503 Service Unavailable

now should trouble-shoot the 503 error issue. when it's resolved, APC issue will be fixed too.

regarding php 503 error, it's because of php process crash. there are many reasons a php process could crash. a simple php script won't cause 503 error.

1st, check the php extensions you've installed. see if able to disable some or all of them to reduce 503 error. please paste here the list of your current extensions(.so)

Yes i received emails too about lsws being restarted because of 503 error.

Here is my loaded extensions :

apache2handler
bcmath
bz2
calendar
Core
ctype
curl
date
dom
ereg
exif
filter
ftp
gd
gettext
hash
iconv
json
libxml
mbstring
mcrypt
mhash
mongo
mysql
mysqli
mysqlnd
odbc
pcre
PDO
pdo_mysql
pdo_sqlite
Phar
Reflection
session
SimpleXML
soap
sockets
SPL
sqlite3
standard
tokenizer
wddx
xml
xmlreader
xmlrpc
xmlwriter
xsl
zip
zlib

Additional information : i have doubled my RAM few hours ago hoping it will solve the issue, but i still have the issue.

 

[NiteWave]

the php extensions, I meant those not build in to lsphp5

please locate the current php.ini, then

#grep extension /path/to/php.ini

to see how many .so are loaded.

 

[semprot]

the php extensions, I meant those not build in to lsphp5

please locate the current php.ini, then

#grep extension /path/to/php.ini

to see how many .so are loaded.

Okay here it is. Thanks.

; Directory in which the loadable extensions (modules) reside.
extension_dir = "/usr/local/lib/php/extensions/no-debug-non-zts-20060613"
zend_extension="/usr/local/IonCube/ioncube_loader_lin_5.2.so"
extension = "pdo.so"
extension = "pdo_sqlite.so"
extension = "sqlite.so"
extension = "pdo_mysql.so"
extension = "ffmpeg.so"
extension="apc.so"
extension="timezonedb.so"
extension = "memcache.so"
;eaccelerator.cache_dir="/tmp/eaccelerator"
;eaccelerator.check_mtime="1"
;eaccelerator.compress="1"
;eaccelerator.compress_level="9"
;eaccelerator.debug="0"
;eaccelerator.enable="1"
;eaccelerator.filter=""
;eaccelerator.optimizer="1"
;eaccelerator.shm_max="0"
;eaccelerator.shm_only="0"
;eaccelerator.shm_prune_period="0"
;eaccelerator.shm_size="16"
;eaccelerator.shm_ttl="0"

 

[NiteWave]

>zend_extension="/usr/local/IonCube/ioncube_loader_lin_5.2.so"
what's your php version ? php 5.2.x ?

 

[semprot]

>zend_extension="/usr/local/IonCube/ioncube_loader_lin_5.2.so"
what's your php version ? php 5.2.x ?

Yes i run 5.2 now.

 

[semprot]

More information,

/var/log/messages:

Oct 6 04:37:43 myusenamehere kernel: lsphp5[8870]: segfault at ffffffffffffffff rip 0000003d7ae0a2ca rsp 00007fff8e9d7188 error 6
Oct 6 04:37:43 myusenamehere kernel: lsphp5[8871]: segfault at ffffffffffffffff rip 0000003d7ae0a2ca rsp 00007fffdf459398 error 6
Oct 6 04:37:43 myusenamehere kernel: lsphp5[8872]: segfault at ffffffffffffffff rip 0000003d7ae0a2ca rsp 00007fff03c42a18 error 6
Oct 6 04:37:43 myusenamehere kernel: lsphp5[8873]: segfault at ffffffffffffffff rip 0000003d7ae0a2ca rsp 00007fff9d566338 error 6
Oct 6 04:37:43 myusenamehere kernel: lsphp5[8874]: segfault at ffffffffffffffff rip 0000003d7ae0a2ca rsp 00007fff569734a8 error 6
Oct 6 04:37:44 myusenamehere kernel: lsphp5[8875]: segfault at ffffffffffffffff rip 0000003d7ae0a2ca rsp 00007fff0b35ad18 error 6
Oct 6 04:37:44 myusenamehere kernel: lsphp5[8876]: segfault at ffffffffffffffff rip 0000003d7ae0a2ca rsp 00007fff46312478 error 6
Oct 6 04:37:44 myusenamehere kernel: lsphp5[8877]: segfault at ffffffffffffffff rip 0000003d7ae0a2ca rsp 00007fff31bda378 error 6
Oct 6 04:37:44 myusenamehere kernel: lsphp5[8878]: segfault at ffffffffffffffff rip 0000003d7ae0a2ca rsp 00007fff8df93ff8 error 6
Oct 6 04:37:44 myusenamehere kernel: lsphp5[8879]: segfault at ffffffffffffffff rip 0000003d7ae0a2ca rsp 00007fffb2e0a488 error 6
Oct 6 04:37:44 myusenamehere kernel: lsphp5[8881]: segfault at ffffffffffffffff rip 0000003d7ae0a2ca rsp 00007fff96509178 error 6
Oct 6 04:37:44 myusenamehere kernel: lsphp5[8882]: segfault at ffffffffffffffff rip 0000003d7ae0a2ca rsp 00007fff0cb0c158 error 6
Oct 6 04:37:45 myusenamehere kernel: lsphp5[8883]: segfault at ffffffffffffffff rip 0000003d7ae0a2ca rsp 00007fff0a422478 error 6
Oct 6 04:37:45 myusenamehere kernel: lsphp5[8889]: segfault at ffffffffffffffff rip 0000003d7ae0a2ca rsp 00007fff893779b8 error 6
Oct 6 04:37:45 myusenamehere kernel: lsphp5[8891]: segfault at ffffffffffffffff rip 0000003d7ae0a2ca rsp 00007fff1c32c978 error 6
Oct 6 04:37:45 myusenamehere kernel: lsphp5[8892]: segfault at ffffffffffffffff rip 0000003d7ae0a2ca rsp 00007fff05b02028 error 6
Oct 6 04:37:45 myusenamehere kernel: lsphp5[8893]: segfault at ffffffffffffffff rip 0000003d7ae0a2ca rsp 00007fffaa76e0f8 error 6
Oct 6 04:37:45 myusenamehere kernel: lsphp5[8894]: segfault at ffffffffffffffff rip 0000003d7ae0a2ca rsp 00007fff298f16e8 error 6
Oct 6 04:37:45 myusenamehere kernel: lsphp5[8895]: segfault at ffffffffffffffff rip 0000003d7ae0a2ca rsp 00007fff32f38b98 error 6
Oct 6 04:37:45 myusenamehere kernel: lsphp5[8896]: segfault at ffffffffffffffff rip 0000003d7ae0a2ca rsp 00007fffce968918 error 6
Oct 6 04:37:45 myusenamehere kernel: lsphp5[8897]: segfault at ffffffffffffffff rip 0000003d7ae0a2ca rsp 00007fff6f7ddba8 error 6

and tons of similar message.

 

[NiteWave]

so php crashed many times.

the main problem should be at php code side. with so many crash, why you still want to enable APC ? 1st priority is to reduce the crashes. will disable APC to stop or reduce crash ?

"Oct 6 04:37:43 myusenamehere kernel: lsphp5[8873]: segfault at ffffffffffffffff rip 0000003d7ae0a2ca rsp 00007fff9d566338 error 6"

if above crashes happens on one user "myusenamehere" only, you can disable it temporarily. and watch some time if php crash/503 error/lsws restart has improved.

another suggestion is to install php 5.3.x or 5.4, move some accounts to run 5.3/5.4. you can run multiple version php at the same time. here's the how-to wiki:
http://www.litespeedtech.com/suppor...peed_wiki:multiple_phps_in_control_panel_lsws

 

[semprot]

so php crashed many times.

the main problem should be at php code side. with so many crash, why you still want to enable APC ? 1st priority is to reduce the crashes. will disable APC to stop or reduce crash ?

"Oct 6 04:37:43 myusenamehere kernel: lsphp5[8873]: segfault at ffffffffffffffff rip 0000003d7ae0a2ca rsp 00007fff9d566338 error 6"

if above crashes happens on one user "myusenamehere" only, you can disable it temporarily. and watch some time if php crash/503 error/lsws restart has improved.

another suggestion is to install php 5.3.x or 5.4, move some accounts to run 5.3/5.4. you can run multiple version php at the same time. here's the how-to wiki:
http://www.litespeedtech.com/suppor...peed_wiki:multiple_phps_in_control_panel_lsws

About the APC, it helps to cache queries in my website and reduce load significantly.

And the problem appeared only about 3 days ago. I've been running APC for months, traffic is stable (no sudden increase), and 3 days ago errors popped up.

Is it possible that crashed is caused by something else? Such as PHP coding error / lack of memory (that i doubt, because my RAM has been doubled now) / misconfigured server?

About PHP 5.3.x or 5.4 multiple installation, does it needed for analyzing the cause of the crash / solve the crash itself?

Thanks.

 

[NiteWave]

Such as PHP coding error / lack of memory (that i doubt, because my RAM has been doubled now) / misconfigured server?

most likely "PHP coding error" / code change

about mis-config server, check if you added
extension="apc.so"
extension = "ffmpeg.so"
extension = "memcache.so"
..
etc recently

 

[semprot]

Problem has been solved. The php crash was caused by mismatched binaries, and the php was rebuilt several times, and finally it is fixed

 

[NiteWave]

Problem has been solved. The php crash was caused by mismatched binaries, and the php was rebuilt several times, and finally it is fixed

great! thanks for update. particularly which binaries ?

 

[semprot]

great! thanks for update. particularly which binaries ?

Here is what my hosting support said

I believe I may have resolved your issues. I was seeing the following when looking into your PHP configuration:

myusername@mybox [/home]# /usr/local/cpanel/bin/rebuild_phpconf --available
Available handlers: suphp dso fcgi cgi none
PHP4 SAPI: cgi
PHP5 SAPI: not installed
SUEXEC: available
RUID2: not available


The rebuild I did earlier should have fixed this, but did not for some reason. I've re-run easyapache to rebuild Apache's PHP binaries, and then rebuilt Litespeed's to match, and everything has settled down. I've since stopped seeing PHP segfaults in messages also, so I believe the issue to be resolved. I'm also seeing the load way down near 1-8 as opposed to 15-23 that it was earlier. Looks like the main issue was the Apache PHP binaries were broken, along with a misconfigured PHP configuration and php.ini. I'm going to run a forced update and re-install of cPanel and it's core perl module in hopes that it fixes whatever had broken the configuration.

Update : after i watched the server, it seems APC is still restarted every 10-25 mins. But no 503 error e-mail on my inbox anymore. And because i've increased my RAM, the effect is not as significant as when my RAM is still small. Will check the litespeed log later.