PHP crashing when APC is enabled

[wanah]

Hello,

We enabled APC for PHP with LS Processgroup mode and on a few sites there have been PHP crashes and core dumps created.

This is with PHP 5.3 which is currently our native version.

Here is the output for dumps for both sites :

#0  0x00000000007dfb6d in do_bind_function ()
#1  0x00000000008259ac in ZEND_DECLARE_FUNCTION_SPEC_HANDLER ()
#2  0x0000000000825750 in execute ()
#3  0x00007fc31c88aaa6 in zend_oe () from /usr/local/Zend/lib/Guard-5.5.0/php-5.3.x/ZendGuardLoader.so
#4  0x00007fc31c639a90 in suhosin_execute_ex (op_array=0x2d389e8, zo=0, dummy=0) at /home/cpeasyapache/src/suhosin/suhosin-0.9.33/execute.c:585
#5  0x0000000000830be2 in ZEND_INCLUDE_OR_EVAL_SPEC_TMP_HANDLER ()
#6  0x0000000000825750 in execute ()
#7  0x00007fc31c88aaa6 in zend_oe () from /usr/local/Zend/lib/Guard-5.5.0/php-5.3.x/ZendGuardLoader.so
#8  0x00007fc31c639a90 in suhosin_execute_ex (op_array=0x2cdf550, zo=0, dummy=0) at /home/cpeasyapache/src/suhosin/suhosin-0.9.33/execute.c:585
#9  0x000000000082c63a in ZEND_INCLUDE_OR_EVAL_SPEC_CONST_HANDLER ()
#10 0x0000000000825750 in execute ()
#11 0x00007fc31c88aaa6 in zend_oe () from /usr/local/Zend/lib/Guard-5.5.0/php-5.3.x/ZendGuardLoader.so
#12 0x00007fc31c639a90 in suhosin_execute_ex (op_array=0x2cdc590, zo=0, dummy=0) at /home/cpeasyapache/src/suhosin/suhosin-0.9.33/execute.c:585
#13 0x000000000082bff4 in ZEND_INCLUDE_OR_EVAL_SPEC_CV_HANDLER ()
#14 0x0000000000825750 in execute ()
#15 0x00007fc31c88aaa6 in zend_oe () from /usr/local/Zend/lib/Guard-5.5.0/php-5.3.x/ZendGuardLoader.so
#16 0x00007fc31c639a90 in suhosin_execute_ex (op_array=0x244c820, zo=0, dummy=0) at /home/cpeasyapache/src/suhosin/suhosin-0.9.33/execute.c:585
#17 0x0000000000830be2 in ZEND_INCLUDE_OR_EVAL_SPEC_TMP_HANDLER ()
#18 0x0000000000825750 in execute ()
#19 0x00007fc31c88aaa6 in zend_oe () from /usr/local/Zend/lib/Guard-5.5.0/php-5.3.x/ZendGuardLoader.so
#20 0x00007fc31c639a90 in suhosin_execute_ex (op_array=0x244c118, zo=0, dummy=0) at /home/cpeasyapache/src/suhosin/suhosin-0.9.33/execute.c:585
#21 0x0000000000830be2 in ZEND_INCLUDE_OR_EVAL_SPEC_TMP_HANDLER ()
#22 0x0000000000825750 in execute ()
#23 0x00007fc31c88aaa6 in zend_oe () from /usr/local/Zend/lib/Guard-5.5.0/php-5.3.x/ZendGuardLoader.so
#24 0x00007fc31c639a90 in suhosin_execute_ex (op_array=0x244bd18, zo=0, dummy=0) at /home/cpeasyapache/src/suhosin/suhosin-0.9.33/execute.c:585
#25 0x0000000000830be2 in ZEND_INCLUDE_OR_EVAL_SPEC_TMP_HANDLER ()
#26 0x0000000000825750 in execute ()
#27 0x00007fc31c88aaa6 in zend_oe () from /usr/local/Zend/lib/Guard-5.5.0/php-5.3.x/ZendGuardLoader.so
#28 0x00007fc31c639a90 in suhosin_execute_ex (op_array=0x244b6c0, zo=0, dummy=0) at /home/cpeasyapache/src/suhosin/suhosin-0.9.33/execute.c:585
#29 0x00000000007fad2f in zend_execute_scripts ()
#30 0x00000000007a3ac7 in php_execute_script ()
#31 0x00000000008885ef in lsapi_module_main ()
#32 0x000000000088884c in processReq ()
#33 0x000000000088908e in main ()

#0  0x00000000007dfb6d in do_bind_function ()
#1  0x00000000008259ac in ZEND_DECLARE_FUNCTION_SPEC_HANDLER ()
#2  0x0000000000825750 in execute ()
#3  0x00007f0912b1baa6 in zend_oe () from /usr/local/Zend/lib/Guard-5.5.0/php-5.3.x/ZendGuardLoader.so
#4  0x00007f09128caa90 in suhosin_execute_ex (op_array=0x366e428, zo=0, dummy=0) at /home/cpeasyapache/src/suhosin/suhosin-0.9.33/execute.c:585
#5  0x0000000000837e4b in ZEND_INCLUDE_OR_EVAL_SPEC_VAR_HANDLER ()
#6  0x0000000000825750 in execute ()
#7  0x00007f0912b1baa6 in zend_oe () from /usr/local/Zend/lib/Guard-5.5.0/php-5.3.x/ZendGuardLoader.so
#8  0x00007f09128caa90 in suhosin_execute_ex (op_array=0x2f9f828, zo=0, dummy=0) at /home/cpeasyapache/src/suhosin/suhosin-0.9.33/execute.c:585
#9  0x000000000084de26 in zend_do_fcall_common_helper_SPEC ()
#10 0x0000000000825750 in execute ()
#11 0x00007f0912b1baa6 in zend_oe () from /usr/local/Zend/lib/Guard-5.5.0/php-5.3.x/ZendGuardLoader.so
#12 0x00007f09128caa90 in suhosin_execute_ex (op_array=0x2f9e398, zo=0, dummy=0) at /home/cpeasyapache/src/suhosin/suhosin-0.9.33/execute.c:585
#13 0x000000000084de26 in zend_do_fcall_common_helper_SPEC ()
#14 0x0000000000825750 in execute ()
#15 0x00007f0912b1baa6 in zend_oe () from /usr/local/Zend/lib/Guard-5.5.0/php-5.3.x/ZendGuardLoader.so
#16 0x00007f09128caa90 in suhosin_execute_ex (op_array=0x2d0b9b0, zo=0, dummy=0) at /home/cpeasyapache/src/suhosin/suhosin-0.9.33/execute.c:585
#17 0x00000000007f109b in zend_call_function ()
#18 0x00000000008123d7 in zend_call_method ()
#19 0x00000000008203be in zend_std_call_getter ()
#20 0x00000000008206a3 in zend_std_read_property ()
#21 0x00000000008238ae in zend_fetch_property_address_read_helper_SPEC_CV_CONST ()
#22 0x0000000000825750 in execute ()
#23 0x00007f0912b1baa6 in zend_oe () from /usr/local/Zend/lib/Guard-5.5.0/php-5.3.x/ZendGuardLoader.so
#24 0x00007f09128caa90 in suhosin_execute_ex (op_array=0x2d0d028, zo=0, dummy=0) at /home/cpeasyapache/src/suhosin/suhosin-0.9.33/execute.c:585
#25 0x000000000084de26 in zend_do_fcall_common_helper_SPEC ()
#26 0x0000000000825750 in execute ()
#27 0x00007f0912b1baa6 in zend_oe () from /usr/local/Zend/lib/Guard-5.5.0/php-5.3.x/ZendGuardLoader.so
#28 0x00007f09128caa90 in suhosin_execute_ex (op_array=0x3663fa0, zo=0, dummy=0) at /home/cpeasyapache/src/suhosin/suhosin-0.9.33/execute.c:585
#29 0x000000000084de26 in zend_do_fcall_common_helper_SPEC ()
#30 0x0000000000825750 in execute ()
#31 0x00007f0912b1baa6 in zend_oe () from /usr/local/Zend/lib/Guard-5.5.0/php-5.3.x/ZendGuardLoader.so
#32 0x00007f09128caa90 in suhosin_execute_ex (op_array=0x365c728, zo=0, dummy=0) at /home/cpeasyapache/src/suhosin/suhosin-0.9.33/execute.c:585
#33 0x000000000084de26 in zend_do_fcall_common_helper_SPEC ()
#34 0x0000000000825750 in execute ()
#35 0x00007f0912b1baa6 in zend_oe () from /usr/local/Zend/lib/Guard-5.5.0/php-5.3.x/ZendGuardLoader.so
#36 0x00007f09128caa90 in suhosin_execute_ex (op_array=0x3bd9458, zo=0, dummy=0) at /home/cpeasyapache/src/suhosin/suhosin-0.9.33/execute.c:585
#37 0x000000000084de26 in zend_do_fcall_common_helper_SPEC ()
#38 0x0000000000825750 in execute ()
#39 0x00007f0912b1baa6 in zend_oe () from /usr/local/Zend/lib/Guard-5.5.0/php-5.3.x/ZendGuardLoader.so
#40 0x00007f09128caa90 in suhosin_execute_ex (op_array=0x322ed08, zo=0, dummy=0) at /home/cpeasyapache/src/suhosin/suhosin-0.9.33/execute.c:585
#41 0x000000000084de26 in zend_do_fcall_common_helper_SPEC ()
#42 0x0000000000825750 in execute ()
#43 0x00007f0912b1baa6 in zend_oe () from /usr/local/Zend/lib/Guard-5.5.0/php-5.3.x/ZendGuardLoader.so
#44 0x00007f09128caa90 in suhosin_execute_ex (op_array=0x322e608, zo=0, dummy=0) at /home/cpeasyapache/src/suhosin/suhosin-0.9.33/execute.c:585
#45 0x000000000084de26 in zend_do_fcall_common_helper_SPEC ()
#46 0x0000000000825750 in execute ()
#47 0x00007f0912b1baa6 in zend_oe () from /usr/local/Zend/lib/Guard-5.5.0/php-5.3.x/ZendGuardLoader.so
#48 0x00007f09128caa90 in suhosin_execute_ex (op_array=0x3232660, zo=0, dummy=0) at /home/cpeasyapache/src/suhosin/suhosin-0.9.33/execute.c:585
#49 0x000000000084de26 in zend_do_fcall_common_helper_SPEC ()
#50 0x0000000000825750 in execute ()
#51 0x00007f0912b1baa6 in zend_oe () from /usr/local/Zend/lib/Guard-5.5.0/php-5.3.x/ZendGuardLoader.so
#52 0x00007f09128caa90 in suhosin_execute_ex (op_array=0x2fcdb10, zo=0, dummy=0) at /home/cpeasyapache/src/suhosin/suhosin-0.9.33/execute.c:585
#53 0x000000000084de26 in zend_do_fcall_common_helper_SPEC ()
#54 0x0000000000825750 in execute ()
#55 0x00007f0912b1baa6 in zend_oe () from /usr/local/Zend/lib/Guard-5.5.0/php-5.3.x/ZendGuardLoader.so
#56 0x00007f09128caa90 in suhosin_execute_ex (op_array=0x3b94150, zo=0, dummy=0) at /home/cpeasyapache/src/suhosin/suhosin-0.9.33/execute.c:585
#57 0x00000000007fad2f in zend_execute_scripts ()
#58 0x00000000007a3ac7 in php_execute_script ()
#59 0x00000000008885ef in lsapi_module_main ()
#60 0x000000000088884c in processReq ()
#61 0x000000000088908e in main ()

The error message I'm getting is :

Core was generated by `lsphp:/home/USERNAME/'.
Program terminated with signal 11, Segmentation fault.

I'm not sure how to interprete this…

Is it zend guard loader causing the issue when apc is enabled ? or is it lsapi_module_main ? or someting else ?

Litespeed isn't crashing or restarting as far as I can tell, just the core dumps being created.

Just beacause I see zend guard loader ass the last .so file loaded does it mean it's this ? Or something else ?

Thanks

 

[NiteWave]

may have to do more tests.

it looks there are 3 extensions involved:
apc.so
ZendGuardLoader.so
suhosin.so

as your thread title suggests, if disable apc, this issue disappear ?
how about disable ZendGuard or suhosin only among above 3 .so, no such issue as well ?

#4 0x00007fc31c639a90 in suhosin_execute_ex (op_array=0x2d389e8, zo=0, dummy=0) at /home/cpeasyapache/src/suhosin/suhosin-0.9.33/execute.c:585

maybe need build suhosin.so again under lsws ? if build under lsws, should point to /home/cpeasyapache/ls.src/...

 

[wanah]

Hello,

I'm under the impression that this is a shared memory issue.

I moved accounts that had issues to phpselector and disabled apc and suhosin for them.

I've just checked and no core dumps but instead I've got the following the the php error logs :

[19-Jan-2014 15:47:15 Europe/Paris] PHP Warning:  PHP Startup: Unable to load dynamic library '/opt/alt/php53/usr/lib64/php/modules/phar.so' - /opt/alt/php53/usr/lib64/php/modules/phar.so: failed to map segment from shared object: Cannot allocate memory in Unknown on line 0
[19-Jan-2014 15:47:15 Europe/Paris] PHP Warning:  PHP Startup: Unable to load dynamic library '/opt/alt/php53/usr/lib64/php/modules/xmlreader.so' - /opt/alt/php53/usr/lib64/php/modules/xmlreader.so: failed to map segment from shared object: Cannot allocate memory in Unknown on line 0
[19-Jan-2014 15:47:15 Europe/Paris] PHP Warning:  PHP Startup: Unable to load dynamic library '/opt/alt/php53/usr/lib64/php/modules/bz2_filter.so' - /opt/alt/php53/usr/lib64/php/modules/bz2_filter.so: failed to map segment from shared object: Cannot allocate memory in Unknown on line 0
[19-Jan-2014 15:47:15 Europe/Paris] PHP Warning:  PHP Startup: Unable to load dynamic library '/opt/alt/php53/usr/lib64/php/modules/posix.so' - /opt/alt/php53/usr/lib64/php/modules/posix.so: failed to map segment from shared object: Cannot allocate memory in Unknown on line 0
[19-Jan-2014 15:47:15 Europe/Paris] PHP Warning:  PHP Startup: Unable to load dynamic library '/opt/alt/php53/usr/lib64/php/modules/mcrypt.so' - /opt/alt/php53/usr/lib64/php/modules/mcrypt.so: failed to map segment from shared object: Cannot allocate memory in Unknown on line 0
[19-Jan-2014 15:47:15 Europe/Paris] PHP Warning:  PHP Startup: Unable to load dynamic library '/opt/alt/php53/usr/lib64/php/modules/xsl.so' - /opt/alt/php53/usr/lib64/php/modules/xsl.so: failed to map segment from shared object: Cannot allocate memory in Unknown on line 0
[19-Jan-2014 15:47:22 Europe/Paris] PHP Warning:  PHP Startup: Unable to load dynamic library '/opt/alt/php53/usr/lib64/php/modules/pdo_sqlite.so' - /opt/alt/php53/usr/lib64/php/modules/pdo_sqlite.so: failed to map segment from shared object: Cannot allocate memory in Unknown on line 0
[19-Jan-2014 15:47:22 Europe/Paris] PHP Warning:  PHP Startup: Unable to load dynamic library '/opt/alt/php53/usr/lib64/php/modules/phar.so' - /opt/alt/php53/usr/lib64/php/modules/phar.so: failed to map segment from shared object: Cannot allocate memory in Unknown on line 0
[19-Jan-2014 15:47:22 Europe/Paris] PHP Warning:  PHP Startup: Unable to load dynamic library '/opt/alt/php53/usr/lib64/php/modules/xmlreader.so' - /opt/alt/php53/usr/lib64/php/modules/xmlreader.so: failed to map segment from shared object: Cannot allocate memory in Unknown on line 0
[19-Jan-2014 15:47:22 Europe/Paris] PHP Warning:  PHP Startup: Unable to load dynamic library '/opt/alt/php53/usr/lib64/php/modules/bz2_filter.so' - /opt/alt/php53/usr/lib64/php/modules/bz2_filter.so: failed to map segment from shared object: Cannot allocate memory in Unknown on line 0
[19-Jan-2014 15:47:22 Europe/Paris] PHP Warning:  PHP Startup: Unable to load dynamic library '/opt/alt/php53/usr/lib64/php/modules/posix.so' - /opt/alt/php53/usr/lib64/php/modules/posix.so: failed to map segment from shared object: Cannot allocate memory in Unknown on line 0
[19-Jan-2014 15:47:22 Europe/Paris] PHP Warning:  PHP Startup: Unable to load dynamic library '/opt/alt/php53/usr/lib64/php/modules/mcrypt.so' - /opt/alt/php53/usr/lib64/php/modules/mcrypt.so: failed to map segment from shared object: Cannot allocate memory in Unknown on line 0
[19-Jan-2014 15:47:22 Europe/Paris] PHP Warning:  PHP Startup: Unable to load dynamic library '/opt/alt/php53/usr/lib64/php/modules/xsl.so' - /opt/alt/php53/usr/lib64/php/modules/xsl.so: failed to map segment from shared object: Cannot allocate memory in Unknown on line 0

So it seems that by enabeling apc for all users I've hit a shared memory limit of some kind.

The server has got plenty of free memory :

free -g
total used free shared buffers cached
Mem: 251 159 92 0 47 56
-/+ buffers/cache: 55 195

(thats 55 GB used and 195 GB available).

PHP's memory limit for this account is set to 512 M

Litespeed's external app memory limit is set to :

Memory soft limit : 122880M
Memory hard limit : 163840M

Cloudlinux's memory limit's are disabled

but unlimit -a gives :

core file size          (blocks, -c) 200000
data seg size           (kbytes, -d) 200000
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 2063089
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) 200000
open files                      (-n) 100
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 35
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Could I be hitting :

max memory size         (kbytes, -m) 200000

?

 

[wanah]

This value was from inside a user's account. Some of the values do seem low though…

The value for root user is :

ulimit -a
core file size          (blocks, -c) 1000000
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 2063089
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 14335
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

 

[wanah]

It was cPanel's fork bomb protection causing theses values :

we now have :

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 2063089
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65535
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 65535
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

We will see how it goes with these settings.

 

[mistwang]

Thanks for the update. Can you please more explicit with the change you made to cPanel to fix it? we may add a wiki entry for how to enable per user SHM.