====== Troubleshooting 503 Errors ====== 503 errors are often caused by a malfunction in PHP, and are not related to Apache or LiteSpeed Web Server. One of the common reasons for a 503 error is a PHP crash. Generally, troubleshooting 503 errors is out of our support scope. Just the same, this wiki will go over basic steps to troubleshoot 503 errors, some common causes of 503 errors, and some examples that show the steps in practice. If your server is running on a control panel, such as cPanel or Plesk, and runs into a 503 error, you should temporarily switch to Apache for a test. Most of the time, it will run into the same 503 issue with Apache. This means that it is //not// a LiteSpeed Web Server issue. Nonetheless, we have provided this wiki to give you some troubleshooting ideas. Please run through the steps outlined below until you find the issue. **If everything runs well with Apache but 503 problems occur only with LiteSpeed, you can log a ticket with us to take a look.** If these troubleshooting steps sound too overwhelming to you, you can engage us through [[https://store.litespeedtech.com/store/cart.php?gid=5|hourly support]]. We will troubleshoot and try our best to fix the issue for you. ===== Quick Troubleshooting Guide ===== Before engaging us for premium support service, you may like to try a few of these simple steps to see if you can fix the error yourself. * Check the phpinfo page of the problematic user account. ([[litespeed_wiki:php:php_info|Create a phpinfo.php file]] if no phpinfo page exists.) * Generally, server ''error_log/stderr.log'' should provide some hints as to the problem. For example, it may be a memory issue. * Disable opcode cache for that PHP version and verify from the phpinfo page. * Disable unsafe PHP extensions such as ZendGuardLoader, Suhosin, ionCube, etc. * Check disk space. * If CloudLinux is used, check the LVE memory limit to see if it needs to be increased. * If CloudLinux is used, check the LVE process limit to see if it needs to be increased. * Try increasing the PHP memory limit. These are just a few simple quick fixes for you to try first. If they don't help, you can refer to the following for detailed steps/reasons, or engage our support as mentioned earlier. ===== Comprehensive Troubleshooting Guide ===== ==== Check the Server Error Log and PHP Error Log ==== For native LSWS setups, the error log is usually named ''error.log''. It may be named ''error_log'' if you are using WHM/cPanel, and the usual location is ''/usr/local/apache/logs/error_log''. PHP may also generate error logs at the location of the script, for example, ''/home/user/public_html/error_log''. The error log helps you identify what kind of problem is occurring and when the problem is occurring. Here is an example of LiteSpeed Web Server error log entries showing a 503 error: 2014-01-28 10:05:51.751 [INFO] [lsphp5] PID: 45143, add child process pid: 45175, procinfo: 0x15656f0 2014-01-28 10:05:51.790 [INFO] Remove pid: 45175, exitcode: 255 2014-01-28 10:05:51.790 [INFO] [66.249.64.1:14152-0#APVH_lsws.com] connection to [/tmp/lshttpd/lsphp5.sock] on request #0, confirmed, 0, associated process: -1, running: 0, error: Connection reset by peer! 2014-01-28 10:05:51.791 [INFO] [lsphp5] PID: 45143, add child process pid: 45177, procinfo: 0x15656f0 2014-01-28 10:05:51.829 [INFO] Remove pid: 45177, exitcode: 255 2014-01-28 10:05:51.829 [INFO] [66.249.64.1:14152-0#APVH_lsws.com] connection to [/tmp/lshttpd/lsphp5.sock] on request #0, confirmed, 0, associated process: -1, running: 0, error: Connection reset by peer! 2014-01-28 10:05:51.831 [INFO] [lsphp5] PID: 45143, add child process pid: 45179, procinfo: 0x15656f0 2014-01-28 10:05:51.869 [INFO] Remove pid: 45179, exitcode: 255 2014-01-28 10:05:51.869 [INFO] [66.249.64.1:14152-0#APVH_lsws.com] connection to [/tmp/lshttpd/lsphp5.sock] on request #0, confirmed, 0, associated process: -1, running: 0, error: Connection reset by peer! 2014-01-28 10:05:51.869 [NOTICE] [66.249.64.1:14152-0#APVH_lsws.com] Max retries has been reached, 503! 2014-01-28 10:05:51.869 [NOTICE] [66.249.64.1:14152-0#APVH_lsws.com] oops! 503 Service Unavailable^M Interpreting these entries: * The time stamp - We can use this time stamp later to find related entries in other logs. * ''Remove pid: 45175, exitcode: 255'' - In this log line, we see the PID of the process that died as well as an exit code. This also tells us that the process did not crash. If it had crashed, instead of an exit code, we would have seen an entry telling us the process had been ''killed by signal''. * ''error: Connection reset by peer!'' - This line tells you that an error just occurred. * ''confirmed'' - The confirmed value tells you whether, in the error reported, the PHP process took the request. Possible values are ''0'' and ''1''. ''0'' means that the PHP process died before taking the request. This suggests that it ran into the problem while parsing the ''php.ini''. * ''associated process'' - The associated process value gives you a PID. You can use this PID to find the process with a problem. If the value is ''-1'', then LSWS does not have its PID, probably because it died before taking the request. * ''Max retries has been reached, 503!'' - In most cases, LSWS will retry when it gets an error. After three ''Connection reset by peer!'' errors, LSWS will then return a 503 error page. This rule of three retries generally holds true. With a POST request, however, if the confirmed value is ''1'' (meaning the error occurred after the request was taken), LSWS will return a 503 after one try. From these entries, we can tell that the above error did not cause a crash. We know what time the error occurred and to which process. We also know that the problem is probably in the php.ini, since the error occurred before the process took the request. Let's look at another example for 503 due to PHP Crash. Grep ''oops'' to locate them. grep oops /error_log This command returns the following output: 2015-12-24 22:17:29.376 [NOTICE] [37.130.227.133:16368] oops! 503 Service Unavailable This indicates that ''37.130.227.133:16368'' has a problem. Try to locate it by running: grep 37.130.227.133:61318 /error_log This command returns: 2015-12-24 22:17:28.185 [INFO] [37.130.227.133:61318] connection to [/tmp/lshttpd/APVH_minfo_Suphp.sock.901] on request #0, confirmed, 1, associated process: 397369, running: 1, error: Connection reset by peer! 2015-12-24 22:17:28.189 [INFO] [37.130.227.133:61318] PID: 381184, add child process pid: 397371, procinfo: 0x2854280 2015-12-24 22:17:28.223 [INFO] [37.130.227.133:61318] connection to [/tmp/lshttpd/APVH_minfo_Suphp.sock.901] on request #0, confirmed, 1, associated process: 397371, running: 0, error: Connection reset by peer! 2015-12-24 22:17:28.223 [NOTICE] [37.130.227.133:61318] Max retries has been reached, 503! 2015-12-24 22:17:28.223 [NOTICE] [37.130.227.133:61318] oops! 503 Service Unavailable 2015-12-24 22:17:28.223 [NOTICE] [37.130.227.133:61318] Content len: 0, Request line: 'GET /ajax/api/hook/decodeArguments? arguments=O%3A12%3A%22vB_dB_Result%22%3A2%3A%7Bs%3A5%3A%22%00*%00db%22%3BO%3A11%3A%22vB_Database%22%3A1%3A%7Bs%3A9%3A%22functions%22%3Ba%3A1%3A%7Bs%3A11%3A%22free_result%22%3Bs%3A6%3A%22system%22%3B%7D%7Ds%3A12%3A%22%00*%00recordset%22%3Bs%3A15%3A%22echo%20'object0r'%22%3B%7D HTTP/1.1' 2015-12-24 22:17:28.223 [NOTICE] [37.130.227.133:61318] Redirect: #1, URL: /index.php 2015-12-24 22:17:28.223 [INFO] [37.130.227.133:61318] abort request..., code: 4 2015-12-24 22:17:28.223 [INFO] [37.130.227.133:61318] connection to [/tmp/lshttpd/APVH_minfo_Suphp.sock.901] on request #0, confirmed, 1, associated process: 397371, running: 0, error: Connection reset by peer! which shows process ''397371'' likely crashed. To verify: grep 397371 /error_log 2015-12-24 22:17:28.189 [INFO] [37.130.227.133:61318] PID: 381184, add child process pid: 397371, procinfo: 0x2854280 2015-12-24 22:17:28.223 [INFO] [37.130.227.133:61318] connection to [/tmp/lshttpd/APVH_minfo_Suphp.sock.901] on request #0, confirmed, 1, associated process: 397371, running: 0, error: Connection reset by peer! 2015-12-24 22:17:28.223 [INFO] Process: 397371, killed by signal: 11, core dump: 126 ''Process: 397371, killed by signal: 11, core dump: 126'' tells us that the 503 error was due to a PHP crash. ''core dump: 126'' means a core file was generated. You may try to use GDB to debug it as shown in step 3 below. Sometime PHP crashes but you will see the line ''core dump: 0'' in your error log which means a core file was not generated. Please follow the process in step 3 to enable the core dump. For CentOS/Redhat when using ABRT, you may see the line ''core dump: 126'' in your error log even though no core file was generated. In this case, you can enable it through ''/etc/abrt/abrt-action-save-package-data.conf''(refer to step 3 for details). Let's look at another example: The error log includes: 2016-08-31 10:31:33.927 [INFO] [136.243.154.102:56558] PID: 946060, add child process pid: 964125, procinfo: 0x5f312a0 2016-08-31 10:31:43.915 [INFO] [136.243.154.102:56558] connection to [/tmp/lshttpd/APVH_notici4s_Suphp.sock.378] on request #0, confirmed, 1, associated process: 964125, running: 1, error: Connection reset by peer! 2016-08-31 10:31:43.916 [INFO] Process: 964125, killed by signal: 9, core dump: 0 The lsphp process was killed by signal: 9 not 11. Most situations like this are caused by an ''out of memory'' error. Check the ''dmesg'' log: dmesg | egrep -i -B100 'killed process' [11486240.579151] Out of memory in UB 1871: OOM killed process 997500 (lsphp) score 0 vm:438884kB, rss:108820kB, swap:0kB [11486249.204813] Out of memory in UB 1871: OOM killed process 997592 (lsphp) score 0 vm:430272kB, rss:99576kB, swap:0kB To resolve this, you should double or triple the lsphp memory limit from the LSWS Admin Console for the related lsphp external app: [[https://www.litespeedtech.com/docs/webserver/config/extapps/lsapi#memSoftLimit|MemorySoftLimit]] and [[https://www.litespeedtech.com/docs/webserver/config/extapps/lsapi#memHardLimit|MemoryHardLimit]] ==== Check for Corresponding Entries in stderr.log and system log ==== The ''stderr.log'' logs errors from the standard error stream. This log can give you additional information about errors that occurred. Using the timestamps and PIDs you've gotten from the error log, you may be able to find relevant errors in the ''stderr.log''. The ''stderr.log'' can usually be found in the same directory as the error log. For LSWS native, this is normally ''/usr/local/lsws/logs/stderr.log''. In WHM/cPanel, it is usually ''/usr/local/apache/logs/stderr.log''. Here are some sample entries we might find in stderr.log: 2014-01-28 10:05:51.789 [STDERR] PHP Fatal error: Directive 'register_globals' is no longer available in PHP in Unknown on line 0 2014-01-28 10:05:51.828 [STDERR] PHP Fatal error: Directive 'register_globals' is no longer available in PHP in Unknown on line 0 2014-01-28 10:05:51.867 [STDERR] PHP Fatal error: Directive 'register_globals' is no longer available in PHP in Unknown on line 0 In the entries above, the error is caused by the directive ''register_globals''. We only need to remove that directive. If you cannot find errors in ''stderr.log'', you may have to resort to testing common causes of PHP errors (addressed below). You may also check the system log since it may help you determine if there was a PHP crash. For example: vi /var/log/message or tail /var/log/message or dmesg | egrep -i -B100 'killed process' or dmesg | egrep -i -B100 'segfault' An example: In the server's error log: 2016-08-08 15:16:22.307 [NOTICE] [176.88.888.88:57670] request already in process for 73 seconds, fail with 503 2016-08-08 15:16:22.307 [NOTICE] [176.88.888.88:57670] oops! 503 Service Unavailable 2016-08-08 15:16:22.307 [INFO] [176.88.888.88:57670] File not found [/home/discountgolfstor/public_html/503.shtml] In the stderr log: 2016-08-08 15:16:22.307 [STDERR] Child process with pid: 20031 was killed by signal: 11, core dump: 0 or 2017-10-27 14:28:45.054 [STDERR] (986548): Child process with pid: 986552 was killed by signal: 6, core dump: 128 In the system log ''/var/log/message'' Aug 8 15:15:55 ssd148 kernel: lsphp5[20011]: segfault at 7ffecbbaaf70 ip 00007fa0b74d9df7 sp 00007ffecbbaaf60 error 6 in libpcre.so.1.2.6[7fa0b74c6000+41000] All logs show the PHP crash. The PHP process was killed by signal 11. "core dump: 0" means no core file was generated during the crash. The system log shows it is a ''pcre'' related crash. If you need to trace the crash in details, you should enable core dump which is explained below. ==== Check phpinfo page ==== You should check (create a phpinfo.php file if it doesn't exist yet) phpinfo page to see php opcode cache, php extentions, memory_limit, php_execution_time etc, PHP versions to gain better idea about PHP environment. ==== Disable PHP Opcode Caches (APC, xCache, eAccelerator) ==== The different opcode caches often have compatibility issues with PHP. These issues may be uncovered when looking through the backtrace of a PHP crash. Often, if you are getting a 503 error, and especially if you see PHP crashing, it is a good idea to try turning off your opcode cache to see if doing so solves the problem. To turn off opcode caching, locate a line in your php.ini files that includes ''opcache.so'' ''apc.so'', ''xcache.so'', or ''eaccelerator.so'' and comment it out. Restart LSWS and try the page in question again. Note that only one opcode cache can be loaded at a time. They don't work together. If you find that the opcode cache is causing the error, you can try upgrading the version of the opcode cache you are using or using a different version of PHP. If neither of these steps fixes the issue, disable the opcode cache and possibly try a different opcode cache. You may also want to submit a bug report to the opcode cache developer. Example: in stderr.log: 2017-07-30 18:36:17.314 [STDERR] zend_mm_heap corrupted "zend_mm_heap corrupted" is almost always related to bugs in PHP OPcache, bugs that can be triggered by any number of things, including other extensions being loaded. You should turn off opcache or disable certain application extensions. === An example === When running XenForo 2.0, the user is getting a lot of 503 errors when accessing XenForo Admin Portal via links like ''https://example.com/forum/admin.php''. However, when switching to Apache, there is no 503 error at all. Error_log: 2018-06-17 17:46:32.104047 [INFO] [136.26.101.82:63425] connection to [/tmp/lshttpd/APVH_canonr_Suphp56.sock.941] on request #4, confirmed, 1, associated process: 8647, running: 1, error: Connection reset by peer! 2018-06-17 17:46:32.104114 [NOTICE] [136.26.11.82:63425] POST request in process stage, fail with 503 2018-06-17 17:46:32.104165 [NOTICE] [136.26.11.82:63425] oops! 503 Service Unavailable 2018-06-17 17:46:32.104175 [NOTICE] [136.26.11.82:63425] Content len: 124, Request line: 'POST /forum/admin.php?add-ons/ThemeHouse-UIX/install HTTP/1.1' 2018-06-17 17:46:32.104183 [INFO] [136.26.101.82:63425] Cookie len: 881, __auc=170dbaf0160d7bd3f2f14d21cb3; _ga=GA1.2.875072536.1515447992; __vrz=1.0.8; __qca=P0-982906617-1515447994559; __gads=ID=091a33f508c4e503:T=1515447998:S=ALNI_MaasaG2m1W1N0yk4ZZzNSzcX4YWhg; __gacid=1830d162-321c-4e8f-b0fe-0e352762f7dd; __sonar=2579824405608507099; _bs5f9d842da81e5abe9750219d590c74d3=1; _bsd5bbb597c02b3c01ca67e41d766da0ea=1; _bs3454a553469fee5822afc67fc8a5ed0a=1; __aac=2; d7s_uid=jhi7lkqwmeiibs; wingify_donot_track_actions=0; __utma=255064090.875072536.1515447992.1528420150.1528838981.27; __utmz=255064090.1528838981.27.2.utmcsr=s1.canonrumors.com|utmccn=(referral)|utmcmd=referral|utmcct=/canon-full-frame-mirrorless-is-definitely-coming-and-the-wait-wont-be-as-long-as-we-thought/; xf_csrf=IFmJ4MhI55zWgX11; xf_user=1%2Cb8GZudHeysjQIDvMgXNol49VdQj6o6E-HYoaHZgY; xf_session=o0INazjGg7Mfiq8vzmTZvF01H0CDH0Tc; xf_session_admin=lgZGkllKEVap1fN2-SHqVe9T09saUfpE 2018-06-17 17:46:32.104195 [INFO] [136.26.11.82:63425] abort request..., code: 4 2018-06-17 17:46:32.104232 [INFO] [136.26.11.82:63425] File not found [/home/canonr/s1.canonrumors.com/503.shtml] Disabling opcode cache in PHP fixes the problem. So why is this not a problem with Apache? Apache ''suphp'' mode runs as CGI, so opcache is already not enabled. That's why it won't crash. When using LSWS with opcache enabled, Xenforo admin crashes. It is bug between Xenforo code and PHP opcode cache. It has nothing to do with LiteSpeed Web Server, hence there is not much we can do from the web server side. Your options are to either disable opcache, or try a different version of PHP and hopefully find a stable version. Also, it couldn't hurt to log a bug report with both Xenforo and PHP developers. ==== Disable unsafe third party modules (ZendGuardLoader, Suhosin, ionCube, etc.) ==== Third party modules also often have compatibility issues as well. To turn off these modules, comment out the line with said extension in your php.ini files. Restart LSWS and try the page in question again. Just as with opcode caches, if you find that a third party module is causing the error, you can try upgrading your version of the module or using a different version of PHP. If neither of these steps fixes the issue, disable the module and consider submitting a bug report to the module's developer. ==== Check disk space( your disk may be full) ==== 503 errors can also be caused by the disk your site is using being full. For example, if your site uses the ''/tmp'' directory, you will also get 503 errors. Use the ''df'' command to check disk space. Check both disk space(''df -h'') and inodes(''df -i''). ==== CloudLinux Limits Reached ===== For CloudLinux users, if a site is limited by either the amount of memory or the number of processes, then the user will receive 503 errors because the server cannot execute the script. Increasing the limits on memory and number of processes for that CageFS user may fix the problem. To understand why LVE may cause such error, please refer to [[https://docs.cloudlinux.com/index.html?lve.html|CloudLinux LVE documentation]]. If the site is limited by CPU or IO, then the site will start responding slower. If the site is limited by memory or number of processes limits, then the user will recieve 500 or 503 errors that server cannot execute the script. ==== PHP memory_limit reached ==== PHP memory_limit may be reached and you should consider to increate PHP memory_limit from php.ini or cpanel multi-PHP ini editor. ==== lsphp process memory limit hit ==== There are two types of memory limits: lsphp process memory limits and php internal memory limits. The PHP internal memory limit is normally controlled by the ''php.ini'' file and errors are normally logged in the error log. For Example, to fix ''PHP: Fatal Error: Allowed Memory Size of 536870912 Bytes Exhausted - 512 MB'', you should increase the memory limit in your ''php.ini'' or the equivalent overriding setting in ''.htaccess'' or within your php script. How to set this up is beyond the scope of this wiki. LSPHP process memory limits are defined in the lsphp external application, which may or may not be logged in the error log. You might find ''out of memory'' entries in the system log. Example 1: 2016-09-07 07:59:25.703 [STDERR] fork() failed, please increase process limit: Cannot allocate memory PHP process limits need to be increased. Example 2: dmesg | egrep -i -B100 'killed process' [11486240.579151] Out of memory in UB 1871: OOM killed process 997500 (lsphp) score 0 vm:438884kB, rss:108820kB, swap:0kB This is also a sign that php process limits were reached and need to be increased. How do you increase the PHP process limit? There are 2 LiteSpeed specific settings for PHP process limits in the LSWS Web Console: * [[https://www.litespeedtech.com/docs/webserver/config/extapps#memSoftLimit|Memory Soft Limit (bytes)]] * [[https://www.litespeedtech.com/docs/webserver/config/extapps#memHardLimit|Memory Hard Limit (bytes)]] The main purpose of this limit is to prevent excessive memory usage because of software bugs or intentional attacks, not to impose a limit on normal usage. Make sure to leave enough headroom, otherwise your application may fail and a 503 error may be returned. In a shared hosting server with memory soft/hard limits set, if a PHP script in an account consumes too much memory, it'll fail (and return a 503 error). In this case, other accounts are not affected. the ideal solution is to optimize the PHP script to consume less memory. A quick and temporary workaround is to raise the soft/hard limit to a large value (for example 8G), to see if the 503 error stops occurring. Here's an example: https://www.litespeedtech.com/support/forum/threads/solved-xenforo-rebuild-attachment-thumbnails-503.12403/ ==== PHP Max_execution_time reached ==== Faulty configurations or directives in your php.ini can cause fatal errors and make PHP processes exit early. php.ini problems generally show an "error: Connection reset by peer!" error and have a confirmed value of ''0'', meaning that the process never took the request. These errors will often be explained in your ''stderr.log''. They may require that you comment out bad directives or fix faulty configurations. Check the PHP ''max_execution_time'' configuration in a phpinfo() page to see if it is set high enough to handle long running requests. ==== The lsphp process is killed unexpectedly by CSF/LFD ==== When a PHP script is executing, if the process is killed by the admin or a process monitoring daemon, it'll result in 503 errors. === CSF/LFD Kill === A WHM/cPanel server has a "ConfigServer Security & Firewall" (i.e.,CSF / LFD) plugin installed, it kills lsphp5 processes from time to time and results in a 503 error. ''/etc/csf/csf.conf'': # This User Process Tracking option sends an alert if any cPanel user process # exceeds the time usage set (seconds). To ignore specific processes or users # use csf.pignore # # Set to 0 to disable this feature PT_USERTIME = "1800" So if an lsphp5 process has run 1800 seconds (30 minutes), it might be caught by csf and killed. In PHP Daemon mode or ProcessGroup mode, it's normal that the parent lsphp5 process keeps running after 30 minutes. When csf/lfd kills an lsphp5 process, it'll leave logs in ''/var/log/lfd.log''. These logs will look similar to the following: Jun 19 16:29:16 evo lfd[18304]: *User Processing* PID:18264 Kill:1 User:xxxxx VM:538(MB) EXE:/usr/local/lsws/fcgi-bin/lsphp-5.4.42 CMD:lsphp5 The timestamp here should match the timestamp listed with the 503 error in ''/usr/local/apache/logs/error_log'': 2015-06-19 16:29:16.370 [NOTICE] [173.245.50.197:61317-0#APVH_pingje.org] oops! 503 Service Unavailable To fix this: Add pexe:/usr/local/lsws/fcgi-bin/lsphp.* to end of ''/etc/csf/csf.pignore'' , then restart csf / lfd # csf -r or in WHM, Home » Plugins » ConfigServer Security & Firewall lfd - Login Failure Daemon "csf.pignore, Process Tracking" Edit "lfd ignore file" append following line pexe:/usr/local/lsws/fcgi-bin/lsphp.* "Restart Lfd" Note: lsphp location may vary depending on your configuration. "pexe:/path/to/lsphp.*" may change accordingly. === CSF/LFD Forkbomb === In rare cases, LSPHP will be killed by PT_FORKBOMB for having too many processes spawned at any given time. This will, in return, end up sending the signal 9/SIGKILL to the main Litespeed process as well. To stop csf/lfd from killing the LSPHP and LiteSpeed processes, the following settings inside of csf.conf should be adjusted: PT_LIMIT - Default: 10 PT_USERPROC - Default: 10 PT_FORKBOMB - Default: 0 PT_INTERVAL - Default 60 These settings will need to be adjusted on a per case basis as each scenario will be different. They should be increased slightly higher than the current values until the issue stops occurring. For the changes to take effect, both csf and lfd will need to be restarted. This can either be done via the WHM Plugin or through a shell terminal with the following command: ''csf -r; service lfd restart'' ==== Misconfiguration of "lsphp" External Application ==== During LiteSpeed Web Server installation, external applications will be automatically set to the best value for general use. However, sometimes the user intentionally removes some of these settings. This causes a misconfiguration and possibly also a 503 error. For a PHP external app, LSWS will normally add the following to it by default: 2047M 2047M 1400 1500 Imagine a user intentionally removes the above without any clear purpose. When the resource limit is not set, LSWS will use the CGI resource limit by default, but the CGI resource limit is too low to meet the requirements. If the user has configured PHP to have 500 max connections: 500 Then it must have 500 child processes: PHP_LSAPI_CHILDREN=500 The "500 max connections" setting requires "process limit" to be set at a minimum of "1000." But since the user incorrectly removed the PHP process limit settings, the above misconfiguration caused a 503 PHP error when running LSWS. We recommend that you do not to remove anything if you are unsure, as it can cause an uncertain problem, including a 503 error. ==== Apache and LiteSpeed using unmatched PHPs could cause problems for control panel users ==== If you use a control panel such as cPanel or Plesk, make sure both PHPs for Apache and LiteSpeed are matching each other. To test this, you can create a phpinfo page under your domain and switch between both web servers to check. If these PHPs do not match you could end up with 503 errors. ==== PHP Binary-Module Mismatch for old systems ==== This case may only happen to some old systems which compiling PHP. Most of the new sytem such as cPanel Easy Apache 4 use rpm/packages instead of php compling hence there should not be such issue anymore. PHP will return an error if one of your modules uses an API that does not match your PHP version. The following is an example of an error shown when there is a binary-module mismatch: Warning: PHP Startup: imap: Unable to initialize module Module compiled with module API=20090626 PHP compiled with module API=20100525 These options need to match in Unknown on line 0 To fix this, you will need to rebuild the module and/or PHP making sure that you are using a version of PHP that works with the module. Also making sure that the correct extension path is used. ==== PHP Module Loading Order ==== Sometimes, the order you load your modules in makes a difference. Switching the order your modules are listed in within your php.ini has been known to fix these issues. ==== "zend_extension" Instead of "extension" and Vice Versa ==== ''zend_extension'' is used for Zend's own extensions, such as frameworks or optimizers (like ionCube, ZendGuardLoader, or ZendOptimizer). ''extension'' is for everything else, such as PEAR, PECL, etc. For Zend's own extensions, the syntax ''zend_extension=/path/to/extensions.so'' must be used in the php.ini. For other extensions, a number of different possibilities might work: - Try ''zend_extension=/full/path/to/extension.so'' - Try ''extension=extension.so''. Where ''extension.so'' is the extension to be loaded, such as ''pdo.so'', or ''pdo_mysql.so''. When you use this syntax, make sure ''extension_dir'' is defined in the ''php.ini'' with the full path to the directory where all the extensions are found. - Comment out the ''extension_dir'' line to let PHP pick a default. ==== ''Too many open files'' error may cause 503 error ==== If you found there are a lot of ''Too many open files!'' errors, which cause 503 errors. grep "Too many open files" error_log 2019-08-01 21:58:03.511076 [INFO] [319934] [157.33.150.151:49873-1#APVH_pcrm.lidopiegade.lv:lsapi] connection to [uds://tmp/lshttpd/APVH_lidopiegade_Suea-php56.sock] on request #0, confirmed, 0, associated process: 0, running: 0, error: Too many open files! 2019-08-01 21:58:03.511078 [INFO] [319934] [uds://tmp/lshttpd/APVH_lidopiegade_Suea-php56.sock] Connection error: Too many open files, adjust maximum connections to 4! 2019-08-01 21:58:03.511085 [NOTICE] [319934] [APVH_lidopiegade_Sulsphp56:]: Failed to open pid file [/tmp/lshttpd/APVH_lidopiegade_Suea-php56.sock.pid]: Too many open files 2019-08-01 21:58:03.511087 [ERROR] [319934] [APVH_lidopiegade_Sulsphp56:]: Failed to lock pid file for [/tmp/lshttpd/APVH_lidopiegade_Suea-php56.sock]: Too many open files 2019-08-01 21:58:03.809177 [ERROR] [319934] [HTAccess] Failed to open [/home/zazzybab/public_html/girls/shannon-stewart/.htaccess]: Too many open files 2019-08-01 21:58:03.810914 [ERROR] [319934] [46.51.216.186:63901] /home/zazzybab/public_html/girls/shannon-stewart/shannon-stewart-playboy-33645.jpg: open() failed, 24:Too many open files 2019-08-01 21:58:05.072915 [NOTICE] [319934] [APVH_lidopiegade_Sulsphp56:]: Failed to open pid file [/tmp/lshttpd/APVH_lidopiegade_Suea-php56.sock.pid]: Too many open files 2019-08-01 21:58:05.072918 [ERROR] [319934] [APVH_lidopiegade_Sulsphp56:]: Failed to lock pid file for [/tmp/lshttpd/APVH_lidopiegade_Suea-php56.sock]: Too many open files 2019-08-01 21:58:05.072932 [NOTICE] [319934] [APVH_birkenfelds_Sulsphp56:]: Failed to open pid file [/tmp/lshttpd/APVH_birkenfelds_Suea-php56.sock.pid]: Too many open files 2019-08-01 21:58:05.072934 [ERROR] [319934] [APVH_birkenfelds_Sulsphp56:]: Failed to lock pid file for [/tmp/lshttpd/APVH_birkenfelds_Suea-php56.sock]: Too many open files 2019-08-01 21:58:05.072949 [NOTICE] [319934] [APVH_vairaksaules_Sulsphp56:]: Failed to open pid file [/tmp/lshttpd/APVH_vairaksaules_Suea-php56.sock.pid]: Too many open files 2019-08-01 21:58:05.072951 [ERROR] [319934] [APVH_vairaksaules_Sulsphp56:]: Failed to lock pid file for [/tmp/lshttpd/APVH_vairaksaules_Suea-php56.sock]: Too many open files You may want to check the current ulimit: ulimit -n 32768 you may consider to increase to a much higher value like: ulimit -n 3276800 Check [[litespeed_wiki:config:increasing-os-file-descriptor-limit|this wiki]] for details. ==== Magento 503 error ==== After (re)installation of LiteMage Cache extension in Magento Connect manager or enable other modules in Magento 2, Magento site may be placed in maintenancence mode and couldn't be accessed and may give 503 errors. Please refer [[litespeed_wiki:cache:litemage:troubleshooting:connect-cache-session-cleaning-error|this]] for details. ==== Some buggy application plugin may cause 503 error ==== Some buggy application plugin, such as WordPress sportspress plugin used to cause 503 errors. Uninstall all or most of WordPress or other applications' plugins, then installing them back one by one may be a good approach to locate the problematic plugin. ==== CloudLinux Imunify360 error ==== Check ''stderr.log''. It shows: *** Error in 'lsphp:/home/ukrainia/public_html/index.php': malloc(): memory corruption (fast): 0x00007fa380d45e40 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x7f574)[0x7fa37c07f574] /lib64/libc.so.6(+0x82c90)[0x7fa37c082c90] /lib64/libc.so.6(__libc_malloc+0x4c)[0x7fa37c0857dc] /opt/cpanel/ea-php54/root/usr/lib64/php/modules/i360.so(+0xb80b)[0x7fa375fbe80b] /opt/cpanel/ea-php54/root/usr/lib64/php/modules/i360.so(map_increment_int_+0xa2)[0x7fa375fbeaa2] If you try the following: **/opt/cpanel/ea-php54/root/usr/lib64/php/modules>** ''mkdir backup'' **/opt/cpanel/ea-php54/root/usr/lib64/php/modules>** ''mv i360.so backup/'' **/opt/cpanel/ea-php54/root/usr/lib64/php/modules>** ''ls -l i360.so backup/'' You will get this result: ls: cannot access i360.so: No such file or directory backup/: total 88 -rwxr-xr-x 1 root root 86568 Nov 15 12:37 i360.so The issue is fixed by disabling CloudLinux's ''i360.so''. This is a CloudLinux bug. Please be aware that it may have been fixed by the CloudLinux team by the time you read this wiki, but it was a cause of 503 errors at one time. ==== Plesk 503 error ==== Some versions of Plesk PHP, such as php71 and php56, have a buggy APCu module that may cause a 503 error on LSWS. Generally, only one object cache or value cache should be enabled. Enabling two or more may cause conflicts between modules. Disabling the APCu module should fix the problem. Navigate to **Plesk Admin > Tools and configuration > PHP configuration > 7.1.x FastCGI application**. Unselect ''apcu'' and press the **TO ACCEPT** button. {{ :litespeed_wiki:php:plesk-503-disable-apcu.png?800 |}} ==== CPanel WebDisk 503 Errors ==== You might see some errors like the following: 2019-03-12 12:14:00.324041 [INFO] [51.68.79.11:54920:proxy] connection to [127.0.0.1:2077] on request #1, confirmed, 0, associated process: 0, running: 0, error: Connection refused! 2019-03-12 12:14:00.324165 [INFO] [51.68.79.11:54920:proxy] connection to [127.0.0.1:2077] on request #1, confirmed, 0, associated process: 0, running: 0, error: Connection refused! 2019-03-12 12:14:00.324194 [NOTICE] [51.68.79.11:54920:proxy] Max retries has been reached, 503! 2019-03-12 12:14:00.324206 [INFO] [http://127.0.0.1:2077] request force stop: too many 503 errors Check ''/etc/apache2/conf/httpd.conf'': ProxyPass "/___proxy_subdomain_webdisk" "http://127.0.0.1:2077" max=1 retry=0 The errors are coming from cPanel WebDisk service. If this service generates too many 503 errors, you might consider disabling it. To disable cPanel WebDisk, you can log into WHM and disable **cPanel DAV Daemon** in the **Service Manager**. ==== chroot is Enabled in LSWS ===== It is not recommended to use ''chroot'' at all. If enabled, it may lead to a 503 error. To fix the problem, in the LiteSpeed Web Server WebAdmin Console, navigate to **Server > General** and set **Enable chroot** to ''No''. ===== Advanced Troubleshooting Guide ===== The above comprehensive guide lists the most likely cases which might lead to 503 errors. In some circumstances, it could be very difficult to locate the root cause of the errors. If PHP crashes, you can enable a core dump and use gdb to debug the core file. You will need to be clear that even if you find some internal PHP bug which has caused the crash, there are limited things you can do about it. You can disable the module or change to another PHP version which may not have such a bug. Reporting the bug to PHP could be an option, but you may not get a solution in a timely manner. If there is a core file generated and you've tried all of the above methods without reaching a solution, you can try to strace the PHP process for a hint. Just be prepared that the strace log could be very very long and overwhelming. You likely need some experience and knowledge to understand what the log entries mean and it can be a challenge even to locate the relevant section of the log. ==== Strace the PHP process ==== You can use the following command to ''strace'' the PHP process to find out what happened. strace -tt -T -f -p For example: 2015-10-26 16:19:28.873 [INFO] [34.154.77.34:48506] connection to [/tmp/lshttpd/APVH_allier_Suphp.sock] on request #0, confirmed, 1, associated process: 791228, running: 1, error: Connection reset by peer! 2015-10-26 16:19:29.399 [INFO] [34.154.77.34:48506] connection to [/tmp/lshttpd/APVH_allier_Suphp.sock] on request #0, confirmed, 1, associated process: 791228, running: 1, error: Connection reset by peer! 2015-10-26 16:19:29.923 [INFO] [34.154.77.34:48506] connection to [/tmp/lshttpd/APVH_allier_Suphp.sock] on request #0, confirmed, 1, associated process: 791228, running: 1, error: Connection reset by peer! 2015-10-26 16:19:29.923 [NOTICE] [34.154.77.34:48506] Max retries has been reached, 503! 2015-10-26 16:19:29.923 [NOTICE] [34.154.77.34:48506] oops! 503 Service Unavailable PHP process ''791228'' caused the 503 error. ''strace'' that process to find out why. strace -tt -T -f -p 791228 Sometimes, it may not be easy to get the ''pid'' since the PHP may run very quickly. In this case, you can try to strace the //parent pid//, but the disadvantage of stracing the parent pid is that you may get too much unrelated information in strace log. That would make it difficult to locate the problem pattern. One solution is to use a script to catch the pid. It depends on which SuEXEC user your PHP will be running and which script will be running ( such as index.php etc). while true; do if mypid=`ps aux | grep $USERNAME | grep lsphp | grep $SCRIPTNAME | grep -v grep | awk '{print $2; }' | tail -1`; then strace -tt -T -f -p $mypid; fi ; done **Note**: ''$USERNAME'' should be replaced with a real username; ''$SCRIPTNAME'' should be replaced by the name of the script that is running, such as ''index.php'' or other scripts. ==== Enable Core Dump ==== If you find that PHP has crashed (as demonstrated by the process being ''killed by signal''), a core dump will allow you to look further into the cause of the crash. To enable a core dump, add the environment value LSAPI_ALLOW_CORE_DUMP=1 to your external application settings (WebAdmin console > Configuration > External App). The next time the application crashes, a core dump will be generated. The core file created can usually be found in the directory holding the PHP script affected. Your Redhat/Centos system may already use ABRT(Automatic Bug Reporting Tool) to generate core dump files. In this case, core files may located in the following directory: /var/spool/abrt/ ABRT configuration files are normally located in the following directory: /etc/abrt/. Please make sure that ''ProcessUnpackaged = yes'' is in /etc/abrt/abrt-action-save-package-data.conf to enable the core dump, otherwise a core file may not be generated even if the web server error log says one was. If you want to disable core file dumps after the debugging, simply change ''ProcessUnpackaged = yes'' back to ''ProcessUnpackages = no''. ''core_pattern'' is used to specify a core dump file pattern name. cat /proc/sys/kernel/core_pattern This shows you the pattern template for the output filename. It may be as simple as core If the first character of the pattern is a ''|'', the kernel will treat the rest of the pattern as a command to run. The core dump will be written to the standard input of that program instead of to a file. By default, ''/proc/sys/kernel/core_pattern'' contains ''core'' and the kernel produces ''core.*'' files in the crashed process's current directory. ABRT packages(Abrt’s C/C++ hook) overwrite the ''/proc/sys/kernel/core_pattern'' file to: |/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e which results in kernel calling ''abrt-hook-ccpp'' **Note:** As noted below, opcode caches are frequently the cause of PHP crashes. If you find that your PHP has crashed, you may want to try turning off any opcode caching you have. This is addressed further [[#opcode_caches_apc_xcache_eaccelerateor|below]]. If, after adding the ''LSAPI_ALLOW_CORE_DUMP'' environment variable, ''stderr.log'' still has entries like: Child process with pid: 22549 was killed by signal: 11, core dump: 0 Then the core dump was not created. Possible causes of this are: * Permissions conflicts - The user running LSPHP doesn't have write permission for the directory the primary PHP script is located in. \\ \\ **For example:** \\ \\ If you check the user who is running LSPHP using \\ \\ ps aux | grep lsphp and get a result like: nobody 60358 0.0 0.0 2434836 752 s000 S+ 4:17 PM 0:00.00 grep lsphp Then the user ''nobody'' is running LSPHP. \\ \\ Navigate to the directory containing the primary PHP script and input: ls -ld . You should get something like: \\ -rwxr-xr-x 3 root root 102 Dec 21 16:47 . In this case, ''root:root'' owns the folder and its permissions are 755. Make note of the owner and these permissions because for security reasons we will be changing these back later. \\ \\ There are two ways to fix the issue under these circumstances. Either by navigating to the directory above the primary PHP script (its parent directory) and changing its owner to ''nobody'' with \\ chown nobody . or by changing the permissions of the directory to give everyone full control with \\ chmod 777 . Regardless of which of these solutions you choose, remember to revert the changes. ==== Analyzing Core Files with The GNU Debugger ==== GNU Debugger (GDB) uses the syntax ''gdb ''. (Your LSPHP binary can usually be found in ''/usr/local/lsws/fcgi-bin''. If you installed LSPHP through a LiteSpeed Repository, LSPHP binary files are usually located in the following directory: ''/usr/local/lsws/lsphp5x/lsphp''(where ''x'' is the PHP version: ''3'' ''4'' ''5'' or ''6'')). Once you have opened the core file with GDB, use the ''bt'' command to print a backtrace of the stack (the steps that the application took leading up to the crash). This will often reveal what caused the crash. Here is a sample backtrace from a program that crashed: Program received signal SIGSEGV, Segmentation fault. 0x000000000061c91b in ?? () (gdb) bt #0 0x000000000061c91b in ?? () #1 0x0000000000641ac3 in zend_stack_push () #2 0x000000000060bbf9 in ?? () #3 0x0000000000611515 in lex_scan () #4 0x000000000061fb60 in ?? () #5 0x0000000000608223 in ?? () #6 0x0000000000614965 in compile_file () #7 0x00007fd3c99eda21 in ?? () from /opt/alt/php55/usr/lib64/php/modules/phar.so #8 0x00007fd3d0993359 in ?? () from /opt/alt/php55/usr/lib64/php/modules/opcache.so #9 0x00007fd3d0994187 in ?? () from /opt/alt/php55/usr/lib64/php/modules/opcache.so #10 0x00000000006b54da in ?? () #11 0x00000000006b68e8 in execute_ex () #12 0x000000000064239c in zend_execute_scripts () #13 0x00000000005e2fb0 in php_execute_script () #14 0x00000000006f29ff in ?? () #15 0x00000000006f2c5c in ?? () #16 0x00000000006f2f85 in ?? () #17 0x00007fd3d37cdd1d in __libc_start_main () from /lib64/libc.so.6 #18 0x0000000000424bc9 in _start () This backtrace shows us that the program crashed soon after accessing the opcode cache. This suggests that the issue is with the opcode cache. The user can then try upgrading the opcode cache or changing their PHP version. If neither of those work, the user should try to disable the opcode cache. ===== The "Auto Fix 503 Error" Feature ===== In LiteSpeed Web Server's WebAdmin Console, there is a feature to **Auto Fix 503 Error**. It specifies whether to try to fix the "503 Service Unavailable" error by restarting the server gracefully. A "503" error is usually caused by malfunctioning external applications and a web server restart can often fix the error temporarily. If **Auto Fix 503 Error** is set to ''Yes'', the server will restart automatically whenever there are more than 30 "503" errors within a 30 seconds span. This feature is enabled by default, but you can disable it by navigating to **LSWS Admin -> Server -> General** and setting **Auto Fix 503 Error** to ''No''. {{ :litespeed_wiki:php:auto-fix-503-error.png?800 |}} **Auto Fix 503 Error** is not a permanent fix. You should not rely on this feature to fix recurring 503 errors. Going through the above steps to find a permanent fix is always recommended. ===== Real World Examples ===== [[litespeed_wiki:php:503-errors:example-1|Bad Directive in php.ini]] The above example uses the first two troubleshooting steps — searching the error log and stderr.log. [[http://trulymanaged.com/blog/error-after-installing-ioncube-on-plesk/|ionCube Loader Path not Correct]] The above example shows the kind of error you might get if your extension path is not correct. [[litespeed_wiki:php:503-errors:example-2|Opcode Cache Crashes PHP]] The above example shows the use of GDB to show that opcode cache was causing a crash. [[litespeed_wiki:php:503-errors:example-3|Opcode Cache Crashes PHP 2]] The above example shows another error that was solved by turning off the opcode cache. [[https://www.litespeedtech.com/support/forum/showthread.php?t=5545|ZenGuardLoader Bug]] The above example shows a case where shutting off ZendGuardLoader solved an issue until Zend was able to fix the bug. [[https://www.litespeedtech.com/support/forum/showthread.php?t=2636|ionCube Wants eAccelerator Loaded as an Extension]] The above example shows how switching ''zend_extension'' for ''extension'' can sometimes solve an issue. [[https://www.litespeedtech.com/support/forum/showthread.php?t=2262|PHP Figures Out Extension Locations by Itself]] The above example shows a case where leaving ''extension_dir'' undefined solved an issue.