Stray rails processes

[gwhite]

LSWS enterprise 3.2.4
Fedore core Linux

I am noticing some stray rails processes that don't appear to be attached to the parent processes at all. When running strace on them I see this output:
accept(0, <unfinished ...>
I do run a restart (lswsctrl restart) at midnight in conjunction with rotating the rails application production log so I assumed they weren't being closed properly from that but on a server where I have turned that off I get the same symptoms. I haven't been able to correlate it with an application deploy restart at all either. Has anyone seen something like this?

Graham

 

[mistwang]

It rarely happen but it could happen when the rails process blocked at accept() call, and the server lost the tracking of the process pid for some reason.
We will try to find a better solution to avoid rails process being blocked with accept() on top what we have done already.

 

[gwhite]

Hi just wondering if you guys had looked into this further at all? I hadn't seen anything in the latest updates. Thanks

 

[mistwang]

Have you updated to the latest ruby-lsapi release?

 

[gwhite]

Hi yes, running with ruby-lsapi 3.1, still seems to be happening. Looking at the process tree they are just running without being attached to any parent. No cpu usage but a little bit of memory. Running strace on one of the pids still returns:
accept(0,
Running with litespeed 3.3.8 currently

 

[mistwang]

Thanks for the update.
I think we will change the listening socket to non-blocking to avoid being stucked there.

 

[mistwang]

Please try our ruby-lsapi 3.2 release which using non-blocking listener socket, ruby process will not be blocked at accept() call any more.

 

[gwhite]

Looks good so far! I'll let you know if anything changes. Thanks

 

[teej_m]

I have a similar issue, where I have ruby-lsapi processes that get spawned, but then stop handling requests. It's started to become an issue, because they hold on to their MySQL connection. At this point, I'm not sure if it's a misconfiguration issue, or some sort of bug. I've got a cron job killing all the ruby processes and restarting litespeed as a stopover, but the issue only gets worse as my traffic ramps up. Any guidance in helping debug the issue would be appreciated!

 

[mistwang]

You need to strace the ruby processes that stop working.

 

[teej_m]

I'll have to say sorry for crying wolf on this one. I tried reinstalling the ruby-lsapi gem (just in case it was using the old one) and I managed to get it working from there. I'll stop by if it seems to be causing issues again!

 

[teej_m]

Change of plans. At the moment I have have a few Ruby-lsapi processes that are using cpu and memory and a mysql connection open. Mysql shows that the connection has been idle for 15+ minutes now. There are a few processes like this, and when strace'd, they show nothing happening.

 

[mistwang]

Maybe they are in a infinity loop.
You can try gdb attach to those processes.

gdb
attach <pid_of_ruby>
bt

Also check
http://www.litespeedtech.com/support/wiki/doku.php?id=litespeed_wiki:ruby_debug

 

[teej_m]

Debugging with gdb has pointed me in the right direction. Thanks for your help.