Apache and Mongrel frustrations: 502, 503, Operation Not Permitted, Proxy Error, & Connection Refused Errors

imarichardson — Fri, 30 May 2008 16:38:53 +0000

For a good deal of time I’ve been experiencing numerous problems relating to mongrel and mongrel_cluster. I’ve received proxy errors from my main site:

Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /.

Reason: Error reading from remote server

Apache/2.2.3 (Red Hat) Server at [myserver.com] Port 443

And when viewing the log files I was receiving errors like:

[Mon May 26 23:51:38 2008] [error] [client xxx.xxx.xxx.xxx] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): *
[Wed May 28 08:59:49 2008] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8050 (127.0.0.1) failed
[Wed May 28 08:59:49 2008] [error] ap_proxy_connect_backend disabling worker for (127.0.0.1)
[Wed May 28 08:59:49 2008] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8051 (127.0.0.1) failed
...
[Wed May 28 09:00:39 2008] [error] proxy: BALANCER: (balancer://url_cluster). All workers are in error state
[Fri May 30 08:15:10 2008] [error] [client xxx.xxx.xxx.xxx] proxy: error reading status line from remote server 127.0.0.1
[Fri May 30 08:15:10 2008] [error] [client xxx.xxx.xxx.xxx] proxy: Error reading from remote server returned by / ...

And this, too: (I tried these instructions too, but didn’t work.)

[Thu May 29 03:47:24 2008] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8000 (127.0.0.1) failed
[Thu May 29 03:47:24 2008] [error] ap_proxy_connect_backend disabling worker for (127.0.0.1)
[Thu May 29 03:47:24 2008] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8001 (127.0.0.1) failed
[Thu May 29 03:47:24 2008] [error] ap_proxy_connect_backend disabling worker for (127.0.0.1)

Additionally, I received “Operation Not Permitted” errors when trying to restart and/or stop the mongrel clusters:

** [out :: 208.112.64.164] Operation not permitted (Errno::EPERM)
** [out :: 208.112.64.164] from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.4/bin/mongrel_rails:200:in `send_signal'
** [out :: 208.112.64.164] from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.4/bin/mongrel_rails:240:in `run'
** [out :: 208.112.64.164] from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.4/bin/../lib/mongrel/command.rb:212:in `run'
** [out :: 208.112.64.164] from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.4/bin/mongrel_rails:281
** [out :: 208.112.64.164] from /usr/local/bin/mongrel_rails:19:in `load'
** [out :: 208.112.64.164] from /usr/local/bin/mongrel_rails:19
** [out :: 208.112.64.164] Sending TERM to Mongrel at PID 18429...

Solution

I was able to resolve the issue by killing all the active Mongrel processes and then restarting manually. You can do this by:

1. Viewing all active mongrel processes:

@imarichardson]$ ps -aef | egrep mongrel
root 3156 1 0 May15 ? 00:00:03 /usr/bin/ruby /usr/bin/mongrel_rails start -d -e production -p 8090 -a 127.0.0.1 -P log/mongrel.8090.pid -c /var/www/apps/[app_directory]/current
root 3159 1 0 May15 ? 00:00:02 /usr/bin/ruby /usr/bin/mongrel_rails start -d -e production -p 8091 -a 127.0.0.1 -P log/mongrel.8091.pid -c /var/www/apps/[app_directory]/current

(The second number is the process id; that is the id you’ll use to kill the process.)

2. Killing each process id

“sudo kill -9 [process id]“, For example, to kill the first process listed above:

@imarichardson]$ sudo kill -9 8090

3. Restarting your Mongrel instances

@imarichardson]$ sudo mongrel_rails cluster::restart --clean -C /var/www/apps/[app_directory]/current/config/mongrel_cluster.yml

I'm a Richardson, RU? » troubleshooting

Apache and Mongrel frustrations: 502, 503, Operation Not Permitted, Proxy Error, & Connection Refused Errors

Solution