TPOV

Every now and then, when you least expect it, your Linux machine will stop or crash. This morning the Apache webserver on one of my Linux ervers, running Red Hat Enterprise Linux 5 Server, gave way. Since my primary concern has always been with Windows, my first idea was to reboot the box. Like in Windows, that ‘solves’ a lot of problems with Linux as well.

 But before I did, I checked Apache’s error_log (located, by default, in /var/log/httpd). The last lines read;

[emerg] (28)No space left on device: Couldn’t create accept lock (/etc/httpd/logs/accept.lock.10897) (5)

Okay, plenty of disk space available (checked that by using df -h). So that’s not real cause of the problem. (Don’t you just love these misleading error messages?)

 The problem appeared to be in the lock file itself; it just couldn’t be created and hence Apache ‘thinks’ it ran out of disk space when in fact the problem is that Apache couldn’t create a semaphore to create the file.

So I checked the semaphores on the system:

 # ipcs -s
—— Semaphore Arrays ——–
key semid owner perms nsems
0×00000000 68681743 apache 600 1
0×00000000 68714515 apache 600 1
0×00000000 68747291 apache 600 1

This is ‘strange’, since Apache isn’t running anymore. So let’s just kill these semaphores:

$ ipcrm -s <semid>

But there are just too many semaphores to delete all of them manually, one by one. A little one liner will do the trick though:

for semid in `ipcs -s | grep apachec | cut -f2 -d” “`; do ipcrm -s $semid; done

Now apache can be started again. But of course, chances are that you will be running out of semaphores again if you run a busy box. You can increase the amount of semaphores by editing the sysctl.conf file in the /etc directory. Add these lines:

kernel.msgmni = 1024
kernel.sem = 250 256000 32 1024

Then, you can load these new values with this command:

sysctl -p

If you want to check your semaphore limits, you can issue this command:

# ipcs -l
—— Shared Memory Limits ——–
max number of segments = 4096
max seg size (kbytes) = 32768
max total shared memory (kbytes) = 8388608
min seg size (bytes) = 1

—— Semaphore Limits ——–
max number of arrays = 1024
max semaphores per array = 250
max semaphores system wide = 256000
max ops per semop call = 32
semaphore max value = 32767

—— Messages: Limits ——–
max queues system wide = 1024
max size of message (bytes) = 8192
default max size of queue (bytes) = 16384

Oh well… It’s ‘solved’ for now. That was a refreshing start of a new day.