Apache: (28)No space left on device: Couldn't create accept lock

[Sun Sep 21 17:37:10 2014] [emerg] (28)No space left on device: Couldn't create accept lock (/var/log/httpd/accept.lock.8411) (5)

下午收到监控邮件有台DirectAdmin服务器Apache服务无法启动，检查错误日记后，发现如上错误。首先df -h检查硬盘是否饱和，然后ipcs -s检查ipc，发现是ipc不足。如下所示

# ipcs -s 
------ Semaphore Arrays -------- 
key semid owner perms nsems 
0x00000000 19234816 apache 600 1 
0x00000000 19267585 apache 600 1 
0x00000000 19300354 apache 600 1 
0x00000000 19398659 apache 600 1 
0x00000000 19431428 apache 600 1 
0x00000000 19464197 apache 600 1 
0x00000000 19562502 apache 600 1
………………

执行如下命令清除然后重启Apache即可

ipcs -s | grep apache | perl -lane 'print `ipcrm sem $F[1]`'
service httpd restart

引起这个问题的原因可能是apache没有被正确的关闭。写了个简单的shell可以加入crontab定期执行

#!/bin/bash
rm -rf /var/log/weed/ipcs.log
ipcs -s >> /var/log/weed/ipcs.log
ipcslist=`grep -c "" /var/log/weed/ipcs.log`
if [ "$ipcslist" -ge "20" ]; then
ipcs -s | perl -ane '/^0x00000000/ && `ipcrm -s $F[1]`'
echo $(date) "Ipc crowded, clean up" >> /var/log/weed/ipcs_clean.log
fi