当我正在工作时，开发的一个员工发微信告诉我，阿里云一台服务登录很慢，登录进去操作很卡顿；
我就立刻登录进去，查看了下进程：（ps -aux）这台有nginx（有反代），还有mysql，握草，还有个tomcat，这台配置很多低，还跑了那么多服务，
虽然是拿来测试用的，也扛不住你们这样糟蹋啊，我（top）了下，尼玛，好几秒才出现；先不管，看那个服务占用CPU资源的，看了下负载（load average: 1.19, 1.39, 1.37）
正常啊。这台配置是：CPU：2核，内存：4G，20M带宽，因为这台nginx用来存放APP中的H5静态页面和图片，带宽大些。
然后我就看下nginx的错误日志，看到了问题：2048 worker_connections are not enough 有很多这样，
又看了nginx.conf配置：
#user nobody;
worker_processes 3;
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#pid logs/nginx.pid;
events {
use epoll;
worker_connections 2048;
}
又看了下阿里云控制台监控，CPU负载有一点点高，为什么和我在远程登录top查看的不一样啊，好纠结，最后参考了阿里云的监控，
我把上面的配置标记的地方增加下，worker_connections 2048 改了4096，增加一倍，
再看日志，没有再报了 2048 worker_connections are not enough
出现这样的了
2018/01/03 17:22:57 [alert] 9795#0: worker process 5924 exited on signal 9
2018/01/03 17:23:20 [alert] 9795#0: worker process 5929 exited on signal 9
查看内核信息（/var/log/messages）:
Jan 3 17:37:31 debug010000002015 kernel: lowmem_reserve[]: 0 0 0 0 Jan 3 17:37:31 debug010000002015 kernel: Node 0 DMA: 2*4kB 2*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15752kB Jan 3 17:37:31 debug010000002015 kernel: Node 0 DMA32: 545*4kB 439*8kB 243*16kB 218*32kB 14*64kB 58*128kB 84*256kB 6*512kB 3*1024kB 1*2048kB 0*4096kB = 54572kB Jan 3 17:37:31 debug010000002015 kernel: Node 0 Normal: 2952*4kB 8*8kB 12*16kB 84*32kB 3*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 16864kB Jan 3 17:37:31 debug010000002015 kernel: 723 total pagecache pages Jan 3 17:37:31 debug010000002015 kernel: 0 pages in swap cache Jan 3 17:37:31 debug010000002015 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jan 3 17:37:31 debug010000002015 kernel: Free swap = 0kB Jan 3 17:37:31 debug010000002015 kernel: Total swap = 0kB Jan 3 17:37:31 debug010000002015 kernel: 1048575 pages RAM Jan 3 17:37:31 debug010000002015 kernel: 67471 pages reserved Jan 3 17:37:31 debug010000002015 kernel: 63109 pages shared Jan 3 17:37:31 debug010000002015 kernel: 916409 pages non-shared Jan 3 17:37:31 debug010000002015 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name Jan 3 17:37:31 debug010000002015 kernel: [ 477] 0 477 2705 147 0 -17 -1000 udevd Jan 3 17:37:31 debug010000002015 kernel: [ 1109] 0 1109 23283 76 0 -17 -1000 auditd Jan 3 17:37:31 debug010000002015 kernel: [ 1131] 0 1131 62369 1538 0 0 0 rsyslogd Jan 3 17:37:31 debug010000002015 kernel: [ 1448] 38 1448 7653 135 1 0 0 ntpd Jan 3 17:37:31 debug010000002015 kernel: [ 1539] 0 1539 20226 228 1 0 0 master Jan 3 17:37:31 debug010000002015 kernel: [ 1554] 89 1554 20289 224 0 0 0 qmgr Jan 3 17:37:31 debug010000002015 kernel: [ 1559] 0 1559 29218 154 0 0 0 crond Jan 3 17:37:31 debug010000002015 kernel: [ 1576] 0 1576 5277 46 0 0 0 atd Jan 3 17:37:31 debug010000002015 kernel: [ 1601] 0 1601 396 68 1 0 0 aliyun-service Jan 3 17:37:31 debug010000002015 kernel: [ 1614] 0 1614 17457 154 0 0 0 login Jan 3 17:37:31 debug010000002015 kernel: [ 1616] 0 1616 1016 22 1 0 0 mingetty Jan 3 17:37:31 debug010000002015 kernel: [ 1618] 0 1618 2661 103 1 -17 -1000 udevd Jan 3 17:37:31 debug010000002015 kernel: [ 1619] 0 1619 1016 22 1 0 0 mingetty Jan 3 17:37:31 debug010000002015 kernel: [ 1621] 0 1621 1016 22 0 0 0 mingetty Jan 3 17:37:31 debug010000002015 kernel: [ 1623] 0 1623 1016 21 0 0 0 mingetty Jan 3 17:37:31 debug010000002015 kernel: [ 1625] 0 1625 1016 22 0 0 0 mingetty Jan 3 17:37:31 debug010000002015 kernel: [10105] 0 10105 16559 182 0 -17 -1000 sshd Jan 3 17:37:31 debug010000002015 kernel: [ 9795] 500 9795 7158 335 0 0 0 nginx Jan 3 17:37:31 debug010000002015 kernel: [19524] 0 19524 2280 123 0 0 0 dhclient Jan 3 17:37:31 debug010000002015 kernel: [31947] 0 31947 16624 164 1 0 0 saslauthd Jan 3 17:37:31 debug010000002015 kernel: [31948] 0 31948 16624 164 1 0 0 saslauthd Jan 3 17:37:31 debug010000002015 kernel: [31949] 0 31949 16624 164 0 0 0 saslauthd Jan 3 17:37:31 debug010000002015 kernel: [31950] 0 31950 16624 164 1 0 0 saslauthd Jan 3 17:37:31 debug010000002015 kernel: [31951] 0 31951 16624 164 0 0 0 saslauthd Jan 3 17:37:31 debug010000002015 kernel: [ 1175] 0 1175 2661 103 1 -17 -1000 udevd Jan 3 17:37:31 debug010000002015 kernel: [25593] 500 25593 27111 116 0 0 0 bash Jan 3 17:37:31 debug010000002015 kernel: [15156] 0 15156 29165 160 0 0 0 wrapper Jan 3 17:37:31 debug010000002015 kernel: [11631] 0 11631 27077 79 0 0 0 mysqld_safe Jan 3 17:37:31 debug010000002015 kernel: [11876] 27 11876 421620 120678 1 0 0 mysqld Jan 3 17:37:31 debug010000002015 kernel: [ 4190] 0 4190 909659 118199 0 0 0 java Jan 3 17:37:31 debug010000002015 kernel: [21183] 0 21183 7684 173 1 0 0 AliYunDunUpdate Jan 3 17:37:31 debug010000002015 kernel: [21235] 0 21235 32183 1526 1 0 0 AliYunDun Jan 3 17:37:31 debug010000002015 kernel: [ 3026] 0 3026 626630 14437 0 0 0 java Jan 3 17:37:31 debug010000002015 kernel: [ 4553] 0 4553 25640 258 1 0 0 sshd Jan 3 17:37:31 debug010000002015 kernel: [ 4555] 500 4555 25640 262 1 0 0 sshd Jan 3 17:37:31 debug010000002015 kernel: [ 4556] 500 4556 27111 117 0 0 0 bash Jan 3 17:37:31 debug010000002015 kernel: [ 4578] 500 4578 25238 26 0 0 0 tail Jan 3 17:37:31 debug010000002015 kernel: [ 5161] 0 5161 25640 257 0 0 0 sshd Jan 3 17:37:31 debug010000002015 kernel: [ 5163] 500 5163 25640 257 0 0 0 sshd Jan 3 17:37:31 debug010000002015 kernel: [ 5164] 500 5164 27111 126 0 0 0 bash Jan 3 17:37:31 debug010000002015 kernel: [ 5421] 0 5421 25640 258 0 0 0 sshd Jan 3 17:37:31 debug010000002015 kernel: [ 5423] 500 5423 25640 258 1 0 0 sshd Jan 3 17:37:31 debug010000002015 kernel: [ 5424] 500 5424 27111 125 0 0 0 bash Jan 3 17:37:31 debug010000002015 kernel: [ 6188] 500 6188 36018 966 0 0 0 python Jan 3 17:37:31 debug010000002015 kernel: [ 6190] 500 6190 283830 277086 0 0 0 nginx Jan 3 17:37:31 debug010000002015 kernel: [ 6194] 500 6194 219105 212345 0 0 0 nginx Jan 3 17:37:31 debug010000002015 kernel: [ 6199] 500 6199 117088 110346 0 0 0 nginx Jan 3 17:37:31 debug010000002015 kernel: [ 6200] 0 6200 25640 263 1 0 0 sshd Jan 3 17:37:31 debug010000002015 kernel: [ 6202] 500 6202 25640 264 0 0 0 sshd Jan 3 17:37:31 debug010000002015 kernel: [ 6203] 500 6203 27111 119 0 0 0 bash Jan 3 17:37:31 debug010000002015 kernel: [ 6222] 500 6222 36887 108 0 0 0 su Jan 3 17:37:31 debug010000002015 kernel: [ 6223] 0 6223 27111 120 1 0 0 bash Jan 3 17:37:31 debug010000002015 kernel: [ 6262] 0 6262 25238 30 0 0 0 tail Jan 3 17:37:31 debug010000002015 kernel: Out of memory: Kill process 6190 (nginx) score 282 or sacrifice child Jan 3 17:37:31 debug010000002015 kernel: Killed process 6190, UID 500, (nginx) total-vm:1135320kB, anon-rss:1108148kB, file-rss:196kB
原因是系统内存不可用导致进程挂掉。
内存不够的原因，只有升级阿里云主机配置了，反应给我上级，等待升级，后续。。。。。。
原文地址：https://www.cnblogs.com/chenheping/p/8184738.html
时间： 2024-10-18 01:05:09