内核3.10，systemctl start docker 被阻塞，没有返回，查看状态为启动中。
某兄弟机器安装docker之后，发现systemctl start docker的时候阻塞，由于排查走了一些弯路，记录如下：
level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
level=info msg="libcontainerd: new containerd process, pid: 46803"
level=warning msg="Docker could not enable SELinux on the host system"
level=info msg="Graph migration to content-addressability took 0.00 seconds"
level=info msg="Loading containers: start."
level=warning msg="Running modprobe nf_nat failed with message: ``, error: exec: \"modprobe\": executable file not found in $PATH"
level=warning msg="Running modprobe xt_conntrack failed with message: ``, error: exec: \"modprobe\": executable file not found in $PATH"
level=info msg="Firewalld running: false"
Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain: iptables failed: iptables --wait -t nat -N DOCKER: iptables v
Perhaps iptables or your kernel needs to be upgraded.
(exit status 3)
docker.service: main process exited, code=exited, status=1/FAILURE
Failed to start Docker Application Container Engine.
根据错误记录，确定是创建iptable的链路规则失败，然后查看iptables --list，会报 获取锁失败，
[[email protected] ~]# iptables --list
Another app is currently holding the xtables lock. Perhaps you want to use the -w option
这种情况一般是前面拿锁写规则的iptables进程没有返回，ps -ef 查看对应的进程，发现如下：
[[email protected] ~]# ps -ef |grep -i iptables
root 14967 14926 0 20:05 ? 00:00:00 /usr/sbin/iptables --wait -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
iptables进程确实没有返回，
查看对应的堆栈和内核代码，确定nat模块需要依赖对应的conntrack模块：
int nf_nat_l3proto_register(const struct nf_nat_l3proto *l3proto)
{
int err;
err = nf_ct_l3proto_try_module_get(l3proto->l3proto);
然后看对应为什么没有加载nf_conntrack-2，发现该环境上的nf_conntrack-2 被backlist了。
另外注意到一个很有趣的问题，在打点测试的时候，发现如下代码：
nf_ct_l3proto_try_module_get(unsigned short l3proto)
{
int ret;
struct nf_conntrack_l3proto *p;
retry: p = nf_ct_l3proto_find_get(l3proto);
if (p == &nf_conntrack_l3proto_generic) {
ret = request_module("nf_conntrack-%d", l3proto);
if (!ret)
goto retry;
return -EPROTOTYPE;
}
return 0;
}
这里retry应该是有问题的，如果request的nf_conntrack模块被backlist，则会出现一直不退出的情况，而这个流程中会不停提交work_struct到workqueue中，大量的无效work被执行。
原文地址：https://www.cnblogs.com/10087622blog/p/12127258.html