【原创】大叔经验分享（98）mesos slave启动失败

mesos slave启动失败，查看状态如下：

# systemctl status mesos-slave
● mesos-slave.service - Mesos Slave
   Loaded: loaded (/usr/lib/systemd/system/mesos-slave.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since Sat 2019-12-28 21:41:50 CST; 13s ago
  Process: 15627 ExecStart=/usr/bin/mesos-init-wrapper slave (code=exited, status=1/FAILURE)
 Main PID: 15627 (code=exited, status=1/FAILURE)

Dec 28 21:41:50 test-003 systemd[1]: Unit mesos-slave.service entered failed state.
Dec 28 21:41:50 test-003 systemd[1]: mesos-slave.service failed.

查看mesos-slave日志如下：

# journalctl -u mesos-slave -f -n 300
...
Dec 28 21:42:10 test-003 mesos-slave[15974]: I1228 21:42:10.604262 15978 group.cpp:831] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
Dec 28 21:42:10 test-003 mesos-slave[15974]: I1228 21:42:10.604274 15978 group.cpp:419] Trying to create path ‘/mesos‘ in ZooKeeper
Dec 28 21:42:10 test-003 mesos-slave[15974]: I1228 21:42:10.602254 15961 slave.cpp:615] Agent resources: [{"name":"ports","ranges":{"range":[{"begin":80,"end":60000}]},"type":"RANGES"},{"name":"cpus","scalar":{"value":8.0},"type":"SCALAR"},{"name":"mem","scalar":{"value":30987.0},"type":"SCALAR"},{"name":"disk","scalar":{"value":95544.0},"type":"SCALAR"}]
Dec 28 21:42:10 test-003 mesos-slave[15974]: I1228 21:42:10.605845 15961 slave.cpp:623] Agent attributes: [  ]
Dec 28 21:42:10 test-003 mesos-slave[15974]: I1228 21:42:10.605868 15961 slave.cpp:632] Agent hostname: test003
Dec 28 21:42:10 test-003 mesos-slave[15974]: I1228 21:42:10.605935 15977 task_status_update_manager.cpp:181] Pausing sending task status updates
Dec 28 21:42:10 test-003 mesos-slave[15974]: I1228 21:42:10.606037 15982 detector.cpp:152] Detected a new leader: (id=‘79‘)
Dec 28 21:42:10 test-003 mesos-slave[15974]: I1228 21:42:10.606160 15981 group.cpp:700] Trying to get ‘/mesos/json.info_0000000079‘ in ZooKeeper
Dec 28 21:42:10 test-003 mesos-slave[15974]: I1228 21:42:10.607014 15975 state.cpp:66] Recovering state from ‘/var/lib/mesos/meta‘
Dec 28 21:42:10 test-003 mesos-slave[15974]: I1228 21:42:10.607070 15975 state.cpp:742] No committed checkpointed resources found at ‘/var/lib/mesos/meta/resources/resources.info‘
Dec 28 21:42:10 test-003 mesos-slave[15974]: I1228 21:42:10.607249 15981 zookeeper.cpp:262] A new leading master ([email protected]:5050) is detected
Dec 28 21:42:10 test-003 mesos-slave[15974]: I1228 21:42:10.646075 15979 slave.cpp:6951] Finished recovering checkpointed state from ‘/var/lib/mesos/meta‘, beginning agent recovery
Dec 28 21:42:10 test-003 mesos-slave[15974]: E1228 21:42:10.649549 15979 slave.cpp:7311] EXIT with status 1: Failed to perform recovery: Incompatible agent info detected.
Dec 28 21:42:10 test-003 mesos-slave[15974]: ecovery
Dec 28 21:42:10 test-003 mesos-slave[15974]: ------------------------------------------------------------
Dec 28 21:42:10 test-003 mesos-slave[15974]: Old agent info:
Dec 28 21:42:10 test-003 mesos-slave[15974]: hostname: "test003"
Dec 28 21:42:10 test-003 mesos-slave[15974]: resources {
Dec 28 21:42:10 test-003 mesos-slave[15974]:   name: "ports"
Dec 28 21:42:10 test-003 mesos-slave[15974]:   type: RANGES
Dec 28 21:42:10 test-003 mesos-slave[15974]:   ranges {
Dec 28 21:42:10 test-003 mesos-slave[15974]:     range {
Dec 28 21:42:10 test-003 mesos-slave[15974]:       begin: 80
Dec 28 21:42:10 test-003 mesos-slave[15974]:       end: 60000
Dec 28 21:42:10 test-003 mesos-slave[15974]:     }
Dec 28 21:42:10 test-003 mesos-slave[15974]:   }
Dec 28 21:42:10 test-003 mesos-slave[15974]: }
Dec 28 21:42:10 test-003 mesos-slave[15974]: resources {
Dec 28 21:42:10 test-003 mesos-slave[15974]:   name: "cpus"
Dec 28 21:42:10 test-003 mesos-slave[15974]:   type: SCALAR
Dec 28 21:42:10 test-003 mesos-slave[15974]:   scalar {
Dec 28 21:42:10 test-003 mesos-slave[15974]:     value: 8
Dec 28 21:42:10 test-003 mesos-slave[15974]:   }
Dec 28 21:42:10 test-003 mesos-slave[15974]: }
Dec 28 21:42:10 test-003 mesos-slave[15974]: resources {
Dec 28 21:42:10 test-003 systemd[1]: mesos-slave.service: main process exited, code=exited, status=1/FAILURE
Dec 28 21:42:10 test-003 systemd[1]: Unit mesos-slave.service entered failed state.
Dec 28 21:42:10 test-003 systemd[1]: mesos-slave.service failed.

注意关键的几行

Dec 28 21:42:10 test-003 mesos-slave[15974]: I1228 21:42:10.646075 15979 slave.cpp:6951] Finished recovering checkpointed state from ‘/var/lib/mesos/meta‘, beginning agent recovery
Dec 28 21:42:10 test-003 mesos-slave[15974]: E1228 21:42:10.649549 15979 slave.cpp:7311] EXIT with status 1: Failed to perform recovery: Incompatible agent info detected.

尝试从/var/lib/mesos/meta恢复，但是失败了，然后进程退出，

# rm -rf /var/lib/mesos/meta/*

将meta目录删除之后再启动mesos slave成功，问题解决；

原文地址：https://www.cnblogs.com/barneywill/p/12178500.html

时间： 2024-10-03 01:01:53

【原创】大叔经验分享（98）mesos slave启动失败的相关文章

【原创】大叔经验分享（57）hue启动coordinator时报错

hue启动coordinator时报错,页面返回undefinied错误框: 后台日志报错: runcpserver.log [13/May/2019 04:34:55 -0700] middleware INFO Processing exception: 'NoneType' object has no attribute 'is_superuser': Traceback (most recent call last): File "/opt/cloudera/parcels/CDH-5.

【原创】大叔经验分享（96）docker启动MySQL报错

docker启动mysql反复重启,通过 # docker logs $container_id 查看日志如下: chown: cannot read directory '/var/lib/mysql/': Permission denied 但是目录权限确认没问题,即使 chmod 777 还是报相同的错误,后来发现是selinux的问题查看状态 # getenforce 临时禁用 # setenforce 0 永久禁用 # vi /etc/selinux/config # SELINUX

【原创】大叔经验分享（23）hive metastore的几种部署方式

hive及其他组件(比如spark.impala等)都会依赖hive metastore,依赖的配置文件位于hive-site.xml hive metastore重要配置 hive.metastore.warehouse.dirhive2及之前版本默认为/user/hive/warehouse/,创建数据库或表时会在该目录下创建对应的目录 javax.jdo.option.ConnectionURLjavax.jdo.option.ConnectionDriverNamejavax.jdo.o

【原创】大叔经验分享（35）lzo格式支持

建表语句 CREATE EXTERNAL TABLE `my_lzo_table`(`something` string)ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' STORED AS INPUTFORMAT 'com.hadoop.mapred.DeprecatedLzoTextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputForma

【原创】大叔经验分享（39）spark cache unpersist级联操作

问题:spark中如果有两个DataFrame(或者DataSet),DataFrameA依赖DataFrameB,并且两个DataFrame都进行了cache,将DataFrameB unpersist之后,DataFrameA的cache也会失效,官方解释如下: When invalidating a cache, we invalid other caches dependent on this cache to ensure cached data is up to date. For

【原创】大叔经验分享（52）ClouderaManager修改配置报错

Cloudera Manager中修改配置可能报错: Incorrect string value: '\xE7\xA8\x8B\xE5\xBA\x8F...' for column 'MESSAGE' at row 1 这是一个mysql的字符集问题,极有可能创建scm数据库时使用默认的latin1编码导致,涉及的表为: CREATE TABLE `REVISIONS` ( `REVISION_ID` bigint(20) NOT NULL, `OPTIMISTIC_LOCK_VERSION`

【原创】大叔经验分享（53）kudu报错unable to find SASL plugin: PLAIN

kudu安装后运行不正常,master中找不到任何tserver,查看tserver日志发现有很多报错: Failed to heartbeat to master:7051: Invalid argument: Failed to ping master at master:7051: Client connection negotiation failed: client connection to master:7051: unable to find SASL plugin: PLAIN

【原创】大叔经验分享（55）hue导出行数限制

/opt/cloudera/parcels/CDH/lib/hue/apps/beeswax/src/beeswax/conf.py # Deprecated DOWNLOAD_CELL_LIMIT = Config( key='download_cell_limit', default=10000000, type=int, help=_t('A limit to the number of cells (rows * columns) that can be downloaded from

【原创】大叔经验分享（58）kudu写入压力大时报错

kudu写入压力大时报错 19/05/18 16:53:12 INFO AsyncKuduClient: Invalidating location fd52e4f930bc45458a8f29ed118785e3(server002:7050) for tablet 4259921cdcca4776b37771659a8cafb3: Service unavailable: Soft memory limit exceeded (at 106.05% of capacity). See htt