nova instance状态:power_state, vm_state, task_state
2015-09-22 Openstack 185
nova instance有3种状态:power_state, vm_state, task_state,分别对应horizon界面上的Power State,Status,Task
Openstack wiki上有介绍:
- power_state is the hypervisor state, loaded “bottom-up” from compute worker;
- vm_state reflects the stable state based on API calls, matching user expectation, revised “top-down” within API implementation.
- task_state reflects the transition state introduced by in-progress API calls.
- power_state and vm_state may conflict with each other, which needs to be resolved case-by-case.
Power_state
power_state反映的是hypervisor的状态。目前都是用libvirt作为hypervisor。
例如,nova会调用libvirt api去查询instance的state
[[email protected] ~]# virsh list --all
Id Name State
----------------------------------------------------
229 instance-00000139 running
230 instance-00000138 running
231 instance-00000137 running
232 instance-00000155 running
- instance-00000136 shut off
- instance-0000013a shut off
[[email protected] ~]# virsh domstate instance-00000136
shut off
在libvirt中,共有几种状态:
- undefined
- defined(也叫stopped)
- running
- paused
- saved
具体解释在这里
下图表示在几种状态的转换关系:
Notes:
- 有的转换依赖于domain(instance)是transient domain还是persistent domain, 如running ---(shutdown)---> defined/undefined。两者的区别在这里。思考:Openstack创建的domain是否都是persistent domain?
virsh create
创建transient domain;virsh define;virsh start
创建、启动persistent domain
How is it updated?
- 调hypervisor api取得的状态总是对的;如果nova数据库中的状态与其不一致,那么更新nova数据库。
- nova有定时任务来检查、更新power_state
- 如果对instance做了操作(task),并且该操作可能影响power_state,那么在该task结束之前需要更新power_state
power_state list
目前(2015/9/21 master)有6种状态:
# nova/compute/power_state.py # NOTE(maoy): These are *not* virDomainState values from libvirt. # The hex value happens to match virDomainState for backward-compatibility # reasons. NOSTATE = 0x00 RUNNING = 0x01 PAUSED = 0x03 SHUTDOWN = 0x04 # the VM is powered off CRASHED = 0x06 SUSPENDED = 0x07 # TODO(justinsb): Power state really needs to be a proper class, # so that we‘re not locked into the libvirt status codes and can put mapping # logic here rather than spread throughout the code STATE_MAP = { NOSTATE: ‘pending‘, RUNNING: ‘running‘, PAUSED: ‘paused‘, SHUTDOWN: ‘shutdown‘, CRASHED: ‘crashed‘, SUSPENDED: ‘suspended‘, }
Vm_state
vm_state反映基于API调用的一种稳定状态,而task_state反映的是一种瞬时状态。
How is it updated?
- 只有当一个task(一个compute api call)结束时,才能更新vm_state。
- 如果成功则同时设置task_state=None,如果失败,vm_state可能不变(task支持rollback),也可能设置为ERROR。
vm_state list
# nova/compute/vm_states.py ACTIVE = ‘active‘ # VM is running BUILDING = ‘building‘ # VM only exists in DB PAUSED = ‘paused‘ SUSPENDED = ‘suspended‘ # VM is suspended to disk. STOPPED = ‘stopped‘ # VM is powered off, the disk image is still there. RESCUED = ‘rescued‘ # A rescue image is running with the original VM image attached. RESIZED = ‘resized‘ # a VM with the new size is active. The user is expected to manually confirm or revert. SOFT_DELETED = ‘soft-delete‘ # VM is marked as deleted but the disk images are still available to restore. DELETED = ‘deleted‘ # VM is permanently deleted. ERROR = ‘error‘ SHELVED = ‘shelved‘ # VM is powered off, resources still on hypervisor SHELVED_OFFLOADED = ‘shelved_offloaded‘ # VM and associated resources are not on hypervisor ALLOW_SOFT_REBOOT = [ACTIVE] # states we can soft reboot from ALLOW_HARD_REBOOT = ALLOW_SOFT_REBOOT + [STOPPED, PAUSED, SUSPENDED, ERROR] # states we allow hard reboot from
Task_state
task_state反映了基于API调用的瞬时状态。
它表示instance当前处于某个API调用中的某个阶段,比vm_state更细化,让用户了解某个action/task执行到哪个步骤了,比如创建instance时会有scheduling -> block_device_mapping -> networking -> spawning
。但调用完之后设置为None。
task_state list
# nova/compute/task_states.py """Possible task states for instances. urrent moment. These tasks can be generic, such as ‘spawning‘, or specific, such as ‘block_device_mapping‘. These task states allow for a better view into what an instance is doing and should be displayed to users/administrators as necessary. """ # possible task states during create() SCHEDULING = ‘scheduling‘ BLOCK_DEVICE_MAPPING = ‘block_device_mapping‘ NETWORKING = ‘networking‘ SPAWNING = ‘spawning‘ # possible task states during snapshot() IMAGE_SNAPSHOT = ‘image_snapshot‘ IMAGE_SNAPSHOT_PENDING = ‘image_snapshot_pending‘ IMAGE_PENDING_UPLOAD = ‘image_pending_upload‘ IMAGE_UPLOADING = ‘image_uploading‘ # possible task states during backup() IMAGE_BACKUP = ‘image_backup‘ # possible task states during set_admin_password() UPDATING_PASSWORD = ‘updating_password‘ # possible task states during resize() RESIZE_PREP = ‘resize_prep‘ RESIZE_MIGRATING = ‘resize_migrating‘ RESIZE_MIGRATED = ‘resize_migrated‘ RESIZE_FINISH = ‘resize_finish‘ # possible task states during revert_resize() RESIZE_REVERTING = ‘resize_reverting‘ # possible task states during confirm_resize() RESIZE_CONFIRMING = ‘resize_confirming‘ # possible task states during reboot() REBOOTING = ‘rebooting‘ REBOOT_PENDING = ‘reboot_pending‘ REBOOT_STARTED = ‘reboot_started‘ REBOOTING_HARD = ‘rebooting_hard‘ REBOOT_PENDING_HARD = ‘reboot_pending_hard‘ REBOOT_STARTED_HARD = ‘reboot_started_hard‘ # possible task states during pause() PAUSING = ‘pausing‘ # possible task states during unpause() UNPAUSING = ‘unpausing‘ # possible task states during suspend() SUSPENDING = ‘suspending‘ # possible task states during resume() RESUMING = ‘resuming‘ # possible task states during power_off() POWERING_OFF = ‘powering-off‘ # possible task states during power_on() POWERING_ON = ‘powering-on‘ # possible task states during rescue() RESCUING = ‘rescuing‘ # possible task states during unrescue() UNRESCUING = ‘unrescuing‘ # possible task states during rebuild() REBUILDING = ‘rebuilding‘ REBUILD_BLOCK_DEVICE_MAPPING = "rebuild_block_device_mapping" REBUILD_SPAWNING = ‘rebuild_spawning‘ # possible task states during live_migrate() MIGRATING = "migrating" # possible task states during delete() DELETING = ‘deleting‘ # possible task states during soft_delete() SOFT_DELETING = ‘soft-deleting‘ # possible task states during restore() RESTORING = ‘restoring‘ # possible task states during shelve() SHELVING = ‘shelving‘ SHELVING_IMAGE_PENDING_UPLOAD = ‘shelving_image_pending_upload‘ SHELVING_IMAGE_UPLOADING = ‘shelving_image_uploading‘ # possible task states during shelve_offload() SHELVING_OFFLOADING = ‘shelving_offloading‘ # possible task states during unshelve() UNSHELVING = ‘unshelving‘
task_state state machine diagram (in-progress):
https://docs.google.com/spreadsheets/d/1uvrFI_L86_tBcZGlE2ck3RnMgtsgjIqYVtWTIT9eD8I/edit#gid=3
http://docs.openstack.org/developer/nova/devref/vmstates.html
http://docs.openstack.org/developer/nova/vmstates.html
Code walk-through
nova-compute定义了如下文件:
- nova/compute/power_state.py
- nova/compute/vm_states.py
- nova/compute/task_states.py
nova-api从自身角度定义了vm_state和task_state的对应关系
- nova/api/openstack/common.py
from nova.compute import task_states from nova.compute import vm_states _STATE_MAP = { vm_states.ACTIVE: { ‘default‘: ‘ACTIVE‘, task_states.REBOOTING: ‘REBOOT‘, # 前面是task_state, 后面是显示在cli/horizon上的‘Status‘ task_states.REBOOT_PENDING: ‘REBOOT‘, task_states.REBOOT_STARTED: ‘REBOOT‘, task_states.REBOOTING_HARD: ‘HARD_REBOOT‘, task_states.REBOOT_PENDING_HARD: ‘HARD_REBOOT‘, task_states.REBOOT_STARTED_HARD: ‘HARD_REBOOT‘, task_states.UPDATING_PASSWORD: ‘PASSWORD‘, task_states.REBUILDING: ‘REBUILD‘, task_states.REBUILD_BLOCK_DEVICE_MAPPING: ‘REBUILD‘, task_states.REBUILD_SPAWNING: ‘REBUILD‘, task_states.MIGRATING: ‘MIGRATING‘, task_states.RESIZE_PREP: ‘RESIZE‘, task_states.RESIZE_MIGRATING: ‘RESIZE‘, task_states.RESIZE_MIGRATED: ‘RESIZE‘, task_states.RESIZE_FINISH: ‘RESIZE‘, }, vm_states.BUILDING: { ‘default‘: ‘BUILD‘, }, vm_states.STOPPED: { ‘default‘: ‘SHUTOFF‘, task_states.RESIZE_PREP: ‘RESIZE‘, task_states.RESIZE_MIGRATING: ‘RESIZE‘, task_states.RESIZE_MIGRATED: ‘RESIZE‘, task_states.RESIZE_FINISH: ‘RESIZE‘, task_states.REBUILDING: ‘REBUILD‘, task_states.REBUILD_BLOCK_DEVICE_MAPPING: ‘REBUILD‘, task_states.REBUILD_SPAWNING: ‘REBUILD‘, }, vm_states.RESIZED: { ‘default‘: ‘VERIFY_RESIZE‘, # Note(maoy): the OS API spec 1.1 doesn‘t have CONFIRMING_RESIZE # state so we comment that out for future reference only. #task_states.RESIZE_CONFIRMING: ‘CONFIRMING_RESIZE‘, task_states.RESIZE_REVERTING: ‘REVERT_RESIZE‘, }, vm_states.PAUSED: { ‘default‘: ‘PAUSED‘, task_states.MIGRATING: ‘MIGRATING‘, }, vm_states.SUSPENDED: { ‘default‘: ‘SUSPENDED‘, }, vm_states.RESCUED: { ‘default‘: ‘RESCUE‘, }, vm_states.ERROR: { ‘default‘: ‘ERROR‘, task_states.REBUILDING: ‘REBUILD‘, task_states.REBUILD_BLOCK_DEVICE_MAPPING: ‘REBUILD‘, task_states.REBUILD_SPAWNING: ‘REBUILD‘, }, vm_states.DELETED: { ‘default‘: ‘DELETED‘, }, vm_states.SOFT_DELETED: { ‘default‘: ‘SOFT_DELETED‘, }, vm_states.SHELVED: { ‘default‘: ‘SHELVED‘, }, vm_states.SHELVED_OFFLOADED: { ‘default‘: ‘SHELVED_OFFLOADED‘, }, } def status_from_state(vm_state, task_state=‘default‘): """Given vm_state and task_state, return a status string.""" task_map = _STATE_MAP.get(vm_state, dict(default=‘UNKNOWN‘)) status = task_map.get(task_state, task_map[‘default‘]) if status == "UNKNOWN": LOG.error(_LE("status is UNKNOWN from vm_state=%(vm_state)s " "task_state=%(task_state)s. Bad upgrade or db " "corrupted?"), {‘vm_state‘: vm_state, ‘task_state‘: task_state}) return status
从status_from_state
方法看,_STATE_MAP
表示vm_state处于某个状态时,所有可能的task_state有哪些,都穷举出来了。这里注意,某些task_state的default值,在nova/compute/task_states.py中没有定义,如:
vm_states.ACTIVE: {
‘default‘: ‘ACTIVE‘,
...}
vm_states.STOPPED: {
‘default‘: ‘SHUTOFF‘,
...}
vm_states.RESIZED: {
‘default‘: ‘VERIFY_RESIZE‘,
...}
vm_states.ERROR: {
‘default‘: ‘ERROR‘,
...}
给定一对vm_state/task_state,status_from_state
能返回一个status字符串,这个status就是cli/horizon上的‘Status‘:
[root@node-129 ~]# nova list | cut -d"|" -f1-6 +--------------------------------------+-------------------------------------------------------+---------+------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ID | Name | Status | Task State | Power State +--------------------------------------+-------------------------------------------------------+---------+------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+ | f6d4e67b-885c-4a10-891f-d91675e52bae | 123 | SHUTOFF | - | Shutdown | 05be07c9-6b2a-4db9-b03b-8322c035816c | 652 | SHUTOFF | - | Shutdown | 19d1308c-f19b-4860-979f-220fd376ef58 | ITMGMT | SHUTOFF | - | Shutdown | c496c710-29f8-46c3-90f8-6c2606d4b252 | Winboo-vm | ACTIVE | - | Running | a27f4030-ece5-4aeb-a84e-70bf0389f81f | abc | ACTIVE | - | Running
比较一下,nova-api列出的"Status"其实比nova-compute的vm_state(14个)多,共20个:
- ACTIVE
- REBOOT
- HARD_REBOOT
- PASSWORD
- REBUILD
- MIGRATING
- RESIZE
- BUILD
- SHUTOFF
- VERIFY_RESIZE
- REVERT_RESIZE
- PAUSED
- SUSPENDED
- RESCUE**
- ERROR
- DELETED
- SOFT_DELETED**
- SHELVED
- SHELVED_OFFLOADED**
- UNKNOWN
状态转移
参考:
- http://docs.openstack.org/developer/nova/devref/vmstates.html
- http://docs.openstack.org/developer/nova/vmstates.html
不同的操作,只有当instance处于特定的vm_state和task_state时才能进行。这个控制不是由nova-api来完成的,而是由nova-compute来决定的:nova/compute/api.py
定义了nova-compute暴露给外部的接口。大多数方法名称就是操作名,如create
,restore
, shelve
, soft_delete
。
基本上所有方法都用check_instance_state
修饰了。这个修饰器限定了该操作在某些vm_state/task_state下才可以进行。比如:
@wrap_check_policy @check_instance_lock @check_instance_state(vm_state=[vm_states.SOFT_DELETED]) def restore(self, context, instance): """Restore a previously deleted (but not reclaimed) instance.""" @wrap_check_policy @check_instance_cell @check_instance_state(vm_state=[vm_states.ACTIVE, vm_states.STOPPED, vm_states.PAUSED, vm_states.SUSPENDED]) def snapshot(self, context, instance, name, extra_properties=None): """Snapshot the given instance.""" @wrap_check_policy @check_instance_lock @check_instance_state(vm_state=set( vm_states.ALLOW_SOFT_REBOOT + vm_states.ALLOW_HARD_REBOOT), task_state=[None, task_states.REBOOTING, task_states.REBOOT_PENDING, task_states.REBOOT_STARTED, task_states.REBOOTING_HARD, task_states.RESUMING, task_states.UNPAUSING, task_states.PAUSING, task_states.SUSPENDING]) def reboot(self, context, instance, reboot_type): """Reboot the given instance."""
所以只要查看check_instance_state
的参数就能确定操作可行与否。