This week I’m hosting a hands-on OpenStack training for some clients. The ability to perform Live migrations of running instances between hosts is one of the things they want to see and I had setup the environment to support this.
Live migrations had been working fine for over a week when it finally decided to throw errors this morning.
The error on the command line when trying to do a live migration:
Normally this would happen if the hosts running nova-compute had different CPU types, but in this case they are all identical (Dell C6320 nodes).
Checked the CPU map in /usr/share/libvirt/cpu_map.xml and the CPU is listed.
Since the CPU’s are the same on all nodes it’s obviously the lookup of that CPU type that fails. So, I tried to disable the check by editing /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py. This resulted in the error disappearing but my instances staying put on whatever host they were originally running on. Not much better.
Finally I started modifying the /etc/nova/nova.conf files on both the controller node and the nova-compute nodes. The changes that fixed it were as follows:
I also have the following settings which may or may not matter in this case:
After restarting nova on both the controller and all three compute nodes, live migrations are working fine again. Not sure why they stopped in the first place, but at least this seems to have done the job.
Checking instances for each node:
Performing the migration:
Verifying that the instance has moved from node2 to node4: