Recovering from a failed bhyve VM deploy/stuck VM (FreeBSD)

Sometimes deploying a VM on a bhyve host fails, and you may get errors like errno = 37 when powering off the VM. It seems like the easiest way to fix this is a host reboot but on the contrary, it is possible to recover from this easily and reliably.

First, there will be a few files left over that will have to go. Obviously you will lose state but at this point you aren't likely to recover it. Delete these files:

• /dev/vmm/vmname
• /path/to/dataset/vmname/run.lock

Once you have done that you can vm destroy -f vmname, but this will not destroy the ZFS dataset if you are using a sparse zvol. Figure out which shambling PID is still holding a lock on the zvol and kill it:

# fuser /dev/zvol/data/vm/vmname/disk0
/dev/zvol/data/vm/vmname/disk0: 20847w
# kill -9 20847

You can now destroy the ZFS dataset and zvol and redeploy.