Lot of VMs are stuck in "Investigate me"

[09:54:34] <tosaraja> we seem to be gettina  a lot of "investiagte me" VMs even though the host seems to be up and running ok. for example worthy-mite has 2 VMs running ok and the rest are in investigate_me
[09:54:44] <tosaraja> i can't just delete them either, as it will cause mac address conflicts
[09:57:59] <frgladho> tosaraja: you can delete them, just not through the opennebula web ui, because that ui says: remove vm from db but leave it running...
...
[10:27:13] <tosaraja> ok, by manually just deleting the VMs with virsh, worthy-mite again has capacity for new VMs and it succesfully already launched one as well
[10:27:31] <tosaraja> so there's now something wrong somewhere in the process, but it's not the host dying on us this time
...

...
[10:30:32] <frgladho> interesting, I did: recover-retry and that succeeded for  INVESTIGATE_ME_415363
[10:30:40] <frgladho> it was stuck in prolog state or so
[10:31:04] <frgladho> this means we need to be more subtle in what we do to re-try/re-cover vms