vmm: Only return from reset driven I/O once event received

The reset system is asynchronous with an I/O event (PIO or MMIO) for
ACPI/i8042/CMOS triggering a write to the reset_evt event handler. The
VMM thread will pick up this event on the VMM main loop and then trigger
a shutdown in the CpuManager. However since there is some delay between
the CPU threads being marked to be killed (through the
CpuManager::cpus_kill_signalled bool) it is possible for the guest vCPU
that triggered the exit to be re-entered when the vCPU KVM_RUN is called
after the I/O exit is completed.

This is undesirable and in particular the Linux kernel will attempt to
jump to real mode after a CMOS based exit - this is unsupported in
nested KVM on AMD on Azure and will trigger an error in KVM_RUN.

Solve this problem by spinning in the device that has triggered the
reset until the vcpus_kill_signalled boolean has been updated
indicating that the VMM thread has received the event and called
CpuManager::shutdown(). In particular if this bool is set then the vCPU
threads will not re-enter the guest.

Signed-off-by: Rob Bradford <rbradford@rivosinc.com>

vmm/src/cpu.rs

@@ -1725,6 +1725,10 @@ impl CpuManager {
     ) {
         self.interrupt_controller = Some(interrupt_controller);
     }
+
+    pub(crate) fn vcpus_kill_signalled(&self) -> &Arc<AtomicBool> {
+        &self.vcpus_kill_signalled
+    }
 }
 
 struct Cpu {

vmm/src/device_manager.rs

@@ -1485,8 +1485,16 @@ impl DeviceManager {
         reset_evt: EventFd,
         exit_evt: EventFd,
     ) -> DeviceManagerResult<Option<Arc<Mutex<devices::AcpiGedDevice>>>> {
+        let vcpus_kill_signalled = self
+            .cpu_manager
+            .lock()
+            .unwrap()
+            .vcpus_kill_signalled()
+            .clone();
         let shutdown_device = Arc::new(Mutex::new(devices::AcpiShutdownDevice::new(
-            exit_evt, reset_evt,
+            exit_evt,
+            reset_evt,
+            vcpus_kill_signalled,
         )));
 
         self.bus_devices
@@ -1585,9 +1593,16 @@ impl DeviceManager {
 
     #[cfg(target_arch = "x86_64")]
     fn add_legacy_devices(&mut self, reset_evt: EventFd) -> DeviceManagerResult<()> {
+        let vcpus_kill_signalled = self
+            .cpu_manager
+            .lock()
+            .unwrap()
+            .vcpus_kill_signalled()
+            .clone();
         // Add a shutdown device (i8042)
         let i8042 = Arc::new(Mutex::new(devices::legacy::I8042Device::new(
             reset_evt.try_clone().unwrap(),
+            vcpus_kill_signalled.clone(),
         )));
 
         self.bus_devices
@@ -1615,6 +1630,7 @@ impl DeviceManager {
                 mem_below_4g,
                 mem_above_4g,
                 reset_evt,
+                vcpus_kill_signalled,
             )));
 
             self.bus_devices