vmm: add memory reserve option to opt out of MAP_NORESERVE

Cloud Hypervisor maps guest RAM with MAP_NORESERVE, so the kernel never
reserves the backing pages at mmap time. On a host whose hugepage pool
cannot satisfy every guest, a VM is created successfully and then takes
a SIGBUS when the guest faults a page the pool can no longer back. This
is the failure mode reported in #5730 and #7387. As noted on #5730,
checking free pool headroom up front is not a reliable fix: another
process can consume pages between the check and the fault.

Add a reserve=on parameter to --memory and --memory-zone (default off,
preserving the current MAP_NORESERVE behaviour). When set, the region
is mapped without MAP_NORESERVE, so the kernel reserves the backing
pages (swap, or huge pages for hugepage-backed memory) at mmap time,
atomically with the mapping. An over-committed configuration then
fails cleanly at VM creation with an mmap ENOMEM instead of crashing
the guest later. Unlike prefault it does not fault the memory in, so
it does not slow down boot.

This mirrors QEMU's memory-backend reserve property, which has the same
name and meaning (reserve=off maps with MAP_NORESERVE). reserve is
threaded through the same mmap paths as the existing prefault option,
and is exposed in the OpenAPI schema, CLI help and docs. The top-level
--memory reserve=on path is unchanged: the default zone is synthesised
from MemoryConfig and inherits its reserve value.

Assisted-by: Claude Code (Opus 4.8)
Signed-off-by: Ian Klemm <hi@ianklemm.de>

docs/memory.md

@@ -20,13 +20,14 @@ struct MemoryConfig {
     hugepages: bool,
     hugepage_size: Option<u64>,
     prefault: bool,
+    reserve: bool,
     thp: bool,
     zones: Option<Vec<MemoryZoneConfig>>,
 }
 ```
 
 ```
---memory <memory>	Memory parameters "size=<guest_memory_size>,mergeable=on|off,shared=on|off,hugepages=on|off,hugepage_size=<hugepage_size>,hotplug_method=acpi|virtio-mem,hotplug_size=<hotpluggable_memory_size>,hotplugged_size=<hotplugged_memory_size>,prefault=on|off,thp=on|off" [default: size=512M,thp=on]
+--memory <memory>	Memory parameters "size=<guest_memory_size>,mergeable=on|off,shared=on|off,hugepages=on|off,hugepage_size=<hugepage_size>,hotplug_method=acpi|virtio-mem,hotplug_size=<hotpluggable_memory_size>,hotplugged_size=<hotplugged_memory_size>,prefault=on|off,reserve=on|off,thp=on|off" [default: size=512M,thp=on]
 ```
 
 ### `size`
@@ -177,6 +178,32 @@ _Example_
 --memory size=1G,prefault=on
 ```
 
+### `reserve`
+
+Specifies if guest memory should be `mmap(2)`-ed _without_ the `MAP_NORESERVE`
+flag, asking the kernel to reserve the backing pages (swap space, or huge pages
+for hugepage-backed memory) for the whole region up front at `mmap` time.
+
+By default Cloud Hypervisor maps guest memory with `MAP_NORESERVE`, so VM
+creation succeeds even when the backing pool cannot satisfy the full guest size.
+The shortfall then surfaces only later, as a `SIGBUS` delivered to the guest when
+it faults a page the pool cannot back. With `reserve=on` the reservation is made
+when the memory is mapped, so an over-committed configuration fails cleanly at VM
+creation with an out-of-memory error instead of crashing the guest at run time.
+This is most useful for hugepage-backed memory, where the huge pages are reserved
+from the pool.
+
+Unlike `prefault`, this does not populate or fault in the memory, so it does not
+slow down boot; it only reserves it.
+
+By default this option is turned off.
+
+_Example_
+
+```
+--memory size=1G,hugepages=on,reserve=on
+```
+
 ### `thp`
 
 Specifies if private anonymous memory for the guest (i.e. `shared=off` and no
@@ -214,12 +241,13 @@ struct MemoryZoneConfig {
     hotplug_size: Option<u64>,
     hotplugged_size: Option<u64>,
     prefault: bool,
+    reserve: bool,
     mergeable: bool,
 }
 ```
 
 ```
---memory-zone <memory-zone>	User defined memory zone parameters "size=<guest_memory_region_size>,file=<backing_file>,shared=on|off,hugepages=on|off,hugepage_size=<hugepage_size>,host_numa_node=<node_id>,id=<zone_identifier>,hotplug_size=<hotpluggable_memory_size>,hotplugged_size=<hotplugged_memory_size>,prefault=on|off,mergeable=on|off"
+--memory-zone <memory-zone>	User defined memory zone parameters "size=<guest_memory_region_size>,file=<backing_file>,shared=on|off,hugepages=on|off,hugepage_size=<hugepage_size>,host_numa_node=<node_id>,id=<zone_identifier>,hotplug_size=<hotpluggable_memory_size>,hotplugged_size=<hotplugged_memory_size>,prefault=on|off,reserve=on|off,mergeable=on|off"
 ```
 
 This parameter expects one or more occurrences, allowing for a list of memory
@@ -423,6 +451,34 @@ _Example_
 --memory-zone id=mem0,size=1G,prefault=on
 ```
 
+### `reserve`
+
+Specifies if the memory for this zone should be `mmap(2)`-ed _without_ the
+`MAP_NORESERVE` flag, asking the kernel to reserve the backing pages (swap space,
+or huge pages for hugepage-backed memory) for the whole zone up front at `mmap`
+time.
+
+By default Cloud Hypervisor maps guest memory with `MAP_NORESERVE`, so VM
+creation succeeds even when the backing pool cannot satisfy the full zone size.
+The shortfall then surfaces only later, as a `SIGBUS` delivered to the guest when
+it faults a page the pool cannot back. With `reserve=on` the reservation is made
+when the memory is mapped, so an over-committed configuration fails cleanly at VM
+creation with an out-of-memory error instead of crashing the guest at run time.
+This is most useful for hugepage-backed zones, where the huge pages are reserved
+from the pool.
+
+Unlike `prefault`, this does not populate or fault in the memory, so it does not
+slow down boot; it only reserves it.
+
+By default this option is turned off.
+
+_Example_
+
+```
+--memory size=0
+--memory-zone id=mem0,size=1G,hugepages=on,reserve=on
+```
+
 ### `mergeable`
 
 Specifies if the pages from this memory zone must be marked as _mergeable_,