* feat(compiler): support registered host-await builtins Allow hosts to register function names at compile time so that calls to those names emit HostAwait instructions directly, enabling natural syntax like fetch(x) instead of __builtin_host_await(x, "fetch"). - Add host_await_builtins map and register_host_await_builtin() to Compiler - Validate arg_count == 1 and reject reserved __builtin_host_await name - Extend determine_call_target() resolution: explicit > registered > user > builtin - Both explicit and registered paths emit identical HostAwait bytecode - Add compile_from_policy_with_host_await() entry point in rules.rs - Extended test harness with HostAwaitBuiltinSpec and args assertion - 9 YAML test cases: suspend/resume, run-to-completion, multiple names, queue, shadowing, object packing, arg_count rejection, reserved name rejection, standard builtin override - Documentation: instruction-set.md, architecture.md * Update src/languages/rego/compiler/function_calls.rs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Mark Birger <birgerm@yandex.ru> * fix(compiler): address PR #667 review feedback on host-await registration - Compiler::register_host_await_builtin now rejects duplicate, empty, and whitespace-only names. Previously a duplicate registration would silently overwrite the existing entry, which could mask the host's own registration mistakes. - YAML test cases added: empty registration list as no-op, duplicate name rejection, empty/whitespace name rejection, out-param (a, out) calling syntax with a single-arg registered builtin, and mixed __builtin_host_await + registered builtins in the same policy consuming from their respective identifier queues. - Test harness: replace assert_eq! on HostAwait argument mismatch with anyhow::Error so mismatches propagate through the case reporter instead of panicking and skipping the harness's normal error path. - YAML comment fix: "Registration panics" -> "Registration fails with an error" (registration returns Err, never panics). Addresses anakrish + Copilot inline review comments on PR #667. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * compiler: split CallTarget::HostAwait into explicit and registered variants Addresses PR #667 review item #8: at the emit site in `compile_function_call`, the discrimination between explicit `__builtin_host_await(arg, id)` and a registered host-awaitable builtin was being recovered by string-comparing `original_fcn_path` against `"__builtin_host_await"`. The information was already known in `determine_call_target` and was being thrown away. Replace the single `CallTarget::HostAwait` variant with two: * `ExplicitHostAwait` (unit) — the two-argument call form. The identifier register comes from the user's second argument. * `RegisteredHostAwait { identifier: String }` — the one-argument call form for registered builtins. The identifier is the registered name and is captured in the variant at recognition time, so the emit site never re-derives it from the function path. This removes the magic-string comparison at the emit site (the source of truth is now `determine_call_target`) and makes both match sites in `compile_function_call` exhaustive over the two forms — adding a third host-await form in the future would force a compile error at every match site instead of silently falling through. Arities are now hardcoded in the `expected_args` extraction (`Some(2)` for explicit, `Some(1)` for registered) rather than carried in the variant; registered builtins are constrained to `arg_count == 1` at registration time, so there is no per-call variability to carry. Bytecode output is unchanged; the full RVM test suite (97 cases) and the registered_host_await suite (15 cases) pass without modification. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(compiler): clarify registered host-await intercepts unqualified calls only PR #667 review (Medium): the docs implied registered host-await names shadow user functions and builtins unconditionally, but determine_call_target matches only the bare original_fcn_path. A package-qualified call such as data.demo.resolve(x) is therefore not intercepted -- it resolves through the normal path like any other call. Rather than expand registration to qualified paths (which would let a registered name leak into every package exposing a same-named rule), document the unqualified-only behavior and pin it with tests. - register_host_await_builtin: doc now states only the unqualified call form is intercepted; qualified calls resolve normally. - determine_call_target: inline comment explaining the deliberate original_fcn_path-only match. - docs/rvm/instruction-set.md: describe qualified-call resolution, including that builtins have no qualified form. - tests: cross-package and same-package qualified calls resolve to the rule; bare-name shadowing of a standard builtin; Unknown-function outcome when no rule exists at the qualified path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(tests): compare host-await argument without re-running process_value PR #667 review (Low): the suspendable test harness compared the host-await argument via process_value(argument), but argument is already a runtime Value. process_value is a YAML-fixture decoder -- it rewrites "#undefined" to Undefined, {set!: [...]} to a set, and errors on a runtime Value::Set. Re-running it on the runtime argument could coerce a legitimate payload into a fixture sentinel (passing for the wrong reason) or error outright on sets. Compare the runtime argument directly against the expected value, which is already decoded once at YAML load time. Add a regression case (registered_builtin_suspendable_set_argument) that passes a set payload: it fails under the old double-processing ("unexpected set in value read from json/yaml") and passes with the fix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(tests): reject `args:` payload expectations in run-to-completion mode PR #667 review (Low): a run-to-completion host-await response could carry an `args:` payload expectation, but RTC execution pre-loads responses and never surfaces the call argument to the harness, so the expectation was parsed and silently dropped. A case with `args: "WRONG"` passed as long as the result matched -- asserting a payload that was never checked. Reject `args:` for run-to-completion fixtures at load time, directing the author to suspendable mode where arguments are validated. Also only build the run-to-completion response vector when the case actually runs in RTC mode, so a suspendable case using the shared host_await_responses field with `args:` is not wrongly rejected. Route the fixture-load error through the same want_error handling used for compilation errors, and add registered_builtin_run_to_completion_rejects_args which now fails loudly instead of passing silently. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(compiler): reject host-await builtin names with surrounding whitespace PR #667 review (Low): register_host_await_builtin rejected all-whitespace names via name.trim().is_empty(), but accepted padded names like " lookup" or "lookup ". Those were inserted into host_await_builtins, but Rego function-call paths produce the trimmed identifier, so a padded registration could never match -- a silent dead registration. Reject any name that is not already trimmed (name != name.trim()) in addition to empty names, and update the error message accordingly. Add test cases for leading and trailing whitespace. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Signed-off-by: Mark Birger <birgerm@yandex.ru> Co-authored-by: Mark Birger <markbirger@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
19 KiB
RVM Instruction Set Reference
This reference captures every opcode emitted by the compiler and executed by
RegoVM. Each instruction is defined in src/rvm/instructions/mod.rs and
implemented by the dispatcher tree in src/rvm/vm/dispatch.rs plus specialised
submodules (arithmetic.rs, loops.rs, functions.rs, rules.rs,
comprehension.rs, virtual_data.rs).
Use this guide to understand operand semantics, parameter tables, and runtime side effects.
Reading the tables
- Operands: registers (
rX), literals (litY), parameter indices (pZ) and immediate values. - Parameters: links into
InstructionData(src/rvm/instructions/params.rs). The compiler stores complex metadata here; instructions reference it by index. - Outcome: mentioned in prose where relevant (
Continue,Return,Break,Suspend).
Load and Move instructions
| Mnemonic | Operands | Behaviour |
|---|---|---|
Load |
dest=rD, literal_idx=litN |
Copies literal N into register D. |
LoadTrue |
dest=rD |
Stores boolean true. |
LoadFalse |
dest=rD |
Stores boolean false. |
LoadNull |
dest=rD |
Stores Value::Null. |
LoadBool |
dest=rD, value |
Stores inline boolean literal. |
LoadData |
dest=rD |
Stores the VM's data value. |
LoadInput |
dest=rD |
Stores the VM's input value. |
Move |
dest=rD, src=rS |
Copies register S into register D. |
Out-of-range literal indices raise VmError::LiteralIndexOutOfBounds. Registers
must have been allocated by the current frame.
Arithmetic and comparison instructions
| Mnemonic | Operands | Behaviour |
|---|---|---|
Add |
dest, left, right |
Numeric addition; undefined operands trigger loop condition checks. |
Sub |
dest, left, right |
Numeric subtraction. |
Mul |
dest, left, right |
Numeric multiplication. |
Div |
dest, left, right |
Numeric division with runtime checks (division by zero errors). |
Mod |
dest, left, right |
Modulo. |
Eq |
dest, left, right |
Equality comparison resulting in Value::Bool. |
Ne |
dest, left, right |
Inequality. |
Lt/Le/Gt/Ge |
dest, left, right |
Ordering comparisons. |
And |
dest, left, right |
Logical conjunction (truthiness semantics). |
Or |
dest, left, right |
Logical disjunction. |
Not |
dest, operand |
Logical negation. |
AssertCondition |
condition |
Fails current loop/rule when the condition is falsey. |
AssertNotUndefined |
register |
Fails when register holds Value::Undefined. |
handle_condition routes through loops.rs to propagate failures to loop and
comprehension contexts. Outside loops it aborts the current rule.
Collection and indexing instructions
| Mnemonic | Operands / Params | Behaviour |
|---|---|---|
ObjectSet |
obj, key, value |
Mutates object in obj with key/value from registers. |
ObjectCreate |
params_index=pN |
Builds object from literal template and register entries. |
ArrayNew |
dest |
Creates empty array. |
ArrayPush |
arr, value |
Appends to array. |
ArrayCreate |
params_index=pN |
Builds array from register list; undefined element ⇒ result undefined. |
SetNew |
dest |
Creates empty set. |
SetAdd |
set, value |
Adds element to set. |
SetCreate |
params_index=pN |
Builds set from register list; undefined element ⇒ result undefined. |
Index |
dest, container, key |
Indexes container with runtime key. |
IndexLiteral |
dest, container, literal_idx |
Indexes container using literal stored in program. |
ChainedIndex |
params_index=pN |
Resolves multi-hop path from root register. |
Contains |
dest, collection, value |
Checks membership; returns Value::Bool. |
Count |
dest, collection |
Returns length or Value::Undefined for unsupported types. |
VirtualDataDocumentLookup |
params_index=pN |
Evaluates data path, invoking rules lazily. |
Parameter structures:
ObjectCreateParamsreuses arrays of literal key/value pairs and register pairs. Literal keys must be sorted to match template order.ArrayCreateParamsandSetCreateParamsstore register lists. The VM checks all referenced registers forValue::Undefinedbefore constructing the collection.VirtualDataDocumentLookupParamsandChainedIndexParamsencodeVec<LiteralOrRegister>path components.LiteralOrRegisteris defined insrc/rvm/instructions/types.rs.
Loop instructions
Loops use dedicated parameter tables (LoopStartParams) and the LoopMode
enum.
| Mnemonic | Operands / Params | Behaviour |
|---|---|---|
LoopStart |
params_index=pN |
Initialises loop context and decides first body iteration. |
LoopNext |
body_start, loop_end |
Finalises iteration, updates accumulators, advances to next element. |
LoopMode values:
Any: succeed on first passing iteration, short-circuit on success.Every: fail on first failing iteration.ForEach: evaluate all iterations, typically for comprehensions or complete rules.
LoopStartParams fields:
collection: source register.key_reg/value_reg: iteration registers (for arrays, key is index).result_reg: accumulator storing loop outcome (boolfor quantifiers).body_start/loop_end: PCs identifying loop boundaries.
The dispatcher converts LoopStartParams into a VM-specific LoopParams used by
both execution modes. In suspendable mode, loops own their own ExecutionFrame.
Comprehension instructions
| Mnemonic | Operands / Params | Behaviour |
|---|---|---|
ComprehensionBegin |
params_index=pN |
Allocates collection builder and iteration context. |
ComprehensionYield |
value_reg, key_reg? |
Emits value (and optional key) into builder. |
ComprehensionEnd |
— | Finalises collection and stores result. |
ComprehensionBeginParams captures:
mode: ComprehensionMode(Set, Array, Object)collection_reg: source register for iterationresult_reg: register that will hold the final collectionkey_reg/value_reg: iteration registersbody_start/comprehension_end: branch targets
Comprehensions manage their own stack (ComprehensionContext) to maintain
ordering guarantees (arrays), uniqueness (sets) or key/value pairing (objects).
Call and return instructions
| Mnemonic | Operands / Params | Behaviour |
|---|---|---|
BuiltinCall |
params_index=pN |
Invokes builtin via resolved function pointer. |
FunctionCall |
params_index=pN |
Invokes function rule. |
CallRule |
dest, rule_index |
Requests rule evaluation with caching. |
RuleInit |
result_reg, rule_index |
Prepares rule accumulator and cache state. |
Return |
value_reg |
Returns value from current function body. |
RuleReturn |
— | Finalises rule evaluation frame. |
DestructuringSuccess |
— | Signals successful destructuring, breaks rule block. |
Parameter tables:
BuiltinCallParams/FunctionCallParamsstore destination register, index into builtin table / rule index, argument count and up to eight argument register numbers.- The VM dynamically resizes registers when a callee requires a larger window
using program metadata (
max_rule_window_size).
Host interaction
| Mnemonic | Operands | Behaviour |
|---|---|---|
HostAwait |
dest, arg, id |
Yields control to host with payload value. |
- Run-to-completion: consumes a response from
host_await_responseskeyed by the identifier register. Missing responses raiseVmError::HostAwaitResponseMissing. - Suspendable: emits
InstructionOutcome::SuspendwithSuspendReason::HostAwait. The host must resume with a value that will be written intodest.
Registered host-await builtins
The compiler can be configured with a list of function names that map directly
to HostAwait instructions. This allows policy authors to write natural
function calls (e.g. lookup(input.account_id)) instead of the raw
__builtin_host_await(payload, identifier) builtin.
Registration is done at compile time via Compiler::compile_from_policy_with_host_await:
let builtins = [("lookup", 1), ("persist", 1)];
let program = Compiler::compile_from_policy_with_host_await(
&compiled_policy, &entry_points, &builtins,
)?;
Each registered name is a (name, arg_count) pair. When the compiler
encounters a call to a registered name, it emits a HostAwait instruction
with:
arg= the first argument registerid= a register loaded with a string literal containing the function name
Both the explicit __builtin_host_await(arg, id) call and a registered
builtin call produce the same HostAwait bytecode instruction. The only
difference is how the id register is populated: explicit calls take it from
the second user-supplied argument, while registered calls auto-generate a
Load instruction for the function name string. The VM cannot distinguish
between the two at runtime.
Resolution order in determine_call_target():
__builtin_host_await(magic 2-argument form)- Registered host-await builtins (matched by bare function name only)
- User-defined functions (matched by package-qualified path)
- Standard builtins (matched by bare function name)
Registered names shadow both user-defined functions and standard builtins.
This means time.parse_duration_ns can be overridden to route through the
host instead of the built-in Rust implementation.
Only unqualified calls are intercepted. Registration matches a call by
the name as written in the policy. A bare call — lookup(x) — is
intercepted and compiled to a HostAwait. A package-qualified call —
data.pkg.lookup(x) — is not intercepted; it is resolved normally, as
if the name were never registered.
# "lookup" is registered as a host-await builtin.
package other
import rego.v1
lookup(k) := k # an ordinary rule that happens to share the name
package demo
import rego.v1
a := lookup(input.k) # intercepted -> HostAwait
b := data.other.lookup(input.k) # NOT intercepted -> calls other.lookup
The qualified form is resolved exactly as it would be without registration:
if a rule exists at that path it is called, otherwise compilation fails with
Unknown function. (A standard builtin like count has no qualified form at
all, so data.pkg.count(x) is always an Unknown function error, registered
or not.)
Argument handling: The HostAwait instruction carries a single arg
register. Registered builtins must use arg_count: 1; the compiler rejects
arg_count > 1 at registration time. To pass multiple values, use object
packing: lookup({"user": x, "resource": y}).
Halt instruction
| Mnemonic | Behaviour | Notes |
|---|---|---|
Halt |
Terminates execution immediately. | Used during debugging or emitted for guard rails. |
When encountered during run-to-completion execution, Halt returns the current
value in register 0.
Parameter data overview
InstructionData (src/rvm/instructions/params.rs) collects all complex
parameter types. Each add_* method returns a u16 index suitable for storing
inside instructions. The VM retrieves tables via get_* accessors.
| Struct | Field | Purpose |
|---|---|---|
LoopStartParams |
mode |
Loop semantics (Any, Every, ForEach). |
collection |
Register holding the iterable collection. | |
key_reg / value_reg |
Registers populated with the current key/value each iteration. | |
result_reg |
Accumulator for loop outcome (bool for quantifiers). |
|
body_start / loop_end |
Instruction pointers delimiting the loop body and exit. | |
BuiltinCallParams |
dest |
Register that receives the builtin result. |
builtin_index |
Slot into builtin_info_table for dispatch. |
|
num_args |
Count of argument registers actually populated. | |
args[8] |
Up to eight registers supplying builtin arguments. | |
FunctionCallParams |
dest |
Register that receives the function rule result. |
func_rule_index |
Rule index for the target function definition. | |
num_args |
Number of argument registers provided. | |
args[8] |
Argument register numbers (unused slots ignored). | |
ObjectCreateParams |
dest |
Destination register for the constructed object. |
template_literal_idx |
Literal template containing all expected keys. | |
literal_key_fields: Vec<(u16, u8)> |
Mapping of literal-key indices to value registers. | |
fields: Vec<(u8, u8)> |
Dynamic key/value register pairs for non-literal keys. | |
ArrayCreateParams |
dest |
Destination register for the array literal. |
elements: Vec<u8> |
Registers providing array elements (order preserved). | |
SetCreateParams |
dest |
Destination register for the set literal. |
elements: Vec<u8> |
Registers providing set members (duplicates dropped at runtime). | |
VirtualDataDocumentLookupParams |
dest |
Destination register for lookup result. |
path_components: Vec<LiteralOrRegister> |
Ordered path traversal steps; mix of literals and register-based keys. | |
ChainedIndexParams |
dest |
Destination register for resolved value. |
root |
Register containing the root object/collection. | |
path_components: Vec<LiteralOrRegister> |
Path components applied relative to the root register. | |
ComprehensionBeginParams |
mode |
Comprehension output type (array, set, object). |
collection_reg |
Source collection register for iteration. | |
result_reg |
Register receiving the final collection. | |
key_reg / value_reg |
Iteration registers (keys optional for arrays/sets). | |
body_start / comprehension_end |
Instruction pointers framing comprehension body and exit. |
All parameter structs derive Serialize/Deserialize and can be stored inside
artifacts. Some contain Vec fields; the compiler is responsible for ensuring
indices remain valid and stable across serialization boundaries.