Add knowledge docs, agent definitions, and skill files

Add comprehensive documentation and GitHub Copilot configuration:

- docs/knowledge/: 17 deep-dive knowledge files covering value semantics,
  RVM architecture, builtins, FFI boundary, feature composition, error
  handling migration, policy evaluation security, Rego semantics,
  interpreter/compiler architecture, Azure Policy/RBAC, engine API,
  time builtins, language extension guide, tooling architecture,
  causality/partial eval, Rego compiler, Azure Policy aliases, and
  telemetry/diagnostics

- .github/agents/: 16 role-specific AI agent definitions (red-teamer,
  semantics-expert, architect, performance-engineer, test-engineer,
  verification-engineer, security-auditor, reliability-engineer,
  support-engineer, ci-engineer, refactorer, api-steward, program-manager,
  demo-engineer, dx-engineer, tech-lead)

- .github/skills/: 6 workflow skill definitions (thorough-review,
  design-alternatives, add-builtin, opa-conformance, security-review,
  verification)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: anakrish <35780660+anakrish@users.noreply.github.com>

docs/knowledge/azure-policy-aliases.md

@@ -0,0 +1,287 @@
+<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->
+<!-- Licensed under the MIT License. -->
+
+# Knowledge: Azure Policy Aliases and Normalization
+
+Deep knowledge about the Azure Policy alias system and ARM resource
+normalization. Read this before modifying alias resolution, the normalizer,
+or the denormalizer.
+
+See also `azure-policy-language.md` for the overall Azure Policy compilation
+pipeline.
+
+## What Aliases Are
+
+Azure Policy uses "aliases" to refer to Azure resource properties in a
+provider-independent way:
+
+```
+Full alias:  Microsoft.Storage/storageAccounts/supportsHttpsTrafficOnly
+Short name:  supportsHttpsTrafficOnly
+ARM path:    properties.supportsHttpsTrafficOnly
+```
+
+The alias system bridges between:
+- **Policy authors** — who write conditions using alias paths
+- **ARM resources** — which have nested JSON structures with varying casing
+
+## Alias Registry
+
+### Loading Sources
+
+**Control-plane aliases** — loaded from Azure provider metadata:
+```
+GET /providers?$expand=resourceTypes/aliases
+```
+Produces `ProviderAliases` with resource type → alias mappings.
+
+**Data-plane aliases** — loaded from data policy manifests for `.Data`
+namespaces (e.g., `Microsoft.KeyVault.Data/vaults/secrets`).
+
+### Registry Structure
+
+```rust
+struct AliasRegistry {
+    // Maps full alias name → alias metadata
+    aliases: BTreeMap<String, AliasInfo>,
+    // Maps resource type → list of aliases
+    resource_type_aliases: BTreeMap<String, Vec<String>>,
+}
+```
+
+The registry provides:
+- Alias path segments (for navigating ARM JSON)
+- Alias type metadata (string, array, object, etc.)
+- Default path mappings when aliases are absent
+
+## Normalization Pipeline
+
+The normalizer transforms ARM resource JSON into a flat structure that
+the policy compiler can evaluate directly.
+
+### Input: ARM Resource JSON
+
+```json
+{
+  "type": "Microsoft.Storage/storageAccounts",
+  "id": "/subscriptions/.../storageAccounts/myaccount",
+  "name": "myaccount",
+  "location": "eastus",
+  "properties": {
+    "supportsHttpsTrafficOnly": true,
+    "networkAcls": {
+      "defaultAction": "Deny",
+      "virtualNetworkRules": [
+        { "id": "/subscriptions/.../subnets/default" }
+      ]
+    }
+  }
+}
+```
+
+### Output: Normalized Resource
+
+```json
+{
+  "type": "microsoft.storage/storageaccounts",
+  "id": "/subscriptions/.../storageAccounts/myaccount",
+  "name": "myaccount",
+  "location": "eastus",
+  "supportshttpstrafficonly": true,
+  "networkacls.defaultaction": "Deny",
+  "networkacls.virtualnetworkrules": [
+    { "id": "/subscriptions/.../subnets/default" }
+  ]
+}
+```
+
+### Normalization Steps
+
+1. **Copy root fields** (lowercased): `type`, `id`, `kind`, `name`,
+   `location`, `identity`, `zones`, `sku`, `plan`, `tags`
+
+2. **Merge properties** — contents of `properties` are merged into the
+   result at the top level
+
+3. **Apply alias path resolution**:
+   - Each alias has a path (e.g., `properties.networkAcls.defaultAction`)
+   - The normalizer navigates the ARM JSON using path segments
+   - The extracted value is placed at the alias short name (lowercased)
+
+4. **Handle sub-resources** — sub-resource types (e.g., extensions on VMs)
+   are extracted from arrays and normalized separately
+
+5. **Array element handling** — `[*]` in alias paths triggers iteration
+   over array elements; each element is normalized independently
+
+6. **Case folding** — all property names are lowercased for
+   case-insensitive matching (Azure ARM is case-insensitive)
+
+### Key Complexity: Case Preservation
+
+ARM JSON casing is preserved through normalization and denormalization.
+The normalizer records original casing to enable round-trip fidelity.
+This matters for Modify/Append effects that construct output JSON.
+
+## Denormalization
+
+The denormalizer converts flat normalized paths back to nested ARM JSON
+structure. This is needed for:
+- **Modify effect** — construct the resource patch to apply
+- **Append effect** — construct fields to add to the resource
+
+### Denormalization Challenge
+
+Given a flat path like `networkacls.defaultaction = "Allow"`, the
+denormalizer must reconstruct:
+
+```json
+{
+  "properties": {
+    "networkAcls": {
+      "defaultAction": "Allow"
+    }
+  }
+}
+```
+
+This requires knowing:
+- Where `properties` nesting begins (alias metadata)
+- Original casing of each path segment
+- Whether intermediate nodes are objects or arrays
+
+## Compiler Integration
+
+### Alias Map
+
+The compiler receives an alias map: `BTreeMap<String, String>` mapping
+alias short names to full ARM paths. This is populated from the
+`AliasRegistry` for the specific resource type being evaluated.
+
+### Field Compilation
+
+When compiling a `field` condition:
+
+```json
+{ "field": "supportsHttpsTrafficOnly", "equals": true }
+```
+
+1. Look up field name in alias map
+2. If found: compile as property access on normalized input
+3. If dynamic (`[concat(...)]`): compile ARM expression, use result as key
+4. Emit `Index`/`IndexLiteral`/`ChainedIndex` instructions
+
+### Metadata Accumulation
+
+During compilation, the compiler tracks:
+- `observed_aliases` — all alias names referenced
+- `observed_field_kinds` — static fields, dynamic fields, `[*]` wildcards
+- `observed_resource_types` — resource types from field conditions
+- `observed_has_dynamic_fields` — whether ARM expressions appear as fields
+
+This metadata supports policy analysis and optimization.
+
+## Wildcard Semantics
+
+### Unbound `[*]` (outside count)
+
+```json
+{ "field": "securityRules[*].destinationPortRange", "equals": "443" }
+```
+
+Implicit `allOf` — **every** element must match. The compiler generates
+a `LoopStart { mode: Every }` instruction.
+
+### Bound `[*]` (inside count)
+
+```json
+{
+  "count": {
+    "field": "securityRules[*]",
+    "where": { "field": "securityRules[*].destinationPortRange", "equals": "443" }
+  },
+  "greaterOrEquals": 1
+}
+```
+
+Iteration with counting — each element is tested, matching ones are
+counted. The compiler generates `LoopStart { mode: Count }`.
+
+### Multi-level Wildcards
+
+```json
+{ "field": "outer[*].inner[*].value" }
+```
+
+Nested loops: outer levels use `ForEach`, innermost carries the semantic
+operator. The compiler maintains a binding stack to track scope.
+
+## `current()` Function
+
+Inside `count.where` blocks, `current()` refers to the current iteration
+element:
+
+```json
+{
+  "count": {
+    "value": "[parameters('items')]",
+    "name": "item",
+    "where": {
+      "value": "[current('item').status]",
+      "equals": "active"
+    }
+  }
+}
+```
+
+The compiler binds the loop variable and makes it accessible via
+`current()` calls in ARM template expressions.
+
+## Existence vs Null
+
+Azure Policy distinguishes between missing fields and null values:
+
+- **Missing field** → `Undefined` in regorus Value system
+- **Null field** → `Value::Null`
+
+For most operators, the compiler emits `CoalesceUndefinedToNull` to
+treat missing as null. The `exists` operator is the exception — it
+specifically tests for field presence:
+
+```json
+{ "field": "optionalProperty", "exists": true }   // Field must be present
+{ "field": "optionalProperty", "exists": false }  // Field must be absent
+```
+
+## Key Invariants
+
+1. **Normalization before compilation** — aliases are resolved during
+   normalization, not at compile time or runtime
+
+2. **Case-insensitive everywhere** — all field name comparisons use
+   lowercased strings
+
+3. **`[*]` context matters** — same syntax has different semantics
+   inside vs outside `count` expressions
+
+4. **Round-trip fidelity** — normalize → denormalize must preserve
+   original ARM JSON casing for Modify/Append effects
+
+5. **Missing = null (mostly)** — `CoalesceUndefinedToNull` is the
+   default; `exists` is the exception
+
+## Common Pitfalls
+
+1. **Alias path segments** — paths like `properties.a.b` must be split
+   correctly. Dots in property names (rare but possible) need escaping.
+
+2. **Sub-resource normalization** — sub-resources have their own type
+   and their own alias set. Don't normalize with parent's aliases.
+
+3. **Array vs scalar** — some aliases point to arrays, others to scalars.
+   The `[*]` wildcard only works on arrays. Applying it to a scalar
+   is a compile-time error.
+
+4. **Dynamic field resolution order** — ARM template expressions in
+   field positions are evaluated at runtime. The alias map must be
+   available at runtime for dynamic alias resolution.

docs/knowledge/azure-policy-language.md

@@ -0,0 +1,203 @@
+<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->
+<!-- Licensed under the MIT License. -->
+
+# Knowledge: Azure Policy Language
+
+Deep knowledge about the Azure Policy language extension in
+`src/languages/azure_policy/`. Read this before modifying Azure Policy
+parsing, compilation, or evaluation.
+
+## How Azure Policy Differs from Rego
+
+| Aspect | Azure Policy | Rego |
+|--------|--------------|------|
+| **Syntax** | JSON-based declarative constraints | Prolog-like logic language |
+| **Compilation** | JSON → AST → RVM bytecode | Source → AST → RVM bytecode |
+| **Logic model** | `allOf`/`anyOf`/`not` combinators | Set comprehensions, rules |
+| **Effects** | Policy decision directives (Deny, Audit, Modify, ...) | Returns values |
+| **Templating** | ARM template expressions `[concat(...)]` | No templating |
+| **Field access** | Direct properties + aliases for resource types | Dot-notation queries |
+
+Despite these differences, Azure Policy compiles to the **same RVM bytecode**
+as Rego. The shared VM executes both languages.
+
+## Directory Structure
+
+```
+src/languages/azure_policy/
+  mod.rs                    Module root
+  parser/                   JSON → PolicyRule AST (6 files)
+  compiler/                 AST → RVM Program (14 files)
+  ast/                      Span-annotated AST types
+  aliases/                  ARM resource alias normalization
+    normalizer/             ARM JSON → flat alias paths
+    denormalizer/           Flat paths → ARM JSON structure
+  expr.rs                   ARM template expression sub-parser
+  strings/                  Case folding, key normalization
+```
+
+## AST Types
+
+### Policy Rule Structure
+
+```
+PolicyRule
+  ├── condition: Constraint    // "if" clause
+  └── then_block: ThenBlock    // "then" clause with effect
+```
+
+### Constraint Hierarchy
+
+```rust
+enum Constraint {
+    AllOf { constraints: Vec<Constraint> },   // AND — all must match
+    AnyOf { constraints: Vec<Constraint> },   // OR — any must match
+    Not { constraint: Box<Constraint> },      // Negation
+    Condition(Box<Condition>),                // Leaf condition
+}
+
+struct Condition {
+    lhs: Lhs,              // What to evaluate (Field, Value, or Count)
+    operator: OperatorNode, // How to compare (19 operators)
+    rhs: ValueOrExpr,       // What to compare against
+}
+```
+
+### 19 Operators
+
+Contains, ContainsKey, Equals, Greater, GreaterOrEquals, Exists, In, Less,
+LessOrEquals, Like, Match, MatchInsensitively, NotContains, NotContainsKey,
+NotEquals, NotIn, NotLike, NotMatch, NotMatchInsensitively.
+
+### Effects
+
+```rust
+enum EffectKind {
+    Deny, Audit, Append, AuditIfNotExists, DeployIfNotExists,
+    Disabled, Modify, DenyAction, Manual, Other,
+}
+```
+
+**Note:** Effect compilation is not yet fully implemented — the compiler
+has stubs for effect handling.
+
+## Compilation to RVM
+
+Azure Policy compiles directly to RVM bytecode through a dedicated compiler:
+
+```rust
+pub fn compile_policy_rule(rule: &PolicyRule) -> Result<Rc<Program>>
+pub fn compile_policy_definition(defn: &PolicyDefinition) -> Result<Rc<Program>>
+pub fn compile_policy_definition_with_aliases(rule, alias_map, modifiable) -> Result<Rc<Program>>
+```
+
+The compiler:
+1. Parses JSON → `PolicyRule` AST
+2. Compiles constraints to RVM instructions (shared VM)
+3. Populates metadata (language annotation "azure_policy", effect info)
+4. Resolves parameter defaults
+5. Optionally resolves aliases
+
+### Compiler State
+
+```rust
+struct Compiler {
+    program: Program,                    // Shared RVM program being built
+    register_counter: u8,               // Register allocation
+    alias_map: BTreeMap<String, String>,// Alias resolution
+    parameter_defaults: Option<Value>,  // Default parameter values
+    cached_input_reg: Option<u8>,       // Cached LoadInput register
+    cached_context_reg: Option<u8>,     // Cached LoadContext register
+}
+```
+
+## Alias System
+
+Azure Policy uses "aliases" to refer to resource properties in a normalized
+way. The alias system has two phases:
+
+### Normalizer
+
+Converts ARM JSON resource representations to flat structures with alias
+paths. Handles:
+- Nested resource properties
+- Sub-resource types (e.g., `Microsoft.Compute/virtualMachines/extensions`)
+- Array element access
+- Case-insensitive property matching
+
+### Denormalizer
+
+Converts flat alias paths back to ARM JSON structure. This is needed for
+Modify/Append effects that need to construct resource representations.
+
+**Key complexity**: Casing must survive round-trip. ARM JSON casing is
+preserved through normalization and denormalization.
+
+## ARM Template Expressions
+
+Azure Policy conditions can contain ARM template expressions:
+
+```json
+{
+    "field": "[concat(field('Microsoft.Storage/storageAccounts/name'), '/default')]",
+    "equals": "[parameters('storageName')]"
+}
+```
+
+The expression parser (`expr.rs`) handles:
+- Recursive descent parsing (`.`, `()`, `[]` operators)
+- Unknown symbols enabled in lexer mode
+- 65,536 character column limit for deeply nested expressions
+- Functions: `concat()`, `field()`, `parameters()`, etc.
+
+## Count Expressions
+
+Azure Policy supports counting with optional `where` clauses:
+
+```json
+{
+    "count": {
+        "field": "Microsoft.Network/networkSecurityGroups/securityRules[*]",
+        "where": { "field": "...", "equals": "..." }
+    },
+    "greater": 0
+}
+```
+
+The compiler handles count with existence-pattern optimization — common
+patterns like "count > 0" can be compiled as existence checks.
+
+## Wildcard Handling
+
+The `[*]` wildcard in field references creates implicit iteration:
+
+```json
+{ "field": "Microsoft.Network/securityRules[*].destinationPortRange" }
+```
+
+When a wildcard is unbound, it creates an implicit `allOf` — the condition
+must hold for ALL elements. The compiler generates appropriate iteration
+code in the RVM.
+
+## Integration Points
+
+Azure Policy integrates with the shared infrastructure:
+- **RVM Program**: compiled output is the same `Program` struct as Rego
+- **Value type**: evaluation uses the same `Value` enum
+- **Engine**: accessible via `Engine::compile_for_target()` when the
+  `azure_policy` feature is enabled
+- **CompiledPolicy**: wraps the RVM program with metadata
+
+## Key Invariants
+
+1. **Case-insensitive matching** — Azure Policy field names are
+   case-insensitive. All comparisons must use case-folded strings.
+
+2. **Alias resolution order** — aliases must be resolved before compilation.
+   Missing aliases produce compile-time errors, not runtime errors.
+
+3. **Wildcard semantics** — `[*]` is implicitly "for all" unless inside a
+   count expression where it becomes "for each".
+
+4. **Effect metadata** — the compiled program must carry effect information
+   in metadata, not in the instruction stream.

docs/knowledge/azure-rbac-language.md

@@ -0,0 +1,154 @@
+<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->
+<!-- Licensed under the MIT License. -->
+
+# Knowledge: Azure RBAC Language
+
+Deep knowledge about the Azure RBAC condition language extension in
+`src/languages/azure_rbac/`. Read this before modifying RBAC evaluation.
+
+## How RBAC Differs from Rego and Azure Policy
+
+| Aspect | Azure RBAC | Azure Policy | Rego |
+|--------|-----------|-------------|------|
+| **Purpose** | Access control conditions | Resource compliance | General policy |
+| **Execution** | Direct interpretation | RVM compilation | RVM or interpreter |
+| **Syntax** | Condition expression strings | JSON constraints | Rego source |
+| **Logic** | AND/OR/NOT + quantifiers | allOf/anyOf/not | Rules + comprehensions |
+| **Builtins** | 40+ ABAC functions | 19 operators | 100+ OPA builtins |
+
+**Key difference**: RBAC uses **direct interpretation** (no RVM compilation).
+It has its own `ConditionInterpreter` that evaluates condition strings directly.
+
+## Directory Structure
+
+```
+src/languages/azure_rbac/
+  mod.rs              Module root
+  interpreter.rs      Direct evaluation engine (66 lines)
+  ast/                Expression types (8 files)
+    expr.rs           ConditionExpr enum — 15+ variants
+    context.rs        EvaluationContext (Principal, Resource, Request, Environment)
+    operators.rs      Operator definitions
+    literals.rs       Literal types (string, number, bool, datetime, time, set, list)
+    references.rs     Attribute references
+    spans.rs          Source location tracking
+  parser/             Condition string → AST (3 files)
+  builtins/           40+ ABAC condition functions (14 files)
+  test_cases/         40+ YAML test files
+```
+
+## Evaluation Context
+
+RBAC evaluation happens against a rich context:
+
+```rust
+struct EvaluationContext {
+    principal: Principal,           // Who is accessing
+    resource: Resource,             // What is being accessed
+    request: RequestContext,        // What action is requested
+    environment: EnvironmentContext, // When/where (time, network)
+    action: Option<String>,         // Control-plane action
+    suboperation: Option<String>,   // Sub-operation identifier
+}
+
+struct Principal {
+    id: String,
+    principal_type: PrincipalType,  // User, Group, ServicePrincipal, MSI
+    custom_security_attributes: Value,
+}
+
+struct Resource {
+    id: String,
+    resource_type: String,
+    scope: String,
+    attributes: Value,
+}
+```
+
+## Expression Types
+
+The RBAC AST represents condition expressions:
+
+```rust
+enum ConditionExpr {
+    Logical(LogicalExpression),           // AND/OR
+    Unary(UnaryExpression),              // NOT, exists, notExists
+    Binary(BinaryExpression),            // Operator comparisons
+    FunctionCall(FunctionCallExpression), // ToLower, Substring, etc.
+    AttributeReference(AttributeReference),  // principal.id, resource.attributes.env
+    ArrayExpression(ArrayExpression),     // ANY/ALL quantifiers
+    Identifier(IdentifierExpression),
+    VariableReference(VariableReference), // Loop variables
+    PropertyAccess(PropertyAccessExpression),
+    // Literals: String, Number, Bool, Null, DateTime, Time, Set, List
+}
+```
+
+## Condition Interpreter
+
+The interpreter evaluates conditions directly (no compilation step):
+
+```rust
+struct ConditionInterpreter<'a> {
+    context: &'a EvaluationContext,
+}
+
+impl ConditionInterpreter {
+    fn evaluate_str(&self, condition: &str) -> Result<bool>
+    fn evaluate_condition_expression(&self, cond: &ConditionExpression) -> Result<bool>
+    fn evaluate_bool(&self, expr: &ConditionExpr) -> Result<bool>
+    fn evaluate_value(&self, expr: &ConditionExpr) -> Result<Value>
+}
+```
+
+### Evaluation Flow
+
+1. Parse condition string → `ConditionExpression` with `ConditionExpr` AST
+2. Recursively evaluate:
+   - **Logical**: AND/OR with short-circuit evaluation
+   - **Unary**: NOT, exists (check if attribute is present), notExists
+   - **Binary**: delegate to `RbacBuiltinEvaluator` for comparison
+   - **Function calls**: evaluate with built-in RBAC functions
+   - **Array expressions**: ANY/ALL quantifiers over collections
+   - **Attribute references**: resolve from evaluation context
+
+## RBAC Builtins (40+ functions)
+
+Organized by category:
+
+| Category | Functions |
+|----------|-----------|
+| **Strings** | StringEquals, StringEqualsIgnoreCase, StringLike, StringMatches, StringNotEquals, ... |
+| **Numbers** | NumericEquals, NumericGreaterThan, NumericInRange, ... |
+| **Booleans** | BoolEquals, BoolNotEquals |
+| **GUIDs** | GuidEquals, GuidNotEquals |
+| **DateTime** | DateTimeEquals, DateTimeGreaterThan, DateTimeInRange, ... |
+| **Time of Day** | TimeOfDayEquals, TimeOfDayGreaterThan, TimeOfDayInRange, ... |
+| **IP** | IpMatch, IpNotMatch, IpInRange |
+| **Lists** | ListContains, ListNotContains, NormalizeList, NormalizeSet |
+| **Actions** | ActionMatches, SubOperationMatches |
+| **Quantifiers** | ANY, ALL, EXISTS |
+
+Each builtin is an enum variant in `RbacBuiltin` used for direct dispatch
+in `BinaryExpression` evaluation.
+
+## Key Invariants
+
+1. **No RVM backend** — RBAC is pure interpretation. Changes to the RVM do
+   not affect RBAC evaluation.
+
+2. **Short-circuit evaluation** — AND/OR evaluate left-to-right and stop
+   early. This is semantically important (not just an optimization).
+
+3. **Attribute resolution** — attributes are resolved from the evaluation
+   context at evaluation time. Missing attributes may produce errors or
+   false depending on the operator.
+
+4. **Case sensitivity** — string comparisons have both case-sensitive and
+   case-insensitive variants. Use the correct one.
+
+## Testing
+
+40+ YAML test files in `test_cases/` provide comprehensive coverage.
+Each test case specifies a condition string, evaluation context, and
+expected result.

docs/knowledge/builtin-system.md

@@ -0,0 +1,181 @@
+<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->
+<!-- Licensed under the MIT License. -->
+
+# Knowledge: Builtin System
+
+Deep knowledge about regorus's builtin function infrastructure. Read this
+before adding, modifying, or debugging builtin functions.
+
+## Registration Pattern
+
+Builtin functions live in `src/builtins/`. Each module exports a `register`
+function that inserts entries into the `BUILTINS` lazy_static registry:
+
+```rust
+// In src/builtins/arrays.rs
+pub fn register(m: &mut BuiltinsMap<&'static str, BuiltinFcn>) {
+    m.insert("array.concat", (concat, 2));
+    m.insert("array.reverse", (reverse, 1));
+    m.insert("array.slice", (slice, 3));
+}
+```
+
+The tuple is `(function_pointer, arity)`. The function signature is:
+
+```rust
+fn concat(span: &Span, params: &[Ref<Expr>], args: &[Value], strict: bool) -> Result<Value>
+```
+
+Parameters:
+- `span`: Source location for error messages
+- `params`: AST expressions (for error reporting, not evaluation)
+- `args`: Evaluated argument values
+- `strict`: Whether strict builtin error mode is enabled
+
+## Registration in BUILTINS
+
+All builtin modules register in `src/builtins/mod.rs` via a `lazy_static!` block:
+
+```rust
+lazy_static::lazy_static! {
+    pub static ref BUILTINS: BuiltinsMap<&'static str, BuiltinFcn> = {
+        let mut m = BuiltinsMap::new();
+        numbers::register(&mut m);
+        strings::register(&mut m);
+        // ...
+        #[cfg(feature = "regex")]
+        regex::register(&mut m);
+        // ...
+        m
+    };
+}
+```
+
+## Feature Gating
+
+Optional builtins must be feature-gated at two levels:
+
+**1. Cargo.toml** — declare the feature and optional dependency:
+```toml
+[features]
+regex = ["dep:regex"]
+```
+
+**2. Registration** — gate the register call:
+```rust
+#[cfg(feature = "regex")]
+regex::register(&mut m);
+```
+
+**3. Composite features** — add to `full-opa` and/or `opa-no-std` if the
+builtin is part of the OPA specification:
+```toml
+full-opa = ["regex", ...]
+opa-no-std = ["regex", ...]  # only if the dep supports no_std
+```
+
+## Argument Validation
+
+Every builtin must validate argument count first:
+
+```rust
+fn concat(span: &Span, params: &[Ref<Expr>], args: &[Value], strict: bool) -> Result<Value> {
+    let name = "array.concat";
+    ensure_args_count(span, name, params, args, 2)?;
+    // ...
+}
+```
+
+Then validate argument types. Use `ensure_*` helpers where available.
+
+## OPA Conformance Requirements
+
+**Error messages must match OPA exactly.** The OPA conformance test suite
+(`tests/opa.rs`) compares error messages literally. This means:
+
+- Function names in errors must match OPA's naming
+- Error message format must match OPA's format
+- Type error descriptions must match OPA's wording
+
+If an error message doesn't match, the conformance test fails. When
+implementing a builtin, compare against the OPA Go source for exact wording.
+
+## Strict vs Non-Strict Mode
+
+When `strict` is `true`:
+- Type errors are hard errors (return `Err(...)`)
+- Missing arguments are hard errors
+
+When `strict` is `false`:
+- Type errors return `Value::Undefined` (the OPA default)
+- This matches OPA's behavior where type mismatches silently fail
+
+## Undefined Argument Handling
+
+Builtins receive `Value::Undefined` when an argument expression evaluates to
+undefined. The interpreter checks this before calling:
+
+```rust
+if args.iter().any(|a| a == &Value::Undefined) {
+    return Ok(Value::Undefined);
+}
+```
+
+However, individual builtins may also need to handle Undefined for specific
+semantic reasons.
+
+## Both Execution Paths
+
+Builtins are shared between the interpreter and the RVM. Both use the same
+`BUILTINS` registry. When adding a builtin:
+
+1. The interpreter calls builtins via `eval_builtin_call()`
+2. The RVM resolves builtins by name from the same registry
+3. No special RVM registration is needed — it's automatic
+
+Test with both `cargo test` (interpreter) and RVM-specific tests.
+
+## Adding a New Builtin: Checklist
+
+1. Create the function in the appropriate `src/builtins/` module
+2. Follow the `(span, params, args, strict) -> Result<Value>` signature
+3. Call `ensure_args_count()` first
+4. Feature-gate if it requires optional dependencies
+5. Register in the module's `register()` function
+6. Add the module's `register()` call in `src/builtins/mod.rs` (feature-gated)
+7. Add to composite features (`full-opa`, `opa-no-std`) if OPA-standard
+8. Write tests (YAML format, see `tests/interpreter/`)
+9. Verify error messages match OPA exactly
+10. Update `docs/builtins.md`
+11. Run `cargo test --test opa` to verify OPA conformance
+12. Run `cargo xtask ci-debug` for full suite
+
+## Builtin Modules
+
+The `~19 modules` in `src/builtins/` cover:
+- `numbers` — arithmetic, rounding, abs, rem
+- `strings` — concat, contains, replace, split, trim, format, sprintf
+- `arrays` — concat, reverse, slice
+- `objects` — get, keys, remove, union, filter
+- `sets` — intersection, union, difference
+- `aggregates` — count, sum, min, max, sort
+- `types` — type_name, is_number, is_string, etc.
+- `encoding` — base64, base64url, hex, json, yaml, urlquery
+- `regex` — match, split, find (feature-gated)
+- `glob` — match (feature-gated)
+- `time` — now_ns, parse_ns, date, clock (feature-gated)
+- `crypto` — hashing functions
+- `graphs` — walk, reachable (feature-gated)
+- `semver` — is_valid, compare (feature-gated)
+- `uuid` — rfc4122 (feature-gated)
+- `net` — cidr_contains, cidr_intersects (feature-gated)
+- `opa` — runtime info (feature-gated)
+
+## LRU Caching
+
+Some builtins use the LRU cache (`src/cache.rs`) for expensive compiled objects:
+- **Regex patterns**: up to 256 cached compiled `regex::Regex` objects
+- **Glob matchers**: up to 128 cached compiled `GlobMatcher` objects
+
+The cache is global, thread-safe (mutex-protected), and configurable via
+`cache::configure()`. The hard cap is 2^16 entries per cache type.

docs/knowledge/causality-and-partial-eval.md

@@ -0,0 +1,241 @@
+<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->
+<!-- Licensed under the MIT License. -->
+
+# Knowledge: Causality and Partial Evaluation
+
+Design considerations for future causality tracking and partial evaluation
+features. These are not yet implemented but the architecture is being
+designed to support them. Read this when making architectural decisions
+that may affect these future capabilities.
+
+## Partial Evaluation
+
+### What It Is
+
+Partial evaluation reduces a policy given **known** inputs while leaving
+**unknown** parts symbolic:
+
+```
+Full policy + known data + unknown input
+    → Simplified policy (only depends on unknown input)
+```
+
+Example:
+```rego
+allow {
+    input.role == "admin"       # Unknown (depends on input)
+    data.feature_enabled        # Known: true
+    input.department in {"eng", "security"}  # Unknown
+}
+```
+
+Partial evaluation with `data.feature_enabled = true`:
+```rego
+allow {
+    input.role == "admin"
+    input.department in {"eng", "security"}
+}
+```
+
+The `data.feature_enabled` check is eliminated because it's always true.
+
+### Use Cases
+
+1. **Policy optimization**: pre-evaluate known parts at compile/load time
+2. **Policy simplification**: show users what a policy means for their context
+3. **Incremental evaluation**: only re-evaluate changed parts
+4. **Query planning**: push policy decisions closer to data sources
+5. **Policy diffing**: compare simplified policies across configurations
+
+### Current Architecture Support
+
+**Scheduler dependency analysis**: The scheduler already identifies which
+statements depend on which variables. Statements that only depend on known
+variables can be evaluated. Statements with unknown dependencies remain
+symbolic.
+
+**RVM register model**: Registers could hold symbolic values alongside
+concrete ones. Instructions that operate on symbolic values produce symbolic
+results.
+
+**Value type extensibility**: The `Value` enum could be extended:
+```rust
+pub enum Value {
+    // ... existing variants ...
+    Symbolic(SymbolicExpr),  // Future: represents an unknown value
+}
+```
+
+**Compilation pipeline**: The hoister and scheduler already separate
+ground-truth computations from data-dependent ones. This separation is
+the foundation for partial evaluation.
+
+### Design Principles
+
+1. **Preserve semantics**: partially evaluated policy must produce identical
+   results to the original when the remaining unknowns are bound.
+
+2. **Undefined handling**: partial evaluation must correctly propagate
+   Undefined through symbolic expressions. This is the hardest part —
+   `not Undefined = true` means symbolic undefined propagation has
+   non-obvious results.
+
+3. **No information loss**: the residual policy must capture all constraints,
+   including those that were partially evaluated.
+
+4. **Composability**: partial evaluation results should be further partially
+   evaluatable as more inputs become known.
+
+### Implementation Considerations
+
+**Phase 1: Ground-truth elimination**
+- Identify statements where all variables are known
+- Evaluate them and replace with results
+- Remove always-true conditions, eliminate always-false rule bodies
+- This is the easiest phase and provides immediate value
+
+**Phase 2: Symbolic propagation**
+- Track symbolic values through expressions
+- Simplify expressions where possible (e.g., `true AND x` → `x`)
+- Handle Undefined propagation symbolically
+- Generate residual policy/program
+
+**Phase 3: Cross-rule analysis**
+- Partially evaluate virtual documents
+- Propagate known rule results into dependent rules
+- Handle default rules in partial context
+
+### Challenges
+
+- **Undefined propagation**: `not (Undefined)` = `true` makes symbolic
+  analysis non-trivial. A symbolic expression that might be Undefined
+  has different semantics under negation.
+
+- **Set/Object construction**: if any element is symbolic, the entire
+  collection construction may need to remain symbolic.
+
+- **Comprehensions**: partial evaluation of comprehensions requires
+  knowing which iterations are ground vs symbolic.
+
+- **Builtins**: some builtins are pure (suitable for partial evaluation),
+  others have side effects or depend on runtime state (`time.now_ns()`).
+
+## Causality Tracking
+
+### What It Is
+
+Causality tracking answers **why** a policy produced its result:
+- Which rules contributed to the decision?
+- What input/data values were decisive?
+- What would need to change to get a different result?
+
+### Use Cases
+
+1. **Audit**: prove why a request was allowed/denied
+2. **Debugging**: understand unexpected policy decisions
+3. **Compliance**: demonstrate that decisions follow documented logic
+4. **Counterfactual**: "what if the user had role X instead of Y?"
+
+### Current Infrastructure
+
+**Coverage tracking** (`coverage` feature):
+- Records which expressions were evaluated
+- Binary: evaluated or not evaluated
+- Doesn't track values or decision flow
+
+**Tracing** (`eval_query(query, tracing=true)`):
+- Captures evaluation steps
+- Provides more detail than coverage
+- Performance cost limits production use
+
+**RVM frame stack** (suspendable mode):
+- Frame-by-frame execution history
+- Instruction-level granularity available via single-step mode
+- Only in suspendable mode (not run-to-completion)
+
+**Active rules stack** (interpreter):
+- Tracks which rules are currently being evaluated
+- Used for cycle detection
+- Could be repurposed for causality
+
+### Design Vision
+
+#### Decision Tree
+
+A tree structure recording the evaluation path:
+
+```
+allow = true
+├── Rule: data.auth.allow (body 1 succeeded)
+│   ├── Statement: input.role == "admin" → true
+│   │   └── input.role = "admin" (from input)
+│   └── Statement: input.active == true → true
+│       └── input.active = true (from input)
+└── Default: data.auth.deny = false (not triggered)
+```
+
+#### Value Provenance
+
+Track where each value came from:
+- `input.role` → from user input
+- `data.allowed_roles` → from data document loaded at path X
+- `count(data.items)` → computed by builtin from data
+
+#### Counterfactual Analysis
+
+"What would change if `input.role` were `"viewer"` instead?"
+- Re-evaluate with modified input
+- Compare decision trees
+- Report which statements changed outcome
+
+### Architecture Implications
+
+1. **Opt-in overhead**: causality tracking adds memory and CPU cost.
+   Must be behind a feature flag or runtime configuration. Never in
+   the hot path for production evaluation.
+
+2. **Value annotation**: Values may need optional metadata:
+   ```rust
+   struct AnnotatedValue {
+       value: Value,
+       provenance: Option<Provenance>,  // Where it came from
+   }
+   ```
+
+3. **Evaluation hooks**: the interpreter/RVM need "observation points"
+   where causality information is recorded. These should be no-ops
+   when tracking is disabled.
+
+4. **Serializable traces**: decision trees and provenance information
+   need to be serializable (JSON) for audit logging and external
+   tooling.
+
+5. **Deterministic replay**: for counterfactual analysis, the evaluation
+   must be deterministic. This means:
+   - `time.now_ns()` must be mockable
+   - Random builtins must be seedable
+   - External data must be snapshotted
+
+### Connection to Partial Evaluation
+
+Causality and partial evaluation complement each other:
+- Partial evaluation identifies the **relevant** parts of a policy
+- Causality tracking explains the **decisions** within those parts
+- Together they answer: "given what we know, what decisions were made and why?"
+
+## Design Principles for Both Features
+
+1. **Keep evaluation logic pure** — side-effect-free functions are easier
+   to partially evaluate and track causally.
+
+2. **Document invariants explicitly** — invariants that hold during
+   evaluation are the foundation for symbolic reasoning.
+
+3. **Prefer exhaustive pattern matching** — every case handled explicitly
+   makes symbolic analysis tractable.
+
+4. **Separate observation from computation** — tracking infrastructure
+   should be orthogonal to evaluation logic.
+
+5. **Correct today, analyzable tomorrow** — current code should be
+   designed so these features can be added without fundamental restructuring.

docs/knowledge/compilation-pipeline.md

@@ -0,0 +1,260 @@
+<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->
+<!-- Licensed under the MIT License. -->
+
+# Knowledge: Compilation Pipeline
+
+Deep knowledge about the scheduler, loop hoisting, and destructuring planner.
+Read this before modifying `src/scheduler.rs` or `src/compiler/`.
+
+## Pipeline Overview
+
+```
+AST (with eidx, sidx, qidx indices)
+    ↓
+Scheduler  — determines statement execution order via topological sort
+    ↓
+LoopHoister — identifies loops to hoist and creates binding plans
+    ↓
+RVM Compiler — generates bytecode using hoisted info (if RVM feature)
+    ↓
+Program — bytecode + literal table + metadata
+```
+
+The interpreter also uses the scheduler and hoister output directly (without
+the RVM compiler step).
+
+## AST Indexing
+
+Every AST node carries an index for O(1) lookup of pre-computed information:
+
+- `Expr.eidx: u32` — unique expression index within a module
+- `LiteralStmt.sidx: u32` — statement index within a query
+- `Query.qidx: u32` — query index within a module
+
+These indices are assigned sequentially during parsing and used as keys into
+lookup tables by the scheduler and hoister.
+
+## Scheduler (`src/scheduler.rs`, ~1,218 lines)
+
+### Purpose
+
+Determine safe statement execution order within rule bodies. Statements may
+define and use variables, creating dependencies:
+
+```rego
+allow {
+    user := input.user           # defines 'user'
+    role := user.role            # uses 'user', defines 'role'
+    role == "admin"              # uses 'role'
+}
+```
+
+The scheduler topologically sorts statements so each statement's dependencies
+are satisfied before it executes.
+
+### Core Data Structures
+
+```rust
+struct Definition<Str> {
+    var: Str,               // Variable being defined (empty string = condition-only)
+    used_vars: Vec<Str>,    // Variables this definition depends on
+}
+
+struct StmtInfo<Str> {
+    definitions: Vec<Definition<Str>>,  // A statement can define multiple vars
+}
+
+struct QuerySchedule {
+    scope: Scope,           // Variable binding information
+    order: Vec<u16>,        // Computed statement execution order
+}
+```
+
+### Scheduling Algorithm
+
+The `schedule()` function performs topological sort:
+
+1. **Build dependency map**: `defining_stmts` maps each variable to the
+   statements that define it
+2. **Initialize**: track `defined_vars` (set), `scheduled` (bool array)
+3. **Process variables in discovery order**:
+   - For each variable, try to schedule all statements that define it
+   - A statement is schedulable when all its `used_vars` are already defined
+   - When a statement is scheduled, all its `defined_vars` become available
+   - This cascades — newly defined vars may unblock other statements
+4. **Handle cycles**: if not all statements scheduled, fall back to source order
+
+**Multi-definition statements**: A single statement can define multiple
+variables (e.g., `x, y := foo()`). These are handled with a queue-based
+approach that processes definitions within the statement iteratively.
+
+**Empty-variable statements**: Condition-only statements (like `x > 10`) use
+an empty string as the variable name. These are re-evaluated whenever any
+variable becomes defined, since they may become schedulable.
+
+### Analysis Pipeline
+
+`Analyzer.analyze()`:
+1. Add rules and aliases to scopes
+2. Gather functions into `FunctionTable`
+3. For each module → for each rule → for each query body:
+   - `analyze_query()` examines each statement
+   - Extracts `StmtInfo` (what variables defined/used)
+   - Calls `schedule()` to get execution order
+   - Stores result in `Schedule` lookup table
+
+## Loop Hoisting (`src/compiler/hoist.rs`, ~914 lines)
+
+### Purpose
+
+Identify iteration patterns that can be pre-computed and optimized:
+
+```rego
+# Before hoisting: interpreter must figure out iteration at runtime
+x[i] > 5    # Is 'i' a bound variable or should we iterate?
+
+# After hoisting: pre-computed as a loop with known structure
+HoistedLoop { key: i, collection: x, loop_type: IndexIteration }
+```
+
+### Core Data Structures
+
+```rust
+struct HoistedLoop {
+    loop_expr: Option<ExprRef>,   // The expression that generates the loop
+    key: Option<ExprRef>,         // Index/key variable
+    value: ExprRef,               // Iteration value
+    collection: ExprRef,          // Collection being iterated
+    loop_type: LoopType,          // IndexIteration or Walk
+}
+
+struct HoistedLoopsLookup {
+    statement_loops: Lookup<Vec<HoistedLoop>>,    // Per-statement loops
+    expr_loops: Lookup<Vec<HoistedLoop>>,         // Per-output-expression loops
+    expr_binding_plans: Lookup<BindingPlan>,       // Per-assignment binding plans
+    query_contexts: Lookup<ScopeContext>,           // Per-query scope info
+}
+```
+
+The `Lookup` type uses 2D indexing: `(module_index, item_index)`.
+
+### What Gets Hoisted
+
+**Index iteration**: `x[i]` where `i` is unbound → iterate over indices of `x`
+
+**Walk builtin**: `walk(input, [path, value])` → tree traversal loop
+
+**NOT hoisted**: `x[i]` where `i` is already bound (just an index access)
+
+### ScopeContext
+
+The hoister tracks variable binding state during analysis:
+
+```rust
+struct ScopeContext {
+    context_type: ContextType,           // Rule/Comprehension/Every/Query
+    bound_vars: BTreeSet<String>,        // All bound variables
+    current_scope_bound_vars: BTreeSet<String>,  // Newly bound in this scope
+    unbound_vars: BTreeSet<String>,      // Declared but not yet bound
+    local_vars: BTreeSet<String>,        // Scheduler-tracked locals
+}
+```
+
+The key method `should_hoist_as_loop()` determines whether a variable access
+should be a loop: true if the variable is unbound, local (per scheduler), or
+not in the bound set.
+
+### Analysis Flow
+
+```
+LoopHoister.populate()
+  → populate_module()
+    → populate_rule()  — bind parameters, extract key/value expressions
+      → populate_query()  — process statements in scheduled order
+        → populate_statement()  — analyze literals, store hoisted loops
+          → analyze_expr()  — recursive expression analysis
+            → detect RefBrack with unbound index → HoistedLoop
+            → detect walk() call → HoistedLoop
+            → detect assignment → BindingPlan
+```
+
+## Destructuring Planner (`src/compiler/destructuring_planner/`)
+
+### Purpose
+
+Create plans for pattern matching in assignments, parameters, and `some...in`:
+
+```rego
+[x, y] := func()          # Array destructuring
+{a: b} := obj             # Object destructuring
+some k, v in collection    # some-in binding
+```
+
+### Plan Types
+
+```rust
+enum DestructuringPlan {
+    Var(Span),                    // Bind value to variable
+    Ignore,                       // Wildcard (_)
+    EqualityExpr(ExprRef),        // Match against expression
+    EqualityValue(Value),         // Match against literal
+    Array { element_plans },      // Recursive array destructuring
+    Object { field_plans, dynamic_fields },  // Recursive object destructuring
+}
+
+enum BindingPlan {
+    Destructuring(DestructuringPlan),
+    Assignment(AssignmentPlan),
+    SomeIn(SomeInPlan),
+    LoopIndex(LoopIndexPlan),
+    Parameter(ParameterPlan),
+}
+```
+
+### Assignment Plans
+
+Two assignment operators have different binding semantics:
+
+- **`:=`** (ColonEquals): Only LHS can bind variables. Strict.
+- **`=`** (Equals): Both sides can bind. Two-pass analysis needed.
+
+### Variable Binding Context
+
+```rust
+trait VariableBindingContext {
+    fn is_var_unbound(&self, var_name: &str, scoping: ScopingMode) -> bool;
+    fn has_same_scope_binding(&self, var_name: &str) -> bool;
+}
+```
+
+`ScopingMode::RespectParent` prevents shadowing. `ScopingMode::AllowShadowing`
+allows it (used for function parameters).
+
+## Key Invariants
+
+1. **Scheduled order must respect dependencies** — if statement B uses a
+   variable defined by statement A, A must execute before B.
+
+2. **Hoisted loops must match runtime behavior** — the hoister's analysis of
+   bound vs unbound must match what the interpreter/RVM sees at runtime.
+
+3. **Binding plans must be complete** — every variable that appears in a
+   destructuring pattern must have a binding plan (Var, Ignore, or Equality).
+
+4. **Lookup indices must be consistent** — the same `(module_index, eidx/sidx/qidx)`
+   must refer to the same AST node across scheduler, hoister, and executor.
+
+## Common Pitfalls
+
+1. **Scope context inheritance** — child contexts (comprehensions, every)
+   inherit bound_vars from parent but have their own new bindings.
+
+2. **Multi-definition statements** — a single `=` can bind variables on
+   both sides, creating complex dependency chains.
+
+3. **Loop hoisting vs bound variables** — `x[i]` is a loop only if `i` is
+   unbound. Mistakenly hoisting a bound index access creates incorrect
+   iteration behavior.
+
+4. **Query schedule vs source order** — the scheduled order may differ from
+   source order. Code that assumes source order will break.

docs/knowledge/engine-api.md

@@ -0,0 +1,179 @@
+<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->
+<!-- Licensed under the MIT License. -->
+
+# Knowledge: Engine API
+
+Deep knowledge about the public `Engine` API (`src/engine.rs`). Read this
+before modifying the engine's public interface or evaluation flow.
+
+## Engine Structure
+
+```rust
+pub struct Engine {
+    modules: Rc<Vec<Ref<Module>>>,              // Loaded policy modules
+    interpreter: Interpreter,                    // Execution engine
+    prepared: bool,                             // Compilation state flag
+    rego_v1: bool,                              // Language version
+    execution_timer_config: Option<ExecutionTimerConfig>,
+    policy_length_config: PolicyLengthConfig,    // File size limits
+}
+```
+
+## Primary API Flow
+
+### 1. Policy Loading
+
+```rust
+pub fn add_policy(&mut self, path: String, rego: String) -> Result<String>
+pub fn add_policy_from_file(&mut self, path: impl AsRef<Path>) -> Result<String>
+```
+
+- Parses Rego source via Lexer → Parser → AST
+- Returns the package name (e.g., `"data.test"`)
+- Sets `prepared = false` to trigger recompilation on next eval
+- Enforces `PolicyLengthConfig` limits
+
+### 2. Data and Input
+
+```rust
+pub fn add_data(&mut self, data: Value) -> Result<()>  // Merge into data document
+pub fn add_data_json(&mut self, data: &str) -> Result<()>
+pub fn set_input(&mut self, input: Value)
+pub fn set_input_json(&mut self, input: &str) -> Result<()>
+pub fn clear_data(&mut self)
+```
+
+`add_data()` merges into the existing data document. It requires the value
+to be an object (checked). Conflict detection on merge.
+
+### 3. Evaluation
+
+| Method | Returns | Use Case |
+|--------|---------|----------|
+| `eval_rule(rule)` | `Value` | Direct rule evaluation (fast) |
+| `eval_query(query, tracing)` | `QueryResults` | OPA-compatible with bindings |
+| `eval_bool_query(query)` | `bool` | Boolean shortcut |
+| `eval_allow_query()` | `bool` | Common deny-by-default pattern |
+| `eval_modules(tracing)` | `Value` | Evaluate all loaded modules |
+
+### 4. Compilation (for repeated evaluation)
+
+```rust
+pub fn compile_for_target(&mut self) -> Result<CompiledPolicy>
+pub fn compile_with_entrypoint(&mut self, rule: &Rc<str>) -> Result<CompiledPolicy>
+```
+
+Returns `CompiledPolicy` — an immutable, precompiled artifact that can be
+evaluated many times with different inputs:
+
+```rust
+let compiled = engine.compile_for_target()?;
+// Later, potentially in a different thread:
+let result = compiled.eval_with_input(input)?;
+```
+
+### 5. Configuration
+
+```rust
+pub fn set_rego_v0(&mut self, enabled: bool)     // Language version
+pub fn set_execution_timer_config(config)         // Timeout limits
+pub fn set_policy_length_config(config)           // File size limits
+pub fn set_strict_builtin_errors(b: bool)         // Error vs Undefined for type mismatches
+pub fn add_extension(name, arity, func)           // Custom functions
+```
+
+## CompiledPolicy
+
+```rust
+pub struct CompiledPolicy {
+    inner: Rc<CompiledPolicyData>,
+}
+
+struct CompiledPolicyData {
+    modules: Rc<Vec<Ref<Module>>>,
+    schedule: Option<Rc<Schedule>>,           // Pre-computed statement order
+    rules: Map<String, Vec<Ref<Rule>>>,       // Rule path → rules
+    default_rules: Map<String, Vec<...>>,     // Default rules
+    imports: BTreeMap<String, Ref<Expr>>,
+    functions: FunctionTable,                 // User-defined functions
+    rule_paths: Set<String>,
+    loop_hoisting_table: HoistedLoopsLookup,  // Pre-computed loop info
+    data: Option<Value>,                      // Preloaded data
+    strict_builtin_errors: bool,
+    extensions: Map<String, (u8, Rc<Box<dyn Extension>>)>,
+}
+```
+
+**Benefits of CompiledPolicy:**
+- Schedule, loop hoisting, and function table pre-computed once
+- Can be cloned cheaply (Rc internals)
+- Supports repeated evaluation with different inputs
+- Thread-safe when using `arc` feature
+
+## Internal Evaluation Flow
+
+When `eval_rule()` is called:
+
+1. **Preparation** (if not `prepared`):
+   - Gather all functions from modules → `FunctionTable`
+   - Run scheduler on all queries → `Schedule`
+   - Run loop hoister → `HoistedLoopsLookup`
+   - Build `CompiledPolicyData`
+   - Set `prepared = true`
+
+2. **Interpreter setup**:
+   - Set data and input on interpreter
+   - Set current module context
+
+3. **Evaluation**:
+   - Find rule in `compiled_policy.rules`
+   - Call `interpreter.eval_rule()`
+   - Return result
+
+## Multiple Module Management
+
+- Modules stored as `Rc<Vec<Ref<Module>>>`
+- Each module declares a package namespace (e.g., `package auth`)
+- Rules qualified by package path: `data.auth.allow`
+- Imports resolve cross-module references
+- Functions tracked globally in `FunctionTable`
+
+## Extensions API
+
+Custom functions can be registered at runtime:
+
+```rust
+engine.add_extension(
+    "custom.check".to_string(),
+    2,  // arity
+    Rc::new(Box::new(|args| -> Result<Value> {
+        // implementation
+    })),
+)?;
+```
+
+Extensions are available to Rego policies as builtin functions.
+
+## Metadata Access
+
+```rust
+pub fn get_packages(&self) -> Result<Vec<String>>      // Package names
+pub fn get_policies(&self) -> Result<Vec<Source>>       // Policy sources
+pub fn get_policies_as_json(&self) -> Result<String>    // JSON representation
+pub fn get_coverage_report(&self) -> Result<Report>     // Code coverage
+```
+
+## Key Design Decisions
+
+1. **Lazy compilation** — policies aren't compiled until first evaluation.
+   `prepared` flag tracks whether compilation is needed.
+
+2. **Data merging** — `add_data()` merges, doesn't replace. Multiple data
+   sources accumulate into the data document.
+
+3. **Input replacement** — `set_input()` replaces, doesn't merge. Each
+   evaluation gets a fresh input.
+
+4. **Clone semantics** — `Engine::clone()` clones all persistent state
+   (policies, data, configuration) but resets runtime state (processed
+   rules, caches). The clone is ready for independent evaluation.

docs/knowledge/error-handling-migration.md

@@ -0,0 +1,194 @@
+<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->
+<!-- Licensed under the MIT License. -->
+
+# Knowledge: Error Handling Migration
+
+Deep knowledge about regorus's error handling patterns and the ongoing
+migration from `anyhow` to `thiserror`. Read this before adding error
+handling to new code or modifying existing error paths.
+
+## Current State
+
+The codebase has two error handling approaches coexisting:
+
+### Legacy: anyhow (widespread)
+
+Most of the codebase uses `anyhow::Result` with `bail!()` and `anyhow!()`:
+
+```rust
+use anyhow::{anyhow, bail, Result};
+
+fn eval_something(&mut self) -> Result<Value> {
+    let v = map.get("key").ok_or_else(|| anyhow!("missing key"))?;
+    if condition_fails {
+        bail!("evaluation failed: {reason}");
+    }
+    Ok(value)
+}
+```
+
+Found in: `src/interpreter.rs`, `src/engine.rs`, `src/parser.rs`,
+`src/lexer.rs`, `src/value.rs`, `src/number.rs`, `src/builtins/`, and most
+other modules.
+
+### Target: thiserror (RVM leads)
+
+The RVM uses strongly typed error enums:
+
+```rust
+use thiserror::Error;
+
+#[derive(Error, Debug, Clone, PartialEq)]
+pub enum VmError {
+    #[error("Execution stopped: exceeded maximum instruction limit of {limit} after {executed} instructions (pc={pc})")]
+    InstructionLimitExceeded { limit: usize, executed: usize, pc: usize },
+
+    #[error("Register index {index} out of bounds (pc={pc}, register_count={register_count})")]
+    RegisterIndexOutOfBounds { index: u8, pc: usize, register_count: usize },
+
+    // ... 30+ variants covering every VM error case
+}
+
+pub type Result<T> = core::result::Result<T, VmError>;
+```
+
+Found in: `src/rvm/vm/errors.rs`
+
+## The VmError Pattern (Reference Implementation)
+
+Key design principles visible in `VmError`:
+
+**1. Every variant carries context:**
+```rust
+InstructionLimitExceeded { limit: usize, executed: usize, pc: usize }
+```
+Not just "limit exceeded" — includes the limit, actual count, and program counter.
+
+**2. Program counter in every variant:**
+```rust
+// Every single variant includes `pc: usize`
+RegisterNotObject { register: u8, value: Value, pc: usize },
+LiteralIndexOutOfBounds { index: u16, pc: usize },
+```
+This is a debugging aid — every error can be traced to the exact instruction.
+
+**3. Exhaustive coverage:**
+30+ variants covering every known error case. No catch-all "Other(String)".
+
+**4. Derives Clone and PartialEq:**
+```rust
+#[derive(Error, Debug, Clone, PartialEq)]
+```
+Clone enables error propagation without ownership transfer. PartialEq enables
+testing error conditions precisely.
+
+**5. Type alias for ergonomics:**
+```rust
+pub type Result<T> = core::result::Result<T, VmError>;
+```
+
+**6. Bridge from anyhow:**
+```rust
+impl From<anyhow::Error> for VmError {
+    fn from(err: anyhow::Error) -> Self {
+        VmError::ArithmeticError { message: format!("{}", err), pc: 0 }
+    }
+}
+```
+This allows the RVM to call into legacy code that returns `anyhow::Result`.
+
+## Migration Strategy
+
+### For New Code
+
+**Always use thiserror.** Define a module-specific error enum:
+
+```rust
+use thiserror::Error;
+
+#[derive(Error, Debug, Clone, PartialEq)]
+pub enum MySubsystemError {
+    #[error("invalid input: {0}")]
+    InvalidInput(String),
+
+    #[error("resource limit exceeded: {current} > {limit}")]
+    ResourceLimitExceeded { current: usize, limit: usize },
+}
+
+pub type Result<T> = core::result::Result<T, MySubsystemError>;
+```
+
+### For Existing Code
+
+When modifying existing functions that use `anyhow`:
+- **Within the same module**: continue with `anyhow` for consistency
+- **At module boundaries**: consider wrapping `anyhow::Error` in a typed variant
+- **Incremental migration**: converting a whole module at once is better than
+  mixing styles within a single module
+
+### Bridge Pattern
+
+When typed-error code calls anyhow code (or vice versa):
+
+```rust
+// Typed → anyhow (automatic via anyhow's From impl)
+fn caller() -> anyhow::Result<Value> {
+    typed_function()?;  // VmError auto-converts to anyhow::Error
+    Ok(value)
+}
+
+// Anyhow → typed (explicit conversion needed)
+fn caller() -> Result<Value, VmError> {
+    anyhow_function().map_err(|e| VmError::Internal {
+        message: format!("{}", e),
+        pc: current_pc,
+    })?;
+    Ok(value)
+}
+```
+
+## Error Message Guidelines
+
+### For OPA Conformance
+
+Builtin error messages **must match OPA exactly** — the conformance test suite
+compares literally. When implementing builtins, check the OPA Go source.
+
+### For Internal Errors
+
+- Include enough context to diagnose without a debugger
+- Include identifiers (register index, PC, rule name, etc.)
+- Don't include sensitive data (user input, policy content)
+- Use structured fields, not string formatting:
+
+```rust
+// ✗ Bad
+#[error("register {0} out of bounds at pc {1}")]
+RegisterOutOfBounds(u8, usize),
+
+// ✓ Good — named fields are self-documenting
+#[error("register index {index} out of bounds (pc={pc}, register_count={register_count})")]
+RegisterIndexOutOfBounds { index: u8, pc: usize, register_count: usize },
+```
+
+## Panic Safety Connection
+
+Error handling is the front line of panic safety. The deny lints forbid
+`unwrap()`, `expect()`, `panic!()`, etc. Every fallible operation must return
+`Result`. This is not just style — in daemon mode, a panic crashes the service.
+
+The error migration makes this stronger: with typed errors, every failure mode
+is enumerated and the compiler ensures all are handled. With `anyhow`, errors
+are opaque and may be accidentally swallowed.
+
+## no_std Compatibility
+
+Both `anyhow` and `thiserror` support `no_std` with `default-features = false`:
+
+```toml
+anyhow = { version = "1.0", default-features = false }
+thiserror = { version = "2.0", default-features = false }
+```
+
+Error types must use `alloc::string::String` instead of `std::string::String`
+and avoid `std::io::Error` without a feature gate.

docs/knowledge/feature-composition.md

@@ -0,0 +1,183 @@
+<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->
+<!-- Licensed under the MIT License. -->
+
+# Knowledge: Feature Composition
+
+Deep knowledge about regorus's feature flag system and the risks of
+non-default feature combinations. Read this before adding features or
+modifying feature-gated code.
+
+## Feature Architecture
+
+### Default Features
+
+```toml
+default = ["full-opa", "arc", "rvm"]
+```
+
+- **`full-opa`**: All OPA-compatible builtins. Implies `std`.
+- **`arc`**: `Arc` instead of `Rc` for thread safety.
+- **`rvm`**: Rego Virtual Machine compilation and execution.
+
+### Composite Features
+
+**`full-opa`** includes: base64, base64url, coverage, glob, graph, hex, http,
+jsonschema, net, opa-runtime, regex, cache, semver, std, time, uuid, urlquery,
+yaml.
+
+**`opa-no-std`** includes: arc, base64, base64url, coverage, graph, hex,
+no_std, opa-runtime, regex, semver, lazy_static/spin_no_std. Note this
+**excludes** builtins that require `std` (glob, time, jsonschema, yaml, etc).
+
+### The no_std / std Boundary
+
+The crate is `#![no_std]` by default with `extern crate alloc`.
+
+- **`std`** feature: enables `std` library, parking_lot, filesystem, threading
+- **`no_std`** feature: enables `lazy_static/spin_no_std` for spinlock-based lazy statics
+
+**These are NOT mutually exclusive in Cargo.** If both are enabled, `std` wins.
+But `no_std` should be tested alone:
+
+```bash
+cargo xtask test-no-std  # Builds for thumbv7m-none-eabi
+```
+
+### The arc Feature
+
+Controls whether shared data uses `Rc` or `Arc`:
+
+```rust
+// In src/lib.rs (conditional type alias)
+#[cfg(feature = "arc")]
+type Rc<T> = alloc::sync::Arc<T>;
+#[cfg(not(feature = "arc"))]
+type Rc<T> = alloc::rc::Rc<T>;
+```
+
+**`arc` is default.** Disabling it gives single-threaded performance but breaks
+thread safety. The FFI crate's contention detection (`contention_checks`)
+requires `arc`.
+
+## Known Pitfalls
+
+### Issue #595 Pattern
+
+Feature combinations that compile individually may fail together. Example:
+a feature adds a dependency that conflicts with `no_std`, or a feature-gated
+module uses `std` types without a feature gate.
+
+**Prevention:**
+- Always test with `--no-default-features` plus minimal feature sets
+- CI checks key combinations explicitly
+
+### Compilation Verification Matrix
+
+When adding or modifying features, verify these combinations compile:
+
+```bash
+# Minimal (no_std, no arc, no rvm)
+cargo check --no-default-features
+
+# no_std with arc
+cargo check --no-default-features --features arc,opa-no-std
+
+# std with arc and rvm (common production config)
+cargo check --no-default-features --features std,arc,rvm
+
+# Everything
+cargo check --all-features
+
+# The full CI suite checks more combinations
+cargo xtask ci-debug
+```
+
+### Feature-Gated Code Correctness
+
+Common mistakes:
+
+**1. Using std types without gate:**
+```rust
+// ✗ Bad — breaks no_std
+use std::collections::HashMap;
+
+// ✓ Good — available in no_std via alloc
+use alloc::collections::BTreeMap;
+
+// ✓ Good — gated when std is required
+#[cfg(feature = "std")]
+use std::path::Path;
+```
+
+**2. Feature implies another but not declared:**
+```rust
+// ✗ Bad — regex module uses std but doesn't declare dependency
+[features]
+regex = ["dep:regex"]  # regex crate needs std!
+
+// ✓ Good — declare the implication
+regex = ["dep:regex"]  # regex default-features=false works in no_std
+```
+
+**3. Conditional compilation in wrong direction:**
+```rust
+// ✗ Bad — dead code when feature absent, no compile error
+#[cfg(feature = "myfeature")]
+fn helper() { ... }
+
+fn caller() {
+    helper();  // ERROR: `helper` doesn't exist without myfeature
+}
+
+// ✓ Good — gate the caller too
+#[cfg(feature = "myfeature")]
+fn caller() {
+    helper();
+}
+```
+
+### docsrs Annotation
+
+Public feature-gated APIs must have the docsrs annotation so docs.rs shows
+which feature is required:
+
+```rust
+#[cfg(feature = "myfeature")]
+#[cfg_attr(docsrs, doc(cfg(feature = "myfeature")))]
+pub fn my_function() -> Result<()> { .. }
+```
+
+## Adding a New Feature: Checklist
+
+1. Add to `[features]` in `Cargo.toml` with optional dependency
+2. Gate the module: `#[cfg(feature = "myfeature")] mod myfeature;`
+3. Gate registration (builtins, languages, etc.)
+4. Gate public API with docsrs annotation
+5. Add to `full-opa` if it's an OPA-standard feature
+6. Add to `opa-no-std` if it works without std
+7. Verify compilation with the matrix above
+8. Run `cargo xtask ci-debug` for the full suite
+9. Consider adding the combination to CI if it's a common configuration
+
+## Dependencies and no_std
+
+When adding dependencies:
+- Check if the crate supports `no_std` (look for `default-features = false`)
+- Use `default-features = false` and enable only needed features
+- If the crate requires `std`, the feature must imply `std`
+- Prefer `core`/`alloc` over external crates where feasible
+
+Current dependency pattern:
+```toml
+serde = { version = "1.0", default-features = false, features = ["derive", "rc", "alloc"] }
+regex = { version = "1.12", optional = true, default-features = false }
+```
+
+## The Rc Type Alias
+
+The crate defines a type alias `Rc` that maps to either `alloc::rc::Rc` or
+`alloc::sync::Arc` based on the `arc` feature. This alias is used throughout
+the codebase — in `Value`, `Number`, and everywhere shared ownership is needed.
+
+**Never use `alloc::rc::Rc` or `alloc::sync::Arc` directly in the core crate.**
+Always use the type alias `Rc` to ensure the `arc` feature works correctly.

docs/knowledge/ffi-boundary.md

@@ -0,0 +1,212 @@
+<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->
+<!-- Licensed under the MIT License. -->
+
+# Knowledge: FFI Boundary
+
+Deep knowledge about regorus's foreign function interface and multi-language
+binding architecture. Read this before modifying `bindings/` or the core
+library's public API.
+
+## Architecture
+
+```
+                    regorus (Rust core library)
+                              │
+                    bindings/ffi/ (base FFI crate)
+                              │
+        ┌────────┬────────┬───┴───┬────────┬────────┐
+        │        │        │       │        │        │
+     C/C++    C#/NuGet  Java   Python   Ruby    WASM
+   (cbindgen) (csbindgen)(jni-rs)(PyO3) (magnus)(wasm-pack)
+     CMake    MSBuild    Maven  maturin bundler  npm
+```
+
+The FFI crate (`bindings/ffi/`) is the **security boundary**. Rust's compiler
+guarantees do not extend across it.
+
+## Opaque Handle Pattern
+
+All Rust objects are exposed to C as opaque pointers:
+
+```rust
+// Rust side
+pub struct RegorusEngine {
+    engine: Handle<::regorus::Engine>,  // Rc<RefCell<>> or Arc<RwLock<>>
+}
+
+#[no_mangle]
+pub extern "C" fn regorus_engine_new() -> *mut RegorusEngine {
+    Box::into_raw(Box::new(RegorusEngine::new(engine)))
+}
+
+#[no_mangle]
+pub extern "C" fn regorus_engine_drop(engine: *mut RegorusEngine) {
+    if let Ok(e) = to_ref(engine) {
+        unsafe { let _ = Box::from_raw(ptr::from_mut(e)); }
+    }
+}
+```
+
+**Invariant:** Every `Box::into_raw()` must have a corresponding `Box::from_raw()`
+in a drop function. Missing drops = memory leaks.
+
+## Null Pointer Validation
+
+Every pointer parameter is validated at the FFI boundary:
+
+```rust
+pub(crate) fn to_ref<'a, T>(t: *mut T) -> Result<&'a mut T> {
+    unsafe { t.as_mut().ok_or_else(|| anyhow!("null pointer")) }
+}
+
+pub(crate) fn from_c_str(s: *const c_char) -> Result<String> {
+    if s.is_null() { bail!("null pointer"); }
+    unsafe { CStr::from_ptr(s).to_str().map_err(|e| anyhow!("invalid utf8: {e}")).map(|s| s.to_string()) }
+}
+```
+
+**Invariant:** No FFI function may dereference a pointer without checking for null.
+
+## Contention Detection
+
+The FFI handle uses configurable locking (`bindings/ffi/src/lock.rs`):
+
+| Feature flags | Handle type | Cost | Safety |
+|---------------|-------------|------|--------|
+| `std` + `contention_checks` | `Arc<RwLock<T>>` | Higher | Detects concurrent access |
+| `std` only | `Rc<RefCell<T>>` | Lower | Single-thread assumption |
+| `no_std` | `Rc<RefCell<T>>` | Lowest | Single-thread only |
+
+The contention error message explicitly tells users to clone:
+> "regorus engine handle is already in use; clone the engine before sharing across threads"
+
+## Panic Containment and Poisoning
+
+**Every FFI entry point wraps in `with_unwind_guard()`** which:
+
+1. Checks if engine is already poisoned → return `RegorusStatus::Poisoned`
+2. Installs a temporary panic hook to capture backtrace
+3. Calls `panic::catch_unwind()` around the function body
+4. If panic caught → permanently poisons engine via `AtomicBool`
+5. Returns `RegorusStatus::Panic` with the captured backtrace
+
+**Once poisoned, the engine is PERMANENTLY dead.** All subsequent calls return
+`RegorusStatus::Poisoned`. There is no recovery. This is intentional — after a
+panic, internal state may be corrupt.
+
+## Result Encoding
+
+All FFI functions return `RegorusResult`:
+
+```c
+typedef struct {
+    RegorusStatus status;        // Ok, Error, Panic, Poisoned, ...
+    RegorusDataType data_type;   // None, String, Boolean, Integer, Pointer
+    char* output;                // Owned by Rust — caller MUST call regorus_result_drop()
+    bool bool_value;
+    long long int_value;
+    void* pointer_value;
+    char* error_message;         // Owned by Rust — freed by regorus_result_drop()
+} RegorusResult;
+```
+
+**CRITICAL:** String ownership transfers to C via `CString::into_raw()`. If the
+caller doesn't call `regorus_result_drop()`, memory leaks.
+
+## Binary Buffer Pattern
+
+For binary data (serialized programs), `RegorusBuffer` transfers Vec ownership:
+
+```rust
+pub struct RegorusBuffer {
+    pub data: *mut u8,
+    pub len: usize,
+    pub capacity: usize,
+}
+```
+
+Created via `RegorusBuffer::from_vec()` (which `mem::forget()`s the Vec),
+freed via `regorus_buffer_drop()` (which reconstructs and drops the Vec).
+
+## Language-Specific Binding Patterns
+
+### C — Raw FFI
+No wrapper. Manual `regorus_result_drop()` and `regorus_engine_drop()` calls.
+Error handling via status code checks.
+
+### C++ — RAII
+`regorus.hpp` wraps with:
+- `Result` class: move-only, destructor calls `regorus_result_drop()`
+- `Engine` class: destructor calls `regorus_engine_drop()`
+- Copy prevention via deleted copy constructor/assignment
+
+### C# — SafeHandle with HandleGate
+Most sophisticated wrapper:
+- `SafeHandle` integrates with .NET finalizer
+- `HandleGate` tracks in-flight operations
+- `DangerousAddRef()`/`DangerousRelease()` pins handle during native calls
+- Dispose waits up to 50ms for in-flight calls to drain
+- Thread-safe concurrent access tracking
+
+### Java — AutoCloseable + JNI
+- Stores opaque `long` pointer (64-bit address)
+- `AutoCloseable` for `try-with-resources` blocks
+- `close()` calls `nativeDestroyEngine()`
+
+### Python — PyO3 Direct Embedding
+- `#[pyclass(unsendable)]` embeds Rust Engine in Python object
+- Python GC owns the object, Rust `Drop` is automatic
+- No separate FFI layer — PyO3 marshals directly
+
+### Go — cgo
+- Stores `*C.RegorusEngine` opaque pointer
+- `defer` for cleanup ordering
+- Manual CString conversion with `C.CString()`/`C.free()`
+
+### Ruby — Magnus Native Extension
+- Rust struct wrapped as Ruby class
+- Ruby GC manages lifecycle via finalizer
+
+### WASM — wasm-pack
+- Compiled to WebAssembly, exposed via JavaScript bindings
+- No pointer management — WASM linear memory handles it
+
+## Custom Allocator Support
+
+The FFI crate supports host-provided allocators:
+
+```rust
+#[cfg(feature = "custom_allocator")]
+extern "C" {
+    fn regorus_aligned_alloc(alignment: usize, size: usize) -> *mut u8;
+    fn regorus_free(ptr: *mut u8);
+}
+```
+
+This allows C#/JVM/Go hosts to provide their own allocator, which is important
+for memory tracking and limit enforcement in managed runtimes.
+
+## Impact of Core API Changes
+
+When changing the core library's public API:
+
+1. **Every binding must be updated** — 9 language targets
+2. **FFI function signature changes** require updating:
+   - `bindings/ffi/src/engine.rs` (or relevant FFI module)
+   - C/C++ headers (auto-generated by cbindgen, but verify)
+   - C# P/Invoke declarations
+   - Java JNI native method declarations
+   - Go cgo function declarations
+   - WASM bindings
+3. **Run `cargo xtask test-all-bindings`** to verify all targets
+4. **New public methods** need FFI wrappers, documentation in all languages
+5. **Behavioral changes** may need binding-level test updates
+
+## Security Considerations
+
+- The FFI boundary is where type safety ends — validate everything
+- Pointer arithmetic for array parameters must check bounds carefully
+- String encoding (UTF-8 vs platform) must be validated at the boundary
+- Panic containment prevents Rust panics from unwinding into C/C++
+- Poisoning prevents use-after-panic of potentially corrupt state
+- Memory ownership must be crystal clear — who allocates, who frees

docs/knowledge/interpreter-architecture.md

@@ -0,0 +1,216 @@
+<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->
+<!-- Licensed under the MIT License. -->
+
+# Knowledge: Interpreter Architecture
+
+Deep knowledge about the tree-walking interpreter (`src/interpreter.rs`).
+This is a 4,400+ line file and the legacy execution path. Read this before
+modifying evaluation logic.
+
+## Core Data Structures
+
+### Interpreter State
+
+```rust
+pub struct Interpreter {
+    compiled_policy: Rc<CompiledPolicyData>,
+    data: Value,                           // Data document (rules materialize here)
+    input: Value,                          // User-provided input
+    with_document: Value,                  // Temporary overrides via `with`
+    scopes: Vec<Scope>,                    // Variable binding stack
+    contexts: Vec<Context>,                // Evaluation context stack
+    processed: BTreeSet<Ref<Rule>>,        // Rules already evaluated
+    processed_paths: Value,                // Data paths already evaluated
+    rule_values: RuleValues,               // Cached rule evaluation results
+    active_rules: Vec<Ref<Rule>>,          // Stack for cycle detection
+    loop_var_values: ExprLookup,           // Loop variable cache
+    builtins_cache: BTreeMap<..., Value>,   // Builtin result cache
+    execution_timer: ExecutionTimer,       // Time limit enforcement
+    extensions: Map<String, (u8, Rc<Box<dyn Extension>>)>,
+    with_functions: BTreeMap<String, FunctionModifier>,
+}
+```
+
+### Context Stack
+
+Each query/rule evaluation pushes a `Context`:
+
+```rust
+struct Context {
+    key_expr: Option<ExprRef>,      // Object comprehension key
+    output_expr: Option<ExprRef>,   // Output value expression
+    value: Value,                   // Accumulated results
+    result: Option<QueryResult>,    // For user queries (bindings + expressions)
+    rule_ref: Option<ExprRef>,      // Reference to current rule
+    rule_value: Value,              // Computed rule value
+    is_compr: bool,                 // Comprehension context
+    is_set: bool,                   // Set rule context
+    is_old_style_set: bool,         // Legacy set syntax
+    early_return: bool,             // Break out of evaluation
+}
+```
+
+Contexts are pushed for: rule bodies, comprehensions, user queries. The
+context determines how results are collected (array, set, object, or query
+result bindings).
+
+### Scope Stack
+
+Variables are tracked in a stack of scopes:
+
+```rust
+type Scope = BTreeMap<SourceStr, Value>;
+```
+
+Each function/rule call pushes a new scope. Variable lookup searches from
+innermost to outermost scope.
+
+## Evaluation Call Hierarchy
+
+```
+eval_rule()                          Entry: evaluate a named rule
+  └─ eval_rule_impl()               Dispatch by rule type (Spec/Default/Func)
+       └─ eval_rule_bodies()         Evaluate rule body alternatives
+            └─ eval_query()          Execute a query (ordered statements)
+                 └─ eval_stmts()     Execute statements in scheduled order
+                      └─ eval_stmt() Single statement dispatch
+                           └─ eval_stmt_impl()
+                                ├─ Expr → eval_expr()
+                                ├─ SomeIn → eval_some_in()
+                                ├─ SomeVars → variable declaration
+                                ├─ NotExpr → negation wrapper
+                                └─ Every → eval_every()
+
+eval_expr()                          Expression dispatcher (25+ variants)
+  ├─ Literals → direct Value
+  ├─ Var/RefDot/RefBrack → eval_chained_ref_dot_or_brack()
+  ├─ BinExpr → eval_bin_expr()
+  ├─ BoolExpr → eval_bool_expr()
+  ├─ ArithExpr → eval_arith_expr()
+  ├─ Call → eval_call()
+  ├─ ArrayCompr/SetCompr/ObjectCompr → eval_*_compr()
+  ├─ Array/Set/Object → eval_array/set/object()
+  └─ AssignExpr → execute_destructuring_plan()
+```
+
+## Rule Evaluation Lifecycle
+
+### 1. Rule Discovery
+
+When code references `data.pkg.rule`, the interpreter calls
+`ensure_rule_evaluated()` which:
+1. Checks if the path has initial data (from `add_data()`)
+2. Looks for rules that define that path in `compiled_policy.rules`
+3. Evaluates those rules if not already in `self.processed`
+
+### 2. Rule Bodies
+
+A rule can have multiple bodies (alternatives). Bodies are evaluated in order.
+**First successful body wins** — remaining bodies are skipped.
+
+```rego
+allow { condition_a }  # Body 1
+allow { condition_b }  # Body 2 — only tried if body 1 fails
+```
+
+### 3. Result Collection
+
+Results are collected into `ctx.value` based on rule type:
+- **Complete rules**: single Value
+- **Partial set rules**: `Value::Set` accumulating members
+- **Partial object rules**: `Value::Object` accumulating key-value pairs
+
+### 4. Data Materialization
+
+`update_rule_value()` navigates the rule's path and inserts the result into
+`self.data`. This is how rules become "virtual documents" accessible via
+`data.pkg.rule`.
+
+**Precedence**: initial data > evaluated rules > default rules.
+
+## Variable Lookup
+
+`lookup_var()` is the main variable resolution function. The search order:
+
+1. Local scopes (innermost to outermost)
+2. `input` document (if name is "input")
+3. `data` document (if name is "data") — triggers lazy rule evaluation
+4. Imported variables from other packages
+5. Returns `Undefined` if not found
+
+**Key subtlety**: Looking up a `data` path may trigger rule evaluation, which
+may trigger further lookups — this is how lazy evaluation chains work.
+
+## The `with` Modifier
+
+`with` temporarily overrides data, input, or functions during evaluation:
+
+```rego
+x = eval { y = f(1) with f as g with data.config as override }
+```
+
+### State Save/Restore Pattern
+
+The interpreter saves 7 fields as a tuple before applying `with`:
+```rust
+(with_document, input, data, processed, processed_paths, with_functions, rule_values)
+```
+
+After applying overrides:
+- `self.processed` is cleared (forces re-evaluation with new context)
+- `self.rule_values` is cleared
+- The expression is evaluated
+- All 7 fields are restored
+
+**Function overrides**:
+- `FunctionModifier::Value(v)` — replace function with constant
+- `FunctionModifier::Function(path)` — replace with another function
+
+## Cycle Detection
+
+The interpreter tracks `active_rules` (a stack of currently-evaluating rules).
+If the same rule appears twice in the stack, a cycle is detected and an error
+is raised with a "depends on" chain for debugging.
+
+## Destructuring Plans
+
+The interpreter executes pre-computed `DestructuringPlan`s for pattern matching
+in assignments and `some...in` bindings:
+
+- `DestructuringPlan::Var` — bind to variable
+- `DestructuringPlan::Ignore` — wildcard `_`
+- `DestructuringPlan::EqualityValue` — match against literal
+- `DestructuringPlan::Array` — destructure array elements
+- `DestructuringPlan::Object` — destructure object fields
+
+Plans are computed at compile time by `src/compiler/destructuring_planner/`.
+
+## Performance-Critical Paths
+
+- **Loop variable caching** (`loop_var_values`): avoids re-evaluating loop
+  expressions on each iteration
+- **Builtin result caching** (`builtins_cache`): memoizes pure builtin calls
+- **Rule processing tracking** (`processed`): prevents redundant evaluation
+- **Execution timer**: cooperative checking with amortized overhead
+
+## Known TODOs in Code
+
+The interpreter has ~15 TODO comments indicating areas of active development:
+- Recursive calls with different values for same expression
+- Type coercion behavior verification
+- With modifier optimization (delay state restore)
+- Variable lookup timing questions
+- Copy optimization for paths
+
+These indicate areas where the code is known to be evolving. Extra care
+is needed when modifying near these comments.
+
+## Connection to RVM
+
+Both the interpreter and RVM:
+- Use the same `BUILTINS` registry
+- Share the `Value` type
+- Use the same `CompiledPolicyData` (schedules, hoisted loops)
+- Produce the same results for the same inputs (semantic equivalence)
+
+When implementing features, they must work in **both** execution paths.

docs/knowledge/language-extension-guide.md

@@ -0,0 +1,199 @@
+<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->
+<!-- Licensed under the MIT License. -->
+
+# Knowledge: Language Extension Guide
+
+How to add new policy languages to regorus. Read this when implementing
+support for a new policy language or modifying the language extension
+architecture.
+
+## Current Architecture
+
+Regorus supports multiple policy languages through `src/languages/`:
+
+```
+src/languages/
+  azure_policy/     JSON-based declarative constraints → RVM bytecode
+  azure_rbac/       Condition expression strings → direct interpretation
+  rego/             Rego source → RVM bytecode (via core compiler)
+```
+
+Each language has its own:
+- **Parser**: language-specific syntax → AST
+- **AST types**: language-specific node types with Span tracking
+- **Compilation or interpretation**: AST → RVM bytecode OR direct evaluation
+- **Feature flag**: compile-time opt-in
+
+### No Shared Trait (Yet)
+
+There is **no common trait** defining language behavior. Each language
+provides its own entry points:
+
+- Azure Policy: `parser::parse_policy_rule()` → `compiler::compile_policy_rule()`
+- Azure RBAC: `parser::parse_condition_expression()` → `ConditionInterpreter::evaluate_str()`
+- Rego: integrated into the core `Engine` via `Lexer → Parser → Interpreter/RVM`
+
+This is an adapter pattern — each language adapts to the shared infrastructure
+in its own way. A formal trait may be introduced as more languages are added.
+
+### Two Execution Strategies
+
+**Strategy 1: Compile to RVM** (Azure Policy, Rego)
+- Parse to language-specific AST
+- Compile to shared `Program` (RVM bytecode)
+- Execute on the shared VM
+- Benefits: shared optimization, serialization, instruction budget enforcement
+
+**Strategy 2: Direct interpretation** (Azure RBAC)
+- Parse to language-specific AST
+- Evaluate directly with a language-specific interpreter
+- Benefits: simpler for expression-oriented languages, no compilation overhead
+
+## Adding a New Language
+
+### Step 1: Feature Flag
+
+```toml
+# Cargo.toml
+[features]
+my_language = ["dep:optional-dep-if-needed"]
+```
+
+### Step 2: Module Structure
+
+```
+src/languages/my_language/
+  mod.rs            Module root, public exports
+  ast/              Language-specific AST types
+    mod.rs          Node types with Span tracking
+  parser/           Language-specific parser
+    mod.rs          Entry point: parse() → AST
+  compiler/         If compiling to RVM (Strategy 1)
+    mod.rs          compile() → Rc<Program>
+  interpreter.rs    If direct interpretation (Strategy 2)
+  builtins/         Language-specific builtin functions (if any)
+```
+
+### Step 3: Register in `src/lib.rs`
+
+```rust
+pub mod languages {
+    #[cfg(feature = "my_language")]
+    pub mod my_language;
+    // ... existing languages
+}
+```
+
+### Step 4: Integration Points
+
+**If compiling to RVM:**
+- Produce a `Program` struct (same as Rego/Azure Policy)
+- Populate metadata with language identifier
+- The shared VM executes the program
+- Benefits from instruction budget, time limits, memory limits
+
+**If direct interpretation:**
+- Implement an interpreter that evaluates against provided context
+- Must enforce resource limits manually (time, memory)
+- Must handle errors consistently with other languages
+
+### Step 5: Engine Integration
+
+Add methods to `Engine` (feature-gated) for loading and evaluating the
+new language:
+
+```rust
+#[cfg(feature = "my_language")]
+pub fn add_my_language_policy(&mut self, source: String) -> Result<()> {
+    let ast = languages::my_language::parser::parse(&source)?;
+    let program = languages::my_language::compiler::compile(&ast)?;
+    // ... integrate with engine
+    Ok(())
+}
+```
+
+## Shared Infrastructure
+
+New languages can reuse:
+
+| Component | Location | What it provides |
+|-----------|----------|-----------------|
+| **Value type** | `src/value.rs` | Shared data representation |
+| **Number type** | `src/number.rs` | High-precision arithmetic |
+| **RVM** | `src/rvm/` | Bytecode execution engine |
+| **Builtins** | `src/builtins/` | Shared builtin functions |
+| **Span** | `src/ast.rs` | Source location tracking |
+| **Limits** | `src/utils/limits/` | Time, memory, execution limits |
+| **Cache** | `src/cache.rs` | LRU caching for compiled patterns |
+| **Engine** | `src/engine.rs` | Policy management, data/input handling |
+
+## Design Considerations for New Languages
+
+### AST Design
+
+- Every node should carry a `Span` for error reporting
+- Use `Ref<T>` (Rc-based) for shared ownership
+- Keep AST types in a dedicated `ast/` module
+
+### Parser Design
+
+- Recursive descent is the standard pattern in regorus
+- Enforce depth limits (default 32) to prevent stack overflow
+- Check memory limits during parsing
+- Track line/column for error messages
+
+### Compilation Design
+
+If targeting the RVM:
+- Allocate registers for intermediate values
+- Use the literal table for constants
+- Define entry points for each evaluatable unit
+- Populate metadata (language name, version, etc.)
+- Run `validate_limits()` on the generated program
+
+### Error Design
+
+- Use `thiserror` for language-specific error types
+- Include source location (Span) in all errors
+- Don't leak sensitive information in error messages
+- Consider error recovery for better diagnostics
+
+### Testing
+
+- Create YAML test cases in `tests/` or language-specific test directory
+- Cover: normal operation, edge cases, error conditions, resource limits
+- Verify against reference implementation if one exists
+
+## Future Directions
+
+### Language Server Protocol (LSP)
+
+The AST and Span infrastructure supports building language servers:
+- **Completion**: AST traversal for scope-aware suggestions
+- **Diagnostics**: Parser/compiler errors with source locations
+- **Go to definition**: Span tracking enables precise navigation
+- **Hover**: AST node identification for type/documentation info
+
+### Linters and Analyzers
+
+The compilation pipeline enables static analysis:
+- **Scheduler output**: dependency analysis for unused variables
+- **Scope analysis**: detect shadowing, unused imports
+- **Type inference**: Value type tracking through expressions
+- **Complexity analysis**: rule depth, statement count, loop nesting
+
+### Partial Evaluation
+
+Not currently implemented but the architecture supports it:
+- The RVM's register-based design could track symbolic values
+- The scheduler's dependency analysis identifies independent subexpressions
+- Compilation could produce partially-evaluated programs with "holes"
+- Design principle: keep evaluation logic pure and side-effect-free
+
+### Causality Tracking
+
+Understanding WHY a policy decision was made:
+- The RVM's instruction-level execution could log decision paths
+- The interpreter's context stack tracks which rules contributed
+- Frame-level tracing in suspendable mode provides execution history
+- Coverage tracking (`coverage` feature) already records evaluated expressions

docs/knowledge/policy-evaluation-security.md

@@ -0,0 +1,199 @@
+<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->
+<!-- Licensed under the MIT License. -->
+
+# Knowledge: Policy Evaluation Security
+
+Deep knowledge about security properties, DoS protection, resource limits,
+and input validation in regorus. Read this before modifying evaluation paths,
+parsers, or resource management.
+
+## Threat Model
+
+Regorus evaluates **untrusted policy code** against **untrusted data**. Both
+may be adversarial. The engine must:
+
+1. **Always terminate** — no infinite loops, no unbounded recursion
+2. **Bound resource usage** — memory, CPU time, instruction count
+3. **Return correct results** — a wrong result is a security vulnerability
+4. **Never crash** — panics in daemon mode crash the service
+5. **Not leak information** — error messages must not expose sensitive data
+
+## Resource Limit Enforcement
+
+### Instruction Budget (RVM)
+
+The primary defense against computation-based DoS:
+
+- **Default**: 25,000 instructions (`src/rvm/vm/machine.rs`)
+- **Enforcement**: checked every iteration in the execution loop
+- **Error**: `VmError::InstructionLimitExceeded`
+- **Configurable**: `set_max_instructions(limit)`
+
+### Execution Time Limits
+
+Wall-clock enforcement via `ExecutionTimer` (`src/utils/limits/time.rs`):
+
+- **Cooperative checking** — the timer is checked periodically, not preemptively
+- **Amortized overhead** — accumulates work units before reading the clock
+  to avoid syscall overhead
+- **Suspended time excluded** — `resume_from_elapsed()` preserves elapsed time
+  across VM suspensions, so only active computation counts
+- **Per-instance override** — each VM can set its own timer config
+- **Error**: `VmError::TimeLimitExceeded`
+
+### Memory Limits
+
+Global memory tracking via `src/utils/limits/memory.rs`:
+
+- **Global atomic limit** — `GLOBAL_MEMORY_LIMIT: AtomicU64`
+- **Throttled checking** — dual strategy to avoid contention:
+  - Stride-based: check every 16 iterations
+  - Delta-based: check when 32 KiB has been allocated since last check
+- **Per-thread flushing** — auto-flush at 1 MiB threshold
+- **Enforcement points**: Value construction, deserialization, parsing
+- **Error**: `VmError::MemoryLimitExceeded`
+
+The `allocator-memory-limits` feature uses mimalloc to enforce at the allocator
+level.
+
+## Input Validation
+
+### Policy Source (`src/lexer.rs`)
+
+Rego source is validated during lexing with configurable limits:
+
+| Limit | Default | Purpose |
+|-------|---------|---------|
+| `max_col` | 1,024 chars | Lines exceeding this are likely minified/attack code |
+| `max_file_bytes` | 1 MiB | Prevents memory exhaustion from huge files |
+| `max_lines` | 20,000 | Prevents excessive parsing time |
+
+Memory limit is also checked after each logical chunk during lexing.
+
+### Parser Depth
+
+The parser enforces expression nesting depth:
+
+- **Default**: `MAX_EXPR_DEPTH = 32` (`src/parser.rs`)
+- Prevents stack overflow from deeply nested expressions like `(((((...)))))`
+- Returns error, not panic
+
+### JSON/YAML Data
+
+Data added via `add_data()` must be an object (checked by `engine.rs`).
+Value construction during deserialization checks memory limits at each node.
+
+### RVM Programs
+
+Compiled programs validated by `validate_limits()` (`src/rvm/program/core.rs`):
+
+| Resource | Limit |
+|----------|-------|
+| Instructions | 65,535 |
+| Literals | 65,535 |
+| Rules | 4,000 |
+| Entry points | 1,000 |
+| Source files | 256 |
+| Builtins | 512 |
+| Path depth | 32 |
+
+These prevent adversarial serialized programs from consuming excessive resources
+during deserialization or execution.
+
+## Recursion Protection
+
+- **Parser**: `MAX_EXPR_DEPTH = 32` for expression nesting
+- **RVM**: `MAX_PATH_DEPTH = 32` for rule path depth
+- **Virtual documents**: `needs_runtime_recursion_check` flag enables detection
+  when `VirtualDataDocumentLookup` instructions are present
+- **Rule evaluation**: processed rules tracked in `self.processed` set to
+  prevent re-evaluation cycles
+
+## DoS via Regular Expressions
+
+Regorus uses the `regex` crate which compiles to a DFA — **no catastrophic
+backtracking**. Protection is layered:
+
+1. DFA-based regex engine (no exponential blowup)
+2. Instruction budget limits total work
+3. Execution time limits bound wall-clock
+4. LRU cache prevents repeated compilation (256 patterns, hard cap 2^16)
+
+## Undefined vs False
+
+**This is a security-critical distinction.** In policy evaluation:
+
+```rego
+allow { input.role == "admin" }
+```
+
+If `input.role` is missing:
+- `input.role == "admin"` → `Undefined` (not `false`)
+- `allow` → `Undefined` (rule body didn't succeed)
+- `not allow` → `true` (because `not Undefined = true`)
+
+A bug that treats `Undefined` as `false` (or vice versa) can change policy
+decisions. Every evaluation path must handle the three-valued logic correctly.
+
+See `docs/knowledge/value-semantics.md` for detailed Undefined propagation rules.
+
+## Supply Chain Security
+
+### Dependency Auditing
+
+The `dependency-audit.yml` workflow runs:
+- **cargo-audit**: checks 6 Cargo.lock files (main + 5 bindings) against
+  RustSec advisories
+- **cargo-deny**: checks 9 manifests for CVEs (advisories) and problematic
+  dependencies (bans)
+- **Schedule**: PRs, main pushes, weekly (Mondays 6 AM), manual dispatch
+
+### Dependency Management
+
+- **Pinned action SHAs**: all GitHub Actions references use full commit SHAs,
+  not mutable tags — prevents supply chain attacks via tag mutation
+- **Locked dependencies**: `Cargo.lock` committed, `cargo fetch --locked` /
+  `--frozen` in CI ensures reproducible builds
+- **Dependabot**: automated weekly updates for Cargo, GitHub Actions, Maven,
+  NuGet, pip, npm, bundler, Go
+- **Minimal dependency surface**: prefer `core`/`alloc` over external crates
+
+### Spectre Mitigation
+
+On Windows (MSVC), the optional `msvc_spectre_libs` dependency links with
+Spectre-mitigated CRT and libraries.
+
+## Panic Safety
+
+The 80+ deny lints in `src/lib.rs` exist not just for style — they prevent
+panics at compile time:
+
+| Denied | Why |
+|--------|-----|
+| `clippy::unwrap_used` | `.unwrap()` panics on `None`/`Err` |
+| `clippy::expect_used` | `.expect()` panics on `None`/`Err` |
+| `clippy::indexing_slicing` | `vec[i]` panics on out-of-bounds |
+| `clippy::arithmetic_side_effects` | `a + b` can overflow and panic |
+| `clippy::panic` | Explicit `panic!()` |
+| `clippy::unreachable` | Explicit `unreachable!()` |
+| `clippy::todo` | Explicit `todo!()` |
+
+In daemon mode, **any panic is a service crash**. The deny lints are the first
+line of defense. The FFI layer's `with_unwind_guard()` is the second — it
+catches panics and poisons the engine (see `docs/knowledge/ffi-boundary.md`).
+
+But panic containment is a last resort. The goal is zero panics in all code
+paths, including error paths, resource exhaustion, and adversarial input.
+
+## Security Review Checklist
+
+When reviewing code for security:
+
+1. **Undefined handling** — does the code correctly distinguish Undefined from false?
+2. **Resource limits** — does new code respect instruction budget, time, memory?
+3. **Input validation** — is untrusted input validated before use?
+4. **Panic paths** — can any code path panic (overflow, indexing, unwrap)?
+5. **Error messages** — do errors avoid leaking policy content or data?
+6. **Recursion** — is recursion bounded?
+7. **Allocation** — can adversarial input cause unbounded allocation?
+8. **Cache behavior** — can cache be poisoned or exhausted?

docs/knowledge/rego-compiler.md

@@ -0,0 +1,286 @@
+<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->
+<!-- Licensed under the MIT License. -->
+
+# Knowledge: Rego Compiler
+
+Deep knowledge about the Rego → RVM bytecode compiler in
+`src/languages/rego/compiler/`. Read this before modifying rule compilation,
+expression codegen, register allocation, or optimization passes.
+
+See also `compilation-pipeline.md` for the scheduler and loop hoisting stages
+that feed into this compiler.
+
+## Module Structure
+
+```
+src/languages/rego/compiler/
+  mod.rs               Compiler struct, scope management, register allocation
+  core.rs              Variable resolution, register helpers, instruction emission
+  program.rs           finish() — default rules, rule info construction, metadata
+  rules.rs             Worklist algorithm, per-definition rule compilation
+  queries.rs           Statement compilation, loop hoisting integration
+  expressions.rs       Expression dispatch, recursive compilation
+  references.rs        Chained reference parsing (obj.a[x].b[y])
+  function_calls.rs    Builtin vs. user-defined function dispatch
+  loops.rs             `every` quantifier, loop mode handling
+  comprehensions.rs    Array/Set/Object comprehension compilation
+  destructuring.rs     Function parameter binding/validation
+  error.rs             Error types with span tracking
+```
+
+## Worklist Algorithm
+
+Rule compilation uses a worklist (depth-first queue) rather than
+recursive descent. This provides three benefits:
+
+1. **Dependency ordering** — rules are compiled in reference order
+2. **Recursion detection** — a call stack tracks in-progress rules
+3. **Deduplication** — already-compiled rules are skipped
+
+```
+while worklist not empty:
+    pop (rule_path, call_stack) from worklist
+    if rule_path in call_stack → compile-time recursion error
+    if rule_path already compiled → skip
+    push rule_path onto call_stack
+    compile all definitions of rule_path
+    mark rule as compiled
+```
+
+When compiling a rule body encounters `CallRule` to another rule, that
+target rule is pushed onto the worklist. This ensures rules are compiled
+in call order.
+
+## Variable Resolution
+
+The compiler resolves variable names through a priority chain
+(`core.rs`):
+
+```
+1. "input"  → emit LoadInput (cached per rule definition)
+2. "data"   → emit LoadData (cached per rule definition)
+3. scope    → use bound register from current scope
+4. fallback → treat as rule call: data.{package}.{name}
+```
+
+**Input/data caching**: `LoadInput` and `LoadData` are emitted at most
+once per rule definition. The cached register is reused for subsequent
+references. The cache is reset between definitions to prevent stale state.
+
+## Register Allocation
+
+### Three-Tier Strategy
+
+**Dispatch window** — initial registers for entry point dispatch and
+temporary work. Sized by `dispatch_window_size`.
+
+**Per-rule window** — max registers within any single rule definition.
+Register 0 is always the result accumulator. The VM allocates a fixed
+frame per rule based on `max_rule_window_size`.
+
+**Per-definition reset** — `register_counter` resets to 0 at each
+definition start. This minimizes frame size and enables tail calls.
+
+### Special Registers
+
+| Register | Purpose |
+|----------|---------|
+| 0 | Rule result accumulator |
+| `current_input_register` | Cached `LoadInput` (per definition) |
+| `current_data_register` | Cached `LoadData` (per definition) |
+| 0..N-1 (functions) | Function parameter bindings |
+
+**Limit**: u8 register counter (max 255). The compiler asserts
+`register_counter < 255`.
+
+## Expression Compilation
+
+Each `Expr` variant maps to one or more RVM instructions:
+
+| Expr | Instructions | Notes |
+|------|-------------|-------|
+| Literal (Num/Str/Bool) | `Load` | Literals go to literal table |
+| `true`/`false`/`null` | `LoadTrue`/`LoadFalse`/`LoadNull` | Special-cased |
+| Var (in scope) | — | Reuse bound register |
+| Var (unresolved) | `CallRule` | Treat as rule reference |
+| RefDot | `IndexLiteral` | Literal key optimization |
+| RefBrack | `Index` or loop | Depends on bound/unbound index |
+| Chained ref | `ChainedIndex` | `obj.a[x].b[y]` → single instruction |
+| ArithExpr | `Add`/`Sub`/`Mul`/`Div`/`Mod` | |
+| BoolExpr | `Eq`/`Ne`/`Lt`/`Le`/`Gt`/`Ge` | |
+| Not | `Not` | |
+| Call (builtin) | `BuiltinCall` | Via builtin_call_params table |
+| Call (user) | `FunctionCall` | Via function_call_params table |
+| ArrayCompr | `ComprehensionBegin..Yield..End` | Mode: Array |
+| SetCompr | `ComprehensionBegin..Yield..End` | Mode: Set |
+| ObjectCompr | `ComprehensionBegin..Yield..End` | Mode: Object |
+| Every | `LoopStart { mode: Every }` | Quantifier loop |
+| SomeIn | `LoopStart` | Iteration with binding |
+| UnaryMinus | `Sub` (0 - x) | |
+
+### Chained References
+
+Multi-level property access like `input.request.headers["content-type"]`
+compiles to a single `ChainedIndex` instruction with parameters:
+
+```rust
+ChainedIndexParams {
+    dest: u8,
+    root: ChainedIndexRoot,     // Var or Expr
+    components: Vec<Component>, // Field(literal_idx) or Expr(register)
+}
+```
+
+This avoids emitting multiple `Index` instructions and intermediate
+registers.
+
+## Rule Type Compilation
+
+### Complete Rules
+
+```rego
+allow := input.admin == true
+```
+
+- Body compiled as normal statements
+- Success: `RuleReturn {}` (stores result in register 0)
+- **Static value optimization**: if all definitions yield the same constant,
+  the rule gets `early_exit_on_first_success = true` — VM stops after
+  first successful definition
+
+### Partial Set Rules
+
+```rego
+ports contains p if { ... }
+```
+
+- Emit `ComprehensionYield { value_reg, key_reg: None }`
+- Result register accumulates a set of all yielded values
+
+### Partial Object Rules
+
+```rego
+people[name] = age if { ... }
+```
+
+- Emit `ComprehensionYield { value_reg, key_reg: Some(k) }`
+- Result register accumulates key-value pairs
+
+### Functions
+
+```rego
+f(x, y) := x + y
+```
+
+- Parameters bound to registers 0..N-1 before body compilation
+- `DestructuringSuccess {}` emitted after parameter validation
+- Consistent parameter count enforced across all definitions
+- After compilation, `FunctionInfo` recorded with param names
+
+## Comprehension Compilation
+
+All comprehensions follow the same pattern:
+
+```
+ComprehensionBegin { mode, collection_reg, body_start, end }
+  [body: hoisted loops → statements → ComprehensionYield]
+ComprehensionEnd {}
+```
+
+Modes: `Array`, `Set`, `Object`. The VM creates the appropriate
+collection type and appends each yielded value.
+
+**Context stack**: the compiler pushes a comprehension context to
+track that yield should go to the comprehension (not the rule).
+
+## Optimization Passes
+
+### Constant Folding
+
+`try_eval_const()` evaluates pure expressions at compile time:
+- Array/Set/Object literals with all-constant elements
+- Index operations on constant collections
+- Result stored in literal table, emitted as `Load`
+
+### Static Value Detection
+
+After compiling all definitions of a complete rule, the compiler checks
+if every definition yields the same static value. If so:
+- `early_exit_on_first_success = true`
+- VM stops after first successful definition body
+- Common pattern: `default allow := false` + `allow := true { ... }`
+
+### Literal Key Optimization
+
+`obj["literal"]` compiles to `IndexLiteral { literal_idx }` instead of
+loading the string into a register and using `Index`. Avoids a register
+allocation and a `Load` instruction.
+
+### Lazy Builtin Indexing
+
+Builtins are assigned indices only when first used during compilation.
+The builtin info table contains only actually-referenced builtins,
+kept in deterministic order (BTreeMap).
+
+## Compile-Time Safety
+
+### Recursion Detection
+
+The worklist's call stack detects compile-time recursion:
+```
+Rule A calls Rule B calls Rule A → error
+```
+This prevents infinite compilation loops for mutually recursive rules.
+
+### Register Overflow
+
+`alloc_register()` asserts `register_counter < 255`. If a rule body
+requires more than 255 registers, compilation fails rather than silently
+wrapping.
+
+## Program Output
+
+The compiler produces `Arc<Program>` containing:
+
+```rust
+struct Program {
+    instructions: Vec<Instruction>,        // Bytecode stream
+    literals: Vec<Value>,                  // Constant value table
+    builtin_info_table: Vec<BuiltinInfo>,  // Referenced builtins
+    rule_infos: Vec<RuleInfo>,             // Rule metadata
+    entry_points: IndexMap<String, usize>, // Rule path → instruction offset
+    instruction_data: InstructionData,     // Extended params tables
+    span_infos: Vec<SpanInfo>,             // Source mapping (1:1 with instructions)
+}
+```
+
+Every instruction has a corresponding `SpanInfo` for source mapping,
+enabling debugging and IDE integration.
+
+## Key Invariants
+
+1. **Register 0 = result** — every rule's result is in register 0
+2. **Input/data cache reset per definition** — prevents stale references
+3. **Worklist ordering** — rules compiled in call-graph order
+4. **Instruction ↔ SpanInfo 1:1** — every instruction has source location
+5. **Literal table is append-only** — indices are stable after emission
+
+## Common Pitfalls
+
+1. **Scope nesting** — comprehensions and `every` push new scopes.
+   Variables bound in inner scopes are not visible in outer scopes.
+
+2. **Hoisted loop coordination** — the compiler must query the hoisting
+   table for each statement to know which loops to emit. Missing a
+   hoisted loop causes incorrect variable binding at runtime.
+
+3. **Multi-definition rules** — each definition resets registers but
+   shares the same `RuleInfo`. The `definitions` array in `RuleInfo`
+   records instruction ranges for each definition.
+
+4. **Function parameter count** — all definitions of a function must
+   have the same number of parameters. The compiler enforces this.
+
+5. **Builtin vs user function** — the compiler must distinguish builtin
+   calls (which use `BuiltinCall` with the builtin registry) from user
+   function calls (which use `FunctionCall` with the rule index).

docs/knowledge/rego-semantics.md

@@ -0,0 +1,230 @@
+<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->
+<!-- Licensed under the MIT License. -->
+
+# Knowledge: Rego Semantics
+
+Deep knowledge about how regorus evaluates Rego policies. Read this before
+modifying `src/interpreter.rs`, `src/scheduler.rs`, `src/compiler/`, or
+any evaluation-related code.
+
+## Evaluation Model
+
+Regorus is a **compile-then-execute** engine. Key passes:
+
+```
+Source → Lexer → Parser → AST → Compiler (scheduling, destructuring, loop hoisting) → Execution
+```
+
+The compiler pre-computes:
+- **Destructuring plans**: how to bind variables from patterns
+- **Schedules**: statement execution order within rule bodies
+- **Loop hoisting**: which iterations can be computed at compile time
+
+Runtime evaluation is then straightforward — no runtime planning.
+
+## Rule Evaluation
+
+### Rule Types
+
+**Complete rules** — produce a single value:
+```rego
+allow = true { input.role == "admin" }
+```
+
+**Partial rules** — can have multiple bodies, first success wins:
+```rego
+allow { input.role == "admin" }
+allow { input.role == "superuser" }
+```
+Bodies are evaluated in order. When one succeeds, remaining bodies are skipped.
+
+**Default rules** — fallback when no rule produces a value:
+```rego
+default allow = false
+```
+Default rules are explicitly skipped during normal rule evaluation. They fire
+only when the path is `Undefined` and no complete rule exists.
+
+**Precedence**: `initial data > evaluated rules > default rules`
+
+### Rule Caching
+
+Evaluated rules are tracked in `self.processed` set to prevent re-evaluation.
+Once a rule has been evaluated for a given context, it won't be re-evaluated
+unless the context changes (e.g., via `with` keyword).
+
+## Unification and Destructuring
+
+Regorus does **NOT use a traditional unification algorithm**. Instead:
+
+1. The **compiler** analyzes patterns and generates `DestructuringPlan`s
+2. At runtime, `execute_destructuring_plan()` matches values against patterns
+3. Returns `true` (match succeeded, variables bound) or `false` (no match)
+
+This is more like pattern matching than Prolog-style unification. There is no
+occurs check, no variable-to-variable binding chains.
+
+## Backtracking
+
+Backtracking in regorus is **limited and explicit** — it only occurs with
+`some...in` expressions:
+
+```rego
+some x in collection
+```
+
+The backtracking mechanism:
+1. Save current scope
+2. Iterate over the collection
+3. For each element, bind variables and evaluate remaining statements
+4. If remaining statements fail, restore scope and try next element
+5. Succeed if any element leads to successful evaluation
+
+**There is no implicit backtracking** in other contexts. Statements in a rule
+body execute sequentially — if one fails, the entire rule body fails (no
+trying alternatives for previous statements).
+
+## Undefined Propagation in Evaluation
+
+### Boolean and Comparison Operations
+
+```
+Undefined <op> anything  →  Undefined
+anything <op> Undefined  →  Undefined
+```
+
+This applies to all binary operations: `==`, `!=`, `<`, `>`, `<=`, `>=`,
+`+`, `-`, `*`, `/`, `%`, `&`, `|`.
+
+### Negation (the subtle case)
+
+```
+not true      →  false
+not false     →  true
+not Undefined →  true
+```
+
+`not Undefined` is `true` because negating "this expression has no value"
+means "the condition is not met" which is truthy. This is correct OPA
+semantics.
+
+### Reference Chains
+
+```rego
+x = input.a.b.c
+```
+
+If `input.a` exists but `input.a.b` doesn't, the entire reference returns
+`Undefined`. The interpreter navigates the path and returns `Undefined` at the
+first missing component.
+
+### Collection Literals
+
+```rego
+arr = [1, x, 3]   # If x is Undefined, arr is Undefined (not [1, 3])
+```
+
+Any `Undefined` element poisons the entire collection literal. This is not
+intuitive but matches OPA semantics.
+
+### Builtin Arguments
+
+```rego
+count(x)   # If x is Undefined, result is Undefined
+```
+
+If any argument to a builtin is `Undefined`, the result is `Undefined`. The
+function is never called.
+
+### Rule Body Statements
+
+When a statement in a rule body evaluates to `Undefined` or `false`, the
+rule body fails. Statements must succeed sequentially:
+
+```rego
+allow {
+    input.role == "admin"    # If Undefined → body fails here
+    input.active == true     # Never reached
+}
+```
+
+## Virtual Documents (Rules as Data)
+
+Rules materialize into the `data` object. When code references `data.pkg.rule`,
+the interpreter:
+
+1. Checks if the path has initial data (from `add_data()`)
+2. If not, looks for rules that define that path
+3. Evaluates those rules (if not already cached)
+4. Returns the result
+
+`ensure_rule_evaluated()` is the trigger — it's called during path navigation
+when a reference might resolve to a rule-defined value.
+
+## The `with` Keyword
+
+`with` temporarily overrides data, input, or functions during evaluation:
+
+```rego
+x = eval { y = f(1) with f as g }
+```
+
+Implementation pattern (save/modify/restore):
+1. Save current state (data, input, processed rules, rule values, with_functions)
+2. Apply overrides — modify `self.with_document` and related state
+3. Clear `self.processed` to allow re-evaluation with new overrides
+4. Evaluate the expression
+5. Restore original state
+
+**Function override types:**
+- `FunctionModifier::Value(v)` — replace function with a constant value
+- `FunctionModifier::Function(path)` — replace function with another function
+
+## Comprehensions
+
+All comprehensions follow the same pattern:
+
+1. Push new context with `output_expr` and collection type
+2. Evaluate the query (generates solutions)
+3. For each solution, evaluate `output_expr` and add to context's collection
+4. Pop context and return accumulated collection
+
+**Array comprehension**: `[expr | query]` → ordered array of expr values
+**Set comprehension**: `{expr | query}` → set of expr values
+**Object comprehension**: `{key: value | query}` → object of key-value pairs
+
+## Scheduling
+
+The scheduler (`src/scheduler.rs`) determines statement execution order within
+rule bodies. This is a **compile-time** optimization that:
+
+1. Analyzes variable dependencies between statements
+2. Orders statements to minimize wasted work
+3. Moves ground-truth checks (constants, type checks) before expensive iterations
+4. Hoists loop-invariant computations
+
+The schedule is pre-computed and stored — the interpreter follows it directly.
+
+## OPA Conformance
+
+Regorus targets faithful OPA semantics. The conformance suite (`tests/opa.rs`)
+runs the official OPA test cases. Key areas where conformance matters:
+
+- **Undefined propagation** — must match OPA exactly
+- **Error messages** — builtin error messages are compared literally
+- **Type coercion** — number handling, string comparison
+- **Rule indexing** — which rules fire for which inputs
+- **Comprehension behavior** — ordering, deduplication
+
+When behavior differs from OPA, it's a bug unless documented as an intentional
+extension (gated behind `rego-extensions` feature).
+
+## Common Pitfalls
+
+1. **Treating Undefined as false** — see value-semantics.md for the full story
+2. **Forgetting `not Undefined = true`** — the most common subtle bug
+3. **Collection literal with Undefined element** — entire collection becomes Undefined
+4. **Rule body short-circuit** — first failing statement stops the body
+5. **Default rule precedence** — defaults only fire when path is truly Undefined
+6. **`with` scope** — overrides only apply to the expression, not siblings
+7. **Virtual document evaluation order** — rules may evaluate lazily

docs/knowledge/rvm-architecture.md

@@ -0,0 +1,200 @@
+<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->
+<!-- Licensed under the MIT License. -->
+
+# Knowledge: RVM Architecture
+
+Deep knowledge about the Rego Virtual Machine. Read this before modifying
+anything in `src/rvm/`. Also see `docs/rvm/architecture.md`,
+`docs/rvm/instruction-set.md`, and `docs/rvm/vm-runtime.md`.
+
+## Overview
+
+The RVM compiles Rego policies to register-based bytecode with fixed-width
+32-bit instructions, then executes them in a virtual machine:
+
+```
+Policy source → Lexer → Parser → AST → Compiler → Program (bytecode) → VM → Value
+```
+
+This is the **strategic execution path** — new optimization and feature work
+focuses on the RVM, not the tree-walking interpreter.
+
+## Directory Structure
+
+```
+src/rvm/
+  instructions/       Instruction definitions (fixed-width 32-bit opcodes)
+  program/
+    core.rs           Program struct — instructions, literals, entry points, rule info
+    serialization/    Binary and JSON format implementations
+    recompile.rs      Recompilation from partial programs
+  vm/
+    machine.rs        RegoVM — registers, stacks, execution state
+    execution.rs      Run-to-completion and suspendable execution loops
+    dispatch.rs       Instruction dispatch
+    loops.rs          Loop iteration (Any, Every, ForEach modes)
+    comprehension.rs  Set/array/object comprehension builders
+    rules.rs          Rule evaluation, caching, call stacks
+    virtual_data.rs   Virtual document lookup and caching
+    state.rs          Register window pooling and state management
+    errors.rs         VmError — strongly typed VM errors
+  tests/              RVM-specific test suites
+```
+
+## Two Execution Modes
+
+### Run-to-Completion
+
+The VM executes instructions sequentially until the program completes or
+errors. No suspension. This is the **fast path** for synchronous policy
+evaluation. Most production use cases.
+
+### Suspendable
+
+The VM can suspend mid-execution and be resumed later:
+
+| Reason | Use case |
+|--------|----------|
+| **HostAwait** | Program needs external data from the host |
+| **Breakpoint** | Debugging support |
+| **SingleStep** | Instruction-by-instruction execution |
+
+The host calls `vm.resume(value)` to continue after suspension. The VM
+preserves its entire execution state across suspend/resume cycles.
+
+**Important:** `SuspendReason` variants that appear in run-to-completion mode
+trigger `VmError::UnsupportedSuspendInRunToCompletion`.
+
+## Frame Stack
+
+The suspendable mode uses an explicit frame stack (`execution_stack`) with
+frame kinds:
+
+| Frame Kind | Purpose |
+|------------|---------|
+| **Main** | Top-level program execution |
+| **Rule** | Rule body evaluation |
+| **Loop** | Collection iteration (Any, Every, ForEach) |
+| **Comprehension** | Set/array/object comprehension building |
+
+Each frame tracks its own:
+- Program counter (PC)
+- Register window (base + count)
+- Saved caller state (for restoration on frame pop)
+
+Frames are pushed on entry and popped on completion. The frame stack is the
+mechanism that makes suspension possible — the entire execution state is
+captured in the stack.
+
+## Register Window Pooling
+
+The VM reuses register vectors to minimize allocation:
+
+- **Pool**: `state.rs` manages a pool of `Vec<Value>` vectors
+- **Window**: Each frame gets a register window (base offset + count)
+- **Reuse**: When a frame pops, its register vector returns to the pool
+- **Predictable**: Allocation pattern is bounded and deterministic
+
+**Invariant:** New VM features MUST participate in register window pooling.
+Do not allocate fresh Vecs for register storage.
+
+## Instruction Budget
+
+The VM enforces a configurable instruction limit to prevent unbounded execution:
+
+- **Default**: 25,000 instructions (`machine.rs`)
+- **Enforcement**: Checked in the execution loop (`execution.rs`)
+- **Configurable**: `set_max_instructions(limit)` allows any `usize` value
+- **Error**: `VmError::InstructionLimitExceeded` when exceeded
+
+This is the primary defense against denial-of-service via crafted policies.
+All new execution paths must respect this budget — do not add loops or
+recursion that bypass the instruction counter.
+
+## Program Serialization
+
+Compiled programs can be serialized for distribution and cached execution.
+
+### Binary Format (Primary)
+
+Compact, fast deserialization. Used for production distribution of pre-compiled
+policies. Implemented via the `postcard` crate.
+
+### JSON Format (Debugging)
+
+Human-readable. Useful for debugging, tooling, and inspection.
+
+### Artifact Structure
+
+The program has two sections:
+
+**Stable section** (always serializable):
+- Source files, entry points, metadata
+- Rule information, builtin references
+- Sufficient to recompile the execution section
+
+**Execution section** (version-sensitive):
+- Instructions, literals, parameter tables
+- May fail to deserialize on format version mismatch
+
+**Recompilation fallback**: If the execution section can't be deserialized
+(e.g., after a regorus version upgrade), it can be recompiled from the stable
+section. This is handled by `recompile.rs`.
+
+### Program Limits
+
+`validate_limits()` in `program/core.rs` enforces hard bounds:
+
+| Resource | Limit |
+|----------|-------|
+| Instructions | 65,535 |
+| Literals | 65,535 |
+| Rules | 4,000 |
+| Entry points | 1,000 |
+| Source files | 256 |
+| Builtins | 512 |
+| Path depth | 32 |
+
+These limits prevent adversarial programs from consuming excessive resources.
+
+## VmError Pattern
+
+The RVM uses strongly typed errors (`src/rvm/vm/errors.rs`):
+
+```rust
+#[derive(Error, Debug, Clone, PartialEq)]
+pub enum VmError {
+    #[error("Execution stopped: exceeded maximum instruction limit of {limit} ...")]
+    InstructionLimitExceeded { limit: usize, executed: usize, pc: usize },
+    // ... 30+ variants
+}
+```
+
+Every error variant includes `pc` (program counter) for debugging. This is the
+reference pattern for strongly typed errors in regorus — new subsystems should
+follow this design.
+
+## Rule Caching
+
+The VM caches rule evaluation results to avoid redundant computation:
+- Rules are identified by index
+- Cache is checked before evaluation
+- Cache size must match rule info count (`VmError::RuleCacheSizeMismatch`)
+
+## Virtual Document Lookup
+
+Virtual documents (rules-as-data) are resolved through `virtual_data.rs`:
+- Paths are navigated through the rule tree
+- Results are cached per-evaluation
+- `needs_runtime_recursion_check` flag enables recursion detection
+
+## Performance Priorities
+
+Optimization focus areas in `src/rvm/vm/`:
+1. **Instruction dispatch** — tight loop, minimal branch overhead
+2. **Register window pooling** — predictable allocation, zero unnecessary allocs
+3. **Rule caching** — avoid redundant evaluation
+4. **Virtual document lookup caching** — avoid redundant path navigation
+5. **Comprehension building** — efficient collection construction
+
+Profile with `benches/` (Criterion) before optimizing.

docs/knowledge/telemetry-and-diagnostics.md

@@ -0,0 +1,193 @@
+<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->
+<!-- Licensed under the MIT License. -->
+
+# Telemetry and Diagnostics
+
+## Overview
+
+regorus evaluates authorization and compliance policies at Azure scale. When a
+policy returns an unexpected result, operators need to understand **why** —
+without reading regorus source code, without reproducing the exact environment,
+and often under time pressure during an incident.
+
+This knowledge file captures the telemetry and diagnostics architecture: what
+exists today, what's planned, and the design principles that guide diagnostic
+features.
+
+## Design Principles
+
+1. **Every decision must be explainable** — "policy X denied request Y because
+   condition Z at policy.rego:42 evaluated to Undefined"
+2. **Errors trace back to policy source** — file, line, column, rule name
+3. **Structured over unstructured** — machine-parseable diagnostics enable tooling
+4. **Zero-cost when off** — diagnostics must not affect evaluation performance
+   when not enabled (compile-time or runtime gating)
+5. **Cloud-scale observability** — span-based tracing that integrates with
+   distributed tracing systems (OpenTelemetry)
+6. **Defense in depth** — no secrets in diagnostics (policy content, input data)
+
+## Current State
+
+### Source Location Tracking (Strong)
+
+Every syntax element carries a `Span` with source file, line, column, and byte
+offset. The `Source::message()` method produces formatted error output:
+
+```
+error: policy.rego:42:5
+  |
+  42 | input.role == "admin"
+  |     ^^^^^^^^^ type mismatch: expected string, got number
+```
+
+This works for **parse and compile errors**. Evaluation errors have partial
+coverage — some carry Span, others lose it during execution.
+
+### Error Types (Comprehensive but Fragmented)
+
+Multiple error hierarchies exist across subsystems:
+
+| Subsystem | Error type | Location tracking |
+|-----------|-----------|-------------------|
+| Lexer/Parser | `Span`-annotated errors | ✅ file:line:col |
+| Rego compiler | `SpannedCompilerError` | ✅ file:line:col |
+| RVM execution | `VmError` (40+ variants) | ⚠️ program counter only |
+| Schema validation | `ValidationError` (20+ variants) | ⚠️ JSON path only |
+| Azure RBAC | `ConditionEvalError` | ⚠️ limited |
+| Interpreter | `anyhow::Error` with context | ⚠️ varies |
+
+**Gap**: RVM errors have a program counter (`pc`) but no reverse mapping to
+policy source location. This is the most critical diagnostic gap — when the VM
+reports `InstructionLimitExceeded at pc=1234`, operators cannot trace back to
+which policy rule was executing.
+
+### Trace Builtin (Exists, Not Exported)
+
+The `trace(msg)` builtin accumulates messages internally via
+`Interpreter::set_traces(bool)`. However:
+- **No public API** to retrieve traces from `Engine`
+- Traces are string-only (not structured)
+- No trace correlation with evaluation steps
+- No RVM equivalent of trace collection
+
+### Print Gathering
+
+`Engine::take_prints()` retrieves accumulated `print()` output. This works
+but is designed for debugging by policy authors, not for operational telemetry.
+
+### Limit Enforcement
+
+Resource limits produce diagnostic VmError variants:
+- `InstructionLimitExceeded { pc, limit }`
+- `MemoryLimitExceeded { usage, limit }`
+- `TimeLimitExceeded { elapsed, limit }`
+
+These include numeric context but not evaluation context (which rule, which
+input).
+
+### Coverage Tracking (Internal Only)
+
+Feature-gated coverage tracking exists in the interpreter but has no public
+API. This could be the foundation for evaluation path diagnostics.
+
+## Planned Capabilities
+
+### Phase 1: Error Traceability (Foundation)
+
+- **PC-to-source mapping**: RVM bytecode instructions should carry source
+  location metadata, enabling reverse mapping from `pc` to policy:line:col
+- **Export trace builtin**: Expose `traces` through the public `Engine` API
+- **Structured errors**: Migrate key errors to structured types with
+  `serde::Serialize` for machine consumption
+- **Evaluation context in limits**: When limits are hit, include the rule name
+  and approximate policy location
+
+### Phase 2: Evaluation Explanation
+
+- **Decision attribution**: "rule `allow` returned true because all conditions
+  in the rule body at policy.rego:15-28 were satisfied"
+- **Undefined explanation**: "rule `allow` was Undefined because `input.role`
+  at policy.rego:18 was not present in the input document"
+- **Causality tracking**: integration with the planned causality system
+  (see `causality-and-partial-eval.md`)
+- **Coverage export**: public API for evaluation path coverage data
+
+### Phase 3: Cloud-Scale Telemetry
+
+- **OpenTelemetry integration**: optional spans for parse, compile, evaluate
+  phases, gated behind a feature flag
+- **Metric hooks**: evaluation count, duration, cache hit rate, rule count —
+  exposed as callbacks or trait implementations
+- **Evaluation replay**: record input + policy + configuration as a
+  deterministic replay bundle for reproduction
+- **Diagnostic verbosity levels**: off / errors-only / summary / detailed / trace
+
+## Review Checklist for Diagnostics
+
+When reviewing code changes, consider:
+
+1. **Error messages**: Do they include source location (file:line:col)?
+   Do they include the rule/function name? Are they actionable without
+   reading regorus source?
+2. **New error paths**: Is the error type structured? Does it carry enough
+   context for diagnosis?
+3. **Evaluation changes**: If this changes what a policy returns, can a user
+   understand why the result changed?
+4. **Resource limits**: When limits trigger, does the error help the operator
+   fix the issue (e.g., "increase instruction limit" or "simplify rule X")?
+5. **RVM changes**: Do new instructions carry source location metadata?
+6. **FFI boundary**: Are errors properly translated for each binding target?
+   Do they preserve diagnostic information across the FFI?
+7. **No secrets**: Error messages must never include policy content or input
+   data values — only paths, types, and structural information.
+
+## Architecture Notes
+
+### Zero-Cost Diagnostics Pattern
+
+Diagnostics should use Rust's zero-cost abstraction patterns:
+
+```rust
+// Feature-gated: zero cost when disabled
+#[cfg(feature = "diagnostics")]
+fn record_evaluation_step(&mut self, rule: &Rule, result: &Value) { ... }
+
+#[cfg(not(feature = "diagnostics"))]
+fn record_evaluation_step(&mut self, _rule: &Rule, _result: &Value) {}
+```
+
+Or runtime-gated with branch prediction hints:
+
+```rust
+if unlikely(self.diagnostics_enabled) {
+    self.record_step(pc, instruction);
+}
+```
+
+### Structured Diagnostic Output
+
+```json
+{
+  "evaluation_id": "uuid",
+  "policy": "rbac.rego",
+  "query": "data.rbac.allow",
+  "result": false,
+  "duration_us": 142,
+  "rules_evaluated": 7,
+  "explanation": [
+    {
+      "rule": "allow",
+      "location": "rbac.rego:15",
+      "result": "undefined",
+      "reason": "input.role not present in input"
+    }
+  ]
+}
+```
+
+### Integration Points
+
+- **Engine API**: `Engine::set_diagnostics(DiagnosticLevel)` + `Engine::take_diagnostics()`
+- **FFI**: `regorusSetDiagnostics()` / `regorusGetDiagnostics()` across all bindings
+- **CLI**: `--diagnostics=detailed` flag for `regorusctl` / evaluation tools
+- **OpenTelemetry**: Optional `tracing` crate integration behind feature flag

docs/knowledge/time-builtins-compat.md

@@ -0,0 +1,155 @@
+<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->
+<!-- Licensed under the MIT License. -->
+
+# Knowledge: Time Builtins Compatibility
+
+Deep knowledge about the time builtin functions, especially the Go
+`time.Parse` compatibility layer. Read this before modifying
+`src/builtins/time/` or any time-related builtins.
+
+## Architecture
+
+```
+src/builtins/
+  time.rs              Main time builtins (303 lines)
+  time/
+    compat.rs          Go time.Parse compatibility layer (1,359 lines)
+    diff.rs            Time difference calculation (83 lines)
+```
+
+`compat.rs` is the single most complex builtin module in the codebase.
+
+## Why Go Compatibility Matters
+
+OPA is written in Go and uses Go's `time.Parse()` function. Go's time parsing
+is fundamentally different from standard approaches:
+
+**Standard (C, Rust, Python)**: format strings with `%Y`, `%m`, `%d` etc.
+
+**Go**: uses a **reference time** as the layout. The reference time is:
+```
+Mon Jan 2 15:04:05 MST 2006
+```
+This specific date/time was chosen because each component is unique:
+- Month: January (1)
+- Day: 2
+- Hour: 15 (3 PM)
+- Minute: 04
+- Second: 05
+- Year: 2006
+- Timezone: MST
+
+OPA test cases use Go layouts, so regorus must parse and format times using
+this same convention to pass conformance tests.
+
+## The compat.rs Module
+
+This is essentially a **Rust port of Go's time parsing logic**. Key functions:
+
+### `parse(layout, value)` → Parsed time
+
+Implements Go's `time.Parse()`:
+1. Scans the layout string for known reference time components
+2. Extracts corresponding values from the input string
+3. Handles timezone parsing, AM/PM, fractional seconds
+4. Returns a Chrono `DateTime` or `NaiveDateTime`
+
+### `format(time, layout)` → Formatted string
+
+Implements Go's `time.Format()`:
+1. Scans the layout string for reference time components
+2. Substitutes actual time values
+3. Handles timezone abbreviation, offset formatting
+
+### `parse_duration(s)` → Duration
+
+Parses Go-style duration strings: `"10h12m45s"`, `"1.5h"`, `"300ms"`.
+Go's duration format is different from ISO 8601.
+
+## Tricky Aspects
+
+### Missing Components
+
+Go's `time.Parse` allows missing year or time components. Chrono is stricter.
+The compatibility layer fills in defaults:
+- Missing year → 0 (or current year depending on context)
+- Missing time → 00:00:00
+- Missing timezone → UTC
+
+### Timezone Parsing
+
+Go has a custom timezone parsing approach that differs from standard timezone
+databases. The compatibility layer handles:
+- Named timezones (MST, EST, PST)
+- Numeric offsets (+0700, -05:00)
+- Legacy formats
+- `parse_legacy_timezone()` for OPA-specific timezone handling
+
+### Fractional Seconds
+
+Go layouts use `.000` for milliseconds, `.000000` for microseconds,
+`.000000000` for nanoseconds. The number of zeros determines precision.
+The parser must count zeros to know the precision.
+
+### Lint Suppressions
+
+`compat.rs` suppresses several lints:
+- `clippy::arithmetic_side_effects` — ported Go code uses arithmetic directly
+- `clippy::unseparated_literal_suffix` — literal style from Go port
+- `clippy::pattern_type_mismatch`
+
+This is intentional — the module is a faithful port and the arithmetic has
+been verified in the original Go implementation.
+
+## Main Time Builtins (`time.rs`)
+
+| Function | Purpose | Complexity |
+|----------|---------|------------|
+| `time.now_ns()` | Current time in nanoseconds | Low |
+| `time.parse_rfc3339_ns()` | Parse RFC 3339 timestamp | Low |
+| `time.parse_ns()` | Parse with Go layout → nanoseconds | High (uses compat.rs) |
+| `time.parse_duration_ns()` | Parse Go duration string | Medium |
+| `time.format()` | Format with Go layout | High (uses compat.rs) |
+| `time.date()` | Extract year/month/day | Medium |
+| `time.clock()` | Extract hour/minute/second | Medium |
+| `time.weekday()` | Day of week string | Low |
+| `time.add_date()` | Date arithmetic | Medium |
+| `time.diff()` | Time difference | Medium |
+
+### Date Arithmetic
+
+`time.add_date()` uses checked arithmetic:
+- `checked_add()` and `checked_sub_months()` for year/month bounds
+- Leap year adjustments
+- Returns `Undefined` on overflow (OPA compatibility)
+
+### Nanosecond Precision
+
+All time functions work with nanosecond timestamps internally.
+`safe_timestamp_nanos()` prevents overflow when converting from seconds
+to nanoseconds.
+
+### Predefined Format Layouts
+
+`layout_with_predefined_formats()` maps OPA layout names to Chrono formats:
+- RFC 3339, RFC 822, RFC 850
+- ANSIC, Unix, Kitchen, Stamp formats
+- These must match OPA's predefined layouts exactly
+
+## OPA Conformance
+
+Time builtins are a rich source of conformance edge cases:
+
+1. **Go layout parsing** must match Go's behavior exactly
+2. **Nanosecond overflow** must return `Undefined`, not error
+3. **Timezone names** must be recognized consistently
+4. **Duration parsing** must handle Go's format (not ISO 8601)
+5. **Date arithmetic** edge cases (Feb 29, month overflow)
+
+## Dependencies
+
+- `chrono` — date/time handling (feature-gated behind `time`)
+- `chrono-tz` — timezone database (feature-gated behind `time`)
+
+Both are optional dependencies. Time builtins are not available in `no_std`
+or `opa-no-std` configurations.

docs/knowledge/tooling-architecture.md

@@ -0,0 +1,222 @@
+<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->
+<!-- Licensed under the MIT License. -->
+
+# Knowledge: Tooling Architecture
+
+How regorus's current architecture supports building language servers, linters,
+analyzers, and other developer tooling. Read this when planning or implementing
+tooling features.
+
+## Foundational Infrastructure
+
+### Span Tracking
+
+Every AST node carries source location information:
+
+```rust
+pub struct Span {
+    pub source: Source,    // File reference (Rc<SourceInternal>)
+    pub line: u32,         // Line number (1-based)
+    pub col: u32,          // Column number (1-based)
+    pub start: u32,        // Byte offset in source
+    pub end: u32,          // End byte offset
+}
+```
+
+This enables precise error reporting, go-to-definition, hover information,
+and diagnostic placement. Every expression, statement, rule, and module
+carries a Span.
+
+### AST Node Types
+
+The AST (`src/ast.rs`) represents the full syntactic structure:
+
+- 25+ `Expr` variants covering all expression types
+- `LiteralStmt` for statements within rule bodies
+- `Rule` with `RuleHead` (Compr, Set, Func) and bodies
+- `Module` with package, imports, and policies
+- `Query` for ordered statement lists
+
+### Expression Indexing
+
+Each node carries indices for O(1) lookup:
+- `Expr.eidx` — unique expression index within module
+- `LiteralStmt.sidx` — statement index within query
+- `Query.qidx` — query index within module
+
+These indices enable efficient mapping between AST nodes and compilation
+artifacts (schedules, hoisted loops, binding plans).
+
+### NodeRef Pattern
+
+AST nodes use `Ref<T>` (Rc-based) with pointer-identity comparison:
+```rust
+type Ref<T> = Rc<T>;
+```
+This enables cheap cloning and sharing of AST subtrees, which is important
+for tooling that needs to maintain multiple views of the AST.
+
+## Language Server Capabilities
+
+### Diagnostics (Errors and Warnings)
+
+**Already available:**
+- Parser errors with Span → precise source location for red squiggles
+- Lexer errors with line/column → tokenization failures
+- Scheduler errors → dependency cycle detection
+- Type errors from builtins → argument type mismatches
+
+**Possible additions:**
+- Unused variable detection (scheduler tracks variable definitions/uses)
+- Unreachable rule detection (via dependency analysis)
+- Shadowing warnings (scope context tracks bindings)
+- Style warnings (naming conventions, rule complexity)
+
+### Completion
+
+**What the AST provides:**
+- Package/import declarations → suggest available packages
+- Variable scope information → suggest in-scope variables
+- Builtin function registry → suggest available builtins
+- Rule paths → suggest available rules from data document
+
+**What the scheduler provides:**
+- Variable dependency analysis → which variables are defined at cursor position
+- Scope boundaries → what's visible in the current context
+
+### Go-to-Definition
+
+**What Span tracking enables:**
+- Every variable reference carries a Span
+- Every rule definition carries a Span
+- Imports link to package declarations
+- Function calls link to function definitions
+
+**Resolution path:**
+1. Find AST node at cursor position (binary search on Span ranges)
+2. Determine node type (variable, function call, import, etc.)
+3. Look up definition in scope (variables), FunctionTable (functions),
+   or module list (imports)
+4. Return definition's Span
+
+### Hover Information
+
+**What the AST provides:**
+- Expression type (from Value type system)
+- Rule documentation (doc comments if added)
+- Builtin function signatures (from BUILTINS registry)
+- Variable origin (which statement defined it)
+
+### Rename/Refactoring
+
+**What expression indexing enables:**
+- Find all references to a variable (scope analysis)
+- Find all call sites for a function (FunctionTable)
+- Find all imports of a package (import analysis)
+
+## Linter Capabilities
+
+### Static Analysis from Scheduler
+
+The scheduler's dependency analysis provides:
+- **Unused variables**: defined but never used
+- **Circular dependencies**: variable cycles within rule bodies
+- **Dead statements**: statements that can never execute (after always-failing stmt)
+
+### Static Analysis from Scope Context
+
+The compiler's scope analysis provides:
+- **Variable shadowing**: same name in nested scope
+- **Unbound variable access**: using a variable before it's defined
+- **Import shadowing**: import overriding a local definition
+
+### Static Analysis from AST
+
+Direct AST inspection can detect:
+- **Rule complexity**: number of statements, nesting depth, comprehension count
+- **Naming conventions**: package names, rule names, variable names
+- **Pattern violations**: using `=` where `:=` is preferred
+- **Deprecated syntax**: v0 patterns that should use v1 syntax
+
+### Type Analysis
+
+While Rego is dynamically typed, partial type inference is possible:
+- Literal types are known at parse time
+- Builtin return types are documented
+- Input/data schema (if provided) constrains types
+- Type conflicts in comparison operations can be detected
+
+## Analyzer Capabilities
+
+### Policy Analysis
+
+- **Entrypoint discovery**: find all rules that can be queried
+- **Data dependency mapping**: which rules depend on which data paths
+- **Input dependency mapping**: which rules depend on which input fields
+- **Cross-module analysis**: how packages interact
+
+### Performance Analysis
+
+- **Instruction count estimation**: from RVM compilation
+- **Loop complexity**: from hoisted loop analysis
+- **Comprehension nesting**: depth of nested comprehensions
+- **Virtual document chains**: how deep rule-as-data chains go
+
+### Security Analysis
+
+- **Undefined propagation paths**: where undefined values could affect decisions
+- **Missing default rules**: rules without fallback values
+- **Unbounded iteration**: loops without explicit bounds
+- **Resource limit coverage**: which evaluation paths enforce limits
+
+## Partial Evaluation (Future)
+
+Partial evaluation reduces a policy given known inputs while leaving unknown
+parts symbolic. This enables:
+
+- **Policy optimization**: pre-evaluate the known parts at compile time
+- **Policy simplification**: show users what a policy "means" for their context
+- **Incremental evaluation**: only re-evaluate changed parts
+
+### Design Considerations
+
+The current architecture supports partial evaluation through:
+- **RVM's register model**: registers could hold symbolic values
+- **Scheduler dependency analysis**: identifies independent subexpressions
+- **Value type**: could be extended with a `Symbolic` variant
+- **Compilation pipeline**: could produce residual programs with "holes"
+
+### Requirements for Implementation
+
+1. **Symbolic Value type**: extend `Value` with symbolic representation
+2. **Partial evaluation pass**: walk AST, evaluate ground subexpressions,
+   leave symbolic subexpressions
+3. **Residual program**: output a simplified policy/program
+4. **Correctness guarantee**: partial evaluation must preserve semantics
+
+## Causality Tracking (Future)
+
+Understanding why a policy produced its result:
+
+### What Exists Today
+
+- **Coverage tracking** (`coverage` feature): records which expressions
+  were evaluated during a query
+- **Tracing** (`eval_query(query, tracing=true)`): captures evaluation steps
+- **RVM frame stack**: in suspendable mode, provides execution history
+- **Active rules stack**: tracks rule evaluation chain
+
+### What's Needed
+
+1. **Decision tree**: which rules contributed to the final result
+2. **Value provenance**: where each value came from (input, data, rule)
+3. **Counterfactual analysis**: "what if this input were different?"
+4. **Human-readable explanations**: translate decision path to English
+
+### Architecture Implications
+
+- Evaluation functions need optional "trace" parameters
+- The Value type may need provenance metadata
+- The RVM could log instruction-level execution traces
+- The interpreter's context stack already tracks rule contributions
+- Memory overhead must be opt-in (not in production fast path)

docs/knowledge/value-semantics.md

@@ -0,0 +1,148 @@
+<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->
+<!-- Licensed under the MIT License. -->
+
+# Knowledge: Value Semantics
+
+Deep knowledge about regorus's `Value` type, `Undefined` propagation, and
+three-valued logic. Read this before modifying `src/value.rs`, `src/number.rs`,
+or any evaluation code.
+
+## The Value Enum
+
+```rust
+pub enum Value {
+    Null,                              // JSON null
+    Bool(bool),                        // JSON boolean
+    Number(Number),                    // u64 | i64 | f64 | BigInt — at least 100-digit precision
+    String(Rc<str>),                   // Shared, cheap to clone
+    Array(Rc<Vec<Value>>),             // Ordered collection
+    Set(Rc<BTreeSet<Value>>),          // Ordered set (no JSON equivalent)
+    Object(Rc<BTreeMap<Value, Value>>),// Keys can be any Value, not just strings
+    Undefined,                         // Absence of value — NOT the same as Null or false
+}
+```
+
+All collection variants use `Rc` (or `Arc` with the `arc` feature). Cloning a
+Value is a refcount bump. Use `Rc::make_mut()` for copy-on-write mutation.
+
+**Implementation note:** Rego does NOT require ordered sets or objects. The
+current use of `BTreeSet` and `BTreeMap` provides deterministic ordering but
+this is an implementation detail, not a semantic requirement. The Value
+representation may change in the future (e.g., to hash-based collections for
+performance). Do not write code that depends on iteration order of Sets or
+Objects — treat them as unordered collections.
+
+## The Number Type
+
+`src/number.rs` represents numbers as one of four internal representations:
+
+| Variant | Range | Use case |
+|---------|-------|----------|
+| `UInt(u64)` | 0 to 2^64-1 | Non-negative integers |
+| `Int(i64)` | -2^63 to 2^63-1 | Negative integers |
+| `Float(f64)` | IEEE 754 | Fractional values |
+| `BigInt(Rc<BigInt>)` | Arbitrary | Overflow from u64/i64 |
+
+**Invariants:**
+- `from_bigint_owned()` normalizes: if a BigInt fits in i64/u64, it stores the
+  smaller representation.
+- Float comparison uses the `Number` type's methods, never raw `==` on f64
+  (denied by `clippy::float_cmp`).
+- `F64_SAFE_INTEGER = 2^53` — beyond this, float loses integer precision.
+- Arithmetic between variants promotes correctly (e.g., UInt + Int → Int or BigInt).
+
+**Never do raw arithmetic on Number internals.** Use the type's methods — they
+handle precision, overflow, and type promotion.
+
+## Undefined: The Critical Concept
+
+**`Undefined` is NOT `false`. `Undefined` is NOT `Null`.** Rego has three-valued
+logic where expressions can be true, false, or undefined (absent).
+
+This is the single richest source of subtle bugs in regorus.
+
+### Propagation Rules
+
+**Boolean and comparison operations** (`src/interpreter.rs:618-676`):
+```
+Undefined <op> anything  →  Undefined
+anything <op> Undefined  →  Undefined
+```
+Both operands must be defined for the operation to produce a result.
+
+**Negation** (`not`):
+```
+not true      →  false
+not false     →  true
+not Undefined →  true    ← THIS IS THE TRAP
+```
+`not Undefined` evaluates to `true` because negating "absence" means "the
+condition wasn't met" which is truthy in Rego. This is correct OPA semantics
+but extremely subtle.
+
+**Reference chains** (`a.b.c`):
+If any intermediate key is missing or Undefined, the entire chain returns
+Undefined. The interpreter navigates the path and returns Undefined at the
+first missing component.
+
+**Collection construction** (Array, Set, Object literals):
+```
+[1, Undefined, 3]  →  Undefined    (entire collection is Undefined!)
+```
+If ANY element in a collection literal is Undefined, the entire collection
+becomes Undefined. This is NOT intuitive — it doesn't skip the undefined
+element, it poisons the whole result.
+
+**Builtin function arguments**:
+```
+builtin(x, Undefined, z)  →  Undefined
+```
+If any argument to a builtin function is Undefined, the result is Undefined.
+The function is never called.
+
+**Rule bodies**:
+When a statement in a rule body evaluates to Undefined, the rule body fails
+(the rule doesn't produce a value for that input). This is Rego's core
+evaluation model — rules are "queries" that succeed or fail.
+
+### Default Rules and Undefined
+
+Default rules only fire when:
+1. No complete rule for the path produced a defined value, AND
+2. The path is Undefined in the data
+
+Precedence: `initial data > evaluated rules > default rules`
+
+### Testing Undefined
+
+Every code path that handles Values must consider:
+1. What if this Value is Undefined?
+2. What if an intermediate value in a chain is Undefined?
+3. What does `not <this expression>` mean when the expression is Undefined?
+4. Does collection construction with an Undefined element behave correctly?
+
+## Value Ordering
+
+Values implement `Ord` with a total order:
+```
+Null < Bool < Number < String < Array < Set < Object < Undefined
+```
+
+Within each variant, natural ordering applies (false < true, numeric order,
+lexicographic for strings, element-wise for collections).
+
+This ordering matters for `Set` and `Object` (which use `BTreeSet`/`BTreeMap`).
+
+## Memory Limits
+
+`Value` construction respects memory limits. The function
+`enforce_limit_anyhow()` is called during deserialization and construction to
+check the global memory limit (see `src/utils/limits/memory.rs`). This prevents
+adversarial JSON payloads from exhausting memory.
+
+## Serialization
+
+- `Set` serializes as JSON array (no JSON equivalent for sets)
+- `Object` keys that aren't strings are serialized as `{"__regorus_key": key, "__regorus_value": value}`
+- `Undefined` should never appear in serialized output (it represents absence)
+- `Number` serialization preserves precision (BigInt as string when needed)