feat: add copilot instructions, workflows, and architecture docs

Co-authored-by: anakrish <35780660+anakrish@users.noreply.github.com>

.github/copilot-code-review-instructions.md

@@ -0,0 +1,155 @@
+<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->
+<!-- Licensed under the MIT License. -->
+
+# Copilot Code Review Instructions for regorus
+
+regorus is a security-critical multi-policy-language evaluation engine used in
+production at Azure scale. Behavioral bugs are security bugs.
+
+## Your Role
+
+You are a thorough, independent reviewer. Use your own judgment to determine
+the best review strategy for each change. Read the diff, understand the intent,
+explore the surrounding code, and consult the knowledge files that are relevant.
+You decide what to focus on, what to investigate deeper, and when the review is
+complete.
+
+Do not follow a rigid checklist. Think freely. The domain knowledge below is
+context to inform your thinking — not a script to execute.
+
+## Severity Categories
+
+Categorize findings so the author can triage effectively:
+
+- 🔴 **Correctness** — wrong result, logic error, behavioral bug
+- 🟠 **Security** — could affect policy evaluation, resource limits, DoS vector
+- 🟡 **Robustness** — panic path, missing error handling, unchecked arithmetic
+- 🔵 **Polish** — code duplication, naming, style, documentation, dead code
+- ⚪ **Nit** — minor style preference (only flag if pattern is inconsistent)
+
+Always flag 🔴 and 🟠 findings. Never dismiss them as minor.
+
+## Multi-Scale Thinking
+
+Good reviews naturally move between scales. Let the change guide you:
+
+- **Line-level** — is this line correct? What if the input is unexpected?
+- **File/concept-level** — does this fit its module? Duplication? Naming?
+  Is the abstraction right? Could this be simpler?
+- **Big picture** — does this affect the evaluation contract? Other subsystems?
+  Bindings? Security posture? Will this surprise a future maintainer?
+
+You decide which scale matters most for each change. A one-line fix in
+`value.rs` may need deep big-picture thinking. A large refactor may mostly
+need file-level polish review.
+
+## Domain Knowledge
+
+This is what makes regorus unique. Internalize this context and let it inform
+your review — but decide for yourself what matters for each specific change.
+
+### Three-Valued Logic and Undefined
+
+regorus uses three-valued logic: `true`, `false`, `Undefined`. This is the
+most common source of subtle bugs.
+
+- `Undefined` is **not** `false` — treating it as false is a bug
+- `not Undefined` evaluates to `true` — correct but surprising
+- Any expression with a potentially-undefined operand needs both-path thinking
+- Default rules exist to handle undefined — consider if one is needed
+
+### Cross-Cutting Impact Vectors
+
+Changes in regorus often have non-obvious ripple effects:
+
+- **9 language bindings** — API changes affect C, C++, C#, Go, Java, Python,
+  Ruby, Rust, and WASM targets. Panic safety is critical at FFI boundaries.
+- **Dual execution paths** — interpreter and RVM must produce identical results
+- **Feature flag matrix** — must compile with `--all-features`,
+  `--no-default-features`, and the `arc` feature (Rc→Arc, RefCell→RwLock)
+- **no_std discipline** — `core::`/`alloc::` by default, `std::` only behind
+  `#[cfg(feature = "std")]`
+
+### Safety Invariants
+
+The codebase enforces these — watch for violations:
+
+- `#![forbid(unsafe_code)]` in core crate (only FFI bindings may use unsafe)
+- 80+ deny lints — `#[allow(...)]` additions need strong justification
+- No `.unwrap()` / `.expect()` / unchecked indexing in library code
+- No unchecked arithmetic — use `checked_add()`, `saturating_mul()`, etc.
+- RVM instruction budget (default 25,000) bounds computation
+- Error handling: `thiserror` in new code, `anyhow` acceptable in existing modules
+
+### Security Awareness
+
+regorus evaluates policy at scale — think adversarially:
+
+- Can an adversarial policy or input cause unbounded computation/memory/recursion?
+- Does this trust external input without validation?
+- Does a dependency change expand the attack surface?
+- Could a behavioral change flip a policy decision in production?
+
+### Telemetry and Diagnostics
+
+regorus aims for cloud-scale debuggability. Consider:
+
+- **Error traceability**: do error messages include source location (file:line:col)?
+  Can an operator trace an error back to the policy rule that caused it?
+- **Structured errors**: are new errors machine-parseable? Do they carry enough
+  context for diagnosis without reading source code?
+- **Diagnostic preservation**: does this change preserve or improve the diagnostic
+  information available to users? Watch for error conversions that lose context.
+- **No secrets in errors**: error messages must never include policy content or
+  input data values — only paths, types, and structural information.
+
+Consult: `telemetry-and-diagnostics.md`
+
+## Polish and Code Quality
+
+Good reviews catch more than bugs. Look for opportunities to improve:
+
+- **Code duplication** — similar logic that should be unified
+- **Naming** — variables that describe how, not what; overly generic type names
+- **Dead code** — commented-out code, unused imports, unjustified `#[allow(dead_code)]`
+- **Missing documentation** — public functions without doc comments, complex
+  algorithms without "why" comments
+- **Simplification** — could this be expressed more clearly or concisely?
+
+## Deep Reference: Knowledge Files
+
+When you need deeper understanding of a subsystem, read the relevant knowledge
+file from `docs/knowledge/`. These contain institutional knowledge that is not
+obvious from the code alone.
+
+| File | Domain |
+|------|--------|
+| `value-semantics.md` | Value types, Undefined propagation, three-valued logic |
+| `rvm-architecture.md` | VM execution modes, frame stack, serialization |
+| `rego-compiler.md` | Rego compilation, worklist algorithm, register allocation |
+| `compilation-pipeline.md` | Scheduler, loop hoisting, destructuring planner |
+| `builtin-system.md` | Builtin registration, feature gating, OPA conformance |
+| `ffi-boundary.md` | Handle pattern, panic containment, 9 binding targets |
+| `feature-composition.md` | Feature flag interactions, no_std boundary |
+| `error-handling-migration.md` | anyhow → thiserror strategy, VmError pattern |
+| `policy-evaluation-security.md` | DoS protection, resource limits, supply chain |
+| `rego-semantics.md` | Evaluation model, backtracking, `with` modifier |
+| `interpreter-architecture.md` | Context stack, scope management, rule lifecycle |
+| `azure-policy-language.md` | Azure Policy evaluation, effects, conditions |
+| `azure-policy-aliases.md` | Alias registry, ARM normalization pipeline |
+| `azure-rbac-language.md` | RBAC condition interpreter, ABAC builtins |
+| `engine-api.md` | Public API surface, add_policy → compile → eval flow |
+| `time-builtins-compat.md` | Go time.Parse compatibility, timezone handling |
+| `language-extension-guide.md` | Adding new policy languages, extensibility |
+| `tooling-architecture.md` | Language server, linter, analyzer patterns |
+| `causality-and-partial-eval.md` | Causality tracking, partial evaluation design |
+
+You decide which files are relevant. Not every review needs every file.
+
+## Review Iteration
+
+Thorough review is iterative. After findings are addressed, review again.
+Each pass catches things the previous one missed. Keep going until no
+significant (🔴🟠🟡) findings remain.
+
+A change is ready when you would trust it in production at scale.

.github/copilot-instructions.md

@@ -0,0 +1,135 @@
+<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->
+<!-- Licensed under the MIT License. -->
+
+# Regorus — Copilot Instructions
+
+> If these instructions conflict with the actual codebase, the code is the
+> source of truth. Flag any discrepancy you notice.
+
+## Identity
+
+Regorus is a **multi-policy-language evaluation engine** written in Rust. Its
+primary language is [Rego](https://www.openpolicyagent.org/docs/latest/policy-language/)
+(Open Policy Agent), with extensible support for additional policy languages via
+`src/languages/`. It is used in **production at scale** where **correctness is
+security-critical** — a bug in policy evaluation can mean `allow` when the
+answer should be `deny`.
+
+**Key properties:**
+- 9 language targets: Rust, C, C (no_std), C++, C#, Go, Java, Python, Ruby, WASM
+- `#![no_std]` by default (`extern crate alloc`), `#![forbid(unsafe_code)]`
+- Two execution paths: tree-walking interpreter and **RVM** (bytecode VM)
+- 80+ deny lints in `src/lib.rs` — no panics, no unchecked indexing, no unchecked arithmetic
+
+**Strategic direction:**
+- **RVM is the strategic execution path** — new optimization work focuses there
+- **Isolated / daemon execution** — long-lived process, clean resource lifecycle
+- **Error migration** — `anyhow` → `thiserror` strongly typed errors (RVM leads)
+- **Formal verification** — Miri (active CI), Z3 and Verus (planned)
+- **Multi-policy-language** — extensible via `src/languages/`, don't disclose specifics
+
+## Deep Knowledge
+
+For complex subsystems, read the knowledge files in `docs/knowledge/` before
+making changes. These capture invariants, edge cases, and institutional
+knowledge that isn't obvious from the code alone:
+
+| File | Covers |
+|------|--------|
+| `value-semantics.md` | Value type, Undefined propagation, three-valued logic |
+| `rvm-architecture.md` | VM execution modes, frame stack, serialization, register pooling |
+| `builtin-system.md` | Builtin registration, feature gating, OPA conformance |
+| `ffi-boundary.md` | Safety across 9 bindings, handles, panic containment, poisoning |
+| `feature-composition.md` | Feature flag interactions, no_std boundary, testing matrix |
+| `error-handling-migration.md` | anyhow → thiserror migration strategy, VmError pattern |
+| `policy-evaluation-security.md` | DoS protection, resource limits, input validation |
+| `rego-semantics.md` | Evaluation model, undefined propagation, backtracking, `with` |
+| `interpreter-architecture.md` | Context stack, scope management, rule lifecycle |
+| `compilation-pipeline.md` | Scheduler, loop hoisting, destructuring planner |
+| `azure-policy-language.md` | Azure Policy evaluation model, effects, alias normalization |
+| `azure-rbac-language.md` | RBAC condition interpreter, ABAC builtins, context model |
+| `engine-api.md` | Public API surface, add_policy → compile → eval flow |
+| `time-builtins-compat.md` | Go time.Parse compatibility, timezone handling |
+| `language-extension-guide.md` | Adding new policy languages, LSP/tooling vision |
+| `tooling-architecture.md` | Language server, linter, analyzer design patterns |
+| `causality-and-partial-eval.md` | Causality tracking and partial evaluation design |
+| `rego-compiler.md` | Worklist algorithm, expression codegen, register allocation |
+| `azure-policy-aliases.md` | Alias registry, ARM normalization/denormalization pipeline |
+| `telemetry-and-diagnostics.md` | Error traceability, structured diagnostics, cloud-scale telemetry |
+
+Also see `docs/rvm/architecture.md`, `docs/rvm/instruction-set.md`,
+`docs/rvm/vm-runtime.md` for RVM internals.
+
+## Essential Coding Rules
+
+**No panics — ever** (deny lints enforce this):
+```rust
+// Use typed errors for new code
+let v = map.get("key").ok_or(MyError::MissingKey("key"))?;
+// Or anyhow in existing modules
+let v = map.get("key").ok_or_else(|| anyhow!("missing key"))?;
+```
+
+**No unchecked indexing** — use `.get()` + `?` or iterate.
+
+**No unchecked arithmetic** — use `checked_add()`, `saturating_add()`, etc.
+
+**no_std discipline** — `use core::` and `alloc::` by default. Only `std::`
+behind `#[cfg(feature = "std")]`.
+
+**Unsafe forbidden** — `#![forbid(unsafe_code)]` in the core crate. Only FFI
+binding crates may use unsafe.
+
+**Error handling** — new modules: `thiserror` enums (see `src/rvm/vm/errors.rs`).
+Existing modules: `anyhow` is acceptable for consistency within the module.
+
+**Feature gating** — gate modules, registrations, and public API. Add `docsrs`
+annotation. Verify non-default combinations compile.
+
+## Build & Test
+
+```bash
+cargo xtask ci-debug          # Full debug CI suite
+cargo xtask ci-release        # Full release CI suite (superset)
+cargo xtask test-all-bindings # All 9 language binding smoke tests
+cargo xtask test-no-std       # Verify no_std builds (thumbv7m-none-eabi)
+cargo xtask fmt               # Format workspace + bindings
+cargo xtask clippy            # Lint workspace + bindings
+cargo test --test opa         # OPA conformance (needs opa-testutil feature)
+```
+
+Git hooks auto-installed by `build.rs`: pre-commit (build+format+clippy),
+pre-push (+ doc tests + no_std + OPA conformance).
+
+## Repository Layout
+
+```
+src/                    Core library (no_std, forbid(unsafe_code))
+  rvm/                  Rego Virtual Machine ← strategic focus
+  languages/            Policy language extensions
+  builtins/             Builtin functions (~19 modules)
+  value.rs              Value type (Null, Bool, Number, String, Array, Set, Object, Undefined)
+  interpreter.rs        Tree-walking interpreter (legacy path)
+  engine.rs             Public API
+bindings/               9 language targets (ffi/, c/, cpp/, csharp/, go/, java/, python/, ruby/, wasm/)
+tests/                  Integration, conformance, domain-specific tests
+docs/                   Grammar, builtins, RVM docs, knowledge base
+xtask/                  Development automation CLI
+benches/                Criterion benchmarks
+```
+
+## Supply Chain Security
+
+- `dependency-audit.yml` — cargo-audit + cargo-deny across all Cargo.lock files
+- Dependabot — weekly updates for Cargo, Actions, Maven, NuGet, pip, npm, bundler, Go
+- All GitHub Actions references use pinned commit SHAs, not mutable tags
+- `cargo fetch --locked` / `--frozen` in CI for reproducible builds
+
+## When Making Changes
+
+1. **Read relevant knowledge files** in `docs/knowledge/` first
+2. **Consider all 9 binding targets** — API changes affect every language
+3. **Both execution paths** — features must work in interpreter AND RVM
+4. **Test Undefined propagation** — `Undefined ≠ false`, test both paths
+5. **Run `cargo xtask ci-debug`** before submitting
+6. **Update docs** — `docs/builtins.md`, `docs/rvm/`, knowledge files as needed

.github/workflows/copilot-config-validation.yml

@@ -0,0 +1,145 @@
+# Copyright (c) Microsoft Corporation. All rights reserved.
+#
+# Validates that Copilot configuration files stay in sync with the codebase.
+# Runs on changes to Copilot config or docs/knowledge/, and weekly to catch drift.
+
+name: Copilot Config Validation
+
+on:
+  pull_request:
+    paths:
+      - '.github/copilot-instructions.md'
+      - '.github/copilot-code-review-instructions.md'
+      - '.github/skills/**'
+      - '.github/workflows/copilot-setup-steps.yml'
+      - 'docs/knowledge/**'
+  push:
+    branches: ["main"]
+    paths:
+      - '.github/copilot-instructions.md'
+      - '.github/copilot-code-review-instructions.md'
+      - '.github/skills/**'
+      - '.github/workflows/copilot-setup-steps.yml'
+      - 'docs/knowledge/**'
+  schedule:
+    # Weekly on Monday at 7:00 AM UTC — catch drift from codebase changes
+    - cron: "0 7 * * 1"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  validate-copilot-config:
+    name: Validate Copilot Configuration
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+
+      - name: Validate YAML syntax
+        run: |
+          echo "Checking copilot-setup-steps.yml..."
+          python3 -c "
+          import yaml, sys
+          with open('.github/workflows/copilot-setup-steps.yml') as f:
+              yaml.safe_load(f)
+          print('  ✓ Valid YAML')
+          "
+
+      - name: Validate knowledge file references
+        run: |
+          echo "Checking that all knowledge files referenced in instructions exist..."
+          # Extract knowledge file references from instructions
+          grep -oP '[a-z-]+\.md' .github/copilot-instructions.md | sort -u > /tmp/referenced.txt
+
+          # List actual knowledge files
+          ls docs/knowledge/*.md 2>/dev/null | xargs -I{} basename {} | sort -u > /tmp/actual.txt
+
+          # Check for references to non-existent files
+          missing=$(comm -23 /tmp/referenced.txt /tmp/actual.txt || true)
+          if [ -n "$missing" ]; then
+            echo "❌ Instructions reference non-existent knowledge files:"
+            echo "$missing"
+            exit 1
+          fi
+          echo "  ✓ All referenced knowledge files exist"
+
+          # Check for knowledge files not referenced in instructions
+          unreferenced=$(comm -13 /tmp/referenced.txt /tmp/actual.txt || true)
+          if [ -n "$unreferenced" ]; then
+            echo "⚠ Knowledge files not referenced in instructions (may be intentional):"
+            echo "$unreferenced"
+          fi
+
+      - name: Validate skill files
+        run: |
+          echo "Checking skill SKILL.md files..."
+          errors=0
+          for skill_dir in .github/skills/*/; do
+            skill_name=$(basename "$skill_dir")
+            skill_file="$skill_dir/SKILL.md"
+
+            if [ ! -f "$skill_file" ]; then
+              echo "❌ $skill_dir missing SKILL.md"
+              errors=$((errors + 1))
+              continue
+            fi
+
+            # Check frontmatter has required fields
+            if ! head -20 "$skill_file" | grep -q "^name:"; then
+              echo "❌ $skill_file missing 'name' in frontmatter"
+              errors=$((errors + 1))
+            fi
+            if ! head -20 "$skill_file" | grep -q "^description:"; then
+              echo "❌ $skill_file missing 'description' in frontmatter"
+              errors=$((errors + 1))
+            fi
+
+            echo "  ✓ $skill_name"
+          done
+
+          if [ $errors -gt 0 ]; then
+            echo "❌ $errors skill validation error(s)"
+            exit 1
+          fi
+          echo "  ✓ All skills valid"
+
+      - name: Check knowledge file freshness indicators
+        run: |
+          echo "Checking for potential staleness..."
+          warnings=0
+
+          # Check if key source files changed more recently than their knowledge files
+          check_freshness() {
+            knowledge_file="$1"
+            shift
+            for src in "$@"; do
+              if [ -f "$src" ] && [ -f "$knowledge_file" ]; then
+                src_commit=$(git log -1 --format=%ct -- "$src" 2>/dev/null || echo 0)
+                doc_commit=$(git log -1 --format=%ct -- "$knowledge_file" 2>/dev/null || echo 0)
+                if [ "$src_commit" -gt "$doc_commit" ] 2>/dev/null; then
+                  echo "⚠ $knowledge_file may be stale — $src changed more recently"
+                  warnings=$((warnings + 1))
+                fi
+              fi
+            done
+          }
+
+          check_freshness docs/knowledge/value-semantics.md src/value.rs
+          check_freshness docs/knowledge/rvm-architecture.md src/rvm/vm/mod.rs
+          check_freshness docs/knowledge/builtin-system.md src/builtins/mod.rs
+          check_freshness docs/knowledge/ffi-boundary.md bindings/ffi/src/lib.rs
+          check_freshness docs/knowledge/engine-api.md src/engine.rs
+          check_freshness docs/knowledge/interpreter-architecture.md src/interpreter.rs
+          check_freshness docs/knowledge/rego-compiler.md src/languages/rego/compiler/mod.rs
+          check_freshness docs/knowledge/compilation-pipeline.md src/scheduler.rs
+
+          if [ $warnings -gt 0 ]; then
+            echo ""
+            echo "⚠ $warnings knowledge file(s) may need updating"
+            echo "  This is informational — not a build failure"
+          else
+            echo "  ✓ No obvious staleness detected"
+          fi

.github/workflows/copilot-setup-steps.yml

@@ -0,0 +1,38 @@
+# Copyright (c) Microsoft Corporation. All rights reserved.
+#
+name: "Copilot Setup Steps"
+
+# Automatically run the setup steps when they are changed to allow for easy
+# validation, and allow manual testing through the repository's "Actions" tab.
+on:
+    workflow_dispatch:
+    push:
+        paths:
+            - .github/workflows/copilot-setup-steps.yml
+    pull_request:
+        paths:
+            - .github/workflows/copilot-setup-steps.yml
+
+jobs:
+    # The job MUST be called `copilot-setup-steps` or it will not be picked up
+    # by Copilot.
+    copilot-setup-steps:
+        runs-on: ubuntu-latest
+
+        permissions:
+            contents: read
+
+        steps:
+            - name: Checkout code
+              uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+
+            - name: Setup Rust toolchain
+              uses: ./.github/actions/toolchains/rust
+
+            - name: Cache cargo
+              uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
+              with:
+                  shared-key: ${{ runner.os }}-regorus
+
+            - name: Fetch dependencies
+              run: cargo fetch --locked