Commit Graph

32 Commits

Author SHA1 Message Date
Anand Krishnamoorthi
49bd3c22f3 feat!: add Rego Virtual Machine (RVM) implementation (#495)
* feat!: add Rego Virtual Machine (RVM) implementation

This commit introduces a register-based virtual machine for executing Rego
policies with bytecode-style instructions. Unlike the existing tree-walking
interpreter, the RVM compiles policies into instruction sequences that operate
on virtual registers, offering better performance and optimization potential.

Core Components:

Instruction Set Architecture:
- Define instruction types for data operations, control flow, and builtins
- Implement instruction parameter encoding and display formatting
- Add instruction parser with comprehensive test coverage

Virtual Machine Engine:
- Register-based execution model with program counter management
- Loop execution supporting iterators, comprehensions, and quantifiers
- Function call handling with argument evaluation and context management
- Rule evaluation with default value resolution and virtual data support
- Arithmetic and comparison operation implementations

Program Representation:
- Program listing builder with instruction sequencing
- Rule tree construction for organizing policy rules
- Binary and JSON serialization for compiled programs
- Recompilation support for program modification

Testing Infrastructure:
- Extensive YAML test suites covering all VM features
- Rust unit tests for VM execution and instruction parsing
- Test suites for loops, comprehensions, builtins, and control flow

BREAKING CHANGE: Introduces new VM execution path alongside interpreter

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>

* docs: add detailed RVM architecture references

Introduce architecture.md explaining program artifacts, serialization, and runtime subsystems.
Document the full opcode catalog in instruction-set.md, including operands, parameter tables, and outcomes.
Walk through execution flow, stacks, and operational guidance in vm-runtime.md, tying the runtime to the new architecture docs.

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>

---------

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2025-11-14 11:43:19 -06:00
Anand Krishnamoorthi
6dc505c88b build: Add xtask automation for binding version management (#491)
* build: Add xtask automation for binding version management

Introduces a dedicated xtask crate that keeps language binding versions
in sync with the core regorus crate, following the workflow pattern used
by rust-analyzer, gitoxide, and ripgrep.

Key features:
- Git-based change detection: compares binding source files against a
  base ref (merge-base with origin/main by default) plus unstaged/
  untracked files to identify which bindings have been modified
- SemVer-aware bumping: binding edits trigger a minor version increment
  (e.g. 0.5.1 → 0.6.0) under pre-1.0 semantics, signaling potential
  breaking changes; clean bindings simply align to the root version
- Multi-language support: updates Cargo manifests (Rust FFI, Java,
  Python, WASM, Ruby), Maven pom.xml (Java), Ruby version constants,
  and C# project files in a single pass
- CI integration: --check mode fails fast when manifests are out of
  sync, ensuring pre-commit and release-plz workflows catch stale
  versions before merge

Integration points:
- release-plz.toml: runs cargo xtask bindings --base-ref origin/main
  after bumping the root crate, so binding versions are updated
  atomically during the release process
- scripts/pre-commit: invokes cargo xtask bindings --check to block
  commits that would leave bindings out of sync
- .cargo/config.toml: defines cargo xtask alias for convenience

Documentation includes inline examples showing how version bumps behave
when bindings are ahead/behind the root, and notes that the minor
field acts as the major version under SemVer 0.y.z initial development
phase.

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>

* build: refresh xtask tooling, workflows, and locks

- cargo xtask bindings: keep the binding version-sync pipeline intact
- cargo xtask update-deps: new helper to regenerate workspace/binding Cargo.lock files
- workflows: auto-detect the Java jar version in CI and temporarily disable the Ruby workflow
- lock files: refresh root + binding snapshots after the dependency sweep

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>

---------

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2025-10-28 16:10:54 -05:00
Tyler Schade
1b0c2d4072 feat: Implement net.cidr_contains builtin (#471)
Major changes:
- Implement the `net.cidr_contains` builtin
- Enable the v0 and v1 test for `net.cidr_contains`
- Add the `netip` crate to standardize CIDR searching and other
  operations

Key Concept:
- Allow users to leverage the `net.cidr_contains` builtin to check
  whether an IPv4 or IPv6 CIDR contains a specified IP address or
  subnet.

Testing:
- All tests passing.

Signed-off-by: tjons <tylerschade99@gmail.com>
2025-09-05 15:21:41 -05:00
Anand Krishnamoorthi
2a0b4ae6b5 feat! Mimalloc as the default allocator (#434)
This change integrates mimalloc as the default memory allocator for Regorus,
delivering significant performance improvements across all evaluation modes
and language bindings.

Technical Implementation:
- Build mimalloc in vendored mode from C sources (following QSharp approach)
- Implement GlobalAlloc trait for seamless Rust integration
- Add optional 'mimalloc' feature flag for conditional compilation
- Add comprehensive ACI benchmarks to measure evaluation performance

Performance Impact:

Rust Engine Evaluation:
- Single-threaded: ~29% improvement (423 vs 328 Kelem/s)
- Multi-threaded: Better scaling with reduced thread contention
- Fresh engines: ~24% improvement (56 vs 45 Kelem/s)

Rust Compiled Policy Evaluation:
- Single-threaded: ~41% improvement (426 vs 303 Kelem/s)
- Multi-threaded: Improved allocation efficiency under contention
- Fresh compilation: ~26% improvement (53 vs 42 Kelem/s)

C# FFI Bindings:
- Engine evaluation: ~27% improvement (279 vs 219 Kelem/s)
- Compiled policies: ~29% improvement (273 vs 211 Kelem/s)
- Better threading characteristics through improved underlying allocation

Key Benefits:
- Reduced allocation-related contention in multi-threaded scenarios
- More consistent performance across different thread counts
- Improved memory allocation efficiency for both native Rust and FFI workloads
- Better scaling characteristics for production deployments

The mimalloc integration provides substantial performance gains while
maintaining full compatibility with existing code through feature flags.

Reference: QSharp allocator implementation
(https://github.com/microsoft/qsharp/tree/main/source/allocator)

Fixes #297

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2025-08-25 15:01:38 -05:00
Anand Krishnamoorthi
d561531613 feat: add multi-threaded evaluation benchmark suite with comprehensive C# implementation (#457)
This commit introduces a complete multi-threaded evaluation benchmark suite for both Rust and C# implementations of Regorus.

- Implemented engine evaluation benchmark with input and engine cloning strategies
- Implemented compiled policy evaluation benchmark with input cloning and shared compiled policy strategies.

- Created EngineEvaluationBenchmark.cs and CompiledPolicyEvaluationBenchmark.cs with time-based execution (3s warmup + 3s evaluation)
- Implemented configuration options matching Rust implementation (useClonedEngines, useSharedPolicies parameters)

- Created markdown analysis documentation with cross-platform performance analysis
- C# seems to achieve 58-89% of Rust performance on test machine.

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2025-08-22 11:40:39 -05:00
Anand Krishnamoorthi
cc917ea75d feat: Complete target system with C# bindings and resource inference (#458)
* feat: Add Schema Registry and Validation Framework

This commit introduces a comprehensive schema registry and validation framework, providing schema-based validation of resources and policy effects.

- Thread-safe, in-memory registry for schema storage and management
- Global registry patterns for effects and resources
- Concurrent access with proper error handling
- Unicode schema names support

- JSON Schema-compliant validation for all primitive types
- Advanced constraint validation (patterns, ranges, length limits)
- Discriminated union support with anyOf schemas
- Detailed error reporting with nested validation paths
- Discriminated subobject validation for polymorphic schemas

- **Registry Tests**: All registry operations
- **Effect Tests**: Policy effect validation
- **Resource Tests**: Resource validation
- **Validation Tests**: Core validation engine
- Thread-safety, error handling, integration scenarios, edge cases

- **Dependencies**: dashmap, once_cell, regex
- **Thread Safety**: Minimal locking with Rc<Schema> sharing
- **Error Types**: TypeMismatch, OutOfRange, PatternMismatch, etc.

- Complete schema registry and validation subsystem
- Comprehensive test coverage
- Foundation for policy validation in Regorus

Benchmarks:

- Criterion benchmarks for basic types, effects and Azure resources
- Performance range: 3.22ns (string) to 34.74µs (Azure VM resource schema validation)
- String withs patterns validation: 30.2µs. Need to explore whether regex caching helps
  bring this down.
- Azure policy effects: 188ns-1.4µs

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>

* feat: Complete target system with C# bindings and resource inference

- Add comprehensive target system with TargetRegistry and target-aware compilation
- Implement resource type inference from policy equality expressions
- Create modular C# bindings with separate wrapper classes for each concept
- Add thread-safe CompiledPolicy with reference counting for safe disposal
- Enhance FFI with detailed error propagation and target functionality
- Create TargetExampleApp demonstrating Azure Policy integration
- Add CI/CD pipeline testing for all C# applications
- Support target definitions with schema validation and resource selectors
- Implement PolicyModule struct and target-aware compilation methods
- Add comprehensive test coverage for target functionality

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>

---------

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2025-08-19 20:23:43 -05:00
Anand Krishnamoorthi
db718654b5 feat: Add Schema Registry and Validation Framework (#456)
* feat: Add Schema Registry and Validation Framework

This commit introduces a comprehensive schema registry and validation framework, providing schema-based validation of resources and policy effects.

- Thread-safe, in-memory registry for schema storage and management
- Global registry patterns for effects and resources
- Concurrent access with proper error handling
- Unicode schema names support

- JSON Schema-compliant validation for all primitive types
- Advanced constraint validation (patterns, ranges, length limits)
- Discriminated union support with anyOf schemas
- Detailed error reporting with nested validation paths
- Discriminated subobject validation for polymorphic schemas

- **Registry Tests**: All registry operations
- **Effect Tests**: Policy effect validation
- **Resource Tests**: Resource validation
- **Validation Tests**: Core validation engine
- Thread-safety, error handling, integration scenarios, edge cases

- **Dependencies**: dashmap, once_cell, regex
- **Thread Safety**: Minimal locking with Rc<Schema> sharing
- **Error Types**: TypeMismatch, OutOfRange, PatternMismatch, etc.

- Complete schema registry and validation subsystem
- Comprehensive test coverage
- Foundation for policy validation in Regorus

Benchmarks:

- Criterion benchmarks for basic types, effects and Azure resources
- Performance range: 3.22ns (string) to 34.74µs (Azure VM resource schema validation)
- String withs patterns validation: 30.2µs. Need to explore whether regex caching helps
  bring this down.
- Azure policy effects: 188ns-1.4µs

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>

* Address PR feedback

- move error to a separate file
- use meaningful var names

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>

* Refactor

- Reusable Registry struct
- Split and simplify tests

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>

---------

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2025-08-14 15:59:30 -05:00
Anand Krishnamoorthi
fc09802bfb deps: Update dependencies across all Cargo.toml files and ignore .sln files (#450)
Dependencies updated:
- cc: 1.2.29 -> 1.2.31
- chrono-tz: 0.10.3 -> 0.10.4
- clap: 4.5.40 -> 4.5.42
- clap_builder: 4.5.40 -> 4.5.42
- clap_derive: 4.5.40 -> 4.5.41
- phf: 0.11.3 -> 0.12.1
- phf_shared: 0.11.3 -> 0.12.1
- rand: 0.9.1 -> 0.9.2
- rb-sys: 0.9.116 -> 0.9.117
- rb-sys-build: 0.9.116 -> 0.9.117
- redox_syscall: 0.5.13 -> 0.5.17
- rustix: 1.0.7 -> 1.0.8
- serde_json: 1.0.140 -> 1.0.142
- windows-targets: 0.53.2 -> 0.53.3
- winnow: 0.7.11 -> 0.7.12

Removed obsolete build dependencies:
- chrono-tz-build, parse-zoneinfo, phf_codegen, phf_generator, rand_core

Updated in: main, ffi, java, python, ruby, and wasm bindings
Added *.sln to .gitignore to exclude Visual Studio solution files

Also fix python publishing pipeline by removing non-existent dependency.

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2025-08-04 19:56:27 -05:00
Anand Krishnamoorthi
9487defa20 chore: Regorus v0.5.0 release (#432)
Also update binding versions and lock files.
Note:
- Ruby binding is not updated
- C# binding is v0.7.0. We will make it match Regorus version later.

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2025-07-08 16:40:06 -05:00
Anand Krishnamoorthi
444b2970a1 feat!: Indexes for nodes in the AST (#414)
Indexes allow associating extra data with nodes in the AST
using an array and then quickly looking up the array to fetch
the extra data.

- Index eidx for expressions
- Index sidx for statements
- Index qidx for queries.

AST nodes are not cloneable. Therefore once a module is created,
it is not possible to accidentally create two nodes with the same
index inadvertently via clone.

Also added IndexChecker in debug builds. When a module is parsed,
it will assert that indexes have been constructed correctly.

AST Cleanup
- Make literal expressions (null, val, number, string etc) also structs
  to match all other expressions
- Merge True and False nodes into a single Bool node.

Also update dependencies.

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2025-06-20 15:09:07 -05:00
Anand Krishnamoorthi
39f10326cc build(deps): Update criterion and other deps (#412)
Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2025-05-23 14:00:14 -05:00
Anand Krishnamoorthi
130f9685fd feat: Updates for Policy Framework (#405)
- Documentation
  - Regorus Engine is intended to be used from a single thread
  - Clone the engine after adding policies and data to use from another thread

- Builtin errors strictness:
  - default to less strict for OPA compatibility
  - Provide API to change strictness

- Expose GetAstAsJson to C#,
  This can allow writing policy validations in C#.

- Use spectre mitigated msvc crt libs (binskim compliance)

- Update dependencies

fixes #404

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2025-04-30 15:33:40 -05:00
Anand Krishnamoorthi
6719456468 build: Update dependencies (#401)
Also bump up C# package version

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2025-04-29 11:06:05 -05:00
Anand Krishnamoorthi
9e43bd9878 fix!: Remove cryptographic builtins (#396)
Cryptographic builtins are removed due to various reasons like FIPS
compliance. Users needing crypto builtins are encouraged to use
extensions.

Deprecated functions are also removed.

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2025-04-16 12:24:19 -07:00
Anand Krishnamoorthi
f46ab5b697 fix!: Fix glob.match behavior in presence of : (#390)
glob.match("api://*/appId", null, "api://foo.com/appId") wasn't
being handled correctly. Switch to globset crate.

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2025-04-09 11:11:57 -07:00
Anand Krishnamoorthi
c28bde3f56 build: Check-in Cargo.lock files and lockdown .net (#384)
Use frozen and locked builds

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2025-03-31 07:55:47 -07:00
Anand Krishnamoorthi
4a2df93ae2 fix!: Remove sha1 dependency (#379)
Removed cryptographically insecure sha1. This existed only for OPA
compatibility.

Also exclude bindings from main workspace

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2025-03-13 12:34:11 -07:00
Anand Krishnamoorthi
c963e477a3 feat: Update to OPA v1.2.0 (#373)
Regorus now defaults to rego v1. `import rego.v1` is no longer needed.
Additionally, `future` keywords are automatically imported.

See
https://www.openpolicyagent.org/docs/latest/v0-upgrade/#changes-to-rego-in-opa-v10
to understand the differences between rego v1 and v0.

BREAKING CHANGE:

v0 style policies will error out by default. To enable v0 behavior, call engine.set_rego_v0(true) before
loading policies.

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2025-03-10 11:56:01 -07:00
Anand Krishnamoorthi
20eece58ed Update binding versions (#287)
Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2024-07-28 12:32:39 +05:30
Anand Krishnamoorthi
46e28b36f8 feat: get_policies: Way to obtain policy files and content (#267)
closes #254

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2024-06-19 00:01:55 -07:00
Anand Krishnamoorthi
ee898e112e Update binding versions for next release (#270)
Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2024-06-18 23:39:12 -07:00
balcanuc
e6f2ec825d rename method from 'Clone' to 'clone' in 'Engine' class to match the java naming convention and definiont in the of java.lang.Object. (#268)
Co-authored-by: Cristi Balcanu <balcanu@amazon.com>
2024-06-18 20:46:21 -07:00
Anand Krishnamoorthi
df98c8d168 Provide ability to get JSON representation of policy AST (#266)
closes #265

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2024-06-08 18:58:30 -07:00
Anand Krishnamoorthi
97914d5596 Revert "chore: release v0.2.0 (#257)" (#258)
This reverts commit ffb79f1b30.
2024-05-30 06:09:47 -07:00
Anand Krishnamoorthi
ffb79f1b30 chore: release v0.2.0 (#257)
v0.2.0

Signed-off-by: Anand Krishnamoorthi <35780660+anakrish@users.noreply.github.com>

---------

Signed-off-by: Anand Krishnamoorthi <35780660+anakrish@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-05-30 05:45:43 -07:00
Anand Krishnamoorthi
d09c445add Update bindings to include newer APIs (#250)
- c, cpp
- csharp
- ffi
- go
- Java
- Python
- WASM

`arc` feature is turned on for all bindings
Use pretty string instead of colored string.

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2024-05-25 10:24:06 -07:00
Anand Krishnamoorthi
9894f00829 Fix bindings and add CI tests (#247)
Also update version numbers of binding Rust projects to match Regorus

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2024-05-22 08:15:42 -07:00
Burak
25ac1b6c95 Update README.md for Java bindings to mention we don't publish to (#246)
Maven Central currently
2024-05-20 08:15:39 -07:00
Anand Krishnamoorthi
de56cce7cb Fix anyhow dependency issues (#208)
- Do not require backtrace feature
- Starting version 1.0.77, anyhow gathers backtrace is std feature (enabled by default)
  is specified even if backtrace feature is not enabled.
  Therefore specify default features as false.
- Specify version 1.0.45 since that is the minimul version required to successfully
  compile regorus

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2024-04-20 09:57:52 -07:00
Anand Krishnamoorthi
0e053832db chore: release (#157)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-02-22 21:18:49 -08:00
Burak
22047287b4 Java publishing (#151)
* bindings/java: Add prefix to native methods

* bindings/java: Add javadocs and missing methods to Engine

* Setup publishing uber-JAR via GitHub workflow

* bindings/java: Update README

* bindings/java: Improve native library loading from JAR

* bindings/java: Fix usage of `working-directory`

* bindings/java: Pass required `distribution` parameter to `actions/setup-java@v4`

* bindings/java: Use Corretto distribution

This is because Microsoft doesn't provide JDK8,
see https://learn.microsoft.com/en-us/java/openjdk/download#openjdk-8.

* bindings/java: Install GCC toolchain for `aarch64-unknown-linux-gnu`

* bindings/java: Upload artifacts with different names from each step

* bindings/java: Upload built JARs to GitHub
2024-02-22 15:00:07 -08:00
Burak
bdb2aba596 Java bindings (#147) 2024-02-18 08:21:58 -08:00