regorus/benches/evaluation/README.md

Regorus Multi-Threaded Evaluation Benchmark

A benchmark suite for measuring the multi-threaded performance of the Regorus policy evaluation engine.

Overview

This benchmark evaluates the throughput and scalability of Regorus policy evaluation across different thread counts and configuration strategies. It measures performance variations between fresh and cloned engine instances, as well as fresh and cloned input data.

Features

• Multi-threaded evaluation testing from 1 to num_cpus * 2 threads
• Configurable engine strategies: Fresh vs. cloned engine instances
• Configurable input strategies: Fresh parsing vs. cloned input data
• Complex policy evaluation using realistic RBAC and data sensitivity policies
• Criterion-based benchmarking with statistical analysis
• Performance metrics including throughput and timing

Benchmark Structure

Test Configurations

The benchmark tests four different configuration combinations:

1. Cloned Engines + Cloned Inputs: Pre-instantiated engines with pre-parsed input data
2. Cloned Engines + Fresh Inputs: Pre-instantiated engines with fresh JSON parsing
3. Fresh Engines + Cloned Inputs: New engine instances with pre-parsed input data
4. Fresh Engines + Fresh Inputs: New engine instances with fresh JSON parsing

Thread Scaling

Tests are performed with thread counts: 1, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32 (up to num_cpus * 2)

Each thread performs 1000 policy evaluations to ensure statistically significant measurements.

Running the Benchmark

Prerequisites

• Rust 1.70+
• Cargo

Execution

Run the complete benchmark suite:

cargo bench evaluation_benchmark

Run specific benchmarks:

# Run only cloned engines with cloned inputs
cargo bench "cloned_engines , cloned_inputs"

# Run only single-threaded tests
cargo bench "1 threads"

Output

Results are generated in the target/criterion/ directory and include:

• Detailed timing statistics
• Throughput measurements (Kelem/s)
• Performance comparison with previous runs
• HTML reports with graphs and analysis

Test Policies

The benchmark uses complex Rego policies that simulate real-world scenarios:

RBAC Policy

• Role-based access control with hierarchical permissions
• User-role-resource mapping
• Action-based authorization

Data Sensitivity Policy

• Multi-level data classification (public, internal, confidential, secret)
• Access level validation
• Clearance-based filtering

Time-based Access Policy

• Business hours validation
• Temporal access control
• Schedule-based permissions

Azure Resource Policies

• VM Deployment: VM size restrictions, regional compliance, security configurations
• Storage Account Security: Encryption requirements, network ACLs, HTTPS enforcement
• Key Vault Access: Service principal validation, soft delete requirements, conditional access
• Network Security Groups: Port restrictions, CIDR validation, priority-based rules

Policy Complexity Features

• Multi-condition validation: Complex nested object property checks
• Network operations: CIDR matching and IP range validation
• Time-based constraints: Timestamp comparisons and business hour logic
• Security compliance: Encryption, authentication, and access control patterns
• Azure Resource Manager: Real-world cloud governance scenarios

Configuration

Benchmark Parameters

• Evaluations per thread: 1000
• Measurement iterations: 100 samples per configuration
• Warm-up time: 3 seconds
• Measurement time: 10 seconds (extended for high thread counts)

Customization

The benchmark can be customized by modifying evaluation_benchmark.rs:

// Adjust evaluations per thread
let evals_per_thread = 1000;

// Modify thread count calculation
let max_threads = num_cpus::get() * 2;

// Configure test scenarios
let scenarios = [
    (true, true),   // cloned_engines, cloned_inputs
    (true, false),  // cloned_engines, fresh_inputs
    (false, true),  // fresh_engines, cloned_inputs
    (false, false), // fresh_engines, fresh_inputs
];

Understanding Results

Metrics

• Total Evaluation Time: Total execution time for all evaluations across all threads (ms)
• Throughput: Evaluations per second measured in Kelem/s
  • Kelem/s: Thousands of elements (policy evaluations) per second
  • Example: 98.71 Kelem/s = 98,710 policy evaluations per second

Interpretation

• Lower time = better performance
• Higher throughput = better performance
• Consistent results across runs indicate stable performance
• Outliers may indicate system interference or measurement variance

Tips

• Run on dedicated hardware for consistent results
• Disable other applications during benchmarking
• Use release builds for accurate performance measurements
• Consider CPU affinity for highly controlled testing

Files

• evaluation_benchmark.rs: Main benchmark implementation
• Results are saved to ../../target/criterion/ directory