mirror of
https://github.com/microsoft/regorus.git
synced 2026-08-05 02:16:11 +00:00
Anand Krishnamoorthi
d561531613
This commit introduces a complete multi-threaded evaluation benchmark suite for both Rust and C# implementations of Regorus. - Implemented engine evaluation benchmark with input and engine cloning strategies - Implemented compiled policy evaluation benchmark with input cloning and shared compiled policy strategies. - Created EngineEvaluationBenchmark.cs and CompiledPolicyEvaluationBenchmark.cs with time-based execution (3s warmup + 3s evaluation) - Implemented configuration options matching Rust implementation (useClonedEngines, useSharedPolicies parameters) - Created markdown analysis documentation with cross-platform performance analysis - C# seems to achieve 58-89% of Rust performance on test machine. Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
136 lines
7.8 KiB
Markdown
136 lines
7.8 KiB
Markdown
# Compiled Policy Evaluation Benchmark Results
|
|
|
|
## Test Environment
|
|
- **Platform**: Apple Silicon (M-Series)
|
|
- **CPU**: 16 cores
|
|
- **Architecture**: ARM64 (aarch64-apple-darwin)
|
|
- **Rust Version**: 1.82.0
|
|
- **Benchmark Framework**: Criterion.rs
|
|
- **Test Data**: 20,000 inputs per evaluation (1000 per thread)
|
|
- **Policy**: Complex authorization policy with nested rules
|
|
|
|
## Benchmark Overview
|
|
|
|
The compiled policy evaluation benchmark tests Regorus compiled policy performance across multiple thread configurations (1-32 threads). It measures throughput (thousands of evaluations per second) for different combinations of compiled policy and input data reuse strategies.
|
|
|
|
## Configuration Combinations
|
|
|
|
1. **Compiled Shared Policies, Cloned Inputs**: Each thread uses shared compiled policies and clones of parsed input data - optimal for performance
|
|
2. **Compiled Shared Policies, Fresh Inputs**: Each thread uses shared compiled policies but parses new inputs each time
|
|
3. **Compiled Per Iteration, Cloned Inputs**: Each thread compiles the policy each iteration but reuses input data
|
|
4. **Compiled Per Iteration, Fresh Inputs**: Each thread compiles new policies and parses new inputs for each iteration
|
|
|
|
## Performance Results
|
|
|
|
### Compiled Shared Policies, Cloned Inputs (Best Performance)
|
|
| Threads | Total Evaluation Time (ms) | Throughput (Kelem/s) |
|
|
|--------:|---------------------------:|---------------------:|
|
|
| 1 | 3.30 | 303 |
|
|
| 2 | 8.53 | 234 |
|
|
| 4 | 18.78 | 213 |
|
|
| 6 | 32.35 | 186 |
|
|
| 8 | 73.12 | 109 |
|
|
| 10 | 108.97 | 92 |
|
|
| 12 | 145.56 | 82 |
|
|
| 14 | 196.14 | 71 |
|
|
| 16 | 248.77 | 64 |
|
|
| 18 | 290.01 | 62 |
|
|
| 20 | 317.16 | 63 |
|
|
| 22 | 348.83 | 63 |
|
|
| 24 | 361.05 | 66 |
|
|
| 26 | 389.70 | 67 |
|
|
| 28 | 418.66 | 67 |
|
|
| 30 | 444.40 | 68 |
|
|
| 32 | 476.53 | 67 |
|
|
|
|
### Compiled Shared Policies, Fresh Inputs
|
|
| Threads | Total Evaluation Time (ms) | Throughput (Kelem/s) |
|
|
|--------:|---------------------------:|---------------------:|
|
|
| 1 | 4.51 | 222 |
|
|
| 2 | 9.77 | 205 |
|
|
| 4 | 23.36 | 171 |
|
|
| 6 | 38.12 | 157 |
|
|
| 8 | 85.02 | 94 |
|
|
| 10 | 133.66 | 75 |
|
|
| 12 | 180.46 | 66 |
|
|
| 14 | 238.23 | 59 |
|
|
| 16 | 318.78 | 50 |
|
|
| 18 | 353.15 | 51 |
|
|
| 20 | 389.29 | 51 |
|
|
| 22 | 459.61 | 48 |
|
|
| 24 | 507.62 | 47 |
|
|
| 26 | 539.43 | 48 |
|
|
| 28 | 554.99 | 50 |
|
|
| 30 | 625.57 | 48 |
|
|
| 32 | 690.55 | 46 |
|
|
|
|
### Compiled Per Iteration, Cloned Inputs
|
|
| Threads | Total Evaluation Time (ms) | Throughput (Kelem/s) |
|
|
|--------:|---------------------------:|---------------------:|
|
|
| 1 | 22.68 | 44 |
|
|
| 2 | 47.99 | 42 |
|
|
| 4 | 108.09 | 37 |
|
|
| 6 | 167.62 | 36 |
|
|
| 8 | 283.17 | 28 |
|
|
| 10 | 418.25 | 24 |
|
|
| 12 | 546.24 | 22 |
|
|
| 14 | 688.79 | 20 |
|
|
| 16 | 951.72 | 17 |
|
|
| 18 | 1060.20 | 17 |
|
|
| 20 | 1223.60 | 16 |
|
|
| 22 | 1342.50 | 16 |
|
|
| 24 | 1445.70 | 17 |
|
|
| 26 | 1676.50 | 15 |
|
|
| 28 | 1765.20 | 16 |
|
|
| 30 | 1939.00 | 15 |
|
|
| 32 | 2197.30 | 15 |
|
|
|
|
### Compiled Per Iteration, Fresh Inputs
|
|
| Threads | Total Evaluation Time (ms) | Throughput (Kelem/s) |
|
|
|--------:|---------------------------:|---------------------:|
|
|
| 1 | 23.95 | 42 |
|
|
| 2 | 49.53 | 40 |
|
|
| 4 | 116.42 | 34 |
|
|
| 6 | 197.35 | 30 |
|
|
| 8 | 293.04 | 27 |
|
|
| 10 | 385.90 | 26 |
|
|
| 12 | 508.82 | 24 |
|
|
| 14 | 679.23 | 21 |
|
|
| 16 | 913.02 | 18 |
|
|
| 18 | 1075.90 | 17 |
|
|
| 20 | 1209.80 | 17 |
|
|
| 22 | 1358.90 | 16 |
|
|
| 24 | 1523.90 | 16 |
|
|
| 26 | 1700.20 | 15 |
|
|
| 28 | 1966.90 | 14 |
|
|
| 30 | 2179.30 | 14 |
|
|
| 32 | 2327.70 | 14 |
|
|
|
|
## Analysis
|
|
|
|
The compiled policy benchmark demonstrates the following performance characteristics:
|
|
|
|
1. **Best Performance**: Compiled shared policies with cloned inputs provide the highest throughput
|
|
2. **Compilation Impact**:
|
|
- Pre-compiled policies: Significantly faster than per-iteration compilation
|
|
- Per-iteration compilation: Major overhead (~7x slower than pre-compiled)
|
|
3. **Scaling Patterns**:
|
|
- Best throughput achieved at 1 thread for shared policy configurations
|
|
- Higher thread counts show performance degradation due to contention
|
|
- Per-iteration compilation shows poor scaling across all thread counts
|
|
4. **Input Processing**: Fresh inputs add ~25-30% overhead across all configurations
|
|
5. **Thread Performance**:
|
|
- Peak performance at 1 thread for most configurations
|
|
- Reasonable performance maintained up to 12-16 threads for shared policies
|
|
- Compiled policies show better thread scaling than per-iteration compilation
|
|
|
|
## Comparison with Engine Evaluation
|
|
|
|
| Configuration | Compiled Policy (1 thread) | Engine Evaluation (1 thread) | Performance Ratio |
|
|
|:---------------------|:--------------------------------|:--------------------------------|------------------:|
|
|
| Shared/Cloned | Best performance | Higher throughput | 0.67x-0.92x |
|
|
| Shared/Fresh | ~27% reduction from optimal | ~30% reduction from optimal | 0.62x-0.97x |
|
|
| Per-iteration/Cloned | ~85% reduction from optimal | ~86% reduction from optimal | 0.80x-0.98x |
|
|
| Per-iteration/Fresh | ~86% reduction from optimal | ~87% reduction from optimal | 0.78x-1.00x |
|
|
|