mirror of
https://github.com/microsoft/regorus.git
synced 2026-08-05 02:16:11 +00:00
Anand Krishnamoorthi
d561531613
This commit introduces a complete multi-threaded evaluation benchmark suite for both Rust and C# implementations of Regorus. - Implemented engine evaluation benchmark with input and engine cloning strategies - Implemented compiled policy evaluation benchmark with input cloning and shared compiled policy strategies. - Created EngineEvaluationBenchmark.cs and CompiledPolicyEvaluationBenchmark.cs with time-based execution (3s warmup + 3s evaluation) - Implemented configuration options matching Rust implementation (useClonedEngines, useSharedPolicies parameters) - Created markdown analysis documentation with cross-platform performance analysis - C# seems to achieve 58-89% of Rust performance on test machine. Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
125 lines
7.1 KiB
Markdown
125 lines
7.1 KiB
Markdown
# Engine Evaluation Benchmark Results
|
|
|
|
## Test Environment
|
|
- **Platform**: Apple Silicon (M-Series)
|
|
- **CPU**: 16 cores
|
|
- **Architecture**: ARM64 (aarch64-apple-darwin)
|
|
- **Rust Version**: 1.82.0
|
|
- **Benchmark Framework**: Criterion.rs
|
|
- **Test Data**: 20,000 inputs per evaluation (1000 per thread)
|
|
- **Policy**: Complex authorization policy with nested rules
|
|
|
|
## Benchmark Overview
|
|
|
|
The engine evaluation benchmark tests Regorus policy evaluation performance across multiple thread configurations (1-32 threads). It measures throughput (thousands of evaluations per second) for different combinations of engine and input data reuse strategies.
|
|
|
|
## Configuration Combinations
|
|
|
|
1. **Cloned Engines, Cloned Inputs**: Each thread uses its own engine and clones of parsed input data - optimal for performance
|
|
2. **Cloned Engines, Fresh Inputs**: Each thread uses its own engine but parses new inputs each time
|
|
3. **Fresh Engines, Cloned Inputs**: Each thread creates a new engine each iteration but reuses input data
|
|
4. **Fresh Engines, Fresh Inputs**: Each thread creates new engines and parses new inputs for each iteration
|
|
|
|
## Performance Results
|
|
|
|
### Cloned Engines, Cloned Inputs (Best Performance)
|
|
| Threads | Total Evaluation Time (ms) | Throughput (Kelem/s) |
|
|
|--------:|---------------------------:|---------------------:|
|
|
| 1 | 3.05 | 328 |
|
|
| 2 | 7.46 | 268 |
|
|
| 4 | 16.10 | 248 |
|
|
| 6 | 25.94 | 231 |
|
|
| 8 | 50.18 | 159 |
|
|
| 10 | 80.27 | 125 |
|
|
| 12 | 106.31 | 113 |
|
|
| 14 | 137.31 | 102 |
|
|
| 16 | 163.91 | 98 |
|
|
| 18 | 182.06 | 99 |
|
|
| 20 | 191.36 | 105 |
|
|
| 22 | 201.51 | 109 |
|
|
| 24 | 217.65 | 110 |
|
|
| 26 | 228.11 | 114 |
|
|
| 28 | 248.17 | 113 |
|
|
| 30 | 264.15 | 114 |
|
|
| 32 | 314.27 | 102 |
|
|
|
|
### Cloned Engines, Fresh Inputs
|
|
| Threads | Total Evaluation Time (ms) | Throughput (Kelem/s) |
|
|
|--------:|---------------------------:|---------------------:|
|
|
| 1 | 4.36 | 229 |
|
|
| 2 | 10.34 | 194 |
|
|
| 4 | 21.98 | 182 |
|
|
| 6 | 34.05 | 176 |
|
|
| 8 | 66.47 | 120 |
|
|
| 10 | 100.78 | 99 |
|
|
| 12 | 141.69 | 85 |
|
|
| 14 | 188.53 | 74 |
|
|
| 16 | 261.27 | 61 |
|
|
| 18 | 285.29 | 63 |
|
|
| 20 | 312.14 | 64 |
|
|
| 22 | 329.42 | 67 |
|
|
| 24 | 347.97 | 69 |
|
|
| 26 | 370.24 | 70 |
|
|
| 28 | 394.75 | 71 |
|
|
| 30 | 419.30 | 72 |
|
|
| 32 | 433.58 | 74 |
|
|
|
|
### Fresh Engines, Cloned Inputs
|
|
| Threads | Total Evaluation Time (ms) | Throughput (Kelem/s) |
|
|
|--------:|---------------------------:|---------------------:|
|
|
| 1 | 22.39 | 45 |
|
|
| 2 | 49.22 | 41 |
|
|
| 4 | 98.09 | 41 |
|
|
| 6 | 160.21 | 37 |
|
|
| 8 | 281.26 | 28 |
|
|
| 10 | 413.61 | 24 |
|
|
| 12 | 578.15 | 21 |
|
|
| 14 | 746.34 | 19 |
|
|
| 16 | 961.44 | 17 |
|
|
| 18 | 1127.70 | 16 |
|
|
| 20 | 1248.40 | 16 |
|
|
| 22 | 1386.90 | 16 |
|
|
| 24 | 1559.70 | 15 |
|
|
| 26 | 1736.30 | 15 |
|
|
| 28 | 1891.80 | 15 |
|
|
| 30 | 2077.00 | 14 |
|
|
| 32 | 2289.30 | 14 |
|
|
|
|
### Fresh Engines, Fresh Inputs
|
|
| Threads | Total Evaluation Time (ms) | Throughput (Kelem/s) |
|
|
|--------:|---------------------------:|---------------------:|
|
|
| 1 | 23.63 | 42 |
|
|
| 2 | 48.82 | 41 |
|
|
| 4 | 102.32 | 39 |
|
|
| 6 | 160.09 | 37 |
|
|
| 8 | 271.21 | 29 |
|
|
| 10 | 397.39 | 25 |
|
|
| 12 | 489.09 | 25 |
|
|
| 14 | 670.33 | 21 |
|
|
| 16 | 884.83 | 18 |
|
|
| 18 | 1044.00 | 17 |
|
|
| 20 | 1174.20 | 17 |
|
|
| 22 | 1330.40 | 17 |
|
|
| 24 | 1480.90 | 16 |
|
|
| 26 | 1679.50 | 15 |
|
|
| 28 | 1873.90 | 15 |
|
|
| 30 | 2070.90 | 14 |
|
|
| 32 | 2325.40 | 14 |
|
|
|
|
## Analysis
|
|
|
|
The benchmark results demonstrate the following performance characteristics:
|
|
|
|
1. **Best Performance**: Cloned engines with cloned inputs consistently deliver the highest throughput
|
|
2. **Configuration Performance Hierarchy**:
|
|
- Cloned engines, cloned inputs: Best performance (optimal configuration)
|
|
- Cloned engines, fresh inputs: ~30% reduction from optimal
|
|
- Fresh engines, cloned inputs: ~86% reduction from optimal
|
|
- Fresh engines, fresh inputs: ~87% reduction from optimal
|
|
3. **Scaling Patterns**:
|
|
- Performance degrades with increased thread count due to contention
|
|
- Best throughput achieved at 1 thread for cloned engine configurations
|
|
- Fresh engine configurations show poor scaling across all thread counts
|
|
4. **Engine Creation Overhead**: Fresh engine creation is a significant performance bottleneck (~7-8x slower than cloned engines)
|
|
5. **Input Processing**: Fresh input generation adds moderate overhead (~30% impact compared to cloned inputs)
|
|
6. **Thread Contention**: Performance degradation occurs with higher thread counts across all configurations |