regorus/benches/evaluation/compiled_policy_evaluation_benchmark.md

Compiled Policy Evaluation Benchmark Results

Test Environment

• Platform: Apple Silicon (M-Series)
• CPU: 16 cores
• Architecture: ARM64 (aarch64-apple-darwin)
• Rust Version: 1.82.0
• Allocator: mimalloc (default allocator)
• Benchmark Framework: Criterion.rs
• Test Data: 20,000 inputs per evaluation (1000 per thread)
• Policy: Complex authorization policy with nested rules

Benchmark Overview

The compiled policy evaluation benchmark tests Regorus compiled policy performance across multiple thread configurations (1-32 threads). It measures throughput (thousands of evaluations per second) for different combinations of compiled policy and input data reuse strategies.

Configuration Combinations

1. Compiled Shared Policies, Cloned Inputs: Each thread uses shared compiled policies and clones of parsed input data - optimal for performance
2. Compiled Shared Policies, Fresh Inputs: Each thread uses shared compiled policies but parses new inputs each time
3. Compiled Per Iteration, Cloned Inputs: Each thread compiles the policy each iteration but reuses input data
4. Compiled Per Iteration, Fresh Inputs: Each thread compiles new policies and parses new inputs for each iteration

Performance Results

Compiled Shared Policies, Cloned Inputs (Best Performance)

Threads	Total Evaluation Time (ms)	Throughput (Kelem/s)
1	2.35	426
2	5.36	373
4	11.70	342
6	20.33	295
8	43.26	185
10	61.93	162
12	79.30	151
14	94.45	148
16	113.39	141
18	154.41	117
20	184.37	108
22	204.00	108
24	220.45	109
26	237.07	110
28	252.58	111
30	273.57	110
32	292.69	109

Compiled Shared Policies, Fresh Inputs

Threads	Total Evaluation Time (ms)	Throughput (Kelem/s)
1	3.34	299
2	7.29	274
4	15.19	263
6	24.90	241
8	49.22	163
10	68.45	146
12	86.55	139
14	104.77	134
16	136.07	118
18	169.05	106
20	198.25	101
22	217.05	101
24	234.75	102
26	254.53	102
28	276.06	101
30	296.12	101
32	318.81	100

Compiled Per Iteration, Cloned Inputs

Threads	Total Evaluation Time (ms)	Throughput (Kelem/s)
1	18.11	55
2	36.89	54
4	75.46	53
6	114.66	52
8	152.80	52
10	192.17	52
12	232.32	52
14	301.47	46
16	380.36	42
18	424.64	42
20	484.76	41
22	531.62	41
24	582.88	41
26	631.39	41
28	671.99	42
30	717.65	42
32	766.05	42

Compiled Per Iteration, Fresh Inputs

Threads	Total Evaluation Time (ms)	Throughput (Kelem/s)
1	19.07	52
2	38.89	51
4	79.52	50
6	120.89	50
8	161.08	50
10	202.37	49
12	244.04	49
14	316.66	44
16	398.02	40
18	449.54	40
20	500.57	40
22	557.97	39
24	605.71	40
26	656.88	40
28	710.03	39
30	741.09	40
32	801.26	40

Analysis

The compiled policy benchmark demonstrates the following performance characteristics with mimalloc as the default allocator:

1. Best Performance: Compiled shared policies with cloned inputs provide the highest throughput
2. Compilation Impact:
   • Pre-compiled policies: Significantly faster than per-iteration compilation
   • Per-iteration compilation: Major overhead (~7-8x slower than pre-compiled)
3. Scaling Patterns with mimalloc:
   • Best throughput achieved at 1 thread for shared policy configurations
   • mimalloc provides better thread scaling characteristics compared to the default allocator
   • Higher thread counts show performance degradation due to contention, but less severe with mimalloc
   • Per-iteration compilation shows poor scaling across all thread counts
4. Input Processing: Fresh inputs add ~30% overhead across all configurations
5. Thread Performance with mimalloc:
   • Peak performance at 1 thread for most configurations
   • Reasonable performance maintained up to 12-16 threads for shared policies
   • Compiled policies show better thread scaling than per-iteration compilation
   • mimalloc helps reduce allocation-related contention in multi-threaded scenarios

Comparison with Engine Evaluation

Multi-Thread Performance Comparison

Configuration	1 Thread (Kelem/s)	4 Threads (Kelem/s)	8 Threads (Kelem/s)
	CP / EE	CP / EE	CP / EE
Shared/Cloned	426 / 423	342 / 406	185 / 341
Shared/Fresh	299 / 309	263 / 297	163 / 266
Per-iteration/Cloned	55 / 56	53 / 54	52 / 53
Per-iteration/Fresh	52 / 53	50 / 51	50 / 51

Threading Efficiency Analysis

Configuration	Low Contention (1-4t)	Medium Contention (6-12t)	High Contention (16+t)
	Avg CP / EE	Avg CP / EE	Avg CP / EE
Shared/Cloned	384 / 414	203 / 329	123 / 250
Shared/Fresh	284 / 302	176 / 235	108 / 201
Per-iteration/Cloned	54 / 55	50 / 52	42 / 42
Per-iteration/Fresh	51 / 52	47 / 50	40 / 40

The compiled policy evaluation shows performance characteristics that are generally comparable to engine evaluation, though with some notable differences. While single-threaded performance is very close between the systems, there are observable impacts from the compilation approach that become more apparent under different threading scenarios.

Key Observations:

• Single-threaded performance: Very close parity between systems, though results may vary between runs
• Threading behavior: Engine evaluation demonstrates better scaling characteristics under higher thread contention (4+ threads)
• Multi-threaded impact: Compiled policies show more pronounced performance degradation under thread contention in shared policy configurations
• Contention resistance: Per-iteration compilation shows more consistent (though lower absolute) performance across thread counts
• Optimal usage: Both systems achieve best results with minimal threading (1-4 threads), though engine evaluation maintains better performance at higher thread counts for shared configurations