feat(azure-policy): implement count/count.where compilation (#688)

Implement the full count loop compiler, replacing the stubs in count.rs,
count_any.rs, and count_bindings.rs with a single consolidated module.

Handles both field-based and value-based count nodes. Field counts walk
the resource via resolve_alias_path then iterate the wildcard array;
value counts operate on an arbitrary collection expression.

For nested wildcard paths like A[*].B[*].C, the compiler emits recursive
ForEach loops, drilling one wildcard level at a time. When an outer
count binding already covers a prefix, the inner loop starts from the
bound element register instead of re-walking from the resource root.

Existence patterns (count > 0, count == 0) are recognized and lowered
to LoopMode::Any, which exits on the first match rather than counting
every element.

Count-binding resolution threads the current-element register through
inner field references and current() calls so that nested conditions
can address fields relative to the loop variable.

Also fixes the bound_len arithmetic in conditions_wildcard.rs with a
cleaner strip_prefix call, and removes the nested-wildcard bail in
split_count_wildcard_path since the compiler now handles them.

PR-PLAN.md

@@ -0,0 +1,313 @@
+# Azure Policy Compiler — PR Submission Plan
+
+Main is the source of truth for RVM, aliases, parser, builtins, RBAC, bindings,
+engine, etc. Only compiler/ code and its tests remain to be submitted.
+
+## Completed
+
+- **PR #686** (`azure-policy-compiler-eval` → `microsoft:main`): 2 commits
+  - Commit 1 (`68d935f`): Compiler skeleton with core types and stubs
+  - Commit 2 (`c17a438`): Condition, expression, field, and template dispatch compilation
+  - Status: Draft, Copilot review clean (0 new comments on latest push)
+  - Files: 14 new files in compiler/, +2,557 lines vs main
+
+- **PR #688** (Count support): 1 squashed commit on `azure-policy-compiler-count`
+  - Full count loop compilation replacing stubs
+  - Status: In review, Copilot comments addressed
+
+## Total remaining (compiler only): 7 files, +4,330 lines vs main
+
+After PR #686: +2,984/-1,211 lines across 14 compiler files (restructuring)
+
+Final state on `azure-policy-compiler`:
+- mod.rs (1,681 LOC) — main pipeline, effects, metadata, emit helpers, aliases
+- count.rs (912 LOC) — count loops, count-as-any, bindings
+- conditions.rs — condition compilation + wildcard allOf
+- fields.rs (385 LOC) — field path compilation
+- template_dispatch.rs (369 LOC) — ARM function dispatch
+- expressions.rs (337 LOC) — expression & JSON value compilation
+- utils.rs (143 LOC) — shared helpers
+- (stubs from PR #686 deleted: core.rs, conditions_wildcard.rs, metadata.rs,
+  effects.rs, effects_modify_append.rs, count_any.rs, count_bindings.rs)
+
+---
+
+## PR 4: Effects + Metadata + File Restructure
+
+### Goal
+Complete the compiler by implementing effects, metadata, and consolidating files
+(core.rs → mod.rs, conditions_wildcard.rs → conditions.rs, etc.).
+
+### Phase A: Implement effects (in effects.rs or mod.rs)
+
+#### Step 1: Implement compile_effect()
+Replace the bail stub with full effect dispatch:
+- Resolve effect kind via `resolve_effect_kind()` (handles parameterized `[parameters('effect')]`)
+- Match on EffectKind: Deny, Audit, Disabled, Append, Modify, AuditIfNotExists, DeployIfNotExists, DenyAction, AddToNetworkGroup
+- Simple effects (Deny, Audit, Disabled): load effect name literal, wrap via `wrap_effect_result()`
+- Detail effects (Modify, Append): call `compile_effect_with_details()` → routes to `compile_modify_details()` or `compile_append_details()`
+- Cross-resource effects (AINE, DINE): call `compile_cross_resource_effect()` which emits `HostAwait` instruction
+
+#### Step 2: Implement wrap_effect_result()
+Replace bail stub:
+- Build structured result object `{ "effect": <name_reg>, "details": <details_reg> }`
+- Uses `Instruction::ObjectNew`, `Instruction::ObjectInsert` sequences
+- When details_reg is None, omit the details field
+
+#### Step 3: Implement Modify/Append details
+In effects_modify_append.rs (or same file depending on restructure):
+- `compile_modify_details()` — iterates `details.operations` array, compiles each modify operation
+- `compile_modify_operation()` — handles addOrReplace/Add/Remove operations with field/value pairs
+- `compile_append_details()` — iterates `details` array items
+- `compile_append_item()` — compiles individual append { field, value } items
+
+#### Step 4: Implement cross-resource effects (AINE/DINE)
+- `compile_cross_resource_effect()` — emits HostAwait instruction to request related resource lookup
+- Sets `resource_override_reg` to the host response register for existenceCondition compilation
+- Compiles `details.existenceCondition` constraint against the related resource
+- Builds structured result with effect name + details (including type, resourceGroupName, etc.)
+
+#### Step 5: Implement effect resolution helpers
+- `resolve_effect_kind()` — if effect node is parameter reference, resolves via `parameter_defaults`
+- `resolve_effect_kind_from_parameter_default()` — extracts effect value from `parameters('effectParam')` expression
+- `resolve_effect_name_from_parameter_default()` — string version
+- `effect_kind_from_string()` — maps lowercase string → EffectKind enum
+- `compile_effect_name_expression()` — compiles runtime effect name from parameter expression
+
+### Phase B: Implement metadata
+
+#### Step 6: Implement metadata recording functions
+Replace no-op stubs in metadata.rs:
+- `record_field_kind()` — `self.observed_field_kinds.insert(name.to_string())`
+- `record_alias()` — `self.observed_aliases.insert(path.to_string())`
+- `record_tag_name()` — `self.observed_tag_names.insert(tag.to_string())`
+- `record_operator()` — maps OperatorKind to string, `self.observed_operators.insert()`
+- `record_resource_type_from_condition()` — if condition is `{ field: "type", equals: X }`, insert X into `observed_resource_types`
+
+#### Step 7: Implement resolve_effect_annotation()
+Replace raw-clone stub:
+- When effect is parameterized, resolve from `parameter_defaults` to get the actual effect name
+- Fall back to `effect.raw` if resolution fails
+
+#### Step 8: Implement populate_compiled_annotations()
+Replace no-op stub:
+- Insert into `program.metadata.annotations`: field_kinds, aliases, tag_names, operators, resource_types (as Value sets)
+- Insert boolean flags: uses_count, has_dynamic_fields, has_wildcard_aliases, has_host_await
+- Set `program.metadata.annotations["effect"]` (already done in init_effect_annotation)
+
+#### Step 9: Implement populate_definition_metadata()
+Replace no-op stub:
+- Extract from PolicyDefinition: display_name, description, mode, category, version, preview flag
+- Insert into `program.metadata.annotations`: parameter_names list, policy_type, policy_id, policy_name
+
+### Phase C: File restructure
+
+#### Step 10: Merge core.rs into mod.rs
+Move all content from core.rs into mod.rs:
+- `Compiler` struct definition
+- `CountBinding` struct definition
+- `compile()` pipeline
+- All register/span/emit helpers
+- All literal/builtin/chained-index helpers
+- All alias resolution functions (`resolve_alias_path`, `strip_fq_prefix`)
+- `patch_end_pc`, `current_pc`, `emit_coalesce_undefined_to_null`, `load_input`, `load_context`
+
+Update all `use super::core::Compiler;` → `use super::Compiler;` in:
+- conditions.rs
+- expressions.rs
+- fields.rs
+- template_dispatch.rs
+
+Delete `core.rs` and remove `mod core;` from mod.rs.
+
+#### Step 11: Merge conditions_wildcard.rs into conditions.rs
+Move 4 functions into conditions.rs:
+- `has_unbound_wildcard_field()`
+- `has_inner_unbound_wildcard_field()`
+- `compile_condition_wildcard_allof()`
+- `compile_allof_loop_inner()`
+
+Delete `conditions_wildcard.rs` and remove `mod conditions_wildcard;` from mod.rs.
+
+#### Step 12: Merge effects/metadata stubs into mod.rs
+If effects.rs and metadata.rs have been implemented as separate files, merge them into mod.rs.
+Alternatively, implement directly in mod.rs.
+
+Delete: effects.rs, effects_modify_append.rs, metadata.rs
+Remove their `mod` declarations from mod.rs.
+
+#### Step 13: Simplify utils.rs
+On the final branch, utils.rs is 143 LOC (current eval has ~429 LOC extensions that were trimmed).
+- Verify `split_count_wildcard_path` matches final version
+- Verify `split_path_without_wildcards` matches
+- Ensure `json_value_to_runtime` has `pub(crate)` visibility
+
+#### Step 14: Apply comment/doc and minor code differences
+Based on comparison, apply these adjustments to match final branch:
+- **expressions.rs**: Import path changes, comment enhancements, minor code tweaks
+- **fields.rs**: Import path changes, documentation expansion
+- **template_dispatch.rs**: Import path change, section header formatting
+- **conditions.rs**: Import changes, `patch_end_pc` return type, documentation additions
+
+### Relevant files
+- `src/languages/azure_policy/compiler/mod.rs` — absorbs core.rs + effects + metadata → grows to ~1,681 LOC
+- `src/languages/azure_policy/compiler/core.rs` — DELETE (merged into mod.rs)
+- `src/languages/azure_policy/compiler/conditions.rs` — absorbs conditions_wildcard.rs content
+- `src/languages/azure_policy/compiler/conditions_wildcard.rs` — DELETE (merged into conditions.rs)
+- `src/languages/azure_policy/compiler/effects.rs` — DELETE (merged into mod.rs)
+- `src/languages/azure_policy/compiler/effects_modify_append.rs` — DELETE (merged into mod.rs)
+- `src/languages/azure_policy/compiler/metadata.rs` — DELETE (merged into mod.rs)
+- `src/languages/azure_policy/compiler/expressions.rs` — import path + minor adjustments
+- `src/languages/azure_policy/compiler/fields.rs` — import path + documentation
+- `src/languages/azure_policy/compiler/template_dispatch.rs` — import path + formatting
+- `src/languages/azure_policy/compiler/utils.rs` — streamline to 143 LOC final version
+
+### Line counts
+- mod.rs: +1,614 (absorbs core.rs, adds effects, metadata, emit helpers, aliases)
+- Delete: core.rs (-367), conditions_wildcard.rs (-199), metadata.rs (-52 stub),
+  effects.rs (-30 stub), effects_modify_append.rs (-6 stub)
+- utils.rs: -320 (functions moved into mod.rs)
+- template_dispatch.rs: +75 (new function dispatches)
+- Effects: Deny, Audit, Modify, Append, DenyAction, AINE, DINE
+- Cross-resource evaluation (host_await)
+- Modify/Append details, effect resolution from parameters
+- Metadata: field kinds, aliases, operators, resource types
+
+### Verification
+1. `cargo build` — all effects/metadata compiled, no stubs remain
+2. `cargo clippy` — remove all `#![allow(dead_code)]` from deleted stubs
+3. `cargo test --features azure_policy` — existing tests still pass
+4. `TEST_CASE_FILTER="effect" cargo test --features azure_policy -- --nocapture`
+5. Verify final file list matches: mod.rs, conditions.rs, count.rs, expressions.rs, fields.rs, template_dispatch.rs, utils.rs (7 files)
+
+---
+
+## PR 5: Test Suite
+
+### Goal
+Add the full YAML-driven test suite: 58 high-level cases + 8 parser cases + alias test data.
+
+### Step 1: Update tests/azure_policy/mod.rs
+Replace the 5-line eval version with the full 700+ line test runner that includes:
+- `TestCase` struct with all fields (host_await, want_details, api_version, request_context, context, etc.)
+- `HostAwaitEntry` struct
+- `YamlTest` struct with aliases/global policy_rule/policy_definition support
+- `yaml_test_impl()` — full evaluation pipeline (parse → compile → normalize → VM execute → assert)
+- Helper functions: `make_input()`, `make_context()`, `yaml_to_regorus_value()`, `lowercase_value_keys()`, `lowercase_json_keys()`, `extract_effect_name()`, `extract_details()`, `extract_details_resource_type()`, `inject_type_field()`
+- `#[test_resources("tests/azure_policy/cases/*.yaml")]` auto-discovery
+- `test_specific_case()` with `TEST_CASE_FILTER` support
+- `DEBUG_LISTING` and `DEBUG_RESOURCE` environment variable support
+- Remove `mod normalization;` (normalization tests already on main)
+
+### Step 2: Add test_aliases.json (if not already present)
+- Verify `tests/azure_policy/aliases/test_aliases.json` exists (it does on eval branch)
+- Add `tests/azure_policy/aliases/versioned_aliases.json` if needed
+
+### Step 3: Create tests/azure_policy/cases/ directory with 74 YAML files
+Add all YAML test case files. Categories:
+
+**Foundation tests (13 files):**
+- aliases.yaml, casing.yaml, effects.yaml, effect_details.yaml, exists.yaml
+- expressions.yaml, fields.yaml, field_wildcard_collect.yaml
+- implicit_allof.yaml, logical_combinators.yaml, modifiable_check.yaml
+- operators.yaml, value_conditions.yaml
+
+**Count tests (1 file):**
+- count.yaml (field count, value count, where clauses, nested, count-as-any)
+
+**Template function tests (3 files):**
+- template_functions.yaml, template_functions_datetime_ip.yaml, template_functions_extra.yaml
+
+**Advanced tests (4 files):**
+- deep_nesting.yaml, type_coercion.yaml, parse_errors.yaml, policy_definition.yaml
+
+**Infrastructure tests (2 files):**
+- azure_policies.yaml, complex_policies.yaml, versioned_normalization.yaml
+
+**E2E real-world policies (51 files):**
+- e2e_aci_*.yaml, e2e_aks_*.yaml, e2e_approved_*.yaml, e2e_asc_*.yaml
+- e2e_automanage_*.yaml, e2e_azupdate_*.yaml, e2e_cmk_*.yaml
+- e2e_container_*.yaml, e2e_cosmos_*.yaml, e2e_custom_*.yaml
+- e2e_datafactory_*.yaml, e2e_dcra_*.yaml, e2e_double_*.yaml
+- e2e_fic_*.yaml, e2e_functionapp_*.yaml, e2e_guest_*.yaml
+- e2e_keyvault_*.yaml, e2e_managed_*.yaml, e2e_monitoring_*.yaml
+- e2e_nic_*.yaml, e2e_nsg_*.yaml, e2e_pg_*.yaml, e2e_portal_*.yaml
+- e2e_servicebus_*.yaml, e2e_shared_*.yaml, e2e_signalr_*.yaml
+- e2e_sql_*.yaml, e2e_ssh_*.yaml, e2e_storage_*.yaml
+- e2e_stream_*.yaml, e2e_tags_*.yaml, e2e_vm_*.yaml, e2e_vnet_*.yaml
+
+### Step 4: Update parser tests if needed
+- Verify `tests/azure_policy/parser_tests/` cases are up to date
+- Check if any new parser test YAML files need to be added (8 files on final branch)
+
+### Step 5: Handle normalization test directory
+- The eval branch has `tests/azure_policy/normalization/` with 13 YAML cases
+- The final branch does NOT have this directory (these tests are already on main)
+- Ensure `mod normalization;` is removed from the test mod.rs if normalization tests shipped in an earlier PR
+
+### Relevant files
+- `tests/azure_policy/mod.rs` — replace with full 700+ line test runner
+- `tests/azure_policy/cases/*.yaml` — 74 new YAML test case files
+- `tests/azure_policy/aliases/test_aliases.json` — verify present
+- `tests/azure_policy/aliases/versioned_aliases.json` — verify present
+- `tests/azure_policy/parser_tests/` — verify/update
+
+### Line counts
+- ~84 azure_policy test files (+32,806/-6,051 across 156 test files total)
+- E2e YAML test suites (74+ cases)
+- External test runner with known-failure tracking
+- Lockdown test policies (9 real-world policies)
+- RVM VM suite updates for changed instruction semantics
+
+### Verification
+1. `cargo test --features azure_policy` — all 74 YAML cases + 8 parser cases pass
+2. `TEST_CASE_FILTER="count" cargo test --features azure_policy -- --nocapture` — count cases pass
+3. `TEST_CASE_FILTER="effect" cargo test --features azure_policy -- --nocapture` — effect cases pass
+4. `TEST_CASE_FILTER="e2e" cargo test --features azure_policy -- --nocapture` — all E2E policies pass
+5. `cargo clippy --features azure_policy --all-targets` — no warnings in test code
+6. `cargo xtask pre-push` — full CI check passes
+
+---
+
+## Execution Order & Dependencies
+
+```
+PR #686 (Skeleton + Conditions) ← merged/in review
+    ↓
+PR #688 (Count) ← in review, builds on PR #686
+    ↓
+PR 4 (Effects + Restructure) ← depends on PR #688 (count bindings used in effects)
+    ↓
+PR 5 (Tests) ← depends on PR 4 (tests exercise full compiler including effects)
+```
+
+PRs #688 and 4 could potentially be combined into one PR if review size is acceptable (~2,000 lines).
+PR 5 is large (~33k lines) but is purely test data — can be reviewed for structure rather than line-by-line.
+
+## Key Decisions
+- All implementation should match the final `azure-policy-compiler` branch state
+- `to_lowercase()` vs `to_ascii_lowercase()`: eval branch already fixed to `to_ascii_lowercase()`; keep that fix (it's better)
+- `patch_end_pc` return type: eval has `Result<()>`, final has `()` — reconcile during restructure
+- Strict path validation in utils.rs: eval has more guard rails; reconcile to match simpler final version
+- `pub(super)` visibility on `emit_policy_operator`: eval has it; final makes it `fn` private — reconcile during merge
+
+## Key Context
+
+### Source branches
+- **`azure-policy-compiler`** — final branch with completed compiler (source of truth for target state)
+- **`azure-policy-compiler-eval`** — worktree at `/tmp/azure-policy-compiler-eval` where PRs are built incrementally
+
+### Build & test commands
+- `cargo fmt` — format
+- `cargo clippy --all-features` — lint
+- `cargo test --all-features -- count` — run count-related tests
+- `cargo xtask pre-commit` — pre-commit hook (build + fmt + clippy)
+- `cargo xtask pre-push` — full CI (pre-commit + doc tests + no_std + full test suite + 2861 OPA tests)
+
+### Git workflow
+- Edit files → `cargo fmt` → `git add -A && git commit --amend --no-edit` → `git push origin <branch> --force`
+- All from `/tmp/azure-policy-compiler-eval` worktree
+
+### Crate constraints
+- `#![deny(clippy::indexing_slicing, clippy::expect_used)]` — cannot use `.expect()` or `[]` indexing
+- `no_std` compatible: use `alloc::{format, string, vec}` imports

src/languages/azure_policy/compiler/conditions_wildcard.rs

@@ -4,6 +4,7 @@
 
 //! Implicit allOf for unbound `[*]` wildcard fields.
 
+use alloc::format;
 use alloc::string::{String, ToString as _};
 use alloc::vec::Vec;
 
@@ -67,12 +68,12 @@ impl Compiler {
         };
 
         if let Some(prefix) = &binding.field_wildcard_prefix {
-            let bound_len = prefix.len().saturating_add(4); // len("prefix") + len("[*].")
+            let lc_prefix = prefix.to_ascii_lowercase();
-            if path.len() > bound_len {
+            let bound_prefix = format!("{}[*].", lc_prefix);
-                if let Some(remainder) = path.get(bound_len..) {
+            if let Some(remainder) = path.to_ascii_lowercase().strip_prefix(&bound_prefix) {
-                    if remainder.contains("[*]") {
+                let remainder = remainder.to_string();
-                        return Ok(Some((binding, remainder.to_string())));
+                if remainder.contains("[*]") {
-                    }
+                    return Ok(Some((binding, remainder)));
                 }
             }
         }

src/languages/azure_policy/compiler/count_any.rs

@@ -1,6 +0,0 @@
-// Copyright (c) Microsoft Corporation.
-// Licensed under the MIT License.
-
-//! Existence-pattern optimization (count → Any loop).
-//!
-//! Stub — real implementation added in a later commit.

src/languages/azure_policy/compiler/count_bindings.rs

@@ -1,40 +0,0 @@
-// Copyright (c) Microsoft Corporation.
-// Licensed under the MIT License.
-#![allow(dead_code)]
-
-//! Count-binding resolution and `current()` references.
-//!
-//! Stub — real implementation added in a later commit.
-
-use anyhow::{bail, Result};
-
-use super::core::{Compiler, CountBinding};
-
-impl Compiler {
-    pub(super) const fn resolve_count_binding(
-        &self,
-        _field_path: &str,
-    ) -> Result<Option<CountBinding>> {
-        _ = self.register_counter;
-        Ok(None)
-    }
-
-    pub(super) fn compile_from_binding(
-        &mut self,
-        _binding: &CountBinding,
-        _field_path: &str,
-        span: &crate::lexer::Span,
-    ) -> Result<u8> {
-        let _ = self;
-        bail!(span.error("count binding compilation not yet implemented"))
-    }
-
-    pub(super) fn compile_current_reference(
-        &mut self,
-        _key: &str,
-        span: &crate::lexer::Span,
-    ) -> Result<u8> {
-        let _ = self;
-        bail!(span.error("current() reference not yet implemented"))
-    }
-}

src/languages/azure_policy/compiler/mod.rs

@@ -8,9 +8,8 @@
 //! - [`core`]: `Compiler` struct, main pipeline, register/emit helpers
 //! - [`conditions`]: constraint / condition / LHS compilation
 //! - [`conditions_wildcard`]: implicit allOf for unbound `[*]` fields
-//! - [`count`]: `count` / `count.where` loops
+//! - [`count`]: `count` / `count.where` loops, existence-pattern optimization,
-//! - [`count_any`]: existence-pattern optimization (count → Any loop)
+//!   count-binding resolution and `current()` references
-//! - [`count_bindings`]: count-binding resolution and `current()` references
 //! - [`expressions`]: template-expression and call-expression compilation
 //! - [`fields`]: field-kind and resource-path compilation
 //! - [`template_dispatch`]: ARM template function dispatch
@@ -23,8 +22,6 @@ mod conditions;
 mod conditions_wildcard;
 mod core;
 mod count;
-mod count_any;
-mod count_bindings;
 mod effects;
 mod effects_modify_append;
 mod expressions;

src/languages/azure_policy/compiler/utils.rs

@@ -1,6 +1,6 @@
 // Copyright (c) Microsoft Corporation.
 // Licensed under the MIT License.
-#![allow(dead_code, clippy::pattern_type_mismatch)]
+#![allow(clippy::pattern_type_mismatch)]
 
 //! Free helper functions used by the Azure Policy compiler.
 
@@ -43,9 +43,6 @@ pub(super) fn split_count_wildcard_path(path: &str) -> Result<(String, Option<St
             path
         )
     })?;
-    if after_wildcard.contains("[*]") {
-        bail!("nested [*] wildcards are not supported: {}", path);
-    }
     let suffix_str = after_wildcard.trim_start_matches('.');
     let suffix = if suffix_str.is_empty() {
         None
@@ -351,7 +348,9 @@ mod tests {
 
     #[test]
     fn wildcard_nested() {
-        split_count_wildcard_path("a[*].b[*].c").unwrap_err();
+        let (prefix, suffix) = split_count_wildcard_path("a[*].b[*].c").unwrap();
+        assert_eq!(prefix, "a");
+        assert_eq!(suffix.as_deref(), Some("b[*].c"));
     }
 
     // -----------------------------------------------------------------------