Commit Graph

106 Commits

Author SHA1 Message Date
Anand Krishnamoorthi
aabf329088 Update READEME.md with current status, grammar etc. (#102)
Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2024-01-11 13:12:15 -08:00
Burak
49bd657a41 Implement builtin time.parse_duration_ns method (#100)
* Implement builtin `time.parse_duration_ns` method

* Parse durations properly in `test.sleep` method
2024-01-08 02:21:35 -08:00
Anand Krishnamoorthi
c0fa1c1a42 Implement import keyword (#101)
Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2024-01-08 01:58:37 -08:00
Anand Krishnamoorthi
3e1ddac2a9 OPA conformance: Pass refheads test suite (#90)
- parser: allow non-string index at any position; not just last
- impl Default for Context
- Fix width of OPA test results table
- Allow non string compoenent anywhere in rule ref; not just as last item.
- Normalize want_result before comparison.
- Ensure that object rules are created even if no definition succeed,
- Sort want_result values for "refheads/general, multiple result-set entries"
  The entries are in reverse order of how OPA and regorus produce.
- Emit PASS status for each OPA testpoint
- Detect rule conflicts

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2024-01-05 11:13:25 -08:00
Anand Krishnamoorthi
25dec7e59b OPA conformance: Ensure that withkeyword OPA tests pass (#88)
- ignore worktrees
- feature guard time module
- Apply with modifiers before evaluating loop expressions
- Support value modifier for functions
- stubs for http.send and io.jwt.decode_verify
- Initialize with-document after initializing init data
- In case of conflict, with modifier override init-data values.
- In case of conflict, subsequent with modifier overrides earlier ones.
- Ensure that zero parameter functions are evaluated and added to document
- opa.runtime builtin
  returns:
   - git commit hash
   - environment vars
   - regorus features enabled
   - builtins available
   - deprecated builtins available
- If `sort_bindings` is specified, sort the bindings in OPA tests
- gather inputs, used vars and comprehensions in with modifiers
- For refs starting with `data`, ensure that modules are evaluated before looking up
  value of the expression. Thie ensures that modules that have only been partly populated
  (E.g via with mods) are completely evaluated before the value is looked up
- Mark rules overridden using with modifiers are evaluated.
- Exclude env vars in opa.runtime.
- Include regorus version in OPA runtime
- update to opa v0.60.0
- scheduler: Handle function refs in with modifers. Error out only if
  a truly undefined ref.
- Handle undefined params, parameter expression evaluation errors before
  applying with modifiers.
- When applying with modifiers, first determine whether the target is a
  function. If so, handle cleanly.
- concat: raise error only in strict mode
- In strict mode, propagate errors raised by function rule execution
  in case of multiple function definitions for same rule
- skip "withkeyword/builtin-builtin: arity 0" test which can never pass.
- When a mock has is being applied, clear with_function so that
  other mocks won't be applied during the evaluation of the mock.
- Ability to specify strictness in tests

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2024-01-04 01:13:04 -08:00
Anand Krishnamoorthi
b470c3fb7f Handle walk builtin as a loop expression (#86)
The walk builtin generates values and implicitly creates a loop over the values.
Hoist walk calls as loops and handle them. Also handle cases where return value
is bound to an extra parameter.

Closes #83
2024-01-01 15:16:57 -08:00
Burak
d2eb3ecd1f Implement most of the builtin time module (#82)
* Implement builtin `time.add_date` method

* Implement builtin `time.clock` method

* Implement builtin `time.date` method

* Implement builtin `time.diff` method

* Implement builtin `time.format` method

* Migrate `time.now_ns` to `chrono`

* Implement builtin `time.parse_ns` method

* Implement builtin `time.parse_rfc3339_ns` method

* Implement builtin `time.weekday` method

* Add conditional `test` module for OPA tests

* Cache result of `time.now_ns`

* Fail in strict mode if timestamp is outside of range

* Add `ensure_i32` util

* Move `diff_between_datetimes` into its own file and include appropriate license
2023-12-31 17:07:39 -08:00
Anand Krishnamoorthi
2292774446 OPA Conformance
- Disable default features of jsonschema to reduce binary size (#85)
- graph.reachable, graph.reachable_paths builtins.
- In progress walk builtin
Note: graph.reachable_paths is buggy in upstream OPA and its semantics
  are not well defined.
  https://github.com/open-policy-agent/opa/issues/5871
  https://github.com/open-policy-agent/opa/issues/6128
- Use correct feature for regex
- Ensure that regorus complies with all features disabled.

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-12-31 13:51:53 -08:00
Anand Krishnamoorthi
84bedda1c6 OPA conformance (#81)
- Switch to scientific crate. Large values are printed in scientific notations.
  Regular values are printed as u64, i64 or f64.
- Skip copying commit hooks in git worktrees
- urlquery.decode, urlquery.encode, urlquery.encode_object
- substring, indexof_n string builtins
- Make sprintf more OPA conformant

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-12-30 08:36:15 -08:00
Anand Krishnamoorthi
e549882b07 More OPA conformance (#77)
- object.union
- object.union_n
- treat negative integers as two separate tokens (Sub and Number)
  when seen in arithmetic expressions
- Ensure that fully query string is parsed
- Evaluate queries in a separate module instead of the last read module.
  This correctly handles queries of the form `x = data.test.y` where x is
  already a ref in `data.test`
- Handle queries producing multiple outputs in test infrastructure
- Add tests for engine
- Add tests locking down valid queries
- Update opa.passing

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-12-27 17:00:40 -08:00
Anand Krishnamoorthi
e61b406547 OPA conformance (#71)
- Remove unnecessary memory allocations
- Add --non-strict flag
- Ensure that only empty modules (ones without rules) are initialzed prior to evaluating rules.
- Record rule as entry for each of its prefixes.
  For example, for a rule a.b.c =... in package test, record it in
  rules["data.test.a"], rules["data.test.a.b"] and rules["data.test.a.b.c"]

  This allows evaluating the correct list of rules based on expessions
  a.b.c, a.b, a, data.test.a.b.c, data.test.a.b, data.test.a

Closes #69
Closes #70

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-12-23 11:55:25 -08:00
Burak
ad3282caf4 Builtin UUID module (#68)
* Implement builtin `uuid.parse` method
* Implement builtin `uuid.rfc4122` method
* Parse timestamps for v2 UUIDs
2023-12-23 09:35:20 -08:00
Burak
1fb144b145 Add tests for builtin string::format_int method (#65) 2023-12-22 09:31:57 -08:00
Anand Krishnamoorthi
f6140b6be5 More builtins and semantic improvements (#66)
- base64 builtins
- base64url builtins
- jsonschema builtins
- json.remove builtin
- handle composite index variables
- use dashu_float since rust_decimal has lesser precision

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-12-21 13:34:48 -08:00
Anand Krishnamoorthi
577e1aa8db More OPA conformance; in-progress: ability to trace interpreter (#63)
Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-12-16 17:50:59 -08:00
Anand Krishnamoorthi
73ee18f002 More OPA conformant semantics (#62)
Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-12-11 09:00:06 -08:00
Anand Krishnamoorthi
8a73b4bef9 Updated readme. Added bundle support. (#61)
Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-12-06 15:47:50 -08:00
Anand Krishnamoorthi
70bf371ebf crypto builtins (#57)
Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-12-06 07:59:04 -08:00
Anand Krishnamoorthi
ab968c2386 Regex and Glob builtins (#56)
Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-12-03 12:08:17 -08:00
Anand Krishnamoorthi
ed3492fd7b Formalize concept of a Number (#55)
Number is implemented using rust_decimal::Decimal which uses a 96 bit mantissa.
TODO:
  a) Support u64, i64 variants
  b) Determine desired semantics for floating-point
  c) Determine desired big integer length
  d) Explore other big int/big float crates

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-12-01 08:45:59 -08:00
Anand Krishnamoorthi
bb0ca29753 Lock down ACI tests and more OPA test folders (#54)
Borrowed from rego-cpp

Significantly (> 10 times) faster execution.

$ cargo test -r  --test aci
aci/mount_device                                  passed    9.597958ms
aci/mount_overlay                                 passed    10.159208ms
aci/scratch_mount                                 passed    8.598875ms
aci/create_container                              passed    10.237292ms
aci/shutdown_container                            passed    6.904084ms
aci/scratch_unmount                               passed    6.530875ms
aci/unmount_overlay                               passed    5.958875ms
aci/unmount_device                                passed    5.657834ms
aci/load_fragment                                 passed    6.049917ms

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-12-01 08:32:11 -08:00
Anand Krishnamoorthi
639ba72c90 Fix scheduling regression (#53)
Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-11-27 10:02:37 -08:00
eric-therond
ef36d9bed5 add full api to engine (#50)
Signed-off-by: eric-therond <eric.therond.fr@gmail.com>
2023-11-27 09:08:40 -08:00
Anand Krishnamoorthi
3514594c56 Use Rc<str> instead of string. (#52)
This reduces std::mem::size_of::<Value>() to 16 bytes.
String values are also efficiently copied like Objects, arrays, sets etc.

When multiple function rules are evaluated, ensure that constant argument values
match before running the rule. If actual parameter does not match the constant parameter,
then the rule is skipped.

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-11-24 17:45:49 -08:00
Anand Krishnamoorthi
4db2270dcf More library functions (#51)
- units.parse, units.parse_bytes
- json.is_valid, json.marshal, json.unmarshal
- yaml.is_valid, yaml.marshal, yaml.unmarshal
- object.subset
- set_diff

* Also print number of errors due to each missing function
* Also lock down fully passing OPA suites

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-11-22 14:19:23 -08:00
Luong-Minh Nguyen
e838d5af65 Added semver.is_valid and semver.compare (#49)
* Completed semver.is_valid and semver.compare
* Addressed PR comments

Signed-off-by: Minh Nguyen <luonguyen@microsoft.com>

---------

Signed-off-by: Minh Nguyen <luonguyen@microsoft.com>
Co-authored-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-11-16 16:05:02 -08:00
Anand Krishnamoorthi
a75f7c51df OPA conformance tests (#45)
Run regorus against OPA test suite.

To run full test suite:
  cargo test --test opa

To run specific folder (e.g semver):
  cargo test --test opa -- semver

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-11-15 17:09:53 -08:00
Anand Krishnamoorthi
72070a7061 Avoid dependency on `source lifetime. (#43)
This allows holding onto objects, caching results etc easily.
However it does introduce the overhead of ref counting.

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-11-15 09:25:12 -08:00
Anand Krishnamoorthi
a1d0f8576a Allow with modifier for builtin and user functions (#42)
Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-11-12 21:39:59 -08:00
Anand Krishnamoorthi
1c492144b3 Special cases of refs to data (#41)
1. Numeric indices will be converted to strings for refs beginning with `data`
   if there is not valid numeric key
2. Error out if input document already contains value for a ref

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-11-11 00:15:56 -08:00
Anand Krishnamoorthi
02c6c6b827 Fix scheduling statements that don't create bindings (#40)
Ensure that definitions are added even if no var is being defined.

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-11-09 16:24:43 -08:00
Anand Krishnamoorthi
0af8b6ea12 Ability to run the OPA testsuite (#39)
`OPA_TESTS_DIR=path/to/testsuite cargo test opa -- --shot-output` to run opa tests.
Failing test cases are saved in target/opa/folder for investigation.

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-11-09 15:11:33 -08:00
Anand Krishnamoorthi
d69b413c8e Engine (#38)
- Avoid lifetime parameter for Source, Span. Use Rc instead.
- Engine for simplified API

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-11-07 23:15:28 -08:00
Anand Krishnamoorthi
a751cd69e1 Use Ref for storing ast nodes in collections. (#37)
cmp, eq is done as pointer comparison which makes retrieval super fast.

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-11-05 22:32:00 -08:00
Anand Krishnamoorthi
d3fd0a3a78 all, any deprecated functions (#35)
Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-11-04 13:22:40 -07:00
Anand Krishnamoorthi
9ca55bcdf8 all, any deprecated functions (#34)
Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-11-02 20:27:40 -07:00
Anand Krishnamoorthi
c53d002347 Improvements (#33)
1. Skip recording undefined variables
2. Parse `in` correctly if it is not imported.
3. base64.decode
4. Handle `with` modifier for qualified data and input.

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-11-01 21:57:06 -07:00
Anand Krishnamoorthi
6228eaab4a Order query expression results (#32)
Expressions are scheduled based on dependencies and thusthe gathered
expression values may not be in the same order as in source.

Reorder to match the source.

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-10-31 13:26:33 -07:00
Anand Krishnamoorthi
519cce5b33 Scheduling of statements in user queries (#31)
Nested queries (comprehensions) are handled correctly.

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-10-31 10:52:51 -07:00
Anand Krishnamoorthi
7a3d5e7e02 eval, lex, parse commands (#30)
Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-10-30 21:45:43 -07:00
Anand Krishnamoorthi
63ecc44a48 eval_user_query for OPA style results (#29)
Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-10-30 10:44:02 -07:00
Anand Krishnamoorthi
800e594d52 Arity for builtins (#28)
Old-style function call

Utility for running opa yaml tests

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-10-24 10:11:50 -07:00
Anand Krishnamoorthi
951bb6b14c Handle chained _ (#27)
Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-10-17 19:32:42 -07:00
Anand Krishnamoorthi
2436467fbd Minimize PR 22 (#26)
* specific functions added to eval different rego components

Signed-off-by: eric-therond <eric.therond.fr@gmail.com>

* allow multiple inputs and results

Signed-off-by: eric-therond <eric.therond.fr@gmail.com>

* prepare_for_eval is necessary to be called

Signed-off-by: eric-therond <eric.therond.fr@gmail.com>

* test with the suggested code examples and clean scopes

Signed-off-by: eric-therond <eric.therond.fr@gmail.com>

* try to refactor first steps of evaluations

Signed-off-by: eric-therond <eric.therond.fr@gmail.com>

* improve coverage and fix clean state internal evaluation

Signed-off-by: eric-therond <eric.therond.fr@gmail.com>

* add getters and setters and fix clean function

Signed-off-by: eric-therond <eric.therond.fr@gmail.com>

* Tests are single input by default. Multi input specified via "many!" marker.

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>

---------

Signed-off-by: eric-therond <eric.therond.fr@gmail.com>
Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
Co-authored-by: eric-therond <eric.therond.fr@gmail.com>
2023-10-09 14:27:31 -07:00
eric-therond
2ba718ba72 improve errors location (#23)
Signed-off-by: eric-therond <eric.therond.fr@gmail.com>
2023-09-27 09:41:30 -07:00
Anand Krishnamoorthi
b7420aceba Fix clippy warning (#25)
Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-09-27 08:40:36 -07:00
eric-therond
8a9652b8d2 negation of an undefined value should return true (#21)
Signed-off-by: eric-therond <eric.therond.fr@gmail.com>
2023-06-29 09:44:19 -07:00
Anand Krishnamoorthi
796da46ae8 Ensure that scopes are cleaned up correctly upon error. (#20)
When evaluating rules, upon error the last pushed scope wasn't being
popped from the stack of scopes. This causes incorrect behavior
when there are multiple definitions for the same rule name.

The fix is to make sure that the scopes are popped manually upon encountering errors.

Once the interpreter logic is locked down, then we need to clean up scope management
using Drop functions so that the cleanup happens even during short circuited return.

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
2023-06-23 08:57:47 -07:00
eric-therond
e08d13df53 fix bitwise.and and add tests (#19)
Signed-off-by: eric-therond <eric.therond.fr@gmail.com>
2023-06-22 11:27:58 -07:00
eric-therond
910ef32ffb support of or-functions (#18)
* support of multiple definitions of the same function
* handle functions producing an error

Signed-off-by: eric-therond <eric.therond.fr@gmail.com>
2023-06-16 09:30:17 -07:00