Proposal - val and context replacements

[TotalTechGeek]

Background

Hey folks! I expect this to be one of the more difficult proposals.

Some previous discussions:

• https://github.com/orgs/json-logic/discussions/10 (Previous Value Proposal)
• https://github.com/orgs/json-logic/discussions/1#discussioncomment-11573464

And some commentary here:

• https://github.com/orgs/json-logic/discussions/16#discussioncomment-11621365

Getting to the point: var, missing and missing_some are controversial operators.

They are controversial because:

• They embed a pathing format into a string within the operator, and this string format is not robust enough to accommodate a handful of keys.
• Some folks believe it might be best to switch to a more common / well-thought out format, like RFC6901
• Embedding the format into the string has been called into question, as JSON Logic is an ast-like setup, which means it might be best to embed the path literally.
• It's difficult to retrofit var with improvements without reinventing RFC6901.

Proposal

• Deprecate var, missing and missing_some into the "Legacy" extension for JSON Logic. This means excluding them from the JSON Logic Core RFC. As a community extension, they still appear in Compat Tables.
• Introduce replacement operators, namely val

val could be an operator where you explicitly define the path in the arguments, making it more AST-like.

By doing this, you could more easily chain things to val, like an rfc6901 custom operator,
{ "val": { "rfc6901": "/hello/world" } }

Another operator we might introduce is exists, which receives the same arguments as val, but returns a boolean

One more operator we'd probably want to introduce is ?? or "coalesce", which is similar to "or", but performs a nullish check rather than a truthiness check. Very similar to https://www.geeksforgeeks.org/postgresql-coalesce/

Arguments

• number / string are used as pathing arguments
• array is something I'd like to leave as placeholder, likely for scope control.
• null / boolean / object are defined as error behavior. I'd like to discuss how we test for errors in another proposal.

Test Cases / Showing how the Operators Work

val

[
    "Test Specification for val",
    {
        "description": "Fetches a value",
        "rule": { "val": "hello" },
        "data": { "hello" : 0 },
        "result": 0
    },
    {
        "description": "Fetches a nested value",
        "rule": { "val": ["hello", "world"] },
        "data": { "hello" : { "world": 1 } },
        "result": 1
    },
    {
        "description": "Fetches a value from an empty key",
        "rule": { "val": "" },
        "data": { "" : 1 },
        "result": 1
    },
    {
        "description": "Fetches a value from a nested empty key",
        "rule": { "val": ["", ""] },
        "data": { "" : { "": 2 } },
        "result": 2
    },
    {
        "description": "Fetches a value from an array",
        "rule": { "val": [1] },
        "data": [1, 2],
        "result": 2
    },
    {
        "description": "Fetches a value from an array in an object",
        "rule": { "val": ["arr", 1] },
        "data": { "arr": [1, 2] },
        "result": 2
    },
    {
        "description": "Fetches a value from a doubly nested empty key",
        "rule": { "val": ["", "", ""] },
        "data": { "" : { "": { "": 3 } } },
        "result": 3
    },
    {
        "description": "Fetches a value from a key that is purely a dot",
        "rule": { "val": "." },
        "data": { "." : 20 },
        "result": 20
    },
    {
        "description": "Fetching a value from null returns null",
        "rule": { "val": "hello" },
        "data": { "hello" : null },
        "result": null
    },
    {
        "description": "Fetching a value from a null fetched value returns null",
        "rule": { "val": ["hello", "world"] },
        "data": { "hello" : null },
        "result": null
    },
    {
        "description": "Fetches the entire context",
        "rule": { "val": [] },
        "data": { "": 21 },
        "result": { "": 21 }
    },
    {
        "description": "Fetches the entire context for a nested key",
        "rule": { "val": "" },
        "data": { "": { "": 22 } },
        "result": { "": 22 }
    },
    {
        "description": "Using val in a map",
        "rule": { "map": [[1,2,3], { "+": [{ "val": [] }, 1] }] },
        "data": null,
        "result": [2,3,4]
    }
]

exists

[
    "Test Specification for exists",
    {
        "description": "Checks if a normal key exists",
        "rule": { "exists": "hello" },
        "data": { "hello" : 1 },
        "result": true
    },
    {
        "description": "Checks if a normal key exists (array)",
        "rule": { "exists": ["hello"] },
        "data": { "hello" : 1 },
        "result": true
    },
    {
        "description": "Checks if a normal key exists (false)",
        "rule": { "exists": "hello" },
        "data": { "world" : 1 },
        "result": false
    },
    {
        "description": "Checks if an empty key exists (true)",
        "rule": { "exists": [""] },
        "data": { "" : 1 },
        "result": true
    },
    {
        "description": "Checks if an empty key exists (false)",
        "rule": { "exists": [""] },
        "data": { "hello" : 1 },
        "result": false
    },
    {
        "description": "Checks if a nested key exists",
        "rule": { "exists": ["hello", "world"] },
        "data": { "hello" : { "world": false } },
        "result": true
    },
    {
        "description": "Checks if a nested key exists (false)",
        "rule": { "exists": ["hello", "world"] },
        "data": { "hello" : { "x": false } },
        "result": false
    },
    {
        "description": "Checks if a null value exists",
        "rule": { "exists": "hello" },
        "data": { "hello" : null },
        "result": true
    }
]

??

[
    "Test Specification for ??",
    {
        "description": "Coalesces a string alone",
        "rule": { "??": ["hello"] },
        "data": null,
        "result": "hello"
    },
    {
        "description": "Coalesces a number alone",
        "rule": { "??": [1] },
        "data": null,
        "result": 1
    },
    {
        "description": "Coalesces a boolean alone",
        "rule": { "??": [true] },
        "data": null,
        "result": true
    },
    {
        "description": "Coalesces an object from context alone",
        "rule": { "??": [{ "val": "person" }]},
        "data": { "person": { "name": "John" } },
        "result": { "name": "John" }
    },
    {
        "description": "Empty behavior",
        "rule": { "??": [] },
        "data": null,
        "result": null
    },
    {
        "description": "Coalesces a string with nulls before",
        "rule": { "??": [null, "hello"] },
        "data": null,
        "result": "hello"
    },
    {
        "description": "Coalesces a string with multiple nulls before",
        "rule": { "??": [null, null, null, "hello"] },
        "data": null,
        "result": "hello"
    },
    {
        "description": "Coalesces a string with nulls after",
        "rule": { "??": ["hello", null] },
        "data": null,
        "result": "hello"
    },
    {
        "description": "Coalesces a string with nulls both before and after",
        "rule": { "??": [null, "hello", null] },
        "data": null,
        "result": "hello"
    },
    {
        "description": "Coalesces a number with nulls both before and after",
        "rule": { "??": [null, 1, null] },
        "data": null,
        "result": 1
    },
    {
        "description": "Uses the first non-null value",
        "rule": { "??": [null, 1, "hello"] },
        "data": null,
        "result": 1
    },
    {
        "description": "Uses the first non-null value, even if it is false",
        "rule": { "??": [null, false, "hello"] },
        "data": null,
        "result": false
    },
    {
        "description": "Uses the first non-null value from context",
        "rule": { "??": [{ "val": ["person", "name"] }, { "val": "name" }, "Unknown Name"] },
        "data": { "person": { "name": "John" }, "name": "Jane" },
        "result": "John"
    },
    {
        "description": "Uses the first non-null value from context (with person undefined)",
        "rule": { "??": [{ "val": ["person", "name"] }, { "val": "name" }, "Unknown Name"] },
        "data": { "name": "Jane" },
        "result": "Jane"
    },
    {
        "description": "Uses the first non-null value from context (without any context)",
        "rule": { "??": [{ "val": ["person", "name"] }, { "val": "name" }, "Unknown Name"] },
        "data": {},
        "result": "Unknown Name"
    }
]

Probable Objections

• Why not make val use RFC6901?

This... is a reasonable alternative. I assume this will be debated, but I like the AST-like nature this enables here, and I like how this allows us to avoid embedding a string parsing spec.

Out of Scope

Scopes

Scopes are out of scope for this proposal. I have experimented with a syntax for this,

{ "val": [[2], "hello"] }

This would ascend two scopes to get "hello", but this is not part of this specific proposal.

[TotalTechGeek]

I'm flagging this as "draft";

I'd like to request input on exists, and define additional test cases before getting approval.

[toddbaert]

Perhaps a test that dictates behavior of exists if the key is defined but its value is null:

{
        "description": "Checks for null values",
        "rule": { "exists": ["hello"] },
        "data": { "hello" : null },
        "result": true // or false, I'm not sure
}

TBH I don't know what the expected result is here... are we talking about the existence of the key or the value the key points at?

The existing missing operator includes the key in it's returned array if the key is present but it's value is null, at least in the playground.

I guess I can see arguments for either, but I think one should be specified unless there's some very clear existing reason one or the other is the obvious choice.

[TotalTechGeek]

I think exists would just be a way to check if the key exists as a property on context, without concern for truthiness / nullishness.

So in your test case, the result would be true

You could already do { "if": [{ "val": "hello"}, ...] } or { "!!": { "val": "hello"} } for the other behavior.

---

Out of proposal note -
{ "filter": [["hello"], { "exists": [[2], { val: [] }] }] }

Would allow you to replicate the behavior of missing with exists if we add scope traversal. However, this construct is a bit verbose.

Back in proposal -

It makes me wonder:

• Should a replacement operator work in the same way?
• Are folks using missing in a way that exists wouldn't suffice / work better?

My "gripe" with missing is that it's kind of a complex operator with a seemingly niche use-case. missing_some even moreso. It's the kind of operation that seems like it'd be best represented with two (I guess in the above case, three) simple operators. Hrm...

[toddbaert]

I think exists would just be a way to check if the key exists as a property on context, without concern for truthiness / nullishness.

I think so too, but I think that should be explicitly specified and then we should add a test like the one above, otherwise there's some ambiguity here, and especially considering how missing works, a naive implementation might check the nullishness (or even worse the falseness) of a value at the specified key, instead of checking for the key's existence.

[TotalTechGeek]

Revised with that specific check.

[dslmeinte]

Couple of remarks w.r.t. val:

• The test cases start with the more exotic ones.
• Why having multiple ways to fetch the whole context: [], null, [null]?
• Test case 1 is essentially the same as the one labeled “Fetches the entire context for a nested key” (Create leaf-coercion-proposal.json #8).

[TotalTechGeek]

The test cases start with the more exotic ones.

Yep; need to revise, I copied this test suite from the other proposal, which was meant to resolve empty keys.

null, [null]

Originally, I was expecting { "val": null } to map to the entire context, but { "val": [] } makes more sense.

I wanted to see if we wanted null to:

• Act as a no-op
• Produce an error
• Some other behavior

Given standard parsing rules of JSON Logic though, null is the same as [null].

[dslmeinte]

Originally, I was expecting { "val": null } to map to the entire context, but { "val": [] } makes more sense.

Absolutely!

Given standard parsing rules of JSON Logic though, null is the same as [null].

I'd say that's all the more reason to not introduce aliases/alternatives.

I wanted to see if we wanted null to:

* Act as a no-op

* Produce an error

* Some other behavior

I'd prefer option 2.

[TotalTechGeek]

I'm going to go ahead and drop the null no-op behavior then. Just makes more sense.

[TotalTechGeek]

A note, potentially for another proposal 😬.

It also seems likely to me that if we introduce val, we'll probably want to discuss introducing some sort of ?? or default or coalesce operator that enables nullish coalescing; or may not always work as desired.

I know we're aiming to keep operator introductions low, but if we're deprecating 3 operators... 😅

[toddbaert]

That would not be fundamentally different than simply using the exists you propose in a condition which then either uses val to extract the same value, or else uses some default, right? Are you just proposing a "syntactical sugar" operator that would do the same? If so I'm reluctant by default, but I could be convinced.

[TotalTechGeek]

default or ?? or coalesce would be roughly equivalent to

Basically, this function is designed to evaluate a list of arguments and return the first non-null argument it encounters.

{ "if": [{ "==": [A, null], B, A] }

That way

{ "var": ["hello.world", 17] } could cleanly be mapped to { "??": [{ "val":["hello", "world"]},17]}

?? doesn't interact with context directly, so it could be used to coerce the output of other operators.

Basically - https://www.geeksforgeeks.org/postgresql-coalesce/

[toddbaert]

Right - I think I understand, but the thing I was trying to confirm was that this wouldn't unlock any new functionality, it'd just be a shortcut... basically:

{ "??": [{ "val":["hello"]}, 17]} is the same as

{
  "if": [
    {
      "==": [
        {
          "val": [
            "hello"
          ]
        },
        null
      ]
    },
    17,
    {
      "val": [
        "hello"
      ]
    }
  ]
}

What I was missing was that previously, var supported defaults with the second arg (I'd forgot that), so I guess you're thinking that a ?? op would make migrations much easier.

[TotalTechGeek]

That is correct 😄

It's also a bit more efficient than the remap, because if any of the arguments was some calculation, it wouldn't need to re-evaluate the expression in both the conditional and the branch.

It'd also be variadic, which would mean you could set up multiple fallbacks, like

{ "??": [{ "val": "a" }, { "val": "b" }, 17] }

[TotalTechGeek]

Hey folks, I've revised the proposal a bit. At the moment, I've included all three operators:

• val
• exists
• ?? (Coalesce)

We could choose to separate ?? into a separate proposal, or rename it from ?? to coalesce, or alias it. I have a general preference for avoiding aliases.

[codetiger]

For val or the current var, the operator is only able to get the value from data. Is there any operator that can access the value from result of another calculation. For example, I can do the below instead of ??
Rule:

{
  "filter": [
    {"var": "array"},
    {"!=": [{"var": ""}, null]}
  ]
}

Data

{"array": [null, "hello", null]}

and get the non-null array, but I'll not be able to access the first item.

[TotalTechGeek]

Are you asking about potentially exploring something like a get operator?

[codetiger]

Yes, just wondering if it has to be a new operator or val operator can have an option to feed the result of another operator, instead of using the data.

[TotalTechGeek]

JLE actually has a get operator, but we can explore if we want to add something like get in another proposal.

I'd like to keep new operator intros low, but currently my thoughts are:

• preserve
• get
• apply
• MAYBE pipe

Could be reasonable operators to add to core.

[TotalTechGeek]

I'm undrafting this, @json-logic/tc, @rmannibucau

Do y'all have any thoughts / objections?

I'm voting with a +1.
I'm assuming @codetiger is also giving this a +1.

This is a bit of a leap, so I'm going to try to aim for an aggregate score of 4 before proceeding (existing voting rules apply, but I'm hoping for decent consensus on this).

To clarify, community votes count towards the aggregate score. Community objections will also be noted, and consensus will be sought out.

This proposal is the result of a community objection.

[TotalTechGeek]

Of some note, I consider this proposal 3/4 that is kinda necessary for us to figure out before we move onto re-evaluating all of the operators and building a robust suite. See: https://github.com/orgs/json-logic/discussions/17#discussioncomment-11707219

We need to decide on what we're building our test-cases for context going forward on.

[TotalTechGeek]

One last point of clarification: With var, missing and missing_some being deprecated to a legacy extension, this does not mean that we expect that organization endorsed implementations will drop support for these operators any time in the near term.

They will be expected to be included in our recommended baseline for implementations, but will not become a part of "JSON Logic Core".

No one's existing logic is expected to be bricked by this change.

[codetiger]

+1

[gregsdennis]

Just so the RFC 6901 option doesn't live in isolation, maybe it's a good idea to include RFC 9535 (JSON Path) as another option.

[TotalTechGeek]

I think I might've created some confusion by showing rfc6901, with the assumption of implicit apply support when you chain operators (one of the upcoming proposals).

That wasn't meant to be one of the operators introduced, only val, ?? and exists, and I was demonstrating how a custom rfc6901 operator could be chained to val.

The "custom" operator would just parse the string and turn it into a path, so "/hello/world" would become ["hello", "world"], "" would become [].

Do we want to include RFC6901 / RFC9535 in core? Or would we prefer to make some of these pathing operators an extension?

[TotalTechGeek]

A follow up on RFC9535; val is currently defined to be a simple operator that returns a single value from a path.

So if there was a JSON Path operator, I don't think it could go through val (at least as its currently defined) to allow it to fetch multiple values. Perhaps that'd be better as a separate operator.

[gregsdennis]

Oh, I didn't read that you intended rfc6901 to be an operator but rather the containing object to be an optional argument to val. Having it as an extension operator does make sense.

[TotalTechGeek]

I believe this proposal has met the TC criteria to move forward; so I'm marking it as approved, but I'd like @toddbaert to weigh in and give the community some time to voice any objections.

TC Votes

• @TotalTechGeek (+1)
• @gregsdennis (+1)
• @codetiger (+1)

With var deprecated, and val approved, we can defining test cases using val.

While I'd like to settle one more proposal (deciding if we're going to support implicit application / operator chaining), I think this opens the door for us to begin the process of evaluating each existing operator.

[gregsdennis]

With var deprecated, and val approved, we can defining test cases using val.

We should retain the existing tests that utilize var, maybe moving them to a "legacy" folder/file.

[TotalTechGeek]

Absolutely!

And these tests will be included in the compat tables (which we will be beefing up to be more like "CanIUse").

[TotalTechGeek]

Important

While I failed to explicitly communicate it, the tests codify that JSON Logic uses zero-based indexing for val and exists operators.

This detail will be important to include in the RFC / formal specification we produce.