confused of paresr callback's behavior of null object: remove or discarded?

[nlohmann]

Discussed in #4336

^{Originally posted by gino0717 April 8, 2024}
hi sir,

I'm studying the document of Parser Callbacks

I replace some of the values to null of the example , then expect the parser would remove the object if the value is null.

the input goes like :

{
        "Image": {
            "Width":  null,
            "Height": 600,
            "Title":  "View from 15th Floor",
            "Thumbnail": {
                "Url":    "http://www.example.com/image/481989943",
                "Height": 125,
                "Width":  null
            },
            "Animated" : false,
            "IDs": [116, 943, 234, 38793]
        }
}

Both "Width" under "Image" and "Thumbnail" are null.

in version 2.1.1 which directly installed from apt-get gives the result just as I expected, both the "Width" disappear due to the null value:

{
    "Image": {
        "Animated": false,
        "Height": 600,
        "IDs": [
            116,
            943,
            234,
            38793
        ],
        "Thumbnail": {
            "Height": 125,
            "Url": "http://www.example.com/image/481989943"
        },
        "Title": "View from 15th Floor"
    }
}

but in version 3.10.5 it gives the result like:

{
    "Image": {
        "Animated": false,
        "Height": 600,
        "IDs": [
            116,
            943,
            234,
            38793
        ],
        "Thumbnail": {
            "Height": 125,
            "Url": "http://www.example.com/image/481989943",
            "Width": <discarded>
        },
        "Title": "View from 15th Floor"
    }
}

the second "Width" in Thumbnail becomes "discarded" , and the "discarded" itself cannot be parsed if I feed the filtered json back to parser by :
json parse_again=json::parse(j_filtered.dump());

it gives

terminate called after throwing an instance of 'nlohmann::detail::parse_error'
what(): [json.exception.parse_error.101] parse error at line 1, column 148: syntax error while parsing value - invalid literal; last read: '"Width":<'

When does the parser give the null object a "discarded" value or just remove the object if filtered by the parser callback? Could I chose which behavior it acts?

here's my original code:

#include <iostream>
#include <iomanip>
#include <nlohmann/json.hpp>

using json = nlohmann::json;

int main()
{
    // a JSON text
    auto text = R"(
    {
        "Image": {
            "Width":  null,
            "Height": 600,
            "Title":  "View from 15th Floor",
            "Thumbnail": {
                "Url":    "http://www.example.com/image/481989943",
                "Height": 125,
                "Width":  null
            },
            "Animated" : false,
            "IDs": [116, 943, 234, 38793]
        }
    }
    )";

    // parse and serialize JSON
    json j_complete = json::parse(text);
    std::cout << std::setw(4) << j_complete << "\n\n";

    // define parser callback
    json::parser_callback_t cb = [](int depth, json::parse_event_t event, json & parsed)
    {
        // skip object elements with key "Thumbnail"
       // if (event == json::parse_event_t::key and parsed == json("Thumbnail"))
         if (event == nlohmann::json::parse_event_t::value and parsed.is_null())
        {
            return false;
        }
        else
        {
            return true;
        }
    };

    // parse (with callback) and serialize JSON
    json j_filtered = json::parse(text, cb);
    std::cout << std::setw(4) << j_filtered << '\n';
    
    json parse_again=json::parse(j_filtered.dump());
}

[nlohmann]

This indeed sounds like a bug. The documentation states

Discarded values in structured types are skipped. That is, the parser will behave as if the discarded value was never read.

This is true for arrays. For objects, we indeed keep the key and leave discarded as value.

I am torn whether fixing this could be a breaking change, so I added the "please discuss" label.