DHIS-21957 Skip data values for closed orgunits before dataValueSets POST

[KatherineWyers]

Problem

The outlier detection stage batches all data values for a dataset into a single dataValueSets POST. If any org unit in that batch is closed (has a closedDate or has an openingDate after the submitted period) DHIS2 correctly rejects the value. But since everything is posted together, the entire batch fails with a 409 Conflict, including valid data for org units that are open and valid data for org units that are closed in a later period but open during that period.

This is expected DHIS2 behaviour. Not a DHIS2 bug.

Jira: DHIS2-21957 https://dhis2.atlassian.net/browse/DHIS2-21957

Fix

Make a chunked call to the API to get the opened and closed dates for the org units.
Create helper functions to determine whether to keep or skip the orgunit:period combination.

should_keep_data_value(): compares a data value's period against an org unit's openingDate and closingDate.

get_organisation_unit_dates_bulk(): fetches the openingDate and closingDate for many org unites in one request. This is chunked, so it doesn't go above the API limit of DHIS2.

filter_and_fetch_closed_org_units(): Fetch the dates for the org units referenced in a batch and filters them by (orgUnit, period) pairs.

This fix is wired into _process_tasks() in cli.py right before posting. Orgunit:period pairs that are filtered out are logged individually and they are displayed to the user as 'org unit closed'.

What is unchanged

the 'deletes' and 'integrity_payloads' are unaffected. This fix only filters the outlier-stage for the 'upserts'. It is worth discussing whether deletes and integrity_payloads also need this same fix.

Testing

All new tests use mocked aiohttp sessions, so no real network or DHIS2 calls.
tests/test_org_unit_filter.py: Testing the boundary cases for should_keep_data_value.
tests/test_api_utils.py: Testing the chunking for get_organisation_unit_dates_bulk

[sonarqubecloud]

Quality Gate failed

Failed conditions
5 New issues
1 New Vulnerabilities (required ≤ 0)
4 New Code Smells (required ≤ 0)
C Security Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE