Experiment status background process revision

[LuiggiTenorioK]

As a follow-up to the status revision that has to be done in Autosubmit in BSC-ES/autosubmit#2947, an update has to be done in the API to better orchestrate the status tracking process.

To better understand what needs to be done, here is a summary of how the API processes all the statuses:

• API gets all the entries of experiment and experiment_status
• For each experiment:
  • Get the modified time of the pkl file (for 4.2.0 this will be replaced by the max time in the modified column of the jobs table)
  • If the retrieved time is between the last 10 minutes, then it will be considered as RUNNING
  • Else if the retrieved time is between the last hour, it will try to look for the modified time of the *_run.log files to see if there is one with the modified time between the last 5 minutes.
  • Else it is set as NOT RUNNING

@ntorqulu to avoid overwriting the DELETED/ARCHIVED statuses from BSC-ES/autosubmit#2980, your idea to first filter the RUNNING (and NOT RUNNING) experiments seems great.

However, now we also have to discuss:

• how to replace the *_run.log files "hearthbeat" to rely less on the file system and more on the DB
• avoid race conditions where the API can overwrite statuses set by Autosubmit (e.g. double check in the API before update for possible outdated statuses)

[ntorqulu]

Thanks for the summary!
I reuse this issue to write all the things that should be considered while solving this:

• API currently overwrites the modified column of table experiment_status with the local timezone, not the UTC one.
  Backend writes in this table following UTC convention, but API overwrites it with the local timezone. Should follow the same standard as the backend

[LuiggiTenorioK]

• API currently overwrites the modified column of table experiment_status with the local timezone, not the UTC one.
  Backend writes in this table following UTC convention, but API overwrites it with the local timezone. Should follow the same standard as the backend

In Autosubmit, the next version (4.1.17) will use the local timezone because of the change I recently made here: BSC-ES/autosubmit#2943. And yes, we need to update the API so it is also time zone aware.

[ntorqulu]

Draft plan for Backend/API experiment status refactor (to discuss, just to have a first idea):

Updated 20260512

• Backend is the main responsible of writting into the experiment_status table
• Backend writes statuses: RUNNING, NOT RUNNING, ARCHIVED and DELETED
• Backend keeps updating last_heartbeat timestamp via a daemon thread every 2 min while experiment is running
• API never writes on experiments flagged as ARCHIVED, DELETED or NOT RUNNING
• API can only evaluate RUNNING experiments to correct backend errors (kill signals stopping the running experiment without gracefull termination).

Allowed experiment states for this issue: RUNNING, NOT RUNNING, ARCHIVED, DELETED
Allowed transitions:

• RUNNING <-> NOT RUNNING
• NOT RUNNING <-> ARCHIVED
• NOT RUNNING -> DELETED
  Forbidden transitions:
• DELETED -> any
• RUNNING -> ARCHIVED , DELETED
• ARCHIVED <-> DELETED

Replace API files heartbeat with DB heartbeat

• experiment_status table now has column last_heartbeat, updated by the backend via a daemon thread for each running experiment
• backend updates this timestamp every 2 minutes while experiment state is RUNNING
• backend sets status RUNNING when run starts and sets status to NOT RUNNING when run: ends (completed), is stopped, or some of the jobs failed (finished)
• API uses heartbeat age to determine if the experiment is RUNNING only for RUNNING and empty status experiments. A last_heartbeat is considered to be stale if current_time - last_heartbeat > 150 min (2.5 min).
  PRIORITY 1: Check heartbeat from the db. If the experiment has last_heartbeat, compute the heartbeat age. If heartbeat age is lower than 150 (2.5 min) and the status is RUNNING, then experiment is RUNNING. Do nothing. If status is RUNNING and heartbeat age is higher or equal than 150 minutes then experiment is NOT RUNNING.
  PRIORITY 2: If last_heartbeat not present, check pickle file age (as done before by the api as a safeguard) against 10 min threshold.
  PRIORITY 3: As a last resort, exhaustive check using log files (as done before by the api as a safeguard).

API only updates the status row if the status changes.

[ntorqulu]

PROBLEMS:

• What happens if Backend and API want to overwrite the same experiment status?
  E.g. Experiment was RUNNING, finishes execution of all his jobs, backend sets it as NOT RUNNING but with a fresh last_heartbeat.
  API sees the fresh heartbeat and checks if heartbeat age < 2 min. The first time it checks it, it could be TRUE. And API sets it to RUNNING again. 5 min later, when it checks again the experiment last_heartbeat, it sets it up to NOT RUNNING.
  I think that API should not check the NOT RUNNING statuses, only the RUNNING ones.

[LuiggiTenorioK]

PROBLEMS:

• What happens if Backend and API want to overwrite the same experiment status?
  E.g. Experiment was RUNNING, finishes execution of all his jobs, backend sets it as NOT RUNNING but with a fresh last_heartbeat.
  API sees the fresh heartbeat and checks if heartbeat age < 2 min. The first time it checks it, it could be TRUE. And API sets it to RUNNING again. 5 min later, when it checks again the experiment last_heartbeat, it sets it up to NOT RUNNING.
  I think that API should not check the NOT RUNNING statuses, only the RUNNING ones.

I was thinking of this issue too. I think that, if there is a heartbeat, the API should not be allowed to transition from NOT RUNNING to RUNNING . Then, it is only allowed to set it as NOT RUNNING.

Also, setting the heartbeat frequency to 2 min seems good. It should be less than half of the frequency the API checks for it (5 min).

[LuiggiTenorioK]

In Autosubmit, the next version (4.1.17) will use the local timezone because of the change I recently made here: BSC-ES/autosubmit#2943. And yes, we need to update the API so it is also time zone aware.

This has been solved in the API by #283