Set expires_at timestamp and tag output files on S3

[nozomione]

Context

Related Issue: #1910

Feature branch: feature/user-dataset-expiration

The portal features two dataset types:

• CCDLDatasets: per-processed datasets for all CCDL projects and metadata available for download (no expiration)
• UserDatasets: custom datasets created and processed by users (expire 7 days after processing)

Once UserDatasets expire, their corresponding computed files in S3 will be deleted, and the portal provides users an option to regenerate (re-process) expired datasets for download.

Problem or idea

Currently, there is no 7-day expiration set up for UserDatasets resulting in generated computed files remaining on S3 indefinitely. This affects the portal UI as the regeneration option will not be available (see the linked related issue for more context).

To address this, we need to implement the expiration feature using AWS's tag-based S3 Lifecycle Policy Rule, which enables automated deletion of expired datasets after 7 days. We also need to set up a cron job that marks UserDatasets to enable dataset regeneration on the portal after 7 day of processing.

As part of configuring the tag-based S3 Lifecycle Policy Rule for object expiration, we'll implement the following steps:

1. Populate Dataset.expires_at upon successful job completion: In the DatasetJobProcessor.on_run_done hook method, set the dataset expires_at timestamp (now + 7 days) before saving it. The expires_at should not populated for CCDLDatasets.

# DatasetJobProcessor

"""
on_run_done: Invoked after all steps have completed successfully.
"""

def on_run_done(self):
    self.job.save()
    self.job.dataset.save() 
    logger.info("Job completed.")

2. Tagging the generated computed file on S3: In the DatasetJobProcessor.upload_new_computed_file workflow step, after successful upload, add object tagging with a key-value pair of dataset-type=user. We'll add a new method to handle tagging in the s3 module (e.g., tag_user_dataset_output_file). This method should be skipped for CCDLDatasets, and raise an exception on tagging failure.

# DatasetJobProcessor.upload_new_computed_file step:

def upload_new_computed_file(self):
    s3.upload_output_file(
        self.job.dataset.computed_file.s3_key, self.job.dataset.computed_file.s3_bucket
    )

3. Add a test to verify the implementation works as expected:

• Assert Dataset.expires_at for UserDatasets when DatasetJobProcessor.on_run_done (mocked) runs
• Ensure the tagging is skipped for CCDLDatasets

Additionally, as part of validation, we'll spin up the dev stack and create one CCDLDataset and two UserDatasetes to locally confirm behavior.

Solution or next step

• Set the expires_at timestamp in DatasetJobProcessor.on_run_done
• Tag the generated computed files upon uploading to S3 during the DatasetJobProcessor.upload_new_computed_file step
• Add a new test for the Dataset.expires_at field and mock if DatasetJobProcessor.on_run_done is called
• Verify the implementation using the dev stack