Use timestap upload instead of semver extracted from filename

[TamimiGitHub]

Overview

Is there a way to version objects based on uploaded timestamp? In other words, avoid versioning objects by pattern-matching filenames to identify version numbers. The use case to this is for situations where S3 content does not follow a strict semver policy for file names. In this scenario, the pipeline is required to be triggered on each S3 upload by taking the latest file (using timestamp as the check behaviour to order every fetch from S3 upon a resource check step).

Use Case

A front-end web application is built with a drag and drop functionality for file upload. On every upload, the pipeline is to be triggered to download and unpack the latest zip folder, and run a set of jobs on the file content producing a report at the end of the pipeline.

Issue

The current behaviour of this resource depends on the filename containing a semver. That is not usually the case.

Potential solution

Have a flag use_timestamp so that when set to true, the resource will use upload time to extract the content (based on a regexp). So something along the lines of
extractions := versions.GetBucketUploadedFiles(command.s3client, request.Source)

Note

If this is not feasible, what is the reason for it? To my understanding of concourse resource types the chronological order for an S3 bucket file upload could be upload timestamp

[vito]

Sounds like an interesting use case.

Are there any rules at all about the filenames? Would it make sense to still configure regexp:, and just add something like the github-release resource's order_by param?:

https://github.com/concourse/github-release-resource#source-configuration

This could be used like so:

source:
  regexp: uploaded/.*
  order_by: time

I guess if there are no rules for the filenames you could just specify .*.

I like this because it builds on the existing regexp and just adds an order_by, which would default to version for today's behavior.

I'd be happy to review a PR for the above behavior if it makes sense to you!

[TamimiGitHub]

Good to know there is an order_by source configuration for the github-release resource! I believe this implementation will solve the issue in place.

And no there are no rules for the filenames, since all files uploaded on the bucket are zip folders so I used (.*).zip as the regexp pattern but I believe a simple .* will also suffice.

And you mean today's behavior of version is based on semver extracted from file name eh? I'll take a look at how the github-release resource implemented the order by time behavior and if we can extract timestamp from the s3 object properties.

[vito]

And no there are no rules for the filenames, since all files uploaded on the bucket are zip folders so I used (.*).zip as the regexp pattern but I believe a simple .* will also suffice.

Cool, so it sounds like this would mesh well with regexp.

And you mean today's behavior of version is based on semver extracted from file name eh? I'll take a look at how the github-release resource implemented the order by time behavior and if we can extract timestamp from the s3 object properties.

Yep! Sounds good. 👍