New S3 operator idea

[sowmyasrinath]

I was thinking to add operators .

1. S3: Operator that returns a list of files that are missing from S3 prefix when compared to another S3 prefix. Like {A-B}
2. S3: Similar to the above operator, have an operator that copies missing files from destination S3 prefix but are present in Source.

Want to get a feedback on both these ideas.

[SasanAhmadi]

Is the second operator is almost like a S3 sync operator?

[sowmyasrinath]

Yes, it would be S3 sync between two S3 buckets.

[vincbeck]

My opinion on that is these 2 examples are very specific to your needs. I dont think it's worth writing an operator for both of these use cases because they are too specific. I am maybe mistaken and feel free to challenge me but I think the 2 examples you gave are more meant to be solve through a DAG than an operator.