Skip to content

feat: format (pretty-print) STAC files TDE-759 #172

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
paulfouquet wants to merge 5 commits into
base: master
Choose a base branch
from
Draft

feat: format (pretty-print) STAC files TDE-759 #172

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d1d4df9
feat: format (pretty-print) STAC files TDE-759
paulfouquet Sep 12, 2023
47343a9
refactor: update format-json command call
paulfouquet Sep 14, 2023
bd4ce0d
fix: command is pretty-print
paulfouquet Sep 27, 2023
6e2e3fc
Merge branch 'master' into feat/stac-format-tde-759
paulfouquet Oct 10, 2023
c416047
Merge branch 'master' into feat/stac-format-tde-759
paulfouquet Oct 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions templates/argo-tasks/format.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: tpl-at-format
Copy link
Contributor

@l0b0 l0b0 Nov 2, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's "at" in this? "Argo task"? Would including "stac" somewhere be clearer, such as "tpl-at-format-stac"?

Also, other templates have names like "create-collection". Is "tpl-at-" redundant?

spec:
templateDefaults:
container:
imagePullPolicy: Always
entrypoint: main
templates:
- name: main
inputs:
parameters:
- name: source
description: Path to format files from
- name: target
description: (Optional) Target directory (testing)
Copy link
Contributor

@l0b0 l0b0 Nov 2, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is empty, does it format files in-place? As a newbie, that might be useful clarification. If we are relying on third party functionality, it might be OK to just link to the (appropriate version of the) reference somewhere.

default: ""
- name: fix-content-type
description: (Optional) Set the content-type to S3 object
default: ""
Copy link
Contributor

@l0b0 l0b0 Nov 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are valid values here? Are these MIME types, as used by the HTTP Content-Type header? What happens if this is empty?

- name: version
description: argo-task Container version to use
default: "v2"

container:
image: "019359803926.dkr.ecr.ap-southeast-2.amazonaws.com/eks:argo-tasks-{{= inputs.parameters.version }}"
command: [node, /app/index.js]
env:
- name: AWS_ROLE_CONFIG_PATH
value: s3://linz-bucket-config/config.json
args:
- "pretty-print"
- "{{= inputs.parameters.source }}"
- "{{= sprig.empty(inputs.parameters.target) ? '' : '--target=' + inputs.parameters.target }}"
- "{{= sprig.empty(inputs.parameters['fix-content-type']) ? '' : '--fix-content-type=' + inputs.parameters['fix-content-type'] }}"
2 changes: 1 addition & 1 deletion workflows/imagery/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ graph TD;
tileindex-validate-->standardise-validate;
standardise-validate-->create-collection;
standardise-validate-->create-overview;
create-collection-->stac-validate;
create-collection-->stac-format-->stac-validate;
create-overview-->create-config;
```

Expand Down
14 changes: 12 additions & 2 deletions workflows/imagery/standardising.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -282,6 +282,16 @@ spec:
value: '{{tasks.get-location.outputs.parameters.location}}'
depends: 'standardise-validate'

- name: stac-format
templateRef:
name: tpl-at-format
template: main
arguments:
parameters:
- name: source
value: "{{tasks.get-location.outputs.parameters.location}}flat/"
depends: "create-collection"

- name: stac-validate
template: stac-validate
arguments:
Expand All @@ -291,8 +301,8 @@ spec:
artifacts:
- name: stac-result
raw:
data: '{{tasks.stac-validate.outputs.result}}'
depends: 'create-collection'
data: "{{tasks.stac-validate.outputs.result}}"
depends: "stac-format"

- name: get-location
template: get-location
Expand Down