Tasks: Outputs: Folders not created automatically. #5544

Grimeton · 2020-05-05T06:32:57Z

Grimeton
May 5, 2020

Summary

Tasks: Outputs: Folders not created automatically.

A brief summary of the problem.

When using outputs in a Task, then the folders of the outputs are not created automatically. The documentation states otherwise:

https://concourse-ci.org/tasks.html#schema.task.outputs

Steps to reproduce

Create a pipeline with a task that is using outputs.
Run the task with something like "ls -al"
See the folders are not there.

Starting from docker-compose up, what steps can be followed to reproduce
the problem?

???

Please use https://gist.github.com for large snippets!

Expected results

Folder's with the name of the output are created.

Actual results

Folder's with the name of the output are not created.

Additional context

None

Triaging info

Concourse version: 6.0.0
Browser (if applicable): Vivaldi
Did this used to work?
According to the documentation: Yes.

jamieklassen · 2020-05-05T12:16:55Z

jamieklassen
May 5, 2020
Collaborator

Unfortunately I couldn't reproduce your issue, but I think it will make a difference if you can provide two more details:

the exact pipeline YAML that you are using, or an anonymized version of it if you have some privacy concerns
a description of your concourse deployment configuration - bosh/k8s/docker/systemd, scale (if relevant), what flags are being provided to the web/worker processes

It also sounds like you had some confusion around the meaning of

Starting from docker-compose up, what steps can be followed to reproduce the problem?

So I'll give an especially-detailed example:

$ docker-compose -f <(cat <<EOF
version: '3'
services:
  concourse-db:
    image: postgres
    environment:
      POSTGRES_DB: concourse
      POSTGRES_PASSWORD: concourse_pass
      POSTGRES_USER: concourse_user
      PGDATA: /database
  concourse:
    image: concourse/concourse:6.0.0
    command: quickstart
    privileged: true
    depends_on: [concourse-db]
    ports: ["8080:8080"]
    environment:
      CONCOURSE_POSTGRES_HOST: concourse-db
      CONCOURSE_POSTGRES_USER: concourse_user
      CONCOURSE_POSTGRES_PASSWORD: concourse_pass
      CONCOURSE_POSTGRES_DATABASE: concourse
      CONCOURSE_EXTERNAL_URL: http://localhost:8080
      CONCOURSE_ADD_LOCAL_USER: test:test
      CONCOURSE_MAIN_TEAM_LOCAL_USER: test
      CONCOURSE_WORKER_BAGGAGECLAIM_DRIVER: overlay
      CONCOURSE_CLIENT_SECRET: Y29uY291cnNlLXdlYgo=
      CONCOURSE_TSA_CLIENT_SECRET: Y29uY291cnNlLXdvcmtlcgo=
EOF
) up -d

Wait for concourse to listen on port 8080
fly -t dev login -c http://localhost:8080 -u test -p test

$ fly -t dev set-pipeline -p pipeline -n -c <(cat <<EOF
jobs:
- name: task-with-output
  plan:
  - task: task
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: {repository: busybox}
      outputs:
      - name: output # <--- Here's the output
      run:
        path: ls
        args: ["-al"]
EOF
)

fly -t dev unpause-pipeline -p pipeline
fly -t dev trigger-job -w -j pipeline/task-with-output

When I follow the above steps, I get the following output at my console:

started pipeline/task-with-output #1

initializing
fetching busybox@sha256:a2490cec4484ee6c1068ba3a05f89934010c85242f736280b35343483b2264b6
e2334dd9fee4 [======================================] 743.0KiB/743.0KiB
running ls -al
total 12
drwxr-xr-x    3 root     root          4096 May  5 12:03 .
drwxr-xr-x    3 root     root          4096 May  5 12:03 ..
drwxr-xr-x    2 root     root          4096 May  5 12:03 output # <--- Here's the output
succeeded

I did my best to infer what you meant from the details you gave, but I wasn't able to reproduce the problem as I understood it. Hopefully we can get to the bottom of this together!

0 replies

vito · 2020-05-05T12:47:56Z

vito
May 5, 2020
Maintainer

Converting this to a Discussion until we can confirm there's a bug afoot.

0 replies

Grimeton · 2020-05-05T14:31:47Z

Grimeton
May 5, 2020
Author

Hello,

Unfortunately I couldn't reproduce your issue, but I think it will make a difference if you can provide two more details:

the exact pipeline YAML that you are using, or an anonymized version of it if you have some privacy concerns

---
resources:



  - name: res-CodeServer-Image-Devel
    type: git
    source:
      uri: ((gitea_checkout_url))
      branch: ((gitea_checkout_branch))
      submodule_credentials:
        - host: ((gitea_host))
          username: ((gitea_username))
          password: ((gitea_password))

  - name: res-cdr-code-server-master
    type: git
    source:
      uri: https://github.com/cdr/code-server.git
      branch: master

  - name: res-CodeServer-Image-Devel-Private-Repository
    type: docker-image
    source:
      ca_certs:
        - domain: ((docker_registry_private_ca_domain))
          cert: ((docker_registry_private_ca_certificate))
      repository: ((docker_registry_private_url))
      username: ((docker_registry_private_username))
      password: ((docker_registry_private_password))




  - name: res-CodeServer-Image-Devel-Version-Ubuntu-16.04
    type: semver
    source:
      branch: master
      driver: git
      initial_version: 0.0.0
      uri: ((version_uri))
      file: versions/ubuntu/16.04/devel/CodeServer-Image-Devel
      username: ((gitea_username))
      password: ((gitea_password))


  - name: res-CodeServer-Image-Devel-Version-Ubuntu-18.04
    type: semver
    source:
      branch: master
      driver: git
      initial_version: 0.0.0
      uri: ((version_uri))
      file: versions/ubuntu/18.04/devel/CodeServer-Image-Devel
      username: ((gitea_username))
      password: ((gitea_password))


  - name: res-CodeServer-Image-Devel-Version-Ubuntu-19.10
    type: semver
    source:
      branch: master
      driver: git
      initial_version: 0.0.0
      uri: ((version_uri))
      file: versions/ubuntu/19.10/devel/CodeServer-Image-Devel
      username: ((gitea_username))
      password: ((gitea_password))


  - name: res-CodeServer-Image-Devel-Version-Ubuntu-20.04
    type: semver
    source:
      branch: master
      driver: git
      initial_version: 0.0.0
      uri: ((version_uri))
      file: versions/ubuntu/20.04/devel/CodeServer-Image-Devel
      username: ((gitea_username))
      password: ((gitea_password))


jobs:


  - name: job-CodeServer-Image-Devel-Ubuntu-16.04
    serial_groups: [ "1" ]
    serial: true
    plan:
      - get: res-CodeServer-Image-Devel-Version-Ubuntu-16.04
        params:
          bump: patch
      - in_parallel:
        - get: res-CodeServer-Image-Devel
          trigger: true
        - get: res-cdr-code-server-master
          trigger: true
      - task: task-Build-CodeServer-Image-Devel-Ubuntu-16.04
        config:
          platform: linux
          image_resource:
            type: docker-image
            source: { repository: ubuntu, tag: 16.04 }
          inputs:
            - name: res-CodeServer-Image-Devel
            - name: res-CodeServer-Image-Devel-Version-Ubuntu-16.04
              path: res-CodeServer-Image-Devel-Version
          outputs:
            - name: res-cs-runtime-root
            - name: res-cs-tarballs
            - name: res-CodeServer-Image-Devel-Version-Ubuntu-16.04
          params:
            B_DISTRIBUTION_ID: "ubuntu"
            B_DISTRIBUTION_VERSION_ID: "16.04"
            B_PUSH_OVERRIDE: "1"
            CC_DRY_RUN: "1"
          run:
            path: "res-CodeServer-Image-Devel/code-server/concourse/concourse.sh"
      - put: res-CodeServer-Image-Devel-Version-Ubuntu-16.04
        params:
          file: res-CodeServer-Image-Devel-Version/version
      - put: res-CodeServer-Image-Devel-Private-Repository
        params:
          build: res-cs-runtime-root
          tag_file: res-cs-runtime-root/tag_file
          additional_tags: res-cs-runtime-root/additional_tags
          build_args_file: res-cs-runtime-root/build_args


  - name: job-CodeServer-Image-Devel-Ubuntu-18.04
    serial_groups: [ "2" ]
    serial: true
    plan:
      - get: res-CodeServer-Image-Devel-Version-Ubuntu-18.04
        params:
          bump: patch
      - in_parallel:
        - get: res-CodeServer-Image-Devel
          trigger: true
        - get: res-cdr-code-server-master
          trigger: true
      - task: task-Build-CodeServer-Image-Devel-Ubuntu-18.04
        config:
          platform: linux
          image_resource:
            type: docker-image
            source: { repository: ubuntu, tag: 18.04 }
          inputs:
            - name: res-CodeServer-Image-Devel
            - name: res-CodeServer-Image-Devel-Version-Ubuntu-18.04
              path: res-CodeServer-Image-Devel-Version
          outputs:
            - name: res-cs-runtime-root
            - name: res-cs-tarballs
            - name: res-CodeServer-Image-Devel-Version-Ubuntu-18.04
          params:
            B_DISTRIBUTION_ID: "ubuntu"
            B_DISTRIBUTION_VERSION_ID: "18.04"
            B_PUSH_OVERRIDE: "1"
            CC_DRY_RUN: "1"
          run:
            path: "res-CodeServer-Image-Devel/code-server/concourse/concourse.sh"
      - put: res-CodeServer-Image-Devel-Version-Ubuntu-18.04
        params:
          file: res-CodeServer-Image-Devel-Version/version
      - put: res-CodeServer-Image-Devel-Private-Repository
        params:
          build: res-cs-runtime-root
          tag_file: res-cs-runtime-root/tag_file
          additional_tags: res-cs-runtime-root/additional_tags
          build_args_file: res-cs-runtime-root/build_args


  - name: job-CodeServer-Image-Devel-Ubuntu-19.10
    serial_groups: [ "3" ]
    serial: true
    plan:
      - get: res-CodeServer-Image-Devel-Version-Ubuntu-19.10
        params:
          bump: patch
      - in_parallel:
        - get: res-CodeServer-Image-Devel
          trigger: true
        - get: res-cdr-code-server-master
          trigger: true
      - task: task-Build-CodeServer-Image-Devel-Ubuntu-19.10
        config:
          platform: linux
          image_resource:
            type: docker-image
            source: { repository: ubuntu, tag: 19.10 }
          inputs:
            - name: res-CodeServer-Image-Devel
            - name: res-CodeServer-Image-Devel-Version-Ubuntu-19.10
              path: res-CodeServer-Image-Devel-Version
          outputs:
            - name: res-cs-runtime-root
            - name: res-cs-tarballs
            - name: res-CodeServer-Image-Devel-Version-Ubuntu-19.10
          params:
            B_DISTRIBUTION_ID: "ubuntu"
            B_DISTRIBUTION_VERSION_ID: "19.10"
            B_PUSH_OVERRIDE: "1"
            CC_DRY_RUN: "1"
          run:
            path: "res-CodeServer-Image-Devel/code-server/concourse/concourse.sh"
      - put: res-CodeServer-Image-Devel-Version-Ubuntu-19.10
        params:
          file: res-CodeServer-Image-Devel-Version/version
      - put: res-CodeServer-Image-Devel-Private-Repository
        params:
          build: res-cs-runtime-root
          tag_file: res-cs-runtime-root/tag_file
          additional_tags: res-cs-runtime-root/additional_tags
          build_args_file: res-cs-runtime-root/build_args


  - name: job-CodeServer-Image-Devel-Ubuntu-20.04
    serial_groups: [ "1" ]
    serial: true
    plan:
      - get: res-CodeServer-Image-Devel-Version-Ubuntu-20.04
        params:
          bump: patch
      - in_parallel:
        - get: res-CodeServer-Image-Devel
          trigger: true
        - get: res-cdr-code-server-master
          trigger: true
      - task: task-Build-CodeServer-Image-Devel-Ubuntu-20.04
        config:
          platform: linux
          image_resource:
            type: docker-image
            source: { repository: ubuntu, tag: 20.04 }
          inputs:
            - name: res-CodeServer-Image-Devel
            - name: res-CodeServer-Image-Devel-Version-Ubuntu-20.04
              path: res-CodeServer-Image-Devel-Version
          outputs:
            - name: res-cs-runtime-root
            - name: res-cs-tarballs
            - name: res-CodeServer-Image-Devel-Version-Ubuntu-20.04
          params:
            B_DISTRIBUTION_ID: "ubuntu"
            B_DISTRIBUTION_VERSION_ID: "20.04"
            B_PUSH_OVERRIDE: "1"
            CC_DRY_RUN: "1"
          run:
            path: "res-CodeServer-Image-Devel/code-server/concourse/concourse.sh"
      - put: res-CodeServer-Image-Devel-Version-Ubuntu-20.04
        params:
          file: res-CodeServer-Image-Devel-Version/version
      - put: res-CodeServer-Image-Devel-Private-Repository
        params:
          build: res-cs-runtime-root
          tag_file: res-cs-runtime-root/tag_file
          additional_tags: res-cs-runtime-root/additional_tags
          build_args_file: res-cs-runtime-root/build_args

a description of your concourse deployment configuration - bosh/k8s/docker/systemd, scale (if relevant), what flags are being provided to the web/worker processes

I'm running your latest docker image on two virtual machines. One time as web version and one time as worker. I'm running a private CA and everything is using those private certificates for authentication.

I think this is a problem on the "inside" of the concourse environment. If the database connection wouldn't work, or there would be other problems connection wise, I'd be able to spot that in the logs and could easily fix it.

It's all working surprisingly well, but I got random problems popping up like the one where the folders aren't created.

It comes and goes, together with things like the Vault problem that I wrote about here:

https://discuss.concourse-ci.org/t/not-able-to-get-secrets-from-vault/2223

At the moment I'm under the impression that when the key's value is actually set to "value" and the value's value is the actual value that I need, that it works. As soon as I change the key's value to something else than "value" it seems to stop working.

But that can be a complete coincidence, as this comes and goes without anything in any log file.

Same with the folder.

Another new problem that I have at the moment, is that the semver-resource is trying to access a file outside the build environment when handing it the file parameter.

Instead of trying to access "/tmp/tmp.[RANDOM]/path/to/file" it tries to access "/tmp/put/path/to/file". This problem hasn't been here 10 minutes ago and now stops my build from working. It's totally odd.

The pipeline I pasted is the latest one, where I'm actually trying to figure this out.

Another interesting thing is, that setting "serial" to "1" has absolutely no effect. All jobs are started together. The only way to get this under control is to use serial_groups and use three different groups where I randomly assign the jobs to, so that I have a maximum of three running jobs.

It's not exactly clear if setting serial to '1" in the job's configuration actually has any effect on the parallel execution of jobs in the pipeline or only affects items inside the job to be run serial. The docs are only talking about "builds" whatever that means in that context...

Also "funny" is that the worker is not able to find an image for "ubuntu:19.10":

while at the same time all the other images are working fine and I can manually pull the Ubuntu image:

The job is sitting there like forever, not erroring out:

I'm thinking about flushing the database, recreating the images and then reload the pipeline to see how that goes and if the problems persist.

KR,

G

0 replies

Grimeton · 2020-05-05T16:10:19Z

Grimeton
May 5, 2020
Author

So,

just for fun:

This one works:

---
resources:
  - name: res-CodeServer-Image-Devel
    type: git
    source:
      uri: ((gitea_checkout_url))
      branch: ((gitea_checkout_branch))
      submodule_credentials:
        - host: ((gitea_host))
          username: ((gitea_username))
          password: ((gitea_password))
  - name: res-cdr-code-server-master
    type: git
    source:
      uri: https://github.com/cdr/code-server.git
      branch: master
  - name: res-CodeServer-Image-Devel-Private-Repository
    type: docker-image
    source:
      ca_certs:
        - domain: ((docker_registry_private_ca_domain))
          cert: ((docker_registry_private_ca_certificate))
      repository: ((docker_registry_private_url))
      username: ((docker_registry_private_username))
      password: ((docker_registry_private_password))
  - name: res-CodeServer-Image-Devel-Version
    type: semver
    source:
      branch: master
      driver: git
      initial_version: 0.0.0
      uri: ((version_uri))
      file: versions/ubuntu/16.04/devel/CodeServer-Image-Devel
      username: ((gitea_username))
      password: ((gitea_password))
jobs:
  - name: job-CodeServer-Image-Devel
    serial_groups: ["1"]
    serial: true
    plan:
      - get: res-CodeServer-Image-Devel-Version
        params:
          bump: patch
      - in_parallel:
          - get: res-CodeServer-Image-Devel
            trigger: true
          - get: res-cdr-code-server-master
            trigger: true
      - task: task-Build-CodeServer-Image-Devel
        config:
          platform: linux
          image_resource:
            type: docker-image
            source: { repository: ubuntu, tag: 16.04 }
          inputs:
            - name: res-CodeServer-Image-Devel
            - name: res-CodeServer-Image-Devel-Version
              path: res-CodeServer-Image-Devel-Version
          outputs:
            - name: res-cs-runtime-root
            - name: res-cs-tarballs
            - name: res-CodeServer-Image-Devel-Version
          params:
            B_DISTRIBUTION_ID: "ubuntu"
            B_DISTRIBUTION_VERSION_ID: "16.04"
            B_PUSH_OVERRIDE: "1"
            CC_DRY_RUN: "1"
          run:
            path: "res-CodeServer-Image-Devel/code-server/concourse/concourse.sh"
      - put: res-CodeServer-Image-Devel-Version
        params:
          file: res-CodeServer-Image-Devel-Version/version
      - put: res-CodeServer-Image-Devel-Private-Repository
        params:
          build: res-cs-runtime-root
          tag_file: res-cs-runtime-root/tag_file
          additional_tags: res-cs-runtime-root/additional_tags
          build_args_file: res-cs-runtime-root/build_args

This one breaks the semvar resource:

---
resources:
  - name: res-CodeServer-Image-Devel
    type: git
    source:
      uri: ((gitea_checkout_url))
      branch: ((gitea_checkout_branch))
      submodule_credentials:
        - host: ((gitea_host))
          username: ((gitea_username))
          password: ((gitea_password))
  - name: res-cdr-code-server-master
    type: git
    source:
      uri: https://github.com/cdr/code-server.git
      branch: master
  - name: res-CodeServer-Image-Devel-Private-Repository
    type: docker-image
    source:
      ca_certs:
        - domain: ((docker_registry_private_ca_domain))
          cert: ((docker_registry_private_ca_certificate))
      repository: ((docker_registry_private_url))
      username: ((docker_registry_private_username))
      password: ((docker_registry_private_password))
  - name: res-CodeServer-Image-Devel-Version-Ubuntu-1604
    type: semver
    source:
      branch: master
      driver: git
      initial_version: 0.0.0
      uri: ((version_uri))
      file: versions/ubuntu/16.04/devel/CodeServer-Image-Devel
      username: ((gitea_username))
      password: ((gitea_password))
jobs:
  - name: job-CodeServer-Image-Devel-Ubuntu-1604
    serial_groups: ["1"]
    serial: true
    plan:
      - get: res-CodeServer-Image-Devel-Version-Ubuntu-1604
        params:
          bump: patch
      - in_parallel:
          - get: res-CodeServer-Image-Devel
            trigger: true
          - get: res-cdr-code-server-master
            trigger: true
      - task: task-Build-CodeServer-Image-Devel-Ubuntu-1604
        config:
          platform: linux
          image_resource:
            type: docker-image
            source: { repository: ubuntu, tag: 16.04 }
          inputs:
            - name: res-CodeServer-Image-Devel
            - name: res-CodeServer-Image-Devel-Version-Ubuntu-1604
              path: res-CodeServer-Image-Devel-Version
          outputs:
            - name: res-cs-runtime-root
            - name: res-cs-tarballs
            - name: res-CodeServer-Image-Devel-Version-Ubuntu-1604
          params:
            B_DISTRIBUTION_ID: "ubuntu"
            B_DISTRIBUTION_VERSION_ID: "16.04"
            B_PUSH_OVERRIDE: "1"
            CC_DRY_RUN: "1"
          run:
            path: "res-CodeServer-Image-Devel/code-server/concourse/concourse.sh"
      - put: res-CodeServer-Image-Devel-Version-Ubuntu-1604
        params:
          file: res-CodeServer-Image-Devel-Version/version
      - put: res-CodeServer-Image-Devel-Private-Repository
        params:
          build: res-cs-runtime-root
          tag_file: res-cs-runtime-root/tag_file
          additional_tags: res-cs-runtime-root/additional_tags
          build_args_file: res-cs-runtime-root/build_args

When you diff both files, you get:

26c26
<   - name: res-CodeServer-Image-Devel-Version
---
>   - name: res-CodeServer-Image-Devel-Version-Ubuntu-1604
37c37
<   - name: job-CodeServer-Image-Devel
---
>   - name: job-CodeServer-Image-Devel-Ubuntu-1604
41c41
<       - get: res-CodeServer-Image-Devel-Version
---
>       - get: res-CodeServer-Image-Devel-Version-Ubuntu-1604
49c49
<       - task: task-Build-CodeServer-Image-Devel
---
>       - task: task-Build-CodeServer-Image-Devel-Ubuntu-1604
57c57
<             - name: res-CodeServer-Image-Devel-Version
---
>             - name: res-CodeServer-Image-Devel-Version-Ubuntu-1604
62c62
<             - name: res-CodeServer-Image-Devel-Version
---
>             - name: res-CodeServer-Image-Devel-Version-Ubuntu-1604
70c70
<       - put: res-CodeServer-Image-Devel-Version
---
>       - put: res-CodeServer-Image-Devel-Version-Ubuntu-1604

They're identical except for their names. The broken ones all have the suffix "-Ubuntu-1604". I checked with RFC40 and it says that all those characters are allowed, while upper case characters would be converted to lover case, which is fine.

There isn't any clash if stuff becomes all lower case.

Besides that I suspected the dot "." to cause trouble. Remove it everywhere, still the same problem.... So either there's a length limit in the variables or something is off.

When stuff works/doesn't work based on this, then it's no wonder I'm experiencing all kinds of odd behavior that comes and goes....

KR,

G.

0 replies

Grimeton · 2020-05-05T16:49:37Z

Grimeton
May 5, 2020
Author

I've tested further...

This one works...

---
resources:
  - name: res-CodeServer-Image-Devel
    type: git
    source:
      uri: ((gitea_checkout_url))
      branch: ((gitea_checkout_branch))
      submodule_credentials:
        - host: ((gitea_host))
          username: ((gitea_username))
          password: ((gitea_password))
  - name: res-cdr-code-server-master
    type: git
    source:
      uri: https://github.com/cdr/code-server.git
      branch: master
  - name: res-CodeServer-Image-Devel-Private-Repository
    type: docker-image
    source:
      ca_certs:
        - domain: ((docker_registry_private_ca_domain))
          cert: ((docker_registry_private_ca_certificate))
      repository: ((docker_registry_private_url))
      username: ((docker_registry_private_username))
      password: ((docker_registry_private_password))
  - name: res-CodeServer-Image-Devel-Version
    type: semver
    source:
      branch: master
      driver: git
      initial_version: 0.0.0
      uri: ((version_uri))
      file: versions/ubuntu/16.04/devel/CodeServer-Image-Devel
      username: ((gitea_username))
      password: ((gitea_password))
jobs:
  - name: job-CodeServer-Image-Devel
    serial_groups: ["1"]
    serial: true
    plan:
      - get: res-CodeServer-Image-Devel-Version
        params:
          bump: patch
      - in_parallel:
          - get: res-CodeServer-Image-Devel
            trigger: true
          - get: res-cdr-code-server-master
            trigger: true
      - task: task-Build-CodeServer-Image-Devel
        config:
          platform: linux
          image_resource:
            type: docker-image
            source: { repository: ubuntu, tag: 16.04 }
          inputs:
            - name: res-CodeServer-Image-Devel
            - name: res-CodeServer-Image-Devel-Version
              path: res-CodeServer-Image-Devel-Version
          outputs:
            - name: res-cs-runtime-root
            - name: res-cs-tarballs
            - name: res-CodeServer-Image-Devel-Version
          params:
            B_DISTRIBUTION_ID: "ubuntu"
            B_DISTRIBUTION_VERSION_ID: "16.04"
            B_PUSH_OVERRIDE: "1"
            CC_DRY_RUN: "1"
          run:
            path: "res-CodeServer-Image-Devel/code-server/concourse/concourse.sh"
      - put: res-CodeServer-Image-Devel-Version
        params:
          file: res-CodeServer-Image-Devel-Version/version
      - put: res-CodeServer-Image-Devel-Private-Repository
        params:
          build: res-cs-runtime-root
          tag_file: res-cs-runtime-root/tag_file
          additional_tags: res-cs-runtime-root/additional_tags
          build_args_file: res-cs-runtime-root/build_args

This one doesn't:

---
resources:
  - name: res-cs-id
    type: git
    source:
      uri: ((gitea_checkout_url))
      branch: ((gitea_checkout_branch))
      submodule_credentials:
        - host: ((gitea_host))
          username: ((gitea_username))
          password: ((gitea_password))
  - name: res-cs-master
    type: git
    source:
      uri: https://github.com/cdr/code-server.git
      branch: master
  - name: res-cs-idpr
    type: docker-image
    source:
      ca_certs:
        - domain: ((docker_registry_private_ca_domain))
          cert: ((docker_registry_private_ca_certificate))
      repository: ((docker_registry_private_url))
      username: ((docker_registry_private_username))
      password: ((docker_registry_private_password))
  - name: res-cs-id-version
    type: semver
    source:
      branch: master
      driver: git
      initial_version: 0.0.0
      uri: ((version_uri))
      file: versions/ubuntu/16.04/devel/CodeServer-Image-Devel
      username: ((gitea_username))
      password: ((gitea_password))
jobs:
  - name: job-cs-id
    serial_groups: ["1"]
    serial: true
    plan:
      - get: res-cs-id-version
        params:
          bump: patch
      - in_parallel:
          - get: res-cs-id
            trigger: true
          - get: res-cs-master
            trigger: true
      - task: task-build-cs
        config:
          platform: linux
          image_resource:
            type: docker-image
            source: { repository: ubuntu, tag: 16.04 }
          inputs:
            - name: res-cs-id
            - name: res-cs-id-version
              path: res-CodeServer-Image-Devel-Version
          outputs:
            - name: res-cs-runtime-root
            - name: res-cs-tarballs
            - name: res-cs-id-version
          params:
            B_DISTRIBUTION_ID: "ubuntu"
            B_DISTRIBUTION_VERSION_ID: "16.04"
            B_PUSH_OVERRIDE: "1"
            CC_DRY_RUN: "1"
          run:
            path: "res-cs-id/code-server/concourse/concourse.sh"
      - put: res-cs-id-version
        params:
          file: res-CodeServer-Image-Devel-Version/version
      - put: res-cs-idpr
        params:
          build: res-cs-runtime-root
          tag_file: res-cs-runtime-root/tag_file
          additional_tags: res-cs-runtime-root/additional_tags
          build_args_file: res-cs-runtime-root/build_args

The diff:

3c3
<   - name: res-CodeServer-Image-Devel
---
>   - name: res-cs-id
12c12
<   - name: res-cdr-code-server-master
---
>   - name: res-cs-master
17c17
<   - name: res-CodeServer-Image-Devel-Private-Repository
---
>   - name: res-cs-idpr
26c26
<   - name: res-CodeServer-Image-Devel-Version
---
>   - name: res-cs-id-version
37c37
<   - name: job-CodeServer-Image-Devel
---
>   - name: job-cs-id
41c41
<       - get: res-CodeServer-Image-Devel-Version
---
>       - get: res-cs-id-version
45c45
<           - get: res-CodeServer-Image-Devel
---
>           - get: res-cs-id
47c47
<           - get: res-cdr-code-server-master
---
>           - get: res-cs-master
49c49
<       - task: task-Build-CodeServer-Image-Devel
---
>       - task: task-build-cs
56,57c56,57
<             - name: res-CodeServer-Image-Devel
<             - name: res-CodeServer-Image-Devel-Version
---
>             - name: res-cs-id
>             - name: res-cs-id-version
62c62
<             - name: res-CodeServer-Image-Devel-Version
---
>             - name: res-cs-id-version
69,70c69,70
<             path: "res-CodeServer-Image-Devel/code-server/concourse/concourse.sh"
<       - put: res-CodeServer-Image-Devel-Version
---
>             path: "res-cs-id/code-server/concourse/concourse.sh"
>       - put: res-cs-id-version
73c73
<       - put: res-CodeServer-Image-Devel-Private-Repository
---
>       - put: res-cs-idpr

Gonna give up for now.

Kr,

G.

0 replies

vito · 2020-05-06T18:31:57Z

vito
May 6, 2020
Maintainer

I think what's gone wrong here is you've got a put step that seems to be trying to access a directory which is only available to the previous task as its input path.

put steps can only access artifacts produced by task outputs and get steps, and the directory names that it sees correspond to the get step name and the task output name, not its path.

e.g.:

plan:
- get: abc
- task: def
  config:
    inputs:
    - name: abc
      path: some-path
    outputs:
    - name: some-output
      path: some-path
    run: # write to some-path/banana
- put: blah
  # will run with...:
  # ./abc/...
  # ./some-output/banana

Also, if you're trying to use an input path as an output (i.e. changing files in-place), you'll need to configure them with the same path: field as above. In your example they had the same name, but the input's path was different. Not sure if that's what you were trying to do though.

0 replies

Grimeton · 2020-05-15T15:04:38Z

Grimeton
May 15, 2020
Author

I'm sorry I have to step away from this for two weeks. I have an event of death in my family and I'm not able to respond to this at the moment.

I will however re-open/come back to this when the problem still persists in two weeks.

Thanks.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Concourse

Tasks: Outputs: Folders not created automatically. #5544

Uh oh!

{{title}}

Uh oh!

Replies: 7 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Concourse

Tasks: Outputs: Folders not created automatically. #5544

Uh oh!

Grimeton May 5, 2020

Summary

Steps to reproduce

Expected results

Actual results

Additional context

Triaging info

Replies: 7 comments

Uh oh!

jamieklassen May 5, 2020 Collaborator

Uh oh!

vito May 5, 2020 Maintainer

Uh oh!

Uh oh!

Grimeton May 5, 2020 Author

Uh oh!

Grimeton May 5, 2020 Author

Uh oh!

Uh oh!

Grimeton May 5, 2020 Author

Uh oh!

vito May 6, 2020 Maintainer

Uh oh!

Grimeton May 15, 2020 Author

Grimeton
May 5, 2020

jamieklassen
May 5, 2020
Collaborator

vito
May 5, 2020
Maintainer

Grimeton
May 5, 2020
Author

Grimeton
May 5, 2020
Author

Grimeton
May 5, 2020
Author

vito
May 6, 2020
Maintainer

Grimeton
May 15, 2020
Author