Skip to content

Parameterized Pipelines

In the preceding section you were asked to place private credentials and personal git URLs into the pipeline.yml files. This would make it difficult to share your pipeline.yml with anyone who had access to the repository. Not everyone needs nor should have access to the shared secrets.

Concourse pipelines can include ((parameter)) parameters for any value in the pipeline YAML file.

Parameters are all mandatory. There are no default values for parameters.

In the lesson's pipeline.yml there are two parameters:

- name: show-animal-names
  - task: show-animal-names
      platform: linux
        type: docker-image
        source: {repository: busybox}
        path: env
        args: []
        CAT_NAME: ((cat-name))
        DOG_NAME: ((dog-name))

If we fly set-pipeline but do not provide the parameters, we see an error when the job is triggered to run:

cd ../parameters
fly -t tutorial set-pipeline -p parameters -c pipeline.yml
fly -t tutorial unpause-pipeline -p parameters
fly -t tutorial trigger-job -j parameters/show-animal-names -w

This will fail with the following error:

Expected to find variables: cat-name

Parameters from fly options

fly -t tutorial set-pipeline -p parameters -c pipeline.yml -v cat-name=garfield -v dog-name=odie
fly -t tutorial trigger-job -j parameters/show-animal-names -w

The output will show that the -v variables were passed into the params section of the show-animal-names task. Values in params sections in turn become environment variables within the task:

running env

Parameters from local file

Alternatively, you can store your parameter values in a local file.

cat > credentials.yml <<YAML
cat-name: garfield
dog-name: odie

Use the --load-vars-from flag (aliased -l) to pass in this file instead of the -v flag. The following command should not modify the pipeline from the preceding step as the resulting pipeline YAML is equivalent.

fly -t tutorial set-pipeline -p parameters -c pipeline.yml -l credentials.yml

Revisiting Publishing Outputs

In the previous lesson Publishing Outputs, there were two user-provided changes to the pipeline.yml. These can now be changed to parameters.

cd ../publishing-outputs

There is an alternate pipeline-parameters.yml that offers two parameters for resource-gist:

- name: resource-gist
  type: git
    branch:      master
    uri:         ((publishing-outputs-gist-uri))
    private_key: ((publishing-outputs-private-key))

Create a credentials.yml with the Gist URL and private key:

publishing-outputs-gist-uri: [email protected]:e028e491e42b9fb08447a3bafcf884e5.git
publishing-outputs-private-key: |-
    -----END RSA PRIVATE KEY-----

Use the --load-vars-from or -l flag to pass the variables into the parameters:

fly -t tutorial set-pipeline -p publishing-outputs -c pipeline-parameters.yml -l credentials.yml
fly -t tutorial unpause-pipeline -p publishing-outputs
fly -t tutorial trigger-job -j publishing-outputs/job-bump-date

Dynamic Parameters and Secret Parameters

Parameters are very useful. They allow you to describe your pipeline.yml in public repositories without embedding variables nor secrets.

There are two downsides to the two approaches above.

  • To change any parameter values requires you to rerun fly set-pipeline. If a value is common across many pipelines then you must rerun fly set-pipeline for them all.
  • The parameter values are not very secret. Anyone with access to the pipeline's team is able to download the pipeline YAML and extract the secrets.

    fly -t tutorial get-pipeline -p parameters

    Shows that the two potentially secret parameters are visible in plain text:

      CAT_NAME: garfield
      DOG_NAME: odie
      path: env

The solution to both of these problems is to use a Concourse Credentials Manager and is discussed in lesson Secret with Credential Manager.