Parallel step options
Parallel steps allow you to group pipeline steps that can run at the same time (concurrently) to reduce build time.
Overview
Set up or run parallel steps
Parallel steps enable you to build and test faster, by running a set of self-contained steps at the same time. The total number of build minutes used by a pipeline will not change if you make the steps parallel. You'll be able to see results sooner but the total build minutes used will calculate based on the total time taken on each step.
There are several ways in which you can use this feature, however, we recommend following these guidelines:
Set an initial step to build the software.
Add a set of parallel steps to test the software.
If you are doing a single build step leading to multiple parallel testing steps, cache external build dependencies as much as possible in the first step, so you don't waste time doing it in each of your testing steps. Then, run all your tests concurrently, they can use the cache.Deploying to multiple similar environments at the same time.
For example:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
pipelines:
default:
- step: # non-parallel step
script:
- ./build.sh
- parallel: # these 2 steps will run in parallel
steps:
- step:
script:
- ./integration-tests.sh --batch 1
- step:
script:
- ./integration-tests.sh --batch 2
- step: # non-parallel step
script:
- ./deploy.sh
Default variables for parallel steps
In addition to the standard pipelines variables, parallel step groups also have the following default variables:
BITBUCKET_PARALLEL_STEP - zero-based index of the current step in the group (such as 0, 1, 2, …).
BITBUCKET_PARALLEL_STEP_COUNT - total number of steps in the group.
Limitations for parallel steps
The 100-step limit on pipelines includes each step in a parallel group.
All environments in a parallel step set must be of the same type. Don’t try to mix production and test environment types in the same set.
Parallel steps can produce and consume artifacts, however, keep in mind the following:
Parallel steps can only use artifacts produced by previous steps, not by steps in the same parallel set.
Steps after the parallel set will get a combination of all files produced.
If parallel steps produce artifacts containing a file at the same location, conflicts are resolved on a per-file basis, and the files generated by the latest step in the YAML file will win.
The number of steps your Bitbucket workspace can run at once - across all running pipelines - varies according to your plan.
Standard and premium workspaces can support 600 steps running concurrently across multiple pipelines.
Free workspaces can support 10 steps runnings concurrently across multiple pipelines.
Once this limit is reached, excess steps will fail.
Parallel step options
The following options can be used to configure parallel step groups:
Parallel
The parallel option allows you to to build and test your code faster by running a list of steps at the same time. The total number of build minutes used by a pipeline will not change if you make the steps parallel, but you'll be able to see the results sooner. The total number of steps you can have in a Pipeline definition is limited to 100, regardless of whether they are running in parallel or serial.
For details on using parallel steps, see Set up or run parallel steps.
Property — parallel
Required — No
Data type — Block of new-line separated key-value pairs (YAML spec - Block Mapping)
Allowed parent properties — default, branches, pull-requests, tags, and custom
Allowed child properties — steps (required) and fail-fast
Example — using the parallel option to run two steps at the same time
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
pipelines:
default:
- step: # sequential step
name: Build
script:
- ./build.sh
- step: # sequential step
name: Build
script:
- ./build.sh
- parallel: # these 2 steps will run in parallel
steps:
- step:
name: Integration 1
script:
- ./integration-tests.sh --batch 1
- step:
name: Integration 2
script:
- ./integration-tests.sh --batch 2
- step: # non-parallel step
script:
- ./deploy.sh
Steps
The steps property contains the lists of steps in a stage or a parallel. At least 1 step is required within the steps list.
Property — steps
Required — Yes
Data type — Block of new-line separated key-value pairs (YAML spec - Block Mapping)
Allowed parent properties — parallel and stage
Allowed child properties — step (required)
Example — using the steps property to list the steps in a parallel group
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
pipelines:
default:
- step: # sequential step
name: Build
script:
- ./build.sh
- step: # sequential step
name: Build
script:
- ./build.sh
- parallel: # these 2 steps will run in parallel
steps:
- step:
name: Integration 1
script:
- ./integration-tests.sh --batch 1
- step:
name: Integration 2
script:
- ./integration-tests.sh --batch 2
- step: # non-parallel step
script:
- ./deploy.sh
Example — using the steps property to list the steps in a stage
1
2
3
4
5
6
7
8
9
10
11
12
13
pipelines:
default:
- stage:
name: Build and test
steps:
- step:
name: Build app
script:
- sh ./build-app.sh
- step:
name: Run unit tests
script:
- sh ./run-tests.sh
Fail-fast
fail-fast can be applied to all parallel steps or to a specific step in a parallel group:
If a step has fail-fast: false, then the step can fail without the whole parallel group stopping.
If a step has fail-fast: true, then the whole parallel group will stop if the step fails.
Property — fail-fast
Required — No
Data type — Boolean
Allowed values — true or false
Default value — false
Allowed parent properties — step and parallel
Example — using fail-fast to stop parallel steps when one fails
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
pipelines:
default:
- step:
name: Build
script:
- ./build.sh
- parallel:
# this option allows a force stop on all running steps if any step fails
fail-fast: true
steps:
- step:
name: Integration 1
script:
- ./integration-tests.sh --batch 1
- step:
name: Integration 2
script:
- ./integration-tests.sh --batch 2
Example — exclude a parallel from fail-fast
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
pipelines:
default:
- step:
name: Build
script:
- ./build.sh
- parallel:
# this option allows a force stop on all running steps if any step fails
fail-fast: true
steps:
- step:
name: Integration 1
script:
- ./integration-tests.sh --batch 1
- step:
name: Integration 2
script:
- ./integration-tests.sh --batch 2
- step:
# option can be disabled for a step
# and its failure won't stop other steps in a group
fail-fast: false
name: Upload metadata
script:
- ./upload-metadata.sh
Was this helpful?