Managing Pipelines
Creating Pipelines
A pipeline requires at least two jobs. To create jobs, see Creating Jobs.
-
Click Projects from the primary navigation menu.
By default, the All Projects page opens with the list of existing projects. -
Click a project in the list to open it.
By default, the Jobs page opens with the list of existing jobs. -
Click Pipelines from the secondary navigation menu to open the project’s pipeline library.
-
Click New pipeline to create a new pipeline.
A page to create your pipeline opens. -
Enter a name, alias, and description for your pipeline.
-
Construct your pipeline by adding at least two jobs.
-
Select Add job and chose a job from the list.
A card containing basic information, such as name, technology, and job category, appears in the graph. -
Select Add job again and chose another job from the list.
Another job card appears in the graph. -
Link the two jobs.
-
Repeat these steps as many times as required. You can also add one or more conditions to build more relevant pipelines. To add conditions, click Add condition when editing your pipeline and link it to your jobs. Remember that a condition must have at least one parent and one child.
-
-
Configure settings to schedule your pipeline run, send email alerts, and set environment variables. You can also add a release note.
-
Click either:
-
Save pipeline to save your changes.
-
The kebab menu
to save your changes and automatically run your pipeline. -
The cross to exit the creation mode without saving your pipeline. A pop-up window appears with a progress bar. You can click the cross to exit faster or click Continue edition to cancel your cancellation.
-
To delete a pipeline, you can either:
A confirmation message appears. Click Delete again to confirm the deletion. Be careful, there is no progress bar to cancel the deletion once it is confirmed. |
Running and Stopping Pipelines
-
Click Projects from the primary navigation menu.
By default, the All Projects page opens with the list of existing projects. -
Click a project in the list to open it.
By default, the Jobs page opens with the list of existing jobs. -
Click Pipelines from the secondary navigation menu to open the project’s pipeline library.
-
You can either:
-
Click Run or Stop in front of the desired pipeline from the library.
-
Open the pipeline details and start or stop the run from its Overview page. Click either Run or Stop depending of the current status of your pipeline.
You can also access this command at the bottom of the secondary navigation menu from the Instances and Versions page.
The pipeline status changes depending on the outcome.
-
A pipeline is
Succeeded
when all its jobs have been successfully executed or skipped. -
A pipeline is
Failed
as soon as one or more of its jobs has failed. -
Jobs that follow a failed job are always
Skipped
.
-
Modifying Pipeline Settings
-
Click Projects from the primary navigation menu.
By default, the All Projects page opens with the list of existing projects. -
Click a project in the list to open it.
By default, the Jobs page opens with the list of existing jobs. -
Click Pipeline from the secondary navigation menu to open the project’s pipeline library.
-
Click a pipeline in the list to open it and access its details.
The pipeline Overview page opens. -
Click the desired setting from the Overview page (a) or in Edit mode (b) to edit it:
Figure 1. Pipeline Settings from the Overview PageFigure 2. Pipeline Settings in the Edit ModeNames are mandatory. They can be up to 255 characters long and must be unique within a project.
The pipeline alias is unique to each pipeline in a project. It allows you to reference a pipeline in your code with a constant key.
The description field is not available in the Edit mode. Descriptions are optional and not restricted. However, it is a good practice to add them. Keep them short and to the point.
Click the switch in the side panel of the setting to enable or disable it. There are two types of execution:
-
The manual run, which requires you to click Run to start the pipeline.
-
The scheduled run, which launches the pipeline according to the schedule you choose.
Scheduled pipelines can also be started manually. The Scheduled run type has three schedule modes: Simple, Shortcut, and Expert.
-
In Simple mode, you can easily specify variables through the user interface. There are many possibilities.
-
In Shortcut mode, you can choose the recurrence of your run on an hourly, daily, weekly, monthly, or annual basis. All other settings are automatic.
-
In Expert mode, you can specify variables using the Cron format. The Cron time string consists of five values separated by spaces:
[minute] [hour] [day of the month] [month] [day of the week]
. They are based on the following information:Table 1. Cron format Descriptor Acceptable values Minute
0 to 59, or *
Hour
0 to 23, or *
Day of the month
1 to 31, or *
Month
1 to 12, or *
Day of the week
0 to 7 (0 and 7 both represent Sunday), or *
The Cron time string must contain entries for each character attribute. If you want to set a value using only minutes, you must have asterisk (*) characters for the other four attributes that you are not configuring. Once you have finished scheduling your run, you will see the summary of your choice written below and the time of the next run.
-
Click the switch in the side panel of the setting to enable or disable it. Alerts are optional and can be set to receive an email when the status of your pipeline changes. They can be sent to multiple email addresses to notify you of the following status changes:
Status Description Requested
The pipeline’s run has been requested and is being executed.
Queued
The pipeline is waiting for the necessary resources to be executed.
Running
The pipeline is up and running.
Failed
The pipeline has crashed.
Stopping
The pipeline is stopping.
Stopped
The pipeline has stopped running.
Succeeded
The pipeline has been successfully executed.
Unknown
The pipeline no longer runs because an error has occurred.
Click the switch in the side panel of the setting to enable or disable it. This setting can be used to:
-
Manage Environment Variables on the Fly
You can create, modify, and delete pipeline environment variables on the fly from the Variables side panel. Changes made from this panel will update the pipeline Environment variables page accordingly.
You do not have to enable the setting for this to work. For more information, see Managing Pipeline Environment Variables on the Fly.
-
Transfer Variables Between Jobs During Pipeline Execution
-
When enabled , it allows you to transfer modified and created variables between jobs during pipeline execution. In addition, you can view a summary table of used, modified, and unchanged variables for each job in the pipeline Overview and Instances pages.
This feature relies on advanced mechanisms that require more resources. These resources would be wasted if not used efficiently. Thus, to have correct optimization, we recommend to disable this option if it has no real use during the execution of the pipeline. -
When disabled , modification and transfer of variables between jobs during pipeline execution is not allowed. You can still read global, project, and pipeline environment variables, but not modify them.
-
The second part of the Overview page indicates whether the setting is enabled or not.
-
Each pipeline instance indicates whether it was executed with the setting enabled or not.
-
For more information, see Creating and Modifying Variables in a Job Output in a Pipeline.
-
-
-
Saving is automatic. You can just press Enter to confirm the pipeline name change, click anywhere nearby to confirm the pipeline alias and description change, and close the side panel to confirm the scheduled run, email alert, and environment variable changes.
Managing Pipeline Environment Variables on the Fly
-
Click Projects from the primary navigation menu.
By default, the All Projects page opens with the list of existing projects. -
Click a project in the list to open it.
By default, the Jobs page opens with the list of existing jobs. -
Click Pipeline from the secondary navigation menu to open the project’s pipeline library.
-
Click a pipeline in the list to open it and access its details.
The pipeline Overview page opens. -
Click the Variables setting in your pipeline from the pipeline Overview page or from its Edit mode.
A panel opens with the existing variables in a code block. -
In the code block, add, modify, or delete variables.
How to set pipeline environment variables?-
A pipeline environment variable includes a key (name), an equal sign, and a value.
-
The name is mandatory. It must begin with a letter and can be up to 128 characters, including alphanumeric characters (a-z) (A-Z) (0-9) and underscores (_).
-
Values can be up to 2,048 characters.
-
A line is equal to an environment variable.
-
A line that begins with a hash sign (#) is a comment line. This content will be ignored.
-
If the same key is defined several times, only the value of the last occurrence of this key will be saved.
-
Environment variables defined as passwords are not listed here, nor can they be defined as such.
⇒ Read the embedded help for more information.
-
These changes will update the pipeline Environment variables page accordingly.
Upgrading Pipelines
-
Click Projects from the primary navigation menu.
By default, the All Projects page opens with the list of existing projects. -
Click a project in the list to open it.
By default, the Jobs page opens with the list of existing jobs. -
Click Pipeline from the secondary navigation menu to open the project’s pipeline library.
-
Click a pipeline in the list to open it and access its details.
The pipeline Overview page opens. -
Click Upgrade pipeline from the Instances or Versions page.
You can also click Edit from the Overview page. The page for editing your pipeline opens.
-
You can:
-
Design your pipeline: Move jobs and conditions anywhere on the design grid.
-
Add or remove jobs and conditions:
-
Click Add job or Add condition and select an item from the list to add a job or a condition to the pipeline.
-
Click the job or condition card and select Remove to remove it from the pipeline.
-
-
Add or remove links:
-
Link the connectors of a job or condition to create connections between them.
-
Click the small grey cross in the middle of a link to remove the link between jobs or conditions.
-
-
Modify settings: Change the pipeline name, alias, scheduled run, email alerts, and environment variables.
Changes made only to metadata such as name, alias, scheduled run, email alerts, and environment variables do not trigger the creation of a new pipeline version. -
Manage the release note: Add or modify the release note.
-
-
Click either:
-
Save pipeline to save your changes.
-
The kebab menu
to save your changes and automatically run your pipeline. -
The cross to exit the creation mode without saving your pipeline. A pop-up window appears with a progress bar. You can click the cross to exit faster or click Continue edition to cancel your cancellation.
-
Your pipeline has been upgraded, and you should see that a new version of it has been created, along with a new instance.
Duplicating Pipelines
-
Click Projects from the primary navigation menu.
By default, the All Projects page opens with the list of existing projects. -
Click a project in the list to open it.
By default, the Jobs page opens with the list of existing jobs. -
Click Pipelines from the secondary navigation menu to open the project’s pipeline library.
-
You can click the kebab menu
at the end of the relevant row from the Pipeline page, or from the Overview page of a pipeline.Figure 3. Duplicate your pipeline from the Pipeline page-
You cannot duplicate an invalid pipeline.
-
You cannot select multiple pipeline versions at the same time and duplicate them.
A window opens.
-
-
Select either:
-
Shallow duplication: This option includes a copy of the conditions and environment variables at pipeline level, and a reference to the original jobs and their relationship to the pipeline.
-
Deep duplication: This option includes a copy of the conditions, of the environment variables at pipeline level, and of the original jobs and their relationship to the pipeline.
-
-
Click Duplicate to confirm the duplication.
Depending on whether you duplicated from the Pipeline page or from the pipeline Overview page, you are returned to either the pipeline library, or the Overview page of the duplicated pipeline, respectively. In addition, a message appears saying that your pipeline has been successfully duplicated.The name and alias of your duplicated pipeline will be
<original pipeline name> (ID starting by X)
and<original pipeline alias> (ID starting by X)
. WhereX
is an incremental number starting by 1.For example, you duplicate the pipeline
Clean Data
. The name of the duplicated pipeline will beClean Data (1)
. If you duplicate the pipelineClean Data
a second time, the name of this second duplicated pipeline will beClean Data (2)
, and so on. If the duplicated pipeline name exceed the 255-character limit, we truncate the last 13 characters. The incremental number is kept.
Deleting Pipeline Instances and Pipeline Versions
-
Click Projects from the primary navigation menu.
By default, the All Projects page opens with the list of existing projects. -
Click a project in the list to open it.
By default, the Jobs page opens with the list of existing jobs. -
Click Pipeline from the secondary navigation menu to open the project’s pipeline library.
-
Click a pipeline in the list to open it and access its details.
The pipeline Overview page opens. -
Click either:
-
Instances to access your pipeline’s instances page and delete instances with their logs from the list.
-
Deleting a pipeline instance deletes all the job instances linked to it, as well as all associated logs.
-
You cannot delete unfinished instances, that is, instances whose status is
Requested
,Queued
,Running
,Stopping
. Wait for the end of the executions and try again. -
You cannot delete an instance that has not yet been cleaned from the orchestrator. Wait a moment and try again.
-
-
Versions to access your pipeline’s versions page and delete versions from the list.
-
Deleting a pipeline version deletes the pipeline and job instances linked to it, as well as all its dependencies, such as logs and conditions.
-
You cannot delete the version tagged as
Current
. You can revert to another version of the pipeline to change the current one and delete it. -
You cannot delete a version that is linked to unfinished instances. Wait for the end of the executions and try again.
-
You cannot delete a version whose instances have not yet been cleaned from the orchestrator. Wait a moment and try again.
-
-
-
You can delete either instances or versions of your pipeline.
You can delete a single pipeline instance or version by clicking the trash can that appears on the line when you hover over the desired element.
You can delete a selection of pipeline instances or versions by selecting them in the list and clicking Delete n instances/versions at the top of the list.
You will then be able to play with the selector by adding or removing elements with different states. You can delete a selection of pipeline instances based on status criteria. To do this, select the desired filter and click Delete n instances at the top of the list.
You will then be able to play with the selector by adding or removing elements with different states. You can delete pipeline instances by date criteria with the All instances older than filter. It will delete all instances and associated logs prior to the selected date.
-
Select the All instances older than filter.
A pop-up window opens with a date picker.
-
Select a date.
-
Click Select instances to confirm your choice.
The corresponding instances are selected in the list.You will then be able to play with the selector by adding or removing elements. -
Click Delete n instances at the top of the list.
-
-
Confirm the pop-up message and the permanent deletion by clicking Delete.
Moving a Pipeline to Another Project
-
Click Projects from the primary navigation menu.
By default, the All Projects page opens with the list of existing projects. -
Click a project in the list to open it.
By default, the Jobs page opens with the list of existing jobs. -
Click Pipelines from the secondary navigation menu to open the project’s pipeline library.
-
You can click the kebab menu
at the end of the relevant pipeline row from the Pipelines page, or from the Overview page of a pipeline.Figure 4. Move a pipeline from the Pipeline pageIn Figure 4, the Move to… action is in orange. This means that some issue(s) are preventing your pipeline from being moved. You can still click on it; It will display a panel that lists the errors.
The possible issues can be as follows:
-
You cannot move a pipeline that has one or more jobs in use in another pipeline. It can be a job that is used in a different version than the current version of another pipeline. Duplicate your pipeline and move the duplicated pipeline to bypass this issue.
-
You cannot move a pipeline whose instances have not yet been cleaned from the orchestrator. Wait a moment and try again.
-
You cannot move a running and scheduled pipeline, as well as a pipeline containing running and scheduled jobs. Wait for the end of the execution(s) and try to remove the scheduling settings. Then try again.
-
The job technology that is used in the pipeline must be available in the target project. If it does not, select another project or modify the settings of the target project.
A pop-up window opens.
-
-
Select the target platform and target project from the list.
-
Click Move to confirm the move.
You will see a message saying that your pipeline has been successfully moved. You can click Go to target project below the success message to be redirected to the pipeline library of the target project. Jobs, conditions, pipeline environment variables, versions, instances, logs, alerts, and resource settings are all included in the moved pipeline. There is no obfuscation of the data. If there is sensitive data, it will be visible in the target project.