Managing Pipelines

Use these tutorials to create, run, modify, upgrade, duplicate, and move your pipelines.

Creating Pipelines

Before you begin:

A pipeline requires at least two jobs. To create jobs, see Creating Jobs.

  1. Click The "Projects" module icon is a folder. Projects from the primary navigation menu.
    By default, the The "All Projects" page icon is a folder and the same as the "Projects" module. All Projects page opens with the list of existing projects.

  2. Click a project in the list to open it.
    By default, the The "Jobs" page icon is a 3D pyramid of three squares. Jobs page opens with the list of existing jobs.

    Navigate to your project
  3. Click The "Pipelines" page icon is represented by several connected jobs. Pipelines from the secondary navigation menu to open the project’s pipeline library.

  4. Click New pipeline to create a new pipeline.
    A page to create your pipeline opens.

    pipeline create new
  5. Enter a name, alias, and description for your pipeline.

  6. Construct your pipeline by adding at least two jobs.

    1. Select Add job and chose a job from the list.
      A card containing basic information, such as name, technology, and job category, appears in the graph.

    2. Select Add job again and chose another job from the list.
      Another job card appears in the graph.

    3. Link the two jobs.

      pipeline link jobs
    4. Repeat these steps as many times as required. You can also add one or more conditions to build more relevant pipelines. To add conditions, click Add condition when editing your pipeline and link it to your jobs. Remember that a condition must have at least one parent and one child.

      pipeline example

  7. Configure settings to schedule your pipeline run, send email alerts, and set environment variables. You can also add a release note.

  8. Click either:

    • Save pipeline to save your changes.

    • The kebab menu The kebab menu icon is three vertical dots.  Save & run to save your changes and automatically run your pipeline.

    • The cross close to exit the creation mode without saving your pipeline. A pop-up window appears with a progress bar. You can click the cross to exit faster or click Continue edition to cancel your cancellation.

To delete a pipeline, you can either:

  • Click delete Delete at the end of the corresponding pipeline row from the The "Pipelines" page icon is represented by several connected jobs. Pipelines page.

  • Click the kebab menu The kebab menu icon is three vertical dots.  delete Delete from the The "Overview" page icon is a square divided into several other squares. Overview page of the corresponding pipeline.

  • Click delete Delete pipeline at the bottom of the secondary navigation menu from the corresponding pipeline The "Instances" page icon is three overlapping squares. Instances and The "Versions" page icon is a folder with an arrow pointing up. Versions page.

A confirmation message appears. Click Delete again to confirm the deletion. Be careful, there is no progress bar to cancel the deletion once it is confirmed.

Running and Stopping Pipelines

You can run or stop your pipelines manually, even when pipelines are scheduled to run.

  1. Click The "Projects" module icon is a folder. Projects from the primary navigation menu.
    By default, the The "All Projects" page icon is a folder and the same as the "Projects" module. All Projects page opens with the list of existing projects.

  2. Click a project in the list to open it.
    By default, the The "Jobs" page icon is a 3D pyramid of three squares. Jobs page opens with the list of existing jobs.

    Navigate to your project
  3. Click The "Pipelines" page icon is represented by several connected jobs. Pipelines from the secondary navigation menu to open the project’s pipeline library.

  4. You can either:

    • Click Run Run or Stop Stop in front of the desired pipeline from the library.

    • Open the pipeline details and start or stop the run from its The "Overview" page icon is a square divided into several other squares. Overview page. Click either Run Run or Stop Stop depending of the current status of your pipeline.

      You can also access this command at the bottom of the secondary navigation menu from the The "Instances" page icon is three overlapping squares. Instances and The "Versions" page icon is a folder with an arrow pointing up. Versions page.

      pipeline run button secondary nav menu

    The pipeline status changes depending on the outcome.

    • A pipeline is Succeeded when all its jobs have been successfully executed or skipped.

    • A pipeline is Failed as soon as one or more of its jobs has failed.

    • Jobs that follow a failed job are always Skipped.

Modifying Pipeline Settings

After its creation, you can always modify your pipeline settings. You can access the settings from the pipeline The "Overview" page icon is a square divided into several other squares. Overview page or from its edit mode.

  1. Click The "Projects" module icon is a folder. Projects from the primary navigation menu.
    By default, the The "All Projects" page icon is a folder and the same as the "Projects" module. All Projects page opens with the list of existing projects.

  2. Click a project in the list to open it.
    By default, the The "Jobs" page icon is a 3D pyramid of three squares. Jobs page opens with the list of existing jobs.

    Navigate to your project
  3. Click The "Pipelines" page icon is represented by several connected jobs. Pipeline from the secondary navigation menu to open the project’s pipeline library.

  4. Click a pipeline in the list to open it and access its details.
    The pipeline The "Overview" page icon is a square divided into several other squares. Overview page opens.

    Navigate to your pipeline
  5. Click the desired setting from the The "Overview" page icon is a square divided into several other squares. Overview page (a) or in edit Edit mode (b) to edit it:

    Pipeline settings from the "Overview" page.
    Figure 1. Pipeline Settings from the Overview Page
    Pipeline settings from the edit mode.
    Figure 2. Pipeline Settings in the Edit Mode
    • 1 – Name

    • 2 – Alias

    • 3 – Description

    • 4 – Scheduled Run

    • 5 – Email Alerts

    • 6 – Variables

    Names are mandatory. They can be up to 255 characters long and must be unique within a project.

    The pipeline alias is unique to each pipeline in a project. It allows you to reference a pipeline in your code with a constant key.

    The description field is not available in the edit Edit mode.

    Descriptions are optional and not restricted. However, it is a good practice to add them. Keep them short and to the point.

    Click the switch in the side panel of the setting to enable or disable it.

    There are two types of execution:

    1. The manual run, which requires you to click Run to start the pipeline.

    2. The scheduled run, which launches the pipeline according to the schedule you choose.

      Scheduled pipelines can also be started manually.

      The Scheduled run type has three schedule modes: Simple, Shortcut, and Expert.

      • In Simple mode, you can easily specify variables through the user interface. There are many possibilities.

        Screenshot of the settings for the scheduled run type in simple mode

      • In Shortcut mode, you can choose the recurrence of your run on an hourly, daily, weekly, monthly, or annual basis. All other settings are automatic.

        Screenshot of the settings for the scheduled run type in shortcut mode

      • In Expert mode, you can specify variables using the Cron format. The Cron time string consists of five values separated by spaces: [minute] [hour] [day of the month] [month] [day of the week]. They are based on the following information:

        Table 1. Cron format
        Descriptor Acceptable values

        Minute

        0 to 59, or *

        Hour

        0 to 23, or *

        Day of the month

        1 to 31, or *

        Month

        1 to 12, or *

        Day of the week

        0 to 7 (0 and 7 both represent Sunday), or *

        The Cron time string must contain entries for each character attribute. If you want to set a value using only minutes, you must have asterisk (*) characters for the other four attributes that you are not configuring.

        Screenshot of the settings for the scheduled run type in expert mode

        Once you have finished scheduling your run, you will see the summary of your choice written below and the time of the next run.

    Click the switch in the side panel of the setting to enable or disable it.

    Alerts are optional and can be set to receive an email when the status of your pipeline changes. They can be sent to multiple email addresses to notify you of the following status changes:

    Status Description

    queued requested Requested

    The pipeline’s run has been requested and is being executed.

    queued requested Queued

    The pipeline is waiting for the necessary resources to be executed.

    spinner Running

    The pipeline is up and running.

    fail Failed

    The pipeline has crashed.

    stop Stopping

    The pipeline is stopping.

    stop Stopped

    The pipeline has stopped running.

    success Succeeded

    The pipeline has been successfully executed.

    unknown Unknown

    The pipeline no longer runs because an error has occurred.

    Click the switch in the side panel of the setting to enable or disable it.

    This setting can be used to:

    • Manage Environment Variables on the Fly

      You can create, modify, and delete pipeline environment variables on the fly from the Variables side panel. Changes made from this panel will update the pipeline settings env vars a Environment variables page accordingly.

      You do not have to enable the setting for this to work.
    • Transfer Variables Between Jobs During Pipeline Execution

      • When enabled settings env vars a, it allows you to transfer modified and created variables between jobs during pipeline execution. In addition, you can view a summary table of used, modified, and unchanged variables for each job in the pipeline The "Overview" page icon is a square divided into several other squares. Overview and The "Instances" page icon is three overlapping squares. Instances pages.

        This feature relies on advanced mechanisms that require more resources. These resources would be wasted if not used efficiently. Thus, to have correct optimization, we recommend to disable this option if it has no real use during the execution of the pipeline.
      • When disabled settings env vars, modification and transfer of variables between jobs during pipeline execution is not allowed. You can still read global, project, and pipeline environment variables, but not modify them.

        • The second part of the The “Overview” page icon is a square divided into several other squares. Overview page indicates whether the setting is enabled or not.

        • Each pipeline instance indicates whether it was executed with the setting enabled or not.

  6. Saving is automatic. You can just press Enter to confirm the pipeline name change, click anywhere nearby to confirm the pipeline alias and description change, and close the side panel to confirm the scheduled run, email alert, and environment variable changes.

Managing Pipeline Environment Variables on the Fly

You can create, modify, and delete pipeline environment variables on the fly using the settings env vars Variables side panel.

The setting does not need to be activated for this to work.
  1. Click The "Projects" module icon is a folder. Projects from the primary navigation menu.
    By default, the The "All Projects" page icon is a folder and the same as the "Projects" module. All Projects page opens with the list of existing projects.

  2. Click a project in the list to open it.
    By default, the The "Jobs" page icon is a 3D pyramid of three squares. Jobs page opens with the list of existing jobs.

    Navigate to your project
  3. Click The "Pipelines" page icon is represented by several connected jobs. Pipeline from the secondary navigation menu to open the project’s pipeline library.

  4. Click a pipeline in the list to open it and access its details.
    The pipeline The "Overview" page icon is a square divided into several other squares. Overview page opens.

    Navigate to your pipeline
  5. Click the settings env vars Variables setting in your pipeline from the pipeline The "Overview" page icon is a square divided into several other squares. Overview page or from its edit Edit mode.
    A panel opens with the existing variables in a code block.

  6. In the code block, add, modify, or delete variables.

    How to set pipeline environment variables?
    • A pipeline environment variable includes a key (name), an equal sign, and a value.

    • The name is mandatory. It must begin with a letter and can be up to 128 characters, including alphanumeric characters (a-z) (A-Z) (0-9) and underscores (_).

    • Values can be up to 2,048 characters.

    • A line is equal to an environment variable.

    • A line that begins with a hash sign (#) is a comment line. This content will be ignored.

    • If the same key is defined several times, only the value of the last occurrence of this key will be saved.

    • Environment variables defined as passwords are not listed here, nor can they be defined as such.

    Read the embedded help for more information.

These changes will update the pipeline settings env vars a Environment variables page accordingly.

Upgrading Pipelines

You can upgrade your pipelines to always get the most out of them. It allows you to modify the name and description, the settings, add and delete jobs and conditions. By upgrading your pipeline, you create a new version of it.

  1. Click The "Projects" module icon is a folder. Projects from the primary navigation menu.
    By default, the The "All Projects" page icon is a folder and the same as the "Projects" module. All Projects page opens with the list of existing projects.

  2. Click a project in the list to open it.
    By default, the The "Jobs" page icon is a 3D pyramid of three squares. Jobs page opens with the list of existing jobs.

    Navigate to your project
  3. Click The "Pipelines" page icon is represented by several connected jobs. Pipeline from the secondary navigation menu to open the project’s pipeline library.

  4. Click a pipeline in the list to open it and access its details.
    The pipeline The "Overview" page icon is a square divided into several other squares. Overview page opens.

    Navigate to your pipeline
  5. Click Upgrade pipeline Upgrade pipeline button from the The "Instances" page icon is three overlapping squares. Instances or The "Versions" page icon is a folder with an arrow pointing up. Versions page.

    You can also click edit Edit from the The "Overview" page icon is a square divided into several other squares. Overview page.

    The page for editing your pipeline opens.

  6. You can:

    • Design your pipeline: Move jobs and conditions anywhere on the design grid.

    • Add or remove jobs and conditions:

      • Click Add job or Add condition and select an item from the list to add a job or a condition to the pipeline.

      • Click the job or condition card and select delete Remove to remove it from the pipeline.

    • Add or remove links:

      • Link the connectors of a job or condition to create connections between them.

      • Click the small grey cross close in the middle of a link to remove the link between jobs or conditions.

    • Modify settings: Change the pipeline name, alias, scheduled run, email alerts, and environment variables.

      Changes made only to metadata such as name, alias, scheduled run, email alerts, and environment variables do not trigger the creation of a new pipeline version.
    • Manage the release note: Add or modify the release note.

  7. Click either:

    • Save pipeline to save your changes.

    • The kebab menu The kebab menu icon is three vertical dots.  Save & run to save your changes and automatically run your pipeline.

    • The cross close to exit the creation mode without saving your pipeline. A pop-up window appears with a progress bar. You can click the cross to exit faster or click Continue edition to cancel your cancellation.

Your pipeline has been upgraded, and you should see that a new version of it has been created, along with a new instance.

Duplicating Pipelines

You can only duplicate the Current Badge for the "Current" status version of your pipeline. There are two duplication levels. Shallow and deep duplication. Both options exclude alerts, scheduling, other pipeline versions, instances, logs, and observability data.

  1. Click The "Projects" module icon is a folder. Projects from the primary navigation menu.
    By default, the The "All Projects" page icon is a folder and the same as the "Projects" module. All Projects page opens with the list of existing projects.

  2. Click a project in the list to open it.
    By default, the The "Jobs" page icon is a 3D pyramid of three squares. Jobs page opens with the list of existing jobs.

    Navigate to your project
  3. Click The "Pipelines" page icon is represented by several connected jobs. Pipelines from the secondary navigation menu to open the project’s pipeline library.

  4. You can click the kebab menu The kebab menu icon is three vertical dots.  clone copy Duplicate at the end of the relevant row from the The "Pipelines" page icon is represented by several connected jobs. Pipeline page, or from the The "Overview" page icon is a square divided into several other squares. Overview page of a pipeline.

    pipeline duplicate library
    Figure 3. Duplicate your pipeline from the The "Pipelines" page icon is represented by several connected jobs. Pipeline page
    • You cannot duplicate an invalid pipeline.

    • You cannot select multiple pipeline versions at the same time and duplicate them.

    A window opens.

  5. Select either:

    • Shallow duplication: This option includes a copy of the conditions and environment variables at pipeline level, and a reference to the original jobs and their relationship to the pipeline.

    • Deep duplication: This option includes a copy of the conditions, of the environment variables at pipeline level, and of the original jobs and their relationship to the pipeline.

  6. Click Duplicate to confirm the duplication.
    Depending on whether you duplicated from the The "Pipelines" page icon is represented by several connected jobs. Pipeline page or from the pipeline The "Overview" page icon is a square divided into several other squares. Overview page, you are returned to either the pipeline library, or the The "Overview" page icon is a square divided into several other squares. Overview page of the duplicated pipeline, respectively. In addition, a message appears saying that your pipeline has been successfully duplicated.

    The name and alias of your duplicated pipeline will be <original pipeline name> (ID starting by X) and <original pipeline alias> (ID starting by X). Where X is an incremental number starting by 1.

    For example, you duplicate the pipeline Clean Data. The name of the duplicated pipeline will be Clean Data (1). If you duplicate the pipeline Clean Data a second time, the name of this second duplicated pipeline will be Clean Data (2), and so on. If the duplicated pipeline name exceed the 255-character limit, we truncate the last 13 characters. The incremental number is kept.

Deleting Pipeline Instances and Pipeline Versions

The instance and version history of your pipeline is too large, and it is not easy to find your little ones. In this case, you may want to delete some instances or versions to streamline these lists and improve your user experience.

  1. Click The "Projects" module icon is a folder. Projects from the primary navigation menu.
    By default, the The "All Projects" page icon is a folder and the same as the "Projects" module. All Projects page opens with the list of existing projects.

  2. Click a project in the list to open it.
    By default, the The "Jobs" page icon is a 3D pyramid of three squares. Jobs page opens with the list of existing jobs.

    Navigate to your project
  3. Click The "Pipelines" page icon is represented by several connected jobs. Pipeline from the secondary navigation menu to open the project’s pipeline library.

  4. Click a pipeline in the list to open it and access its details.
    The pipeline The "Overview" page icon is a square divided into several other squares. Overview page opens.

    Navigate to your pipeline
  5. Click either:

    • The "Instances" page icon is three overlapping squares. Instances to access your pipeline’s instances page and delete instances with their logs from the list.

      • Deleting a pipeline instance deletes all the job instances linked to it, as well as all associated logs.

      • You cannot delete unfinished instances, that is, instances whose status is Requested, Queued, Running, Stopping. Wait for the end of the executions and try again.

      • You cannot delete an instance that has not yet been cleaned from the orchestrator. Wait a moment and try again.

    • The "Versions" page icon is a folder with an arrow pointing up. Versions to access your pipeline’s versions page and delete versions from the list.

      • Deleting a pipeline version deletes the pipeline and job instances linked to it, as well as all its dependencies, such as logs and conditions.

      • You cannot delete the version tagged as Current Badge for the "Current" status. You can revert to another version of the pipeline to change the current one and delete it.

      • You cannot delete a version that is linked to unfinished instances. Wait for the end of the executions and try again.

      • You cannot delete a version whose instances have not yet been cleaned from the orchestrator. Wait a moment and try again.

  6. You can delete either instances or versions of your pipeline.

    • Delete a single instance or version

    • Delete a selection of instances or versions

    • Delete a selection of instances based on filters

    • Delete instances by date

    You can delete a single pipeline instance or version by clicking the trash can delete that appears on the line when you hover over the desired element.

    pipeline deletion instance single

    You can delete a selection of pipeline instances or versions by selecting them in the list and clicking Delete n instances/versions at the top of the list.

    You will then be able to play with the selector by adding or removing elements with different states.

    job deletion instance selection pipeline versions deletion selection

    You can delete a selection of pipeline instances based on status criteria. To do this, select the desired filter and click Delete n instances at the top of the list.

    You will then be able to play with the selector by adding or removing elements with different states.
    job deletion instance filter

    You can delete pipeline instances by date criteria with the All instances older than calendar plus filter. It will delete all instances and associated logs prior to the selected date.

    1. Select the All instances older than calendar plus filter.

      job deletion instance date

      A pop-up window opens with a date picker.

    2. Select a date.

    3. Click Select instances to confirm your choice.
      The corresponding instances are selected in the list.

      You will then be able to play with the selector by adding or removing elements.
    4. Click Delete n instances at the top of the list.

  7. Confirm the pop-up message and the permanent deletion by clicking Delete.

Moving a Pipeline to Another Project

You can move a pipeline from one project to another and on a different platform from the pipeline library or the pipeline The "Overview" page icon is a square divided into several other squares. Overview page. Jobs, conditions, pipeline environment variables, versions, instances, logs, alerts, and resource settings are all included in the moved pipeline.

  • This is not a duplication. When moving a pipeline to another project, the pipeline disappears from the source project.

  • Any project you have access to can move a pipeline in and out.

  • This feature does not support external jobs and connections.

  • You cannot select multiple pipelines at the same time and move them.

  • If the name and alias of a pipeline and pipeline job(s) already exist in the target project, they are renamed with an increment in brackets.

  1. Click The "Projects" module icon is a folder. Projects from the primary navigation menu.
    By default, the The "All Projects" page icon is a folder and the same as the "Projects" module. All Projects page opens with the list of existing projects.

  2. Click a project in the list to open it.
    By default, the The "Jobs" page icon is a 3D pyramid of three squares. Jobs page opens with the list of existing jobs.

    Navigate to your project
  3. Click The "Pipelines" page icon is represented by several connected jobs. Pipelines from the secondary navigation menu to open the project’s pipeline library.

  4. You can click the kebab menu The kebab menu icon is three vertical dots.  project move Move to… at the end of the relevant pipeline row from the The "Pipelines" page icon is represented by several connected jobs. Pipelines page, or from the The "Overview" page icon is a square divided into several other squares. Overview page of a pipeline.

    pipeline move to library
    Figure 4. Move a pipeline from the The "Pipelines" page icon is represented by several connected jobs. Pipeline page

    In Figure 4, the project move Move to… action is in orange. This means that some issue(s) are preventing your pipeline from being moved. You can still click on it; It will display a panel that lists the errors.

    pipeline move to error panel

    The possible issues can be as follows:

    • You cannot move a pipeline that has one or more jobs in use in another pipeline. It can be a job that is used in a different version than the current version of another pipeline. Duplicate your pipeline and move the duplicated pipeline to bypass this issue.

    • You cannot move a pipeline whose instances have not yet been cleaned from the orchestrator. Wait a moment and try again.

    • You cannot move a running and scheduled pipeline, as well as a pipeline containing running and scheduled jobs. Wait for the end of the execution(s) and try to remove the scheduling settings. Then try again.

    • The job technology that is used in the pipeline must be available in the target project. If it does not, select another project or modify the settings of the target project.

    A pop-up window opens.

  5. Select the target platform and target project from the list.

    pipeline move to

  6. Click Move to confirm the move.
    You will see a message saying that your pipeline has been successfully moved. You can click Go to target project below the success message to be redirected to the pipeline library of the target project. Jobs, conditions, pipeline environment variables, versions, instances, logs, alerts, and resource settings are all included in the moved pipeline. There is no obfuscation of the data. If there is sensitive data, it will be visible in the target project.