Saagie pipelines

Use these tutorials to create, modify, and run a Saagie pipeline.

1. Create a Saagie pipeline

Together, let’s create the pipeline Regional Branch Sales Data in the project Dunder Mifflin Paper Sales.

Let’s assume that:

  • you’ve completed the tutorial Create a Saagie project where you created the project Dunder Mifflin Paper Sales, and

  • you’ve completed the tutorial Create a Saagie job where you created the job Scranton Branch Sales.

1.1. Create second example job

This section is optional - you can ignore it except if the job Scranton Branch Sales in the project Dunder Mifflin Paper Sales does not already exist on the platform.

A pipeline requires at least two jobs, so you’ll need to create another one.

  1. Follow the Create a Saagie job tutorial to create a second job, replacing all instances of Scranton with Utica.

Create second job for pipeline tutorial

1.2. Navigate to pipelines

Start by navigating to the pipelines homepage within your example project.

  1. Go to the Projects module from the primary navigation menu.

  2. Go to the Dunder Mifflin Paper Sales project.

    Navigate to your project
  3. Go to the Pipelines homepage from the secondary navigation menu.

  4. Select the New pipeline button.

    Verify location and start creating new pipeline

1.3. Enter pipeline name and description

  1. Enter the name of your pipeline: Regional Branch Sales Data

  2. Enter a description of your pipeline: Fetch paper sales data about regional branches.

Enter name and description for pipeline

1.4. Construct pipeline

Let’s construct a pipeline containing our two jobs, Scranton Branch Sales and Utica Branch Sales.

  1. Select the Add job button.

    Add job to pipeline
  2. Choose the Scranton Branch Sales job.

    Choose job to add to pipeline
    1. A card labeled Scranton Branch Sales with the Python technology logo and the Extraction label will appear in pipeline orchestration window.

    2. Drag the job card where you want.

  3. Select the Add job button.

  4. Choose the Utica Branch Sales job.

    Choose second job to add to pipeline
    1. A similar card will appear in pipeline orchestration window, this time labeled Utica Branch Sales.

    2. Drag the job card where you want.

      Pipeline with two jobs added
  5. Enable email alerts.

    Enable pipeline alerts
    1. Choose alerts for when the pipeline status changes to failed and succeeded.

      Set pipeline alerts
  6. Enable scheduling.

    Enable pipeline scheduling
    1. Schedule your pipeline to run every other day at midnight UTC +01:00 Europe/Paris.

      Set pipeline scheduling
  7. Enable release notes.

    Enable pipeline release note
    1. Add this note: This pipeline is a test to show that I can create a pipeline and everything that goes into it!

      Set pipeline release note
  8. Choose one of the following options to finish the pipeline creation:

    1. Select the Save pipeline button to save all modifications.

      Save pipeline
    2. select the Save & run button to save your modifications and automatically run the last version.

      Save & run pipeline
    3. select the Exit cross button to abandon all modifications.

      Exit pipeline creation

Your pipeline is ready to go!

2. Run a pipeline

Now that you’ve created a pipeline on your Saagie platform, let’s run it.

Let’s assume that you’ve completed the tutorial Create a Saagie pipeline where you created the pipeline Regional Branch Sales Data.

2.1. Navigate to pipeline

Start by navigating to the pipeline.

  1. Go to the Projects module from the primary navigation menu.

  2. Go to the Dunder Mifflin Paper Sales project.

    Navigate to your project
  3. Go to the Pipelines homepage from the secondary navigation menu.

  4. Go to the Regional Branch Sales Data pipeline.

    You can run the pipeline directly from the list of pipelines. For the purposes of this tutorial, however, let’s go to our pipeline’s homepage to run it.
    Navigate to your project

2.2. Run pipeline

  1. Select the Run button located in the secondary navigation panel.

    You can run a pipeline from all pages concerning that pipeline: overview, instances, and versions.
  2. If the pipeline ran successfully, you’ll see a Succeeded badge above the Run button.

Run a pipeline

Did you notice the changes to the secondary navigation panel after selecting Run?

  • The status area changes as the job runs.

  • The Run button became a Stop button.

    Job or pipeline run in progress

3. Modify a pipeline’s settings

Together, let’s modify the settings for the pipeline Regional Branch Sales Data in the project Dunder Mifflin Paper Sales.

Let’s assume that:

  • you’ve completed the tutorial Create a Saagie project where you created the project Dunder Mifflin Paper Sales, and

  • you’ve completed the tutorial Create a Saagie pipeline where you created the pipeline Regional Branch Sales Data.

3.1. Navigate to pipeline

Start by navigating to the pipeline.

  1. Go to the Projects module from the primary navigation menu.

  2. Go to the Dunder Mifflin Paper Sales project.

    Navigate to your project
  3. Go to the Pipelines homepage from the secondary navigation menu.

  4. Go to the Regional Branch Sales Data pipeline.

    Navigate to your project

3.2. Go to settings

  1. Notice the section on the pipeline’s overview page displaying the current settings: pipeline name, description, run type, and alerts. We will modify some of this information.

  2. Go to Pipeline settings.

    You can access a pipeline’s settings from all pages concerning that pipeline: overview, instances, and versions.
Navigate to pipeline settings

3.3. Change name and description

We won’t actually change the name or description of the pipeline, but note that you can change either of these at any time.

Leave pipeline name and description

3.4. Change email alerts

Alerts are already enabled for this pipeline. Let’s add an alert when this pipeline succeeds or get the unknown status, and remove the alert when a pipeline is killed.

  1. Select the Succeeded checkbox.

  2. Select the Unknown button.

  3. Make no changes to the Failed checkbox.

  4. Clear the Killed checkbox.

Modify email alerts

3.5. Change run type

Finally, let’s modify the pipeline’s scheduled run.

  1. Scheduled run should already be selected.

  2. Choose Shortcut.

  3. Choose @weekly.

    Notice that a weekly run in shortcut mode will run every Sunday at midnight.
  4. Select the timezone UTC +01:00 Europe/Paris

  5. Select the Save settings button.

Modify pipeline run type

Your pipeline settings are updated. Great work!

4. Upgrade a pipeline

Let’s upgrade the pipeline Regional Branch Sales Data in the project Dunder Mifflin Paper Sales.

Let’s assume that:

  • you’ve completed the tutorial Create a Saagie project where you created the project Dunder Mifflin Paper Sales, and

  • you’ve completed the tutorial Create a Saagie pipeline where you created the pipeline Regional Branch Sales Data.

4.1. Create a complex pipeline example

We’ll upgrade this pipeline by adding job, so start by creating three jobs and one condition.

  1. Follow the Create a Saagie job tutorial to create three jobs, replacing all instances of Scranton with Albany and repeat with Rochester, Northeast and New York corporate HQ.

Create third job for pipeline tutorial

4.2. Navigate to pipeline

Next, navigate to our example pipeline.

  1. Go to the Projects module from the primary navigation menu.

  2. Go to the Dunder Mifflin Paper Sales project.

    Navigate to your project
  3. Go to the Pipelines homepage from the secondary navigation menu.

  4. Go to the Regional Branch Sales Data pipeline.

    Navigate to your project

4.3. Begin pipeline upgrade

Let’s upgrade the pipeline.

  1. Notice some of the information displayed about this pipeline version, specifically which jobs run in the pipeline and this version’s release note. Some of this information will change with the pipeline upgrade, which will pass this pipeline from version 1 to version 2.

  2. Select the Upgrade pipeline button.

    You can upgrade a pipeline from all pages concerning that pipeline: overview, instances, and versions.
Upgrade pipeline

4.4. Select pipeline upgrades

Let’s add a third job to the pipeline.

  1. Select the Add Job button in the middle of the pipeline orchestration window.

    Add job to pipeline
  2. Choose the Albany Branch Sales job.

    Choose job to add to pipeline
    1. A card labeled Albany Branch Sales with the Python technology logo and the Extraction label will appear in pipeline orchestration window.

    2. Drag the job card where you want.

  3. Select the Add condition button.

    Add condition to pipeline
    1. Drag the condition where you want.

    2. Link the Scranton Branch and Utica Branch jobs to the condition input.

      Add condition to two jobs pipeline
  4. Select the Add job button.

  5. Choose the New York corporate HQ job.

    Choose second job to add to pipeline
    1. A similar card will appear in pipeline orchestration window, this time labeled New York corporate HQ.

    2. Drag the job card where you want.

    3. Assuming you already created the following jobs and added to the pipeline: Rochester branch, Albany branch, Northeast branch.

    4. Link jobs and condition as follows:

      Pipeline complex
  6. Enable release notes.

    Enable pipeline release note
    1. Add this note: This a complex pipeline including condition and parallel branches.

      Set pipeline release note for third job
  7. Choose one of the following options to finish the pipeline creation:

    1. Select the Save pipeline button to save all modifications.

      Save pipeline
    2. select the Save & run button to save your modifications and automatically run the last version.

On the pipeline’s home screen, it will show Version 2 underneath the title Current version, along with the release note.
You can also see jobs and conditions in the pipeline orchestration window.

Focused view of changes made in this upgrade

You’ve now upgraded to a new version of your pipeline. Well done!