Saagie pipelines

Use these tutorials to create, modify, and run a Saagie pipeline.

1. Create a Saagie pipeline

Together, let’s create the pipeline Regional Branch Sales Data in the project Dunder Mifflin Paper Sales.

Let’s assume that:

  • you’ve completed the tutorial Create a Saagie project where you created the project Dunder Mifflin Paper Sales, and

  • you’ve completed the tutorial Create a Saagie job where you created the job Scranton Branch Sales.

1.1. Create second example job

A pipeline requires at least two jobs, so you’ll need to create another one.

  1. Follow the Create a Saagie job tutorial to create a second job, replacing all instances of Scranton with Utica.

Create second job for pipeline tutorial

1.2. Navigate to pipelines

Start by navigating to the pipelines homepage within your example project.

  1. Go to the Projects module from the primary navigation menu.

  2. Go to the Dunder Mifflin Paper Sales project.

    Navigate to your project
  3. Go to the Pipelines homepage from the secondary navigation menu.

  4. Select the New pipeline button.

    Verify location and start creating new pipeline

1.3. Enter pipeline name and description

  1. Enter the name of your pipeline: Regional Branch Sales Data

  2. Enter a description of your pipeline: Fetch paper sales data about regional branches.

  3. Select the Continue button.

Enter name and description for pipeline

1.4. Construct pipeline

Let’s construct a pipeline containing our two jobs, Scranton Branch Sales and Utica Branch Sales.

  1. Select the Add Job button in the middle of the pipeline orchestration window.

    Add job to pipeline
  2. Choose the Scranton Branch Sales job.

    Choose job to add to pipeline
  3. A card labeled Scranton Branch Sales with the Python technology logo and the Extraction label will appear in pipeline orchestration window.

  4. Select the icon Icon to add a project to end of a pipeline to the side of the Scranton Branch Sales card.

    Pipeline with one job added
  5. Choose the Utica Branch Sales job.

    Choose second job to add to pipeline
  6. A similar card will appear in pipeline orchestration window, this time labeled Utica Branch Sales.

  7. Select the Continue button.

    Pipeline with two jobs added

1.5. Pipeline settings

This page might look familiar to you from the job creation process.

Choose the following settings:

  1. Enable email alerts for when the pipeline status changes to failed and succeeded.

  2. Add a release note to your pipeline: This pipeline is a test to show that I can create a pipeline and everything that goes into it!

  3. Schedule your pipeline to run every other day at midnight CET (which is 10:00 pm or 22h00 UTC).

  4. Select the Create pipeline button.

Pipeline settings

Your pipeline is ready to go!

2. Run a pipeline

Now that you’ve created a pipeline on your Saagie platform, let’s run it.

Let’s assume that you’ve completed the tutorial Create a Saagie pipeline where you created the pipeline Regional Branch Sales Data.

2.1. Navigate to pipeline

Start by navigating to the pipeline.

  1. Go to the Projects module from the primary navigation menu.

  2. Go to the Dunder Mifflin Paper Sales project.

    Navigate to your project
  3. Go to the Pipelines homepage from the secondary navigation menu.

  4. Go to the Regional Branch Sales Data pipeline.

    You can run the pipeline directly from the list of pipelines. For the purposes of this tutorial, however, let’s go to our pipeline’s homepage to run it.
    Navigate to your project

2.2. Run pipeline

  1. Select the Run button located in the secondary navigation panel.

    You can run a pipeline from all pages concerning that pipeline: overview, instances, and versions.
  2. After about a minute, refresh your page. (Pipeline run times vary.)

  3. If the pipeline ran successfully, you’ll see a Succeeded badge above the Run button.

Run a pipeline

Did you notice the changes to the secondary navigation panel after selecting Run?

  • The status area changes as the job runs.

  • The Run button became a Stop button.

    Job or pipeline run in progress

3. Modify a pipeline’s settings

Together, let’s modify the settings for the pipeline Regional Branch Sales Data in the project Dunder Mifflin Paper Sales.

Let’s assume that:

  • you’ve completed the tutorial Create a Saagie project where you created the project Dunder Mifflin Paper Sales, and

  • you’ve completed the tutorial Create a Saagie pipeline where you created the pipeline Regional Branch Sales Data.

3.1. Navigate to pipeline

Start by navigating to the pipeline.

  1. Go to the Projects module from the primary navigation menu.

  2. Go to the Dunder Mifflin Paper Sales project.

    Navigate to your project
  3. Go to the Pipelines homepage from the secondary navigation menu.

  4. Go to the Regional Branch Sales Data pipeline.

    Navigate to your project

3.2. Go to settings

  1. Notice the section on the pipeline’s overview page displaying the current settings: pipeline name, description, run type, and alerts. We will modify some of this information.

  2. Go to Pipeline settings.

    You can access a pipeline’s settings from all pages concerning that pipeline: overview, instances, and versions.
Navigate to pipeline settings

3.3. Change name and description

We won’t actually change the name or description of the pipeline, but note that you can change either of these at any time.

Leave pipeline name and description

3.4. Change email alerts

Alerts are already enabled for this pipeline. Let’s add an alert when this pipeline succeeds, and remove the alert when a pipeline is killed.

  1. Select the Succeeded checkbox.

  2. Clear the Killed checkbox.

  3. Make no changes to the Failed checkbox.

Modify email alerts

3.5. Change run type

Finally, let’s modify the pipeline’s scheduled run.

  1. Scheduled run should already be selected.

  2. Choose Shortcut.

  3. Choose @weekly.

    Notice that a weekly run in shortcut mode will run every Sunday at midnight UTC.
  4. Select the Save settings button.

Modify pipeline run type

Your pipeline settings are updated. Great work!

4. Upgrade a pipeline

Let’s upgrade the pipeline Regional Branch Sales Data in the project Dunder Mifflin Paper Sales.

Let’s assume that:

  • you’ve completed the tutorial Create a Saagie project where you created the project Dunder Mifflin Paper Sales, and

  • you’ve completed the tutorial Create a Saagie pipeline where you created the pipeline Regional Branch Sales Data.

4.1. Create third example job

We’ll upgrade this pipeline by adding another job, so start by creating a third job.

  1. Follow the Create a Saagie job tutorial to create a third job, replacing all instances of Scranton with Albany.

Create third job for pipeline tutorial

4.2. Navigate to pipeline

Next, navigate to our example pipeline.

  1. Go to the Projects module from the primary navigation menu.

  2. Go to the Dunder Mifflin Paper Sales project.

    Navigate to your project
  3. Go to the Pipelines homepage from the secondary navigation menu.

  4. Go to the Regional Branch Sales Data pipeline.

    Navigate to your project

4.3. Begin pipeline upgrade

Let’s upgrade the pipeline.

  1. Notice some of the information displayed about this pipeline version, specifically which jobs run in the pipeline and this version’s release note. Some of this information will change with the pipeline upgrade, which will pass this pipeline from version 1 to version 2.

  2. Select the Upgrade pipeline button.

    You can upgrade a pipeline from all pages concerning that pipeline: overview, instances, and versions.
Upgrade pipeline

4.4. Select pipeline upgrades

Let’s add a third job to the pipeline.

  1. Select the icon to the side of the Utica Branch Sales card.

    You can add jobs anywhere in the pipeline.
    Add job to pipeline
  2. Choose the Albany Branch Sales job.

    Choose third job to add to pipeline
  3. A card labeled Albany Branch Sales with the Python technology logo and the Extraction label will appear in pipeline orchestration window. This means the job was added to the pipeline successfully.

  4. Select the Continue button.

    Pipeline with three jobs added

4.5. Add a new release note

Finally, let’s add a release note to describe what changed with this upgrade.

  1. Select the Add a release note for this pipeline version checkbox. An empty field will appear.

  2. Enter the release note text: Added job Albany Branch Sales to pipeline.

  3. Select the Upgrade pipeline button.

Add release note to pipeline upgrade

On the pipeline’s home screen, it will show Version 2 underneath the title Current version, along with the release note. You can also see the job Albany Branch Sales in the pipeline orchestration window.

Focused view of changes made in this upgrade

You’ve now upgraded to a new version of your pipeline. Well done!