Saagie pipelines

Use these tutorials to create, modify, and run a Saagie pipeline.

1. Create a Saagie pipeline

Together, let’s create the pipeline Regional Branch Sales Data in the project Dunder Mifflin Paper Sales.

Let’s assume that:

you’ve completed the tutorial Create a Saagie project where you created the project Dunder Mifflin Paper Sales, and
you’ve completed the tutorial Create a Saagie job where you created the job Scranton Branch Sales.

1.1. Create second example job

A pipeline requires at least two jobs, so you’ll need to create another one.

Follow the Create a Saagie job tutorial to create a second job, replacing all instances of Scranton with Utica.

1.2. Navigate to pipelines

Start by navigating to the pipelines homepage within your example project.

Go to the Projects module from the primary navigation menu.
Go to the Dunder Mifflin Paper Sales project.
Go to the Pipelines homepage from the secondary navigation menu.
Select the New pipeline button.

1.3. Enter pipeline name and description

Enter the name of your pipeline: Regional Branch Sales Data
Enter a description of your pipeline: Fetch paper sales data about regional branches.
Select the Continue button.

1.4. Construct pipeline

Let’s construct a pipeline containing our two jobs, Scranton Branch Sales and Utica Branch Sales.

Select the Add Job button in the middle of the pipeline orchestration window.
Choose the Scranton Branch Sales job.
A card labeled Scranton Branch Sales with the Python technology logo and the Extraction label will appear in pipeline orchestration window.
Select the icon to the side of the Scranton Branch Sales card.
Choose the Utica Branch Sales job.
A similar card will appear in pipeline orchestration window, this time labeled Utica Branch Sales.
Select the Continue button.

1.5. Pipeline settings

This page might look familiar to you from the job creation process.

Choose the following settings:

Enable email alerts for when the pipeline status changes to failed and succeeded.
Add a release note to your pipeline: This pipeline is a test to show that I can create a pipeline and everything that goes into it!
Schedule your pipeline to run every other day at midnight CET (which is 10:00 pm or 22h00 UTC).
Select the Create pipeline button.

Your pipeline is ready to go!

2. Run a pipeline

Now that you’ve created a pipeline on your Saagie platform, let’s run it.

Let’s assume that you’ve completed the tutorial Create a Saagie pipeline where you created the pipeline Regional Branch Sales Data.

2.1. Navigate to pipeline

Start by navigating to the pipeline.

Go to the Projects module from the primary navigation menu.
Go to the Dunder Mifflin Paper Sales project.
Go to the Pipelines homepage from the secondary navigation menu.
Go to the Regional Branch Sales Data pipeline.

You can run the pipeline directly from the list of pipelines. For the purposes of this tutorial, however, let’s go to our pipeline’s homepage to run it.

2.2. Run pipeline

Select the Run button located in the secondary navigation panel.

You can run a pipeline from all pages concerning that pipeline: overview, instances, and versions.
After about a minute, refresh your page. (Pipeline run times vary.)
If the pipeline ran successfully, you’ll see a Succeeded badge above the Run button.

Did you notice the changes to the secondary navigation panel after selecting Run?

The status area changes as the job runs.
The Run button became a Stop button.

3. Modify a pipeline’s settings

Together, let’s modify the settings for the pipeline Regional Branch Sales Data in the project Dunder Mifflin Paper Sales.

Let’s assume that:

you’ve completed the tutorial Create a Saagie project where you created the project Dunder Mifflin Paper Sales, and
you’ve completed the tutorial Create a Saagie pipeline where you created the pipeline Regional Branch Sales Data.

3.1. Navigate to pipeline

Start by navigating to the pipeline.

Go to the Projects module from the primary navigation menu.
Go to the Dunder Mifflin Paper Sales project.
Go to the Pipelines homepage from the secondary navigation menu.
Go to the Regional Branch Sales Data pipeline.

3.2. Go to settings

Notice the section on the pipeline’s overview page displaying the current settings: pipeline name, description, run type, and alerts. We will modify some of this information.
Go to Pipeline settings.

You can access a pipeline’s settings from all pages concerning that pipeline: overview, instances, and versions.

3.3. Change name and description

We won’t actually change the name or description of the pipeline, but note that you can change either of these at any time.

3.4. Change email alerts

Alerts are already enabled for this pipeline. Let’s add an alert when this pipeline succeeds, and remove the alert when a pipeline is killed.

Select the Succeeded checkbox.
Clear the Killed checkbox.
Make no changes to the Failed checkbox.

3.5. Change run type

Finally, let’s modify the pipeline’s scheduled run.

Scheduled run should already be selected.
Choose Shortcut.
Choose @weekly.

Notice that a weekly run in shortcut mode will run every Sunday at midnight UTC.
Select the Save settings button.

Your pipeline settings are updated. Great work!

4. Upgrade a pipeline

Let’s upgrade the pipeline Regional Branch Sales Data in the project Dunder Mifflin Paper Sales.

Let’s assume that:

you’ve completed the tutorial Create a Saagie project where you created the project Dunder Mifflin Paper Sales, and
you’ve completed the tutorial Create a Saagie pipeline where you created the pipeline Regional Branch Sales Data.

4.1. Create third example job

We’ll upgrade this pipeline by adding another job, so start by creating a third job.

Follow the Create a Saagie job tutorial to create a third job, replacing all instances of Scranton with Albany.

4.2. Navigate to pipeline

Next, navigate to our example pipeline.

Go to the Projects module from the primary navigation menu.
Go to the Dunder Mifflin Paper Sales project.
Go to the Pipelines homepage from the secondary navigation menu.
Go to the Regional Branch Sales Data pipeline.

4.3. Begin pipeline upgrade

Let’s upgrade the pipeline.

Notice some of the information displayed about this pipeline version, specifically which jobs run in the pipeline and this version’s release note. Some of this information will change with the pipeline upgrade, which will pass this pipeline from version 1 to version 2.
Select the Upgrade pipeline button.

You can upgrade a pipeline from all pages concerning that pipeline: overview, instances, and versions.

4.4. Select pipeline upgrades

Let’s add a third job to the pipeline.

Select the icon to the side of the Utica Branch Sales card.

You can add jobs anywhere in the pipeline.
Choose the Albany Branch Sales job.
A card labeled Albany Branch Sales with the Python technology logo and the Extraction label will appear in pipeline orchestration window. This means the job was added to the pipeline successfully.
Select the Continue button.

4.5. Add a new release note

Finally, let’s add a release note to describe what changed with this upgrade.

Select the Add a release note for this pipeline version checkbox. An empty field will appear.
Enter the release note text: Added job Albany Branch Sales to pipeline.
Select the Upgrade pipeline button.

On the pipeline’s home screen, it will show Version 2 underneath the title Current version, along with the release note. You can also see the job Albany Branch Sales in the pipeline orchestration window.

Focused view of changes made in this upgrade

You’ve now upgraded to a new version of your pipeline. Well done!