Saagie pipelines

Use these tutorials to create, modify, and run a Saagie pipeline.

1. Create a Saagie pipeline

Together, let’s create the pipeline Regional Branch Sales Data in the project Dunder Mifflin Paper Sales.

Let’s assume that:

you’ve completed the tutorial Create a Saagie project where you created the project Dunder Mifflin Paper Sales, and
you’ve completed the tutorial Create a Saagie job where you created the job Scranton Branch Sales.

1.1. Create second example job

This section is optional - you can ignore it except if the job Scranton Branch Sales in the project Dunder Mifflin Paper Sales does not already exist on the platform.

A pipeline requires at least two jobs, so you’ll need to create another one.

Follow the Create a Saagie job tutorial to create a second job, replacing all instances of Scranton with Utica.

1.2. Navigate to pipelines

Start by navigating to the pipelines homepage within your example project.

Go to the Projects module from the primary navigation menu.
Go to the Dunder Mifflin Paper Sales project.
Go to the Pipelines homepage from the secondary navigation menu.
Select the New pipeline button.

1.3. Enter pipeline name and description

Enter the name of your pipeline: Regional Branch Sales Data
Enter a description of your pipeline: Fetch paper sales data about regional branches.

1.4. Construct pipeline

Let’s construct a pipeline containing our two jobs, Scranton Branch Sales and Utica Branch Sales.

Select the Add job button.
Choose the Scranton Branch Sales job.
1. A card labeled Scranton Branch Sales with the Python technology logo and the Extraction label will appear in pipeline orchestration window.
2. Drag the job card where you want.
Select the Add job button.
Choose the Utica Branch Sales job.
1. A similar card will appear in pipeline orchestration window, this time labeled Utica Branch Sales.
2. Drag the job card where you want.
Enable email alerts.
1. Choose alerts for when the pipeline status changes to failed and succeeded.
Enable scheduling.
1. Schedule your pipeline to run every other day at midnight UTC +01:00 Europe/Paris.
Enable release notes.
1. Add this note: This pipeline is a test to show that I can create a pipeline and everything that goes into it!
Choose one of the following options to finish the pipeline creation:
1. Select the Save pipeline button to save all modifications.
2. select the Save & run button to save your modifications and automatically run the last version.
3. select the Exit cross button to abandon all modifications.

Your pipeline is ready to go!

2. Run a pipeline

Now that you’ve created a pipeline on your Saagie platform, let’s run it.

Let’s assume that you’ve completed the tutorial Create a Saagie pipeline where you created the pipeline Regional Branch Sales Data.

2.1. Navigate to pipeline

Start by navigating to the pipeline.

Go to the Projects module from the primary navigation menu.
Go to the Dunder Mifflin Paper Sales project.
Go to the Pipelines homepage from the secondary navigation menu.
Go to the Regional Branch Sales Data pipeline.

You can run the pipeline directly from the list of pipelines. For the purposes of this tutorial, however, let’s go to our pipeline’s homepage to run it.

2.2. Run pipeline

Select the Run button located in the secondary navigation panel.

You can run a pipeline from all pages concerning that pipeline: overview, instances, and versions.
If the pipeline ran successfully, you’ll see a Succeeded badge above the Run button.

Did you notice the changes to the secondary navigation panel after selecting Run?

The status area changes as the job runs.
The Run button became a Stop button.

3. Modify a pipeline’s settings

Together, let’s modify the settings for the pipeline Regional Branch Sales Data in the project Dunder Mifflin Paper Sales.

Let’s assume that:

you’ve completed the tutorial Create a Saagie project where you created the project Dunder Mifflin Paper Sales, and
you’ve completed the tutorial Create a Saagie pipeline where you created the pipeline Regional Branch Sales Data.

3.1. Navigate to pipeline

Start by navigating to the pipeline.

Go to the Projects module from the primary navigation menu.
Go to the Dunder Mifflin Paper Sales project.
Go to the Pipelines homepage from the secondary navigation menu.
Go to the Regional Branch Sales Data pipeline.

3.2. Go to settings

Notice the section on the pipeline’s overview page displaying the current settings: pipeline name, description, run type, and alerts. We will modify some of this information.
Go to Pipeline settings.

You can access a pipeline’s settings from all pages concerning that pipeline: overview, instances, and versions.

3.3. Change name and description

We won’t actually change the name or description of the pipeline, but note that you can change either of these at any time.

3.4. Change email alerts

Alerts are already enabled for this pipeline. Let’s add an alert when this pipeline succeeds or get the unknown status, and remove the alert when a pipeline is killed.

Select the Succeeded checkbox.
Select the Unknown button.
Make no changes to the Failed checkbox.
Clear the Killed checkbox.

3.5. Change run type

Finally, let’s modify the pipeline’s scheduled run.

Scheduled run should already be selected.
Choose Shortcut.
Choose @weekly.

Notice that a weekly run in shortcut mode will run every Sunday at midnight.
Select the timezone UTC +01:00 Europe/Paris
Select the Save settings button.

Your pipeline settings are updated. Great work!

4. Upgrade a pipeline

Let’s upgrade the pipeline Regional Branch Sales Data in the project Dunder Mifflin Paper Sales.

Let’s assume that:

you’ve completed the tutorial Create a Saagie project where you created the project Dunder Mifflin Paper Sales, and
you’ve completed the tutorial Create a Saagie pipeline where you created the pipeline Regional Branch Sales Data.

4.1. Create a complex pipeline example

We’ll upgrade this pipeline by adding job, so start by creating three jobs and one condition.

Follow the Create a Saagie job tutorial to create three jobs, replacing all instances of Scranton with Albany and repeat with Rochester, Northeast and New York corporate HQ.

4.2. Navigate to pipeline

Next, navigate to our example pipeline.

Go to the Projects module from the primary navigation menu.
Go to the Dunder Mifflin Paper Sales project.
Go to the Pipelines homepage from the secondary navigation menu.
Go to the Regional Branch Sales Data pipeline.

4.3. Begin pipeline upgrade

Let’s upgrade the pipeline.

Notice some of the information displayed about this pipeline version, specifically which jobs run in the pipeline and this version’s release note. Some of this information will change with the pipeline upgrade, which will pass this pipeline from version 1 to version 2.
Select the Upgrade pipeline button.

You can upgrade a pipeline from all pages concerning that pipeline: overview, instances, and versions.

4.4. Select pipeline upgrades

Let’s add a third job to the pipeline.

Select the Add Job button in the middle of the pipeline orchestration window.
Choose the Albany Branch Sales job.
1. A card labeled Albany Branch Sales with the Python technology logo and the Extraction label will appear in pipeline orchestration window.
2. Drag the job card where you want.
Select the Add condition button.
1. Drag the condition where you want.
2. Link the Scranton Branch and Utica Branch jobs to the condition input.
Select the Add job button.
Choose the New York corporate HQ job.
1. A similar card will appear in pipeline orchestration window, this time labeled New York corporate HQ.
2. Drag the job card where you want.
3. Assuming you already created the following jobs and added to the pipeline: Rochester branch, Albany branch, Northeast branch.
4. Link jobs and condition as follows:
Enable release notes.
1. Add this note: This a complex pipeline including condition and parallel branches.
Choose one of the following options to finish the pipeline creation:
1. Select the Save pipeline button to save all modifications.
2. select the Save & run button to save your modifications and automatically run the last version.

On the pipeline’s home screen, it will show Version 2 underneath the title Current version, along with the release note.
You can also see jobs and conditions in the pipeline orchestration window.

Focused view of changes made in this upgrade

You’ve now upgraded to a new version of your pipeline. Well done!