Release notes

March 2022

Here are the highlights of new and updated features for this release:

  1. Product Updates (v2.2.1)
    The v.2.2.1 version of the Saagie product has been released with the following features:

    • Breaking changes in the Saagie API behavior have been made.

    • The user experience in pipelines has been improved.

    • Multi-context management is now available for App technologies in the Saagie’s official repository of the Technology Catalog.

    • Bugs in the pipeline execution have been fixed.

  2. Saagie’s Technology Repository Updates
    New technology versions have been added and others have been deprecated.

Product Updates (v2.2.1)

Saagie API behavior

Breaking changes have been made to the Saagie GraphQL API.

The following deprecated endpoints have been removed from the https://<your_saagie_host>/projects/api/platform/<your_platform_id>/graphql: endpoint server:

  • technologies(projectId: UUID, jobCategory: String): [Technology!]

  • technology(id: UUID): Technology

  • technologiesVersions(technologyId: UUID!): [TechnologyVersion!]!

These endpoints have been replaced with the following in the https://<your_saagie_host>/gateway/api/graphql: endpoint server:

  • technologiesByIds(ids: [UUID!]!): [Technology!]!

  • technology(id: UUID!): Technology

User experience improvements in pipelines

User experience improvements have been made to the pipeline design interface:

  • Adding jobs to a pipeline now uses smart positioning on the design grid to prevent jobs from stacking on top of each other when adding multiple jobs in a row.

  • You can automatically rearrange jobs and conditions to the most optimized layout based on the defined pipeline structure by clicking the magic wand.
    This saves you from having to manually drag jobs and conditions into the pipeline design grid to arrange them legibly.

  • Error handling has been improved by preventing the creation of inconsistent pipelines starting or ending with a condition.

  • In addition to the error messages displayed when trying to save an invalid pipeline, the jobs and conditions causing the errors are now clearly highlighted on the design grid so you can quickly identify the cause of the issue.

Multi-context management of Saagie’s official repository App technologies

Similarly to how Job technologies appear in the Saagie repository, all contexts associated with an App technology are now listed under a single entry.

Apps multi-context menu

This improves the readability in the Catalog and Project setting interfaces and allows you to quickly select the App technology context to use when installing an App from the Saagie’s official repository.

Existing Apps have been migrated to a multi-context architecture and former mono-context App technologies have been disabled.

Bug fixes in pipeline execution

Fixes have been added to pipelines to solve execution issues:

  • Pipeline instances now take into account downstream jobs when their parent job returns a “Succeeded” status.

  • The time zone selected when running a scheduled pipeline is now saved at the pipeline creation.

Saagie’s Technology Repository Updates

New technology versions
The following technology versions have been added to the official Saagie technology repository:

  • Python 3.10

  • Jupyter Notebook with Python 3.10

  • JupyterLab with Python 3.10

  • Kibana 7.15.1

Don’t forget to update your Saagie repositories to keep your technologies up to date.

Deprecated technologies
The following technology versions have been deprecated:

  • Jupyter Notebook for Spark 2.4

  • Jupyter Notebook with Python 3.6

Obsolete technologies are tagged as deprecated to indicate that these runtime contexts will be removed.
Jobs, pipelines, and apps using a deprecated technology will work until further notice, but you will not be able to create new ones.
We recommend that you upgrade your technology to use a more up-to-date runtime context.

January 2022

Here are the highlights of new and updated features for this release:

  1. Saagie known issues

  2. Deprecated technologies

Saagie known issues

Deprecated technologies

The following technologies have been deprecated:

  • R 4.0

  • MLFlow Server 1.16

  • RStudio 3.6.2

  • Kibana 5.6.3

  • Kibana 6.4.3

  • Kibana 6.8.4

Obsolete technologies are tagged as deprecated to indicate that those runtime contexts will be removed soon.
Jobs, pipelines, and apps using a deprecated technology should be upgraded to use a more up-to-date runtime context.

December 2021

This update focuses on new features, added technologies, user experience improvements and bug fixes.

Here are the highlights of the update:

  1. Conditional pipelines,

  2. Technology deprecation management in UI,

  3. CPU/RAM limit per job,

  4. Timezone & DST management in job and pipeline scheduling,

  5. Physical deletion of projects, jobs & other assets,

  6. New technology versions,

  7. Fixes to Spark technology,

  8. Auto-refresh of job, pipeline & app status.

Conditional pipelines

Users can now create conditional pipelines allowing the run of multiple jobs in parallel as well as branching conditioned by the previous job’s success or fail status.

A single pipeline now has the ability to cover multiple execution scenarios, giving users a more centralized control and monitoring over their complete workflows.

Conditional-pipeline

Technology deprecation management in UI

Deprecated technologies are now tagged as such in technology repositories, and when used by a job.

Saagie tags obsolete technologies as deprecated in its official repository to indicate users that those runtime contexts will be removed after 6 months. Jobs using a deprecated technology should be upgraded to use a more up-to-date runtime context.

Deprecated technology

CPU/RAM limit per job

Resource allocation quotas can now be set for each job to control their resource consumption.

Users can set both guaranteed resources and limits for CPU and RAM usage. Setting guaranteed resources ensures that this amount of resources will be allocated to the job during its execution. On the other hand, limits prevent the job from over-consuming resources during its execution, therefore allowing to share platform resources across jobs properly.

Resource-quotas

Timezone & DST management in job and pipeline scheduling

Users can now select timezones when setting up scheduled runs for jobs and pipelines.

Time also automatically adjusts for daylight-saving time.

Timezone & Scheduling

Physical deletion of projects, jobs, and other assets

Users can now physically delete projects, jobs, pipelines, environment variables, and Docker credentials from their platforms. Those assets were previously only archived and were still taking up storage resources, which could cause saturation issues.

New technology versions

The following technologies versions have been added:

  • Spark 3.1.2 Java/Scala 11, 3.1.2 python 3.7

  • R 4.1

  • R Studio 4.1

Don’t forget to synchronize your Saagie repositories to keep them up to date.

Fixes to Spark technology

Fixes have been applied to all Spark runtime contexts available in Saagie repositories. They solve an issue where a Spark job’s status could appear as Succeeded in a pipeline, even though the job had actually failed.

  1. To apply these fixes, users need to synchronize their Saagie repositories.

  2. Users having specified the command spark-submit --deploy-mode cluster in the command line used by their Spark jobs will also need to remove it as Spark now needs to be deployed in client mode.

Without those two actions, Spark jobs will fail.

Auto-refresh of job, pipeline & app status

The status of jobs, pipelines and apps now automatically refreshes without needing any user input. This streamlines user experience as users could previously think a job, pipeline or app was stuck in “queued” upon its run, when in reality its status was updating but wasn’t being displayed in the UI unless a user manually updated it.

October 2021

Here are the highlights of new and updated features for this release:

  1. New technologies versions

  2. Deprecated technologies

New technologies versions

The following technologies versions have been added:

  • MLFlow Server 1.20

  • Grafana 8.2

  • Jupyterlab notebook with Python 3.8 and 3.9 kernels

  • Python 3.8

  • Python 3.9

  • Talend Java_11

  • Bash Debian-10-Buster

  • Bash Debian-10-Buster-aws

  • Bash Debian-10-Buster-azure

  • Bash Debian-10-Buster-gcp

Don’t forget to synchronize your Saagie repositories to keep them up to date.

Deprecated technologies

The following technologies have been deprecated:

  • Talend Java_8

  • Bash Debian-9-stretch

  • Bash Debian-9-stretch-aws

  • Bash Debian-9-stretch-azure

  • Bash Debian-9-stretch-gcp

Jobs, pipelines and apps using them will work until further notice, but you will not be able to create new ones, we recommend using others contexts/versions when available.

September 2021

This release focuses on bug fixes, platform stability, security enhancements and user experience improvements.

Here are the highlights of new and updated features for this release:

  1. Customizable default platform sender email

  2. Sample project available in new platform installation

  3. New technologies versions

  4. Deprecated technologies

Customizable default platform sender email

Now the default platform sender email address noreply@saagie.com is configurable when installing or updating your platform.

Sample project available in new platform installation

A new sample project including three jobs is now available in every new platform installation to help you get started with our basic concepts right after your first login.

You can upgrade or delete this project and/or its jobs at your convenience.

New technologies versions

The following technologies versions have been added or upgraded:

  • MLFlow Server 1.16

  • Spark 2.4.8 (replaces Spark 2.4.5)

  • Spark 3.0.3 (replaces Spark 3.0.0)

  • Spark 3.1.2 Java/Scala 11, 3.1.2 python 3.7

Don’t forget to synchronize your Saagie repositories to keep them up to date.

Deprecated technologies

The following technologies have been deprecated:

Jobs, pipelines and apps using them will work until further notice, but you will not be able to create new ones, we recommend using others contexts/versions when available.
Table 1. Deprecated job technologies
Job technology Deprecated contexts

Java/Scala

7, 8

Python

2.7, 3.5, 3.6

R

3.4.4, 3.5.3, 3.6.2

Spark

2.4 Java/Scala 8
2.4 Java/Scala 11
2.4 Python 2.7
2.4 Python 3.5
2.4 Python 3.6
2.4 Python 3.7
3.0 Java/Scala 8
3.0 Python 3.5
3.0 Python 3.6
3.1 python 3.6
3.1 Java/Scala 8

Table 2. Deprecated app technologies
App technology Deprecated contexts

OrientDB

2.1.25, 2.2.37, 3.0.35

R Studio

3.4.2

Zeppelin

0.7.3

August 2020

Here are the highlights of new and updated features for this release:

  1. Revamped user interface

  2. New documentation site

  3. Platform URL changes

  4. Configurable persistent storage size for apps

  5. Prometheus monitoring

  6. New technology version added since April 2020: R 4.0

Revamped user interface

Saagie revamped its user interface to streamline your user experience. The new layout makes it easier to see where you are and to transition from one location to another within Saagie.

Visit the user interface section of our user documentation to learn more.

Here’s a quick summary:

Overview of new user interface
  1. Primary navigation menu

  2. Secondary navigation menu

  3. Button to collapse navigation menus

  4. Platforms menu

Access to your user profile is now part of the primary navigation menu.

New documentation site

We are in the process of migrating all of our documentation to our new site, docs.saagio.io. Hint: you’re on it now!

For those who need Legacy or Version 1 documentation, go to Saagie Legacy (Version 1) from this site’s primary navigation menu.

Keep in mind that V1 documentation is no longer maintained, though we do respond to common questions from time to time.

Platform URL changes

Now that the Projects module is no longer in beta, the pattern for platform URLs has changed.

URLs must now follow this pattern: prefix-suffix[.subdomain].domain.com. Saagie will contact your platform administrator to perform this change.

We discuss DNS entries in our installation documentation:

Table 3. DNS entry components
Component Details Example

prefix
Required

12 characters maximum.
Only letters and numbers are allowed (no hyphens or special characters).

This might be your company name or a shortened version of your company name.

dunder

suffix
Required

For Saagie to function properly, your DNS entry must contain a suffix.

workspace

domain
Required

Company’s domain to which your Saagie platform will be connected.

dundermifflin.com

URL: dunder-workspace.dundermifflin.com

subdomain
Optional

Allowed as many subdomains after prefix-suffix as needed or as your company’s policy requires.

paper

URL with subdomain: dunder-workspace.paper.dundermifflin.com

Configurable persistent storage size for apps

You can now configure the persistent storage size when creating your apps. Persistent storage stores an app’s data between instances.

You can set this value between 64MB and 128MB.

Storage size can only be set when creating an app. It cannot be modified later.
Users can configure persistent storage size in apps

Prometheus monitoring

Users can now monitor various Saagie metrics with Prometheus. These metrics are exposed through Saagie’s HTTP API, allowing Ops teams to centralize and visualize them through Prometheus’s multiple modes of graphing and dashboarding.

During the Saagie installation process, you can enable or disable this exposure.

To learn more about this feature, please visit the operations documentation.

New technology version

New technology version added since April 2020: R 4.0

Users can now create jobs using R 4.0 inside the Projects module.

March 2020

What’s new?

  1. Improved navigation in projects thanks to a breadcrumb trail and new persistent buttons.

  2. Information about technologies and their integrated versions available at a glance when creating new projects.

  3. New Activity menu to get an overview of the last running instances of jobs and pipelines across a platform.

  4. Ability to run jobs on a GPU for customers with the GPU option enabled.

  5. New technology versions added since January 2020: PySpark 3.7, Spark 2.4.5, R 3.6.2, Python 3.7.

Improved navigation in Projects

The Projects user interface now displays a breadcrumb trail at the top of the screen.

Navigation breadcrumbs

Users can easily keep track of where they are inside a project and quickly go back a few steps, greatly streamlining navigation inside Projects.

In addition, new persistent buttons allow users to easily refresh a job or pipeline status and display their settings.

Technology details when creating a project

When creating a project and selecting technologies to add inside, users can now view a quick description of each technology as well as a list of the framework versions available in Saagie.

Quick description of technology details

Previously, users could only view available versions once the project was created. This new feature ensures they have all the appropriate information to select the right technologies for their project.

Activity menu

Users can now access an Activity menu to get a quick overview of the last running instances of jobs and pipelines inside each of their projects during the last 12 hours.

Activity menu for last 12 hours for jobs and pipelines

In a production-ready environment, this feature can be really useful to quickly react to failures in the execution of jobs and pipelines, in conjunction with the email alerts which already notify users when a job or pipeline changes status (i.e. goes from “running” to “success” or “failed”).

Running jobs on GPU

Customers with the GPU option enabled on their platform can now run Python jobs on a GPU node.

Run Python jobs on a GPU node

When creating a job, users can select run on GPU and Saagie will automatically allocate GPU resources for this job. This feature is particularly useful for certain types of jobs, like deep learning algorithms for instance, where the use of a GPU can greatly accelerate processing times.

New technology versions

Four new technology versions made available since January 2020:

  • PySpark 3.7

  • Spark 2.4.5 (replaces Spark 2.4)

  • R 3.6.2

  • Python 3.7 (contains PyTorch 1.3)