2023

Here are the highlights of new and updated features for this release:

Find the Saagie CI/CD GitHub Action on the GitHub marketplace. It is designed to make it easier to set up the CI/CD process on your Saagie platform, and provides you with a set of customizable options to upgrade your jobs and pipelines.

For more information, see CI/CD Action for Saagie DataOps Platform.

For more information, see Managing Storage Spaces.

From the pipeline library or its Overview page, you can now duplicate the Current version of your pipeline.

This avoids you to start from scratch and improves your productivity.

For more information, see Duplicating Pipelines.

A new default dashboard has been added to SUM. It gives you information on the capacity planning of your jobs and pipelines.

For more information, see the Saagie – Next Scheduling default dashboard.

You now have a solution for resuming a pipeline that has failed. There are many reasons why a pipeline may not be completed. It can be costly and time consuming to completely restart it. Resuming a pipeline allows you to pick up where the pipeline stopped and complete the remaining jobs.

For more information, see Resume a Pipeline.

Read our new how-to section to learn how to do machine learning with BigQuery using Python in Saagie. To do this, we will look at sentiment analysis applied to movie reviews in the IMDB (Internet Movie Database) and follow each step of this process. Our goal is to determine the polarity of the reviews, that is, whether they are positive or negative.

This section will include several articles. One article for each step of the sentiment analysis process. The article Send Data to BigQuery is the first in a series of four. Stay tuned!

For more information, see Create Machine Learning Pipelines With BigQuery in Saagie.

Learn how to push models in the Hugging Face’s Model Hub with Saagie. Then use them with our Saagie Hugging Face Model Server add-on. This add-on includes the Saagie HF ModelServer TextCLF app, which is designed to facilitate the deployment and prediction of Hugging Face deep learning models for text classification.

For more information, see our article on how to Push Models to Hugging Face With Saagie.

The access to Saagie Resources Monitoring (SRM) has been enhanced for a better user experience.

To access it, you can now click Resource Details in the secondary navigation menu of the Monitoring module. It will open SRM in a new tab.

For more information, see About Saagie Resources Monitoring.

The following technologies have been added or deprecated in the Saagie official technology repository:

Here are the highlights of new and updated features for this release:

Saagie Resources Monitoring (SRM) is a set of graphs providing an overview of your cluster’s resource consumption. SRM gives you a global view of RAM and CPU consumption for nodes, jobs and apps of your cluster through several graphs. It is based on Grafana. With its custom dashboard, you can quickly visualize and analyze RAM and CPU consumption in visual form.

For more information, see About Saagie Resources Monitoring.

Moving jobs from one project to another was already possible, but only between projects on the same platform. You can now move jobs from one project to another on a different platform.

From the job library or its Overview page, click the kebab menu   Move to… and enter the required information. The moved job keeps its versions, instances, logs, packages, alerts, and resource settings.

This avoids you to start from scratch and improves your productivity.

For more information, see Moving a Job to Another Project.

From your job’s Overview and Instances pages, you can now see the execution time of the running and terminated job, along with the different types of status it has gone through.

This allows you to determine the performance of your job. If it is not effective enough, you can optimize it accordingly.

From the Instances and Versions pages of your pipeline, you can now delete instances and versions. This allows you to streamline your list, improve your user experience, and maintain control over storage. You can either delete a single instance or version, a selection of versions or instances with or without filters.

For more information, see Deleting Pipeline Instances and Versions.

The feature to delete job instances has been improved. You can now delete job instances with their logs using a date picker. From your job’s Instances page, select the All instances older than filter to delete all instances prior to the selected date.

This will streamline the list and improves your user experience.

For more information, see Deleting Job Instances and Job Versions.

The job version deletion feature has been improved. You can now delete versions of a job based on tag criteria. From the Versions page of your job, select the desired filter to delete versions accordingly.

This will streamline the list and improves your user experience.

For more information, see Deleting Job Instances and Job Versions.

You can now integrate your Saagie projects into a CI/CD pipeline using our Saagie Python API. By including the source code of your jobs and pipelines in a leading Git tool like GitHub, you can enable CI/CD across all Saagie platforms, from development to production. These development best practices, such as pull changes, review, compare, or commit, can help you better control changes and thus ensure the integrity and consistency of your production environment.

For more information, see Saagie CI/CD.

You can use ChatGTP to generate your job description. Click Generate with ChatGPT above the description field to send your request to ChatGPT.

To enable the option, you must upgrade Saagie to the latest version. When configuring saagiectl, you will have to answer new prompts about the use of OpenAI. This will be asked when configuring your cluster settings.

For more information, see Generating a Job Description With ChatGPT.

You can now use the Jupyter Notebook app with generative AI, such as ChatGPT, SageMaker, or Bedrock. A new app called JupyterLab+GenAI 4.0 Python 3.10 has been added to the Saagie official technology repository for use with a generative AI.

For more information, see Use Generative AI in Jupyter Notebook

You can now use the VS Code app with generative AI pair programmers, such as GitHub Copilot and Genie. Use the VS Code Python 4.15.0 app context to have this feature.

For more information, see Use VS Code Powered by Generative AI

The Saagie Hugging Face Model Server add-on is an app designed to facilitate the deployment and prediction of Hugging Face deep learning models for text classification.

For more information, see Saagie Hugging Face Model Server.

The Saagie Code Search add-on is an app designed to help you search and retrieve Python code snippets from a default codebase or code repositories hosted on GitHub.

For more information, see Saagie Code Search.

This new version of Saagie is compatible with Kubernetes 1.25.x.

If you plan to upgrade your cluster to Kubernetes 1.25.x and already have a Saagie installation on it, you must upgrade Saagie to the latest version BEFORE upgrading to Kubernetes 1.25.x.

Be careful, Kubernetes v1.25 comes with several major changes, including the removal of PSPs (Pod Security Policies). As a reminder, your administrator is responsible for your clusters and their security. As such, they are also responsible for removing PSPs, as well as any other Kubernetes resources removed in v1.25. For more information, see the official Kubernetes documentation on Kubernetes Removals and Major Changes In 1.25.

Also, if your cluster is hosted on Microsoft AKS, and you want to use Saagie on an AKS v1.25.x environment, you need to reinforce the cgroups_v1.

Why? For now, Saagie is still using cgroup_v1. Although Saagie is compatible with Kubernetes 1.25.x, we will not be natively compatible with AKS as Microsoft has forced the transition from AKS to Kubernetes v1.25 on OS cgroup_v2.

To reinforce the cgroups_v1, refer to Azure’s README.md file on GitHub: Revert Kubernetes 1.25 to cgroup v1.

For more information on the compatible versions of Kubernetes, see Compatible Kubernetes Versions.

The Monitoring and Operations modules have been restructured.

The Monitoring module have been deleted. As a reminder, this module was composed of the Platform Overview page. This page provided you with an overview of node consumption and reservations for the selected platform.

The Operations module have been renamed Monitoring. It stays the same as before, except for the name. For more information, see Monitoring Module.

The following technologies have been added to the Saagie official technology repository:

Here are the highlights of new and updated features for this release:

From the job’s Instances page, you can now delete instances and associated logs to streamline the list, improve your user experience, and maintain control over storage. You can either delete a single instance, a selection of instances, or a selection of instances based on status filters.

For more information, see Deleting Job Instances and Job Versions.

From the job’s Versions page, you can now delete versions to streamline the list and improve your user experience. You can either delete a single version, or a selection of versions.

For more information, see Deleting Job Instances and Job Versions.

From the job library or its Overview page, you can now duplicate the Current version of your job.

This avoids you to start from scratch and improves your productivity.

For more information, see Duplicating Jobs

To increase the reliability of job and app execution, better share limited resources with others, and guarantee simultaneous execution our internal system has been enhanced.

Default values for CPU and RAM resources have been defined to all technologies and contexts in Saagie’s Technology Catalog, except for external technologies. These values ensure greater platform stability. You can see the details by clicking the technology in Catalog > Repositories > Saagie.

These values also exist at the technology context level and can override the values defined at the technology level. You can configure them when creating a job or app, or by modifying the Resources setting of your job or app.

In addition, the catalog schemas have been updated with new optional fields to add default values to the technologies in your custom repositories. If this field is left blank, the default values will be 1 CPU and 500 MB RAM. For more information, see Type-Specific Attribute Tables.

You can use our Python package saagieapi, which implements Python API wrappers to easily interact with the Saagie platform in Python.

For more information, see Saagie Python API documentation.

Each technology has its own metadata.yaml file composed of a variety of attributes requiring different types of values.

The parser is sensitive to float ambiguity when the attribute expects a value of type string. This has a particular impact on the technology version number. For example, if you have Python 3.10, it will be read as 3.1 and not 3.10.

To remove this ambiguity in version 2023.03 of the Technology Catalog, you must:

This concerns all attributes requiring a string value.

Jobs lasting more than 15 minutes were automatically assigned the Unknown status. They now end with an appropriate status.

To solve performance issues of the app History page, a pagination has been implemented. Events are loaded progressively rather than all at once, improving page loading time and fluidity.

In addition, the timeline display on the app Overview page has also been modified accordingly. If your app history contains too many events, only the most recent will be displayed. Part of the beginning of the timeline will be grayed out to indicate that the oldest events cannot be displayed.

The following technologies have been added or deprecated in the Saagie official technology repository:

Here are the highlights of new and updated features for this release:

New resource monitoring elements have been added to monitor resource consumption of your cluster and pipelines.

At the cluster level, you can access the Operations module to see an overview of your cluster. This page displays the number of projects, jobs, pipelines, and apps created on each platform, as well as resource capacity metrics for CPU and RAM for each node in the platform.

In the Overview and Instances page of pipelines, you can access graphs displaying runtime and resource consumption metrics.

This added focus on resource monitoring in Saagie will allow data engineers and platform administrators to have a complementary view of clusters and pipelines to track performance and better optimize resource usage on their platforms.

You can now create new type of conditions to build more relevant pipelines:

These new conditions will allow you to implement advanced intelligence in your pipelines.

For more information, see About Conditions in Pipelines.

Saagie is now available on Google Cloud Platform (GCP).

The following technologies have been added to the official Saagie technology repository:

Here are the highlights of new and updated features for this release:

New resource monitoring pages have been added throughout Saagie to monitor resource consumption, from a platform level down to a specific item.

At the platform level, you can access the Monitoring module to see an overview of your platform.

This page displays the number of projects, jobs, pipelines, and apps created on the selected platform, as well as resource capacity metrics for CPU and RAM for each node in the platform.

In the Overview page of jobs and apps, you can access new graphs displaying runtime and resource consumption metrics for the last running instance.

Besides the resource consumption limits that can already be defined for jobs and apps, Saagie’s focus on monitoring will help data engineers and platform administrators quickly identify bottlenecks, debug memory-starved jobs and apps, and better optimize resource usage on the platform.

The new Saagie Usage Monitoring add-on can be installed on your platforms as an app, to monitor:

This app, based on Grafana, is available as an app technology in the Saagie’s official technology repository and can be installed in any project.

For more information, see Saagie Usage Monitoring.

In addition to existing environment variables that are set at the global or project levels, you can now create environment variables inside a pipeline and use them to transfer information between jobs during a pipeline execution.

These variables can be dynamically modified by jobs as the pipeline execution progresses, with a table displaying for each job the input and output values of variables.

This feature allows you to build smarter pipelines and paves the way to conditions based on a pipeline environment variables.

For more information, see the Pipeline Overview Page.

Saagie will now be installed with a ready-to-use example project, which goal is to propose an intelligent learning pipeline able to detect feelings on movie reviews. It is accessible from your platform’s project library.

For more information, see Starting With the Saagie Project Example

This new version of the Saagie installer is now also compatible with Kubernetes versions 1.23.x and 1.24.x.

For more information on supported versions of Kubernetes, see System Requirements.

For clarity on the product version you are using, it will now follow a new naming convention made up of the year, and the product version increment for the year. For this version, it is 2023.01.

The following technologies have been added in the official Saagie technology repository: