2023
April 2023
Here are the highlights of new and updated features for this release:
-
Product Updates (2023.02)
A new version of Saagie has been released with the following features:-
New elements have been created to monitor resource consumption of pipelines and your cluster.
-
Pipeline functionality has been enhanced to include more advanced orchestration logic, such as conditions on environment variables and job status.
-
Saagie now supports Google Cloud Platform (GCP).
-
-
Saagie’s Technology Repository Updates
New technologies have been added.
Product Updates (2023.02)
Cluster and Pipeline Resource Monitoring
New resource monitoring elements have been added to monitor resource consumption of your cluster and pipelines.
At the cluster level, you can access the Operations module to see an overview of your cluster.
This page displays the number of projects, jobs, pipelines, and apps created on each platform, as well as resource capacity metrics for CPU and RAM for each node in the platform.
In the Overview and
Instances page of pipelines, you can access graphs displaying runtime and resource consumption metrics.
This added focus on resource monitoring in Saagie will allow data engineers and platform administrators to have a complementary view of clusters and pipelines to track performance and better optimize resource usage on their platforms.
Smart Conditions in Pipelines
You can now create new type of conditions to build more relevant pipelines:
-
Conditions based on environment variables
-
Conditions based on job status
These new conditions will allow you to implement advanced intelligence in your pipelines.
For more information, see About Conditions in Pipelines.
Saagie With Google Cloud Platform (GCP)
Saagie is now available on Google Cloud Platform (GCP).
Saagie’s Technology Repository Updates
The following technologies have been added to the official Saagie technology repository:
Technologies | New contexts |
---|---|
Dataiku DDS |
|
dbt |
|
Google Cloud Data Transfer |
|
Google Cloud Dataflow |
|
Python |
|
Do not forget to synchronize your Saagie repositories to keep them up to date. |
January 2023
Here are the highlights of new and updated features for this release:
-
Product Updates (2023.01)
A new version of Saagie has been released with the following features:-
New elements to monitor resource consumption have been created.
-
A new add-on, called Saagie Usage Monitoring, can be deployed as an app inside projects.
-
Pipeline functionality has been enhanced to allow context propagation between jobs in a pipeline.
-
Saagie will now be installed with a ready-to-use example project, which goal is to propose an intelligent learning pipeline able to detect feelings on movie reviews.
-
Saagie now supports Kubernetes
1.23.x
and1.24.x
. -
The product version naming pattern has changed.
-
-
Saagie’s Technology Repository Updates
New technology versions and external job technologies have been added.
Product Updates (2023.01)
Resource Monitoring
New resource monitoring pages have been added throughout Saagie to monitor resource consumption, from a platform level down to a specific item.
At the platform level, you can access the Monitoring module to see an overview of your platform.
This page displays the number of projects, jobs, pipelines, and apps created on the selected platform, as well as resource capacity metrics for CPU and RAM for each node in the platform.
If node isolation has not been configured for your platform, the |
In the Overview page of jobs and apps, you can access new graphs displaying runtime and resource consumption metrics for the last running instance.
Besides the resource consumption limits that can already be defined for jobs and apps, Saagie’s focus on monitoring will help data engineers and platform administrators quickly identify bottlenecks, debug memory-starved jobs and apps, and better optimize resource usage on the platform.
Saagie Usage Monitoring
The new Saagie Usage Monitoring add-on can be installed on your platforms as an app, to monitor:
-
The amount of jobs and apps created, with their high-level metadata.
-
Metrics on the execution time and status of jobs and pipelines.
-
Metrics on the global usage of the storage volume associated with Saagie.
This app, based on Grafana, is available as an app technology in the Saagie’s official technology repository and can be installed in any project.
This app requires some configuration to work.
Click the information icon |
As this app is designed to display cross-project metrics, Saagie recommends deploying it in a dedicated administration project. |
For more information, see Saagie Usage Monitoring.
Context Propagation Between Jobs in Pipelines
In addition to existing environment variables that are set at the global or project levels, you can now create environment variables inside a pipeline and use them to transfer information between jobs during a pipeline execution.
These variables can be dynamically modified by jobs as the pipeline execution progresses, with a table displaying for each job the input and output values of variables.
This feature allows you to build smarter pipelines and paves the way to conditions based on a pipeline environment variables.
For more information, see the Pipeline Overview Page.
Saagie Project Example
Saagie will now be installed with a ready-to-use example project, which goal is to propose an intelligent learning pipeline able to detect feelings on movie reviews. It is accessible from your platform’s project library.
For more information, see Starting With the Saagie Project Example
Kubernetes 1.23.x and 1.24.x Support
This new version of the Saagie installer is now also compatible with Kubernetes versions 1.23.x
and 1.24.x
.
For more information on supported versions of Kubernetes, see System Requirements.
Product Version Naming Convention
For clarity on the product version you are using, it will now follow a new naming convention made up of the year, and the product version increment for the year.
For this version, it is 2023.01
.
Saagie’s Technology Repository Updates
The following technology versions and external job technologies have been added to the official Saagie technology repository:
Technologies | New contexts |
---|---|
Bash |
|
Java/Scala |
|
Talend |
|
GCP Cloud Functions |
|
GCP Cloud Run |
|
Technologies | New contexts |
---|---|
Apache Superset |
|
Grafana |
|
Metabase |
|
MLFlow Server |
|
Saagie Usage Monitoring |
For Saagie |
Do not forget to synchronize your Saagie repositories to keep them up to date. |