How Does Saagie Work?
Saagie consists of numerous components that work together in stacks. Kafka manages communication between stacks and between components within stacks. Stacks use MongoDB for data storage.
The Global diagram per stack illustrates the stacks and how they work together. The criticality of each component is also represented. Stacks and criticality are described in the sections below.

Saagie Stacks and Components
Saagie has several stacks:
-
The Ingress stack exposes all other stacks.
-
The Authentication stack manages users, groups, and authorizations.
-
The Orchestration stack manages the execution of jobs, apps, and pipelines in Kubernetes.
-
The Datalake Governance stack describes and qualifies the data and data structure using Hadoop.
Ingress Stack
The Ingress
stack exposes pods outside of Kubernetes.
The following pods are exposed through ingress-nginx-controller
:
-
admin-ui
-
authentication
-
conforama
-
datasetaccess
-
datasetaccess-ui
-
governance
-
login
-
profile
-
projects-and-jobs
-
projects-and-jobs-api
-
security
-
settings
-
technology-manager
-
traefik
If a request doesn’t reach a pod listed above, ingress-nginx-defaultbackend
provides a fallback path.
Authentication Stack
The Authentication
stack is responsible for logins and the management of users and groups.
This stack is composed of both Saagie and third-party components:
-
auth
is a CRUD component for users and groups.auth
communicates with thesecurity
pod to get authorizations associated with groups of a given user. -
authentication
usesKeycloak
to change passwords and manage tokens. This component also triggers verification emails sent to users. -
security
retrieves group authorizations. It uses theidmacl
pod to know which groups are associated with a user.security
calls theauthentication
pod to check a token’s validity.security
is a component called by other external components, such asprojects-and-jobs-api
. -
idmacl
is an abstraction layer for the users and groups persistence system. According to the Saagie offer you chose, this persistence could be an external LDAP, such as OpenLDAP or ActiveDirectory, orKeycloak
. -
Keycloak
is a third-party service used to manage tokens, such as creation and check.Keycloak
also keeps current sessions open related to these tokens. -
admin-ui
is the user interface that manages users and groups. -
login
is the user interface that manages connection to the platform (login and password). -
profile
manages users profiles, such as email and job.profile
callsauthentication
to trigger emails sent to users.
Orchestration Stack
The Orchestration
stack is responsible for executing jobs, apps, and pipelines, as well as everything related to environment variables and Docker credentials.
There are two API entry points that receive requests from Saagie users: projects-and-jobs-api
and conforama
.
Each platform has its own Kubernetes namespace containing a MinIO server. Each project also has a Kubernetes namespace containing a MinIO server and Argo. All executable elements—jobs, apps, and pipelines—are executed in the namespace of the corresponding project.
Due to the namespaces system, projects are isolated from other projects, just as platforms are isolated from other platforms.
Saagie components that are not in the project or platform namespaces are in the <installationId>
namespace.
Where <installationId>
must be replaced with your installation ID, which must match the prefix you have determined for your DNS entry.
This stack is composed of several components with different roles and features:
-
projects-and-jobs
is the user interface for Saagie'sProjects
section. -
projects-and-jobs-api
is the API used by theprojects-and-jobs
component. -
project-k8s-controller
is a Kubernetes controller (along with the CRDProject
created by Saagie) allowing the creation and update of project namespaces. -
platform-k8s-controller
is a Kubernetes controller (along with the CRDPlatform
created by Saagie) allowing the creation and update of platform namespaces. -
conforama
is an HTTP API that saves files for a platform. -
technology-manager
allows users to manage their own technologies and repositories within Saagie. -
Fluent Bit
is a KubernetesDaemonSet
that allows Docker logs to be read in order to extract jobs and apps logs and make them available inprojects-and-jobs-api
. -
scredz
is used byprojects-and-jobs-api
for Docker credentials concerns. -
Traefik
is an ingress controller that allows HTTP access to apps from outside Saagie. It contains a sidecar container responsible for verifying the access rights for the requested app.
Data Lake Governance Stack
The Data lake governance
stack is an application used to manage, document, and qualify your data lake.
It allows you to manage user domains, provenances, trust levels, and data status.
The stack is connected to the data lake, enabling it to extract necessary information.
The Data lake governance
stack is composed of a few components:
-
governance
is the main component managing all data lake documentation and qualifications. -
datasetaccess-ui
is the user interface that manages dataset access rights. -
datasetaccess
is the API used bydatasetaccess-ui
. -
rule-manager
applies the access rights defined indatasetaccess
to the data lake.
Component Criticality
Each technical component used by Saagie has a level of criticality depending on its role. The impact varies when these components fail or are shut down.
Criticality Level | Color | Meaning |
---|---|---|
Minor |
Yellow |
Any Anomaly making it impossible for the customer to use one or more non-essential features of the solution. |
Major |
Orange |
Anomaly reducing the use of the solution by preventing the use of certain essential functions. |
Critical |
Red |
Anomaly making total use of the solution impossible. |
The tables below show the criticality level and the impact on the platform of a failure for each component.
Component | Criticality | Impact |
---|---|---|
|
Critical |
No access to Saagie API. |
|
Minor |
No default error page. |
Component | Criticality | Impact |
---|---|---|
|
Critical |
Authentication stack is unusable, rendering Saagie unusable. |
|
Critical |
Authentication stack is unusable, rendering Saagie unusable. |
|
Critical |
Authentication stack is unusable, rendering Saagie unusable. |
|
Critical |
Authentication stack is unusable, rendering Saagie unusable. |
|
Critical |
Authentication stack is unusable, rendering Saagie unusable. |
|
Minor |
Cannot manage users and groups. |
|
Critical |
Cannot login to Saagie. |
|
Minor |
Cannot manage user profiles, such as jobs and email. |
Component | Criticality | Impact |
---|---|---|
|
Major |
Projects and jobs user interface is unavailable. |
|
Major |
Cannot create jobs, pipelines, and scheduled jobs. |
|
Major |
Impossible to create projects. |
|
Minor |
Impossible to create platforms. |
|
Minor |
Impossible to create, modify, or delete files on the MinIO platform. Doesn’t block usage by jobs and apps. |
|
Minor |
No consequence. |
Fluent Bit |
Minor |
No logs for jobs and apps. |
|
Minor |
Impossible to create, update, or delete Docker credentials. Doesn’t block usage by jobs and apps. |
|
Major |
Impossible to access app ports. |
Component | Criticality | Impact |
---|---|---|
|
Major |
Cannot access Governance. |
|
Major |
Cannot manage dataset access. |
|
Major |
Cannot manage dataset access or use governance properly. |
|
Major |
Cannot grant authorizations to the data lake. When |
Component | Criticality | Impact |
---|---|---|
|
Major |
Impossible to run jobs and apps in the corresponding projects. |
|
Major |
Impossible to run jobs and apps in the corresponding projects. |
|
Critical |
Numerous Saagie components cannot work, rendering Saagie unusable. |
|
Critical |
Numerous Saagie components cannot work, rendering Saagie unusable. |
|
Critical |
Numerous Saagie components cannot work, rendering Saagie unusable. |
|
Critical |
Numerous Saagie components cannot work, rendering Saagie unusable. |
|
Critical |
Saagie user interface is unusable. |
|
Minor |
Metrics are not collected. |