References
Attribute Reference Tables
The parser is sensitive to float ambiguity when the attribute expects a value of type string .
To remove this ambiguity, you must write id: "3.10" instead of id: 3.10 .
This concerns all attributes of type string .
|
Common Attributes Table
The following attributes are common to jobs, apps, and external jobs.
Attribute | Description | Default value | ||
---|---|---|---|---|
|
The version of the file format.
|
none |
||
|
The unique identifier of the technology in the repository. |
none |
||
|
The name of the technology as it will appear in the user interface. |
none |
||
|
The description of the technology as it will appear in the user interface. |
none |
||
|
Makes the technology unusable when the answer is |
|
||
|
Indicates whether it is a job, an app or an external job technology.
|
none |
||
|
The name of the image as it appears in the Saagie Design System. |
none |
||
|
Defines the different versions or features of the technology. |
none |
||
|
The context’s unique identifier of your technology. |
none |
||
|
The name of the context as it will appear in the user interface. |
none |
||
|
The description of the context as it will appear in the user interface. Multiline file description
|
none |
||
|
Makes the context unusable when the answer is
|
|
||
|
Labels the context as Recommended when the response is
|
|
||
|
Informs that the context is not stable when the response is
|
none |
||
|
Informs that the context is deprecated when a depreciation date is specified.
|
none |
If you encounter problems with one of these fields, contact Saagie Support. |
-
If you set
available: false
, then the technology will appear as Disabled in the UI, regardless of the value of thetrustlevel
anddeprecationDate
attributes. -
If you set only
available: true
, then no specific label will be displayed next to the technology in the UI. -
If you set
available: true
in addition of:-
deprecationDate: "2021-06-19T10:30:50Z"
, then the technology will appear as Deprecated in the UI, regardless of the value of thetrustLevel
attribute. -
trustLevel: experimental
, then the technology will appear as Experimental in the UI. -
trustLevel: stable
, then the technology will appear as Stable in the UI.
-
-
If you set
recommended: true
in addition to the above rules, then the technology will also appear as Recommended in the UI.
Type-Specific Attribute Tables
The following attributes apply to jobs only.
Attribute | Description | Default value | ||
---|---|---|---|---|
|
The recommended CPU limit and request.
|
|
||
|
The recommended memory limit and request in
|
|
||
|
The list of the features available in the job creation form. |
none |
||
|
Information about the Docker image of a technology. |
none |
||
|
The Docker image name of the technology. |
none |
||
|
The Docker image version (or tag). |
|
||
|
The recommended CPU limit and request.
|
|
||
|
The recommended memory limit and request in
|
|
Jobs have at least one context that contains its own list of features. In each context, there can be several features, but only one of each type.
A feature is defined by the following values:
Attribute | Description | Possible values |
---|---|---|
|
Defines the feature type. |
|
|
Information displayed next to a field. |
Character string |
|
Indicates whether if a field is required or not. |
|
|
Information that appears next to a field and provides examples of how to complete that field. |
Character string |
|
Example of a field’s default value. |
Character string |
The following attributes apply to apps only.
Attribute | Description | Default value | ||
---|---|---|---|---|
|
The recommended CPU limit and request.
|
|
||
|
The recommended memory limit and request in
|
|
||
|
The basic description of the application. |
none |
||
|
The background color for the app card in the user interface. |
none |
||
|
The context release notes. |
none |
||
|
Information about the Docker image of a technology. |
none |
||
|
The Docker image name of the technology. |
none |
||
|
The Docker image version (or tag). |
|
||
|
The list of available open ports for a Docker container. |
none |
||
|
The port number. |
none |
||
|
The port label, with a description of its use. |
none |
||
|
The name of the environment variable that will contain the URL generated by Saagie. |
none |
||
|
The list of Docker volume paths used by the application. |
none |
||
|
The volume path.
|
none |
||
|
The recommended CPU limit and request.
|
|
||
|
The recommended memory limit and request in
|
|
||
|
The path to the repository of the technology |
none |
The following attributes apply to external jobs only.
External jobs are not run in Saagie but use external services instead. When used with Saagie, external technologies keep their properties and use the same execution mode as internal jobs, such as status, logs, and pipelines. |
Attribute | Description | Default value | ||
---|---|---|---|---|
|
The unique identifier of the associated connection type.
|
none |
||
|
The parameters of the external job. |
none |
||
|
The parameter’s unique identifier of your external job. |
none |
||
|
The type of parameter.
|
none |
||
|
The name of the parameter as it will appear in the user interface. |
none |
||
|
The description text of the parameter, which appears in the tooltips of the user interface. |
none |
||
|
Indicates if the parameter is mandatory for the job to be valid. |
none |
||
|
The list of parameters on which the current parameter depends. |
none |
||
|
The unique identifier of the parameter on which the current parameter depends. |
none |
||
|
The default value of the parameter.
|
none |
||
|
The list of available values. |
none |
||
|
The unique identifier of the listed values.
|
none |
||
|
The name of the value as it will appear in the user interface.
|
none |
||
|
Defines how the values should be retrieved to be available in the drop-down list. |
none |
||
|
The relative path to the JavaScript file containing the function to get the values. |
none |
||
|
The name of the exported function to execute to get the values in the drop-down list. |
none |
||
|
The configuration of each action to execute the external job’s life cycle.
|
none |
||
|
Provides a method to start the external job. |
none |
||
|
The relative path to the JavaScript file containing the function to start the external job. |
none |
||
|
The name of the exported function to execute to start the external job. |
none |
||
|
Provides a method to stop the external job.
|
none |
||
|
The relative path to the JavaScript file containing the function to stop the external job. |
none |
||
|
The name of the exported function to execute to stop the external job. |
none |
||
|
Provides a method to get the external job status. |
none |
||
|
The relative path to the JavaScript file containing the function to get the external job status. |
none |
||
|
The name of the exported function to execute to get the external job status. |
none |
||
|
Provides a method to get the logs of the external job.
|
none |
||
|
The relative path to the JavaScript file containing the function to get the external job logs. |
none |
||
|
The name of the exported function to execute to get the external job logs. |
none |
The connection type specifies how to establish a connection with the external service.
It must be declared in a metadata.yaml
file with the declaration of its parameters.
This metadata is used by Saagie to create a connection to your input values that will be provided to the external job scripts.
Attribute | Description | Default value | ||
---|---|---|---|---|
|
The version of the file format.
|
none |
||
|
Indicates the type of metadata in the file.
|
none |
||
|
The connection type’s unique identifier in the repository. |
none |
||
|
The name of the connection type as it will appear in the user interface. |
none |
||
|
The connection type parameters. |
none |
||
|
The parameter’s unique identifier in your connection type. |
none |
||
|
The type of parameter.
|
none |
||
|
The name of the parameter as it will appear in the user interface. |
none |
||
|
The description text of the parameter, which appears in the tooltips of the user interface. |
none |
||
|
Indicates if the parameter is mandatory for the connection to be valid. |
none |
||
|
The list of parameters on which the current parameter depends. |
none |
||
|
The unique identifier of the parameter on which the current parameter depends. |
none |
||
|
The default value of the parameter.
|
none |
||
|
The list of available values. |
none |
||
|
The unique identifier of the listed values.
|
none |
||
|
The name of the value as it will appear in the user interface.
|
none |
||
|
Provides a method to check the connection of the external job.
|
none |
||
|
The relative path to the JavaScript file containing the function to check the connection of the external job. |
none |
||
|
The name of the exported function to execute to check the connection of the external job. |
none |
External Jobs Configuration
The configuration of external technologies requires a basic understanding of JavaScript. |
Dynamic Select
The DYNAMIC_SELECT
parameter type is only used at the creation and the modification of the external job.
The script’s function must return an array of objects containing the attributes id
and label
typed as string.
It must be declared in the metadata.yaml
file of the external job and its function must be exposed by a JavaScript file embedded in the repository .zip
file.
parameters:
- type: DYNAMIC_SELECT
id: jobDefinition
label: Job Definition
mandatory: true
dynamicValues:
script: path-to-the-script/job-form.js
function: getBatchJobs
exports.getBatchJobs = async ({connection}) => {
const client = buildClient(connection);
const {data} = await client.batch.describeJobDefinitions();
return data?.jobDefinitions
?.filter(({status}) => status === 'ACTIVE')
?.map(({jobDefinitionArn, jobDefinitionName}) => ({
id: jobDefinitionArn,
label: jobDefinitionName,
})) ?? [];
};
Life Cycle Actions
There are four actions used to navigate and manage the life cycle of an external job instance.
For each action, there must be a function exposed by a JavaScript file embedded in the repository .zip
file.
→ start({ connection, parameters }): payload
The start
action is used to execute the job.
The script’s function must return the payload that will help subsequent calls identify the newly executed job instance ;
The returned payload must be a simple JavaScript object containing all the information needed for the other actions.
The action must be called with an object argument, containing information about the connection and parameter values.
Argument | Type | Default value |
---|---|---|
|
|
none |
const axios = require('axios');
const axiosHttp = require('axios/lib/adapters/http');
const client = axios.create({
adapter: axiosHttp
});
exports.start = async ({ connection, parameters }) => {
// Make a fetch request to start the job using axios
const result = await client.get(`${connection.url}/datasets/${parameters.dataset}/projects/${parameters.project}/processes/${parameters.process}/run`);
return { jobId: result.data.jobId }; // the payload associated to the running job
}
→ stop({ connection, parameters, payload })
The stop
action is used to stop a running job.
The script’s function can return a payload that will help the call to the getLogs
function to identify the instance of the stopped job.
This payload will be merged with the payload returned by the call to the start
function.
The action must be called with an object argument, containing information about the connection and parameter values, as well as the payload returned by the call to the start
function.
Argument | Type | Default value |
---|---|---|
|
|
none |
const axios = require('axios');
const axiosHttp = require('axios/lib/adapters/http');
const client = axios.create({
adapter: axiosHttp
});
exports.stop = async ({ connection, parameters, payload }) => {
// Make a fetch request to stop the job
await client.get(`${connection.url}/datasets/${parameters.dataset.id}/projects/${parameters.project}/processes/${parameters.process}/instance/${payload.jobId}/stop`);
}
→ getStatus({ connection, parameters, payload }): string
The getStatus
action is used to retrieve the current status of a job.
The script’s function must return the string
representation of the status.
The action must be called with an object argument, containing information about the connection and parameter values, as well as the payload returned by the call to the start
function.
Argument | Type | Default value |
---|---|---|
|
|
none |
const axios = require('axios');
const axiosHttp = require('axios/lib/adapters/http');
const client = axios.create({
adapter: axiosHttp
});
const JobStatus = {
AWAITING: 'AWAITING',
REQUESTED: 'REQUESTED',
QUEUED: 'QUEUED',
RUNNING: 'RUNNING',
SUCCEEDED: 'SUCCEEDED',
KILLING: 'KILLING',
KILLED: 'KILLED',
FAILED: 'FAILED'
};
exports.getStatus = async ({ connection, parameters, payload }) => {
// Make a fetch request to get the current status of the job
const status = await client.get(`${connection.url}/datasets/${parameters.dataset}/projects/${parameters.project}/processes/${parameters.process}/instance/${payload.jobId}/getStatus`);
if (status.isRunning) {
// And return the response along with the current status of the job
return JobStatus.RUNNING;
}
return JobStatus.UNKNOWN;
}
→ getLogs({ connection, parameters, payload }): stream
The getLogs
action is used to retrieve the logs of a running job.
The script’s function must return a data stream of the logs.
The action must be called with an object argument, containing information about the connection and parameter values, as well as the payload returned by the call to the start
function.
Argument | Type | Default value |
---|---|---|
|
|
none |
const axios = require('axios');
const axiosHttp = require('axios/lib/adapters/http');
const client = axios.create({
adapter: axiosHttp,
responseType: 'stream'
});
exports.getLogs = async ({ connection, parameters, payload }) => {
console.log('GET LOG INSTANCE:', payload.jobId);
// Make a fetch request to get the logs of the job
const { data } = await client.get(`${connection.url}/datasets/${parameters.dataset}/projects/${parameters.project}/processes/${parameters.process}/instance/${payload.jobId}/getLogs`);
return data;
};
The written JavaScript code is run by Node.js and you must have version 16
or later.
The script will be run in a constrained and isolated environment where not all APIs and node modules will be available.
Thus, the script must be built in such a way that it contains all the required dependencies at runtime.
The Saagie SDK provides an example of a properly configured build.
In addition, you can use the CLI (Command Line Interface) tool developed by Saagie to simplify your development process.
Code written for an external technology can be provided in one or more files. Since each script must contain all its dependencies itself, it is recommended to have a JavaScript file for each function. |
Folder hierarchy example
Here is an example of an external technology exposing two contexts, with two different file organizations:

# [...]
actions:
start:
script: instanceActions.js
function: start
stop:
script: instanceActions.js
function: stop
getStatus:
script: instanceActions.js
function: getStatus
# [...]
exports.start = async ({ connection, parameters }) => {
// Logic to start the external job instance.
}
exports.stop = async ({ connection, parameters, payload }) => {
// Logic to stop the external job instance.
}
exports.getStatus = async ({ connection, parameters, payload }) => {
// Logic to get the job instance status.
}
# [...]
actions:
start:
script: start.js
function: run
stop:
script: stop.js
function: run
getStatus:
script: getStatus.js
function: run
# [...]
exports.run = async ({ connection, parameters, payload }) => {
// Logic to get the job instance status.
}
exports.run = async ({ connection, parameters }) => {
// Logic to start the external job instance.
}
exports.run = async ({ connection, parameters, payload }) => {
// Logic to stop the external job instance.
}
Check Connection
The checkConnection
action is used to verify that Saagie is able to connect to the external service via the connection credentials.
The script’s function must return a JavaScript object with the ok
field as a boolean
.
When activated, a Check connection button appears in the user interface. |
The checkConnection
action must be declared in the metadata.yaml
file of the connection type and its function must be exposed by a JavaScript file embedded in the repository .zip
file.
actions:
checkConnection:
script: path-to-the-script/check-connection.js
function: checkConnection
exports.checkConnection = async ({connection}) => {
const client = buildClient(connection);
await client.sts.getCallerIdentity();
return {
ok: true
};
};
Metadata file examples
Here are some examples of metadata.yaml
files for each type of technology.
You can find more examples in the Saagie technology repository. |
Example for Java/Scala 8
, 11
, and 13
.
version: "v1"
type: JOB
id: java-scala
label: Java/Scala
available: true
icon: java-scala
contexts:
- id: 11
label: 11
available: true
recommended: true
dockerInfo:
image: saagie/java-scala
version: 11-1.10.0_merge_certified_experimental
trustLevel: "stable"
job:
features:
- type: COMMAND_LINE
label: "Command line"
mandatory: true
comment: "Linux shell command to launch the job."
defaultValue: java -jar {file} arg1 arg2
- type: ARTIFACT
label: "Package"
mandatory: true
comment: "Compatible upload file : .jar"
- type: SCHEDULER
label: "Scheduled"
mandatory: true
- id: 13
label: 13
available: true
recommended: false
dockerInfo:
image: saagie/java-scala
version: 13-1.10.0_merge_certified_experimental
trustLevel: "experimental"
job:
features:
- type: COMMAND_LINE
label: "Command line"
mandatory: true
comment: "Linux shell command to launch the job."
defaultValue: java -jar {file} arg1 arg2
- type: ARTIFACT
label: "Package"
mandatory: true
comment: "Compatible upload file : .jar"
- type: SCHEDULER
label: "Scheduled"
mandatory: true
- id: 8
label: 8
available: true
recommended: false
dockerInfo:
image: saagie/java-scala
version: 8-1.10.0_merge_certified_experimental
trustLevel: "stable"
job:
features:
- type: COMMAND_LINE
label: "Command line"
mandatory: true
comment: "Linux shell command to launch the job."
defaultValue: java -jar {file} arg1 arg2
- type: ARTIFACT
label: "Package"
mandatory: true
comment: "Compatible upload file : .jar"
- type: SCHEDULER
label: "Scheduled"
mandatory: true
Example for Jupyter Notebook.
version: v1
type: APP
id: jupyter-doc
label: Jupyter Notebook
baseline: Data Science Notebook
backgroundColor: "#E87A35"
description: "The Jupyter Notebook is an open-source web application..."
available: true
icon: jupyter
customFlags: []
contexts:
- id: "6.1.1"
label: Jupyter Notebook
releaseNotes: "Prevent inclusion of requests_unixsocket on Windows"
available: true
trustLevel: stable
dockerInfo:
image: saagie/jupyter-python-nbk
version: v2
ports:
- port: 8888
name: Notebook
rewriteUrl: false
basePath: SAAGIE_BASE_PATH
volumes: ["/notebooks-dir"]
Examples for AWS Lambda.
-
A metadata file declaring the AWS connection.
metadata.yamlversion: v2 id: aws-connection label: AWS Connection description: Connection to AWS type: CONNECTION_TYPE parameters: - type: TEXT id: aws_access_key_id label: Access Key ID mandatory: true - type: PASSWORD id: aws_secret_access_key label: Secret Access Key mandatory: true - type: TEXT id: region label: Region comment: "AWS region. Example: us-east-1" mandatory: true actions: checkConnection: script: path-to-the-script/check-connection.js function: checkConnection
-
A metadata file declaring the AWS Lambda parameters and scripts.
metadata.yamlversion: v2 id: aws-lambda label: aws-lambda description: "Run code without thinking about servers. Pay only for the compute time you consume. Only lambda with Kinesis, SQS and DynamoDB Stream. Api Version: 2015-03-31" #available: true type: EXTERNAL_JOB icon: job contexts: - id: functions label: functions description: "Only lambda with Kinesis, SQS and DynamoDB Stream." recommended: true trustLevel: experimental connectionTypeId: aws-connection parameters: - type: DYNAMIC_SELECT id: functions label: Functions comment: Only lambda with Kinesis, SQS and DynamoDB Stream. mandatory: true dynamicValues: script: ./functions/module.js function: getFunctions actions: start: script: ./functions/module.js function: start stop: script: ./functions/module.js function: stop getStatus: script: ./functions/module.js function: getStatus getLogs: script: ./functions/module.js function: getLogs
The referenced JavaScript files must be part of the repository .zip file. |