References

1. Attribute reference tables

Technologies can have a variety of attributes. Some are common to jobs, apps, and external jobs, while others are type-specific.

Common attributes table

The following attributes are common to jobs, apps, and external jobs.

Attribute Description Default value

version
Required
String

The version of the file format.

The supported versions are v1 and v2, except for external jobs which are only allowed in the v2 file format.

none

id
Required
String

The technology’s unique identifier in the repository.

none

label
Required
String

The name of the technology as it will appear in the user interface.

none

description
Optional
String (multiline)

The description of the technology as it will appear in the user interface.

none

available
Optional
Boolean

Makes the technology unusable when the answer is false, that is, it’s no longer possible to create new jobs and apps or to update existing ones with this technology.
However, jobs and apps that already use it continue to work.

false

type
Required
String

Indicates whether it is a job, an app or an external job technology.

The supported types are JOB, APP and EXTERNAL_JOB.

none

icon
Optional
String

The name of the image as it appears in the Saagie Design System.

none

contexts
Required
Array

Defines the different versions or features of the technology.

none

contexts[]
.id

Required
String

The context’s unique identifier of your technology.

none

contexts[]
.label

Required
String

The name of the context as it will appear in the user interface.

none

contexts[]
.description

Optional
String (multiline)

The description of the context as it will appear in the user interface.

Multiline file description
description: >
  Several lines of text,
  with some "quotes" of various 'types',
  plus another line at the end.

none

contexts[]
.available

Optional
Boolean

Makes the context unusable when the answer is false, that is, it’s no longer possible to create new jobs and apps or to update existing ones with this context.
However, jobs and apps that already use it continue to work.

You should be aware that when applying state-related attributes, some states take priority over others. For more information, see the State-related attribute rules.

false

contexts[]
.recommended

Optional
Boolean

Labels the context as Recommended when the response is true.

You should be aware that when applying state-related attributes, some states take priority over others. For more information, see the State-related attribute rules.

false

contexts[]
.trustLevel

Required
String

Informs that the context is not stable when the response is experimental.

  • The supported values are stable or experimental.

  • You should be aware that when applying state-related attributes, some states take priority over others. For more information, see the State-related attribute rules.

none

contexts[]
.deprecationDate

Optional
Timestamp

Informs that the context is deprecated when a depreciation date is specified.
The value must be a past date with the format yyyy-mm-ddThh:mm:ssZ.
Leave the attribute value empty if the technology is not deprecated.

You should be aware that when applying state-related attributes, some states take priority over others. For more information, see the State-related attribute rules.

none

If you encounter problems with one of these fields, contact Saagie Support.
  • If you set available: false, then the technology will appear as Disabled in the UI, regardless of the value of the trustlevel and deprecationDate attributes.

  • If you set only available: true, then no specific label will be displayed next to the technology in the UI.

  • If you set available: true in addition of:

    • deprecationDate: "2021-06-19T10:30:50Z", then the technology will appear as Deprecated in the UI, regardless of the value of the trustLevel attribute.

    • trustLevel: experimental, then the technology will appear as Experimental in the UI.

    • trustLevel: stable, then the technology will appear as Stable in the UI.

  • If you set recommended: true in addition to the above rules, then the technology will also appear as Recommended in the UI.


Type-specific attribute tables

  • Job attributes table

  • App attributes table

  • External job attributes table

The following attributes apply to jobs only.

Attribute Description Default value

contexts[]
.job
.features

Optional
-

The list of the features available in the job creation form.

none

contexts[]
.dockerInfo

Required
-

Information about the Docker image of a technology.

none

contexts[]
.dockerInfo
.image

Required
String

The Docker image name of the technology.

none

contexts[]
.dockerInfo
.version

Required
String

The Docker image version (or tag).

latest


List of features of the job creation form

Jobs have at least one context that contains its own list of features. In each context, there can be several features, but only one of each type.

A feature is defined by the following values:

Attribute Description Possible values

type

Defines the feature type.

  • COMMAND_LINE

  • ARTIFACT

  • SCHEDULER

  • GPU

  • DOCKER_INFO

label

Information displayed next to a field.

Character string

mandatory

Indicates whether if a field is required or not.

  • true

  • false

comment

Information that appears next to a field and provides examples of how to complete that field.

Character string

defaultValue

Example of a field’s default value.

Character string

The following attributes apply to apps only.

Attribute Description Default value

baseline
Optional
String

The basic description of the application.

none

backgroundColor
Optional
String

The background color for the app card in the user interface.

none

contexts[]
.releaseNotes

Optional
String (multiline)

The context release notes.

none

contexts[]
.dockerInfo

Required
-

Information about the Docker image of a technology.

none

contexts[]
.dockerInfo
.image

Required
String

The Docker image name of the technology.

none

contexts[]
.dockerInfo
.version

Required
String

The Docker image version (or tag).

latest

contexts[]
.ports

Required
Array

The list of available open ports for a Docker container.

none

contexts[]
.ports[]
.port

Required
Integer

The port number.

none

contexts[]
.ports[]
.name

Optional
String

The port label, with a description of its use.

none

contexts[]
.ports[]
.basePath

Optional
String

The name of the environment variable that will contain the URL generated by Saagie.

none

contexts[]
.volumes

Optional
Array

The list of Docker volume paths used by the application.

none

contexts[]
.volumes[]

-
String

The volume path.

The volume path attribute is required only if you use the volumes attribute above.

none

The following attributes apply to external jobs only.

External jobs are not run in Saagie but use external services instead. When used with Saagie, external technologies keep their properties and use the same execution mode as internal jobs, such as status, logs, and pipelines.
Attribute Description Default value

contexts[]
.connectionTypeId

Required
String

The unique identifier of the associated connection type.

The connection type specifies how to establish a connection with the external service and must be declared in a metadata.yaml file.

none

contexts[]
.parameters

Required
Array

The parameters of the external job.

none

contexts[]
.parameters[]
.id

Required
String

The parameter’s unique identifier of your external job.

none

contexts[]
.parameters[]
.type

Required
String

The type of parameter.

The supported types are TEXT, DYNAMIC_SELECT, STATIC_SELECT, and TOGGLE.

none

contexts[]
.parameters[]
.label

Required
String

The name of the parameter as it will appear in the user interface.

none

contexts[]
.parameters[]
.comment

Optional
String

The description text of the parameter, which appears in the tooltips of the user interface.

none

contexts[]
.parameters[]
.mandatory

Optional
String

Indicates if the parameter is mandatory for the job to be valid.

none

contexts[]
.parameters[]
.dependsOn

Optional
Array

The list of parameters on which the current parameter depends.

none

contexts[]
.parameters[]
.dependsOn[]

Optional
String

The parameter’s unique identifier on which the current parameter depends.

none

contexts[]
.parameters[]
.defaultValue

Optional
String

The default value of the parameter.

Only for TEXT, STATIC_SELECT, and TOGGLE parameter types.
For the TOGGLE parameter type, the default value must be true or false.
For the STATIC_SELECT parameter type, the default value must be the unique identifier of one of the values listed in the staticValues attribute.

none

contexts[]
.parameters[]
.staticValues

Required for the STATIC_SELECT parameter type
String

The list of available values.

none

contexts[]
.parameters[]
.staticValues[]
.id

Required for the STATIC_SELECT parameter type
String

The unique identifier of the listed values.

It will not be visible in the user interface but will be used as a value in the associated JavaScript file.

none

contexts[]
.parameters[]
.staticValues[]
.label

Required for the STATIC_SELECT parameter type
String

The name of the value as it will appear in the user interface.

It will not be processed in the associated JavaScript file but the value of the id attribute will be.

none

contexts[]
.parameters[]
.dynamicValues

Required for the DYNAMIC_SELECT parameter type
String

Defines how the values should be retrieved to be available in the drop-down list.

none

contexts[]
.parameters[]
.dynamicValues
.script

Required for the DYNAMIC_SELECT parameter type
String

The relative path to the JavaScript file containing the function to get the values.

none

contexts[]
.parameters[]
.dynamicValues
.function

Required for the DYNAMIC_SELECT parameter type
String

The name of the exported function to execute to get the values in the drop-down list.

none

contexts[]
.actions

Required
-

There are four actions. For each action, there must be a function exposed by a JavaScript file embedded in the repository .zip file.

none

contexts[]
.actions
.start

Required
-

Provides a method to start the external job.

none

contexts[]
.actions
.start
.script

Required
String

The relative path to the JavaScript file containing the function to start the external job.

none

contexts[]
.actions
.start
.function

Required
String

The name of the exported function to execute to start the external job.

none

contexts[]
.actions
.stop

Optional
-

Provides a method to stop the external job.

If you don’t provide this method, you will not be able to stop the external job from the user interface.

none

contexts[]
.actions
.stop
.script

Required
String

The relative path to the JavaScript file containing the function to stop the external job.

none

contexts[]
.actions
.stop
.function

Required
String

The name of the exported function to execute to stop the external job.

none

contexts[]
.actions
.getStatus

Required
-

Provides a method to get the external job status.

none

contexts[]
.actions
.getStatus
.script

Required
String

The relative path to the JavaScript file containing the function to get the external job status.

none

contexts[]
.actions
.getStatus
.function

Required
String

The name of the exported function to execute to get the external job status.

none

contexts[]
.actions
.getLogs

Optional
-

Provides a method to get the logs of the external job.

If you don’t provide this method, you will not be able to get the logs from the user interface.

none

contexts[]
.actions
.getLogs
.script

Required
String

The relative path to the JavaScript file containing the function to get the external job logs.

none

contexts[]
.actions
.getLogs
.function

Required
String

The name of the exported function to execute to get the external job logs.

none


Connection type attributes

The connection type specifies how to establish a connection with the external service. It must be declared in a metadata.yaml file with the declaration of its parameters.

This metadata is used by Saagie to create a connection to your input values that will be provided to the external job scripts.

Attribute Description Default value

version
Required
String

The version of the file format.

The only supported version is v2.

none

type
Required
String

Indicates the type of metadata in the file.

The only supported version is CONNECTION_TYPE.

none

id
Required
String

The connection type’s unique identifier in the repository.

none

label
Required
String

The name of the connection type as it will appear in the user interface.

none

parameters
Required
Array

The connection type parameters.

none

parameters[]
.id

Required
String

The parameter’s unique identifier in your connection type.

none

parameters[]
.type

Required
String

The type of parameter.

The supported types are TEXT, PASSWORD, STATIC_SELECT, and TOGGLE.

none

parameters[]
.label

Required
String

The name of the parameter as it will appear in the user interface.

none

parameters[]
.comment

Optional
String

The description text of the parameter, which appears in the tooltips of the user interface.

none

parameters[]
.mandatory

Optional
String

Indicates if the parameter is mandatory for the connection to be valid.

none

parameters[]
.dependsOn

Optional
Array

The list of parameters on which the current parameter depends.

none

parameters[]
.dependsOn[]

Optional
String

The parameter’s unique identifier on which the current parameter depends.

none

parameters[]
.defaultValue

Optional
String

The default value of the parameter.

Only for TEXT, STATIC_SELECT, and TOGGLE parameter types.
For the TOGGLE parameter type, the default value must be true or false.
For the STATIC_SELECT parameter type, the default value must be the unique identifier of one of the values listed in the staticValues attribute.

none

parameters[]
.staticValues

Required for the STATIC_SELECT parameter type
String

The list of available values.

none

parameters[]
.staticValues[]
.id

Required for the STATIC_SELECT parameter type
String

The unique identifier of the listed values.

It will not be visible in the user interface but will be used as a value in the associated JavaScript file.

none

parameters[]
.staticValues[]
.label

Required
String

The name of the value as it will appear in the user interface.

It will not be processed in the associated JavaScript file but the value of the id attribute will be.

none

actions
.checkConnection

Optional
-

It must be declared in a metadata.yaml file and its function must be exposed by a JavaScript file.

none

actions
.checkConnection
.script

Optional
String

The relative path to the JavaScript file containing the function to check the connection of the external job.

none

actions
.checkConnection
.function

Optional
String

The name of the exported function to execute to check the connection of the external job.

none

2. External jobs configuration

External technologies require a more advanced configuration with the creation of JavaScript files including functions to manage the life cycle of external job instances and to check the connection with the external service.

The configuration of external technologies requires a basic understanding of JavaScript.

Dynamic select

The DYNAMIC_SELECT parameter type is only used at the creation and the modification of the external job.
The script’s function must return an array of objects containing the attributes id and label typed as string.

It must be declared in the metadata.yaml file of the external job and its function must be exposed by a JavaScript file embedded in the repository .zip file.

metadata.yaml
    parameters:
      - type: DYNAMIC_SELECT
        id: jobDefinition
        label: Job Definition
        mandatory: true
        dynamicValues:
          script: path-to-the-script/job-form.js
          function: getBatchJobs
job-form.js
exports.getBatchJobs = async ({connection}) => {
    const client = buildClient(connection);
    const {data} = await client.batch.describeJobDefinitions();
    return data?.jobDefinitions
        ?.filter(({status}) => status === 'ACTIVE')
        ?.map(({jobDefinitionArn, jobDefinitionName}) => ({
            id: jobDefinitionArn,
            label: jobDefinitionName,
        })) ?? [];
};

Life cycle actions

There are four actions used to navigate and manage the life cycle of an external job instance. For each action, there must be a function exposed by a JavaScript file embedded in the repository .zip file.

  • Start the job

  • Stop the job

  • Get the job status

  • Get job logs

start({ connection, parameters }): payload

The start action is used to execute the job. The script’s function must return the payload that will help subsequent calls identify the newly executed job instance ; The returned payload must be a simple JavaScript object containing all the information needed for the other actions.
The action must be called with an object argument, containing information about the connection and parameter values.

Argument Type Default value

params

object { connection, parameters }

none

Example of JavaScript to start a job:
const axios = require('axios');
const axiosHttp = require('axios/lib/adapters/http');

const client = axios.create({
  adapter: axiosHttp
});

exports.start = async ({ connection, parameters }) => {
  // Make a fetch request to start the job using axios
  const result = await client.get(`${connection.url}/datasets/${parameters.dataset}/projects/${parameters.project}/processes/${parameters.process}/run`);
  return { jobId: result.data.jobId }; // the payload associated to the running job
}

stop({ connection, parameters, payload })

The stop action is used to stop a running job. The script’s function can return a payload that will help the call to the getLogs function to identify the instance of the stopped job. This payload will be merged with the payload returned by the call to the start function.
The action must be called with an object argument, containing information about the connection and parameter values, as well as the payload returned by the call to the start function.

Argument Type Default value

params

object { connection, parameters, payload }

none

Example of JavaScript to stop a job:
const axios = require('axios');
const axiosHttp = require('axios/lib/adapters/http');

const client = axios.create({
  adapter: axiosHttp
});

exports.stop = async ({ connection, parameters, payload }) => {
  // Make a fetch request to stop the job
  await client.get(`${connection.url}/datasets/${parameters.dataset.id}/projects/${parameters.project}/processes/${parameters.process}/instance/${payload.jobId}/stop`);
}

getStatus({ connection, parameters, payload }): string

The getStatus action is used to retrieve the current status of a job. The script’s function must return the string representation of the status.
The action must be called with an object argument, containing information about the connection and parameter values, as well as the payload returned by the call to the start function.

Argument Type Default value

params

object { connection, parameters, payload }

none

Example of JavaScript to get the status of a job:
const axios = require('axios');
const axiosHttp = require('axios/lib/adapters/http');

const client = axios.create({
  adapter: axiosHttp
});

const JobStatus = {
  AWAITING: 'AWAITING',
  REQUESTED: 'REQUESTED',
  QUEUED: 'QUEUED',
  RUNNING: 'RUNNING',
  SUCCEEDED: 'SUCCEEDED',
  KILLING: 'KILLING',
  KILLED: 'KILLED',
  FAILED: 'FAILED'
};

exports.getStatus = async ({ connection, parameters, payload }) => {
  // Make a fetch request to get the current status of the job
  const status = await client.get(`${connection.url}/datasets/${parameters.dataset}/projects/${parameters.project}/processes/${parameters.process}/instance/${payload.jobId}/getStatus`);

  if (status.isRunning) {
    // And return the response along with the current status of the job
    return JobStatus.RUNNING;
  }

  return JobStatus.UNKNOWN;
}

getLogs({ connection, parameters, payload }): stream

The getLogs action is used to retrieve the logs of a running job. The script’s function must return a data stream of the logs.
The action must be called with an object argument, containing information about the connection and parameter values, as well as the payload returned by the call to the start function.

Argument Type Default value

params

object { connection, parameters, payload }

none

Example of JavaScript to get the logs of a job:
const axios = require('axios');
const axiosHttp = require('axios/lib/adapters/http');

const client = axios.create({
  adapter: axiosHttp,
  responseType: 'stream'
});

exports.getLogs = async ({ connection, parameters, payload }) => {
  console.log('GET LOG INSTANCE:', payload.jobId);

  // Make a fetch request to get the logs of the job
  const { data } = await client.get(`${connection.url}/datasets/${parameters.dataset}/projects/${parameters.project}/processes/${parameters.process}/instance/${payload.jobId}/getLogs`);

  return data;
};

JavaScript code
The written JavaScript code is run by Node.js and you must have version 16 or later.

The script will be run in a constrained and isolated environment where not all APIs and node modules will be available. Thus, the script must be built in such a way that it contains all the required dependencies at runtime.
The Saagie SDK provides an example of a properly configured build. In addition, you can use the CLI (Command Line Interface) tool developed by Saagie to simplify your development process.

Code written for an external technology can be provided in one or more files. Since each script must contain all its dependencies itself, it’s recommended to have a JavaScript file for each function.

Folder hierarchy example
Here is an example of an external technology exposing two contexts, with two different file organizations:

Folder hierarchy example
Figure 3: Folder hierarchy example
  • 1.0

  • 1.1

context.yaml
# [...]
  actions:
    start:
      script: instanceActions.js
      function: start
    stop:
      script: instanceActions.js
      function: stop
    getStatus:
      script: instanceActions.js
      function: getStatus
# [...]
instanceActions.js
exports.start = async ({ connection, parameters }) => {
  // Logic to start the external job instance.
}

exports.stop = async ({ connection, parameters, payload }) => {
  // Logic to stop the external job instance.
}

exports.getStatus = async ({ connection, parameters, payload }) => {
  // Logic to get the job instance status.
}
context.yaml
# [...]
  actions:
    start:
      script: start.js
      function: run
    stop:
      script: stop.js
      function: run
    getStatus:
      script: getStatus.js
      function: run
# [...]
getStatus.js
exports.run = async ({ connection, parameters, payload }) => {
  // Logic to get the job instance status.
}
start.js
exports.run = async ({ connection, parameters }) => {
  // Logic to start the external job instance.
}
stop.js
exports.run = async ({ connection, parameters, payload }) => {
  // Logic to stop the external job instance.
}

Check connection

The checkConnection action is used to verify that Saagie is able to connect to the external service via the connection credentials.
The script’s function must return a JavaScript object with the ok field as a boolean.

When activated, a Check connection button appears in the user interface.
Click the button to start the verification. If the connection is valid, a green tick appears. Similarly, if the connection isn’t valid, a red cross appears.

screen check connection button

The checkConnection action must be declared in the metadata.yaml file of the connection type and its function must be exposed by a JavaScript file embedded in the repository .zip file.

metadata.yaml
actions:
  checkConnection:
    script: path-to-the-script/check-connection.js
    function: checkConnection
check-connection.js
exports.checkConnection = async ({connection}) => {
    const client = buildClient(connection);
    await client.sts.getCallerIdentity();
    return {
        ok: true
    };
};

3. Metadata file examples

Here are some examples of metadata.yaml files for each type of technology.

You can find more examples in the Saagie technology repository.
  • Job technology

  • App technology

  • External technology

Example for Java/Scala 8, 11, and 13.

metadata.yaml
version: "v1"
type: JOB
id: java-scala
label: Java/Scala
available: true
icon: java-scala
contexts:
  - id: 11
    label: 11
    available: true
    recommended: true
    dockerInfo:
      image: saagie/java-scala
      version: 11-1.10.0_merge_certified_experimental
    trustLevel: "stable"
    job:
      features:
        - type: COMMAND_LINE
          label: "Command line"
          mandatory: true
          comment: "Linux shell command to launch the job."
          defaultValue: java -jar {file} arg1 arg2
        - type: ARTIFACT
          label: "Package"
          mandatory: true
          comment: "Compatible upload file : .jar"
        - type: SCHEDULER
          label: "Scheduled"
          mandatory: true
  - id: 13
    label: 13
    available: true
    recommended: false
    dockerInfo:
      image: saagie/java-scala
      version: 13-1.10.0_merge_certified_experimental
    trustLevel: "experimental"
    job:
      features:
        - type: COMMAND_LINE
          label: "Command line"
          mandatory: true
          comment: "Linux shell command to launch the job."
          defaultValue: java -jar {file} arg1 arg2
        - type: ARTIFACT
          label: "Package"
          mandatory: true
          comment: "Compatible upload file : .jar"
        - type: SCHEDULER
          label: "Scheduled"
          mandatory: true
  - id: 8
    label: 8
    available: true
    recommended: false
    dockerInfo:
      image: saagie/java-scala
      version: 8-1.10.0_merge_certified_experimental
    trustLevel: "stable"
    job:
      features:
        - type: COMMAND_LINE
          label: "Command line"
          mandatory: true
          comment: "Linux shell command to launch the job."
          defaultValue: java -jar {file} arg1 arg2
        - type: ARTIFACT
          label: "Package"
          mandatory: true
          comment: "Compatible upload file : .jar"
        - type: SCHEDULER
          label: "Scheduled"
          mandatory: true

Example for Jupyter Notebook.

metadata.yaml
version: v1
type: APP
id: jupyter-doc
label: Jupyter Notebook
baseline: Data Science Notebook
backgroundColor: "#E87A35"
description: "The Jupyter Notebook is an open-source web application..."
available: true
icon: jupyter
customFlags: []
contexts:
  - id: "6.1.1"
    label: Jupyter Notebook
    releaseNotes: "Prevent inclusion of requests_unixsocket on Windows"
    available: true
    trustLevel: stable
    dockerInfo:
      image: saagie/jupyter-python-nbk
      version: v2
    ports:
      - port: 8888
        name: Notebook
        rewriteUrl: false
        basePath: SAAGIE_BASE_PATH
    volumes: ["/notebooks-dir"]

Examples for AWS Lambda.

  1. A metadata file declaring the AWS connection.

    metadata.yaml
    version: v2
    id: aws-connection
    label: AWS Connection
    description: Connection to AWS
    type: CONNECTION_TYPE
    
    parameters:
      - type: TEXT
        id: aws_access_key_id
        label: Access Key ID
        mandatory: true
      - type: PASSWORD
        id: aws_secret_access_key
        label: Secret Access Key
        mandatory: true
      - type: TEXT
        id: region
        label: Region
        comment: "AWS region. Example: us-east-1"
        mandatory: true
    
    actions:
      checkConnection:
        script: path-to-the-script/check-connection.js
        function: checkConnection
  2. A metadata file declaring the AWS Lambda parameters and scripts.

    metadata.yaml
    version: v2
    id: aws-lambda
    label: aws-lambda
    description: "Run code without thinking about servers. Pay only for the compute time you consume. Only lambda with Kinesis, SQS and DynamoDB Stream. Api Version: 2015-03-31"
    #available: true
    type: EXTERNAL_JOB
    icon: job
    
    contexts:
      - id: functions
        label: functions
        description: "Only lambda with Kinesis, SQS and DynamoDB Stream."
        recommended: true
        trustLevel: experimental
        connectionTypeId: aws-connection
        parameters:
          - type: DYNAMIC_SELECT
            id: functions
            label: Functions
            comment: Only lambda with Kinesis, SQS and DynamoDB Stream.
            mandatory: true
            dynamicValues:
              script: ./functions/module.js
              function: getFunctions
        actions:
          start:
            script: ./functions/module.js
            function: start
          stop:
            script: ./functions/module.js
            function: stop
          getStatus:
            script: ./functions/module.js
            function: getStatus
          getLogs:
            script: ./functions/module.js
            function: getLogs