Create and manage technologies

This documentation introduces the technology catalog, which allows new technologies to be saved to Saagie.

1. Introduction

The technology catalog allows users to add their own technologies to Saagie.

Complete the following steps to add a new technology:

Technology creation process
Figure 1: Technology creation process

2. Repository

The first step is to create a repository. There are two ways to do this: fetch from a URL or upload a zip archive.

2.1. Fetch from URL

The repository must be exposed using a .zip file accessible to Saagie by URL. You can use any HTTP server to do so. (example: Nginx)

For example, Saagie’s technology repository is exposed using a Github .zip file accessible by URL: https://github.com/saagie/technologies/releases/latest/download/technologies.zip.

Add the new repository by providing the following information:

  1. a unique name (255 characters maximum)

  2. a URL leading to the .zip file

2.2. Upload zip archive

Upload a .zip file containing the technologies. This option is useful if you are on offline mode or if you want to test your repositoryquicly.

Add the new repository by providing the following information:

  1. a unique name (255 characters maximum)

  2. a .zip file

When a repository is created, the initial synchronization is performed automatically based on the information in the .zip file.

3. Technology files

Saagie uses two types of technologies: jobs and apps. To add a new technology, you’ll need to:

  • Create a metadata.yaml file (or files).

  • Archive your file or files in a .zip file.

Saagie supports both .yml and .yaml extensions.

Below are some examples. You can find more examples in the Saagie technologies repository.

3.1. Technology file examples

3.1.1. Job technology

Example for Java/Scala.

The contexts correspond to Java versions 8, 11, and 13.

metadata.yaml
version: "v1"
type: JOB
id: java-scala
label: Java/Scala
available: true
icon: java-scala
contexts:
  - id: 11
    label: 11
    available: true
    recommended: true
    dockerInfo:
      image: saagie/java-scala
      version: 11-1.10.0_merge_certified_experimental
    trustLevel: "stable"
    job:
      features:
        - field: COMMAND_LINE
          label: "Command line"
          mandatory: true
          comment: "Linux shell command to launch the job."
          defaultValue: java -jar {file} arg1 arg2
        - field: ARTIFACT
          label: "Package"
          mandatory: true
          comment: "Compatible upload file : .jar"
        - field: SCHEDULER
          label: "Scheduled"
          mandatory: true
  - id: 13
    label: 13
    available: true
    recommended: false
    dockerInfo:
      image: saagie/java-scala
      version: 13-1.10.0_merge_certified_experimental
    trustLevel: "experimental"
    job:
      features:
        - field: COMMAND_LINE
          label: "Command line"
          mandatory: true
          comment: "Linux shell command to launch the job."
          defaultValue: java -jar {file} arg1 arg2
        - field: ARTIFACT
          label: "Package"
          mandatory: true
          comment: "Compatible upload file : .jar"
        - field: SCHEDULER
          label: "Scheduled"
          mandatory: true
  - id: 8
    label: 8
    available: true
    recommended: false
    dockerInfo:
      image: saagie/java-scala
      version: 8-1.10.0_merge_certified_experimental
    trustLevel: "stable"
    job:
      features:
        - field: COMMAND_LINE
          label: "Command line"
          mandatory: true
          comment: "Linux shell command to launch the job."
          defaultValue: java -jar {file} arg1 arg2
        - field: ARTIFACT
          label: "Package"
          mandatory: true
          comment: "Compatible upload file : .jar"
        - field: SCHEDULER
          label: "Scheduled"
          mandatory: true

3.1.2. App technology

Example for Jupyter notebook.

metadata.yaml
version: v1
type: APP
id: jupyter
label: Jupyter Notebook
baseline: Data Science Notebook
backgroundColor: "#E87A35"
description: "The Jupyter Notebook is an open-source web application..."
available: true
icon: jupyter
customFlags: []
contexts:
  - id: "6.1.1"
    label: Jupyter Notebook
    releaseNotes: "Prevent inclusion of requests_unixsocket on Windows"
    available: true
    trustLevel: stable
    dockerInfo:
      image: saagie/jupyter-python-nbk
      version: v2
    ports:
      - port: 8888
        name: Notebook
        rewriteUrl: false
        basePath: SAAGIE_BASE_PATH
    volumes: ["/notebooks-dir"]

3.2. Attributes

Technologies can have a variety of attributes. Some are common to jobs and apps while others are specific to either jobs or apps.

3.2.1. Common attributes

The following attributes are common to jobs and apps.

Table 1. Common attributes
Attribute Details Default value

version
Required
String

File format: metadata.yaml
The version is determined by Saagie.

Current version: v1

id
Required
String

Technology’s unique identifier in the repository.

none

label
Optional
String

Name of the technology as it will appear in the User Interface.

id value
(unless another is provided)

type
Required
String

Indicate whether it is a job or app technology.

JOB or APP

icon
Optional
String

Name of the image as it appears in the Saagie Design System.

none

available
Optional
Boolean

Masks the technology to Saagie platform users for whom the response is false.

true

description
Optional
String (multiline)

Description of the technology as it will appear in the User Interface.

none

If a field is not described in this table, that means the field is for internal use only.
If you encounter problems with one of these fields, contact Saagie Support.

3.2.2. Job attributes

The following attributes apply to jobs only.

Table 2. Job attributes
Attribute Details Default value

contexts
Required
Array

none

none

contexts[]
.id

Required
String

The unique identifier of the context in your technology.

none

contexts[]
.label

Optional
String

Name of the context as it will appear in the User Interface.

id value
(unless another is provided)

contexts[]
.description

Optional
String (multiline)

The description of the context displayed in the User Interface.
See Multiline file description below.

none

contexts[]
.available

Optional
Boolean

Set to false to prevent Saagie from displaying the context to users.

true

contexts[]
.recommended

Optional
Boolean

Set to true to recommend this context to users.

false

contexts[]
.trustLevel

Optional
String

Set to experimental to warn the user that the context is not stable.

stable or experimental

contexts[]
.endpoint

Configuration for the endpoint creation form.
This form allows the user to create one or more endpoint connections for this technology context.
See Endpoint file example below.

none

contexts[]
.endpoint
.checkConnection

Provide a method to check the status of the endpoint connection.

none

contexts[]
.endpoint
.checkConnection
.script

Optional
String

Relative path to JavaScript file where there is a function to check the status of the endpoint connection.

none

contexts[]
.endpoint
.checkConnection
.function

Optional
String

The name of the exported function to run when the check endpoint connection action is called.

If no function name is provided, the default export of the script will be used.

contexts[]
.endpoint
.features

Refer to the list of available Features in the endpoint creation form.

none

contexts[]
.job

Configuration for the job creation form.
This form allows the user to create one or more job for this technology context.

none

contexts[]
.job
.features

Refer to the list of available Features in the job creation form.

none

contexts[]
.instance

Configuration for the job instances.

none

contexts[]
.instance
.actions

Configuration for each instance action.

none

contexts[]
.instance
.actions
.onStart

Provide a method to start the job.

none

contexts[]
.instance
.actions
.onStart
.script

Required
String

Relative path to JavaScript file where there is a function to start the job.

none

contexts[]
.instance
.actions
.onStart
.function

Optional
String

The name of the exported function to execute when the start job action is called.

If no function name is provided, the default export of the script will be used.

contexts[]
.instance
.actions
.onStop

Provide a method to stop the job.

none

contexts[]
.instance
.actions
.onStop
.script

Optional
String

Relative path to JavaScript file where there is a function to stop the job.

none

contexts[]
.instance
.actions
.onStop
.function

Optional
String

The name of the exported function to execute when the stop job action is called.

If no function name is provided, the default export of the script will be used.

contexts[]
.instance
.actions
.getStatus

Provide a method to get the job status.

none

contexts[]
.instance
.actions
.getStatus
.script

Required
String

Relative path to JavaScript file where there is a function to get the job status.

none

contexts[]
.instance
.actions
.getStatus
.function

Optional
String

The name of the exported function to execute when the get job status action is called.

If no function name is provided, the default export of the script will be used.

contexts[]
.instance
.actions
.getLogs

Provide a method to get the job logs.

none

contexts[]
.instance
.actions
.getLogs
.script

Optional
String

Relative path to JavaScript file where there is a function to get the job logs.

none

contexts[]
.instance
.actions
.getLogs
.function

Optional
String

The name of the exported function to execute when the get job logs action is called.

If no function name is provided, the default export of the script will be used.

Example files: Job attributes
Multiline file description
description: >
  Several lines of text,
  with some "quotes" of various 'types',
  plus another line at the end.
Endpoint file example
endpoint:
  checkConnection:
    script: ./src/checkConnection.js
    function: checkConnection
  features:
    - type: URL
      name: url
      label: Endpoint URL
      required: true
      helper: e.g. https://amazingtech.company.com/api/v1

3.2.3. App attributes

The following attributes apply to apps only.

Table 3. App attributes
Attribute Details Default value

baseline
Optional
String

The baseline description of the application.

none

backgroundColor
Optional
String

The background color for the app card in the user interface.

none

contexts
Required
Array

none

none

contexts[]
.id

Required
String

The unique identifier of the context in your application.

none

contexts[]
.label

Optional
String

Name of the context as it will appear in the user interface.

id value
(unless a different value is provided)

contexts[]
.description

Optional
String (multiline)

The description of the context displayed in the user interface.
See Multiline file description below.

none

contexts[]
.available

Optional
Boolean

Set to false to prevent Saagie from displaying the context to users.

true

contexts[]
.releaseNotes

Optional
String (multiline)

Release notes for this context.

none

contexts[]
.recommended

Optional
Boolean

Set to true to recommend this context to users.

false

contexts[]
.trustLevel

Optional
String

Set to experimental to indicate that the context is not stable.

stable or experimental

contexts[]
.dockerInfo

Required

Information about a technology’s Docker image.

none

contexts[]
.dockerInfo
.image

Required
String

Name of technology’s Docker image.

none

contexts[]
.dockerInfo
.version

Required
String

Version (or tag) of the Docker image.

latest

contexts[]
.ports

Required
Array

List open ports available to a Docker container.

none

contexts[]
.ports[]
.port

Required
Integer

Port number.

none

contexts[]
.ports[]
.name

Optional
String

Label the port, describing what it is used for.

none

contexts[]
.ports[]
.basePath

Optional
String

Name of the environment variable that will contain the URL generated by Saagie.

none

contexts[]
.volumes

Optional
Array

List of Docker volumes paths used by the application.

none

contexts[]
.volumes[]

String

Volume path.
Required* only if using volumes.

none

3.3. Features

All external technology jobs have at least one context that contains its own list of features.

A feature is defined by the following values:

Table 4. Features
Attribute Details Possible values

field

Type of field that will be displayed in the User Interface

COMMAND_LINE
ARTIFACT
SCHEDULER
GPU
DOCKER_INFO

label

Information displayed next to a field.

Character string

mandatory

Indicates whether or not a field is required.

true or false

comment

Information appearing next to a field that provides examples for how to complete the field.

Character string

defaultValue

Example of a field’s default value, such as a command line.

Character string

3.4. Create a .zip file

Once you’ve created your metadata.yml files, you need to archive them in a .zip file.

Your .zip file can contain multiple types of files. Only files named metadata.yaml or metadata.yml will be processed.

For example, you can create a .zip file containing all of the content of the current technologies/ directory using the following command:

zip technologies.zip -r technologies/

4. External technologies

External technology jobs are not run in Saagie, but use external services instead. When used with Saagie, external technologies keep their properties and use the same mode of execution as internal jobs (such as status, logs, and pipelines).

Using external technologies
Figure 2: External technologies
This documentation assumes a basic understanding of JavaScript, which is used to integrate external technologies.

4.1. Setting up an external technology

In order to set up an external technology, you’ll need to provide information about how to connect to the external service.

There are three major parts:

  1. Endpoints correspond to connection points to external services.

  2. Jobs correspond to configuration information used to launch an instance of a job.

  3. Instances correspond to actions used to navigate and manage the life cycle of an instance for an external job.

There are also external features available to endpoints and jobs.

4.1.1. Endpoints

This configuration feature allows you to decide which fields you’ll complete when creating an endpoint. It is composed of a list of features that do not launch JavaScript scripts.

Features available for endpoints are: URL, TEXT, and PASSWORD.

Endpoint example in context.yaml
# [...]
endpoint:
  features:
    - type: URL
      name: url
      label: Endpoint URL
      required: true
      helper: e.g. https://amazingtech.company.com/api/v1

    - type: TEXT
      name: login
      label: Login

    - type: PASSWORD
      name: password
      label: Password
# [...]
Endpoint

4.1.2. Jobs

This configuration feature allows you to decide which fields you’ll complete when creating a job. It is composed of a list of features.

Job example in context.yaml
# [...]
job:
  features:
    - type: ENDPOINT
      name: endpoint
      label: Endpoint
      required: true

    - type: SELECT
      name: dataset
      label: Dataset
      required: true
      options:
        script: ./src/entityForm.js
        function: getDatasets
      dependsOn:
        - connection

    - type: SELECT
      name: project
      label: Project
      required: true
      options:
        script: ./src/entityForm.js
        function: getProjects
      dependsOn:
        - dataset

    - type: SELECT
      name: process
      label: Process
      required: true
      options:
        script: ./src/entityForm.js
        function: getProcesses
      dependsOn:
        - project
# [...]
Job

4.1.3. Instances

There are four actions used to navigate and manage the life cycle of an external job’s instance:

Folder hierarchy example
Figure 3: Folder hierarchy example
1.0/context.yaml
# [...]
instance:
  actions:
    onStart:
      script: instanceActions.js
      function: start
    onStop:
      script: instanceActions.js
      function: stop
    getStatus:
      script: instanceActions.js
      function: getStatus
# [...]
1.0/instanceActions.js
exports.start = async ({ job, instance }) => {
  // Logic to start the external job instance.
}

exports.stop = async ({ job, instance }) => {
  // Logic to stop the external job instance.
}

exports.getStatus = async ({ job, instance }) => {
  // Logic to get the job instance status.
}
1.1/context.yaml
# [...]
instance:
  actions:
    onStart:
      script: start.js
    onStop:
      script: stop.js
    getStatus:
      script: getStatus.js
# [...]
1.1/start.js
exports = async ({ job, instance }) => {
  // Logic to start the external job instance.
}
1.1/stop.js
exports = async ({ job, instance }) => {
  // Logic to stop the external job instance.
}
1.1/getStatus.js
exports = async ({ job, instance }) => {
  // Logic to get the job instance status.
}
Start: start({ job, instance }): Response

The start method contains the code that will run the job. It should return a success or an error, using the custom Response object implemented in the sdk library.

This method should be called with one object argument, holding information about the job and the instance.

Table 5. start
Argument Type Default value

params

object { job, instance }

none

start example
const axios = require('axios');
const { Response } = require('@saagie/sdk');

exports.start = async ({ job, instance }) => {
  try {
    // Make a fetch request to start the job using axios
    await axios.get(`${job.featuresValues.endpoint.url}/datasets/${job.featuresValues.dataset.id}/projects/${job.featuresValues.project.id}/processes/${job.featuresValues.process.id}/run`);

    // And return the response
    return Response.success();
  } catch (error) {
    // Handle the error
    return Response.error('Fail to start the job', { error })
  }
}
Stop: stop({ job, instance }): Response

The stop method contains the code that will stop a running job. It should return a success or an error, using the custom Response object implemented in the sdk library.

This method should be called with one object argument, holding information about the job and the instance.

Table 6. stop
Argument Type Default value

params

object { job, instance }

none

stop example
const axios = require('axios');
const { Response } = require('@saagie/sdk');

exports.stop = async ({ job, instance }) => {
  try {
    // Make a fetch request to stop the job
    await axios.get(`${job.featuresValues.endpoint.url}/datasets/${job.featuresValues.dataset.id}/projects/${job.featuresValues.project.id}/processes/${job.featuresValues.process.id}/stop`);

    // And return the response
    return Response.success();
  } catch (error) {
    // Handle the error
    return Response.error('Fail to stop the job', { error })
  }
}
getStatus: getStatus({ job, instance }): Response

The getStatus method contains the code that will get the current status of a job. It should return a success with the current job status (from the Status enum in @saagie/sdk lib), or an error, using the custom Response object implemented in the sdk library.

getStatus should be called with one object argument holding information about the job and the instance.

Table 7. getStatus
Argument Type Default value

params

object { job, instance }

none

getStatus example
const axios = require('axios');
const { Response, Status } = require('@saagie/sdk');

exports.getStatus = async ({ job, instance }) => {
  try {
    // Make a fetch request to get the current status of the job
    const status = await axios.get(`${job.featuresValues.endpoint.url}/datasets/${job.featuresValues.dataset.id}/projects/${job.featuresValues.project.id}/processes/${job.featuresValues.process.id}/getStatus`);

    if (status.isRunning) {
      // And return the response along with the current status of the job
      return Response.success(Status.RUNNING);
    }

    return Response.success(Status.UNKNOWN);
  } catch (error) {
    // Handle the error
    return Response.error(`Fail to get the status of job "${name}"`, { error })
  }
}
getLogs: getLogs({ job, instance }): Response

The getLogs method contains the code that will get the logs of a running job. It should return a success with the current job logs, or an error, using the custom Response object implemented in the sdk library.

getLogs should be called with one object argument holding information about the job and instance.

Table 8. getLogs
Argument Type Default value

params

object { job, instance }

none

getLogs example
const axios = require('axios');
const { Response, Stream, Log } = require('@saagie/sdk');

/**
 * Logic to retrieve the external job instance logs.
 * @param {Object} params
 * @param {Object} params.job - Contains job data including featuresValues.
 * @param {Object} params.instance - Contains instance data including the payload returned in the start function.
 */
exports.getLogs = async ({ job, instance }) => {
  try {
    console.log('GET LOG INSTANCE:', instance);

    // Make a fetch request to get the logs of the job
    const { data } = await axios.get(
      `${job.featuresValues.endpoint.url}/api/demo/datasets/${job.featuresValues.dataset.id}/logs`,
    );

    // And return the response along with the logs of the job
    // Return an array of Log with the following parameters:
    // - The line of log
    // - The kind of output : STDERR and STDOUT will have different behavior in the UI for example.
    // - The date in ISO 8601 format.
    return Response.success(data.logs.map((item) => Log(item.log, item.output === 'err' ? Stream.STDERR : Stream.STDOUT, item.date)));
  } catch (error) {
    return Response.error(`Fail to get logs of job "${name}"`, { error })
  }
};
Params

params is the argument containing information about the job being managed, whether a job is being run, stopped, or monitored via status or logs.

Table 9. params
Argument Type Details

params.job

object

Contains the job data.

params.job
.featuresValues

object

Contains all the values from the declared features.

params
.instance

object { id, payload? }

Contains the identifier of the instance and the optional value that may be returned by the start function.

4.1.4. External Features

External features are used by jobs and endpoints.

Table 10. External features
Feature Details

URL
code

Field with URL type validation.
All shared configurations are available for this feature.

TEXT
code

Open field where users can give any value they want.

PASSWORD
code

Field with password type validation.

TEXTAREA
code

Open larger field where users can give any value they want.

ENDPOINT
code

Select box with the previously-created endpoints.

SELECT
code

Select a value from a list of options. This list can be hard-coded or dynamically-loaded using scripts.

SELECT option: Hard-coded
code

Hard-coded options can be provided using an array made of id and label. The label is what the user will see on the User Interface.

SELECT option: dynamic
code

Dynamic options system provide a way to call a web service to get the options. Use script and function keys to provide a path to a script and a function to call.

RADIO option: Hard-coded
code

Hard-coded options must be provided using an array made of label and value. The label is what the user will see on the User Interface, while value is what the user will send.

Code examples
URL feature code
- type: URL
  name: url
  label: Endpoint URL
  required: true
  helper: e.g. https://amazingtech.company.com/api/v1
  value:
    script: ./jobForm.js
    function: getEndpointURL
  dependsOn:
    - dependency-name
TEXT feature code
- type: TEXT
  name: login
  label: Login
  required: true
  helper: e.g. This is a text
  value:
    script: ./jobForm.js
    function: getLogin
  dependsOn:
    - dependency-name
PASSWORD feature code
- type: PASSWORD
  name: password
  label: Password
  required: true
  helper: Please provide a password
  dependsOn:
    - dependency-name
TEXTAREA feature code
- type: TEXTAREA
  name: description
  label: Description
  required: true
  helper: e.g. This is a text area
  value:
    script: ./jobForm.js
    function: getDescription
  dependsOn:
    - dependency-name
ENDPOINT feature code
- type: ENDPOINT
  name: endpoint
  label: Endpoint
  required: true
  helper: Please select the endpoint to use
  dependsOn:
    - dependency-name
SELECT feature code
# Using script
- type: SELECT
  name: dataset
  label: Dataset
  required: true
  options:
    script: ./src/script.js
    function: getDatasets
  dependsOn:
    - endpoint

# Using hard coded values
- type: SELECT
  name: dataset-hard-coded
  label: Dataset Hard Coded
  required: true
  options:
    - id: first
      label: First
    - id: second
      label: Second
    - id: third
      label: Third
  dependsOn:
    - endpoint
Hard-coded options code
- type: SELECT
  name: project
  options:
    - id: first
      label: First
    - id: second
      label: Second
    - id: third
      label: Third
Dynamic options code
- type: SELECT
  name: project
  options:
    script: ./entityForm.js
    function: getProjects
RADIO feature code
- type: RADIO
  name: region
  label: DataCenter Region
  required: true
  options:
    - label: Europe West
      value: eu-west
    - label: Europe East
      value: eu-east
Shared configurations

Each feature has multiple configuration keys. Most of the keys are basic, with type being more complicated.

Table 11. Shared configurations
Attribute Details Default value

type
Required
String

Available feature types are described in the previous table.

none

name
Required
String

The feature name should be unique in the context.

none

label
Optional
String

The label of the feature displayed in the User Interface.

name value
(unless another is provided)

helper
Optional
String

The helper displayed on the User Interface to clarify the feature.

none

required
Optional
Boolean

The requirement of the feature.

false

dependsOn
Optional
Array

Identifies any feature that must be successfully filled before this feature shows up in the User Interface.

none

Shared configurations code
- type: TEXT
  name: login

- type: PASSWORD
  name: password
  dependsOn:
    - login
Specific configurations

Some feature types have additional configurations available.

Table 12. Specific configurations
Attribute Details Default value

value.script
Optional
String
(TEXT, URL)

JavaScript file path to call to get the input value autofill.

none

value.function
Optional
String
(TEXT, URL)

Name of the function to call. If none is given, the function should be the script’s default export.

none

options.script
Optional
String
(SELECT)

JavaScript file path to call to get the select options.

none

options.function
Optional
String
(SELECT)

Name of the function to call to get the select options. If none is given, the function should be the script’s default export.

none

4.2. Developing an external technology

4.2.1. JavaScript code

The written JavaScript code is run by Node.JS. You must have the version 12.15.0 or later, as well as axios version 0.19.2.

Written code for an external technology can be provided in one or multiple files. Generally, each action to be developed corresponds to one JavaScript function.

Create a project:

npm init -y
npm install axios@0.19.2 @saagie/sdk

4.2.2. Saagie SDK Command Line Interface

You can use the CLI tool developed by Saagie to simplify your development process. For more information, refer to our SDK repository.

5. Synchronization

Synchronization allows users to update Saagie with the technologies present in their repository. Users can add new technologies, update existing ones or delete existing ones.

Synchronization methods:

  • Initial synchronization occurs automatically when a repository is created, according to the information in the .zip file.

  • Manual synchronization is described in detail below.

Before synchronization can take place, you must have created the following: technology file, .zip file, and your repository. If you have not created these yet, return to the Introduction before continuing.

5.1. Manual synchronization

When a user selects Synchronize now in a repository:

  • All client repository technologies will be compared with the ones in the .zip file:

  • If a technology is present in both the client repository and the .zip file, it will be updated with the information from the metadata.yaml. Example: Python and Jupyter in fig. 3.

  • Else if a technology is only present in the .zip file, it will be created as a new technology in the client repository. All users with access to this repository will now have access to the new technology. Example: Spark in fig. 3.

  • Else if a technology is only present in the client repository, it will be removed from the client repository. Example: Java-Scala in fig. 3.

  • At the technology contexts level, the same synchronization rules are applied.

If a technology is removed during the synchronization process, no user with access to the client repository will be able to access to this technology anymore.
Repository synchronization
Figure 4: Repository synchronization

6. Congratulations

Once you have completed each step, your new technologies are available in Projects. Saagie users can use these technologies to create jobs and apps from a repository to which they have access.