Repositories

1. Library

When you first navigate to the Technology Catalog, you’ll land in the library of repositories.

Screenshot of the catalog library homepage with five sections outlined

  1. The Technology Catalog is a cross-platform module. You cannot navigate between platforms directly from the catalog.

  2. The breadcrumbs in the top bar display your current location.

  3. The top bar title is the title of the current page. In this case, you’re on Repositories.

  4. Also in the top bar are buttons to:

    • Update (refresh) your page

    • Add a repository

  5. Use the search bar to search for repositories by name only.

  6. Finally, you’ll see a collection of cards featuring of all of the repositories available on this platform. The cards feature the name of each repository, how many technologies are included in the repository, and when the repository was last updated.

Notice the icons next to the word Updated.

  • The arrows indicate a repository is fetched from a URL.

  • The cloud with an upward-facing arrow indicates a repository is uploaded with a .zip file.

Screenshot of the catalog library cards calling out sync and upload icons

2. Overview

The overview page is the first page you’ll reach when you navigate to a repository. It displays essential information about the repository and links to other important repository details.

Screenshot of the repository overview homepage with seven sections outlined

  1. The second-level navigation column has links to the repository’s overview and history pages.

  2. The breadcrumbs in the top bar show which repository you’re viewing. You can also use the breadcrumbs as navigation.

  3. The top bar title is the title of the current page.

  4. Also in the top bar are buttons to:

    • update your page,

    • access the settings for your repository, and

    • update the technologies in your repository.

  5. In the body of the page layout, there is a section featuring details about the repository, including:

    • the type of repository,

    • when the repository was added,

    • who added the repository, and

    • the repository’s URL or the name of the .zip file.

  6. The next section in the body of the page provides details about the repository’s status, including:

    • the repository’s last update,

    • who updated the repository,

    • the number of technologies changed with the last update, and

    • the total number of times the repository has been updated.

  7. The final section in the body of the page features details about the technologies available in the repository. Refer to the section on technologies (later on this page) for more details.

3. Settings

Settings for repositories in the Technology Catalog are limited. They include:

  • Name

  • URL (fetch from URL repositories only)

  • Delete repository

3.1. Name

  1. Change the name of the repository.

    • Names cannot exceed 255 characters.

    • You cannot repeat repository names.

3.2. URL

For fetch from URL repositories only, there are two additional settings.

  1. Update your repository’s URL.

  2. Automatically update your technologies after saving the changes to the repository’s settings.

    Choosing to automatically update your technologies will only apply to this instance of saving your settings.
    Catalog settings highlighting URL and automatic update checkbox

3.3. Delete repository

The settings section has a tab called the Danger Zone.

Catalog settings Danger Zone section

In the Danger Zone, you can choose to delete your repository.

You will be prompted to confirm the deletion, and there is also a small amount of time while the deletion is being processed during which you can cancel the deletion.

Deleting a repository will disable all jobs and apps using technologies from that repository.

4. History

On a repository’s history page, you can view all updates made in that repository and details about each event.

Screenshot of the repository history homepage with five sections outlined

  1. The side of the history timeline displays when each update occurred, as well as who performed the update.

  2. Each update starts with a brief description of the changes to the repository. In the example image, 26 technologies were updated.

  3. The timeline also displays the total number of times a repository has been updated.

  4. Reverts are performed from the catalog history page. Refer to the section on repository reverts (later on this page) for more information.

  5. The details section lists:

    • each technology,

    • the change that happened to each technology (indicated by an icon),

    • whether the technology can be used by jobs or apps, and

    • which contexts of the technology were changed.

4.1. Changes to technologies

There are four possible changes that can happen to technologies.

  1. Update: The technology Nifi was updated, and more specifically, its context Nifi 1.9.2.

    Snipit of an updated technology

  2. Add: The technology RStudio 3.4.2 was added.

    Snipit of an added technology

  3. Delete: The technology Spark History Server was deleted.

    Snipit of a deleted technology

  4. Replace: The technology Zeppelin Notebook was updated, replacing the context Zeppelin 0.9.0 with context Zeppelin 0.7.3.

    Snipit of a replaced technology

5. Technologies

Here are the technologies and their contexts that have been tested and approved by Saagie. This list will evolve as we test more technologies and their contexts.

You can add technologies that do not appear on this list, but be aware that we have not tested them.
Table 1. Job technologies
Job technology Available contexts Deprecated contexts Disabled contexts

Bash

Debian-10-buster
Debian-10-buster-aws
Debian-10-buster-azure
Debian-10-buster-gcp

none

Debian-8-jessie
Debian-9-stretch
Debian-9-stretch-aws
Debian-9-stretch-azure
Debian-9-stretch-gcp

Java/Scala

11

none

7, 8

Python

3.7, 3.8, 3.9

none

2.7, 2.7.15
3.5, 3.6, 3.6.6

SQOOP

1.4.6

none

none

Spark

2.4 Java/Scala 11
2.4 Python 3.7
3.0 Java/Scala 11
3.0 python 3.7
3.1 Java/Scala 11
3.1 python 3.7

none

2.4 Java/Scala 8
2.4 Python 2.7
2.4 Python 3.5
2.4 Python 3.6
3.0 Java/Scala 8
3.0 Python 3.5
3.0 Python 3.6
3.1 Java/Scala 8
3.1 python 3.6

Talend

Use_Java_11

Use_Java_8

none

R

4.1.0

4.0

3.4.4, 3.5.3, 3.6.2, 3.6.3

Generic

Docker

none

none
Table 2. App technologies
App technology Available contexts Deprecated contexts Disabled contexts

CloudBeaver

1.2.0

none

none

Dash

2.0.0

none

none

Docker image

Any Docker image

none

none

Grafana

8.2

none

none

Jupyter Notebook

Python 3.6

none

none

Jupyter Notebook for Spark

2.4 Python 3.6

none

none

JupyterLab Notebook

Python 3.8, Python 3.9

none

none

Kibana

7.6.2

5.6.3
6.4.3, 6.8.4

5.6.16
6.4.1, 6.8.1
7.4.0, 7.4.1, 7.4.2, 7.5.1

MLFlow Server

1.20

1.16

none

Nifi

Any Nifi context

none

none

OrientDB

3.1.7

none

2.1.25, 2.2.37
3.0.35

RStudio

4.1.0

3.6.2

3.4.2

Zeppelin

0.9.0

none

0.7.3

6. Update repository

Performing updates and reverts in your Technology Catalog can have unexpected consequences for existing jobs, pipelines, and apps.

If a job, pipeline, or app uses a technology or technology context that is changed or deleted because of an update or a revert, it will not be able to run. Depending on the situation, you’ll need to modify or possibly delete and recreate the job, pipeline, or app before continuing.

Updating your repository can include a combination of three events:

  1. Add: If you added new technologies to your repository or .zip archive, they’ll be added to the repository on your Saagie platform.

  2. Update: If you updated existing technologies in your repository or .zip archive, they’ll be updated in the repository on your Saagie platform.

  3. Delete: If you deleted technologies from your repository or .zip archive, they’ll be deleted from the repository on your Saagie platform.

If you add, update, or delete a technology by mistake, you can revert to the previous version of the repository.

There are slight differences between updating a URL repository and a ZIP repository:

  • URL: Updating a URL repository synchronizes the repository on your Saagie platform with the repository fetched from the URL.

  • ZIP: Updating a ZIP repository requires the upload of a new .zip archive.

When updating technologies, there is also a button to edit the URL for your repository. Selecting this button will open repository settings where you can modify the URL.

7. Revert to a previous update

Performing updates and reverts in your Technology Catalog can have unexpected consequences for existing jobs, pipelines, and apps.

If a job, pipeline, or app uses a technology or technology context that is changed or deleted because of an update or a revert, it will not be able to run. Depending on the situation, you’ll need to modify or possibly delete and recreate the job, pipeline, or app before continuing.

You might find yourself in a situation where you need to revert to a previous update of your repository.

You can only revert to the update that immediately precedes the current version of your repository. If you need to revert to an earlier update, revert several times until you reach the correct version.

Reverting to a previous update deletes the current update, and that update is no longer available. This action cannot be undone. If you realize later that you need the deleted update, perform a new update with those contents.

Screenshot of the confirmation box to confirm a revert