Managing Jobs

Use these tutorials to create, run, modify, upgrade, duplicate, and move your jobs.

Creating Jobs

You can create jobs to perform tasks and launch them individually or as part of a data pipeline.

Creating a job can also be done via our CI/CD process. For more information, see Saagie CI/CD.
Before you begin:

Before creating the job, you must create your code file with the actions you want your job to perform. In this tutorial, we will use Python technology and create a file called hello-scranton.py.

  1. Open a new file in your preferred text editor.

  2. Copy and paste the following code into your file:

    print("Hello, Scranton Branch!")
  3. Save the file as hello-scranton.py.

  1. Click The "Projects" module icon is a folder. Projects from the primary navigation menu.
    By default, the The "All Projects" page icon is a folder and the same as the "Projects" module. All Projects page opens with the list of existing projects.

  2. Click a project in the list to open it.
    By default, the The "Jobs" page icon is a 3D pyramid of three squares. Jobs page opens with the list of existing jobs.

    Navigate to your project
  3. Click New job from the The "Jobs" page icon is a 3D pyramid of three squares. Jobs page to create a new job.
    The New job page opens.

  4. Fill in the creation form with the required information and click Continue between each step.

    1. Enter a name, define an alias, and add a description.

    2. Select your job technology.

    3. Depending on whether you are creating an embedded or external job:

      • Embedded Job

      • External Job

      Choose the technology context, upload your code file, and enter your shell command.

      Choose the technology context, select the external connection from the Connection list or create it on the fly if you do not already have one, and fill the following fields.

    4. Configure settings to schedule your job run, send email alerts, and set your job resources. You can also add a release note.

  5. Click Create job to confirm the creation.
    The The "Overview" page icon is a square divided into several other squares. Overview page of your job opens with a message saying that your job has been created.

To delete a job, you can either:

  • Click the kebab menu The kebab menu icon is three vertical dots.  delete Delete from the The "Overview" page icon is a square divided into several other squares. Overview page of the corresponding job.

  • Click delete Delete job in the secondary navigation menu from any other page of your job.

A confirmation message appears. Click Delete again to confirm the deletion. Be careful, there is no progress bar to cancel the deletion once it is confirmed.

You cannot delete a job that is part of a pipeline.

Generating a Job Description With ChatGPT

You can use ChatGPT to generate your job description. This feature is only available for Spark with a Python context, Bash, R, Sqoop, and Python job technologies.

This option must have been enabled by your administrator during the configuration of the Saagie installer, saagiectl. They must have answered the prompt to use OpenAI when configuring your cluster settings.
  1. Click The "Projects" module icon is a folder. Projects from the primary navigation menu.
    By default, the The "All Projects" page icon is a folder and the same as the "Projects" module. All Projects page opens with the list of existing projects.

  2. Click a project in the list to open it.
    By default, the The "Jobs" page icon is a 3D pyramid of three squares. Jobs page opens with the list of existing jobs.

    Navigate to your project
  3. Click a job in the list to open it and access its details.
    The job The "Overview" page icon is a square divided into several other squares. Overview page opens.

  4. Click Generate with ChatGPT above the description field from the The "Overview" page icon is a square divided into several other squares. Overview page to send your request to ChatGPT.

    It only works with .py, .sh, .r, and .bash files and command lines, and it does not work with archive files.

    A pop-up message appears to inform you that the code will be sent to OpenAI and thus will not be secured anymore by Saagie.

  5. Click Generate description to confirm and generate your job description with ChatGPT.
    The description is displayed in the Description field. A check mark appears in front of the text Generated with ChatGPT job chatgpt generated description to indicate that the description is from ChatGPT.

    Once a description has been generated, you cannot regenerate it. Unless you modify it manually or upgrade your job. When you manually modify the description, it is no longer consider as generated by ChatGPT. The checkmark disappears, and you can regenerate the job description with ChatGPT. When you upgrade your job, you can update its description by clicking refresh Update.

    job chatgpt description

Running and Stopping Jobs

You can run or stop your jobs manually, even when the jobs are scheduled to run.

  1. Click The "Projects" module icon is a folder. Projects from the primary navigation menu.
    By default, the The "All Projects" page icon is a folder and the same as the "Projects" module. All Projects page opens with the list of existing projects.

  2. Click a project in the list to open it.
    By default, the The "Jobs" page icon is a 3D pyramid of three squares. Jobs page opens with the list of existing jobs.

    Navigate to your project
  3. You can either:

    • Click Run Run or Stop Stop in front of the desired job from the library.

    • Open the job details and start or stop the run from its The "Overview" page icon is a square divided into several other squares. Overview page. Click either Run Run or Stop Stop depending of the current status of your pipeline.

      You can also access this command at the bottom of the secondary navigation menu from the The "Instances" page icon is three overlapping squares. Instances and The "Versions" page icon is a folder with an arrow pointing up. Versions page.

    The job status changes depending on the outcome.

Modifying Job Settings

You can change your job settings at any time after creating it. You can access the settings from the The "Overview" page icon is a square divided into several other squares. Overview page of the job.

  1. Click The "Projects" module icon is a folder. Projects from the primary navigation menu.
    By default, the The "All Projects" page icon is a folder and the same as the "Projects" module. All Projects page opens with the list of existing projects.

  2. Click a project in the list to open it.
    By default, the The "Jobs" page icon is a 3D pyramid of three squares. Jobs page opens with the list of existing jobs.

    Navigate to your project
  3. Click a job in the list to open it and access its details.
    The job The "Overview" page icon is a square divided into several other squares. Overview page opens.

  4. Click the desired setting from the The "Overview" page icon is a square divided into several other squares. Overview page to edit it.

    Job settings from the "Overview" page.
    • 1 – Name

    • 2 – Alias

    • 3 – Description

    • 4 – Scheduled Run

    • 5 – Email Alerts

    • 6 – Resources

    Names are mandatory. They can be up to 255 characters long and must be unique within a project.

    The job alias is unique to each job in a project. It allows you to reference a job within another job and can be used to pass information between jobs during pipeline execution. However, to make this work, the settings env vars Variables setting must be enabled.

    Descriptions are optional and not restricted. However, it is a good practice to add them. Keep them short and to the point.

    You can also use ChatGPT to generate your job description by clicking Generate with ChatGPT (a). For more information, see Generating a Job Description With ChatGPT.

    This feature is only available for Spark with a Python context, Bash, R, and Python job technologies. It works with .py, .sh, .r, and .bash files and command lines, but it does not work with archive files.
    Click the switch in the side panel of the setting to enable or disable it.

    There are two types of execution:

    1. The manual run, which requires you to click Run to start the job.

    2. The scheduled run, which launches the job according to the schedule you choose.

      Scheduled jobs can also be started manually.

      The Scheduled run type has three schedule modes: Simple, Shortcut, and Expert.

      • In Simple mode, you can easily specify variables through the user interface. There are many possibilities.

        Screenshot of the settings for the scheduled run type in simple mode

      • In Shortcut mode, you can choose the recurrence of your run on an hourly, daily, weekly, monthly, or annual basis. All other settings are automatic.

        Screenshot of the settings for the scheduled run type in shortcut mode

      • In Expert mode, you can specify variables using the Cron format. The Cron time string consists of five values separated by spaces: [minute] [hour] [day of the month] [month] [day of the week]. They are based on the following information:

        Table 1. Cron format
        Descriptor Acceptable values

        Minute

        0 to 59, or *

        Hour

        0 to 23, or *

        Day of the month

        1 to 31, or *

        Month

        1 to 12, or *

        Day of the week

        0 to 7 (0 and 7 both represent Sunday), or *

        The Cron time string must contain entries for each character attribute. If you want to set a value using only minutes, you must have asterisk (*) characters for the other four attributes that you are not configuring.

        Screenshot of the settings for the scheduled run type in expert mode

        Once you have finished scheduling your run, you will see the summary of your choice written below and the time of the next run.

    Click the switch in the side panel of the setting to enable or disable it.

    Alerts are optional and can be set to receive an email when the status of your job changes. They can be sent to multiple email addresses to notify you of the following status changes:

    Status Description

    queued requested Requested

    The job’s run has been requested and is being executed.

    queued requested Queued

    The job is waiting for the necessary resources to be executed.

    spinner Running

    The job is up and running.

    fail Failed

    The job has crashed.

    A failed job can go into an out of memory Out Of Memory (OOM) state, which is an extension of the Failed state. The OOM state can be due to a lack of memory (RAM).

    stop Stopping

    The job is stopping.

    stop Stopped

    The job has stopped running.

    success Succeeded

    The job has been successfully executed.

    unknown Unknown

    The job no longer runs because an error has occurred.

    For embedded jobs only.

    Click the switch in the side panel of the setting to enable or disable it.

    CPU and RAM resources are optional, but are recommended. They can be specified for optimal execution.

    This works the same way for CPU and RAM resources. The only difference is that you can choose between GB and MB units for RAM.

    The consumption of your job can be managed by guaranteed resources, that is, the minimum amount of resource requested, and limited resources, that is, the maximum amount of resource that can be consumed.

    Default values are already defined by Saagie at the technology level, except for external technologies. These values are mandatory. They also exist at the technology context level, and can override the values defined at the technology level. You can configure them when creating a job or app, or by modifying the Resources Icon for CPU and RAM resources. setting of your job or app.

    When you create a job, CPU and RAM resource management is enabled by default with predefined values. In other words, based on the defaults defined at technology level, Saagie automatically assigns resource requests and limits to your job. These values can be adjusted to suit your needs.

    If the resource capacity is not defined, the defaults that are defined at the technology level will be assigned at the technology context level.

    Automatic adjustments can be made to avoid inconsistent configurations. If you try to set a guaranteed value that is greater than the limit value, a note appears to inform you that the guaranteed value has been adjusted. If you try to set a limit value smaller than the guaranteed value, a note appears to inform you that the limit value has been adjusted (a).

    RAM limit adjustment message.

    In addition, if the guaranteed value and the limit value are not optimal, a message is displayed with the appropriate values for an optimal configuration (b).

    CPU recommendation message.

    When you change CPU and RAM resources, your job is automatically restarted.
  5. Saving is automatic. You can just press Enter to validate the job name change, click anywhere nearby to confirm the job alias and description change, and close the side panel to validate the scheduled run, email alert, and resource changes.

Upgrading Jobs

You can upgrade your jobs to always get the most out of them. By upgrading your job, you create a new version of it.

Upgrading a job can also be done via our CI/CD process. For more information, see Saagie CI/CD.
Before you begin:

You must update your code file or create a new one.

  1. Open your file in your preferred text editor.

  2. Copy and paste the following code into your file:

    print("Hi there, Scranton Branch!")
  3. Save the file as hi-scranton.py.

  1. Click The "Projects" module icon is a folder. Projects from the primary navigation menu.
    By default, the The "All Projects" page icon is a folder and the same as the "Projects" module. All Projects page opens with the list of existing projects.

  2. Click a project in the list to open it.
    By default, the The "Jobs" page icon is a 3D pyramid of three squares. Jobs page opens with the list of existing jobs.

    Navigate to your project
  3. Click a job in the list to open it and access its details.
    The job The "Overview" page icon is a square divided into several other squares. Overview page opens displaying information on the last update, such as when it took place, by whom, the technology and execution context used, the package, and the job logs.

  4. Click Upgrade job Upgrade job button from the The "Instances" page icon is three overlapping squares. Instances or The "Versions" page icon is a folder with an arrow pointing up. Versions page.

    You can also click edit Edit from the The "Overview" page icon is a square divided into several other squares. Overview page of the job.

    The Upgrade job page opens.

  5. Enter the information you want to change and click Continue between each step. You can:

    1. Depending on whether you are upgrading an embedded or external job:

      • Embedded Job

      • External Job

      Choose the technology context, upload your code file, and enter your shell command.

      Choose the technology context, select the external connection from the Connection list or create it on the fly if you do not already have one, and fill the following fields.

    2. Add a release note to briefly explain your changes.

  6. Click Save job to save your changes and exit the job upgrade settings.
    Your job has been upgraded. You should see that a new version of it has been created, along with a new instance.

Duplicating Jobs

You can duplicate only the Current Badge for the "Current" status version of your job. The duplicated job will include a copy of the technology, package, and resource limits. The copy excludes alerts, scheduling, links to associated pipelines, observability data, versions, instances, and logs.

  1. Click The "Projects" module icon is a folder. Projects from the primary navigation menu.
    By default, the The "All Projects" page icon is a folder and the same as the "Projects" module. All Projects page opens with the list of existing projects.

  2. Click a project in the list to open it.
    By default, the The "Jobs" page icon is a 3D pyramid of three squares. Jobs page opens with the list of existing jobs.

    Navigate to your project
  3. You can click the kebab menu The kebab menu icon is three vertical dots.  clone copy Duplicate at the end of the relevant job row from the The "Jobs" page icon is a 3D pyramid of three squares. Jobs page, or from the The "Overview" page icon is a square divided into several other squares. Overview page of a job.

    • You cannot duplicate an invalid job.

    • You cannot select multiple job versions at the same time and duplicate them.

    Depending on whether you duplicated from the The "Jobs" page icon is a 3D pyramid of three squares. Jobs page or from the job The "Overview" page icon is a square divided into several other squares. Overview page, you are returned to either the job library, or the The "Overview" page icon is a square divided into several other squares. Overview page of the duplicated job, respectively. In addition, a message appears saying that your job has been successfully duplicated.

    The name and alias of your duplicated job will be <original job name> (ID starting by X) and <original job alias> (ID starting by X). Where X is an incremental number by 1.

    For example, you duplicate the job Albany Branch Sales. The name of the duplicated job will be Albany Branch Sales (1). If you duplicate the job Albany Branch Sales a second time, the name of this second duplicated job will be Albany Branch Sales (2), and so on. If the duplicated job name and its alias exceed the 255-character limit, we truncate the last 13 characters. The incremental number is kept.

Deleting Job Instances and Job Versions

The instance and version history of your job is too large, and it is not easy to find your little ones. In this case, you may want to delete some instances or versions to streamline these lists and improve your user experience.

  1. Click The "Projects" module icon is a folder. Projects from the primary navigation menu.
    By default, the The "All Projects" page icon is a folder and the same as the "Projects" module. All Projects page opens with the list of existing projects.

  2. Click a project in the list to open it.
    By default, the The "Jobs" page icon is a 3D pyramid of three squares. Jobs page opens with the list of existing jobs.

    Navigate to your project
  3. Click a job in the list to open it and access its details.
    The job The "Overview" page icon is a square divided into several other squares. Overview page opens.

  4. Click either:

    • The "Instances" page icon is three overlapping squares. Instances to access your job’s instances page and delete instances with the logs from the list.

      • You cannot delete instances that are part of existing pipeline instances.

      • You cannot delete unfinished instances, that is, instances whose status is Requested, Queued, Running, Stopping. Wait for the end of the executions and try again.

      • You cannot delete an instance that has not yet been cleaned from the orchestrator. Wait a moment and try again.

    • The "Versions" page icon is a folder with an arrow pointing up. Versions to access your job’s versions page and delete versions from the list.

      • Deleting a job version deletes the job instances linked to it, as well as all its dependencies.

      • You cannot delete a version in which a job instance is linked to a pipeline instance.

      • You cannot delete the version tagged as Current Badge for the "Current" status. You can revert to another version of the job to change the current one and delete it.

      • You cannot delete a version that is linked to unfinished instances. Wait for the end of the executions and try again.

      • You cannot delete a version whose instances have not yet been cleaned from the orchestrator. Wait a moment and try again.

  5. You can delete either instances or versions of your job.

    • Delete a single instance or version

    • Delete a selection of instances or versions

    • Delete a selection of instances or versions based on filters

    • Delete instances by date

    You can delete a single job instance or version by clicking the trash can delete that appears on the line when you hover over the desired element.

    job deletion instance single

    You can delete a selection of job instances or versions by selecting them in the list and clicking Delete n instances/versions at the top of the list.

    You will then be able to play with the selector by adding or removing elements with different states.

    job deletion instance selection job deletion version selection

    You can delete a selection of job instances based on status criteria, or a selection of job versions based on tag criteria. To do this, select the desired filter and click Delete n instances/versions at the top of the list.

    You will then be able to play with the selector by adding or removing elements with different states.

    job deletion instance filter job deletion version filter

    You can delete job instances by date criteria with the All instances older than calendar plus filter. It will delete all instances and associated logs prior to the selected date.

    1. Select the All instances older than calendar plus filter.

      job deletion instance date

      A pop-up window opens with a date picker.

    2. Select a date.

    3. Click Select instances to confirm your choice.
      The corresponding instances are selected in the list.

      You will then be able to play with the selector by adding or removing elements.
    4. Click Delete n instances at the top of the list.

  6. Confirm the pop-up message and the permanent deletion by clicking Delete.

Moving a Job to Another Project

You can move a job from one project to another and on a different platform from the job library or the job The "Overview" page icon is a square divided into several other squares. Overview page. Packages, versions, instances, logs, alerts, and resource settings are all included in the moved job.

  • This is not a duplication. When moving a job to another project, the job disappears from the source project.

  • Any project you have access to can move a job in and out.

  • This feature does not support external jobs.

  • You cannot select multiple jobs at the same time and move them.

  • If the job name and alias already exist in the target project, they are renamed with an increment in brackets.

  1. Click The "Projects" module icon is a folder. Projects from the primary navigation menu.
    By default, the The "All Projects" page icon is a folder and the same as the "Projects" module. All Projects page opens with the list of existing projects.

  2. Click a project in the list to open it.
    By default, the The "Jobs" page icon is a 3D pyramid of three squares. Jobs page opens with the list of existing jobs.

    Navigate to your project
  3. You can click the kebab menu The kebab menu icon is three vertical dots.  project move Move to… at the end of the relevant job row from the The "Jobs" page icon is a 3D pyramid of three squares. Jobs page, or from the The "Overview" page icon is a square divided into several other squares. Overview page of a job.

    • The job technology must be available in the target project. If it does not, select another project or modify the settings of the target project.

    • You cannot move a job that is or was part of a pipeline. You can duplicate your job and move the duplicated job.

    • You cannot move a running and scheduled job. Wait for the end of the execution(s) and/or try to remove the scheduling settings. Then try again.

    • You cannot move a job whose instances have not yet been cleaned from the orchestrator. Wait a moment and try again.

    A pop-up window opens.

  4. Select the target platform and target project from the list.
    By default, it will search for the available categories:

    • If one of them is available, it assigns it. It can be changed if others are available.

    • If no category is available, you will be asked to change the target project or edit the selected project settings. You can click edit the technologies’ settings for this project to access and change the project settings directly.

      job move to

  5. Click Move to confirm the move.
    You will see a message saying that your job has been successfully moved. You can click Go to target project below the success message to be redirected to the job library of the target project. Packages, versions, instances, logs, alerts, and resource settings are all included in the moved job. There is no obfuscation of the data. If there is sensitive data, it will be visible in the target project.