Saagie Usage Monitoring Default Dashboards

Saagie Usage Monitoring (SUM) comes with defaults dashboards.
A dashboard is a set of one or more panels organized and arranged into one or more rows. Dashboards and panels allow you to show your data in visual form.

When installing SUM, you must set several environment variables, including the MONITORING_OPT environment variable. This variable allows you to choose the default configuration of your dashboards by choosing between the following values:

  • SAAGIE: Choose this value if you want to monitor only Saagie jobs, apps, and pipelines. This is the default value.

  • SAAGIE_AND_DATALAKE: Choose this value if you want to monitor Saagie jobs, apps, and pipelines, as well as your HDFS data lake.

  • SAAGIE_AND_S3: Choose this value if you want to monitor Saagie jobs, apps, and pipelines, as well as your S3 buckets.

If you choose the SAAGIE option, you will have the following three dashboards:

If you choose the SAAGIE_AND_DATALAKE option, you will have the dashboards included with the SAAGIE option, and additional dashboards with information about your HDFS storage space.

If you choose the SAAGIE_AND_S3 option, you will have the dashboards included with the SAAGIE option, and an additional dashboard with information about your S3 storage space.

Dashboards are accessible from the Dashboards tab:

saagie usage monitoring dashboards

Saagie – Job Count

This page gives you information about all the jobs and apps of your platforms.

saagie usage monitoring job count

You can see information on the total number of jobs and apps on your Saagie platforms, their distribution and evolution in number per project, and details for each app and job.

Click a legend item of a graph to display only the corresponding information. You can also view multiple items at once by pressing ctrl and clicking the desired legend items. To reset the graph, click a legend item twice.

Saagie – Jobs Failed Timeline

This page gives you information about failed jobs.

saagie usage monitoring jobs failed timeline

Click a dot in the graph to display a tooltip with the job instance details. You can see the job name, the date the job instance failed, its status, and its execution time. You can click open external Job to open the instance of the job in Saagie.

Red dots are for failed jobs and yellow dots are for killed jobs.

Saagie – Jobs and Pipelines

This page gives you information about the execution of your jobs and pipelines.

saagie usage monitoring jobs and pipelines

You can see information on the evolution of job status, the jobs and pipelines execution time, the average execution time for each job and its success rate.

  • Hover over the items of the different graphs to display tooltips with more details.

  • Click a legend item of a graph to display only the corresponding information. You can also view multiple items at once by pressing ctrl and clicking the desired legend items. To reset the graph, click a legend item twice.

S3 – Global Usage

This page gives you information about the storage space you have on your S3.

saagie usage monitoring s3 global usage

You can see information about the total space used by all your S3 buckets, their total number of objects and the size of each bucket. The size of your S3 buckets is presented through a table and a graph.

Data Lake - Disk Space Explorer

This page gives you information about the storage space you have on your HDFS, especially on the first level directories.

saagie usage monitoring data lake disk space explorer

You can see information about the size of first level directories in the Folder size dashboard, and about the number of files per first level directories in the Number of files dashboard.

Hover over the items of the different graphs to display tooltips with more details.

Data Lake - Global Usage

This page gives you information about the overall storage space you have on your HDFS.

saagie usage monitoring data lake global usage

This page allows you to see the history of the size of the directories, and the number of files. You can also see information about the average file size, and the storage space used compared to the overall capacity.

See also