About Saagie Usage Monitoring

The Saagie Usage Monitoring (SUM) add-on is an app based on Grafana that can be used to monitor job, app, and pipelines metrics for your projects.

It is available in the official Saagie repository and you can install it in any project. However, since the app is designed to display cross-project metrics, Saagie recommends deploying it in a dedicated administration project.

Its installation requires some configuration to work. Your user account must have view rights on at least all projects, and you must set several environment variables. For more information, see the README help file.

Architecture and Operating Mode

SUM is based on the combination of several components, including a Grafana app, a Python job, a Cron, a TTYD or a pgAdmin, and either an internal or external PostgreSQL.

saagie usage monitoring internal postgresql architecture
Figure 1. SUM with an internal PostgreSQL
saagie usage monitoring external postgresql architecture
Figure 2. SUM with an external PostgreSQL and a pgAdmin app

  1. The Python job retrieves data from jobs and pipelines through the Saagie API, and imports it into a PostgreSQL database initiated when SUM starts.

    Data can also be imported from S3 or HDFS storage, which must be configured with environment variables when installing SUM. For more information, see the README help file.

  2. The Cron runs the Python job every hour to synchronize the data.

  3. The PostgreSQL database feeds your alerts and dashboards by transmitting the retrieved data to the Grafana.

  4. TTYD and pgAdmin allow you to write your queries.

    You can also use the pgAdmin app to look into the PostgreSQL and analyze the data model.

External PostgreSQL Database

To use an external PostgreSQL database, you must define the connection parameters via the corresponding environment variables in Saagie:

  • SAAGIE_PG_HOST: PostgreSQL host
    → The default value is localhost.

  • SAAGIE_PG_PORT: PostgreSQL port
    → The default value is 5432.

  • SAAGIE_PG_USER: PostgreSQL user
    → The default value is supervision_pg_user.

  • SAAGIE_PG_PASSWORD: PostgreSQL password
    → The default value is ` `.

  • SAAGIE_PG_DATABASE: PostgreSQL database
    → The default value is supervision_pg_db.