Prometheus

Saagie uses the monitoring system Prometheus to expose metrics through Saagie’s HTTP API. During the Saagie installation process, you can enable or disable this exposure.

1. Exposed components

Each exposed component is accessible by its own HTTP path. The list of available paths in Saagie can be found in the prometheus Kubernetes Ingress of the saagie-common namespace.

To list the paths, run the following command:

kubectl -n saagie-common get ingress <ingress_name> -o jsonpath='{range .spec.rules[*].http.paths[*]}{.path}{"\n"}{end}' (1)
1 Replace <ingress_name>.

To ingress prometheus, you should receive the following result:

/prometheus/projects-and-jobs/jmx/
/prometheus/jobchestrator/jmx/
/prometheus/conforama/jmx/
/prometheus/platform-k8s-controller/jmx/
/prometheus/scredz/jmx/
/prometheus/auth/jmx/
/prometheus/authentication/jmx/
/prometheus/idmacl/jmx/
/prometheus/profile/jmx/
/prometheus/security/jmx/
/prometheus/settings/jmx/
/prometheus/ingress-controller-apps-bouncer/jmx/
/prometheus/technology-manager/jmx/
/prometheus/reporter/jmx/
/prometheus/governance/jmx/
/prometheus/dataset-access/jmx/
/prometheus/rule-manager/jmx/

To ingress prometheus-kafka, you should receive the following result (number of replicas will vary):

/prometheus/kafka-broker/jmx/0/
/prometheus/kafka-broker/jmx/1/
/prometheus/kafka-broker/jmx/2/
/prometheus/kafka-schema-registry/jmx/
/prometheus/kafka-zookeeper/jmx/0/
/prometheus/kafka-zookeeper/jmx/1/
/prometheus/kafka-zookeeper/jmx/2/

Paths follow this model:

/prometheus/<component>/<exporter>/{<replicas>/} (1) (2) (3)
1 Components might have a sub-component as well, appearing as <component-subcomponent> For example, the path /prometheus/ingress-controller-apps-bouncer/jmx/ refers to the component ingress-controller with the sub-component apps-bouncer.
2 exporter is defined according to the component technology.
3 replicas depend on the component and may or may not be present.

2. Prometheus Server configuration

Below, you’ll find an example of a Prometheus Server configuration (prometheus.yml) used to connect to Saagie’s Prometheus exporters:

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  - job_name: 'saagie-projects'
    metrics_path: '/prometheus/projects/jmx/'
    scheme: 'https'
    basic_auth:
      username: 'prometheus'
      password: 'password'
    static_configs:
      - targets: ['SAAGIE_HOST']
  - job_name: 'saagie-jobchestrator'
    metrics_path: '/prometheus/jobchestrator/jmx/'
    scheme: 'https'
    basic_auth:
      username: 'prometheus'
      password: 'password'
    static_configs:
      - targets: ['SAAGIE_HOST']
...

3. How to collect Prometheus metrics

Use an HTTP call to collect Prometheus metrics.

The following command line gives an example with the curl tool, using metrics from the projects component:

curl -u <prometheus_user>:<prometheus_password> https://<saagie-host>/prometheus/<component>/<exporter>/{<replicas>/}' (1)
1 Replace <saagie-host> with your Saagie domain and <prometheus_user> and <prometheus_password> with your Prometheus credentials.

To learn more about Prometheus use, refer to Prometheus documentation.

It is not recommended to make more than one request per endpoint per 30 second period to the Prometheus API.