Operational logs

Saagie generates logs for all components so that IT teams can monitor component behavior. Logs are generated from a component’s standard output. The IT team in charge of maintaining your Kubernetes cluster can collect the logs.

Saagie JVM components use the following tools to format logs:

  • LOGBack logging framework, with an XML file to configure common patterns.

  • We also use specific tools from the Elasticsearch suite in our examples. You can use Elasticsearch tools or other similar tools.

JVM components use an additional configuration to modify the log levels via an XML file.

The project-k8s-controller pod uses both Golang and shell script. There are no tools to format logs, but generated logs follow the same format as other components, except for {THREAD}.
Logs are in Coordinated Universal Time (UTC) exclusively.

1. GDPR compliance

Saagie complies with the General Data Protection Regulation (GDPR). Operational logs collect specific personal data from users in order to ensure the security and traceability of the product.

The data collected can be viewed by the logs of each microservice.

Saagie operational logs collect usernames.

2. Log levels

Log levels are divided by level of criticality:

Table 1. Log levels
Level Description

DEBUG

A general debugging event. Provides precision to a log, such as object details for Saagie support.

INFO
default setting

Monitors main service input, output, and instructions so that you can follow component execution.

WARN

An event that could result in an error. Intervention might be needed.

ERROR

An error in the component.

3. Log patterns

Saagie operational logs follow this pattern:

[OPERATIONAL-{LOG_VERSION}] {DATEFORMAT_PATTERN_UTC} {LOG_LEVEL} [{THREAD}] - {COMPONENT_NAME}[{PACKAGE.CLASS}:{LINE_NUMBER}] {LOG_MESSAGE} -[{LOG_METADATA}]- {EXCEPTION}

The log pattern features:

  • {LOG_VERSION}: current log version

  • {DATEFORMAT_PATTERN_UTC}: all log times are in UTC

  • {LOG_LEVEL}: INFO, DEBUG, WARN, ERROR

  • {THREAD}: thread name

  • {COMPONENT_NAME}: component for which you are retrieving logs

  • {PACKAGE.CLASS}:{LINE_NUMBER}: package and class, followed by line number

  • {LOG_MESSAGE}: describes ongoing activity

  • {LOG_METADATA}: describes metadata, such as realm, id, and action

  • {EXCEPTION}: stacktrace of an error; only present in the case of an error

Regardless of the log level, sensitive information (such as passwords) is not logged for security reasons.

4. Retrieve logs

To retrieve the operational logs generated by a component manually, run the following kubectl command:

# See the logs generated by component's pods.

kubectl -n <namespace> logs <pod_name> | grep "OPERATIONAL" (1) (2)
1 Replace <namespace> with the name of your namespace.
2 Replace <pod_name> with the name of the pod for which you want to retrieve logs.
Components can run on several pods simultaneously. Review each pod’s logs for complete information about your component.

5. Working with logs

There are several external tools that facilitate automatic log retrieval and make it easier for you to use the logs. Let’s take a look at some log examples and how to work with them.

The following examples feature tools from the Elasticsearch suite:

  • Filebeat: retrieve logs

  • Logstash: parse logs

  • Elasticsearch: optimized storage

Additionally, while we do not include an example, you might want to use Kibana or a similar tool to visualize and use your logs.

You are not limited to Elasticsearch tools. Feel free to substitute them for your favorites.

5.1. Parse logs

You can retrieve parsed logs using Logstash.

This pattern can be used for third-party applications to retrieve relevant information. Below is an example using Logstash.

\[%{WORD:log_type}-%{WORD:log_type_version}\] %{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log_level} ?( )\[%{DATA:thread}\] - %{NOTSPACE:component_name}\[%{DATA:class}\] (?m)%{GREEDYDATA:message} -\[%{DATA:logs_metadata}\]-( (?m)%{GREEDYDATA:exception})?

We use a logstash plugin called kv to generate key/value pairs from the payload named logs_metadata. Follow this pattern:

  kv{
    source => "logs_metadata"
    value_split => "="
    trim_value => ","
  }

5.2. Multiline logs

Some logs can be multiline, such as stacktraces of exceptions.

The following example shows how to retrieve logs using a Filebeat configuration, including retrieving multiline logs:

  filebeat.yml: |-
    filebeat.inputs:
    - type: container
      paths:
        - "/var/log/containers/*_saagie-common_*.log" (1)
      include_lines: ['^\[AUDIT-V[0-9]+]', '^\[OPERATIONAL-V[0-9]+']
      multiline.pattern: '^[[:space:]]+(at|\.{3})[[:space:]]+\b|^Caused by:'
      multiline.negate: false
      multiline.match: after

    output.logstash:
      hosts: ["logstash:8080"]
      ssl.enabled: false
1 saagie-common is the namespace in this example. Modify it with your namespace.

Now these logs can be parsed using a tool such as Logstash.

These are only examples and will need to be adapted to your needs.

6. Modify log configuration

6.1. Modify log level

The XML files are mounted in the Kubernetes ConfigMap named <component_name>-config.

  1. Find your ConfigMap.

    # List all ConfigMaps with settings to modify log levels.

kubectl -n <namespace> get configmap (1)
    1 Replace <namespace> with the name of your namespace.

  2. Find your ConfigMap on the list produced.

    • ConfigMaps are listed in this format: <component_name>-config.

  3. Open your ConfigMap file.

    # Open the ConfigMap so that you can modify it.

kubectl -n <namespace> edit configmap <component_name>-config (1) (2)
    1 Replace <namespace> with the name of your namespace.
    2 Replace <component_name> with the name of your component.

  4. Modify your ConfigMap’s log level.

    1. Your XML file will look something like this:

        <logger name="io.saagie" level="info" additivity="false"> (1)
    <appender-ref ref="OPERATIONAL"/>
  </logger>
      1 Notice the level value info.

  5. Change level from info to a different level, such as debug.

  6. Save your changes.

  7. Restart the pod.

Here is an example with the namespace saagie-common and the component projects-and-jobs:

kubectl -n saagie-common edit configmap saagie-common-projects-and-jobs-config

6.2. Configure logs according to package or class

Logs are highly customizable. Here are two examples of customized configurations.

Refer to LOGBack’s documentation to learn more about possible configurations.

  1. Retrieve logs only for specific packages and classes.

      <logger name="io.saagie.projectsandjobs.infra.adapter.primary.graphql.resolver" level="debug" additivity="false"> (1)
    <appender-ref ref="OPERATIONAL"/>
  </logger> (2)
    1 You’ll only retrieve debug-level logs for the package io.saagie.projectsandjobs.infra.adapter.primary.graphql.resolver.
    2 You will not retrieve logs for any other package.

  2. Set different log level defaults for logs for different packages and classes.

      <logger name="io.saagie" level="info" additivity="false"> (1)
    <appender-ref ref="OPERATIONAL"/>
  </logger>

  <logger name="io.saagie.projectsandjobs.infra.adapter.primary.graphql.resolver" level="debug" additivity="false"> (2)
    <appender-ref ref="OPERATIONAL"/>
  </logger>
    1 You’ll retrieve all info-level logs for the entire platform,
    2 as well as debug-level logs for the specific package.
You’ll need to restart the pod to apply this modification.