Operational logs
Saagie generates logs for all components so that IT teams can monitor component behavior. Logs are generated from a component’s standard output. The IT team in charge of maintaining your Kubernetes cluster can collect the logs.
Saagie JVM components use the following tools to format logs:
-
LOGBack logging framework, with an XML file to configure common patterns.
-
We also use specific tools from the Elasticsearch suite in our examples. You can use Elasticsearch tools or other similar tools.
JVM components use an additional configuration to modify the log levels via an XML
file.
The |
Logs are in Coordinated Universal Time (UTC) exclusively. |
1. GDPR compliance
Saagie complies with the General Data Protection Regulation (GDPR). Operational logs collect specific personal data from users in order to ensure the security and traceability of the product.
The data collected can be viewed by the logs of each microservice.
Saagie operational logs collect usernames.
2. Log levels
Log levels are divided by level of criticality:
Level | Description |
---|---|
|
A general debugging event. Provides precision to a log, such as object details for Saagie support. |
|
Monitors main service input, output, and instructions so that you can follow component execution. |
|
An event that could result in an error. Intervention might be needed. |
|
An error in the component. |
3. Log patterns
Saagie operational logs follow this pattern:
[OPERATIONAL-{LOG_VERSION}] {DATEFORMAT_PATTERN_UTC} {LOG_LEVEL} [{THREAD}] - {COMPONENT_NAME}[{PACKAGE.CLASS}:{LINE_NUMBER}] {LOG_MESSAGE} -[{LOG_METADATA}]- {EXCEPTION}
The log pattern features:
-
{LOG_VERSION}
: current log version -
{DATEFORMAT_PATTERN_UTC}
: all log times are in UTC -
{LOG_LEVEL}
: INFO, DEBUG, WARN, ERROR -
{THREAD}
: thread name -
{COMPONENT_NAME}
: component for which you are retrieving logs -
{PACKAGE.CLASS}:{LINE_NUMBER}
: package and class, followed by line number -
{LOG_MESSAGE}
: describes ongoing activity -
{LOG_METADATA}
: describes metadata, such as realm, id, and action -
{EXCEPTION}
: stacktrace of an error; only present in the case of an error
Regardless of the log level, sensitive information (such as passwords) is not logged for security reasons. |
4. Retrieve logs
To retrieve the operational logs generated by a component manually, run the following kubectl command:
# See the logs generated by component's pods.
kubectl -n <namespace> logs <pod_name> | grep "OPERATIONAL" (1) (2)
1 | Replace <namespace> with the name of your namespace. |
2 | Replace <pod_name> with the name of the pod for which you want to retrieve logs. |
Components can run on several pods simultaneously. Review each pod’s logs for complete information about your component. |
5. Working with logs
There are several external tools that facilitate automatic log retrieval and make it easier for you to use the logs. Let’s take a look at some log examples and how to work with them.
The following examples feature tools from the Elasticsearch suite:
-
Filebeat
: retrieve logs -
Logstash
: parse logs -
Elasticsearch
: optimized storage
Additionally, while we do not include an example, you might want to use Kibana
or a similar tool to visualize and use your logs.
You are not limited to Elasticsearch tools. Feel free to substitute them for your favorites. |
5.1. Parse logs
You can retrieve parsed logs using Logstash.
This pattern can be used for third-party applications to retrieve relevant information. Below is an example using Logstash.
\[%{WORD:log_type}-%{WORD:log_type_version}\] %{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log_level} ?( )\[%{DATA:thread}\] - %{NOTSPACE:component_name}\[%{DATA:class}\] (?m)%{GREEDYDATA:message} -\[%{DATA:logs_metadata}\]-( (?m)%{GREEDYDATA:exception})?
We use a logstash plugin called kv
to generate key/value pairs from the payload named logs_metadata
.
Follow this pattern:
kv{
source => "logs_metadata"
value_split => "="
trim_value => ","
}
5.2. Multiline logs
Some logs can be multiline, such as stacktraces of exceptions.
The following example shows how to retrieve logs using a Filebeat configuration, including retrieving multiline logs:
filebeat.yml: |-
filebeat.inputs:
- type: container
paths:
- "/var/log/containers/*_saagie-common_*.log" (1)
include_lines: ['^\[AUDIT-V[0-9]+]', '^\[OPERATIONAL-V[0-9]+']
multiline.pattern: '^[[:space:]]+(at|\.{3})[[:space:]]+\b|^Caused by:'
multiline.negate: false
multiline.match: after
output.logstash:
hosts: ["logstash:8080"]
ssl.enabled: false
1 | saagie-common is the namespace in this example.
Modify it with your namespace. |
Now these logs can be parsed using a tool such as Logstash.
These are only examples and will need to be adapted to your needs. |
6. Modify log configuration
6.1. Modify log level
The XML files are mounted in the Kubernetes ConfigMap
named <component_name>-config
.
-
Find your ConfigMap.
# List all ConfigMaps with settings to modify log levels. kubectl -n <namespace> get configmap (1)
1 Replace <namespace> with the name of your namespace. -
Find your ConfigMap on the list produced.
-
ConfigMaps are listed in this format:
<component_name>-config
.
-
-
Open your ConfigMap file.
# Open the ConfigMap so that you can modify it. kubectl -n <namespace> edit configmap <component_name>-config (1) (2)
1 Replace <namespace> with the name of your namespace. 2 Replace <component_name> with the name of your component. -
Modify your ConfigMap’s log level.
-
Your XML file will look something like this:
<logger name="io.saagie" level="info" additivity="false"> (1) <appender-ref ref="OPERATIONAL"/> </logger>
1 Notice the level value info
.
-
-
Change level from
info
to a different level, such asdebug
. -
Save your changes.
-
Restart the pod.
Here is an example with the namespace saagie-common
and the component projects-and-jobs
:
kubectl -n saagie-common edit configmap saagie-common-projects-and-jobs-config
6.2. Configure logs according to package or class
Logs are highly customizable. Here are two examples of customized configurations.
Refer to LOGBack’s documentation to learn more about possible configurations. |
-
Retrieve logs only for specific packages and classes.
<logger name="io.saagie.projectsandjobs.infra.adapter.primary.graphql.resolver" level="debug" additivity="false"> (1) <appender-ref ref="OPERATIONAL"/> </logger> (2)
1 You’ll only retrieve debug-level logs for the package io.saagie.projectsandjobs.infra.adapter.primary.graphql.resolver
.2 You will not retrieve logs for any other package. -
Set different log level defaults for logs for different packages and classes.
<logger name="io.saagie" level="info" additivity="false"> (1) <appender-ref ref="OPERATIONAL"/> </logger> <logger name="io.saagie.projectsandjobs.infra.adapter.primary.graphql.resolver" level="debug" additivity="false"> (2) <appender-ref ref="OPERATIONAL"/> </logger>
1 You’ll retrieve all info-level logs for the entire platform, 2 as well as debug-level logs for the specific package.
You’ll need to restart the pod to apply this modification. |