Systemd, is a tool to manage processes and systems in linux. It offers a very powerful logging solution to log service related information through its journal system. The journalctl service provides a way to access and manipulate the data stored in the systemd journal.

The main goal of this systemd’s journal is to have a centralized log management, merge logs from multiple sources to a single and easy to access source. The Journald daemon is responsible for collecting and managing log messages from a variety of components such as the kernel, initrd and other services.

The main advantage of using systemd journal includes dynamic log displays, arbitrary output formats and the ability to replace or complement existing syslog implementation. This centralized approach gives an efficient log analysis, which enables the users to have log data presented according to one’s needs.

Setting the System Time

Before getting into journalctl, it is important to make sure the system time settings are correct. Systemd provides the timedatectl tool for managing time-related configs. Use it to list all available time zones and set the appropriate time zone for your servers.

timedatectl list-timezones
sudo timedatectl set-timezone <zone>
timedatectl status

Basic Log Viewing

To view logs stored by the journald daemon, utilize the journalctl command. When executed without options, it displays all journal entries in a pager. You can navigate through the logs, which include information from the early boot process, kernel, initrd, and application standard error and output.

journalctl

To display timestamps in UTC instead of local time, use the –utc flag:

journalctl --utc

Journal Filtering by Time

Displaying Logs from the Current Boot

A common scenario is viewing logs from the current boot. The -b flag achieves this:

journalctl -b

For previous boots, utilize the –list-boots option to identify available boots and display logs from a specific boot using the -b flag along with the boot ID.

journalctl --list-boots
journalctl -b <boot-ID>

Time Windows

To filter logs within specific time ranges, use the –since and –until options. Time values can be absolute or relative, allowing flexibility in defining time windows.

journalctl --since "YYYY-MM-DD HH:MM:SS"
journalctl --since yesterday
journalctl --since 09:00 --until "1 hour ago"

Filtering by Message Interest

By Unit

Filtering by unit (service) is useful for focusing on specific components. Use the -u option to retrieve logs related to a particular unit.

journalctl -u nginx.service
journalctl -u nginx.service --since today
journalctl -u nginx.service -u php-fpm.service --since today

By Process, User, or Group ID

Filter logs by process ID (_PID), user ID (_UID), or group ID (_GID). This can be valuable for isolating logs associated with specific processes or users.

journalctl _PID=<process-ID>
journalctl _UID=<user-ID> --since today
journalctl -F _GID  # Display available group IDs
journalctl _GID=<group-ID>

By Component Path

Filter logs by providing an executable path. This is helpful when focusing on entries related to a specific executable.

journalctl /usr/bin/bash

Displaying Kernel Messages

Retrieve kernel messages using the -k or –dmesg flags:

journalctl -k
journalctl -k -b -5  # Display messages from five boots ago

By Priority

Filter logs by priority using the -p option, showing entries at or above the specified level.

journalctl -p err -b

Modifying the Journal Display

Truncate or Expand Output

Adjust how journalctl displays data by using the –no-full option to truncate output or the -a flag to display all information.

journalctl --no-full
journalctl -a

Output to Standard Out

By default, journalctl uses a pager. Use the –no-pager option to output directly to standard output, facilitating further processing.

journalctl --no-pager

Output Formats

Change the output format using the -o option with format specifiers such as json, json-pretty, short, etc.

journalctl -b -u nginx -o json
journalctl -b -u nginx -o json-pretty

Active Process Monitoring

Journalctl can serve as a real-time log monitoring tool.

Displaying Recent Logs

Use the -n option to display a specific number of recent log entries.

journalctl -n
journalctl -n 20

Following Logs

Actively follow logs as they are written using the -f flag, similar to tail -f.

journalctl -f

Journal Maintenance

Finding Current Disk Usage

Check the current disk usage of the journal with the –disk-usage flag:

journalctl --disk-usage

Deleting Old Logs

Shrink the journal by specifying a size with –vacuum-size or a cutoff time with –vacuum-time.

sudo journalctl --vacuum-size=1G
sudo journalctl --vacuum-time=1years

Limiting Journal Expansion

Configure journal growth limits in the journald.conf file, using options like SystemMaxUse, SystemKeepFree, etc.

Conclusion

The Journalctl command, coupled with systemd’s journal, offers a more robut solution for log management and analysis. The flexibility it offers, coupled with various filtering and formatting options, empowers the admins to efficiently navigate and extract valuable insights from system and app logs. Understanding this abilities of Journalctl enhances ability to troubleshoot, monitor, and maintain a systemd-based system effectively.