Kernel logging: APIs and implementation
Kernel API
Logging within the kernel is performed using the printk function
int printk( const char * fmt, ... );
The kernel code simply defines the log level as the first argument of the message, as illustrated in the following example for a critical message:
printk( KERN_CRIT "Error code %08x.\n", val );
Note that the first argument is not an argument at all, as no comma (,) separates the level (KERN_CRIT) from the format string.
Table 1. Log levels, symbolics, and uses
| Symbolic | String | Usage |
| KERN_EMERG | <0> | Emergency messages (precede a crash) |
| KERN_ALERT | <1> | Error requiring immediate attention |
| KERN_CRIT | <2> | Critical error (hardware or software) |
| KERN_ERR | <3> | Error conditions (common in drivers) |
| KERN_WARNING | <4> | Warning conditions (could lead to errors) |
| KERN_NOTICE | <5> | Not an error but a significant condition |
| KERN_INFO | <6> | Informational message |
| KERN_DEBUG | <7> | Used only for debug messages |
| KERN_DEFAULT | <8> | Default kernel logging level |
| KERN_CONT | <9> | Continuation of a log line (avoid adding new time stamp) |
Note that if a caller does not provide a log level within printk, a default of KERN_WARNING is automatically used.
Printk source code is as follows:
static char buf[1024];
extern int vsprintf(char * buf, const char * fmt, va_list args);
int printk(const char *fmt, ...)
{
va_list args;
int i;
va_start(args, fmt);
i=vsprintf(buf,fmt,args);
va_end(args);
console_print(buf);
return i;
}
At this point, you've explored the API used to insert log messages into the kernel ring buffer. Now, let's look at the method used to migrate data from the kernel into the host.
Kernel logging and interface
Access to the log buffer is provided at the core through the multi-purpose syslog system call.
The syslog call serves as the input/output (I/O) and control interface to the kernel's log message ring buffer. From the syslog call, an application can read log messages (partial, in their entirety, or only new messages) as well as control the behavior of the ring buffer (clear contents, set the level of messages to be logged, enable or disable console, and so on). Figure 1 provides a graphical illustration of the logging stack with some of the major components discussed.
Figure 1. Kernel logging stack identifying the major components
The syslog call (called do_syslog within the kernel in ./linux/kernel/printk.c) is a relatively small function that provides the ability to read and control the kernel ring buffer. Note that in glibc 2.0, this function is called klogctl because of overuse of the term syslog, which refers to a variety of calls and applications. The prototype function (in user space) for syslog and klogctl is defined as:
int syslog( int type, char *bufp, int len );
int klogctl( int type, char *bufp, int len );
The type argument communicates the command to perform and is associated with an optional buffer with its length. Some commands (such as clearing the ring buffer) ignore the bufp and len arguments.The complete list of SYSLOG commands is shown in Table 2.
Table 2. Commands implemented with the syslog/klogctl system call
| Command/opcode | Purpose |
SYSLOG_ACTION_CLOSE (0) |
Close the log (unimplemented) |
SYSLOG_ACTION_OPEN (1) |
Open the log (unimplemented) |
SYSLOG_ACTION_READ (2) |
Read from the log |
SYSLOG_ACTION_READ_ALL (3) |
Read all messages from the log (non-destructively) |
SYSLOG_ACTION_READ_CLEAR (4) |
Read and clear all messages from the log |
SYSLOG_ACTION_CLEAR (5) |
Clear the ring buffer |
SYSLOG_ACTION_CONSOLE_OFF (6) |
Disable printks to the console |
SYSLOG_ACTION_CONSOLE_ON (7) |
Enable printks to the console |
SYSLOG_ACTION_CONSOLE_LEVEL (8) |
Set level of messages the console accepts |
SYSLOG_ACTION_SIZE_UNREAD (9) |
Return the number of unread characters in the log |
SYSLOG_ACTION_SIZE_BUFFER (10) |
Return the size of the kernel ring buffer |
Implemented above the syslog/klogctl layer, the kmsg proc file system is a I/O path (implemented in ./linux/fs/proc/kmsg.c) that provides a binary interface for reading log messages from the kernel buffer. This is commonly read by a daemon (klogd or rsyslogd) that consumes the messages and passes them to rsyslog for routing to the appropriate log file (based on its configuration).
User space applications
User space provides a number of access points for reading and managing kernel logging. Let's begin with the lower-level interfaces (such as the /proc file system configuration elements), and then expand to the higher-level applications.
The /proc file system exports more than just a binary interface for accessing log messages (kmsg). It also presents a number of configuration elements both related and independent of those discussed through the syslog/klogctl. Listing 1 shows an exploration of these parameters.
Listing 1. Exploring the printk configuration parameters in /proc
mtj@ubuntu:~$ cat /proc/sys/kernel/printk 4 4 1 7 mtj@ubuntu:~$ cat /proc/sys/kernel/printk_delay 0 mtj@ubuntu:~$ cat /proc/sys/kernel/printk_ratelimit 5 mtj@ubuntu:~$ cat /proc/sys/kernel/printk_ratelimit_burst 10
From Listing 1, the first entry defines the log levels currently used in the printk API. These log levels represent the console log level, default message log level, minimum console log level, and default console log level. The printk_delay value represents the number of milliseconds to delay between printk messages (to add readability in some scenarios). Note here that it's set to zero, and it cannot be set through /proc. The printk_ratelimit defines the minimum length of time allowed between messages (currently defined as some number of kernel messages every 5 seconds). The number of messages is defined by printk_ratelimit_burst (currently defined as 10). This is particularly useful if you have a chatty kernel but a bandwidth-constrained console device (such as over a serial port). Note that within the kernel, rate limiting is caller controlled and is not implemented within printk. A printk user who desires rate limiting calls the printk_ratelimit function.
The dmesg command can also be used to print and control the kernel ring buffer. This command uses the klogctl system call to read the kernel ring buffer and emit it to standard output (stdout). The command can also be used to clear the kernel ring buffer (using the -c option), set the level for logging to the console (the -n option), and define the size of the buffer used to read the kernel log messages (the -s option). Note that if the buffer size is not specified, dmesg identifies the proper buffer size using the SYSLOG_ACTION_SIZE_BUFFER operation to klogctl.
the mother of all logging applications is syslog, a standardized logging framework that is implemented in major operating systems (including Linux® and Berkeley Software Distribution [BSD]). syslog has its own protocol used to convey event notification messages over a variety of transport protocols (dividing components into originators, relays, and collectors). In many cases, all three are implemented in a single host. In addition to syslog's many interesting features, it specifies how logging information is collected and filtered as well as where to store it. syslog has gone through numerous changes and evolved. You've probably heard of syslog, klog, or sysklogd. In more recent distributions of Ubuntu, a new version of syslog called rsyslog is used (based upon the original syslog), which refers to the reliable and extended syslogd.
The rsyslogd daemon, through its configuration file in /etc/rsyslog.conf), understands the /proc file system kmsg interface and uses it to extract kernel logging messages. Note that internally, all log levels are written through /proc/kmsg so that instead of the kernel defining which log levels to transport, the task is left to rsyslog itself. The kernel log messages are then stored in /var/log/kern.log (among other configured files). In /var/log, you'll find a plethora of log files that include general message and system-related calls (/var/log/messages), system boot log (/var/log/boot.log), authentication logs (/var/log/auth.log), and others.
Although the logs are available for your review, you can also use them for automated audits and forensics. A variety of log file analyzers exists for troubleshooting or compliance with security regulations and automatically look for problems using techniques such as pattern recognition or correlation analysis (even across systems).
