Linux System Admin

Posted on | Edited on | In

这个课程的收获就是boot stages, kernel的升级以及linux logging的种类，使用。特别是journald，来自systemd。我从其他文章中补充了一些内容，包括loginctl.

Boot

Linux booting process:

  • firmware stage (BIOS or UEFI)
  • boot loader stage (grub2)
  • kernel stage (ramdisk -> root filesystem)
  • initialization stage (systemd)

/boot directory is about kernel. grub configuration file:

1
2
3
# providing boot menu and excuting kernel
# -N: show line number
sudo less -N /boot/grub2/grub.cfg

讲了一下如何定制grub 的kernel 菜单选项 to add custom boot entry，grub 的菜单会在开机时的图形界面显示。可以在开机时更改kernel line加入systemd rescue or emergency target, refer here:

Kernel

Upgrade kernel version for CentOS:

1
2
3
4
5
6
7
8
9
# uname -r
sudo yum list installed kernel-*
# see new kernel available
sudo yum list available kernel

# update
sudo yum update -y kernel
# then reboot and check kernel version
sudo reboot

The above steps usually cannot help much because the lack of latest version in official repo. We need third-party repo, see this artical for help: How to Upgrade Linux Kernel in CentOS 7

Because we use SSH session to upgrade the kernal, so we are not able to select the kernel version on boot menu, we can do it by configuring the grub2:

1
2
3
4
5
6
7
# check kernel index list, index starts from 0
sudo awk -F\' '$1=="menuentry " {print $2}' /etc/grub2.cfg

CentOS Linux (5.4.125-1.el7.elrepo.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.31.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.25.1.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-adbe471f40421bfbf841690042db23fd) 7 (Core)

Switch to 5.4.125-1 version:

1
2
3
4
5
# set kernel index 0
sudo grub2-set-default 0
# reconfig boot loader code
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo reboot

After rebooting, check the kernel version:

1
uname -r

Switch back to old kernel version is easy:

1
2
3
4
5
# # set kernel index 2, see above index list
sudo grub2-set-default 2
# reconfig boot loader code
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo reboot

注意kernel version 和 OS version不一样，比如查看CentOS OS version:

1
2
3
cat /etc/centos-release
# or
rpm -qa centos-release

Linux Logging

Linux has 2 logging systems，这 2 个logging systems can run parallelly, or you can use journal alone.

  • rsyslog (persistent logs, can log remotely)
  • journald (nonpersistent by default)

syslog vs rsyslog vs syslog-ng: Basically, they are all the same, in the way they all permit the logging of data from different types of systems in a central repository, each project trying to improve the previous one with more reliability and functionalities.

Different logs for differnet purpose, some for failed jobs, some for cron jobs, etc. The rsyslog is a daemon:

1
systemctl status rsyslog

/etc/rsyslog.conf is the configuration file, see section under #### RULES ####. For example, anything beyond mail, authpriv and cron is logged in /var/log/messages, in the below file, cron.* means messages of all priorities will be logged (debug, info, notice, warn, err, crit, alert, emerg), cron.warn will log warn and above:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

Log rotate config is in /etc/logrotate.conf and there is a cron job for rotate /etc/cron.daily/logrotate.

If you want to log message to system log file, use logger command:

1
2
3
4
# you will see it in /var/log/messages
logger "hello"
# -p: priority
logger -p local4.info " This is a info message from local 4"

How to search and view rsyslog, see this article: Linux uses a set of configuration files, directories, programs, commands and daemons to create, store and recycle these log messages. The default location for log files in Linux is /var/log.

If you check with ls -ltr -S /var/log, the lastlog file may have a big size, way bigger than the disk space, it is a sparse file.

At the heart of the logging mechanism is the rsyslog daemon. This service is responsible for listening to log messages from different parts of a Linux system and routing the message to an appropriate log file in the /var/log directory. It can also forward log messages to another Linux server.

/var/log/messages 可以用vim等工具正常查看. command who, last其实是使用了/var/run/utmp and /var/run/wtmp

journalctl have the same logs as in rsyslogd, from here, persistent journal can replace rsyslogd.

1
2
3
4
5
# persist journald by making a dir
sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald
# you will see journal records here
ls -l /var/log/journal

Or enable in /etc/systemd/journald.conf, set Storage=persistent.

You can specify date ranges:

1
2
3
4
5
journalctl --since "2020-12-11 15:44:32"
# time left off is 00:00:00 midight
journalctl --since "2020-10-01" --until "2020-10-03 03:00"
journalctl --since yesterday
journalctl --since 09:00 --until "1 hour ago"

Some useful commands:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
# list boots
journalctl --list-boots
# check last boot journal
# -b -1: last boot
sudo reboot
sudo journalctl -b -1

# combine
journalctl -u nginx.service -u php-fpm.service --since today

# pid, uid, gid
journalctl _PID=8088
journalctl _UID=33 --since today

# -F: show available values
journalctl -F _GID
journalctl -F _UID

# check executable
journalctl /usr/bin/bash

# display only kernel message
journalctl -k

# by priority, can use number or name
#0: emerg
#1: alert
#2: crit
#3: err
#4: warning
#5: notice
#6: info
#7: debug
journalctl -p err -b

# the same as tail -n/-f
journalctl -n 10
journalctl -f

# disk usage
journalctl --disk-usage
# shrink
sudo journalctl --vacuum-size=1G
sudo journalctl --vacuum-time=1years

You can use right arrow key to see full entry if it is too long.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# print all on stdout, no pager with less
journalctl --no-pager

# -o output format
#cat: Displays only the message field itself.
#export: A binary format suitable for transferring or backing up.
#json: Standard JSON with one entry per line.
#json-pretty: JSON formatted for better human-readability
#json-sse: JSON formatted output wrapped to make add server-sent event compatible
#short: The default syslog style output
#short-iso: The default format augmented to show ISO 8601 wallclock timestamps.
#short-monotonic: The default format with monotonic timestamps.
#short-precise: The default format with microsecond precision
#verbose: Shows every journal field available for the entry, including those usually hidde 
#internally.
journalctl -b -u nginx -o json
journalctl -b -u nginx -o json-pretty

Linux Session

Other capabilities, like log management and user sessions are handled by separated daemons and management utilities (journald/journalctl and logind/loginctl respectively).

Get info about user and the processes he is running before:

1
2
3
4
5
6
7
8
# list sessions
loginctl list-sessions

# session status 
# you can see the user action history
loginctl session-status [session id]
loginctl show-session [session id]
loginctl kill-session [session id]
0%