Top

A Linux User Reference

Search tips
  • search ignores words that are less than 4 characters in length
  • searches are case insensitve
  • if a search does not return anything try it in Boolean mode then Query expansion mode by checking the appropriate radio button e.g. searching for 'cron' in just the Administration category returns nothing - presumably because the 50% threshold is reached. Boolean mode ignores this threshold so a search for 'cron' returns several hits
  • in Boolean mode preceding a word with a '+' means the result must include that word, a '-' means it must not
  • in Boolean mode '+crontab -anacron' means match articles about crontab that DO NOT mention anacron
  • to match a phrase e.g. 'manage system' check the Boolean mode radio button and enclose the phrase in quotes "some phrase ..."
  • in Query expansion mode the search context is expanded beyond the keywords you entered - relevancy of hits may well be degraded

ADMINISTRATION

System log

  • Table of system log related files
    File name Description
    /dev/log Unix domain socket to/from which local syslog messages are read
    /var/run/syslogd.pid The file containing the process id of syslogd
    /etc/syslog.conf Main configuration file for the syslogd daemon
    /sbin/syslogd Part of sysklogd, supports system local and remote logging
    /sbin/klogd Part of sysklogd, supports kernel message logging/trapping can run as a standalone or client of syslogd
    /proc/kmsg One Source for kernel messages klogd
    /var/run/klogd.pid The file containing the process id of klogd
    /boot/System.map (or /System.map, /usr/src/linux/System.map) Default locations for kernel system maps
  • Syslog facilities and priorities

    Logging is based on a combination of facility and priority. Facilities are basically a means of grouping message types.

    Facility Priority
    auth (or security) 1 - debug
    authpriv 2 - info
    cron 3 - notice
    daemon 4 - warning (or warn)
    ftp 5 - err (or error)
    kern 6 - crit
    lpr 7 - alert
    mail 8 - emerg (or panic)
    mark
    news
    syslog
    user
    uucp
    local0
    local1
    .....
    local7

    Alternative facility keywords (or .....) are deprecated and should not be used, mark is for internal use.

  • Example syslog configuration file
    /etc/syslog.conf
    Format: <facility>.<priority>   [action]
    

    Where [action] can be

    A regular file

    Full pathname. '-' prefix to omit syncing file after each update.

    Named Pipes

    Used as a destination for log messages, | .

    Terminal & Console

    Terminal such as /dev/console.

    Remote Machine

    Send messages to another host, (@)hostname, messages are not forwarded from the receiving host.

    List of Users

    Comma-separated list of logged in users, * for all logged in users (uses wall command).

    Sample configuration - /etc/syslog.conf

    # Log all kernel messages to the console
    kern.*                                                            /dev/console
    
    # log all facilities (*.) of level info or higher except
    # mail,authpriv and cron
    *.info;mail.none;authpriv.none;cron.none                          /var/log/messages
    
    # The authpriv file has restricted access
    authpriv.*                                                        /var/log/secure
    
    # Log all the mail messages in one place
    mail.*                                                           -/var/log/maillog
    
    # Log cron stuff
    cron.*                                                            /var/log/cron
    
    # Everybody gets emergency messages
    *.emerg                                                           *
    
    # Save news errors of level crit and higher in a special file
    uucp,news.crit                                                    /var/log/spooler
    
    # Save boot messages also to boot.log
    local7.*                                                          /var/log/boot.log
    
  • Some further examples
    /etc/syslog.conf
    # Send critical and higher to remote host logsrv
    # and to the console
    kern.*                                                            /var/log/kernel
    kern.crit                                                         @logsrv
    kern.crit                                                         /dev/console
    
    # Send info, notice and warning messages to /var/log/kernel-info
    kern.info;kern.!err                                               /var/log/kernel-info
    
    # Store all mail messages except info priority in /var/log/mail
    mail.*;mail.!=info                                                /var/log/mail
    
  • Syslog security features
    auth,authpriv
    auth

    This facility reports on authentication and authorisation messages from system commands e.g. pam, ...

    authpriv

    This facility reports non system authorisation messages e.g. tcpd, su, sudo, ...

    Enable 'tcpd' logging - /etc/syslog.conf

    auth,authpriv.*            /var/log/auth.log
    
  • Enable sudo logging

    Disabled by default on some distros.

    Touch, create the log file if it does not exist

    # touch /var/log/sudo
    

    Configure syslog to log to sudo log file - /etc/syslog.conf

    local2.debug               /var/log/sudo
    

    Restart syslog

    # kill -HUP <syslogd PID>
    
  • Recording login attempts
    /var/log/wtmp, /var/log/btmp

    Successful (wtmp) and unsuccessful (btmp) login attempts are recorded .

    They are not clear text files, can examine them using the last (wtmp) and lastb (btmp) commands. _lastb requires root privileges.

  • System logging and kernel messages

    sysklogd provides two system utilities which provide support for system logging and kernel message trapping. Support of both internet and unix domain sockets allows for local and remote logging.

    System logging is provided by a version of syslogd. Support for kernel logging is provided by the klogd utility which allows kernel logging to be conducted in either a standalone fashion or as a client of syslogd.

  • System logging daemon
    /sbin/syslogd

    Linux system log daemon.

    Send signal(s) to syslogd

    # kill <-SIGNAL> `cat /var/run/syslogd.pid`
    
    Signal Action
    SIGHUP Re-initialise. All open files are closed, configuration file is reread and the syslog facility is re-started.
    SIGTERM The syslogd will die.
    SIGINT or SIGQUIT If debugging is enabled these are ignored, otherwise syslogd will die.
    SIGUSR1 Switch debugging on/off. Only if syslogd is started with the -d debug option.
    SIGCHLD Wait for children if some were born, because of 'wall'ing messages.
  • Kernel logging daemon
    /sbin/klogd

    System daemon which intercepts and logs Linux kernel messages.

    Send signal(s) to syslogd

    # kill <-SIGNAL> `cat /var/run/klogd.pid`
    
    Signal Action
    SIGHUP Close kernel log sources and terminate gracefully
    SIGINT Close kernel log sources and terminate gracefully
    SIGKILL Close kernel log sources and terminate gracefully
    SIGTERM Close kernel log sources and terminate gracefully
    SIGTSTP Stop kernel logging. Close log sources and spin in an idle loop
    SIGCONT Start kernel logging - reinitialises
    SIGUSR1 Reload the kernel module symbols
    SIGUSR2 Reload both static kernel symbols and the kernel module symbols
  • Log rotation
    /etc/logrotate.conf

    Control the size of log files, usually run as a cron job. General idea is that log files are periodically backed up and a new log is started. Several generations of a log are kept, when a log ages to the last generation it may be archived.

    The configuration file /etc/logrotate.conf defines how log rotating and archiving should happen.

    Example configuration

    # rotate log files weekly
    Weekly
    
    # keep 4 weeks worth of backlogs
    Rotate 4
    
    # create new (empty) log files after rotating old ones
    Create
    
    # compress log files
    Compress
    
    # RPM packages drop log rotation information into this directory
    Include /etc/logrotate.d
    
    # no packages own wtmp -- we'll rotate them here
    /var/log/wtmp {
        missingok
        monthly
        create 0664 root utmp
        rotate 1
    }
    
    # system-specific logs may be also be configured here.  AN EXAMPLE BELOW
    /var/log/messages {
        rotate 5                              # Five backups are kept
        mail logsave@pinguino                 # When the oldest backup ages out mail to logsave@pinguino
        size 100k                             # Rotated after it reaches 100KB in size
        postrotate                            # A script to run after the rotation is complete
           /usr/bin/killall -HUP syslogd      # Script restarts the syslogd daemon
        endscript                             # Statement is required to terminate the script
                                              # also required if a prerotate script is present
    }
    
  • Monitor a log file
    /usr/bin/tail
    # tail -f /var/log/<logfile>
    
  • Log a message to syslog
    /usr/bin/logger

    Log some messages to syslog

    # logger -p user.info "mark test"
    
    # logger -f /var/log/messages "mark test - file only"
    
    # logger -f /var/log/messages -t ---- "mark test - file and tag"
    

    View the log entries

    # tail /var/log/messages
    Nov 23 14:34:59 localhost mark: mark test
    Nov 23 14:36:55 localhost logger: mark test - file only
    Nov 23 14:37:40 localhost ----: mark test - file and tag
    
  • Configure a Syslog server

    Syslog server uses port 514/UDP. Reserve/configure the port in /etc/services if not already there.

    Listen on network for remote 'syslogd' connections

    # syslogd -r
    

    Configure a syslog server on Debian - /etc/default/syslogd

    # For remote UDP logging use SYSLOGD="-r"
    SYSLOGD="-r"
    

    Configure a syslog server on Fedora, Redhat - /etc/sysconfig/syslog

    # Options to syslogd
    # -m 0 disables 'MARK' messages.
    # -r enables logging from remote machines
    # -x disables DNS lookups on messages received with -r
    SYSLOGD_OPTIONS="-m 0 -r"
    
    # Options to klogd
    # -2 prints all kernel oops messages twice; once for klogd to decode, and
    #    once for processing with 'ksymoops'
    # -x disables all klogd processing of oops messages entirely
    # See klogd(8) for more details
    KLOGD_OPTIONS="-x"
    
    SYSLOG_UMASK=077
    # set this to a umask value to use for all log files as in umask(1).
    # By default, all permissions are removed for "group" and "other".
    

    Allow UDP traffic through to syslog server, modify any Firewalls rules that block UDP/514 accordingly.

  • Syslog client configuration

    For this example f64local is the syslog server. Clients will forward configured syslog facilities to the syslog server.

    In client's /etc/syslog.conf

    kern.*      @f64local
    users.*     @f64local
    .....
    

    Restart syslogd on client(s)

    # /etc/init.d/sysklogd restart
    or
    $ sudo kill -SIGHUP `cat /var/run/syslogd.pid`
    
  • Rsyslog
    • an enhanced 'syslogd' supporting, among others, MySQL, PostgreSQL, failover log destinations, syslog/tcp, fine grain output format control, high precision timestamps, queued operations and the ability to filter on any message part
    • implements a modular design

    The articles that follow will be brief so please read the manual if you are looking to gain greater understanding and know-how.

    Are you running it?

    $ ps aux | grep syslog[Dd]
    syslog    1090  0.0  0.0 249476  1616 ?        Sl   13:43   0:00 rsyslogd -c5
    

    Installed by default on at least Mint 14 (debian, ubuntu based).

    '-c5' sets the version compatibility as explained a bit below.

    Debian, Mint 13 default file - /etc/default/rsyslog

    # Options for rsyslogd
    # -m 0 disables 'MARK' messages (deprecated, only used in compat mode < 3)
    # -r enables logging from remote machines (deprecated, only used in compat mode < 3)
    # -x disables DNS lookups on messages received with -r
    # -c compatibility mode
    # See rsyslogd(8) for more details
    RSYSLOGD_OPTIONS="-c5"
    
  • Rsyslog configuration file
    • configured via the rsyslog.conf file, typically found in /etc.
    • by default, 'rsyslogd' reads the file /etc/rsyslog.conf. This may be changed by command line option "-f".

    (Debian based) Configuration files directory - /etc/rsyslog.d

    20-ufw.conf  50-default.conf
    
    • configuration files consists of statements, line spacing does NOT matter
    • old style (sysklogd & legacy rsyslog), line spacing does matter
    • comments start with '#' or enclosed in '/* ..... */'
    • directives are processed from the top of rsyslog.conf to the bottom => sequence matters

    Flow Control Statements

    if expr then ... else ...    - conditional execution
    stop                         - stops processing the current message
    call                         - calls a ruleset (just like a subroutine call)
    continue                     - a NOP, useful e.g. inside the then part of an if 
    

    Data Manipulation Statements

    set                          - sets a user variable
    unset                        - deletes a previously set user variable 
    

    Inputs

    Every input requires an input module to be loaded and a listener defined for it. Full details → rsyslog modules documentation.

    Outputs

    Outputs are also called "actions". Some are pre-loaded, others must be loaded just like inputs.

    An action is invoked via the action(type="type" ...) object. Type is mandatory and must contain the name of the plugin to be called (e.g. "omfile" or "ommongodb"). Other paramters may be present.

    Rulesets and Rules

    A rule is a way how rsyslog shall process a specific message. Usually, there is a type of filter (if-statement) in front of the rule. Complex nesting of rules is possible, much like in a programming language.

    Rulesets are containers for rules. A ruleset can be "bound" (assigned) to a specific input. Full details &rarr rsyslog rulesets documentation

    For quick reference, rulesets are defined as follows:

    ruleset(name="rulesetname") {
        action(type="omfile" file="/path/to/file")
        action(type="..." ...)
        /* and so on... */
    }
    
  • Log remote messages to file
    rsyslog.conf

    This is a slimmed down version of the rsyslog newbie guide.

    Objective

    configure 'rsyslog' to recive UDP messages, to filter them depending on the IP of the host and to store them in a file.

    Steps:

    1. configure the module first - Module (load="im-type of protocol-")
    2. configure the input - Input (type="im-protocol of input-“port=”-number of port-")
    3. configure the filter - if ... then conditional statement
    4. configure the action - Action (type="omfile” File=”-filename-")
    Module (load=”imudp”)
    Input (type=”imudp” port=”514”)
    If $fromhost-ip == “172.19.1.135” then [
       Action (type=”omfile” file=”/var/log/network1.log”)
    ]
    
  • Ruleset example
    rsyslog.conf

    A ruleset is a set of rules bound to an input via the input option ruleset="-rulesetname-".

    Bind a ruleset 'rs1' to a input

    Input (type=”imudp” port=”514” ruleset=”rs1”)
    

    Ruleset to store all messages in file /var/log/network1.log

    Module (load=”imudp”)
    Input (type=”imudp” port=”514” ruleset=”rs1”)
    Ruleset (name=”rs1”) {
       Action (type=”omfile” file=”/var/log/network1.log”)
    }
    
  • Legacy syslog entries
    rsyslog.conf

    Taken from the debian based Mint 14 default configuration - /etc/rsyslog.d/50-default.conf

    .....
    # First some standard log files.  Log by facility.
    #
    auth,authpriv.*                  /var/log/auth.log
    *.*;auth,authpriv.none          -/var/log/syslog
    kern.*                          -/var/log/kern.log
    mail.*                          -/var/log/mail.log
    
    # Set up your local facilies as before e.g.
    #
    local1.*                        -/var/log/dhcp
    local3.err                       /var/log/myapp
    
    # Logging for the mail system.  Split it up so that
    # it is easy to write scripts to parse these files.
    #
    #mail.info                      -/var/log/mail.info
    #mail.warn                      -/var/log/mail.warn
    mail.err                         /var/log/mail.err
    ....
    

    By default mail is logged twice .....