Top

A Linux User Reference

Search tips
  • search ignores words that are less than 4 characters in length
  • searches are case insensitve
  • if a search does not return anything try it in Boolean mode then Query expansion mode by checking the appropriate radio button e.g. searching for 'cron' in just the Administration category returns nothing - presumably because the 50% threshold is reached. Boolean mode ignores this threshold so a search for 'cron' returns several hits
  • in Boolean mode preceding a word with a '+' means the result must include that word, a '-' means it must not
  • in Boolean mode '+crontab -anacron' means match articles about crontab that DO NOT mention anacron
  • to match a phrase e.g. 'manage system' check the Boolean mode radio button and enclose the phrase in quotes "some phrase ..."
  • in Query expansion mode the search context is expanded beyond the keywords you entered - relevancy of hits may well be degraded

ADMINISTRATION

Troubleshooting

  • Hardware analysis tools
    Tool Description
    dmesg Prints kernel ring buffer
    lspci Lists all PCI devices.
    lsmod Lists loaded kernel modules.
    lsusb Lists USB devices.
    lspnp Lists Plug-and-Play devices.
    lshw Lists hardware.
  • Filesystem analysis

    Directories particularly important to look at for startup and runtime problems.

    Directory Function
    /proc/ Is the virtual filesystem with information on processes and system status, the guts of a running system
    /var/log/ If something goes wrong, helpful information may be found in a file here e.g. boot, messages, kern.log
    / On some systems bootup files like vmlinuz and initrd.img may be here rather than /boot/
    /boot Stores files directly used in the kernel boot process
    /lib/modules/kernel Modules, nested under this directory in subdirectories named for the current kernel version (if you boot multiple kernel versions, multiple directories should exist)
  • System identification analysis tools

    Tools to see what is going on at the moment - there are more such as ps , top.

    Tool Description
    lsof Lists open files
    ltrace Trace library calls
    lsmod List currently loaded modules
    insmod Loads kernel modules
    rmmod Removes kernel modules
    modprobe Higher level insmod, rmmod, lsmod wrapper
    uname Prints system information (kernel version, etc.)
    strace Traces system calls
    strings Find strings in binaries
  • Network analysis tools

    Many available - the basics. I always use the OSI 7 layer model as a reference, starting at the bottom and working my way up.

    Tool Description
    ping Test network reachability
    ifconfig List configured interfaces
    dig Name service utility
    route Routing tables
    ipchains Firewall settings - /var/log/messages
    iptables Firewall settings - /var/log/messages
    iwlist List wireless configuration
    netstat View network statistics
    lsof -i List open internet files
    tcpdump Network sniffers (wireshark ...)
    traceroute Traceroute
  • Access an unbootable system
    /usr/sbin/chroot

    A 'chroot' can be used to move back into the damaged environment after bootstrapping from an alternate root file system (such as from installation media, or a LiveCD).

    Run command or interactive shell with special root directory

    chroot <newroot> [command...]
    
  • The init process

    Few things go wrong, if it does, usually a misconfiguration in /etc/inittab or whatever the distribution uses.

    A common problem (once)

    'init: ID "process name" respawning too fast : disabled for 5 minutes'
    

    The process/application maybe segfaulting and restarting

    The process/application may not even exist

    The problem lies with the process/application not init

  • None hardware or application specific problems

    Problems that do not relate to hardware or specific applications are generally down to misconfiguration of environment e.g. keyboard mappings, garbled console, wrong home directory, incorrect path, incorrect prompt string, wrong editor, ...

    Valid login shells - /etc/shells

    /usr/bin/es
    /usr/bin/ksh
    /bin/ksh
    /usr/bin/rc
    /usr/bin/tcsh
    ....
    /bin/false
    

    Different login shells read different configuration files on startup e.g. 'bash' reads different files when invoked as a login shell as opposed to an interactive shell.

  • The bash shell environment

    Bash can be invoked in several different ways. It's global configuration file /etc/profile is always read.

    Invoked shell Files read
    login bash reads ~/.bash_profile, ~/.bash_login and ~/.profile
    interactive (cmd-line) bash reads ~/.bashrc
    in a script bash reads $BASH_ENV environment variable
    as 'sh' bash reads filename listed in $ENV variable
  • Kernel run-time parameters
    /sbin/sysctl

    These generally enable or disable a feature or allow for system tuning that may improve performance. Misconfiguration can obviously cause problems.

    The parameters available are those listed under /proc/sys/

    List variables that can be controlled and their current setting

    # sysctl -a
    .....
    net.core.netdev_budget = 300
    net.core.warnings = 1
    net.core.somaxconn = 128
    abi.vsyscall32 = 1
    crypto.fips_enabled = 0
    .....
    
    # sysctl -a | wc -l
    657                            (There are one or two configurable kernel parameters!)
    
  • Set sysctl parameters at boot time
    /etc/sysctl.conf
    .....
    # Controls IP packet forwarding
    net.ipv4.ip_forward = 0
    
    # Controls source route verification
    net.ipv4.conf.default.rp_filter = 1
    .....
    # sysctl -a | grep ipv4.ip
    net.ipv4.ip_forward = 0
    net.ipv4.ip_default_ttl = 64
    net.ipv4.ip_no_pmtu_disc = 0
    ........
    net.ipv4.ip_local_port_range = 32768 61000
    net.ipv4.ipfrag_secret_interval = 600
    net.ipv4.ipfrag_max_dist = 64
    

    There are further 'sysctl' examples at configure for routing - Network configuration (Routing), and manage file handles - Security (Hardening).

  • Dynamic libraries
    /etc/ld.so.conf

    /lib/ and /usr/lib/ are cached.

    To include directories in system wide search path add there directory names to /etc/ld.so.conf and run 'ldconfig' as root.

    Again the set up structure may vary depending on the distibution – in Mint /usr/local/lib is included via ...

    $ more /etc/ld.so.conf 
    include /etc/ld.so.conf.d/*.conf 
    
    $ more /etc/ld.so.conf.d/libc.conf 
    # libc default configuration 
    /usr/local/lib 
    
  • Authorisation problems

    Generally down to misconfiguration of config files especially for PAM and SELinux.

    Check

    /var/log/messages
    /var/log/auth.log
    
  • File access problems

    Check file permissions and ownership and

    /etc/passwd
    /etc/shadow
    /etc/group
    /etc/gshadow
    
  • System logging problems

    Problems with logging are generally down to /etc/syslog.conf configuration or logging configuration in an applications configuration file. Intermittent events may be the results of scripts running periodically.

    /etc/syslog.conf can achieve a degree of control over what events are logged and where. Events may not necessarily be logged immediately on receipt, maybe a few seconds delay.

    Check

    cron
    anacron
    /etc/syslog.conf
    crontab files
    
  • Crontab problems

    Most problems are not to do with 'cron' but with permissions and configuration. Scripts to be run need the appropriate permissions.

    /etc/cron.d/ contains crontab files, often system related, these are not scripts and require the user argument.

    -rw-r--r--   1 root root   267 2007-11-20 20:40 smolt
    -rw-r--r--   1 root root   192 2007-08-20 10:15 sysstat
    
    User crontabs - '/var/spool/crontabs/[user]'

    Common problems are permissions, environment related. Users often do not have the correct permissions especially if using their own script(s). The environment that 'cron' runs under is different from the users.

    Check that files exist

    /etc/cron.allow
    /etc/cron.deny
    

    If 'cron.allow' file exists and a [user] is not in it - they cannot use 'crontab'.

  • Printing problems
    printer drivers

    'lpstat' will report an error if it is unable to connect to 'cupsd'.

    Printer drivers

    Check availability at Open printing.org to see if your printer is supported.

    List the available devices or drivers known to the CUPS server

    $ lpinfo -v
    network socket
    .....
    network http
    network ipp
    network lpd
    direct scsi
    serial serial:/dev/ttyS0?baud=115200
    network smb
    

    Check if printer driver is available

    # lpinfo -m | grep -i "C64"
    Foomatic:Epson-Stylus_C64-gutenprint-ijs-simplified.5.0.ppd Epson 
    Stylus C64 Foomatic/gutenprint-ijs-simplified.5.0
    Foomatic:Epson-Stylus_C64-gutenprint-ijs.5.0.ppd Epson Stylus C64 
    Foomatic/gutenprint-ijs.5.0
    

    Make sure to check with Linux Foundation - open printing (via link in this article).

  • Printing
    ensure CUPS server is running
    # ps -ef | grep -[Cc]upsd
    
    # /etc/init.d/cupsys status                       (Script may be called 'cups', ...)
     Status of Common Unix Printing System: cupsd is not running.
    
    # /etc/init.d/cupsys start                        (Script may be called 'cups', ..)
     * Starting Common Unix Printing System: cupsd            [OK]
    
  • Receive error message when queuing a job for printing
    # lpstat -a                     (or)
    
    # lpc status                    (To check that the printer is accepting jobs)
    
  • A queued job does not print
    # lpstat -p
    
    # lpc status                    (To check that the printer is accepting jobs)
                                    (May need to move the job to another printer)
    
  • Remote printer problems
    • Check that it still exists on the remote system and that it is operational.
    • May need to update the configuration file to allow a particular user or remote system to print on the printer.
    • May need to ensure that the firewall allows remote printing requests.
    • May need to verify that you have the right driver.