|dmesg||Prints kernel ring buffer|
|lspci||Lists all PCI devices.|
|lsmod||Lists loaded kernel modules.|
|lsusb||Lists USB devices.|
|lspnp||Lists Plug-and-Play devices.|
- Hardware analysis tools
- Filesystem analysis
Directories particularly important to look at for startup and runtime problems.
Directory Function /proc/ Is the virtual filesystem with information on processes and system status, the guts of a running system /var/log/ If something goes wrong, helpful information may be found in a file here e.g. boot, messages, kern.log / On some systems bootup files like vmlinuz and initrd.img may be here rather than /boot/ /boot Stores files directly used in the kernel boot process /lib/modules/kernel Modules, nested under this directory in subdirectories named for the current kernel version (if you boot multiple kernel versions, multiple directories should exist)
- System identification analysis tools
Tools to see what is going on at the moment - there are more such as ps , top.
Tool Description lsof Lists open files ltrace Trace library calls lsmod List currently loaded modules insmod Loads kernel modules rmmod Removes kernel modules modprobe Higher level insmod, rmmod, lsmod wrapper uname Prints system information (kernel version, etc.) strace Traces system calls strings Find strings in binaries
- Network analysis tools
Many available - the basics. I always use the OSI 7 layer model as a reference, starting at the bottom and working my way up.
Tool Description ping Test network reachability ifconfig List configured interfaces dig Name service utility route Routing tables ipchains Firewall settings - /var/log/messages iptables Firewall settings - /var/log/messages iwlist List wireless configuration netstat View network statistics lsof -i List open internet files tcpdump Network sniffers (wireshark ...) traceroute Traceroute
- Access an unbootable system
A 'chroot' can be used to move back into the damaged environment after bootstrapping from an alternate root file system (such as from installation media, or a LiveCD).
Run command or interactive shell with special root directory
chroot <newroot> [command...]
- The init process
Few things go wrong, if it does, usually a misconfiguration in /etc/inittab or whatever the distribution uses.
A common problem (once)
'init: ID "process name" respawning too fast : disabled for 5 minutes'
The process/application maybe segfaulting and restarting
The process/application may not even exist
The problem lies with the process/application not init
- None hardware or application specific problems
Problems that do not relate to hardware or specific applications are generally down to misconfiguration of environment e.g. keyboard mappings, garbled console, wrong home directory, incorrect path, incorrect prompt string, wrong editor, ...
Valid login shells - /etc/shells
/usr/bin/es /usr/bin/ksh /bin/ksh /usr/bin/rc /usr/bin/tcsh .... /bin/false
Different login shells read different configuration files on startup e.g. 'bash' reads different files when invoked as a login shell as opposed to an interactive shell.
- The bash shell environment
Bash can be invoked in several different ways. It's global configuration file /etc/profile is always read.
Invoked shell Files read login bash reads ~/.bash_profile, ~/.bash_login and ~/.profile interactive (cmd-line) bash reads ~/.bashrc in a script bash reads $BASH_ENV environment variable as 'sh' bash reads filename listed in $ENV variable
- Kernel run-time parameters
These generally enable or disable a feature or allow for system tuning that may improve performance. Misconfiguration can obviously cause problems.
The parameters available are those listed under /proc/sys/
List variables that can be controlled and their current setting
# sysctl -a ..... net.core.netdev_budget = 300 net.core.warnings = 1 net.core.somaxconn = 128 abi.vsyscall32 = 1 crypto.fips_enabled = 0 ..... # sysctl -a | wc -l 657 (There are one or two configurable kernel parameters!)
- Set sysctl parameters at boot time
..... # Controls IP packet forwarding net.ipv4.ip_forward = 0 # Controls source route verification net.ipv4.conf.default.rp_filter = 1 ..... # sysctl -a | grep ipv4.ip net.ipv4.ip_forward = 0 net.ipv4.ip_default_ttl = 64 net.ipv4.ip_no_pmtu_disc = 0 ........ net.ipv4.ip_local_port_range = 32768 61000 net.ipv4.ipfrag_secret_interval = 600 net.ipv4.ipfrag_max_dist = 64
There are further 'sysctl' examples at configure for routing - Network configuration (Routing), and manage file handles - Security (Hardening).
- Dynamic libraries
/lib/ and /usr/lib/ are cached.
To include directories in system wide search path add there directory names to /etc/ld.so.conf and run 'ldconfig' as root.
Again the set up structure may vary depending on the distibution – in Mint /usr/local/lib is included via ...
$ more /etc/ld.so.conf include /etc/ld.so.conf.d/*.conf $ more /etc/ld.so.conf.d/libc.conf # libc default configuration /usr/local/lib
- Authorisation problems
Generally down to misconfiguration of config files especially for PAM and SELinux.
- File access problems
Check file permissions and ownership and
/etc/passwd /etc/shadow /etc/group /etc/gshadow
- System logging problems
Problems with logging are generally down to /etc/syslog.conf configuration or logging configuration in an applications configuration file. Intermittent events may be the results of scripts running periodically.
/etc/syslog.conf can achieve a degree of control over what events are logged and where. Events may not necessarily be logged immediately on receipt, maybe a few seconds delay.
cron anacron /etc/syslog.conf crontab files
- Crontab problems
Most problems are not to do with 'cron' but with permissions and configuration. Scripts to be run need the appropriate permissions.
/etc/cron.d/ contains crontab files, often system related, these are not scripts and require the user argument.
-rw-r--r-- 1 root root 267 2007-11-20 20:40 smolt -rw-r--r-- 1 root root 192 2007-08-20 10:15 sysstat
- User crontabs - '/var/spool/crontabs/[user]'
Common problems are permissions, environment related. Users often do not have the correct permissions especially if using their own script(s). The environment that 'cron' runs under is different from the users.
Check that files exist
If 'cron.allow' file exists and a [user] is not in it - they cannot use 'crontab'.
- Printing problems
'lpstat' will report an error if it is unable to connect to 'cupsd'.
- Printer drivers
Check availability at Open printing.org to see if your printer is supported.
List the available devices or drivers known to the CUPS server
$ lpinfo -v network socket ..... network http network ipp network lpd direct scsi serial serial:/dev/ttyS0?baud=115200 network smb
Check if printer driver is available
# lpinfo -m | grep -i "C64" Foomatic:Epson-Stylus_C64-gutenprint-ijs-simplified.5.0.ppd Epson Stylus C64 Foomatic/gutenprint-ijs-simplified.5.0 Foomatic:Epson-Stylus_C64-gutenprint-ijs.5.0.ppd Epson Stylus C64 Foomatic/gutenprint-ijs.5.0
Make sure to check with Linux Foundation - open printing (via link in this article).
ensure CUPS server is running
# ps -ef | grep -[Cc]upsd # /etc/init.d/cupsys status (Script may be called 'cups', ...) Status of Common Unix Printing System: cupsd is not running. # /etc/init.d/cupsys start (Script may be called 'cups', ..) * Starting Common Unix Printing System: cupsd [OK]
- Receive error message when queuing a job for printing
# lpstat -a (or) # lpc status (To check that the printer is accepting jobs)
- A queued job does not print
# lpstat -p # lpc status (To check that the printer is accepting jobs) (May need to move the job to another printer)
- Remote printer problems
- Check that it still exists on the remote system and that it is operational.
- May need to update the configuration file to allow a particular user or remote system to print on the printer.
- May need to ensure that the firewall allows remote printing requests.
- May need to verify that you have the right driver.