Top

A Linux User Reference

Search tips
  • search ignores words that are less than 4 characters in length
  • searches are case insensitve
  • if a search does not return anything try it in Boolean mode then Query expansion mode by checking the appropriate radio button e.g. searching for 'cron' in just the Administration category returns nothing - presumably because the 50% threshold is reached. Boolean mode ignores this threshold so a search for 'cron' returns several hits
  • in Boolean mode preceding a word with a '+' means the result must include that word, a '-' means it must not
  • in Boolean mode '+crontab -anacron' means match articles about crontab that DO NOT mention anacron
  • to match a phrase e.g. 'manage system' check the Boolean mode radio button and enclose the phrase in quotes "some phrase ..."
  • in Query expansion mode the search context is expanded beyond the keywords you entered - relevancy of hits may well be degraded

ADMINISTRATION

Backup & install

  • What to backup?

    Often includes:

    /home/, /etc/, /usr/local/ maybe /root/ and /boot/
    

    and maybe parts of

    /var/log/ and /var/mail/
    
  • Backup strategies

    In general there are three approaches to backups.

    Full backup

    A complete backup, usually of a whole filesystem, directory or group of related files.

    Takes the longest time to achieve so often used with one of the following approaches.

    Differential or cumulative backup

    A backup of all things that have changed since the last full backup.

    Recovery requires the last full backup plus the latest differential backup.

    Incremental backup

    A backup of only those changes since the last incremental backup.

    Recovery requires the last full backup plus all of the incremental backups (in order) since the last full backup.

  • Using raw devices
    • One way to make a full backup of a filesystem is to make an image of the partition on which it resides.
    • A raw device, such as /dev/hda1 or /dev/sda2, can be opened and read/write as a sequential file using the 'dd' command.
  • Convert and copy a file
    /bin/dd
    • Copy a file, converting and formatting according to the operands.
    • In it's simplest form, dd copies an input file to an output file, where either file may be a raw device.

    Copy the contents of /dev/sda3 to file

    # dd if=/dev/sda3 of=backup-1
    2040255+0 records in
    2040255+0 records out
    1044610560 bytes (1.0 GB) copied, 49.3103 s, 21.2 MB/s 
    

    Copy and compress the contents of /dev/sda3 to file

    # dd if=/dev/sda3 | gzip > backup-2
    2040255+0 records in
    2040255+0 records out
    1044610560 bytes (1.0 GB) copied, 117.723 s, 8.9 MB/s
    

    Restoring a partition using 'dd'

    # dd if=backup-1 of=/dev/sdc7
    2040255+0 records in
    2040255+0 records out
    1044610560 bytes (1.0 GB) copied, 44.0084 s, 23.7 MB/s
    
  • Backup using an ext2/3/4 linux filesystem aware utility
    /sbin/dump

    Dump can be used for full, differential or incremental backups. You may need to install it - (debian) apt-get install dump

    Make a full backupof a device e.g. /dev/sda3

    # dump -0 -f backup-4 -j -u /dev/sda3
        DUMP: Date of this level 0 dump: Sun Jul 8 16:47:47 2007
        DUMP: Dumping /dev/sda3 (/grubfile) to backup-4
        .....
        DUMP: Date of this level 0 dump: Sun Jul 8 16:47:47 2007
        DUMP: Date this dump completed: Sun Jul 8 16:47:57 2007
        DUMP: Average transfer rate: 819 kB/s
        DUMP: Wrote 12260kB uncompressed, 7377kB compressed, 1.662:1
        DUMP: DUMP IS DONE 
    

    full-backup (-0), to output file (-f), using bzlib compression (-j) and outputting dump info (-u)

    What filesystems need to be dumped

    $ dump -W
    Last dump(s) done (Dump '>' file systems):
      /dev/sda5 (     /) Last dump: never
      /dev/sda2 ( /boot) Last dump: never
      /dev/sda8 ( /home) Last dump: never
      /dev/sda10    (  /tmp) Last dump: never
      /dev/sda7 (  /usr) Last dump: never
      /dev/sda9 (  /var) Last dump: never
      /dev/sda11    (/archives) Last dump: never
      /dev/sdb1 (/media/usball) Last dump: never
    
  • Restore files or file systems from backups made with dump
    /sbin/restore

    There are many options. restore is installed automatically when dump is installed.

    Checking the contents of 'backup-5' dump archive file

    # restore -t -f backup-5
    Dump tape is compressed.
    Dump    date: Sun Jul 8 16:55:46 2007
    Dumped from: Sun Jul 8 16:47:47 2007
    ..... 
    

    Restore a file from 'backup-6' dump archive file

    # restore -i -f backup-6
    Dump tape is compressed.
    restore >
    restore > ls new*
    newerfile
    newfile
    restore > add newerfile
    restore > extract
    You have not read any volumes yet.  Unless you know which volume your file(s) are \
    on you should start with the last volume and work towards the first.
    Specify next volume # (none if no more volumes): 1
    Set owner/mode for '.'? [yn] y
    restore > q 
    
  • Archiving utilities
    • Useful for partial backups, both 'cpio' and 'tar' are more commonly used for this purpose.
    • Both can write archives to files or to devices such as tape or floppy disk and both can restore from such archives.
    • 'tar' is more commonly used as it handles complete directories better.
    • GNU 'tar' supports both 'gzip' and 'bzip' compression.
  • Copy files to and from archives
    /bin/cpio
    • Operates in 'copy-out' mode to create an archive, 'copy-in' mode to restore an archive and 'copy-pass' mode to copy a set of files from one location to another.
    • Input is a list of files provided on stdin.
    • Output is either to stdout, to a device or file. See man page.
    • Advantage over 'tar' is that it can both add and remove files to and from an archive.
    • In copy-pass mode, 'cpio' copies files from one directory tree to another

    Backup all php files in my home directory

    $ find ~ -depth -name '*.php' -print0 | cpio --null -o > my-phpfiles-cpio
    79812 blocks 
    
    /* Options used
     * '-depth'         tells find to process directory contents before the 
     *                  directory itself.
     * '-print0'        find generates null-terminate strings for file names.
     * '--null'         lets cpio read null-terminated string formats.  This
     *                  ensures correct handling of file names that have 
     *                  embedded blank or newline characters.
     * '-o | --create'  cpio copy-out mode. 
     */
    

    Make a copy of 'linuxref' directory tree in 'linuxref-pass' directory

    $ du -s linuxref
    7548    linuxref
    
    $ find linuxref/ -depth -print0 | cpio -p --null --make-directories \
    --preserve-modification-time linuxref-pass 
    13540 blocks
    
    $ du -s linuxref-pass/linuxref
    7548    linuxref-pass/linuxref 
    

    Create a file archive on a tape device

    # find /home -print |cpio -ocBv /dev/rmt0 
    
    /* Options used:
     * 'o'     reads from stdin and copies files to stdout – copy-out mode
     * 'c'     reads and writes header information in ASCII character form. 
               Improves portability
     * 'B'     performs block input and output using 512 bytes to a record
     * 'v'     verbose about operations
     */
    

    List file archive entries on a tape device

    # cpio -itcvB 
    
    /* Options used
     * 't'     list contents of archive
     */
    

    Retrieve a file from a tape drive

    # cpio -icvdBum file.name  
    
    /* Options used:
     * 'u'     copies unconditionally.  An older file will overwrite a newer one
     * 'm'     restore modification times on files
     */
    

    Copy '*.txt' files into an archive

    $ ls *.txt | cpio -o > txt-archive
    1 block
    
    $ cat txt-archive | cpio -itv
    -rw-rw-r--   1 mark     mark            3 Nov 15 23:13 bert.txt
    -rw-rw-r--   1 mark     mark           68 Nov 15 23:03 fred2.txt
    -rw-rw-r--   1 mark     mark           18 Nov 15 23:07 fred.txt
    1 block
    

    Extract all files matching 'fred*' from archive

    $ rm fred*
    $ ls
    bert.txt  txt-archive 
    
    $ cat txt-archive | cpio -idv "fred*"
    fred2.txt
    fred.txt
    1 block
    
    $ ls
    bert.txt fred2.txt  fred.txt  txt-archive
    
  • GNU version of the tar archiving utility
    /bin/tar
    • GNU version of the 'tar' archiving utility.
    • In contrast to 'cpio', 'tar' creates the directory hierarchy automatically.

    Create a gzip compressed archive with a label

    $ tar cvfz ~/archive-1.tgz -V "Docs directory" ~/Documents 
    

    Extract a gzip compressed archive

    $ tar xvfz nginx-0.7.65.tar.gz 
    

    Check/test the archive

    $ tar tvf ~/archive-1.tgz
    v--------- 0/0          0 2007-11-23 18:11:47 Docs directory--Volume Header--
    drwxr-xr-x mark/mark    0 2007-11-22 14:12:35 home/mark/Documents/
    drwxrwxr-x mark/mark    0 2007-11-16 18:48:03 home/mark/Documents/lpi102/ 
    

    Comparing archives and files using 'tar'

    $ tar -cvf htmls.tar ./*.php               (Archive the current directory)
    
    $ tar --diff -f htmls.tar --directory .    (Compare the archive with the current dir)
                                               (No difference as nothing has changed)
    $ touch ./lpi102-idx.php                   (Change modification timestamp on a file)
    
    $ tar --diff -f htmls.tar --directory .    (Compare again)
    ./lpi102-idx.php: Mod time differs         (Difference has been identified) 
    

    Backup/archive /home filesystem to tape

    # tar -cvf /dev/rmt0 /home 
    

    Backup/archive /home filesystem to a hard drive/removable media

    # tar -cvf /mnt/backup/2005-08-12.tar /home 
    

    Backup/archive /home filesystem to a compressed file archive

    # tar -cv /home | gzip - > /mnt/backup/2005-08-12.tgz 
    

    Locate a file on tape archive

    # tar -tvf /dev/rmt0 | grep file.name 
    

    Retrieve a particular file from a tape archive

    # tar -xvf /dev/rmt0 file.name 
    
  • Control magnetic tape drive operation
    /bin/mt

    Some common operations:

    Command Description
    eof, weof Write count EOF marks at current position.
    fsf Forward space count files. Tape is positioned on first block of next file.
    bsf Backward space count files. Tape is positioned on first block of next file.
    fsr Forward space count records.
    bsr Backward space count records.
    seek Seek to block number count.
    rewind Rewind the tape.
    offline, rewoffl Rewind the tape and, if applicable, unload the tape.
    status Print status information about the tape unit.
    retension Rewind the tape, then wind it to the end of the reel, then rewind it again.
    erase Erase the tape.


    Verify a tar archive written to tape - assume tape is rewound

    # tar -tf /dev/st0             (Auto rewinding SCSI tape device)
    # tar tf /dev/nst0             (Non rewinding SCSI tape device)
    # mt -f /dev/nst0 fsf 1        (Position tape head on the first block of next file)
    # tar tf /dev/nst0
    # mt -f /dev/st0 rewind 
    

    This ONLY shows that the backup media is readable. The data is not verified.

    If multiple archives on a single tape use a non-rewinding device enabling one to skip forward.

  • Common linux compression utilities
    /bin/zcat, /bin/gunzip, /bin/gzip
    • Compress or expand files. See man page for full details.
    • To create a single archive file with multiple members so that members can later be extracted independently, use an archiver such as 'tar' or 'zi'.

    Add multiple files to an archive

    $ gzip -c file1  > foo.gz
    $ gzip -c file2 >> foo.gz
    

    The 2nd. example appends a file to an existing archive.

    Increase an archives compression ratio

    Create an archive

    $ gzip -c dbview* > pics.gz
    

    Check it

    $ gzip -tv pics.gz                             
    pics.gz:  OK
    

    Get comression info

    $ gzip -lv pics.gz                            
    method  crc     date  time    compressed    uncompressed  ratio uncompressed_name
    defla 2ce22bc6 Jan 22 19:33      2890793           59914 -4724.8% pics
    

    Repack with best compression method

    $ gzip -cd pics.gz | gzip -9 > new.pics.gz
    

    See the result

    $ gzip -lv new.pics.gz 
    method  crc     date  time    compressed    uncompressed  ratio uncompressed_name
    defla a00af8a2 Jan 22 19:35      2888111         3103158   6.9% new.pics
    

    Can achieve better compression with

    $ cat file1 file2 | gzip > foo.gz 
    

    than with

    $ gzip -c file1 file2 > foo.gz 
    

    Decompress an archive

    $ gunzip -c foo
    $ gzip -dc pics 
    
  • A block-sorting file compressor
    /bin/bzip2, /bin/bunzip2
    • 'bzip2', 'bunzip2' and 'bzcat' are really the same program.
    • 'bzcat' decompresses files to stdout.
    • 'bzip2' compresses files in blocks, usually 900kB long. Each block is handled independently. If damage occurs to one or more blocks it may be possible to recover data from the undamaged blocks.
    • 'bzip2recover' recovers data from damaged 'bzip2' files. See man pages for full details on all of the above.
    • 'bzip2recover' searches for blocks in .bz2 files and writes each block out into its own .bz2 file. 'bzip2 -t' can then be used on each file to test its integrity. If ok the it can be decompressed.

    There are a couple of examples at the end of the tarball's section below.

  • Tape archives
    tarballs

    tarballs are compressed tar files, they can be unpacked using either two separate commands or a single command.

    Unpack a tarball

    $ gunzip source-file.tar.gz
    

    Or with two commands

    $ gzip -d source-file.tar.gz
    $ tar -xf source-file.tar 
    

    Unpack a tarball - single command examples

    $ tar -zxf source-file.tar.gz
    $ tar -zxf source-file-1.tar.gz source-file-2.tgz source-file-3.tar.Z
    $ tar -jxf source-file.tar.bz2
    
    $ bzip2 -dc source-file.tar.bz2 | tar -x
    $ bzip2 -dc source-file.tar.bz2 | tar -xf -
    
    $ zcat source-file.tar.gz | tar -x
    
    $ gunzip -c source-file.tar.gz | tar -xf - 
    

    Create a tarball - archive all files and sub-directories in ./www

    $ tar -cvf html-code.tar ./www/        (Results in file: html-code.tar)
    $ gzip html-code.tar                   (Results in file: html-code.tar.gz)
    $ bzip2 html-code.tar                  (Results in file: html-code.tar.bz2) 
    
  • Install from a source tarball
    • Source files come in compressed files - tarballs
    • tarballs are independent of a distribution's Package Manager.
    • Various file extensions exist - .tar.gz, tar.Z, .tgz, .taz, tar.bz2, ...
    Basic install overview
    Obtain the tarball
    Unpack the tarball
    Read the README files
    Run the configure script - if exists
    Run make
    Run make install

    Run any configure script (usually by running it in the build top directory)

    # ./configure 
    

    Build the application from source by running

    # make 
    

    Test the build by running (if the 'test' target exists)

    # make test 
    

    Install modules if modules are built (as root)

    # make modules_install 
    

    Install the application (as root)

    # make install 
    

    Clean up the build directory

    # make clean 
    

    Does not remove configuration, 'dot' files. Clean up the build directory after problems

    # make clean 
    # make mrproper
    

    This removes all built files to date including '.configure' files. Start build process from beginning.

  • Remote copy script example
    /bin/cp

    Copies to local media which includes NFS mounts.

    Backup script example using 'cp' (or just as easily 'scp')

    #!/bin/bash
    # File: backup-daily.sh
    # ++ Run this on a daily cron job ++
    
    # Could be part of an incremental backup schedule.
    # copy all the files in /home/ that have been modified in the lst day
    # -- first make sure the target directories exist
    for d in `find /home -type d` 
    do 
     mkdir -p /mnt/backup$d
    done
    
    # -- then copy all the recently modified files (one day)
    for f in `find /home -mtime -1`
    do 
     cp $f /mnt/backup$f
    done 
    
  • Secure copy of files between networked hosts
    /usr/bin/scp
    • Copies files between hosts on a network.
    • A replacement for 'rcp'. Uses 'ssh' for data transfer, uses the same authentication and provides the same security as 'ssh'.
    • Unlike 'rcp', 'scp' will ask for passwords or passphrases if they are needed for authentication.
    • Copies between two remote hosts are permitted.

    Recursively secure copy a remote directory to a local directory

    $ scp -r <user>@<remote-hostname>:<directory path> <local directory>
    

    You will be prompted once for the remote systems user password. Securely copy a file from one remote machine to another remote machine

    $ scp <user>@<remote-hostname>:<path/to/file> <user>@<remote-hostname>:<path/to/destination> 
    

    You will be prompted twice, once for each remote systems user password

  • Faster, flexible replacement for rcp
    /usr/bin/rsync

    Secure copy the firewall's logfile, using 'sshd' and 'blowfish cipher', to current directory

    # rsync -avPe 'ssh -c blowfish' firewall:/var/log/suspect.log* . 
    

    Copy file from local system to remote system

    # rsync -t myfile.tar 192.168.0.2:/backups/system1
    

    Copy file from remote system to local system

    # rsync 192.168.0.2:/backups/system1/myfile.tar . 
    
  • Incremental backup example
    /usr/bin/rsync
    • Provides fast incremental file transfer, often the best tool for automated remote backups.
    • It can optionally enforce two-way synchronisation.
    • It can also automatically remove locally deleted files from the remote backup.

    Example backup script

    #!/bin/sh
    # This script does personal backups to a rsync backup server. You will end up with a 7 day
    # rotating incremental backup. The incrementals will go into subdirs named after the day 
    # of the week, and the current full backup goes into a directory called "current"
    # directory to backup
    BDIR=/home/$USER
    
    # excludes file - this contains a wildcard pats of files to exclude
    EXCLUDES=$HOME/cron/excludes
    
    # the name of the backup machine
    BSERVER=owl
    
    # your password on the backup server
    export RSYNC_PASSWORD=XXXXXX
    BACKUPDIR=`date +%A`
    OPTS="--force --ignore-errors --delete-excluded --exclude-from=$EXCLUDES --delete --backup 
         --backup-dir=/$BACKUPDIR -a"
    export PATH=$PATH:/bin:/usr/bin:/usr/local/bin
    
    # the following line clears the last weeks incremental directory
    [-d $HOME/emptydir] || mkdir $HOME/emptydir
    rsync --delete -a $HOME/emptydir/ $BSERVER::$USER/$BACKUPDIR/
    rmdir $HOME/emptydir
    
    # now the actual transfer
    rsync $OPTS $BDIR $BSERVER::$USER/current