Top

A Linux User Reference

Search tips
  • search ignores words that are less than 4 characters in length
  • searches are case insensitve
  • if a search does not return anything try it in Boolean mode then Query expansion mode by checking the appropriate radio button e.g. searching for 'cron' in just the Administration category returns nothing - presumably because the 50% threshold is reached. Boolean mode ignores this threshold so a search for 'cron' returns several hits
  • in Boolean mode preceding a word with a '+' means the result must include that word, a '-' means it must not
  • in Boolean mode '+crontab -anacron' means match articles about crontab that DO NOT mention anacron
  • to match a phrase e.g. 'manage system' check the Boolean mode radio button and enclose the phrase in quotes "some phrase ..."
  • in Query expansion mode the search context is expanded beyond the keywords you entered - relevancy of hits may well be degraded

HARDWARE

RAID

  • RAID management utilities, RAID levels

    There were at least a couple of ways of implementing Linux Software RAID -

    raidtools

    Utilities to support 'old-style' RAID disks

    raidtools2

    Utilities to support 'new-style' RAID disks

    mdadm

    Newer tool which can usually operate without the need for a separate configuration file. In most distributions it is supplanting raidtools/raidtools2. Can be used to create, manage, and monitor MD devices

    Most common raid levels

    RAID-0    Disk striping
    RAID-1    Mirroring
    RAID-4    Striping with parity. Parity information on a single disk
    RAID-5    Striping with parity.  Parity information distributed across disks
    
  • Raidtools/2 configuration file
    /etc/raidtab

    the configuration file consists of

    • multiple sections, one for each md device which is being configured.
    • each section begins with the raiddev keyword and consists of a list of keyword value pairs.
    • the order of items in the file is important, this list is not exhaustive.

    File format

    <option> white space(s) <value>
    
    Option Value
    raiddev Virtual disk partition (/dev/md?), the device that mkfs and fsck work with, is mounted in the same way as an actual hard disk partition
    raid-disk Underlying partitions used to build a RAID. Marked as partition type 0xFD
    spare-disk Spare disks (typically one) that are brought online when a RAID disk fails
    nr-raid-disks Number of raid devices in the array, there should be count raid-disk entries later in the file
    nr-spare-disks Number of spare devices in the array; there should be spare-disk entries later in the file. Spare disks may only be used with RAID4 and RAID5
    persistent-superblock Newly created RAID arrays should use a persistent superblock, can be 0 (no) or 1 (yes). It is a small disk area allocated at the end of each RAID device which helps the kernel to safely detect RAID devices even if disks have been moved between SCSI controllers
    parity-algorithm Use with RAID5:
    'left-asymmetric', 'right-asymmetric', 'left-symmetric', or 'right-symmetric'
    'left-symmetric' is the one that offers maximum performance on typical disks with rotating platters
    chunk-size The stripe size in kilobytes
    raid-disk Position in the raid array at which the most recently defined device is inserted
    spare-disk Position in the spare disk array at which the most recently defined device is inserted
  • RAID-0 configuration example
    /etc/raidtab

    Defines a RAID-0 virtual device called '/dev/md0'.

    Disk striping

    raiddev                 /dev/md0
    raid-level              0
    nr-raid-disks           2
    nr-spare-disks          0
    chunk-size              32
    persistent-superblock   1
    device                  /dev/sda2
    raid-disk               0
    device                  /dev/sdb2 
    raid-disk               1
    

    The first 32 KB chunk of '/dev/md0' is allocated to '/dev/sda2', the next 32 KB go on '/dev/sdb2', the third chunk is on '/dev/sda2', ...

    Once a RAID device has been configured it needs to be created and mounted.

  • RAID-1 configuration example
    /etc/raidtab
    • data written to '/dev/md0' will be saved on both '/dev/sdb6' and '/dev/sdc5'
    • '/dev/sdd5' is configured as a hot spare

    Disk mirroring

    raiddev                 /dev/md0
    raid-level              1
    nr-raid-disks           2
    nr-spare-disks          1
    persistent-superblock   1
    device                  /dev/sdb6
    raid-disk               0
    device                  /dev/sdc5
    raid-disk               1
    device                  /dev/sdd5
    spare-disk              0
    

    In the event '/dev/sdb6' or '/dev/sdc5' malfunctions, '/dev/sdd5' will be populated with data and brought online as a replacement.

    Once a RAID device has been configured it needs to be created and mounted.

  • RAID-5 configuration example
    /etc/raidtab
    • RAID-5 requires at least three drives
    • it can survive a single drive failure

    When a drive fails it drops into degraded mode which can dramatically impact I/O performance until a spare-disk can be brought online and repopulated.

    Striping with parity, parity information distributed across disks

    raiddev                 /dev/md0
    raid-level              5
    nr-raid-disks           7
    nr-spare-disks          0
    persistent-superblock   1
    parity-algorithm        left-symmetric
    chunk-size              32
    device                  /dev/sda3
    raid-disk               0
    device                  /dev/sdb1
    raid-disk               1
    device                  /dev/sdc1
    raid-disk               2
    device                  /dev/sdd1
    raid-disk               3
    device                  /dev/sde1
    raid-disk               4
    device                  /dev/sdf1
    raid-disk               5
    device                  /dev/sdg1
    raid-disk               6
    

    No spare disk configured in this setup - see RAID-1 example for spare-disk setup

    Once a RAID device has been configured it needs to be created and mounted.

    RAID-5 performance is greatly enhanced by providing the filesystem with correct stride value.

    For example, if the chunk size is 32 KB, it means that 32 KB of consecutive data will reside on one disk. If an ext2 filesystem has 4 KB block size then there will be eight filesystem blocks in one array chunk.

    Set the stride value

    # mke2fs -b 4096 -R stride=8 /dev/md0
    # mount /dev/md0 /home
    
  • Create a raidtools/2 raid device
    /sbin/mkraid

    Creates a RAID device - /dev/md?.

    mkraid [options] raiddevice
    
    Options:
     -c conf_file               Use an alternate file to default /etc/raidtab
     -f | --force               Initialise devices even if they appear to have data on them
     -o | --upgrade             Upgrade older arrays to current kernel's RAID version
     -V | --version
    

    Make an array

    # mkraid /dev/md0
    

    'mkraid' requires '/etc/raidtab' (default conf file)

  • Start one or more raidtools/2 md devices
    /sbin/raidstart
    raidstart [options] raiddevice
    
     -a | --all                 Apply to all configs in configuration file options:
     -c conf_file               Use an alternate file to default /etc/raidtab
     -h | --help
     -V | --version
    
    persistent-superblock

    enabling the kernel's persistent-superblock feature allows the kernel to start the RAID automatically at boot time.

    new-style RAID uses the persistent superblock and is supported in 2.4 and 2.6 kernels.

  • Stop one or more raidtools/2 md devices
    /sbin/raidstop
    raidstop [options] raiddevice
    
    Options:
     -a | --all                 Apply to all configs in configuaration file
     -c conf_file               Use an alternate file to default /etc/raidtab
     -h | --help
     -V | --version
    
  • Software RAID management application
    mdadm

    On Debian/Ubuntu requires 'initramfs-tools' and 'mdadm' packages along with the following kernel modules:

    md, linear, multipath, raid0, raid1, raid5, raid6, raid10
    
    • the configuration file */etc/mdadm/mdadm.conf~ does NOT require editing
    • place holders can be used for non-existent disks/devices.
  • mdadm mirroring (RAID-1) example
    • two disks will be used in the array - '/dev/sda' and '/dev/sdb'
    • 'dev/sda' is currently in use
    • /boot on '/dev/sda1' will be mirrored to '/dev/sdb1'

    ** Please note that the software is forever being upgraded and the command syntax has no doubt changed by now however the overall approach remains the same.

    Step (1)

    Need to ensure that there are no 'left-overs' of any previous RAID configurations on the disk(s) that will be used in the array. /proc/mdstat - gives informations about md devices status.

    Check for previous RAID installations, current status of any arrays - /proc/mdstat

     # cat /proc/mdstat
     Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
     unused devices: <none>
    

    The system has no RAID arrays configured.

    Zero the disk partition - for each partition to be used in the array (/dev/sdb)

     # mdadm --zero-superblock /dev/sdb1
     mdadm: Unrecognised md component device - /dev/sdb1
    

    This error is displayed if there are NO remains from a previous RAID installation. If there are remains then the partition superblock will be zeroed.

    Step (2)

    Create a raid device

    Use a place holder

      # mdadm --create /dev/md0 --level=1 --raid-disks=2 missing /dev/sdb1
    

    In this scenario '/dev/sda' is in use aka. missing.

    Check the status of the newly configured RAID-1 array

      # cat /proc/mdstat
      Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
      md0 : active raid1 sdb1[1]
             144448 blocks [2/1] [_U]
      unused devices: <none>
    

    [_U] or [U_] means that the array is degraded. This is to be expected as a place holder was used for '/dev/sda'.

    [UU] means the array is ok.

    Step (3)

    System is now aware of new array - md0. Create filesystems on the RAID and configure for boot.

    Create fs and update 'mdadm.conf

      # mkfs.ext2 /dev/md0
      # cp /etc/mdadm/mdadm.conf /etc/mdadm/mdadm.conf_orig
      # mdadm --examine --scan >> /etc/mdadm/mdadm.conf         (updates /etc/mdadm/mdadm.conf)
    
    

    Reflect changes in /etc/fstab

      .....
      /dev/md0      /boot    ext3    defaults     0  2
      .....
    
    

    Reflect changes in boot file - /boot/grub/menu.lst

     .....
     title           Debian GNU/Linux, kernel 2.6.18-4-486 RAID (hd1)
     root            (hd1,0)
     kernel          /vmlinuz-2.6.18-4-486 root=/dev/md0 ro
     initrd          /initrd.img-2.6.18-4-486
     savedefault
     title           Debian GNU/Linux, kernel 2.6.18-4-486 RAID (hd0)
     root            (hd0,0)
     kernel          /vmlinuz-2.6.18-4-486 root=/dev/md0 ro
     initrd          /initrd.img-2.6.18-4-486
     savedefault
    
     # kopt=root=/dev/md0 ro
     .....
    
    

    GRUB2

    **This is based on researched bits and pieces - I have not configured it in the real world as with the GRUBv1 example above.

    Add 'insmod mdraid' to allow volume native addressing e.g. '/dev/md0' -> (md0), 1st. partition '/dev/md01' -> (md0,1)

    Add a grub2 custom menuentry - /etc/grub.d/40_custom

      .....
      menuentry "Ubuntu mirror" {
              insmod ext2
              insmod mdraid
              set root='(md0)'
              linux /boot/vmlinuz-3.8.0-33-generic root=UUID=<uuid-xxxx> ro
              initrd /boot/initrd.img-3.8.0-33-generic
      }
    
    

    Install grub to both disks

      # grub-install --recheck --debug /dev/sda
      # grub-install --recheck --debug /dev/sdb
    

    '--debug' is optional

    Step (4)

    Update ramdisk

      # update-initramfs -u
    
    
  • Man page examples
    more current

    Is device an array or part of one

    mdadm --query /dev/name-of-device
    

    Outputs brief information about the device.

    Assemble and start all arrays listed in the default config file

    mdadm --assemble --scan
    

    Typically, command will go in a system startup file.

    Shut down all arrays that can be shut down

    mdadm --stop --scan
    

    Typically, command will go in a system shutdown script.

    Monitor all arrays, polling every 2 minutes

    mdadm --follow --scan --delay=120
    

    Does so, if (and only if) there is an Email address or program given in the standard config file,.

    Create /dev/md0 as a RAID1 array consisting of /dev/hda1 and /dev/hdc1

    mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/hd[ac]1
    

    Create a prototype config file

    echo 'DEVICE /dev/hd*[0-9] /dev/sd*[0-9]' > mdadm.conf
    mdadm --detail --scan >> mdadm.conf
    

    Populates config file with currently active arrays (from partitions of IDE or SCSI drives). This file should be reviewed before being used as it may contain unwanted detail.

    Find arrays that COULD be assembled from existing disks

    echo 'DEVICE /dev/hd[a-z] /dev/sd*[a-z]' > mdadm.conf
    mdadm --examine --scan --config=mdadm.conf >> mdadm.conf
    

    Stores what it finds in config file format - review and edit id choose to use as an actual config file.

    Print a brief listing of RAID superblocks

    mdadm --examine --brief --scan --config=partitions
    mdadm -Ebsc partitions
    

    Creates a list of devices by reading /proc/partitions.

    Fail component devices and remove from array

    mdadm /dev/md4 --fail detached --remove detached
    

    This will take place on all devices that make up /dev/md4.

    Get some help

    mdadm --create --help
    mdadm --config --help
    mdadm --help
    
  • Disk failure replacement example

    Scenario: RAID-1, /dev/md0 (/dev/sda, and /dev/sdb), /dev/sdb fails

    (1) Fail and remove disk

    Simulate a hard drive (soft) failure and remove from array

    # mdadm --manage /dev/md0 --fail /dev/sdb1
    # mdadm --manage /dev/md0 --remove /dev/sdb1
    

    Remove drive from system.

    (2) Add replacement disk to system

    Copy partition table from working disk to the new one - non GPT (old)

    # sfdisk -d /dev/sda | sfdisk /dev/sdb           (Use '--force' if an error occurs)
    

    Copy partition table from working disk to the new one - GPT

    $ apt-cache search gdisk
    gdisk - GPT fdisk text-mode partitioning tool
    
    sgdisk --replicate=/dev/sdb /dev/sda
                                                (or)
    sgdisk -R /dev/sdb /dev/sda
    

    'sgdisk' is part of the 'gdisk' tool.

    (3) Add replacement disk to the RAID array**

    If a new disk then this next step will not be required Remove any remains of a previous RAID array from the replacement disk

    # mdadm --zero-superblock /dev/sdb1
    

    For each partition on the disk.

    Add /dev/sdb1 to the RAID array

    # mdadm -a /dev/md0 /dev/sdb1               (maybe dated)
                                                (or)
    # mdadm --manage /dev/md0 --add /dev/sdb1
    
    (4) Install boot loader to replacement disk

    Only necessary if the disk being mirrored is a boot disk

    Install GRUBv1 in the MBR of the replacement disk

    # grub
    grub> root (hd1,0)
    grub> setup (hd1)
    grub> quit
    

    Install GRUBv2 in the MBR of the replacement disk

    $ sudo grub-install --modules="raid" /dev/sdb
    

    If a RAID level where more than one disk can become the boot disk then need to install on all the disks.

    '--modules="raid"' is required, GRUB will be unable to boot from a fail-over disk if that disk has not had the raid module installed on it. (Apparently loops without ever getting to the grub boot prompt)

    Update initramfs

    $ sudo update-initramfs -u