The Daily Admin

Tip #1: /proc/sys/dev/raid/{speed_limit_max,speed_limit_min} kernel variables

The /proc/sys/dev/raid/speed_limit_min is config file that reflects the current "goal" rebuild speed for times when non-rebuild activity is current on an array. The speed is in Kibibytes per second (1 kibibyte = 2¹⁰ bytes = 1024 bytes), and is a per-device rate, not a per-array rate . The default is 1000.

The /proc/sys/dev/raid/speed_limit_max is config file that reflects the current "goal" rebuild speed for times when no non-rebuild activity is current on an array. The default is 100,000.

To see current limits, enter:

sysctl dev.raid.speed_limit_min
sysctl dev.raid.speed_limit_max

NOTE: The following hacks are used for recovering Linux software raid, and to increase the speed of RAID rebuilds. Options are good for tweaking rebuilt process and may increase overall system load, high cpu and memory usage.

To increase speed you can increase these values. Here are the ones that work great on my machine:

 
echo 100000 > /proc/sys/dev/raid/speed_limit_min
echo 200000 > /proc/sys/dev/raid/speed_limit_max

These sets the min value to 10,000 K/sec and the max value to 20,000 K/sec.  I have a quad core machine with 16 Gig of ram, and they seem to be about the best for me.  Feel free to tinker with the values, and check which ones give you the best performance by running:

cat /proc/mdstat

Tip #2: Set read-ahead option

Set readahead (in 512-byte sectors) per raid device. The syntax is:
# blockdev --setra 65536 /dev/mdX


Where /dev/mdX is the device identifier for your Sofware RAID device.  Below is an example of the command I run to  set the read-ahead to 32 MIB:

blockdev --setra 65536 /dev/md127

Tip #3: Set stripe-cache_size for RAID5 or RAID 6

This is only available on RAID5 and RAID6 and boost sync performance by 3-6 times. It records the size (in pages per device) of the stripe cache which is used for synchronising all write operations to the array and all read operations if the array is degraded. The default is 256. Valid values are 17 to 32768. Increasing this number can increase performance in some situations, at some cost in system memory. Note, setting this value too high can result in an "out of memory" condition for the system. Use the following formula:

memory_consumed = system_page_size * nr_disks * stripe_cache_size

To set stripe_cache_size to 16 MiB for /dev/md127, type:

echo 16384 > /sys/block/md127/md/stripe_cache_size

To set stripe_cache_size to 32 MiB for /dev/md127, type:

echo 32768 > /sys/block/md127/md/stripe_cache_size

Tip #4: Disable NCQ on all disks

The following will disable NCQ on /dev/sda,/dev/sdb,..,/dev/sde using bash for loop

 for i in sd[abcde]
do
  echo 1 > /sys/block/$i/device/queue_depth
done

Tip #5: Bitmap Option

Bitmaps optimize rebuild time after a crash, or after removing and re-adding a device. Turn it on by typing the following command, making sure you change the device identifer to what ever your RAID is:

mdadm --grow --bitmap=internal /dev/md127

Once array rebuild or fully synced, disable bitmaps:

mdadm --grow --bitmap=none /dev/md127

After running those commands, my speeds increase dramatically.  I went from 13184 K/sec to about 95108 K/sec!  Hope that helps you save some time o your raid rebuilds.

This entry was posted on Sunday, October 5th, 2014 at 10:54 pm by CubedRoot and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.