Helpful tips to speed up a Linux sofware RAID rebuild
Tip #1: /proc/sys/dev/raid/{speed_limit_max,speed_limit_min} kernel variables
The /proc/sys/dev/raid/speed_limit_min is config file that reflects the current "goal" rebuild speed for times when non-rebuild activity is current on an array. The speed is in Kibibytes per second (1 kibibyte = 210 bytes = 1024 bytes), and is a per-device rate, not a per-array rate . The default is 1000.
The /proc/sys/dev/raid/speed_limit_max is config file that reflects the current "goal" rebuild speed for times when no non-rebuild activity is current on an array. The default is 100,000.
To see current limits, enter:
sysctl dev.raid.speed_limit_min
sysctl dev.raid.speed_limit_max
NOTE: The following hacks are used for recovering Linux software raid, and to increase the speed of RAID rebuilds. Options are good for tweaking rebuilt process and may increase overall system load, high cpu and memory usage.
To increase speed you can increase these values. Here are the ones that work great on my machine:
echo 100000 > /proc/sys/dev/raid/speed_limit_min
echo 200000 > /proc/sys/dev/raid/speed_limit_max
These sets the min value to 10,000 K/sec and the max value to 20,000 K/sec. I have a quad core machine with 16 Gig of ram, and they seem to be about the best for me. Feel free to tinker with the values, and check which ones give you the best performance by running:
cat /proc/mdstat
Tip #2: Set read-ahead option
Set readahead (in 512-byte sectors) per raid device. The syntax is:
# blockdev --setra 65536 /dev/mdX
Where /dev/mdX is the device identifier for your Sofware RAID device. Below is an example of the command I run to set the read-ahead to 32 MIB:
blockdev --setra 65536 /dev/md127
Tip #3: Set stripe-cache_size for RAID5 or RAID 6
This is only available on RAID5 and RAID6 and boost sync performance by 3-6 times. It records the size (in pages per device) of the stripe cache which is used for synchronising all write operations to the array and all read operations if the array is degraded. The default is 256. Valid values are 17 to 32768. Increasing this number can increase performance in some situations, at some cost in system memory. Note, setting this value too high can result in an "out of memory" condition for the system. Use the following formula:
memory_consumed = system_page_size * nr_disks * stripe_cache_size
To set stripe_cache_size to 16 MiB for /dev/md127, type:
echo 16384 > /sys/block/md127/md/stripe_cache_size
To set stripe_cache_size to 32 MiB for /dev/md127, type:
echo 32768 > /sys/block/md127/md/stripe_cache_size
Tip #4: Disable NCQ on all disks
The following will disable NCQ on /dev/sda,/dev/sdb,..,/dev/sde using bash for loop
for i in sd[abcde]
do
echo 1 > /sys/block/$i/device/queue_depth
done
Tip #5: Bitmap Option
Bitmaps optimize rebuild time after a crash, or after removing and re-adding a device. Turn it on by typing the following command, making sure you change the device identifer to what ever your RAID is:
mdadm --grow --bitmap=internal /dev/md127
Once array rebuild or fully synced, disable bitmaps:
mdadm --grow --bitmap=none /dev/md127
After running those commands, my speeds increase dramatically. I went from 13184 K/sec to about 95108 K/sec! Hope that helps you save some time o your raid rebuilds.
How to test for Bash ShellShock
If you want to test each exploit individually without running the script above, feel free! They are listed below.
Exploit 1 (CVE-2014-6271)
There are a few different ways to test if your system is vulnerable to shellshock. Try running the following command in a shell.
env x='() { :;}; echo vulnerable' bash -c "echo this is a test"
If you see "vulnerable" you need to update bash. Otherwise, you should be good to go.
Exploit 2 (CVE-2014-7169)
Even after upgrading bash you may still be vulnerable to this exploit. Try running the following code.
env X='() { (shellshocker.net)=>\' bash -c "echo date"; cat echo; rm ./echo
If the above command outputs the current date (it may also show errors), you are still vulnerable.
Exploit 3 (???)
Here is another variation of the exploit. Please leave a comment below if you know the CVE of this exploit.
env X=' () { }; echo hello' bash -c 'date'
If the above command outputs "hello", you are vulnerable.
Exploit 4 (CVE-2014-7186)
bash -c 'true <<EOF <<EOF <<EOF <<EOF <<EOF <<EOF <<EOF <<EOF <<EOF <<EOF <<EOF <<EOF <<EOF <<EOF' || echo "CVE-2014-7186 vulnerable, redir_stack"
A vulnerable system will echo the text "CVE-2014-7186 vulnerable, redir_stack".
Exploit 5 (CVE-2014-7187)
(for x in {1..200} ; do echo "for x$x in ; do :"; done; for x in {1..200} ; do echo done ; done) | bash ||
echo "CVE-2014-7187 vulnerable, word_lineno"
A vulnerable system will echo the text "CVE-2014-7187 vulnerable, word_lineno".
Exploit 6 (CVE-2014-6278)
shellshocker='() { echo You are vulnerable; }' bash -c shellshocker
You shouldn't see "You are vulnerable", if you're patched you will see "bash: shellshocker: command not found"
Exploit 7 (CVE-2014-6277)
bash -c "f() { x() { _;}; x() { _;} <<a; }" 2>/dev/null || echo vulnerable
If the command outputs "vulnerable", you are vulnerable.
If you've tested your system, please leave a comment below. Don't forget to include your bash version and what OS you're running. Type bash --version for bash, and cat /etc/*release* for your OS.
---------------------------------------------------------------------------------
To check for the CVE-2014-6271 vulnerability
env x='() { :;}; echo vulnerable' bash -c "echo this is a test"
it should NOT echo back the word vulnerable.
To check for the CVE-2014-7169 vulnerability
(warning: if yours fails it will make or overwrite a file called /tmp/echo that you can delete after, and need to delete before testing again )
cd /tmp; env X='() { (a)=>\' bash -c "echo date"; cat echo
it should say the word date then complain with a message like cat: echo: No such file or directory. If instead it tells you what the current datetime is then your system is vulnerable.
To check for CVE-2014-7186
bash -c 'true <<EOF <<EOF <<EOF <<EOF <<EOF <<EOF <<EOF <<EOF <<EOF <<EOF <<EOF <<EOF <<EOF <<EOF'
|| echo "CVE-2014-7186 vulnerable, redir_stack"
it should NOT echo back the text CVE-2014-7186 vulnerable, redir_stack.
To check for CVE-2014-7187
(for x in
{1..200}
;
do echo "for x$x in ; do :";
done;
for x in
{1..200}
;
do echo done
;
done)
| bash || echo "CVE-2014-7187 vulnerable, word_lineno"
it should NOT echo back the text CVE-2014-7187 vulnerable, word_lineno.
Create Apple Time Capsule for Time Machine backups using Centos 6.5
I have been looking for the parts required to put together so that I can backup all the macs on a linux server hosted here in my lab. What follows is the my “yum ready” instructions.
- Start With Centos 6.5 Minimal Distribution
- We need to create the file space where the Time Machine is going to save the files. I chose /home/tony/timemachine
- Create the user:
adduser lynn #create user
passwd lynn #set password - Create the path:
mkdir -p /home/lynn/timemachine - Set ownership:
chown -R lynn:lynn /home/lynn
- Create the user:
- Install wget
yum -y wget - Install the EPEL Repository
wget http://www.mirrorservice.org/sites/dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpmyum localinstall epel-release-6-8.noarch.rpm - Use Yum to install the required packages:
yum -y install netatalk avahi dbus nss-mdns - Set services to start at boot:
chkconfig netatalk on
chkconfig messagebus on
chkconfig avahi-daemon on - Now configure netatalk by posting this line at the very bottom of the file:
- -transall -uamlist uams_randnum.so,uams_dhx.so,uams_dhx2.so -nosavepassword -advertise_ssh(Yes, just like that) - Edit /etc/netatalk/AppleVolumes.default and create the network shares. You’ll need one for each laptop you want to backup:
/home/lynn/timemachine allow:lynn options:usedots,upriv,tm dperm:0775 fperm:0660 cnidscheme:dbd - Next edit /etc/nsswitch.conf
hosts: files mdns4_minimal dns mdns mdns4 -
Create /etc/avahi/services/afpd.service with the contents:
<?xml version=”1.0″ standalone=’no’?>
<!DOCTYPE service-group SYSTEM “avahi-service.dtd”>
<service-group>
<name replace-wildcards=”yes”>%h</name>
<service>
<type>_afpovertcp._tcp</type>
<port>548</port>
</service>
<service>
<type>_device-info._tcp</type>
<port>0</port>
<txt-record>model=TimeCapsule</txt-record>
</service>
</service-group> - Remove the ssh service from Avahi
mv /etc/avahi/services/ssh.service /etc/avahi/services/ssh.service.disabled -
IPTABLES are running by default, so add the following to open up some ports and then save the iptables config:
iptables -I INPUT -p udp –dport 548 -j ACCEPT
iptables -I INPUT -p tcp –dport 548 -j ACCEPT
iptables -I INPUT -p tcp –dport 5353 -j ACCEPT
iptables -I INPUT -p udp –dport 5353 -j ACCEPT
iptables -I INPUT -p udp –dport 5354 -j ACCEPT
iptables -I INPUT -p tcp –dport 5354 -j ACCEPTservice iptables save -
Start the required services:
service avahi-daemon start service messagebus start service netatalk start
When you open Time Machine on your mac (mine is a MacBook Air running Mavericks), click the locked padlock to allow changes, which also enables the add/remove a backup disk. Click Add/remove and you should see your network Time Machine disk.
Avoiding CPU Speed Scaling – Running CPU At Full Speed
If you have a server on collocation with the super-fast CPU, that doesn't mean that you get it running at full power.
Here is a preamble. When doing inventory of Linux servers, that I'm administering, I've found that some of them are running on lower CPU speed, then they could. This can be easily checked with this command:
#grep -E '^model name|^cpu MHz' /proc/cpuinfo
What you can see:
model name : Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz
cpu MHz : 1600.000
...
or
model name : Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz
cpu MHz : 1596.000
...
or even
model name : Intel(R) Core(TM)2 CPU 4400 @ 2.00GHz
cpu MHz : 1000.000
model name : Intel(R) Core(TM)2 CPU 4400 @ 2.00GHz
cpu MHz : 600.000
Oops, we are paying for 2 Core 2Ghz CPU that runs on 600Mhz on one core and 1000Mhz on another!!!
There will be other lines for all the CPUs/cores/threads, probably with the same values.
This feature is nice, if we are running workstation, but what I've noticed, we do have the same CPU throttling on Ubuntu Server 10.04 builds and on CentOS 5.3, 5.4 and 5.5 builds (thus on RedHat too).
After hours of digging google, I've found that:
- this problem is very common
- there are several bug reports about this issue
- this is not BIOS settings problem, because on dual boot systems, CPU runs at full speed on Windows
- there are no 100% working solutions or they are too difficult to find
- this is not a bug, but a 'feature' of the new kernels, it is implemented differently on 2.6.18 (CentOS) and 2.6.32 (Ubuntu).
Here is a tip how to disable it on running system:
1) Check that 'kondemand' thread is running, run as root: "pgrep -lf ondemand"
the output should be like:
# uname -a
Linux boston07 2.6.18-164.6.1.el5 #1 SMP Tue Nov 3 16:18:27 EST 2009 i686 i686 i386 GNU/Linux
# pgrep -lf ondemand
1444 kondemand/0
1445 kondemand/1
2) Check that current cpu speed differs from the maximum:
# grep -E '^model name|^cpu MHz' /proc/cpuinfo
model name : Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz
cpu MHz : 1596.000
model name : Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz
cpu MHz : 1596.000
3) Change CPU governor from 'ondemand' to 'performance' for all CPUs/cores, run as root:
for CPUFREQ in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do [ -f $CPUFREQ ] || continue; echo -n performance > $CPUFREQ; done
4) Check that your changes have been applied:
# grep -E '^model name|^cpu MHz' /proc/cpuinfo
model name : Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz
cpu MHz : 2394.000
model name : Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz
cpu MHz : 2394.000
5) If you are running 'cpuspeed', 'cpufreqd', 'powerd' or other daemons, that can control CPU stepping, just stop them, if you really need to run your system on 100% of the CPU speed.
On CentOS:
# service cpuspeed stop
6) On Linux 2.6.32 (On RedHat 6, and Oracle Unbreakable Linux 6) remove CPU scaling kernel modules:
# lsmod | grep ondemand
cpufreq_ondemand 8764 0
freq_table 3751 2 cpufreq_ondemand,acpi_cpufreq
# rmmod cpufreq_ondemand acpi_cpufreq freq_table
Ensure that no 'kondemand' kernel threads are running:
# pgrep -lf kondemand
#
7) To make changes permanent (on reboot):
- On Ubuntu, modify /etc/init.d/ondemand script:
change this
echo -n ondemand > $CPUFREQ
to this:
echo -n performance > $CPUFREQ
OR ALTERNATIVELY just remove all references to ondemand from /etc/rc?.d/
rm -f /etc/rc?.d/S99ondemand
- On CentOS, just create a new script /etc/init.d/ondemand:
#! /bin/bash
#
# ondemand sets cpu govermor
#
# chkconfig: 2345 10 90
#
# description: Set the CPU Frequency Scaling governor to "performance"
#
### BEGIN INIT INFO
# Provides: $ondemand
### END INIT INFO
PATH=/sbin:/usr/sbin:/bin:/usr/bin
case "$1" in
start)
for CPUFREQ in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
do
[ -f $CPUFREQ ] || continue
echo -n performance > $CPUFREQ
done
;;
restart|reload|force-reload)
echo "Error: argument '$1' not supported" >&2
exit 3
;;
stop)
;;
*)
echo "Usage: $0 start|stop" >&2
exit 3
;;
esac
then enable it:
chmod +x /etc/init.d/ondemand
chkconfig --add ondemand
service ondemand restart
I'm using 'ondemand' name of the script, this may be a little bit misleading (because really it is a 'performance'), but you may change it.
Reference here: http://www.servernoobs.com/avoiding-cpu-speed-scaling-in-modern-linux-distributions-running-cpu-at-full-speed-tips/?doing_wp_cron=1390110978.2860989570617675781250#
Install Handbrake on Fedora 18
Unfortunately, there are no repos that contain Handbrake pre-built for Fedora. That's not always a bad thing, it just means we need to download and compile it on our machines. It's not as hard as it sounds! Here's how I did it:
We need to get any and all dependencies for Handbrake. I found these by checking the readme and a few other places on the web:
Lets get the development stuff first:
sudo yum groupinstall "Development Tools"
Now lets get all the other dependencies for Handbrake:
sudo yum install libass-devel libsamplerate-devel libogg-devel libtheora-devel libvorbis-devel yasm zlib-devel bzip2-devel fribidi-devel dbus-glib-devel libgudev1-devel webkitgtk-devel libnotify-devel gstreamer-devel gstreamer-plugins-base-devel libsamplerate-devel svn
Once we have all those installed, we can download a copy of Handbrake using SVN:
svn checkout svn://svn.handbrake.fr/HandBrake/trunk hb-trunk
OK. Now we have everything we need to begin compiling lets change to the proper directory and compile!
cd hb-trunk
./configure
cd build
gmake
make install
That's all there is to it! You should now have Handbrake installed and ready for use. Please post below if you come across any problems.
FreeNX session timeout
With the launch of the newest NoMachine Windows NX client, you may have problems when trying to connect to a linux server that is running FreeNX. Occasionally the session will timeout upon connecting with very obscure error messages. Add this to you node.conf and restart FreeNX.
Try the following:
Open your node.conf file for editing:
vi /etc/nxserver/node.conf
Add the following at the bottom of the file:
ENABLE_2_0_0_BACKEND="1"
Restart your FreeNX server so the changes take effect:
service freenx-server restart
OR
/etc/init.d/freenx-server restart
Try connecting from you Windows machine 🙂
Add, Remove, and adjust disks and file systems in AIX 6.1
You can use the built in tool called SMITTY to resize files systems in AIX. Supply SMITTY with the FS argument to work with the file systems. See instructions below for a walk through of adjusting /opt
Launch Smitty by typing: smitty fs
Arrow down to the third option "Add / Change / Show / Delete File Systems" and press enter.
Select "Enhanced Journaled File System" and press enter.
Arrow down to the third option labled "Change / Show Characteristics of an Enhanced Journaled File System" and press enter.
This will bring up a selection box of the available partitions and file systems that can be edited. Use the arrow keys to select the one you want to modify, and press enter.
Now you will see all available options for editing. To adjust the size of the file system, arrow down to the line labeled "Number of Units"
In this example, I have adjusted the file system to 12582912 units which is equivalent to 6 GB.
Once you have entered the desired size, press the enter key to accept.
Give the system a few seconds to adjust the size, and if successful you will see a message similar to the one below.
Once you see this confirmation, press the F10 to exit smitty.
