Making RAID55 with Encryption + more

MAKE RAID 55 OUT OF 12 DISKS + luks encryptions + progress bars + troubleshooting Raid + more

#############################################################################################

ABSTRACT CONCEPT: Going to make four raid 5s and those will act as “disks”/members to a big raid 5. For one test I will lay ext3 on it, and then for another I will make an encrypted system. I will run some disk speeds etc. All this on my Virtual Machine. NOTE THIS WILL VERY VERY SLOW DUE TO THE LAYERS AND LAYERS OF ENCRYPTION. RAID 55 IN REALWORLD IS PROBABLY VERY UNFEASABLE AND NOT USEABLE – HOWEVER I WROTE THIS TO SHOW YOU IT EXISTS AND HOW TO MAKE IT.

==TOPICS YOU WILL LEARN:==

* make more loopback devices then are given by the system

* Making raids

* Making hybrid raids

* make a member of raid not a spare and making auto-read-only go away – without hurting data – if there was any on it at least, in this example there was no data but if there was i show you how to make a spare active without hurting data and with that randomly the auto-read-only status went away

* progress bars with dd by using “pv” and “bar”

* filesystem checks with ext2,3,4

* doing read and write speed test

* Encrypting a device block level

* Changing password/passphrase on encrypted device

==MAP OF THE 2 PATHS ILL TAKE:==

* You can look at these as layers too from physical on the left to logical on the right

Disks->Loopback Image->4 Raids5 ->Hybrid Raid5->ext3

Disks->Loopback Image->4 Raids5 ->Hybrid Raid5->Encryption Block Level->ext4

==SIDE NOTE:==

All of the commands that you should type are prefixed with # Thats if you want to follow this guide. I dont know why you would its just a hypothetical turned to a possibility turned to a means turned to an I did it. I guess you can get the I did it as well by following this. Also I hope I didnt leave out any of the little commands – like “cd” ing into the right directory – If I did forgive me and let your logic figure it out.

################

===THE SYSTEM===

################

Here are the stats of my machine.

# cat /proc/cpuinfo

processor : 0

vendor_id : GenuineIntel

cpu family : 6

model : 42

model name : Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz

stepping : 7

cpu MHz : 2975.309

cache size : 6144 KB

fdiv_bug : no

hlt_bug : no

f00f_bug : no

coma_bug : no

fpu : yes

fpu_exception : yes

cpuid level : 5

wp : yes

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc up pni monitor ssse3 lahf_lm

bogomips : 5950.61

clflush size : 64

cache_alignment : 64

address sizes : 36 bits physical, 48 bits virtual

power management:

Not much memory but no worries

# cat /proc/meminfo

MemTotal: 384332 kB

I have some Swap Space

# swapon -s

Filename Type Size Used Priority

/dev/sda5 partition 392188 48792 -1

/dev/sdb1 partition 979928 0 -2

I have 2 disk sda and sdb. SDA is the root its small, its partitioned into I cant remember how many partitions. But here is the output – Its 3 partitions okay.. 🙂

cat /proc/partitions

major minor #blocks name

8 0 8388608 sda

8 1 7993344 sda1

8 2 1 sda2

8 5 392192 sda5

8 16 8388608 sdb

8 17 979933 sdb1

8 18 7405965 sdb2

# df -h

Filesystem Size Used Avail Use% Mounted on

/dev/sda1 7.6G 4.7G 2.5G 66% /

tmpfs 188M 0 188M 0% /lib/init/rw

udev 183M 216K 183M 1% /dev

tmpfs 188M 0 188M 0% /dev/shm

/dev/sdb2 7.1G 4.1G 2.4G 64% /vol

I will make the raid 5s out of images that will lay in disk

#########################

===BUILDING THE RAID55===

#########################

===WHAT IS THIS?===

– we are going to make it out of disk images

– make 12 image files filled with zero bits for 300 MB – that will emulate a 300 MB disk

– make a loopback device out of them with losetup

– since linux only comes with 8 loop backs we need a few more

– Then we make the mdadm commands from loop devices to make 4 raid 5s

– then we make a raid 5 out of raids (That are also raid 5). These raid out of raids are called hybrid raids.

===INTERPRETATION:===

RAID 55

RAID xy

Y is the first raid

X is the raid made out of the raids

===THE STEPS===

We are going to make 12 mini disk emulations with empty image files – like isos and use loopbacks to make them considered devices – (I heard though mdadm doesnt need them to be loops – I could of just fed the image files to mdadm – oh well – good thought Ill try that another time but wont write an article on it). Loopbacks are used to mount cdrom images etc. And also my favorite use is testing filesystems, big volumes with lvm, raid with mdadm, encryption with cryptosetup, etc – so basically testing block layer stuff and above.

We need 5 more loopback devices but linux comes with only 8 (0 thru 7) i need to have loopback devices to go up to 12

NOTE we are not using loop0 because I wanted the count to start at 1.

First see what loopbacks you need to create in my case it only went up to 7 so i needed to make 8,9,10,11, and 12:

# ls -l /dev/loop*

MAKE THEM:

# mknod -m660 /dev/loop8 b 7 8

# mknod -m660 /dev/loop9 b 7 9

# mknod -m660 /dev/loop10 b 7 10

# mknod -m660 /dev/loop11 b 7 11

# mknod -m660 /dev/loop12 b 7 12

THE HAVE THE WRONG OWNERSHIP OF root.root (user.group) SO CHANGE IT TO MATCH THE ORIGINAL LOOPBACK WHICH IS root.disk:

# chown root.disk /dev/loop8

# chown root.disk /dev/loop9

# chown root.disk /dev/loop10

# chown root.disk /dev/loop11

===MAKE LOOPBACK DEVICES===

First I will make the loopback devices with dd its nice to get a progress bar – the program for that is bar. Have to download the .deb for it:

# cd /tmp

# apt-get update

# apt-get wget

# wget http://downloads.sourceforge.net/clpbar/bar_1.10.9_i386.deb?modtime=1183036896&big_mirror=0

Get the file name with “ls” note my filename literally was like this:

# dpkg -i bar_1.10.9_i386.deb?modtime=1183036896&big_mirror=0

# mkdir -p /vol/raid

# cd /vol/raid

Im going to make a for loop that makes the image files and makes them into loopback devices with progress bars so we can see how far we are with the process:

# for i in 1 2 3 4 5 6 7 8 9 10 11 12; do echo “===ab$i===” && dd if=/dev/zero bs=1M count=300 | bar -s 300m -of ab$i && losetup /dev/loop$i ab$i;done;

TRANSLATION OF THAT INTO ENGLISH:

-NOTE DONT DO THESE COMMANDS THATS WHY I DIDNT INCLUDE A # WITH IT, ITS JUST AN EXPLANATION OF THE FOR LOOP IN ENGLISH-

This is what happens for the first iteration of that loop. After this it repeats 11 more times to make the rest:

echo “===ab1===”, THIS IS FOR A TEXT HEADER SO I KNOW ITS WORKING ON Ith ITERATION

dd if=/dev/zero/ bs=1M count=300 | bar -s 300m -of ab1, THIS IS TO MAKE A 300 MEG FILE WITH A PROGRESS BAR THE FILE WILL JUST BE ALL 0 BITS

losetup /dev/loop1 /ab1, THIS IS TO ASSOCIATE THE IMAGE WITH A LOOPBACK DEVICE – A NECESSARY STEP FOR IT TO BE A DEVICE (but not too. necessary for mdadm – but I wont go into that – because I already did above in some parenthsis sets.)

echo “===ab2===”

dd if=/dev/zero/ bs=1M count=300 | bar -s 300m -of ab2

losetup /dev/loop2 /ab2

echo “===ab3===”

dd if=/dev/zero/ bs=1M count=300 | bar -s 300m -of ab3

losetup /dev/loop3 /ab3

ECTERA… ECTERA… YOU GET THE IDEA REPEATS UNTIL LOOPBACK 12…

===MAKE RAIDS (four RAID5s)===

# mdadm -Cv -c64 -l5 -n3 -e1.2 /dev/md1 /dev/loop1 /dev/loop2 /dev/loop3

# mdadm -Cv -c64 -l5 -n3 -e1.2 /dev/md2 /dev/loop4 /dev/loop5 /dev/loop6

# mdadm -Cv -c64 -l5 -n3 -e1.2 /dev/md3 /dev/loop7 /dev/loop8 /dev/loop9

# mdadm -Cv -c64 -l5 -n3 -e1.2 /dev/md4 /dev/loop10 /dev/loop11 /dev/loop12

===MAKE HYBRID RAID 55===

# mdadm -Cv -c128 -l5 -n4 -e1.2 /dev/md0 /dev/md1 /dev/md2 /dev/md3 /dev/md4

===CONFIRM===

# cat /proc/mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]

md0 : active (auto-read-only) raid5 md4[4](S) md3[2] md2[1] md1[0]

1842624 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/3] [UUU_]

md4 : active raid5 loop12[3] loop11[1] loop10[0]

614272 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/2] [UU_]

[=====>……………] recovery = 25.0% (76800/307136) finish=2.6min speed=1449K/sec

md3 : active raid5 loop9[3] loop8[1] loop7[0]

614272 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/2] [UU_]

[======>…………..] recovery = 31.1% (96000/307136) finish=2.1min speed=1609K/sec

md2 : active raid5 loop6[3] loop5[1] loop4[0]

614272 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/2] [UU_]

[=======>………….] recovery = 38.3% (118000/307136) finish=1.7min speed=1764K/sec

md1 : active raid5 loop3[3] loop2[1] loop1[0]

614272 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/2] [UU_]

[=========>………..] recovery = 45.3% (139504/307136) finish=1.4min speed=1893K/sec

unused devices: <none>

FAIL!! BUT NO WORRIES WE CAN FIX

NOTICE WITH MD0 THE RAID WE ARE CONCERNED WITH DISK MARKED AS SPARE AND ALSO auto-read-only

===MAKE SPARE AN ACTIVE MEMBER AND NOT A SPARE===

SO STOP RAID AND RECREATE WITH –assume-clean (if had data on it would recreate the raid and not hurt the data – make sure device order is the same, if not and you do an add – add causes resync – you will destroy the data – BUT if you do assume-clean and put wrong order of members just try again with another assume-clean, as long as you dont “–add” a drive you will not resync so data will not get hurt, Just cant resync with wrong order.) Anyhow..

# mdadm -S /dev/md0

# mdadm -Cv /dev/md0 –assume-clean –level=5 –chunk=128 –raid-devices=4 /dev/md1 /dev/md2 /dev/md3 missing

# mdadm -a /dev/md0 /dev/md4

cat /proc/mdstat

(or)

watch -n2 cat /proc/mdstat

(or)

watch -n2 “(cat /proc/mdstat | grep recovery)”

cat /proc/mdstat

LOOK ITS ALL FIXED AND ITS NOT AUTO READ ONLY

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]

md0 : active raid5 md4[4] md3[2] md2[1] md1[0]

1842432 blocks super 1.2 level 5, 128k chunk, algorithm 2 [4/4] [UUUU]

md4 : active raid5 loop12[3] loop11[1] loop10[0]

614272 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]

md3 : active raid5 loop9[3] loop8[1] loop7[0]

614272 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]

md2 : active raid5 loop6[3] loop5[1] loop4[0]

614272 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]

md1 : active raid5 loop3[3] loop2[1] loop1[0]

614272 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]

WIN!!!

NOTE THE SYSTEM IS CHILL:

# w

06:31:37 up 17 days, 1:15, 4 users, load average: 0.28, 2.18, 2.03

USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT

root pts/1 172.20.18.115:S. 00:01 3:13m 0.86s 0.85s /bin/bash

root pts/4 172.20.18.115:S. 00:44 5:35m 0.01s 0.01s /bin/bash

root pts/5 172.20.18.115:S. 00:46 3:03m 0.02s 0.02s /bin/bash

root pts/6 172.20.18.115:S. 03:28 0.00s 0.12s 0.00s w

################################

===2 PATHS: ext3 AND dm-crypt===

################################

##########

—EXT3—

##########

# mkfs.ext3 /dev/md0

# mkdir /mnt/r55

# mount /dev/md0 /mnt/r55

# df -h | grep -i “filesys\|md0”

Filesystem Size Used Avail Use% Mounted on

/dev/md0 1.8G 35M 1.7G 3% /mnt/r55

Very slow build but it works 🙂

Here are the numbers 🙂

–TEST SPEED WITH EXT3–

# cd /mnt/r55

-WRITE TEST:-

ZEROS DATA:

# dd if=/dev/zero of=testfile0 bs=8k count=10000; sync;

Hit CONTROL-C after 5 seconds to get results

30703616 bytes (31 MB) copied, 7.94884 s, 3.9 MB/s

RANDOM DATA:

# dd if=/dev/urandom of=testfileR bs=8k count=10000; sync;

Hit CONTROL-C after 5 seconds to get results

65994752 bytes (66 MB) copied, 21.8919 s, 3.0 MB/s

-READ TEST:-

ZEROS DATa:

# dd if=testfile0 of=/dev/null

29368320 bytes (29 MB) copied, 0.780934 s, 37.6 MB/s

RANDOM DATA:

dd if=testfileR of=/dev/null

30842880 bytes (31 MB) copied, 1.05709 s, 29.2 MB/s

OTHER READ TEST:

# hdparm -Tt /dev/md0

Timing cached reads: 16280 MB in 2.00 seconds = 8158.98 MB/sec

Timing buffered disk reads: 42 MB in 3.13 seconds = 13.42 MB/sec

# hdparm -Tt –direct /dev/md0

Timing O_DIRECT cached reads: 6928 MB in 2.00 seconds = 3466.79 MB/sec

Timing O_DIRECT disk reads: 58 MB in 3.30 seconds = 17.55 MB/sec

Now to prepare for next section with cryptography:

First remove the other ext3

# cd /

# umount /dev/dm0

################################

—CRYPTO-SYSTEM: BLOCK LEVEL—

################################

Citation 1: http://www.cyberciti.biz/hardware/howto-linux-hard-disk-encryption-with-luks-cryptsetup-command/

Citation 2: http://www.markus-gattol.name/ws/dm-crypt_luks.html

This is block level encryption so it will take /dev/md0 and encrypt it, then when we decrypt it it will give us a new device with a name of our choosing “crypt55” in this example. Then we can lay a file system on top of it, but before that write zeros to it just to ensure healthy encryption.

The crypt setup starts here

# apt-get install cryptsetup

Default options for cryptsetup -y for verify passphrase so we know we are typing the right password. -v for verbose. Notice you have to type YES in capital letters to continue. passphrase I chose was 1234 nothing special. Note when you type the password/passphrase its blank and doesnt display what you type, I just included it here for no reason.

# cryptsetup -y -v luksFormat /dev/md0

WARNING!

========

This will overwrite data on /dev/md0 irrevocably.

Are you sure? (Type uppercase yes): YES

Enter LUKS passphrase: 1234

Verify passphrase: 1234

Command successful.

Then we need to make a mapping to the device we will use. We can give it any name this, I called it “crypt55”. So basically “crypt55” will map to /dev/md0. /dev/md0 is the encrypted device, and crypt55 is the unecrypted device which gets used. So basically UNENCRYPTED /dev/mapper/crypt55 –> ENCRYPTED /dev/md0 . We need to enter the passphrase accurately. This is essentially where we “login” with our passphrase. Just for fun at first I put the wrong password to make sure it fails – like it would not lol 🙂 – and it did so then it asks for it and I enter the correct 1234

# cryptsetup luksOpen /dev/md0 crypt55

Enter passphrase for /dev/md0: 4321

No key available with this passphrase.

Enter passphrase for /dev/md0: 1234

Note you can see all your opened crypt drives after you map them – they are in dev mapper. Before the luksOpen command this folder only had the “control” device. The control device is always there.

# ls -l /dev/mapper

total 0

crw——- 1 root root 10, 59 Feb 16 05:16 control

lrwxrwxrwx 1 root root 7 Mar 5 07:02 crypt55 -> ../dm-0

To see information on the cryptsetup and get an idea of what the default encryptions are:

# cryptsetup -v status crypt55

/dev/mapper/crypt55 is active:

cipher: aes-cbc-essiv:sha256

keysize: 256 bits

device: /dev/md0

offset: 2304 sectors

size: 3682560 sectors

mode: read/write

Command successful.

First we are going to write all zeros to this “crypt55” drive which will inturn put alot of random data on /dev/md0.

# dd if=/dev/zero of=/dev/mapper/crypt55

(or even better)

BUT THAT WILL TAKE LONG AND WHO KNOWS WHEN IT WILL BE DONE LETS MONITOR WITH pv ANOTHER TOOL bar WORKS TOO – EXAMPLE OF bar WAS ABOVE. “pv -h” SHOWS ALL THE OPTIONS WE ARE GOING TO USE THESE:

-p, –progress show progress bar

-t, –timer show elapsed time

-r, –rate show data transfer rate counter

-e, –eta show estimated time of arrival (completion)

-b, –bytes show number of bytes transferred

# pv -tpreb /dev/zero | dd of=/dev/mapper/crypt55 bs=128M

38.6MB 0:00:07[3.57MB/s][ <=> ]

Note how the <=> is the location of the progress it starts on the left and goes to the right just like any progress bar. HOWEVER it comes back and then it goes the forward. AHH its not a progress bar. Its not a progress bar, its a mistake!

The speeds im seeing: 3,9,2,3,11 Mb/s

On a lighter note – I opened up another shell to see CPU use:

# w

load average: 12.68, 6.26, 2.78

12.68!!!!!!!!!!! wow that means to do the current job without queuing the processor this crappy processor would have to be 13 times stronger/faster/harder working.

Memory use was like this – regular memory almost completly used up and cache/swap free to run around:

# free -m

total used free shared buffers cached

Mem: 375 365 10 0 37 234

-/+ buffers/cache: 93 282

Swap: 1339 51 1288

When it was over this is how it looked. Notice how the progress bar is like smack down in some random spot :/ I realize I could probably use another pv arg. to make it more progressy but eh.

1.76GB 0:10:34 [2.83MB/s][ <=> ]

0+14386 records in

0+14385 records out

1885470720 bytes (1.9 GB) copied, 639.592 s, 2.9 MB/s

Total Time 10 minutes 34 seconds at 2.9 MB /s just at writing zeros. Sounds just as fast as the ext3 filesystem from above without any encryption – however this encryption is the default encryption its probably slow.

Just to heat things up Im making ext4 for this one. After all it should be “better” then ext3. If you ask me its all about “zfs” as soon as they get all the quirks out of it. zfs has encryption and raid built into it, so thats pretty sweet. Ill make an article on it later.

# mkfs.ext4 /dev/mapper/crypt55

# mount /dev/mapper/crypt55 /mnt/r55

Right after the mount it was still happy “w” showed a load average of 0.17 for the 1st minute average (thats the number on the very left on the load average triple digit numbers – so the full load average numbers were: load average: 0.17, 0.26, 1.52 – 1st minute average, 5 min average, 15 min average

–SPEED TESTS–

# cd /mnt/r55

-WRITE TEST:-

ZEROS DATA:

# dd if=/dev/zero of=testfile0 bs=8k count=10000; sync;

Hit CONTROL-C after 5 seconds to get results

81920000 bytes (82 MB) copied, 35.2622 s, 2.3 MB/s

RANDOM DATA:

# dd if=/dev/urandom of=testfileR bs=8k count=10000; sync;

Hit CONTROL-C after 5 seconds to get results

81920000 bytes (82 MB) copied, 68.4873 s, 1.2 MB/s

-READ TEST:-

ZEROS DATa:

# dd if=testfile0 of=/dev/null

81920000 bytes (82 MB) copied, 14.6555 s, 5.6 MB/s

RANDOM DATA:

# dd if=testfileR of=/dev/null

81920000 bytes (82 MB) copied, 5.95958 s, 13.7 MB/s

infotinks

My Notes, Articles & Guides for Linux, Windows and Networking.

Making RAID55 with Encryption + more

Leave a Reply Cancel reply