MAKE SPARE AN ACTIVE MEMBER AND NOT A SPARE WITH RAID5 AND MAKE AUTO-READ-ONLY GO AWAY
THIS FIX WORKS WITHOUT HURTING YOUR DATA IF YOU FOLLOW THE STEPS 🙂

WHAT WAS THE PROBLEM:
Highlighted in green is the problem a SPARE DISK marked with S. Notice how the raid was magically auto-read-only (i dont know if thats related to it being a spare, most probably definetly not 🙂 ) Any how this fix will fix that too. The fix is highlighted in Yellow. Order of drives is important too, thats in blue highlight.

# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active (auto-read-only) raid5 sdd1[4](S) sdc1[2] sdb1[1] sda1[0]
1842624 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/3] [UUU_]

THE FIX:
SO STOP RAID AND RECREATE WITH –assume-clean (if had data on it would recreate the raid and not hurt the data – make sure device order is the same, if not and you do an add – add causes resync – you will destroy the data – BUT if you do assume-clean and put wrong order of members just try again with another assume-clean, as long as you dont “–add” a drive you will not resync so data will not get hurt, Just cant resync with wrong order.) Anyhow..

Get the chunk size and the rest of the settings for your fix line – we need to match the yellow line with the same settings as the old raid so we dont hose the data – in this case chunk size
# mdstat -D /dev/md0
Look at the super block in the good drive (technically with this problem they are all good except as I understand sdd being the spare shouldnt have a superblock – i might be wrong though -any how that doesnt matter) – look at the information of mdadm -E to get more settings so that you match the raid accordingly. The main things to match are chunk size and in which way its symmetric in this case and most cases (99.99%) they are all left symmetric – because thats the best type of symmetricity for a raid. If the symmetricity of your raid is not left-sym. then add the -p in your mdadm –assume-clean line and find the right argument for -p to match your symmetricity I KNOW that -pls is to set left-symmetric but we never use it as thats what the default setting is with mdadm.
# mdstat -E /dev/sda1

Stop RAID before Fix
# mdadm -S /dev/md0

order of recreation is key so look at output of the mdstat and notice the device orders it in the brackets it goes from right to left (opposite of english). Highlighed in blue is what Im talking about.

# cat /proc/mdstat

# mdadm -Cv /dev/md0 –assume-clean –level=5 –chunk=128 –raid-devices=4 /dev/sda1 /dev/sdb1 /dev/sdc1 missing
or
# mdadm -Cv /dev/md0 –assume-clean -l5 -c128 -n4 /dev/sd[abc]1 missing

Readd the disk ONLY if your sure thats the right order or if DATA does NOT matter (how to know if it works? Mount it, if you see data you win!, If you use MDADM inside your LVM run lvscan and if you see your LV then you win! If you dont retry by starting at stopping the raid again):

# mdadm -a /dev/md0 /dev/sdd1

WATCH PROGRESS OF RESYNC:
# cat /proc/mdstat
(or)
# watch -n2 cat /proc/mdstat
(or)
# watch -n2 “(cat /proc/mdstat | grep recovery)”