I have a 3 RAID arrays on my Linux server.
root@master:~# lsblk -b
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 21474836480 0 loop
loop1 7:1 0 10737418240 0 loop
sda 8:0 0 512110190592 0 disk
├─sda1 8:1 0 1048576 0 part
└─sda2 8:2 0 512107741184 0 part
└─md0 9:0 0 511972474880 0 raid1 /
sdb 8:16 0 512110190592 0 disk
├─sdb1 8:17 0 1048576 0 part
└─sdb2 8:18 0 512107741184 0 part
└─md0 9:0 0 511972474880 0 raid1 /
sdc 8:32 0 3000592982016 0 disk
└─sdc1 8:33 0 3000591450112 0 part
└─md127 9:127 0 9001368551424 0 raid5
└─vg0-lv0 252:0 0 11001479036928 0 lvm /mnt/bigdrive
sdd 8:48 0 4000787029504 0 disk
├─sdd1 8:49 0 3000591450112 0 part
│ └─md127 9:127 0 9001368551424 0 raid5
│ └─vg0-lv0 252:0 0 11001479036928 0 lvm /mnt/bigdrive
└─sdd2 8:50 0 1000193654784 0 part
└─md126 9:126 0 2000116776960 0 raid5
└─vg0-lv0 252:0 0 11001479036928 0 lvm /mnt/bigdrive
sde 8:64 0 4000787030016 0 disk
├─sde1 8:65 0 3000591450112 0 part
│ └─md127 9:127 0 9001368551424 0 raid5
│ └─vg0-lv0 252:0 0 11001479036928 0 lvm /mnt/bigdrive
└─sde2 8:66 0 1000193654784 0 part
└─md126 9:126 0 2000116776960 0 raid5
└─vg0-lv0 252:0 0 11001479036928 0 lvm /mnt/bigdrive
sdf 8:80 0 4000787030016 0 disk
├─sdf1 8:81 0 3000591450112 0 part
└─sdf2 8:82 0 1000193654784 0 part
md0 is a mirror raid and is irrelevant to my problem.
My other 2 raids are RAID 5 arrays.
root@master:~# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4] [raid0] [raid10]
md126 : active raid5 sdd2[0] sde2[1]
1953239040 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [UU_]
bitmap: 0/8 pages [0KB], 65536KB chunk
md127 : active raid5 sdd1[1] sde1[2] sdc1[0]
8790398976 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [UUU_]
bitmap: 2/22 pages [8KB], 65536KB chunk
md0 : active raid1 sdb2[1] sda2[0]
499973120 blocks super 1.2 [2/2] [UU]
unused devices: <none>
root@master:~# mdadm --detail /dev/md126
/dev/md126:
Version : 1.2
Creation Time : Tue May 21 19:23:02 2024
Raid Level : raid5
Array Size : 1953239040 (1862.75 GiB 2000.12 GB)
Used Dev Size : 976619520 (931.38 GiB 1000.06 GB)
Raid Devices : 3
Total Devices : 2
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Tue Jun 11 19:07:38 2024
State : clean, degraded
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Consistency Policy : bitmap
Name : master:2 (local to host master)
UUID : 266af7f7:8915136a:c8630864:2c514c63
Events : 3201
Number Major Minor RaidDevice State
0 8 50 0 active sync /dev/sdd2
1 8 66 1 active sync /dev/sde2
- 0 0 2 removed
root@master:~# mdadm --detail /dev/md127
/dev/md127:
Version : 1.2
Creation Time : Tue May 21 19:22:19 2024
Raid Level : raid5
Array Size : 8790398976 (8.19 TiB 9.00 TB)
Used Dev Size : 2930132992 (2.73 TiB 3.00 TB)
Raid Devices : 4
Total Devices : 3
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Tue Jun 11 19:05:40 2024
State : clean, degraded
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Consistency Policy : bitmap
Name : master:1 (local to host master)
UUID : cb895040:3fa006aa:88ea4323:15194452
Events : 25871
Number Major Minor RaidDevice State
0 8 33 0 active sync /dev/sdc1
1 8 49 1 active sync /dev/sdd1
2 8 65 2 active sync /dev/sde1
- 0 0 3 removed
As seen above, one array has 4 drives and one has 3. Both arrays are degraded with one drive missing each.
I have inserted a new drive "/dev/sdf" that has 2 partitions that exactly matches the sizes of the missing "drives".
The status of the 2 arrays looks identical to me.
Now, when I try to add the new/replaced drives to the 2 arrays, only one works (!!!).
root@master:~# mdadm --manage /dev/md127 --add /dev/sdf1
mdadm: add new device failed for /dev/sdf1 as 4: Invalid argument
root@master:~# mdadm --manage /dev/md126 --add /dev/sdf2
mdadm: added /dev/sdf2
I am even able to add the "/dev/sdf1" drive to the "/dev/md126" array (even though it's too big), so there is no problem with the drive itself.
I am really at a loss here. No idea why "/dev/127" refuses to accept the drive and throws "Invalid argument" without any further details to why...
UPDATE:
root@master:~# cat /sys/block/md126/md/dev-*/state | xargs
in_sync in_sync
root@master:~# cat /sys/block/md127/md/dev-*/state | xargs
in_sync in_sync in_sync
In dmesg, I see this error. No idea why it thinks it would have a superblock already as I am adding a new drive:
md: sdf1 does not have a valid v1.2 superblock, not importing!
md: md_import_device returned -22
cat /sys/block/md126/md/dev-*/state | xargsandcat /sys/block/md127/md/dev-*/state | xargs? Please add these to your questionmissingas one of the parameters you need to replace it in the appropratedev-*subdirectory withremove. For exampleecho remove >/sys/block/md126/md/dev-YYY/state