megacli - How to Replace a Failed Disk

My monitoring started to alert me about a failing disk recently.

Replacing a failed disk is quite straight forward. I have first identified the device:

┬─[root@hv1:~/s/megacli]─[09:42:10 AM]─[G:main=]
╰─>$ ./10-lsi.sh errors
Slot Number: 0
Slot Number: 1
Slot Number: 2
Slot Number: 3
Slot Number: 4
Slot Number: 5
Slot Number: 6
Slot Number: 7
Slot Number: 8
Slot Number: 9
Slot Number: 10
Slot Number: 11
Media Error Count: 333
Other Error Count: 33
Slot Number: 12
Slot Number: 13
Slot Number: 14
Slot Number: 15
Slot Number: 16
Slot Number: 17
Slot Number: 18
Slot Number: 19
Slot Number: 20
Slot Number: 21
Slot Number: 22
Slot Number: 23
LSI Custom Script
{
    "pd-number": "7",
    "enclosure-id": "2",
    "slot-number": "11",
    "enc-position": "1",
    "device-id": "35",
    "wwn": "5000CCA0252FFFDF",
    "media-error-count": "333",
    "other-error-count": "33",
    "predict-fail-count": "0",
    "pd-type": "SAS",
    "raw-size": 600126116593.664,
    "sector-size": "512",
    "logical-size": "512",
    "firmware-state": "Online,SpunUp",
    "serial": "xxxxxxxxxxx",
    "device-speed": "6.0Gb/s",
    "link-speed": "6.0Gb/s",
    "drive-temp": "29C",
    "vd-id": "2"
}
Python Custom to JSON Script

1) Locating the Drive

We will locate the drive to make sure that we are replacing the correct disk.

/opt/MegaRAID/MegaCli/MegaCli64 -PdLocate -start -physdrv[2:11] -a0
Adapter: 0: Device at EnclId-2 SlotId-11  -- PD Locate Start Command was successfully sent to Firmware 

Exit Code: 0x00

2) Phisically Removing the Drive

After locating the drive we can remove it and replace with a spare one.

3) Rebuilding the Array

My MegaRaid card is configured to auto-rebuild my arrays in case of a disk failure.

We can check the status of the rebuild with the command below:

/opt/MegaRAID/MegaCli/MegaCli64 -PDRbld -ShowProg -PhysDrv[2:11] -aALL
Rebuild Progress on Device at Enclosure 2, Slot 11 Completed 91% in 0 Minutes.

Exit Code: 0x00

Resources

LSI MegaRAID - megacli - thatlinuxbox.com