Dell PowerEdge R510

iDrac6 Recovery Through TFTP and Serial

The History:
This week I had a Dell PowerEdge R510’s iDrac completely die on me; I attempted repairs with several utilities that Dell gives out on their site and all of them ended with failure. I thought it might have been because I upgrade the iDrac from an old version to the latest, without components like the BIOS or NIC, that the iDrac communicates with, being upgraded as well. After upgrading everything, iDrac still was not working, after a few days of messing with it, I found out through piecing together several sites how to force the iDrac in recovery mode to do a TFTP repair, writing a new image to it.

The symptoms:
The system used the Windows iDrac Updater, which stated the update had competed successfully. I then, remotely, told the system to reboot; it shut down and never came back up. When I physically went to the server, it was at the BIOS start screen stating “Error Communicating with iDrac. Press F1 to continue, or F2 for System Setup.” In restarting the server I found that “System Services” were disabled. Then the system would go through normal boot sequence, but when it tried to communicate with the iDrac it would fail then restart the server. After restarting, it would allow a full boot, but would give that same “Press F1 to continue, or F2 for System Setup” message. Thus the server would not boot without physical intervention at the machine.

This is a Dell PowerEdge R510, I attempted to upgrade the iDrac from 1.3.* to 1.6.5.

The Fix:
We need to get to the iDrac’s serial recovery mode, and then we can recover the system.

  1. Reboot the system, and after the system resets itself for not being able to reach iDrac go into “System Setup”, the F2 key
  2. Hit down until you select “Serial Communication”, enter that menu
  3. Set the following settings:
    • Serial System Setup Settings
    • Serial Communication : On With Console Redirection via COM2
    • Serial Port Address : Serial Device 1=COM1, Serial Device2=COM2
    • External Serial Connector : Serial Device 1
      • This could be Remote Access Device, but that gave me problems (I may have had a bad serial cable)
    • Failsafe Baud Rate : 115200
      • For the 11G servers this is the default baud rate
    • Remote Terminal Type : VT100/VT220
    • Redirect After Boot : Enable
  4. Then rebooted the system. I got Windows to start by manually hitting F1
  5. At this point you need to go to support.dell.com, lookup downloads for your system, then under “Embedded Server Management” there is “iDRAC6 Monolithic Release 1.97” (or whatever version is newest)
  6. There are several versions, for my system I got “iDRAC6_1.97_A00_FW_IMG.exe (50 MB)”
  7. After downloading, running this file will extract “firmimg.d6” and a readme file.
    • The readme has no useful information in it, it just tells you to search for the user guide
  8. The “firmimg.d6” file needs to be placed on a TFTP server that the iDrac can hit
  9. Using Putty in Windows I connected the COM2 at 115200 Baud, this is the iDrac being redirected. Connect to your systems Com2 however you can
    • Note all this is being done on the server and nothing is done on a other machine, I had TFTP running on this Windows system
  10. Hitting enter should show a recovery menu
    • Unfortunately I did not save pictures of the recovery screen, some of the next menu options may not be the exact wording
  11. I had DHCP on the network my iDrac was sitting on so I hit 9 to get a IP address, this can also be set manually
  12. Hit 7 to change the TFTP server IP address
  13. Now hit the option that says “Firmware Upgrade”, this will go to the TFTP server specified, download the firmware, and reinstall all pieces of the iDrac from that file. It takes about 5 minutes.
  14. Keep in mind you are in your OS, for me Windows, while the iDrac and its system upgrades and reboots
  15. After it reboots successfully the recovery console stops getting data, I was next to the server, when the iDrac reboots the fans go to full speed then calm back down. That’s how I was able to tell it restarted
  16. Now you can use the RACADM commands if open manage/iDrac tools are installed, or reboot and you should see “System Services” back online, then you can change the IP of the iDrac like normal

Everything should work now and the world is happy!