iDrac6 Recovery Through TFTP and Serial

The History:
This week I had a Dell PowerEdge R510’s iDrac completely die on me; I attempted repairs with several utilities that Dell gives out on their site and all of them ended with failure. I thought it might have been because I upgrade the iDrac from an old version to the latest, without components like the BIOS or NIC, that the iDrac communicates with, being upgraded as well. After upgrading everything, iDrac still was not working, after a few days of messing with it, I found out through piecing together several sites how to force the iDrac in recovery mode to do a TFTP repair, writing a new image to it.

The symptoms:
The system used the Windows iDrac Updater, which stated the update had competed successfully. I then, remotely, told the system to reboot; it shut down and never came back up. When I physically went to the server, it was at the BIOS start screen stating “Error Communicating with iDrac. Press F1 to continue, or F2 for System Setup.” In restarting the server I found that “System Services” were disabled. Then the system would go through normal boot sequence, but when it tried to communicate with the iDrac it would fail then restart the server. After restarting, it would allow a full boot, but would give that same “Press F1 to continue, or F2 for System Setup” message. Thus the server would not boot without physical intervention at the machine.

This is a Dell PowerEdge R510, I attempted to upgrade the iDrac from 1.3.* to 1.6.5.

The Fix:
We need to get to the iDrac’s serial recovery mode, and then we can recover the system.

  1. Reboot the system, and after the system resets itself for not being able to reach iDrac go into “System Setup”, the F2 key
  2. Hit down until you select “Serial Communication”, enter that menu
  3. Set the following settings:
    • Serial System Setup Settings
    • Serial Communication : On With Console Redirection via COM2
    • Serial Port Address : Serial Device 1=COM1, Serial Device2=COM2
    • External Serial Connector : Serial Device 1
      • This could be Remote Access Device, but that gave me problems (I may have had a bad serial cable)
    • Failsafe Baud Rate : 115200
      • For the 11G servers this is the default baud rate
    • Remote Terminal Type : VT100/VT220
    • Redirect After Boot : Enable
  4. Then rebooted the system. I got Windows to start by manually hitting F1
  5. At this point you need to go to support.dell.com, lookup downloads for your system, then under “Embedded Server Management” there is “iDRAC6 Monolithic Release 1.97” (or whatever version is newest)
  6. There are several versions, for my system I got “iDRAC6_1.97_A00_FW_IMG.exe (50 MB)”
  7. After downloading, running this file will extract “firmimg.d6” and a readme file.
    • The readme has no useful information in it, it just tells you to search for the user guide
  8. The “firmimg.d6” file needs to be placed on a TFTP server that the iDrac can hit
  9. Using Putty in Windows I connected the COM2 at 115200 Baud, this is the iDrac being redirected. Connect to your systems Com2 however you can
    • Note all this is being done on the server and nothing is done on a other machine, I had TFTP running on this Windows system
  10. Hitting enter should show a recovery menu
    • Unfortunately I did not save pictures of the recovery screen, some of the next menu options may not be the exact wording
  11. I had DHCP on the network my iDrac was sitting on so I hit 9 to get a IP address, this can also be set manually
  12. Hit 7 to change the TFTP server IP address
  13. Now hit the option that says “Firmware Upgrade”, this will go to the TFTP server specified, download the firmware, and reinstall all pieces of the iDrac from that file. It takes about 5 minutes.
  14. Keep in mind you are in your OS, for me Windows, while the iDrac and its system upgrades and reboots
  15. After it reboots successfully the recovery console stops getting data, I was next to the server, when the iDrac reboots the fans go to full speed then calm back down. That’s how I was able to tell it restarted
  16. Now you can use the RACADM commands if open manage/iDrac tools are installed, or reboot and you should see “System Services” back online, then you can change the IP of the iDrac like normal

Everything should work now and the world is happy!

141 comments

  1. Thanks you saved my day!
    I ran the update live dvd (om74-firmware-live) to update the firmware on a R710 server.
    everything wend fine, till it startet hanging at the iDRAC6 update (while the fans where on full speed) waited for +3 hours. Than restarted the server.
    The Server showed the same problem you described better jet your solution worked!
    I didn’t have a OS on the box, so i used the live dvd to access the serial console (“screen /dev/ttyS1 115200”). It had some issues displaying all of the serial output correctly but thanks to you i knew that 9, 7 + ip and 10 would do the trick. In case some of the characters that are send are not display ignore that, and assume they are there.

    Thanks again!!

    1. hey, i copied the recovery menu,
      With the path i took:

      ==== Firmware Recovery Option ====

      1. Bootloader Version ……. 1.13.7
      2. Ethernet Adapter ……… AMEA
      3. MAC Address ………….. bc:30:5b:e8:2c:1b

      4. IP Address …………… 192.168.0.120
      5. Subnet Mask ………….. 255.255.255.0
      6. Gateway IP Address ……. 192.168.0.1

      7. TFTP Server IP Address … 192.168.0.100
      8. Image File Name ………. firmimg.d6

      9. Enable DHCP Client
      10. Execute Firmware Upgrade
      11. Reset

      12. Bypass Firmware Recovery
      – This option reverts serial device 2 to be connected
      to the external serial connector.

      ECOVER> 9
      Wait for auto-negotiation complete … OK 100MB – Full Duplex
      BOOTP broadcast 1
      *** Unhandled DHCP Option in OFFER/ACK: 28
      *** Unhandled DHCP Option in OFFER/ACK: 28
      DHCP client bound to address 192.168.xx.yy

      RECOVER> 7

      Please enter TFTP Server IP Address ==> 192.168.xx.zz

      RECOVER> 10

      Downloading image from TFTP server …

      Using AMEA device
      TFTP from server 192.168.xx.zz; our IP address is 192.168.xx.yy
      Filename ‘firmimg.d6’.
      Load address: 0x1000000
      Loading: 52 MB
      Bytes transferred = 55364664 (34ccc38 hex)

      Checking and flashing N-1 image …

      Checking partition of N-1 image … Done!

      Checking image header CRC … OK
      Checking platform ID …….. OK
      Checking kernel image CRC … OK
      Checking rootfs image CRC … OK
      Checking u-boot image CRC … OK
      Skipping u-boot update …… YES
      Copying kernel image …….. OK
      Copying rootfs ………….. OK
      Done!

      Syncing up persistent storage …

      Checking and flashing N image …

      Checking partition of N image … Done!

      Checking image header CRC … OK
      Checking platform ID …….. OK
      Checking kernel image CRC … OK
      Checking rootfs image CRC … OK
      Checking u-boot image CRC … OK
      Skipping u-boot update …… YES
      Copying kernel image …….. OK
      Copying rootfs ………….. OK
      Done!

      Erasing private storage …

      Un-Protected 12 sectors

      Erasing Bank 0: 100%
      Erased 12 sectors
      Protected 12 sectors

      Erasing bootloader environment …

      Un-Protected 1 sectors

      Erasing Bank 0: 100%
      Erased 1 sectors

      Writing Bank 0: 100%
      Protected 1 sectors

      Hit any key to stop reset: 0

      ÿ
      ——
      Then the menu stops. and after about 2 Minutes the fan speed goes down.

  2. Hi Guys,

    After 2 day’s (many long reboots) trying to fix this nasty issue, I found this solution and it works! Thanks a lot.
    All other solutions failed. On a T610 I had ESX 5.5 installed. To fix it, I installed a windows server 2008 evaluation to see if the windows solutions worked. There are quit some people having a issue with drac with no working solution. Installing the TFTP server and putty and simply let DHCP assign a IP and and use my own IP as TFTP server, the process was completed within minutes. It’s funny that even Dell does not mention such a great solution.

    Thanks a lot for sharing

    Best Regards

    Henk

  3. I’m so glad I found this guide, however there is a little issue.
    After setting the Serial config in the BIOS, as shown here: https://buildingtents.files.wordpress.com/2014/04/screen-shot-2014-04-24-at-12-21-03-pm.png?w=620&h=124

    The COMS2 port dissapears from Windows Server 2008 upon booting up.
    COMS2 shows up again when It’s set to “On Without Console Redirection”.

    I’ve tried connecting via The External (Serial port 1), and Virtual Serial port in Windows, still nothing.

    I’m going to try around with the Serial settings in the BIOS to get it to show up.

    1. Did the serial settings give you any luck? I was thinking you could go into Device Manager on Windows and see what serial ports it is giving you. Perhaps for some reason it is creating COM3 or something similar?

      1. Well, as it turns out, Even tho I can’t see COM2, apparently It’s being used!

        So, that made me think:
        Maybe It’s hidden.

        And sure enough, It is.

        Unfortunately, It thinks It’s unplugged, so I think iDrac6 has kicked the bucket.

        Looks like I’m going to have to replace the Motherboard in my Dell T610.

  4. Hi, im so glad to find this article.. Everythinks works, But when i start the Firmware Update, the output says *** device is not ready ***

    Any more ideas? Replace Mainboard??

    Output:

    RECOVER> 10

    Downloading image from TFTP server …

    Using LOM0 Gb1 device
    TFTP from server 192.168.25.8; our IP address is 192.168.25.183
    Filename ‘firmimg.d6’.
    Load address: 0x1000000
    Loading: 52 MB
    Bytes transferred = 54574136 (340bc38 hex)

    Checking and flashing image …

    *** device is not ready ***

    Erasing persistent storage …

    Un-Protected 12 sectors

    Erasing Bank 0: 100%
    Erased 12 sectors
    Protected 12 sectors

    Erasing bootloader environment …

    Un-Protected 1 sectors

    Erasing Bank 0: 100%
    Erased 1 sectors

    Writing Bank 0: 100%
    Protected 1 sectors

    Hit any key to stop reset: 0

    1. One thing you could try. Unplug the server, the hold the power button for ten seconds. I know that sounds crazy, and a classic IT thing to say; but I spoke to a dell systems engineer and he told me todo that and it fixed my problems before. By holding the button for ten seconds when the system doesnt have power. It tells the battery powered iDrac and BIOS to flush everything running and clear all settings, then reboot when they get power again.

      1. Hi,
        I had the same problem (“*** device is not ready ***”). Tried to hold power for 10 seconds with AC unplugged, but it got no better.
        I tried to post in the Dell community but nobody has any idea yet.
        Any ideas or hints appreciated.

      2. Unfortunately I dont have any recommendations 😦 That should reset the chip and then the console should allow recovery mode to work. Sorry :/

      3. I replaced the DRAC (the little module with plastic cover that plugs into the motherboard, not the one that has the RJ-45 connector) and it works now. Hope this helps someone googling around for this problem.

      1. Hi danberk, thank you. I tried many times, but no avail. Maybe I need to replace. Should I need another idrac express or I need the whole motherboard?

        Thank You,
        Tiago

      2. Sorry to hear that, I definitely have had some I just couldn’t recover, can you login to the idrac at all? I’ve been able once to use https://www.dell.com/support/home/us/en/19/drivers/driversdetails?driverid=8f07j&oscode=ws8r2&productcode=poweredge-r610 the repair program to block level replace the broken parts and get it back online. It’s a real longshot. If you have a 11th Feb or lower, you can replace the chip I believe, 12 and up, it’s usually on the motherboard.

  5. Hi,
    I am so glad to find this article after i been toss a few days.

    I have the same problem on R610. I do every step i could done from here, but i can’t enter iDrac recovery mode!!

    What can i do ?

    Here is what i do with:
    1. set the BIOS like this article describe.
    2. using a usb to serial port adapter on a laptop running windows 8.1 and the serial port adapter works fine.
    3. using null modem cable connected R610 COM2(serial, NOT VGA) and laptop’s usb to serial port adapter.
    4. I could see R610 booting up information, but if the server not power on and booting up, i can see nothing and i can’t enter recovery mode.

    Question:
    1. Do i need another PC connect to the Server (R610) in COM2?
    2. what cable should i use for connection of the console redirection?
    3. Is the Server(R610) needed a OS? such as running on Windows or Linux?

    Thanks..!

    1. To answer your questions in order 🙂
      1. No I used the server itself to read the comm port.
      2. No cable its virtual!
      3. I had windows on my server, but Linux should work too, just need to get the Linux idrac package.

      1. Quick Question, after the redirection, how did you connect to coms port 2?
        With Putty in Windows on the Server itself?
        I’ve done everything however it shows up, however it is ‘disconnected’ and so putty simply can not connect to coms port 2.
        Maybe I’m missing something….

      2. Mine was comm 2, you may want to check device manager, it could be a different number for you. I used putty with the settings listed.

      3. THANKS FOR YOUR REPLY.
        I will install a windows Server 2008 on that Server then try again with your way.
        Whether succeed or failed, I will reply your here.

        I better take a rest for now, it’s 5:09am here.
        hehe

      4. Same problem like here others reply, I can see COM2 on my windows server 2008 on that Server.

        There’s even no hidden COM2! I installed OMSA on windows server 2008, it can’t detect iDrac too (No idrac options).

        Is that something I missed?
        Do I have to replace any hard part on that Server ?

      5. I didn’t replace any part. I was able to communicate over com 2 then, make sure you hit enter after connecting to see if it gives you anything? Also you could try unplugging the box, and holding the power button for 10 seconds; then plugging back in. That hard resets idrac.

  6. Hi, I used Dell’s live DVD 7.3 to update the server r610 and got it hanged on DRAC update, after reboot front LCD went dark and it fails to communicate to drac now.
    Whatever I do, there is nothing related to drac recovery from ANY com ports. Been switching those com settings in bios back and forth without success, the only output I can get is booting text, nothing else there! In windows there is NO COM2 etc. This is so frustrating.

  7. Just got it fixed!
    Dell R610. Real pain.

    Set in BIOS:

    Serial Communication: On without Console Redirection
    Serial Port Address: Serial Device1=COM2, Serial Device2=COM1 (NOTE, reversed then in author’s post)
    External Serial Connector: Serial Device 1
    115200

    Find in google a live Windows (any kind). I used some creepy Gandalf’sWin7x86PE.iso and it was good enough. Write to USB or connect ISO directly via KVM (my case).

    Boot to Windows, configure networking, download Putty. If in Device Manager you see Unknown Device, just do right click and click update driver (setup networking upon doing this).
    Note: live iso works in RAM only, all the files or drivers you added will be lost after reboot.

    You should see COM1 and COM2 both unhidden!

    Connect to COM1 via putty, selecting Flow Control: None and Rate: 115200 !!!!
    If port is empty, try COM2. In my case COM1 was the one.

    You’ll see the recovery menu. FINALLY!!!

    1. Hi Viktor, I’m going through this same problem on my R410. I have tried your alternate config for the serial but although now I am able to see COM2 in windows 2008 r2, neither COM2 nor COM1 show any output in putty.

      Are you certain you got this to work with the option “Serial Communication: On without Console Redirection” I can’t see how it would work without console redirection, but if it worked for you that is good because this is the only way I can see COM2 in windows…

      Thanks I hope you can see this reply and let me know.

  8. i am having this issue, however, when I connect a keyboard or a mouse nothing I don’t seem to have any power or connection to it. suggestions?

    1. Are you using the front ports, or a port on the back? I have seen the ribbon on some dell servers going to the front ports fail, or they could be disabled in the BIOS?

  9. DanBerk,

    Your a legend, Had no Front Lights, No iDRAC, No Lifestyle Controller, and System Services Disabled after a failed upgrade ( upgrade failed due to iDRAC SD Card not being seated and iDrac firmware going from 1.95 to 1.98 )
    Built New OS HDD, and Installed, then went through this exact procedure with Putty and 3C Daemon ( TFTP). All worked a treat. Used Firmware iDRAC 1.95 ( Last known good for my server)
    Thankyou, Thankyou, Thankyou

  10. can i do this with my r720 ? i try many times with your procedure but still no luck 😦 , still nothing on my putty screen , is there any thing that i need to setup more on bios with my r720 ?

    1. Have you tried unplugging both power supplies, and holding power for 30 seconds? On the newer idracs that forces a hard reset. Or there is an idrac repair package you can run on the OS?

  11. Hello.
    I have the same problem with R720
    I configured serial communication but i don’t have field “External Serial Connector”.
    When i try connect to com2 i see only black screen.
    I haven’t done try unplugging both power supplies yet.
    I tried reset NVRAM, unplugged power cables during 3 minutes, but it didn’t work.
    What about “IDRAC repair package” i didn’t see it. Can you send link, please.
    Will Can this packet worke if My OS isn’t see IDRAC module?
    Please help.

    1. Have you tried running the normal iDrac update package against it? The iDrac is mounted during updates, and maybe it would see it?

  12. I have an R520
    I am having the same issue when I connect to COM2 using putty. Just a blank screen… I also don’t see the “External Serial connector” option.
    I tried resetting NVRAM, draining the flea power, running the update package.
    Also, I have noticed that the bios reports that the server is an R420…. I have tried several BIOS revs and this is still the case. 😦

  13. h, I have run into the idrac communication failure problem after an unsuccessful firmware udpate via web gui. i found this site, and while i have made some progress – got putty to connect to serial com2, see menu options detailed above. did install tftp (server currently running winserver2016 on poweredge r710). i feel like i am missing something simple. in which directory do i save the img file to, so that that i can load via putty? OP says “The “firmimg.d6” file needs to be placed on a TFTP server that the iDrac can hit” but where is that exactly?? thanks for any help. i feel like i am close to getting this thing working and updated… but soooo far away at the same time.

    1. Hi!
      The tftp server will have a folder that it serves files out of, put it in the root of that folder and it’ll become tftp:///firming.d6
      If you have made a webserver before it’s like that, files in the folder become root of the web/tftp server.
      Dan

  14. Hi danberk!!

    We have R420 with this problem.
    The config Bios is all right, we did. But next steps we need some help:

    1- Can We use CENTOS LiveCD to copty the firming.d7 file? (this case we will get the file with hosted on my notebook); or need use the instalation OS on the server (we have SXI with Windows7 e Ubuntu Server)

    2- With LiveCD we get the file with TFTP, so, how can the update will happen? We need just reboot the server?

    Thank you

    1. Hi,

      1-Centos as long as you have the image file is fine, I have used live environments a lot. Getting to the serial port is just different.

      2-The idrac has local storage, the tftp transfer takes it from your computers memory and storage to the idrac flash, which is a completely different, out of band OS.

      Just make sure to get the right image for your model of server/idrac. 🙂

      Dan

  15. If I use COM1 with redirection, I am able to see the output but not the menu. If I use COM2, I am not able to see anything except for a few off words. I am using putty to telnet to serial. What am I doing wrong? Thanks Dan. My idrac is not responding on a R320.

  16. Hi,

    I have R320 server, I put a serial cable (on the same server) attached to IDRAC network port.
    I follow your step, but neither com1 nor com2 are working using putty and I didn’t see the recovery windows.
    Please I need your help.

      1. Server has a high fan noise and server’s LCD is off.
        Server has VMWARE ESXI installed. there is no windows installed.
        SW0700: iDRAC is not ready.

      2. Have you tried, turning it off, powering down, then wit hit unplugged, holding the power for 30 seconds to drain idrac power and force a reset.

  17. Dell tech support helped me update iDRAC firmware, seems used wrong version (iDRAC8_2.10 for my iDRAC7). The update failed and I can’t access iDRAC anymore (initializing error, fan runs fully, noisy, after twice rebooting, goes to above screen “Press F1 to continue, or F2 for System Setup”.
    No OS on my T420. As mentioned above, using TFTP and serial COM, should I set up a TFTP on my laptop and connect both iDRAC port and laptop LAN to network (switcher), in addition to connecting Serial COM to laptop via DB9?
    Thanks in advance.

    1. I haven’t tried to access the menu externally, also have you tried shutting down, pulling the power cords, then holding the power button for 30 seconds to flush and reset idrac?

      1. I already tried it, but no luck.
        iDRAC port doesn’t not blink, the front service tag sceeen no light anymore. Unplug idrac card and drain power, boots again, the little screen lights again but nothing on it.
        With you instruction, in my case, to connect to TFTP which is on laptop, I need network connection to T420 and laptop. How set up the network? Does serial com know TFTP as well? I am fraid I cannot get t420’s IP setup.
        Thanks

  18. Thanks.
    Dell site gives ‘bootable depository’ to update whole fireware package, but seems it is Linux version. Did you try it on your problem? Not sure if is for windows too.

    1. I had a 11th gen 510, that one I was able to upload in windows, you can get a Linux live CD and boot it without installing anything to run that package!

      1. Today, I installed Windows server 2011 on it with CD, and installed WinTFTP.
        Setting COM as you showed, using PuTTy, tried both COM2 and COM1 connection, but the terminal couldn’t get anything. I checked COM in device management, they all show “work properly”, I even change their speed rate to 115200. Not lucky.
        Any suggestions about this serial way?

        Next maybe try Dell bootable depository.

      2. This page was originally written for 11th gen, you as well as some others seem to have the same thing where it doesn’t come up on the com port, I tried it on an 12th gen that worked and it didn’t come up either, there must be another trick.
        Have you tried opening the com port the holding the info button to force an idrac reset?

      3. Resetting idrac by holding infor button worked well before this idrac-upgrade. After the failure of update, it is out of control. Tech support concluded yesterday my idrac is faulty, we have to replace the motherboard. I am not convinced.
        I read idrac7-8 guide again (about page66), seems putting bootable firmware.m7 on embedded SD card, the system will load and reset idrac. Let me try it now.

      4. Sorry. It is page76, “rollback firmware-recovering iDRAC” with firming.d7.
        Idrac7-8-lifecycle-controller-v2606060_user-guide.

  19. Hey there, I think your blog might be having browser compatibility issues.

    When I look at your website in Firefox, it looks
    fine but when opening in Internet Explorer, it has some
    overlapping. I just wanted to give you a quick heads up!
    Other then that, terrific blog!

    1. I couldn’t get a 12th gen to the same place, and unfortunately haven’t seen a great solution other than praying during hard idrac reset by holding power without any power

  20. It would be useful if you’d make it clearer in the article that people should not connect the server to an other laptop that has a serial port as that won’t work. I’ve been trying that for days and have wasted time and only realized now that you are running Putty INSIDE the OS running on the server! Had that been clearer in the article to begin with I wouldn’t have wasted so much time.
    Connected via a serial cable to an other laptop all that I’m allowed to see is the post messages but I can’t actually get into the rac recovery mode. I tried all the settings in the BIOS but to no avail. So, I know that my connection works because I can see POST messages but I can’t get into the recovery console.

  21. Thanks for the solution. I got an initialization error on t320. I am able boot it, but I don’t have any luck with tftp and putty.

    I installed tfpd server on t320 using http://tftpd32.jounin.net/. Pointed tfpd server to using local IP address 192.168.1.220.
    Tried putty COM1 and COM2 and nothing shows up on the screen. Any suggestions?

    1. A t320 is 12th gen, so its iDrac will respond a bit different, have you tried running one of the packages like “DELL iDRAC7 1.66.65, ESM_Firmware_3F4WV_WN64_1.66.65_A00.EXE” That should be able to overwrite a bad firmware with a fixed one. Did you try shutting down, unplugging, and holding power for 30 seconds to force a full power drain on the iDrac?

  22. Dell R720 PowerEdge while booting facing this issue :

    CPLD version : 103

    Alert! Power required exceeds PSU wattage. Check PSU and system configuration.
    Alert! Continuing system boot accepts risk that system may power download without warning.

    Warning! iDRAC initialization error. Continuing to boot may yield unexpected power loss due to unknown power inventory and various Management, Inventory, LCD, or System Error Reporting.

    Management Engine Mode : Recovery
    management Engine Recovery version : 0002.0001
    Recovery Patch : 0005
    Recovery Build : 00A7
    Strick the F1 key to continue, F2 to run the system setup program

    ____________________________________________________________________________
    When I press F1 or F2 to get system setup or get boot manager is stuck and happens nothing

    it do not boot to windows which is already installed in it, any idea please …………..?

  23. A Dell Poweredge R720, while bios boot i face to this error :
    CPLD version : 103

    Alert! Power required exceeds PSU wattage. Check PSU and system configuration.
    Alert! Continuing system boot accepts risk that system may power downloa without warning.

    Warnning! iDRAC initialization error. Continuing to boot may yield unexpeted power loss due to unknown power inventory and various Managment, Inventory, LCD, or System Error Reporting.

    Management Engine Mode : Recovery
    management Engine Recovery version : 0002.0001
    Recovery Patch : 0005
    Recovery Build : 00A7
    Strick the F1 key to continue, F2 to run the system setup program
    ____________________________________________________________________________

    while bios loading lifecycle is disabled and when i press F2 to system setup or F11 for manager its says entering to system setup or Entering boot manager but after the above error and i press F1 and F2 nothing happens, the above error screen stuck and nothing happens. it also do not boot from windows server which is already installed in it

    any idea or advice please…. ?

    1. Have you tried holding the information button for 30 seconds to reset the idrac? Also going g into the BIOS via F2, then going to the iDrac settings and lifecycle controller may show if it’s in recovery mode. You may also want to use the iDrac settings menu to erase the iDrac if you can

  24. I did 30 second information button not result, when i press F2 to get system setup it says Entering while bios loading but after that facing this error :
    CPLD version : 103

    Alert! Power required exceeds PSU wattage. Check PSU and system configuration.
    Alert! Continuing system boot accepts risk that system may power download without warning.

    Warning! iDRAC initialization error. Continuing to boot may yield unexpected power loss due to unknown power inventory and various Management, Inventory, LCD, or System Error Reporting.

    Management Engine Mode : Recovery
    management Engine Recovery version : 0002.0001
    Recovery Patch : 0005
    Recovery Build : 00A7
    Strick the F1 key to continue, F2 to run the system setup program
    _______________________________________________________

    even i press F1 and F2 both nothing happens, it do not let to entering in system bios or continue to installed os, it just freeze on the above error that’s it.

    1. Have you tried unplugging both power supplies, then holding power for 30 seconds to force a baseband reset? Do you have any big GPU or pci cards?

      1. I did not 30 second holding power button with unplugging both power supplies and there is not GPU but there are 3 other pci cards, I will test both this evening.

  25. another point that I forgot to mention is ( when the server first boots up after configuration memory complete, initialization idrac face to error : “initialization idrac do not responding” and server restarts

  26. i did the above tow points, removing all pci cards and holding for 30 second power button without PSUs, not result…….

    1. Have you tried removing both power supplies, then reseating just one and seeing if the power error occurs? Unfortunately I don’t have a ton of tools for a box in that state :/

      1. Yes also I did without power supplies no result, but I have found another method by googling, If I got any result will share.

  27. I’ve tried to do this from Linux, but neither /dev/ttyS0 or ttyS1 would show anything.
    Does this means that Linux is unaware of BIOS redirection and initialize the ttyS* without taking care of BIOS settings, or does this means that It’s also dead for me using this method? (ssh responds (card gets an ip thru DHCP) but refuses my passwords and web interface is down and lspci doesn’t see the iDRAC card.
    I’m trying to access using:
    echo -n ” > /dev/ttyS1
    screen /dev/ttyS1 115200

    then I type enter and… nothing
    same for ttyS0 (com1; come2 is ttyS1)

    1. You need to turn the setting on in the BIOS then it should show up as a device. If you can see the ttyS* device, then the bios is presenting it to Linux. Have you tried the racadm commands from the Dell Repo? You could override the username/password there.

  28. Hi Danberk,

    I have a Dell R410 and tried your solution but putty does not show me anything, just a balck screen.
    Do i need to plug in a serial cable to theiDrac port?
    What network settings d i need to use?
    I have using normal server 2016, is that the issue?

    Your assistance is greatly appreciated?

    Thanks

  29. Has anyone got this issue fixed with a 12th Gen server? I have an R720 doing the same thing. I have tried holding the power button with power supplies disconnected multiple times. When I try the serial redirect from within the OS (Server 2016), I get a black screen. Not sure I am doing something wrong on that front. Lastly, I was able to utilize a 3rd party warranty and get a motherboard replacement and it is the exact same issue! Is my luck so bad we got a refurb that had the same issue to begin with?! I tried to strip down the config too. all raid controllers and PCI cards removed. The only thing in it is some HDD, the CPUs, and memory….

    1. On a 720 I think you can get a idrac repair package, that may do something? 2 in a row is odd. Can you get into the life cycle controller?

  30. Hi, I have similar problem with m620, its stuck on initializing. I cant press any button because i see only “initializing”. Is there any option to start bios and recovery idrac

  31. Just want to add to this for anyone who comes across similar issues on a PowerEdge R720/R720XD – if the amber light on the rear of the server is flashing then put the firmimg.d7 file on a FAT formated SD card and put the SD card in the slot at the back. The flashing light should turn solid and 5 or so minutes later iDRAC should be back up and running.

  32. First, I’m SUPER impressed that this article from 2014 is still active in the comments. danberk, you’re a wizard.

    PowerEdge R710 Gen2
    iDRAC 6 Enterprise (lights active)
    Front panel display – appears inactive (no lights or activity)
    Windows Server 2012 R2 (eval installed only for this task)
    FAT-formatted SD w/ firmimg.d6 image
    TFTP w/ firmimg.d6 image

    Experiencing the issues described in the article above. I’ve exhausted my internet-ing abilities and finally landed on this article. I’m hung up on step 13, iDRAC firmware upgrade can’t seem to find the TFTP server I established on the server itself. I dropped the firewall entirely and I can reach it from other computers on my network, however, the iDRAC can’t seem to find it. Putty works for me on COM2 and I can see the Recovery menu and interact with it, however, I can’t get it to load the firmware. I get the following:

    Downloading image from TFTP server …
    Using AMEA device
    TFTP from server 192.168.1.19; our IP address is 192.168.1.23
    Filename ‘firmimg.d6’
    Load address: 0x1000000
    Loading: 0 B T:200

    I finally cancel the process at T:200 and the load progress never changed (remains at 0 B). As a result of canceling the process I get the following:

    Checking and flashing N-1 image …
    Checking partition of N-1 image … Done!
    *** unknown image version ***
    *** fail to update image ***
    *** fail to execute firmware upgrade ***

    Rebooting the server w/ the SD card doesn’t seem to do anything at all. Help? I feel like if I could just get the iDRAC to see the TFTP server and download the image, I’d probably be golden.

    Yup, I’ve tried …
    – removing power cords and holding the power button for 30+ seconds
    – just about everything support.dell.com had to offer
    – being extra nice to the server
    – yelling profanities at the server
    – apologizing to the server

    Thanks in advance.

      1. I can’t get the .usc file to unzip as mentioned in the instructions, so I haven’t been able to get it to run as stated in their documentation. 😦

      2. Oh, snap. It’s [firmware upgrade] working now. Here’s the difference … hosted tftp from Linux vs Windows. That was it. Thanks again for the great article!

        NO MORE FANS AT 100% … AAAAAMAAAAAZING
        *happy dance*

    1. Hello

      I have the same problem and would like to try iDrac recovery via Serial Communication and TFTP Server. Could you please to provide more detailed instruction with screenshots? It is very critical for me now. Thanks in advance.

  33. Hello

    Could anybody please help me to repair iDRAC? I have the same problem with server DELL R720dx. I would like to have more detailed instruction with screenshots because I am beginning administrator and some points is not understood for me. Now company where is located this server requires me to buy a new server.

      1. I did not tried a repair package. I havenext error SW0700: iDRAC is not ready. I can not get into the life cycle controller. I do not have any OS on the server. What is the first step for me?

      2. You should look at some of the other comments in this thread, some things like holding power with the server unplugged for 30 seconds can help in that situation.

      3. I already tried it but it was not success… I would like to try iDrac recovery via Serial Communication. Could you please to provide more detailed instruction with screenshots? It is very critical for me now. Thanks in advance.

    1. One,

      The following assumes you’ve configured the hardware of your server via the instructions in danberk’s blog post above.

      Start by making a bootable USB/Flash drive (I recommend something like Ubuntu Linux). See this article for instructions: https://tutorials.ubuntu.com/tutorial/try-ubuntu-before-you-install#0

      Once you get your bootable Ubuntu flash drive, then you can boot the server (F11 for the BIOS Boot Manager). When the Boot Manager appears, scroll down to “Hard Drive C:” and a new menu will appear with your flash drive listed. Select the flash drive to boot Ubuntu.
      In Ubuntu, get tftp up and running by following the guide I found: https://linuxhint.com/install_tftp_server_ubuntu/
      You’ll want to do BOTH the server and client (testing tftp server) portions. Once you have a functional tftp server, you can download the firmimg.d6 image and save it to the root of your tftp server.
      Install Putty (https://www.linuxtechi.com/install-putty-in-ubuntu/) and you should be able to follow from step #9 of danberk’s blog above going forward.

      I’m no expert, but I’ll try my best to help / clarify as needed.

      Good luck!

      1. Hello Outrigger

        Need I install Ubuntu or load system from flash drive? If need to load system from flash drive so I cannot to start service TFTP. Please help me.

      2. One, you’ll need a bootable Operating System, but it can be installed on the flash drive instead of on the server itself. The link I provided walks you through downloading and installing Ubuntu on the flash drive. You can then boot the server from that flash drive and follow the rest of the instructions. Does that help clarify?

      3. Hello Outrigger

        I do not have recovery menu when I connected via putty to iDrac Console Redirection. I tried to connect to COM1 and COM2 but nothing. What is the problem?

      4. One, walk me through it. You get a connection on COM2, but no menu? Did you press Enter at the blank screen to reveal the menu?

      5. Yes, I pressed Enter but was only blank screen. Now I tried to boot from Windows liveDVD but had the same as in Linux. I can sent to you photos from BIOS Serial Communication settings for check.

      6. One, I’m so sorry … if those instructions aren’t helpful, I’m not sure how else I can help. I’m very new at this as well and was only able to get my Enterprise iDRAC operational again thanks to the instructions from danberk. Good luck

      7. Outrigger,
        Do I need to connect the server to network for the start of recovery menu or it is not necessary?
        Thanks in advance

      8. One,
        Not to my knowledge. The Recovery Menu should be present on the iDRAC itself. Depending on where you place the image file, you may need network/Internet access.

  34. Morning danberk and others! I’ve made it to step 9, but having issues connecting to the iDRAC via COM2 with putty. I’m not quite sure what I’m missing? Here’s some background info of what I’ve done so far:

    Dell R720
    iDRAC 7; firmware was 1.4x

    – I tried updating the firmware via the web UI. While uploading the .d7 file, it froze and crashed. I think this is what is causing the issues…
    – Booted up to the Dell Centos SLI bootable linux to try and manually flash the fw. I did this on our other R610 and it worked great (following this guide: https://robwillis.info/2018/12/manually-updating-the-firmwares-on-a-dell-poweredge-r610/). Doesn’t seem to work here for the R720 and says it can’t find the controller. Same when trying to update the BIOS and LCC and such: https://imgur.com/A7RWeFg
    – I’m seeing the same SWC0700 error that others are: https://imgur.com/a/GOHk7Or
    – Tried firmwares for everything surfing the Dell support site
    – Found out about RACADM and it can apparently reset it to default with the command “racadm racresetcfg” but I couldn’t figure out how to get it to actually install/work: https://imgur.com/X4v9o6K – I’m not the best with Linux unfortunately and have a VERY basic knowledge of things in bash
    – Found your blog here and set the BIOS up: https://imgur.com/h7xsSBO
    – Got MediCat Windows 10 PE bootable os on a flash drive and fired it up and installed putty on it and googled up what TFTP was and found it as a tiny little app that creates a TFTP server on your system: https://imgur.com/JKVdn7C
    – I noted your comment that you did everything on the SAME computer, because initially I thought a USB > RS232 serial cable was going to be needed with the BIOS changes and stuff.
    – When trying to connect to the COM2 port, it fails. I’m assuming that if it had connected is when I would be able to go to step 10 and see the recovery mode CLI and be able to press 9and 10, etc etc.
    – I saw the comment from Jacob Murphy about his COM ports being hidden/having issues and I’m wondering if I’m in that spot. Like if that failed fw update killed the whole iDRAC maybe?

    Any help is super appreciated in advance 🙂
    Thanks

  35. Morning danberk and others! I’ve made it to step 9, but having issues connecting to the iDRAC via COM2 with putty. I’m not quite sure what I’m missing? Here’s some background info of what I’ve done so far:

    Dell R720
    iDRAC 7; firmware was 1.4x

    – I tried updating the firmware via the web UI. While uploading the .d7 file, it froze and crashed. I think this is what is causing the issues…
    – Booted up to the Dell Centos SLI bootable linux to try and manually flash the fw. I did this on our other R610 and it worked great (following this guide: https://robwillis.info/2018/12/manually-updating-the-firmwares-on-a-dell-poweredge-r610/). Doesn’t seem to work here for the R720 and says it can’t find the controller. Same when trying to update the BIOS and LCC and such: https://imgur.com/A7RWeFg
    – I’m seeing the same SWC0700 error that others are: https://imgur.com/a/GOHk7Or
    – Tried firmwares for everything surfing the Dell support site
    – Found out about RACADM and it can apparently reset it to default with the command “racadm racresetcfg” but I couldn’t figure out how to get it to actually install/work: https://imgur.com/X4v9o6K – I’m not the best with Linux unfortunately and have a VERY basic knowledge of things in bash
    – Found your blog here and set the BIOS up: https://imgur.com/h7xsSBO
    – Got MediCat Windows 10 PE bootable os on a flash drive and fired it up and installed putty on it and googled up what TFTP was and found it as a tiny little app that creates a TFTP server on your system: https://imgur.com/JKVdn7C
    – I noted your comment that you did everything on the SAME computer, because initially I thought a USB > RS232 serial cable was going to be needed with the BIOS changes and stuff.
    – When trying to connect to the COM2 port, it fails. I’m assuming that if it had connected is when I would be able to go to step 10 and see the recovery mode CLI and be able to press 9and 10, etc etc – https://imgur.com/apM0p4d
    – I saw the comment from Jacob Murphy about his COM ports being hidden/having issues and I’m wondering if I’m in that spot. Like if that failed fw update killed the whole iDRAC maybe?

    Any help is super appreciated in advance 🙂
    Thanks!

    1. Hello,

      First have you tried powering off the box, unplugging, then holding power for 30 seconds to force the idrac to reload? You can also go into Linux and look for /dev/ttyS# that is the Linux equivalent to Com-something and the Linux drivers may work out of the box, win pe can be hit and miss

      1. Thanks for the quick reply bro. Yeah, unfortunately I tried the power off strategy already 😦 Ok, sweet I’ll try getting into Linux and seeing what’s up. I called for service, got a mobo replacement and it did the exact same thing, including the SWC0700 error. Apparently, getting another board soon. Will report back!

      2. ***I’m sorry for that other post. Apparently I’m awful at hitting the reply button.

        Is that ISO also referred to as the SUU? I found a video of a guy using that and booting into the Dell centos pe. He ran it through the console and it searched and updated everything compatible. I tried that originally before replacing the board and it bombed out. It searched and found the BIOS and some hdd firmware and other stuff, but when typed like “sudo ./suu -e” (I think the -e arg was for update only) it tried on each hardware piece and failed.

        I love your example though. I’m going to try that as soon as possible. Currently, I got the board replaced once and the board was dead. It wouldn’t even light up the front info panel display thing. 2nd one they shipped in here actually works, but we’re getting a voltage error now. They have PSU’s on order to swap those out. I think we might have practically a new server by the time were done here LOL.

      3. I think it’s slightly different? This is a no touch iso you just boot and it goes through and does everything last time I used it 🙂

      4. Dang that sounds frickin awesome. No touch upgrade has gotta be the best! Can’t wait to try it. I’ll report back asap 🙂

      5. An anti-climatic end to it all. They replaced the mobo 3 times. I guess third time was the charm. Came with a DRAC7 license as well, so that was nice. Unfortunately, didn’t get to try out any of the cool stuff, but I’ll run the no-touch firmware update you linked me next week when I can schedule more downtime. Thanks again for all of your help and posting this blog!

    2. Is that ISO also referred to as the SUU? I found a video of a guy using that and booting into the Dell centos pe. He ran it through the console and it searched and updated everything compatible. I tried that originally before replacing the board and it bombed out. It searched and found the BIOS and some hdd firmware and other stuff, but when typed like “sudo ./suu -e” (I think the -e arg was for update only) it tried on each hardware piece and failed.

      I love your example though. I’m going to try that as soon as possible. Currently, I got the board replaced once and the board was dead. It wouldn’t even light up the front info panel display thing. 2nd one they shipped in here actually works, but we’re getting a voltage error now. They have PSU’s on order to swap those out. I think we might have practically a new server by the time were done here LOL.

    3. An anti-climatic end to it all. They replaced the mobo 3 times. I guess third time was the charm. Came with a DRAC7 license as well, so that was nice. Unfortunately, didn’t get to try out any of the cool stuff, but I’ll run the no-touch firmware update you linked me next week when I can schedule more downtime. Thanks again for all of your help and posting this blog!

      1. Thanks Dan for this, and to everyone else with comments.

        I have two identical R410 servers. Running Windows Server 2019 Datacenter Edition. Idrac Express and idrac enterprise installed on both.

        Server1 has the idrac communication failure as mentioned with all the fan speed issues. I got the groovy menu on putty. I have the firmware image file.

        Anyone got a link to an idiot guide on how to set up a tftp server please? I can’t make it connect. Idrac nic mode is shared. Firewalls are off. (I tried solarwinds free tftp server and it doesn’t connect on the other server.) Maybe I need to run tftp on the server itself?

        If this doesn’t work, I’m getting a replacement motherboard. There is nothing wrong with the idrac express or enterprise modules as I swapped them with the identical server.

      2. Hello!

        Have you made sure the file is in the correct folder, as well as set the allowed IP range under security? https://appuals.com/how-to-setup-and-configure-tftp-server-on-windows-10/ goes over that setting.
        In networking terms, I don’t think you want to run it on the same server because of the shared NIC, you may run into issues where the network is confused about traffic going out then back in the same port. From server 1 in windows can you connect to the tftp server with a client like tftpd32? Just to figure out if it’s the board having issues or the server. I have always used tftpd32 (or 64 now) for tftp server needs. Remember like http, tftp will have it’s own root folder, so c:\tftp-root\file, becomes tftp://your-ip/file

  36. Update while it is still fresh in my mind.
    I have reinstalled TFTP server on a Windows 10 machine following the instructions. Still have no idea how to use a client. Thanks for the link anyway. The image file is in the correct folder.
    I go to another windows machine and open up file explorer. In the address bar I put tftp://192.168.1.xx of the windfows 10 machine and file explorer asks me what to do with the file. I am guessing that works.
    So then I get to the putty stage and I see the prompts as others have seen. It would not connect.Then I read the bit about DHCP. So I tried to make it DHCP and the negotiation fails.
    So then I pull out both the idrac express card and the enterprise card. And this time no prompt on putty. So right now, I am trying to run it with just the idrac express card.

    My situation is…
    idrac IP responds to ping only when both the express and enterprise cards are removed.

    Weird

    At the moment, I can run the server without idracs and the fans are 75% loud. Or install idrac cards that I cannot connect to and have the fans 199% loud.
    For comparison, the working server is 50% loud at 100% CPU load.

    I will popst again, but so far, a new system board may be the best option.

  37. So with only the idrac express installed, I get this

    Downloading image from TFTP server …

    Using LOM0 Gb1 device
    TFTP from server 192.168.1.9; our IP address is 192.168.1.210
    Filename ‘firmimg.d6’.
    Load address: 0x1000000
    Loading: 54 MB
    Bytes transferred = 56826936 (3631c38 hex)

    Checking and flashing N-1 image …

    Checking partition of N-1 image … *** fail to read ***
    *** fail to read block from device ***
    *** fail to read MBR ***
    *** fail to activate partition ***

    *** fail to swap to N-1 image ***

    I downloaded a fresh copy and got the firmimg.d6 file from the Dell website just now.

    I am going to power off and try with the enterprise as well.

    Going down……

    1. So with the idrac enterprise installed, COM2 doesn’t go to the shared NIC any more. It goes to the dedicated NIC. I am now downloading and it is slower as the NIC is only 100Mb/s

      I will update when done.

  38. Last update for today as it is late.

    Checking partition of N-1 image … *** fail to read ***
    *** fail to read block from device ***
    *** fail to read MBR ***
    *** fail to activate partition ***

    *** fail to swap to N-1 image ***

    So I pulled out the idrac express and tried another one. Still the same results.

    Also this evening, I eventually found the jumper to clear the NVRAM (It is at the back, near the idrac express). With a live power cord, but no fans running, I shorted the 2 connectors that were open for 10 seconds.

    I will try again tomorrow but if this doesn’t work, I thik it might be time for a new main board.

    Sorry for all the spam everyone. I had to type it all out as I went along in case I forgot anything.

    1. OK. So after trying various things and doing some more reading, I got it to the stage where the fans were quieter and both idrac cards installed. This was caused be downgrading the BMC to an older version.
      I also went through the BIOS on both systems side by side.
      Also did the flea power and NVRAM tricks.
      Still no putty to get the firmware recovered.
      So as a final solution, I took the idracs from the working server and put them in the broken one. Broken server still broken. Working server still working beautifully. Also broken server back to being noisy.
      Tried a few more firmware updates and it is telling me that I cannot apply them as Server 2019 Datacenter is too new.
      The closest I got to a good system was quiet fans with the idrac cards installed and system services available. However when I compared with the working server, there was no platform update option. Could not install lifecycle repair or much else.
      Now giving up. There are tested used motherboards on eBay for 30GBP. As long as the revision/part numbers are the same, I just need to use the Service Tag change utility and that should be everything.
      Once again to everyone here for sharing, thank you very much. This is a week of my life I will never get back, so at least this was all done during lockdown.

      1. That happens, we have all had some boards not work unfortunately. High fans usually does mean something in bios and idrac can’t communicate.

  39. Hi! Quick question that’s entirely off topic. Do you know how to make your site mobile friendly?
    My site looks weird when browsing from my iphone4.
    I’m trying to find a theme or plugin that might be able
    to correct this problem. If you have any recommendations,
    please share. Thanks!

    1. Just a bit off topic 😉 I am just running stock wordpress.com, and I picked one of their themes. I have very few plugins installed. If you go to your WP Admin, under appearance there is a section for Mobile that has to be enabled. Hope that helps.

  40. Hello everyone,
    I’ve been trying to do this on a Poweredge 720 but did not have a OS installed and have been having trouble even installing a Windows Server 2008 OS on the drives, getting some kind of RAID error which is another hell of a issue to fix.
    One last idea I have is trying to make a “Windows to go” OS on a portable hard drive and then booting through USB onto that OS.
    All I want is to get to the console and see if I can actually get into the serial communication of the iDRAC port. I have yet to test it, but does anyone know if this would even work?

    1. I have done the same and it’s worked before but I was using Windows PE for embedded devices, the only issue with win pe is that I didn’t have device manager so I had to guess the COM number

      1. Thanks for the information, I will definitely test and report back to see if I’m able to get to the recovery menu. Does Windows PE support installing a program like Putty? And wouldn’t the COM number be just COM2?

      2. WinPE is odd, but you can easily get portable versions of things to run usually, then you are right it’s guessing COM1 2 or 3

  41. Hi I just experienced a failed iDRAC upgrade on my R320 and now it’s exhibiting the same issues as described in the post. Do these steps work for the 12th generation iDRACs? It doesn’t have an SD card slot.

    1. I would first try to do the unplug, and hold power for 30 seconds trick to attempt to reset the idrac. Then if that doesnt work I would go and get the http://downloads.dell.com/published/pages/poweredge-r320.html “Block Repair Package” that goes and in an OS block repairs the idrac storage. It usually brings it back to an old idrac/lifecycle version but can get you back online. You can also try the idrac upgrade again from the lifecycle controller if you can get into that.

      1. how do I install this if both the idrac and lifecycle controller aren’t working? it’s a .usc file. the bios says ‘lifecycle controller disabled’

      2. after doing some more research, it seems that “block repair package” repairs the lifecycle controller through the idrac. but both are not working.

        https://ctrlaltdell.wordpress.com/2013/08/12/hard-hard-reset-of-a-dell-idrac67-and-lifecycle-controller-plus-another-troubleshooting-tip/

        so how can I repair the idrac and lifecycle controller on a 12th generation poweredge? the dell update ISO hangs at collecting system inventory and your guide didn’t work

      3. Unfortunately I dont have any ready advice for you on that, this guide is for 11th gen, and the 12th gen iDrac is a bit different. By chance are your fans running very high? That can be a sign the iDrac and motherboard arent getting a link up. The lifecycle controller wont let you get into it on boot? Have you tried to go in the bios and make sure its not in recovery mode? There are some times these boards just get in a state and need replaced which is sad, you can see tales of others here who just ended up replacing it :/

  42. Hi all, one more moment, If you gonna to restore it using firmimg.d7 on SD, try to do it with oldest one.. in case, server will apply only newest version of FW. Not same, not older, newest then actual installed.

Leave a Reply to ktirmnih Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s