Reverse Engineering Dell iDRAC6 Express: Fan Control
Part II: The Silence of the Fans
As previously mentioned, the Dell PowerEdge T710's stock Pulse Width Modulated (PWM) fans are controlled by the iDRAC6 & BMC firmware data files. I haven't had time to properly address these components but luckily there is another quick method that we can use to silence the stock fans. Later on I also tried swapping out the stock fans (YouTube video) with much quieter Arctic F9 PWM models but this introduced other problems.
As it turns out, running
/avct/util/pwm_test will override the iDRAC firmware fan control (mostly determined by BMC firmware and the
/bin/fullfw processes) and allow us to set the fan RPM to a reasonable level, somewhere around 1,300 - 1,500 RPM, down from an average of 2,000 - 2,400 RPM. This is well above the 1,080 RPM "critical threshold" set by the iDRAC6 BMC. As you can imagine, the lower spin rate causes the noise level to drop significantly while still keeping the server cool at my regular work load.
pwm_test as root via iDRAC SSH or Telnet connection. These commands can be inserted into the init sequence if you would like to have them persist across RAC resets.
Note that we have to keep this process running in the background or
fullfw will resume control.
$ cd /avct/util $ ./pwm_test 0 0 0 1 255 & $ ./pwm_test 0 1 0 1 255 & $ ./pwm_test 0 2 0 1 255 & $ ./pwm_test 0 3 0 1 255 &
Swapping stock fans for Arctic F9s
The Arctic F9 fans spin at a much lower rate than the stock fans rate of 1,000 - 6,000 RPM. With Arctic F9 you can expect anywhere between 550 - 1,800 RPM. Without running
pwm_test the fans spin down below the critical threshold mentioned above and the iDRAC reacts to this by cycling the fan speed up before returning it to regular levels. As soon as the RPMs return to "regular levels" they fall below the threshold which causes a vicious cycle.
To combat this I used
/avct/util/FanTachTestAP to adjust the interpreted tachometer output frequency at 1 pulse per revolution. As the spec for PWM fans is typically 2 pulses per revolution this effectively causes the software to double the calculated RPM rate. With the "numbers" back into the 2,000+ RPM range I was able to avoid the vicious cycling caused by the iDRAC firmware.
I found an acceptable fan speed/noise level using the following commands:
$ cd /avct/util $ ./pwm_test 0 0 4 -1 50 & $ ./pwm_test 0 1 4 -1 50 & $ ./pwm_test 0 2 4 -1 50 & $ ./pwm_test 0 3 4 -1 50 & $ ./FanTachTestAP config all 1
To incorporate these commands into the iDRAC firmware, I created an init script at
/etc/sysapps_script/S_0000_local.sh (link to my script) and then placed it into the init sequence by editing
/etc/init.d/postinit_adv.sh and placing it into the list immediately after the line that reads
/etc/sysapps_script/S_5378_idrac_fully_ready.sh. Unfortunately it takes the iDRAC a while after booting to reach this point and you'll hear the fans cycle up and down until then, creating error entries in the DRAC system log.
You'll notice a departure from the previous article in my init script. I've moved the netcat startup to this script instead of including it in the stock Telnet init script. This wasn't for any particular reason except that I wanted as many of my customizations to appear in one place should I have to incorporate these changes into a newer version of the iDRAC6 firmware.
I'll cover customizing the fan sensor and PWM control kernel modules in the next part.
Dylan Daugherty wrote...
At 2018-01-21 01:54
Thank you for sharing this work!
It nicely quiets down an R610 (1U) with the 40mm fans of doom
Matt Adams wrote...
At 2018-01-21 02:23
Awesome, glad to hear it helped Dylan!
Phil Green wrote...
At 2018-04-18 16:10
Been looking for a mod to lower the Dell t710 PowerEdge BMC Firmware Fan Threshold, but everything above means nothing to me, is there a modified firmware zip file i can just install, I tried installing Noctua fans but the PSU just got louder.
Aymen Djellal wrote...
At 2018-09-30 17:42
Hello ! Thank you for this ! It helped me a lot ! I have a Dell R810, and I have a slight problem you might be able to help me with. Applying the ./pwm_test 0 0 0 1 255 & for every fan worked great. But every 10min (9min30sec to be exact), the fans go to maximum speed for about 30 seconds, then go back to normal. (from 1300 RPM to 13 000 RPM) And its every 9.5min (+- a few seconds) Any idea what is happening ?
Matt Adams wrote...
At 2019-12-16 23:24
Phil: Sorry, I can't redistribute the firmware images and I wouldn't want to see you bricking your BMC with a bad image from me. I imagine the PSU became louder after you installed your fans because they are returning a different PWM signal than expected.
Aymen: Keep trying to adjust the parameters to "pwm_test" - I expect that the R810 is just different enough that you will have to experiment to get what you want.
Leave a Comment