Air Crash Investigation: gone in 6 seconds

“What goes up, must come down”. I had an opportunity to face this old truth 2 weeks ago, when my 600mm quadcopter felt from 30 meters straight into water.

It was supposed to be a simple flight:

  1. take off
  2. fly few hundred meters away with GPS assist
  3. engage Return To Home and land

Dozens of missions like this in last few months, so really, nothing special. Unfortunately, like I mentioned above, not everything went as planned. 2 minutes after take off, UAV lost stability and went down with screaming motors.

Since I was recording everything using EZ-GUI, my cheap telemetry rig and LTM protocol, I had a last known position on a map.

flight on map

Terrain was quite rough and it took me about 10 minutes to get there. And another 10 minutes to realize, that UAV felt into 2m wide water trough (pretty decent accuracy). When I recovered my drone, it was in pretty decent shape, not counting smoke coming from few places.

drone removed from water

drone drying up

There were two open questions:

  1. Which elements survived?
  2. Why it happened?

Answer to the first question is: looks like almost 90% of UAV survived. Confirmed losses are: 1 ESC, gimbal controller, broken propeller.

Answer to the second question is: faulty ESC that got on fire during flight. Why I know it? Because I’ve run an Air Crash Investigation

Evidences

Evidence #1

First clue leading to a conclusion that crash was caused by a faulty ESC was the fact, that only one ESC was burned. All other ESCs and motors survived and after drying out were working without problems (including motor connected to burned ESC).

emax 30A ESC SimonK Series on fire

emax 30A ESC SimonK on fire

Unfortunately this clue did not proven that ESC started to burn before it was submerged in water. After all, smoke was still coming from UAV when I pulled it out. Here comes…

Evidence #2

Blackbox log of the whole flight. I was able to see data flight controller was receiving and how it was reacting to it.

blackbox investigation 01

Until 02:04 [minutes:seconds from arm] log was not showing any traces of strange behavior. This is out time marker 0. All future timestamps will be relative to 02:04.

blackbox investigation 02

At 00:00.400ms gyro started to record stronger than usual rotation on PITCH and ROLL that quickly became constant oscillation with a 2Hz frequency.

blackbox investigation 03

Flight controller started to compensate by increasing power on motor #4 and reducing it on motor #1. So, either motor #4 was producing less thrust than before, or motor #1 was producing more. This was sufficient until time marker 00:02:330ms when periodic corrections were not enough to keep UAV stable. It’s worth to mention that before 00:02:330ms flight controller was able to keep attitude error below 3 degrees and requested output from motor #4 was not bigger than 65%. If situation would not got worse, UAV would be able to continue flight.

blackbox investigation 04

Starting from 00:02.674ms flight controller started to request 100% power output on motor #4. UAV was unable to keep desired inclination, but until 00:03.162ms was able to keep inclination error below 7 degrees.

blackbox investigation 05

Starting from 00:03.162ms total thrust was not enough to keep desired altitude, and to compensate, FC started to request more thrust on other motors.

Starting from 00:03.665ms it was clear that situation is beyond recovery. Stability was lost, ROLL inclination raised above 30 degrees and was growing rapidly. Altitude stared to drop.

blackbox investigation 06

At 00:04.156ms UAV was inverted for the first time.

At 00:04:239ms FC started to request full power on motors #2, #3 and #4, what helped for a fraction of a second in terms on altitude correction. Did not helped rotation that started to exceed 600 degrees per second on ROLL and PITCH. UAV was tumbling and accelerating towards ground.

blackbox investigation 07

Everything ended at 00:06.306ms when log ends. UAV has hit water and power supply was disturbed.

Now, guess which ESC hasย burned? Yes, it was ESC #4, the one that appeared to loose power from the very beginning.

Why it took more that 6 seconds?

6 seconds is a lot of time. Much more than required to fell 30 meters. And for a half of that time UAV was stable and maintained altitude. ESC was failing gradually.

Most probable explanation is that ESC in deed started to burn somewhere before 02:04. Growing temperature triggered BLHeli Thermal Protection routine. When Thermal Protection engages, ESC tries to lower MOSFET/microcontroller temperature by lowering power output. Unfortunately, this does not works well in multirotors. Flight Controller required more and more power to keep stability and altitude. The very moment ESC started to fail, UAVs fate was sealed…

10 thoughts on “Air Crash Investigation: gone in 6 seconds

  1. Hello again, Dziku!
    Just my two cents:
    Are you using the BEC outputs coupled together on the Flight Controller? Most of the users do this and till now it is considered to be safe in practice (accordind to RCgroups). As a HW engineer I cant’ recommend this because it can lead to thermal runaways due to output voltage offsets (especially on cheaper ESCs). I can elaborate this phenomenon if you wish.
    Just my two cents, like I said…
    Zsolt

    1. No, I’m using separate BEC, BECs from ESCs are disconnected.
      But, please elaborate this phenomenon, this will be useful for everyone since people only say: no not couple multiple BEC, but they do not say why it should not be done

      1. Well, in general electrical power sources are not allowed to be put in parallel if their output voltage is not 100% equal. It can be done only if there is an external “supervisor” circuit to equalize their voltage.
        Just imagine this: we have 2pcs 5000mah batteries, one fully charged, one fully depleted. Internal resistance (+cable) ~0.05 Ohm. If we have, let’s say 5V between the 2 batteries charge state, it will result in 5V/ 0.05 Ohm = 100Amp current!!! Fireworks…
        ESCs use similar low output impedance “voltage regulator bricks”. The output voltage difference can be ~0.1V for these 5V bricks, even more for cheap ESCs. When coupling their output, the more this difference, more these regulators will try to impose their prescribed (internal reference) voltage. In other words, the fight will be bigger and the BECs eventually will heat up and may led (in theory) to catastrophic results.
        So, there is no reason to put more BECs in parallel: 1..2A output current is more than enough for your electronics (Rx and FC), so don’t try to get more by putting them in parallel. The so called “redundancy” is simply not true (if you can make safe connections), as one can imagine: these bricks are very safe on their own… until you don’t make some stupid thing with them. Of course, you can by expensive ESCs, with (maybe) better output voltage control – eventually they can be 100% equal on good QC models – but, hey – would you trust it in time and risk it? If something can led to accident in theory, then it’s only a question of time ๐Ÿ™‚ I wouldn’t fly a ticking bomb.
        And maybe there are not direct evidences to pointing this directly as root cause of many crashes, the users usually don’t do “air crash investigations” ๐Ÿ™‚ so the causes usually remain unknown ๐Ÿ™ They simply replace the broken/burnt component and go forward…
        Kudos for Dziku for his scientific approach ๐Ÿ™‚
        So, when connecting the ESC leads to your FC, always remove the (+) wire and leave only one to power the FC. And of course, be sure to secure somehow all the connectors, not to be loose in time – soldering directly is the best practice if you master it.

  2. Yeah … and I forgot the most important issue about some dubious ESCs:
    The “5V/2A” ESCs usually have 2pcs 5V/1A regulator bricks mounted in parallel, but via two “equalizing” resistor. This trick usually works very well, till some very clever chinese guys discovered that they can make a huge economy by omitting those 0.01 cent resistors, so they simply put the two bricks in parallel, so then can by default heat up the whole board, without additional negligence of the user.
    Unfortunately, i saw many designs like this… so it’s a good idea to use some tested / dissected ESCs.

  3. BTW, Dziku, can you make some close-ups of the remaining ESC boards (the carbonized one is just useless), to identify those resistors? I’m really interested in this issue because I’m waiting 12pcs similar ESCs ๐Ÿ™ (“SimonK/30A, no brand”). If you doesn’t have any spare, I will do it when they arrive.

  4. Good point, but this paralleling not necessarily enough to totally destroy an ESC IMHO. It can heat up the board and this can led to other nasty things… (The L78M05 are current protected on their output so an 1W.. max 2W thermal dissipation is not yet enough to carbonize that way the whole board)
    As an official “investigator” I can presume a couple of causes, though:
    1. Sudden output short (at the motor) – not plausible if the connections are made carefully and after the incident nothing points at this
    2. Desoldered/interrupted/defective electrolytic capacitor. Without a proper filtering, the MOSFETS can do very nasty things… I experienced it on my first quad after a freefall from 30m. Nothing burnt, just the esc stopped because a desoldered capacitor. After this (and after being stupefied by measuring it) I replaced all electrolytics with brand ones, each of them measured by me (capacitance, series resistance-ESR-at 1kHz)
    After all, the manufacturer can make most of the economy on this component – after CPU, this is the most expensive component on the ESC.
    Note that the pro’s recommend mounting extra capacitors if the power wires are too lenghty . Good ones ๐Ÿ™‚
    I think there are topics on RCgroups regarding this issue.
    The whole ESC can enter in strange oscillations whitout proper filtering. If two opposite MOSFETs will be open at the same time, despite the good driving signal from CPU, the power wires will be practically in short, resulting in something similar what you can see.
    This is my highest probability tip, according to the pics.

Leave a Reply

Your email address will not be published.