Using Mayhem for API to Fuzz etcd

forallsecure.com
5 min read
fairly easy
Last week, I found and fixed two crashing bugs in etcd , the distributed key-value store used (among other things) to manage the state of kubernetes clusters. I'm excited to have been able to contribute a bit to such an excellent project! I didn't specifically set out to work on ...
Using Mayhem for API to Fuzz etcd

J. David Lowe ·

Last week, I found and fixed two crashing bugs in etcd, the distributed key-value store used (among other things) to manage the state of kubernetes clusters. I'm excited to have been able to contribute a bit to such an excellent project!

I didn't specifically set out to work on etcd. I work on Mayhem for API, and my goal was just to integrate the etcd server into our internal fuzz testing harness. We use this harness to validate changing versions of Mayhem against unchanging versions of third party services. While we report any serious findings responsibly, this part of our system is mostly interested in Mayhem's behavior, not that of the service we're fuzzing.

Here's the thing: I know almost nothing about etcd. But it's an especially easy project to start fuzzing for two reasons: it already has a thorough openapi spec, and it defaults to a completely unauthenticated mode, meaning we can fuzz the entire API without setting up credentials.

Fuzzing etcd

I built and started etcd like this (and I wish all software were this easy to start!):

etcd$ ./build.sh etcd$ bin/etcd

And then I started a 5 minute Mayhem run, like this:

$ mapi run etcd 300 rpc.swagger.json --url http://localhost:2379

What's great is that, even knowing almost nothing about etcd, once I started fuzzing, Mayhem uncovered crashing bugs in less than a minute.

Part Zero

Mayhem for API is still a young project. Crashing bugs—where the service we're fuzzing falls over completely—are something we haven't faced often enough to handle very cleanly until now. In other words, before I could fix etcd crashing, I needed to make Mayhem usefully report the crashes, the way it already usefully reports lots of other kinds of issue.

When I first started fuzzing etcd, it crashed within seconds. However, Mayhem treated the observations it made when etcd died (closed connections and refused connections, mostly) as transient, and eventually gave up... without…
J. David Lowe
Read full article