🏊‍♀️🚴‍♀️🏃‍♀️Mood: Excited and nervous for my race tomorrow

🎵 Soundtrack: This is Lana Del Ray

📚 Reading Github Issues

Started today by reading the 15 new github issues since yesterday. Same trends as always: people care about compiling and some fuzzing stuff.

I am starting to recognize the people who comment on the github issues the most. I feel like I am figuring out who is a golang maintainer the hard way.

🐱 Go fuzzing -> fuzzing in go

Yesterday I got some fuzzing working with the tool go-fuzz, which has better documentation than the native fuzzing beta feature. Today my goal was to get the same examples working with the native fuzzing on gotip.

First: update gotip with the latest from the fuzzing branch with gotip download dev.fuzz

To keep it simple here, I kept my code and test all in one file.

// parser_test.go

package parser

import (
	"testing"
)

func Parse(data string) {
	if len(data) == 5 {
		panic("oh no! bad size!")
	}
}

func FuzzParse(f *testing.F) { // this function is called the "fuzz target"
	// seed initial corpus
	f.Add("meow")
	f.Add("amelia")

	f.Fuzz(func(t *testing.T, a string) { // this is the "fuzz function" it is run for each test input
		  Parse(a)
		}
	})
}

Then I ran gotip test -fuzz=FuzzParse

Got some output:

$ gotip test -fuzz=FuzzParse

found a crash, minimizing...
fuzzing, elapsed: 0.1s, execs: 27 (218/sec), workers: 4, interesting: 0
--- FAIL: FuzzParse (0.12s)
        panic: oh no! bad size!
        goroutine 11 [running]:
        runtime/debug.Stack()
        ...

And I see the following generated files:

$ tree
.
├── parser.go
├── parser_test.go
└── testdata
    └── corpus
        └── FuzzParse
            └── 91e831e695d7ff994bac5052298b9399302af9ad44338b0f740f8c96e97847e8


$ cat testdata/corpus/FuzzParse/91e831e695d7ff994bac5052298b9399302af9ad44338b0f740f8c96e97847e8
go test fuzz v1
string("woeow")

Now that this failure is recorded in the corpus, it will run whenever I run gotip test, even when I don’t run fuzzing. Which we can see here:

$ gotip test

--- FAIL: FuzzParse (0.00s)
    --- FAIL: FuzzParse/testdata/fuzz/FuzzParse/91e831e695d7ff994bac5052298b9399302af9ad44338b0f740f8c96e97847e8 (0.00s)
panic: oh no! bad size!
goroutine 9 [running]:
...

This will continue failing until I fix the bug. Pretty cooool!

When I ran this with go-fuzz, the it minimized my crasher to “00000”. I was surprised to see the crasher be “woeow”, that doesn’t seem minimized to me. And I know that the minimizer is running because it is printing out found a crash, minimizing... after the test run. I ran through these steps a few times. Between runs I ran rm -rf testdata to clear the data.

Here are some examples (can you tell I had fun running it?):

  • string("\x01\xe9eow")
  • string("f\xa6\x94\x87,")
  • string("m$\x80\x00K")
  • string("amelS")
  • string("a\x1dlia")
  • string("\x00\x02\x00\x00\x00")
  • string("meo'w")
  • string("MEow\xbc")
  • string("aQQ\x89\x9f")

🧐 I think I might try to look deeper into the minimizer on Monday. This is not how I expected it to work. These do not look minimized to me.

Biggest differences between go-fuzz and native fuzzing

I did only play around with these for a couple days. No promise for correctness here.

Fuzzing types

  • Go-fuzz: only makes fuzzy bytes.
  • Native fuzz: can generate fuzzy values for bytes, strings, and ints.

Number of Fuzzing Args

  • Go-fuzz: only fuzz one value at a time.
  • Native fuzz: can fuzz multiple values on each run.

Generating the Corpus

  • Go-fuzz: user makes one file per item
  • Native fuzz: uses the f.Add function at the beginning of the fuzz target.

Number of Runs

  • Go-fuzz: by default (seems to) keep running forever, even after it finds a crasher
  • Native fuzz: by default stops at the first crasher (will eventually have a flag to continue)

Minimizing

  • Go-fuzz: beautiful minimizing
  • Native fuzz: not-so-beautiful minimizing

How to run

  • Go-fuzz: have to build a zip file first and then run
  • Native fuzz: no pre-build step! Just run go test with the fuzz flag.

🔍 Fuzzing Real Functions

I searched the CloudFoundry github org for instances of the word “parse” to try to find functions suitable for fuzzing.

I ended up writing some fuzzing tests for:

🐞 I didn’t find any bugs or crashers, but I did learn how difficult it is to write good fuzzing functions. For example the ParseQueryParam function expects an argument of type url.Values. The fuzzer will not automatically make complex structs like url.Values, so I had to take the fuzzers generated values and make the struct myself. But in doing so I made a lot of decisions about the input data, which seemed to take away from the randomness of it all.

🐜 Minimizing…

I had a little bit of time left in my day, so I decided to peak at the minimizer code. Turns out it does SOME minimizing, but not the type I was expecting based on my experience with go-fuzz. It will shorten the length of the data, but it does not try to simplify the characters. Maybe this is something I could contributeeeeeeeeee.

I went to write the test that I wanted (yay TDD!), but couldn’t get them to run. So I asked my very first question in the gopher slack (after googling and reading the docs, of course). Someone responded in a way that it still tool me 30 more minutes of reading stack overflow to understand. But I figured it out!

You can’t use go test for go itself. You have to use go tool dist test to test the go distribution

So I ran this to find out the name of the test (since my initial guess of internal/foo wasn’t working):

$ gotip tool dist test -list

and then ran the tests:

$ gotip tool dist test go_test:internal/fuzz

##### Testing packages.
ok  	internal/fuzz	0.155s

ALL TESTS PASSED (some were excluded)

🎉 Yay fast feedback loops!

💡 I should PR this to the contribution guide, because this was waaaay to hard to figure out and the guide only suggests running a script that takes about 20 minutes to run allll the tests.

Self notes for Monday

  • Congrats on your race I’m sure you did great!
  • Continue looking into improving the minimizer to simplfy characters. Look at how go-fuzz is doing it.
  • Make a PR to the contribution guide on how to run individual tests.