Hey there! I’m Jared and I’m proud to announce that I’ll be working with the Haskell Summer of Code project; specifically, on the project “Bringing Sanity to the GHC Performance Test-suite.” It was a bit of a roller-coaster getting to this point but I’m quite excited to get cracking.
For those who want the nitty-gritty details on the project and wish to read my exact proposal, I’ve put it on my GitHub and you can view it here. For those who want a less technical introduction, here’s a bit of introduction about the project itself, the languages I’ll be working in, and what I aim to do:
First off, GHC is the compiler of the programming language Haskell. It’s pretty neat and does lots of cool stuff; unfortunately, one of the things it isn’t super great at right now is running performance tests. The performance tests do two things: first, they make sure that when people change the compiler, they don’t accidentally make it slower; secondly, they make sure that when people change the compiler, they don’t accidentally make the generated code less efficient. Right now, the performance test situation requires manually generating “target numbers,” which are the same for every machine and operating system–this renders them mostly useless unless you submit your patch to the build server that has a standardized testing area.
This is a giant pain in the neck when you’re just trying to hack on the compiler yourself and get something implemented. Worse yet, because Windows, MacOS, and Linux typically all have slightly different characteristics, the performance number has to have a wide enough error tolerance to account for the operating system differences; this means that you can introduce, say, a 10% performance regression and have the tests pass because that error margin was needed to account for OS differences. Over time, those hidden regressions can pile up if they’re not diligently caught. So now, in order to really contribute meaningful code to the compiler, a contributor has to:
That’s an impressive amount of busy-work and nonsense that a programmer has to keep up with just to not make the compiler suck. We’re not even getting to actually improving the compiler! Clearly, this is a situation that needs to change.
The goal, at the end of this, is to have a much improved situation all around.
How are we gonna achieve all of this? I’m glad you asked! The first goal is going to be to introduce a new test modifier to the testsuite driver (a python program that runs the tests). This will dump collected measurements into a git note. At this point, I won’t be messing with any of the program’s logic itself yet except for potentially starting to refactor the performance metrics out of the test-driver. After that, a few tests will be moved over to use the new test modifier and make sure that everything’s working; things will still be manual, but when everything works, all that remains is to automate the git note population and usage–then the most basic and essential goals of the project will be done.
The plucky hero (me) and his brave companion will adventure forth to fix this and bring sanity to the GHC performance test-suite! Will they succeed? Will they fail? Find out on this blog! It’ll be covering the ups and downs of the journey towards sanity. (And as a note to the technical users: I will be committing to a git branch in the GHC repo: wip/perf-testsuite. You’ll be able to see progress there, too.)