<- top
-|--- Back Story ---|-

I'm surrounded by talented coders, but every so often I meet one who mishandles bugs.

These coders are not the type to throw their hands in the air when there's a problem. They are smart and dedicated -- their reaction is to dive in, eager to put things right. They begin with some ideas about what the problem might be, they attempt a few fixes...

An hour later, several patches have been applied but things are still broken. The origins of the bug are now murky, because 20 additional lines of code have been piled on, and one refactor to "increase reliability". 3 hours later they're frustrated, clicking angrily on Stack Overflow, writing commit messages that just say "fuck my life".

They eventually "fix" the bug, but without gaining any insight into the underlying problem. This cycle repeats, and months later they have a brittle, poorly understood stack, with much cargo culting and many a race condition.

I know this problem well, because once upon a time, my technique was just this. Not only did it diminish my ability to write great software, but I was also missing out on some of the best learning opportunities of my career.

Luckily, I encountered a handful of wise mentors who showed me a different path: a way of life where bugs are transcendental, open-ended learning experiences, where "solutions" are used in favor of "fixes", and where we constantly question each other, in order to better ourselves.

-|--- Thesis ---|-

If handled correctly, bugs are among the most fruitful learnings you will ever come across. They will give you deep insight into how things work under the hood -- the kind of wisdom that talented hackers seek out, the kind of things you can't find in textbooks. These learnings are like great spells: find them, learn them, become more powerful...

When you encounter an aggravating, nasty, despicable bug -- don't get mad! Thank the bug. Cherish it, love it, learn from it, then destroy it mercilessly.

What follows is a method for dealing with bugs that is quite different from merely "fixing". Fixing means you did something, and now the problem is gone; fixing is not a good habit to get into. Rather than fixing, the next section is devoted to "solving": a habit that has made me a happier, healthier, and more inquisitive hacker.

-|--- Method ---|-

The "solving" method has only a few steps:

As a sidenote, if one of your colleagues has just solved a bug -- try to understand it! Ask questions like: "how did you solve it?", "what was the problem?", "did you write a test for it?". Not only will that increase your own understanding of the stack, but it will spur them into thinking more deeply about the problem. These questions are not a sign of doubt, but of respect. Asking them will cause everyone to hold each other to a higher standard.

And now, lets talk about the *how*, the so-called...

-|--- Spells ---|-

Now the question is: how do you quickly assess a problem, without (necessarily) looking at the source code or putting in additional logging? How do you walk into any situation, equipped to evaluate it and fix it? Even if it's code you're unfamiliar with ..?

For me, the answer is to have proficiency with a handful of "universal tools" -- lightweight unix tools that can be applied in many different situations. Even without being a systems expert (which I am certainly not), using these tools will actually push you into learning more and more, until one day you find that you're a hacker, with a solid understanding of the kernel, virtual memory, syscalls, tcp/ip, disk I/O, event loops, watchdog timers -- the sky's the limit! And with each new learning, more is revealed, and steadily the magic of these systems in your mind will grow, and you will be asking all sorts of new questions. Pushing boundaries that haven't been pushed!

But I digress. To bring it back down to earth, here are the most frequently used items from my toolbox (in rough order of importance): ps, top, pstree, procfs, strace/dtrace/ltrace, netstat, tcpdump, wireshark, curl, socat, and nc.

-|--- More ---|-

Less universal, though equally awesome: gdb, gcore, valgrind, perftools, jstack.

-|--- Even More ---|-

Infrequently used tools include: strings, c++filt, nm, objdump, size, readelf. These are primarily for debugging issues that arise during compilation of software, or for finding out why a certain shared library doesn't contain some symbol, etc. I won't go into detail on them, but I suggest you check them out.

-|--- Conclusion ---|-

Bugs are frustrating, but anyone can come to love them, and to experience a certain giddy delight when they find a good one and vanquish it. To master the art of this way, there are many freely-available tools of impeccable unix pedigree -- they are your spells to cast. Use them wisely! And next time you find yourself working on a bug, ask yourself: "am I *solving* this problem, or am I merely *fixing* it?".