Official Hashrocket Blog
-
Or, how Hashrocket displays pair programming gravatars on github.
The Peculiar Problem of Pairs
At Hashrocket, we pair program all the time. We also use git exclusively for source control. This presents a problem:
gitonly supports one “commiter,” but we’d like to give credit to both developers. Here’s how we hacked it.Author vs. Committer
First, a bit of background:
gitdoes support the notion of an “author” as distinct from a “committer.” This feature handles the situation where some external third-party developer has submitted a patch to a source maintainer. The maintainer, as the authoritative and github-credentialed “committer,” can commit the changeset for the external developer, who is set as the “author.” Both the author and the committer are recorded in the commit data.We did think about setting one pair developer as the author, and the other as the committer. But, then we thought of something we think is even sexier.
Set the Pairs in the Author Fields
Unfortunately,
gitonly supports one “author” as well. But, the author fields do provide us a workable place to hack in our pair attribution solution: we concatenate and store both pair developers usernames in the those author fields. In theBASHshell, that goes a little something like this:bash$ export GIT_AUTHOR_NAME='Jon Larkowski and Tim Pope' bash$ export GIT_AUTHOR_EMAIL=dev+l4rk+tpope@example.comThe convention we use for
GIT_AUTHOR_EMAILis the form:dev+pair1+pair2@example.comwhere “dev” is short for “developer.”devis just an email alias that goes to every developer at Hashrocket. The pairs are added in alphabetical order by github username. We hack our usernames into the email address using the “plus-notation” feature of email addresses.Configure the Committer
Committer for a Developer’s Laptop
As for the committer field, it’s just the developer whose machine you’re pairing on. This allows us to tell which user this commit came from, while still indicating the pair authors. Also, it allows us to simply use that developer’s github credentials for pushing up changes. So the
git log --pretty=fullafter a commit looks something like:commit 3eb7b3a155f11e56cf406f2b09dc2fd98e92b532 Author: Jon Larkowski and Tim Pope <dev+l4rk+tpope@example.com> Commit: Jon Larkowski <jon@example.com> Ebony and ivory, live together in perfect harmony.Committer for a Shared Pairing Workstation
We do have one twist on the committer. More and more, we’re using communal desktop machines at Hashrocket. We have a couple Macs Pro and a couple Macs Mini (I assure you, those are the proper irregulars plural, look it up). On these shared machines, we name the committer after the machine’s name. This allows us to know which machine a commit came from, but it’s not attached to any particular developer’s name.
bash$ git config --get-regexp user\.* user.email dev+some_workstation@example.com user.name Some WorkstationAlso, note that the shared development workstation will need to have its own public key registered on github, under your organization's main github account, not your own personal one.
github & Gravatar Photos
github displays a Gravatar photo next to a username when displaying commit information. This helps add to the “social” feel of github.
Single Developer Gravatar Photo
For you as a lone developer, github just looks up that gravatar based on the email address you gave github when you signed up.
Developer Pair Gravatar Photo
Now for a pair, we need a unique email address that github can use to look up a photo on gravator. This is where our pair developer email address convention comes into play. We just add that
dev+pair1+pair2@example.comas a new email address under our main corporate gravatar account, which is maybe signed up underinfo@example.comfor instance.Gravator lets you have as many email addresses as you one under one main umbrella account. It’s a bit spammy, in that every developer will receive a gravatar confirmation email whenever a new pair permutation signs up. But, that sounds like a “First World problem,” if I ever heard one.
Say, “Cheese!”
We typically just snap our pair photos using Photo Booth.app, since we’re likely pairing at that very moment on a Mac with a built-in iSight camera. Oh, and since we’re picky, we make sure that
pair1is on the left andpair2is on the right of the photo.
Hitch
Setting
gitenvironment variables by hand every time you switch pairs could get tedious. Ro and Les wrote a little gem called “Hitch” that makes this easy.Install it with:
sudo gem install therubymug-hitchTo set it up, follow the prompts and answer the questions:
hitchrcWhen you start pairing with someone, just call:
hitch your_pairs_github_usernameTo code solo:
unhitchTo pick from the list of your previous pairs:
hitch -iJust running hitch, with no arguments, will tell you who you are pairing with. Pro tip: Add it to your
Bonus, can hitch/unhitch from withinbashprofile to see it when you start a shell.vim, i.e.:!hitch github_username
Hardcore BASH Scripting Action
Here's a BASH script that accomplishes the same thing as hitch.
To set up a pair:
bash$ pair fry bender
Bender Bending Rodriguez and Philip J. FryTo enter a pair that's not a declared variable at the top of the script:
bash$ pair fry hypnotoad="All Hail the Hypnotoad"
Philip J. Fry and All Hail the HypnotoadTo persist a pairing, even between shell invocations:
bash$ pair -w zoidberg bender
Bender Bending Rodriguez and Dr. John ZoidbergUse pair with no arguments to see who's paired. Use pair -w with no arguments to clear a persistent pairing.
-
BizConf Venue Teaser from Hashrocket on Vimeo
-
Pair programming does present some unique problems, one that we encountered at Hashrocket was commit attribution: a commit message would be identified with one member of the pair only. This is not exactly tragic, but from the perspective of a passioniate developer, having commit credit (and accountability) is a critical and visible part of the ethos.
Tim Pope had the brilliant idea of using a group email address with tags (such as group+you@example.com) to identify the commits a pair made to git. A little bash script fu lead to a nifty command line utility that let us track pairs with our git commits. With that script as a starting point, Ro Samour wrote a pure ruby implementation:
hitchhitchwill modify your git configuration to reflect who you are pairing with. Here is ahitchcommit message usinggit log --pretty=full:commit e518dd0637e7d1d77d3bd79a645e5d0bc93eae2d Author: Les Hill and Rogelio Samour <bleep+bleep+bleep@hashrocket.com> Commit: Les Hill and Rogelio Samour <bleep+bleep+bleep@hashrocket.com> Adding show note iconThis is a commit message from
hitchon github:
hitchis open source and is available from github.Getting started
You will need to install the
hitchgem from github1:% sudo gem install therubymug-hitchYou will need a group email account with a service such as GMail that allows email tags2 (as an example,
newsis a tag in this email address yourname+news@example.com which is considered to be the same email address as yourname@example.com):You really want a Gravatar for your group email account so that the commits show the custom gravatar along with your commit messages. For every pair, register a new gravatar and associate it with a tagged email address of the form:
dev+github_user_1+github_user_2@example.comThe github usernames should appear in the local part of the address in alphabetical order after the group address local part, each github username seperated by a
+.Initialize
hitchby runninghitchrcand giving it your name, github username, and group email address when prompted:% hitchrcUsing
hitchTo pair with another developer whose github username is
pair:% hitch pairhitchwill prompt you if it does not recognize github username of your pair and save it for later use.To unpair and code solo:
% unhitchTo see who you are currently paired with:
% hitch1 Add
github.comto your gem sources withgem sources -a http://gems.github.com/2 Read about email tags on Wikipedia
-
Some people have asked why I've got a Hashrocket tattoo on my calf. The reasons are pretty biographical; 'ware ye the history contained herein. Credit for the photo goes to Travis Schmeisser.
Each Wednesday Hashrocket has a midweek get-together called Hashrocket Hot Hackers Hump Day Happy Hour (or 6H). It was a chilly January evening and there were almost a dozen rocketeers milling about at the local martini bar when Sal casually asked if I wanted to go get a Hashrocket tattoo with him.
Of course, inebriated as I was, there wasn't much chance I was going to turn down the idea of getting a Hashrocket tattoo.
Yet, there was a time when I considered tattoos silly things that a person gets to show how edgy he or she is, or to indicate an extreme level of I-will-kick-your-ass. Now I've got one. So why choose to get a Hashrocket tattoo?
I've become quite enamored of Hashrocket since I arrived in Atlantic Beach at the end of March in 2008. I was still a recovering burn-out when I came down here, and somehow Hashrocket refilled my spiritual-coding cup. That sounds extreme, but it's just the way things are.
I spent five years at a county-level government IT shop as a web programmer. I serviced sixteen department websites and also wrote an intranet from the ground up that served 1700 employees while I was there. My boss was lost on anything past FrontPage and for help I had only a string of limited-engagement, part-time assistants of varying levels of skill.
I burned out. Totally and completely. I threw away all of my computer gear and went back to auditing hotels overnight and contemplatively staring at the moon. After about a year, I had the realization that software is what I do, and no matter how burned I felt or how much I wished otherwise, it seemed that was the value I would provide to society.
I began the slow road back to development by working as a part-time webmaster at a non-profit. Then Tiger turned me on to Ruby on Rails, and things started to happen inside of me. A strange sensation that, after experiencing a few times I was able to place: happiness. I was happy writing ruby code. I was happy using the rails framework. Just typing out each line of code somehow made me feel good.
That's how I came to be a Rails consultant in Madison, Wisconsin. My first paid site was completed in November, 2007, and I've never looked back.
When Tiger invited me to come down and see how Hashrocket does things in March of 2008, I was excited to see the magic sauce that had both he and Lark raving about the company. I didn't expect to be offered a job, but I was, and it's been the best thing that's happened to me.
In many of the same ways that ruby and rails took away the pain of coding for me, Hashrocket has taken away the pain of work and replaced it with happiness. Pair programming has made me a more effective and efficient programmer. Communicating all the time has taken away all the bad conversations, because nothing has time to fester. Test-driven development gives me a level of confidence in my code that makes me unafraid to change even systems I haven't looked at in ages. I feel encouraged to excel as an individual rockstar within the community, even on the Hashrocket clock. As Les Hill (in the photo, on the right) is wont to remind us in his blog posts, working at Hashrocket is like attending an ongoing seminar.
All of these reasons, from my discovery of Ruby on Rails through joining Hashrocket and even becoming a presenter at local user groups, this is why I have a Hashrocket tattoo on my calf. If I never have another experience that I want to commemorate with a tattoo, I'm glad I've had this one. -
Corey Haines at Hashrocket / RMM from Hashrocket on Vimeo.
Join us in welcoming Corey Haines (aka The Software Journeyman) to Hashrocket for some pair programing with Stephen Caudill. Corey Haines travels around the country pair programming with whoever will feed him a scrumptious vegetarian meal and give him a spot to put his air mattress.
In the video, Corey arrives at Hashrocket (accompanied by Cory Foy) and introduces himself to the team. Then Obie, Corey and crew proceed to discuss software craftmanship and the project that Corey and Stephen will be working on for the duration of the week: RMM - the Rails Maturity Model application. A great conversation ensues that really hits home on what RMM is all about. -
While recently discussing if a power law applies to bugs at the ongoing seminar I sentimentally call work, I noted a corollary to Linus’s Law.
Linus’s Law states:
Given enough eyeballs, all bugs are shallow
The corollary is:
Most bugs fall to the second pair of eyes
That is, just having one other developer look at a bug will likely resolve it. Of course, developer’s who are pair programming already know this and are ahead of the game.
On a related note, pretty much every developer has been in the following situation. You are called over by a collegue to look over a bug, often accompanied by the statement, ‘This should work.’ You give a cursory examination and you resolve the problem with a quick change, often one that is obvious (after you pointed it out!)
If you come away from that experience thinking ‘Fresh pair of eyes’ or some such, good for you, you passed the test.
If you come away from that experience thinking ‘I am so awesome!’ or some such, not so good for you, you are probably not even competent.
-
One of the most persistent and pernicious stumbling blocks I see when a team is starting out with Agile1 is equating story points and velocity directly with hours.
Often this stumbling block is not apparent immediately; it often crops up after a few iterations have passed. As an example, a stakeholder who was new and unsure about agile, was eager to have a release a month into development as an affirmation of the decision to go with an agile approach. After we had prioritized the feature set for the release, he turned to me and said, “Let’s see, we have 78 points in the release, so we can deliver the release four weeks from today.”
My answer, “Not exactly”, surprised him.
If the velocity of the team is an evidence-based number using the teams own story point estimations, how could such a statement be incorrect? There are two related reasons such a statement needs to be qualified to be useful, the Planning Fallacy and Velocity is not Destiny2.
The Planning Fallacy
As described in the Overcoming Bias blog, when students were asked in an experiment to forecast when they would complete an assignment:
“Even when asked to make a highly conservative forecast, a prediction that they felt virtually certain that they would fulfill, students’ confidence in their time estimates far exceeded their accomplishments.”
More generally, this phenomenon is known as the “planning fallacy”. The planning fallacy is that people think they can plan, ha ha.Since giving up on planning is not an option, we can make our estimates better through various techniques, one of which is to plan based on similarity to other known efforts, instead of estimations of time.
a cross-cultural study, found that Japanese students expected to finish their essays 10 days before deadline. They actually finished 1 day before deadline. Asked when they had previously completed similar tasks, they responded, “1 day before deadline.”
Agile mitigates the planning fallacy by eschewing estimates in hours or days and using story points3. Story points are abstract esitmations based on similarity; a 4 point story should be of the same relative size or complexity as any other 4 point story. Time is not a factor with story points.
Velocity is not Destiny
Given that story points yield better estimates, and velocity is an evidence based aggregate number smoothing out variations in individual stories, why qualify saying we can deliver four weeks from today? Velocity is a continuous measurement of the past performance of your team and is not a constant. The velocity of your team is a dynamic variable giving you direct and constant insight into how they are progressing. A future iteration has no velocity, as it has not been (and cannot yet be) measured. We can make predictions about future iterations based on velocity, but we cannot make any iron-clad guarantees.
An analogy might help explain how to think of velocity. You are taking a long automobile trip you have never taken before, where you will leave at 9 a.m. and travel 250 miles to arrive at your destination. Your car will accurately report your average velocity on the trip. 125 miles into the trip, your car reports an average velocity of 62.5 miles per hour, and you are currently on the highway going 70 miles per hour. How long do you have left, and how much money would you bet that you are accurate within one minute? Intuitively, most people understand that although the very clear math says you are 2 hours away from your destination, there are too many pitfalls such as traffic, accidents, road work, weather conditions, signals, etc to make a prediction that is accurate to the minute. (With your final average velocity, the very clear math will yield an exact and accurate result.)
Making the Release
There are many pitfalls in any software project, the one which was making me cautious about our release was a common one for novice stakeholders, Release Induced Focus. RIF is the tendancy for novice stakeholders to only really pay attention when they have an immediate release pending, leading to a spike in the number of stories rejected and bugs filed, and a corresponding drop in velocity. Given an experienced stakeholder who is rigorous about acceptance, the bug tax on velocity is taken out of velocity over time; with RIF the entire bug tax on velocity is paid in one or two velocity sapping iterations.
After explaining this, and using velocity as a predictor, we adjusted our plan for the release. We reprioritized the backlog to front load all the required features, moved unneeded features out of the release, and had the stakeholders exercise more rigor when doing their acceptance. The release was made in four weeks, with the team and stakeholders delivering tangible value without the long hours and anxiety provoked by more static approaches to software project management.
Velocity is a Dynamic Measure
Velocity is a continous measurement of your team’s past performance over time; all of the previously known and unknown risk variables have been surfaced, made known, become constants, and reduced to one accurate numeric measure. What velocity does not tell you is what the future holds. To quote every mutual fund prospectus ever written:
Past Performance is no Guarantee of Future Results
Understanding that velocity is accurate and dynamic is key to using it for iteration planning. Most other software project management methodologies pretend that the project is static in all aspects, that all variables are known and quantified. Agile acknowledges that a software project is a dynamic system, and provides simple evidence based tools to manage the dynamism inherent in building new software.
1 I use the term to encompass the variations of Agile methodologies that use story points and velocity.
2 I first heard this poetic phrase from Amy Kline, an awesome scrum master and friend.
3 Peter Jackson has written a great summary of why we use story points in The Case for Story Points .
-
Hashrocket has a few core philosphies that are essentially required for team members. Agile development processes, an adherence to ruby and rails best practices and idioms, and a firm belief in pair programming are three of the biggies. Of these three, pair programming is the technique that has changed me most, and required the most change in my thinking. And not surprisingly, pair programming has proven the most critical to my success at Hashrocket.
I knew of pair programming before I came down to check out the Hashrocket way. And I even knew some people who personally advocated pairing. But I was that guy. You know, the one who does his best work in the morning with his headphones on and drinking an obscenely large coffee. The guy who types madly for a few select hours a day and, during those few hours, produces as much as regular-speed developers do over the course of a day.
There was definitely a transition period when I would only have a pair for half a day here and there where I was still that guy even though I was with Hashrocket. But ultimately, I have come to realize three distinct situations in which pairing is far superior to working solo that have cemented my belief that pair programming is superior to solo programming.
First, a big part of coding is constantly seeing the big picture. I can't tell you how many times I've sat down to solve a problem and been blind-sided by a particularly difficult routine to write, or maybe a supremely complex data model. Or maybe I find that I have to monkey-patch some existing code. These situations inevitably make me sit and spin my wheels for a minute... sometimes they even made me get up out of my desk and waste some time thinking about solutions. Occasionally I would give up and leave early for the day. With a pair sitting next to me, however, a simple discussion of the situation typically leads to a more obvious solution. In worst case scenarios, we are still more productive by keeping different parts of the solution in each of our heads.
Other times, that obscenely large coffee is just a bit too much and I get going a little too fast. I make simple typos or logic mistakes that I don't catch until I've already started running tests, and then I end up wasting 30 seconds waiting for tests to tell me that I've entered a period instead of a comma. As an over-caffeinated developer, I can't tell you how frustrating these little typos and large amounts of wasted time become. With a pair by my side, it's very rare thing that I get to the end of any line of code without one of us noticing a typo.
The last situation that I've noticed an improvement in development speed while with a pair is after lunch. I don't know about anyone else, but I get very yawny after lunch. So instead of spinning my gears alone and staring off into space, my pair and I typically find myself getting into more code discussions, or letting my pair drive while I keep an eye out for typos. Either way, I'm way more productive than I would be while browsing digg.com.
For all of these reasons, I've become a huge believer in pair programming. And since becoming a believer in pairing, I've found myself extending the idea of pairing to other areas of my life: I pair at the gym with my gym buddy, I have a haircut buddy, and I typically line up my schedules for things like DMV runs with other folks.
But the real reason I wanted to talk about pairing today was my latest pair adventure: presenting. I'll be honest; I think I'm a terrible presenter. There was a time in High School when I won awards for giving speeches. That was almost a decade ago, though, and I'm a little more than frightened about getting in front of a large group of people.
Being at Hashrocket, it's no surprise that I'm surrounded by people way smarter than me. In this situation, having a more senior pair is quite possibly the best thing I could have asked for in my career. And also, being part of a brain trust such as we are, it's only natural that all of us Rocketeers should be presenting at a few conferences a year. And let me tell you, I'm not the only Rocketeer terrified of getting up in front of a crowd.
So my solution: present two topics at once in a stream-lined talk about two different topics. Jax CodeCamp is coming up, and I'm hoping to present a talk discussing distributed development with git and rapid web application development with rails. As a side effect we'll also be discussing pair programming, but that should be in a mostly meta sense. The idea is for one of us to create an application and commit it to a git repo and from then on to ping-pong back and forth, making incremental application improvments and highlighting the uses of git in distributed development while we do so.
The benefits of doing so should be two-fold: first, to show how awesome ruby on rails and git can be, and second to decrease the likelihood that either me or the other speaker will run out of the room screaming.
Wish me luck?
