What the Fork was that? A forking post mortem.

As many of you already know there was a 'hiccup'  in the bitcoin network this week.  It caused a flurry of articles, some good, some panicked and others that were outright alarmist.

To be honest, while I was aware of the issue as it transpired it did not register with me as much of threat.  To me it was just a temporary problem, one that we have seen before and undoubtedly will see again.  While it was slightly unusual that there had to be any sort of human interaction, it was far from an unexpected event and had been planned for well in advance.

What Happened?

The technical description to what happened is an unintentional Hard Fork.  I have detailed the blocks and times in a simple graphic (to the right) if you want to follow it more closely as I go into what happened.

A hard fork occurs when there are a sufficient number of bitcoin clients on the network that disagree on the rules about how blocks are created and recorded in the universal block chain and it leads to a split in the chain, one set of bitcoin clients follow one split in the chain and another set follows another split in the chain.

When talking about bitcoin we call these splits a fork, the analogy is that it is like a fork in the road that one group follows the right fork and another follows the left fork.  In theory there can be more than one split in the chain to follow leading to multiple forks, but that did not happen in this instance.

This particular fork occurred because version 0.7.2  and version 0.8 bitcoind (the bitcoin reference client) differed slightly in how they thought a block should be built.  Version 0.8 though that blocks should be allowed to be a certain size and version 0.7.2 disagreed.

When bitcoind version 0.8 was released nearly a month ago on the 19th of February the intention was for all miners to replace their older version 0.7.2 clients (released 14th of December 2012).  Intentions being what they are though, it seems that many did not.  There are a variety of reasons not to upgrade but this set the stage for what was about to happen.

If you wanted to point a finger of blame at something, you would be firmly pointing your finger at this version of Block 225430 mined by Slush's pool that was mined at 10:46:35 PM on the 11th of March, 2013 (all block times are in UTC).  All in all it is not an unusual block except for the fact that it was rather large, 974.6865234375 KB to be exact.

If you were like me and weren't watching the bitcoin protocol using a 0.7.2 client at the time you would have missed what happened next.  If you were like me, and watching with a version 0.8 client you would have seen what I saw, block 225430 was mined and added to the block chain on top of block 225429, followed by block  225431 at 10:46 pm then block 225432 at 10:56 pm and so on.

However if you were watching with a version 0.7.2 client you would have seen block 225429 at 9:33 pm and then a block 225430 at 10:39 that was rejected because it was too large.  In fact, you would have seen this new block 225430 rejected several times as other 0.8 clients insisted that it was valid but your 0.7.2 client would continue rejecting it (and subsequent blocks that were based off that block) because it was just too big as far as 0.7.2 was concerned.

This could (and some could argue should)  have gone on for ever effectively forcing people to upgrade to version 0.8 in order to be able to continue to use and validate the block chain, but that is not what happened, what happened next was a hard fork.

1 hour and 9 minutes after the last valid block for 0.7.2 (block 225429) a new version of block 225430 was mined and reported to the network by the IP address 46.250.25.35 creating a new version of the chain (or a fork) that could be followed by version 0.7.2 clients, meanwhile version 0.8 clients had already mined another few blocks (all the way to block 225535).

A fight then ensued between the 0.7.2 clients and the 0.8 clients each reporting that they had the valid block chain, 0.7.2 insisted that 0.8's version of block 225430 was invalid and 0.8 insisted that its chain was valid and longer than 0.7.2's version so it was the main chain.

Who did it affect?

Most would not have noticed the problem if it weren't for the considerable press coverage, in the end it was only miners (solo, or the pool that they belong to) and merchants that were affected.

Miners were affected because there were apparently two valid chains to work on and depending on the client that they were running they could be working on either the 0.7.2 or 0.8 version of the chain and any block rewards would only be valid in that version of the chain.

Merchants were probably the most affected, in effect they for a short amount of time were running with two general ledgers and any particular transaction might of shown up in either version, possibly neither or even both!

What should I have done?

Unless you are a solo miner or a bitcoin merchant the safest and proper action you should have taken is nothing.  It did not matter if you were running 0.8 or 0.7.2, in fact it does not matter even now (but you should probably upgrade when 0.8.1 comes out, not because of the fork, just because upgrading your software is a good thing to do).

If you were a solo miner on the other hand you should have downgraded to version 0.7.2 just for the mean time.

If you were a merchant you would have been well served by following MtGox's lead of suspending all transactions until the problem was resolved.

How was it fixed?

Truth be told, sooner or later it would have fixed itself as people with 0.7.2 clients realised they that they were not getting the latest blocks on their chain and upgraded to 0.8 but that may have taken days (or possibly even weeks).

What ended up happening is The Bitcoin Project lead by Gavin Andresen released a post asking all miners to downgrade their bitcoin software to 0.7.2 effectively killing the 0.8 chain because the 0.7.2 chain would not accept the fork block 225430.

Alternatively Gavin could of asked everyone to upgrade to 0.8.0 and bitcoin would have just continued to run with the new chain including the large block at 225430.

Did anyone lose anything?

Yes.  Not directly through the fork itself though, but because of the panic that arose.  It could have been coincidental but there was a large sell off in bitcoin about the same time that the fork was discovered.

The value quickly recovered after the problem was properly identified and it ended up being less than a 5% drop, but for the person/persons who did the sell off (it amounted to about 25% of bitcoins value) they would have unfortunately taken quite a loss in the panic.

Will it happen again?

Yes, in fact it already has.

There are two versions of  blocks 225564, 225637, 22575, 225830 and 225853 already, these are what we call soft forks or forks that solve themselves without any interaction from us.  They are caused by two separate miners solving the same block at or about the same time.  The bitcoin network resolves these block conflicts by choosing the block that is the largest and then continuing the chain using that block.  The other block not chosen is then called an orphan block and is removed from the network automatically.

What have we learned?

While this hard fork was unintentional it has been a useful lesson.  Sooner or later we are going to want to do this on purpose to change the way the bitcoin network works and that will require a hard fork.  There are many reasons that we will want to do this, just to name a few off the top of my head that have been talked about recently are:

• Make the block size smaller so a the full chain does not take as much hard drive space
• Make the block size larger to enable more transactions per block (and less fees)
• Increase the decimal point in bitcoin from 8 places (changing the smallest denomination from 0.00000001 to something like 0.00000000000001)

There are more reasons for a hard fork, and some reasons that have not even been imagined yet but they are all valid in their own way and would require us all to agree to do a hard fork.

We have also learned not to panic (yeah right).

Conclusion.

While I don't want to lessen the importance of this event, it was serious, it did require intervention and it did impact on the price of bitcoin.  I would also like to point out that it was a good test run for bitcoin, sooner or later it had to happen, and later we may force it to happen ourselves.