No CommentsWhile Bitcoin Examiner last talked about the major 0.9 release that is coming back in July, the core development team has been working hard towards the conclusion of the update. In the meantime, they have fixed two minor bugs and reviewed over 300 pull requests.
This Thursday (24), Gavin Andresen, the chief scientist at the Bitcoin Foundation, decided to publish an update to let us know how the team’s work is going, highlighting some changes that are already part of the 0.9 source tree. Besides, he also talks about some features that the team hopes to implement soon. However, there’s still no forecast regarding the release date of the update.
Andresen started by talking about the possibility of embedding data in the blockchain.
Some people feel that the blockchain should be a public resource available to use for whatever people like, as long as they pay sufficient transaction fees to make it worthwhile for miners to store their transactions. Others feel that the blockchain should only contain the data necessary to validate Bitcoin transactions, and any other data should be stored separately.
And while the debates raged, clever people found lots of creative ways of embedding data into the blockchain.
The foundation’s chief scientist admits that he merged the pull request #2738 – “Relay OP_RETURN data TxOut as standard transaction type”, which means adding a new standard transaction type, which permits small amount of data to be attached to a transaction, in the form of an additional TxOut that is provably prunable, according to the proposal – “with some reluctance”.
The goal is to give people a way to do what they clearly want to do, which is “associate extra data with a transaction that is secured by the blockchain”, but in a responsible and balanced way. This request lets developers “associate up to 80 bytes of arbitrary data with their transactions by adding an extra ‘immediately prune-able’ zero-valued output”.
Why 80 bytes? Because we imagine that most uses will be to hash some larger data (perhaps a contract of some sort) and then embed the hash plus maybe a little bit of metadata into the output. But it is not large enough to do something silly like embed images or tweets.
Why allow any bytes at all? Because we can’t stop people from adding one or more ordinary-looking-but-unspendable outputs to their transactions to embed arbitrary data in the blockchain.
What do I mean by “immediately prune-able?” The form of the up-to-80-byte transaction output (“OP_RETURN <data>”) is such that it can never be used as an input for another transaction– so it can theoretically be forgotten by everybody except for machines that want to keep a full record of every single transaction (“archive nodes”). That is a big improvement over the various hacks people are using today to associate data with their transactions, and will be more important in the future when we implement code that saves disk space by keeping only unspent transaction outputs and not every old block.
The core code has no easy way of creating these new transaction outputs– you have to create them yourself using the raw transactions API. And there are no plans to display the data in Bitcoin-Qt, so you don’t have to worry about somebody sending you a few millibits and attaching a short-but-annoying message to the transaction.
From a theme that is still controversial, we move on to autotools. Andresen explains that the team “switched from using hand-coded, platform-specific Makefiles and qmake to automatically generated with autotools Makefiles“. And lets not forget about the work done by Mike Hearn, who helped to make the payment protocol more complete by adding a very necessary FAQ section, posted to the bitcointalk.org forum.
Moving on from what has been done to the list of work in progress, Andresen highlights some features that “hopefully going to make it into the 0.9 release” like the no-wallet mode and bitcoin-cli.
bitcoind is increasingly being used as an “edge router” — used as a firewall between the full Bitcoin peer-to-peer network and some custom wallet or mining software, to filter out any invalid blocks or transactions that might tickle bugs in the custom software’s implementation of the Bitcoin protocol.
The 0.9 release should contain Jeff Garzik’s “-disablewallet” mode, which lets bitcoind run entirely without a wallet, making startup faster and using less run-time memory.
We are (slowly) moving from a monolithic code base that does everything (wallet, RPC, network, blockchain storage/validation) towards more modular internals with fewer dependencies between the different pieces. Another change you should see in the 0.9 release is moving away from the bitcoind executable functioning both as a server and as a RPC client– Wladimir J. van der Laan split the RPC client functionality (“tell the running bitcoin daemon to do THIS”) into a separate executable, ‘bitcoin-cli’. The RPC client code will eventually be removed from bitcoind, but will be kept for backwards compatibility for a release or two.
After this point, he moves on to talk about the optimization of the download process of the blockchain. Finally, this feature will enable a choice when downloading the entire chain, making the full download optional. “Once it has been thoroughly tested, we hope to merge headers-first sync for the 0.9 release. The idea of headers-first sync is to download the blockchain in two stages”, he writes. These two stages are:
1. Download just block headers from your peers. This is very quick, because each block header is just 80 bytes.
2. Once you’ve got one or more chains of block headers, pick the one with the most proof-of-work and then fetch just the full blocks for that chain.
Getting the headers first makes it possible to implement parallel download of blocks from multiple peers, which increases performance. And it has security and privacy benefits, too–an attacker can’t make you waste space or time storing or serving an ‘orphan’ blockchain fragment that doesn’t connect to the real blockchain.
Finally, we move on to the possibility of developing smarter transaction fees, since the current ones are hard-coded into the Bitcoin-Qt/bitcoind wallet software and the related rules are a “collection of heuristics that evolved over the last four years”. Basically, Andresen calls the the fee-handling code “a complicated mess that will stop working as soon as transaction volume doubles a couple of more times”. So, it desperately need some improvements.
In theory, it should be simple: transaction fees should to be set by a give-and-take between the people creating transactions (who want to pay as little as possible) and the people validating and storing the transactions (miners, who would like to be rewarded with higher fees).
And the code that miners are using today to select transactions is simple: they fill their blocks with the highest fee-per-kilobyte transaction (and highest priority transactions, if they choose to set aside some space in their blocks for free transactions).
I’ve been working on teaching the wallet code to estimate how low a fee (or priority) a transaction needs, at the moment it is sent, to be accepted by miners and included in the next block or three. The estimates are based on watching transactions as they are broadcast on the network and keeping track of which of those transactions are accepted into blocks.
The danger with estimating transaction fees is miners have an incentive to try to game the estimate to make transaction fees higher. For example, if the estimate was based on the average transaction fee for all transactions in the last N blocks, miners could add very-high-fee pay-to-self transactions to the blocks that they mine to drive up the average. However, by only considering fees for transactions that have been broadcast on the network that threat is eliminated– miners could broadcast very-high-fee pay-to-self transactions, but would end up paying those high transaction fees to other miners. The transaction estimation code also uses median transaction fees, not averages, to make it much harder for a minority of transactions to influence transaction fees.
If most of the miners got together and agreed to all broadcast lots of high transaction fee transactions they could succeed in driving up the average transaction fee… but that isn’t going to happen because it would be easy for miners to “free ride” by breaking the agreement and just collecting the higher transaction fees for themselves.
All of the above will work well for fully-validating software like Bitcoin-Qt/bitcoind. More thinking is needed to figure out how lightweight simplified payment verification software (like Multibit) can better estimate transaction fees, because today they can’t calculate how much transaction fee each transaction is paying.
All that being said, what about some predictions? Well, like we mentioned in the beggining of the article, Gavin Andresen doesn’t make any, so we’ll just have to wait until the release is finished and implemented. “Our bottleneck is still code review and testing”, he says, inviting all the C++ experts to write some unit tests and help to eliminate as many bugs as possible before the release.
Andresen also invites anyone who is “comfortable running bleeding-edge code from the command line, downloading binaries created by the pull-tester, running them on the -testnet, and reporting any bugs (or reporting ‘works for me on Ubuntalicious 11.11 64-bit’ in the pull request)” to participate.
But there even other ways to help: “write a test plan for a pull request that you’d like to see pulled. If you’re really motivated to help get something into Bitcoin-Qt/bitcoind, offer some Bitcoin bounties to the first person (or people) to run through a test plan and report success or failure; I’ve had good success recruiting people from the #bitcoin IRC channel to do testing of major features by giving out modest (e.g. 0.11 BTC) bounties”.
