Demystifying "Recent Blockhash" Expiration

Searching for answers within contradicting documentation

Background

Solana is a high-performance blockchain platform designed to support decentralized applications and crypto-currencies. It is known for its innovative approach to maintaining consensus across its network of nodes, using a mechanism known as Proof of History. Proof of History is a unique consensus model that allows Solana to keep track of the order of events and the time that has passed between them without relying on external time references. This is achieved by using a sequential preimage resistant hash that is continuously run, creating a historical record that proves that a specific event has occurred at a specific moment in time.

Blockhashes play a crucial role in this process. A blockhash is a unique identifier for a particular block in the Solana blockchain. It is generated by hashing the content of the block and the previous block’s hash. This ensures the integrity of the data and provides a reference point for validating transactions. This post aims to explain the roles of the “recent blockhash” in Solana transactions and try to make sense of how it exactly works, despite contradicting literature. This post is inspired by this thread on Stack Exchange, currently viewed only 150 times.

Slots & Blocks

Before diving deep into why blockhashes are important, we must understand the concepts of slots and blocks. Slots serve as the clock of Solana. Each slot represents a duration of time, currently set to 400ms, but usually ranges from 400-600ms in practice. During each slot, a selected validator based on a predetermined schedule, known as the leader, is tasked with producing a block. A block contains a set of transactions along with some metadata. Whether a slot contains a block depends on whether the scheduled leader actually proposed a block at that slot and whether the proposed block was verified by other validators by getting the required number of votes. Therefore, it is essential to understand that during each slot, a block may nor may not be produced and at most one block is produced. This is the reason why the slot and block height on Solana diverges: block height will always lag behind slots, and the difference will only increase over time.

Given the current slot and block height of Solana mainnet, more than 92% of slots have successfully produced blocks. Due to this high parity, “slots” and “blocks” have sometimes been used interchangeably in the Solana community, which can easily confuse new developers or users entering the space. If you would like to read more about slots, blocks, epochs, and validation, Helius has an excellent blog post that dives deeper into these concepts.

Transactions

Transactions can be though of as small packets of data that are propagated through the Solana validation infrastructure (transactions are actually converted to UDP packets by RPC servers before sending to validators). Each packet contains data about the instructions to execute, accounts to read from or write to, and a “recent blockhash”. These data make up all the necessary information to determine what the transaction is supposed to do and if the transaction is valid or not.

When a transaction is submitted to the network via the “sendTransaction” RPC method, it is first sent to an RPC server. The RPC server checks the incoming request and, if it is a valid Solana transaction, then send the transaction packet to the current and next validator on the leader schedule. The validator’s Transaction Processing Unit (TPU) receives the transaction, verifies the transaction, executes it, and propagates it to other validators in the network. In the verification step, not only are the signatures checked, but the slot of the recent blockhash must be less than 152 slots old for the transaction to be valid. But wait, is this really how this works?

Recent Blockhash

In the context of transactions, blockhashes are used as a form of cryptographic timestamp. When a transaction is created, it includes a “recent blockhash”. This blockhash is used to proves that the transaction was created recently and is not a replay of an old transaction. For instance, if the same transaction (including its recent blockhash) was submitted twice (by mistake or by RPC server retrying transaction), the network can tell that they are identical, and were not intended to be executed twice. This solution to double processing avoids the usage of transaction counters, like nonces on Ethereum, that can often lead to later transactions getting stuck waiting for earlier ones.

Using a recent blockhash also allows the network to discard old transactions efficiently, as recent blockhashes are only valid for a certain number of blocks. Once the recent blockhash expires, the user does not have to worry about the transaction possibly going through in the future. So what determines when a recent blockhash should expire? Let us explore the official Solana documentation to find out.

Solana Documentation

Beginning by reading about “How does transaction expiration work?” on the official Solana documentation, we quickly learn that

If the validator can’t find a slot number for the blockhash or if the looked up slot number is more than 151 slots lower than the slot number of the block being processed, the transaction will be rejected.

In addition, we are presented with pseudocode and an example scenario

Transaction has expired pseudocode: currentBankSlot > slotForTxRecentBlockhash + 151

Transaction not expired pseudocode: currentBankSlot - slotForTxRecentBlockhash < 152

Let’s walk through a quick example:

1. A validator is producing a new block for slot #1000

2. The validator receives a transaction with recent blockhash 1234... from a user

3. The validator checks the 1234... blockhash against the list of recent blockhashes leading up to its new block and discovers that it was the blockhash for slot #849

4. Since slot #849 is exactly 151 slots lower than slot #1000, the transaction hasn’t expired yet and can still be processed!

5. But wait, before actually processing the transaction, the validator finished the block for slot #1000 and starts producing the block for slot #1001 (validators get to produce blocks for 4 consecutive slots).

6. The validator checks the same transaction again and finds that it’s now too old and drops it because it’s now 152 slots lower than the current slot :(

Later in the same page, this behavior is reiterated multiple times:

Currently, Solana clusters require that transactions use blockhashes that are no more than 151 slots old.

As mentioned before, blockhashes expire after a time period of only 151 slots…

Based on this information, it should be pretty clear that a recent blockhash will be valid for 151 slots. After these 151 slots elapse, any transaction with this recent blockhash would be dropped and not executed. However, if we dive deeper into the other pages of the official Solana documentation we find contradicting descriptions. For instance in “How RPC Nodes Broadcast Transactions”, we learn that

By default, RPC nodes will try to forward transactions to leaders every two seconds until either the transaction is finalized or the transaction’s blockhash expires (150 blocks or ~1 minute 19 seconds as of the time of this writing).

Hmm, this seems interesting. So the blockhash expires not based on the slot, but based on the number of blocks that elapsed? In the terminology definition page for “skipped slot”, we find corroborating descriptions again:

A past slot that did not produce a block, because the leader was offline or the fork containing the slot was abandoned for a better alternative by cluster consensus. A skipped slot will not appear as an ancestor for blocks at subsequent slots, nor increment the block height, nor expire the oldest recent_blockhash.

Since the documentation is contradiction itself, it is not easy to determine if the recent blockhash expires by slot count or by block height. Since we can not blindly trust either description, the only way to find out is to understand the source code. Based on the code, it seems like the latter is true: the blockhash can be used for a transaction if it is within the 150 most recent blockhashes.

Consequences of Bad Documentation

This use of the “recent blockhash” is core to the Solana blockchain and is part of what sets Solana apart from other chains. However, the behavior was explained in multiple contradictory ways on the official documentation. Contradictions can create confusion, erode trust in the reliability of the documentation, and result in wasted time as developers struggle to reconcile conflicting information. This situation may lead to a lack of confidence in the overall quality and reliability of the product, making it challenging for users and developers to navigate and utilize the blockchain with confidence.

Moreover, incorrect information in the official documentation can easily spread in the community. Here are just a few examples where it has happened with this topic:

https://omniatech.io/pages/solana-transaction-confirmation/

https://blog.fordefi.com/warping-time-for-solana-defi-reconciling-expiring-nonces-and-institutional-policy-controls

https://docs.thornode.io/guides/navigating-through-congestion-understanding-transaction-confirmation-and-performance-in-solana

Confusing blocks and slots have also led to some blatantly incorrect code samples on the official documentation for “Customizing Rebroadcast Logic”:

  const blockhashResponse = await connection.getLatestBlockhashAndContext();
  const lastValidBlockHeight = blockhashResponse.context.slot + 150;

  let blockheight = await connection.getBlockHeight();
  while (blockheight < lastValidBlockHeight) {
    connection.sendRawTransaction(rawTransaction, {
      skipPreflight: true,
    });
    await sleep(500);
    blockheight = await connection.getBlockHeight();
  }

Recommendations

Although in this example, it does not matter for most users if the recent blockhash expires after 151 slots or 150 blocks, as most applications just fetch the latest blockhash to ensure the best chance of a successful transaction. However, there are some niche cases where this matters. For instance, using an older blockhash that is about to expire can achieve an “immediate-or-kill” transaction or specify an exact time window within which the transaction can execute without worrying about the “maxRetries” implementation. In addition, if slots are used to determine when a transaction is dropped, one may find themselves surprised about the transaction actually succeeding when they thought it was already dropped.

Furthermore, the documentation for RPC methods “isBlockhashValid” and “getLatestBlockhash” are equally confusing. One may think that a “valid” blockhash would be one that can still be used in transactions. Apparently, a blockhash is defined as “valid” if it is within the 300 most recent blocks, not 150 as one would think. I believe some additional context could be given either in the method description or in the terminology for “blockhash”.

All in all, while it may seem like a minor concern, this issue serves as a reminder of the crucial importance of maintaining good documentation. This post does not suggest improvements to the network in terms of fees or scheduling. Instead, it suggests improving Solana by placing more emphasis on maintaining accurate documentation. Although the official documentation is open source, the content should be vetted before merging and updates should be frequently made to reflect protocol changes. For this particular issue, I will not submit a pull request to the docs, but I encourage anyone who has made it this far to do so. For me, this serves as a sign that at least someone is reading :)

Finally, in the spirit of Solana Scribes, I want to echo 0xMert’s sentiment on the importance of writing, which led to the creation of this hackathon. Clear and accurate documentation is paramount for ecosystem growth and attracting builders. It serves as a crucial guide for developers, easing onboarding and fostering efficient contributions. Well-documented projects not only accelerate collaboration but also attract a diverse community, promoting innovation. Beyond immediate benefits, clear documentation reduces barriers to entry, encourages knowledge sharing, and establishes a foundation for long-term sustainability, contributing to a robust and thriving ecosystem.