Golden Encyclopedia | What is the Bitcoin memory pool? How does it work?

Source: Bitcoin Magazine; Translated by Wuzhu, Golden Finance

Everyone who has used Bitcoin has interacted with the mempool. So, what is the mempool?

From a technical perspective, there is no such thing as a "unique" memory pool. Each Bitcoin full node operates its own memory pool, which is a cache of valid Bitcoin transactions that have been broadcast to the network but have not yet been confirmed in a block. Nodes exchange messages with each other to see which transactions they have or do not have, and to exchange transactions they do not possess.

Each memory pool is essentially an independent "island" with its own set of unconfirmed transactions, and sometimes even its own configuration variables and settings. The size of the memory pool can be configured, with a default of 300 MB. Additionally, there is a minimum fee rate that adjusts dynamically and can be configured to a specific value. This is used to determine which transactions will be kicked out of the memory pool when it is full and more transactions continue to flow in. There are also some other configurable options, such as the datacarrier and datacarriersize options, which affect transactions containing OP_RETURN outputs.

Different nodes run memory pools for various reasons, so their demands also differ. Ultimately, the satisfaction of these demands is achieved through all nodes synchronously operating their respective memory pools and interacting with each other.

Think of each mempool as a real pool that is connected to each other by an underlying channel. The larger the mempool, the deeper the underlying pool. Miners, exchanges, block explorers, these will be the deepest pools. They all have their own motivations and want to understand every unconfirmed transaction waiting to be packaged into chunks. miners, to ensure that they have the most profitable transaction in the next block. exchanges, to ensure that they are aware of all pending transactions. Block explorers, because their entire service is trying to present the dataset about the blockchain and the mempool as completely as possible. Your average node really just needs to be deep enough to contain the highest rate portion of the "mempool".

Now imagine each transaction as a drop of liquid. The higher the fee, the greater the density of the liquid. This liquid flows through channels between mining pools, and after reaching each pool, a drop of liquid received will be replicated and sent through the channel to any other mining pool that has not yet received that liquid. As the mining pools fill up, when the liquid overflows, the liquid with lower density (lower fee) will overflow the edge of the pool first.

In the end, some lucky miners will scoop out a certain amount of liquid from the bottom of the mining pool and pour it into the latest glass trough, forming a long winding glass trough filled with liquid, which will stay there forever (blockchain). This is just an intuitive way to think about the system and cover most of its dynamics.

This arrangement of interconnected mining pools offers different uses for different users.

Trader

When users conduct transactions, the memory pool serves two purposes. First and foremost, it sends their transactions to miners. If a transaction does not enter a miner's memory pool, it cannot be packaged into a block. Memory pools are interconnected and share transactions, ensuring that once a transaction is placed into one memory pool, it will eventually enter the memory pools of all miners. Having a strong and decentralized network that can ensure transactions ultimately reach all miners from users, regardless of changes in network connectivity and fragmentation, is invaluable.

The second use is cost estimation, which is especially important for Layer 2 users, as they need to ensure that response transactions to invalid states can be confirmed in a timely manner. By simply looking at the fee rates of transactions in these blocks, costs can be estimated to some extent, but this does not provide any information about the state of the mempool after the latest block. It cannot explain sudden spikes, the influx of speculators into the mempool, or the ongoing surge of transactions that hasn't yet ended. Due to the inability to view the mempool, cost estimation cannot ensure whether it has taken into account the current state of pending transactions.

Recipient

When you receive Bitcoin, your node will validate the transaction as well as the entire block that contains the transaction. The payment to you will be broadcast and eventually enter the miners' memory pool, where the miners will find a block that will be broadcast to the network, and then your node will download and validate that block.

But the actual way it works is not like this (unless you disable the node's memory pool and run in block-only mode). Your node verifies each transaction when it first receives it in the memory pool and caches it as a valid Bitcoin transaction. When a miner finds a block, they actually only forward the block header and a small amount of compressed information (due to lack of a better simple explanation), which can be used to determine which transactions are included in the block. Then, your node will fetch the pre-verified transactions, verify the block header, and if all passes, will forward the "compact block".

This optimization is essentially why miners no longer rely on centralized and permissioned relay networks, such as FIBRE maintained by Matt Corrallo and the short-lived Falcon Network. Due to the slower relay speeds of peer-to-peer networks, miners in the past had to connect to the Falcon Network to ensure low block relay latency with other miners.

Miner

Miners clearly want to see everything. They are profit-driven entities that want to filter out transactions with the highest fees from the largest possible set of pending transactions. This is how they maximize profits and earn income to continue expanding their business and remain competitive.

They actually obtain funds from the memory pool. Their motivation to acquire any valid paid transaction is so strong that they have historically, currently, and almost certainly in the future established numerous systems, even informal arrangements available in society, aimed at allowing users to submit transactions directly to miners instead of through an open peer-to-peer network.

Block Explorers, On-chain Analytics Tools, etc.

They, like miners, hope to see every pending transaction that has been created and broadcast to the world. The main difference between the two is that miners profit directly by collecting fees from these transactions, while blockchain explorers and analytics companies profit indirectly by displaying, analyzing, and providing information analysis on these transactions in profitable products.

I cannot list any specific examples involving cached memory pool data, but it is well known that on-chain analysis companies regularly purchase on-chain transaction activity metadata obtained privately. They also operate witch Bitcoin nodes, which connect as broadly as possible with nodes throughout the network in order to narrow down the range of nodes that initially broadcast the transaction.

Block explorers also profit from the visualization of blockchain and mempool data, and their entire business model revolves around this point. If these information can be presented in a practical or novel way, then accessing more data and displaying it to users means that more potential profit information can be obtained.

Information Desires to Flow

All these different types of users benefit from "one" public memory pool, and the reason is simple: information flows freely among them. As long as there are sufficient fees that pass through the minimum relay filter, it conforms to consensus and does not pose a legitimate risk of denial of service or resource exhaustion to any single node, it can provide value for disseminating information among various users in each individual memory pool in the network.

If there is no fully functional public memory pool, the only viable alternative for all these different individual users is a centralized solution, or a chaotic situation that is difficult to manage, where there is a hasty and disorderly attempt to build fragmented public memory pools, each of which needs to be tracked separately by the users.

This not only raises concerns about manipulating fee rate data, deceiving users, and the extractable value for miners due to private relay transactions. Without a healthy and open public mempool, these are the issues that Bitcoin must face.

In the following articles, I will explore these issues, as well as the different types of memory pool filters and their reasons for existence.