# Attested TLS in the Wild
import { Accordion, Accordions } from 'fumadocs-ui/components/accordion';
# Introduction
Trusted Execution Environments (TEEs) are a practical foundation for confidential computing: they protect code and memory at runtime, even when the surrounding host is not fully trusted.
But distributed systems are remote by default, so the question becomes: how does a client verify that it's really talking to a TEE endpoint, that the channel terminates inside that TEE (no eavesdropping), and that the endpoint is running the expected software?
Transport Layer Security (TLS) provides part of the answer: channel confidentiality and authenticity. A client can authenticate a server through a public key (Web PKI). However, TLS alone doesn't show that the key is usable only from inside the TEE. To get that guarantee, you need a way to prove what runtime generated that key, and to bind those runtime claims to the specific live session.
This is where attested TLS (aTLS) comes in: bind remote attestation evidence to the live TLS session so peers can verify transport identity, TEE authenticity, and expected runtime state.
}>
Put differently, aTLS answers one question: does this session key belong to this measured software instance?
In this post, we describe how we handled this in practice while migrating a network of TEEs from libp2p to a new transport. We first shipped a working TEE-to-TEE mutual attestation flow, then mapped it to the aTLS draft and reference implementation to understand what we matched, what we didn't, and what the extra mechanisms buy you under different threat models.
In this article, when we refer to aTLS we are always talking about **Post-Handshake** Attested TLS, as that's the pattern we are using in our implementation.
There are 2 more variants of aTLS currently in discussion within the IETF: **Pre-Handshake** and **Intra-Handshake,** both covered by the FOSDEM talk referenced below.
# The Remote TEE Trust Problem
For a remote peer, three properties must hold at the same time to extend TEE guarantees such as confidentiality and integrity to the network channel:
1. **Channel confidentiality and integrity.** Traffic must be unreadable and tamper-evident in transit, otherwise sensitive inputs and outputs can be leaked or modified.
2. **Endpoint authenticity and in-TEE termination.** The remote endpoint must be the intended TEE-backed peer, not a substituted endpoint or a man-in-the-middle. To guarantee this, encryption must terminate inside the TEE (i.e. plaintext only exists within the enclave's protected memory).
3. **Runtime integrity against policy.** The remote TEE must be running software state that matches policy; otherwise you may be talking to a genuine TEE running the wrong code.
TLS gives you an encrypted channel and proof of key ownership; remote attestation gives you evidence about runtime state. aTLS is the standard way to make those proofs refer to the same live session.
# aTLS at a High Level
## Quick (m)TLS Refresher
TLS 1.3 starts with a handshake that negotiates fresh session keys and authenticates certificate key ownership. In mutual TLS, both peers present certificates and prove possession of their private keys by signing a challenge.
This gives you an encrypted and authenticated channel, addressing the first 2 problems, although it doesn't really provide any assurances that you are talking to a TEE directly (second half of problem 2) or that you are indeed communicating to the *exact* TEE you are expecting (problem 3).
## What aTLS Adds
The easiest way to reason about aTLS is as a composition of three independent proofs that must all hold at the same time.
TLS proves channel confidentiality and peer key ownership. Attestation proves what code is running in the TEE. Binding proves that the attestation evidence belongs to this exact secure channel, not some other session.
This security shape is the core idea behind aTLS. The specific message formats and primitives can vary, but the outcome should be the same: channel key ownership is cryptographically bound to attested software identity.
With that model in place, we can now move from generic protocol properties to the concrete T+ deployment that motivated our implementation.
# The T+ Confidential Exchange
T+ ([@tplus\_cx](http://x.com/tplus_cx)) is a multi-node confidential exchange with different service roles, each running in its own TEE. In this setup, end-to-end attestation is a networking requirement: peers need confidential channels, proof that the remote endpoint is a genuine TEE peer, and proof that its running expected software, to ensure full trade privacy. In the T+ system, **critical communication flows are fully end-to-end attested, and thus confidential**.
That requirement shaped our transport design, which we are describing next.
## The TEE-to-TEE Flow We Shipped
After a mutual TLS handshake establishes an encrypted channel and verifies peer certificates, both peers run an attestation exchange to complete the custom handshake.
Each side extracts the other peer's certificate public key from the active TLS session, asks the local TEE runtime for their own evidence bound to that session, and exchanges attestation payloads over the encrypted channel, verifying that the received payload is valid against the extracted peer's certificate public key and TEE runtime state.
Since each node generates a self-signed peer certificate at startup, the transport identity is ephemeral and tied to the TEE's lifetime. Making use of this allows us to guarantee that the only entity in possession of the certificate and that could thus complete the handshake *must* be the intended TEE peer we just verified.
In TDX (and similar TEE platforms), this attestation evidence is called a **quote**: a hardware-signed statement that includes measurements, including cryptographic digests of the code, configuration, and initial state loaded into the TEE. In addition, a quote also carries caller-supplied input data, which is how we bind it to TLS identity: the caller includes the TLS cert public key as input data.
**This is how the channel is proven to be secure: the quote attests to the fact that the code and the runtime state is what we expect, and it also binds to the ephemerally generated and unique, self-signed certificate**.
Here is the flow in diagram form:
In our current deployment this is mutual attestation, so each side verifies the other before admitting protocol traffic. To keep the diagram readable, the sequence below shows one direction only. The reverse direction is symmetric.
In production, we also bind a stable identity (for example the node operator) to the ephemeral TLS session so auditing and policy stay stable even as transport keys rotate.
With that flow in place, the integration point became clear: the attestation check should run after transport setup, but before protocol traffic is accepted.
## Why ConnectionHooks Were the Right Migration Point
[`msg-rs`](https://github.com/chainbound/msg-rs), our Rust messaging transport library, recently introduced connection hooks that run after transport setup and before protocol traffic. Connection hooks make it trivial to implement application-layer handshakes, which is exactly where attestation belongs.
If you want more context on `msg-rs`, we covered it in previous entries such as [Linkem](https://engineering.chainbound.io/linkem) and [Introducing Flowproxy](https://engineering.chainbound.io/introducing-flowproxy).
[`ConnectionHook`](https://docs.rs/msg-socket/latest/msg_socket/hooks/trait.ConnectionHook.html) was also the migration lever for us: it let us re-implement trust-boundary behavior that previously lived in fork-specific libp2p code as an explicit, reusable integration point in `msg-rs`. This preserved T+'s existing attestation requirement while keeping transport attestation out of business logic.
If code is more your *vibe* than prose, here's a short example taken from the library:
```rust
/// Runs after transport setup, before protocol traffic.
/// Return Ok(io) to accept, Err to reject and close.
trait ConnectionHook {
type Error;
async fn on_connection(&self, io: Io) -> HookResult;
}
```
```rust
impl ConnectionHook for AtlsServerHook
where
Io: AsyncRead + AsyncWrite + Send + Unpin + 'static + TlsCerts,
F: Fn(&Bytes) -> bool + Send + Sync + 'static,
{
type Error = ServerHookError;
async fn on_connection(&self, io: Io) -> HookResult {
// Obtain the peer certificate from the underlying transport
let client_cert = io.peer_cert();
let mut conn = Framed::new(io, Codec::new_server());
// Wait for the client to send their attestation
let msg = conn.next().await
.ok_or(Error::hook(ServerHookError::ConnectionClosed))?;
let Message::Attestation(payload) = msg? else {
return Err(Error::hook(ServerHookError::ExpectedAuthMessage));
};
// Validate the client's attestation against the certificate used
// for the TLS session
if !self.validate_attestation(payload, client_cert) {
conn.send(auth::Message::Reject).await?;
return Err(Error::hook(ServerHookError::Rejected));
}
conn.send(Message::Authenticated).await?;
// remove Framed wrapper and return the, now attested, IO
Ok(conn.into_inner())
}
}
```
```rust
// Server: validate incoming attestations
let pub_socket = PubSocket::new(Tcp::default())
.with_connection_hook(AtlsServerHook::new());
// Client: send attestation on connect
let sub_socket = SubSocket::new(Tcp::default())
.with_connection_hook(AtlsClientHook::new());
```
With this abstraction in place, we can attach an attestation hook to the sockets that need trust enforcement. If attestation fails, the connection is rejected immediately before the application can make use of it.
# From Our Implementation to aTLS
With that working baseline in place, we mapped it against the aTLS draft to see where it aligned, where it diverged, and why.
The current IETF draft standardizes this binding in a portable, interoperable way. Like our flow, aTLS runs attestation after a normal TLS 1.3 handshake.
The difference is in the machinery it uses, which rests on three building blocks:
* **Exported Keying Material (EKM) ([RFC 5705](https://datatracker.ietf.org/doc/html/rfc5705))**: a value derived from the TLS master secret that is unique per session. Including it in the attestation evidence binds the produced quote to the *session.*
* **Exported Authenticators ([RFC 9261](https://datatracker.ietf.org/doc/html/rfc9261))**: a standard message format that lets a TLS peer prove an additional identity after the handshake, reusing TLS's own structure. aTLS uses this as the envelope for attestation evidence.
* **RATS Conceptual Message Wrapper ([draft](https://datatracker.ietf.org/doc/draft-ietf-rats-msg-wrap/))**: a uniform container for platform-specific attestation evidence (e.g. a TDX quote) with type metadata, so verifiers
don't need custom parsing per platform.
Useful references for aTLS itself:
* [IETF draft](https://datatracker.ietf.org/doc/draft-fossati-seat-expat/)
* [Reference implementation](https://github.com/tls-attestation/attestation-exported-authenticators)
* [FOSDEM 2026 Talk](https://fosdem.org/2026/schedule/event/GHGFBM-attestedtls/)
## Cert-Pubkey Binding vs EKM in Controlled Deployments
Comparing our implementation to the draft, one difference mattered most: the standard binds evidence to the session using EKM, while our version bound it to the TLS certificate public key. The rest of this section is about when that difference changes the security story.
Our baseline intuition was cert-pubkey binding: generate the TLS keypair inside the TEE, put the cert pubkey hash into the attestation quote input, verify measurements and policy, and trust the session.
**Is that enough, or do you need EKM?**
In our case, that question is evaluated under explicit assumptions: a controlled peer set, startup-generated self-signed certs, and TEEs expected to run the same measured software.
### Why Cert-Pubkey Binding Can Be Sufficient
For a controlled deployment where cert keys are generated inside the TEE and never leave it, the TLS handshake already proves key possession, and only the TEE can satisfy that proof.
The quote then proves expected measurements, while quote input data ties that quote to the cert key used in TLS. Under this model, an outside attacker cannot replay a quote and complete the handshake without also possessing the key, so cert-pubkey binding is a sound baseline.
### Why aTLS Still Prefers EKM
EKM adds value in broader or stricter models. If key bytes are exfiltrated through side channels, an attacker could complete TLS handshakes with stolen key material.
EKM is unique per session, so captured attestation cannot be replayed across sessions even with the same cert key. Standards also need an interoperable channel binding mechanism that does not assume a TEE-generated cert lifecycle.
So this is not either-or. EKM is a stronger and more general binding primitive. The practical question is when you need that extra strength.
## Threat-model Menu: Pick What You Need
In practice, the mapping became a menu: pick a threat model, then pick the mechanism that addresses it.
| Mechanism | What it protects against | When to use | Cost |
| ------------------------------------- | --------------------------------------------------------------------- | ----------------------------------------------------------- | ----------- |
| Quote input bound to TLS cert pubkey | Detached quote replay with unrelated cert/key | Baseline for TEE-generated cert deployments | Low |
| Challenge nonce | Reusing old evidence without proving freshness of attestation request | If freshness must be explicit beyond handshake liveness | Low |
| EKM binding (RFC 5705) | Session replay even under key exfiltration scenarios | High assurance, mixed trust, non-TEE-managed cert lifecycle | Low-medium |
| Mutual attestation | One-sided trust in asymmetric topologies | Any topology where both peers need TEE guarantees | Medium |
| CMW evidence wrapping | Platform lock-in and custom evidence formats | Multi-platform or multi-org deployments | Low |
| Full RFC 9261 Exported Authenticators | Incompatibility with other implementations | Required for cross-implementation interoperability | Medium-high |
A pragmatic rollout strategy is to start with cert-pubkey binding and a strict measurement policy, add a nonce when you need explicit freshness, then add EKM for defense-in-depth, and finally adopt RFC 9261 plus CMW when interoperability becomes a requirement.
# Other Sightings
### Confer's Private Inference
The pattern shows up outside TLS as well. Confer's [Private inference](https://confer.to/blog/2026/01/private-inference/) writeup explains how they can serve fully private AI inference. They use [Noise](https://noiseprotocol.org/) Pipes instead of TLS, but the core idea is the same: bind attestation evidence to the cryptographic channel that carries protocol traffic.
Their use case is one-directional, where the client verifies the inference endpoint. Our deployment uses mutual attestation, where both peers verify each other.
The attestation topology differs, but the security shape is the same: verify quote authenticity, bind evidence to the channel handshake keys, and validate measurements before trusting application data.
### Flashbots & MEV
Another example is Flashbots' [attested-tls-proxy](https://github.com/flashbots/attested-tls-proxy), a reverse HTTP proxy that runs a post-handshake attestation exchange after a normal TLS 1.3 handshake, using ALPN (Application-Layer Protocol Negotiation) to signal that attestation is expected.
Its binding event covers both the cert-pubkey and EKM rows of our threat-model menu! It does so by combining a hash of the TLS leaf certificate public key with TLS exporter material.
Flashbots use aTLS extensively throughout their confidential MEV products to provide their users with verifiable privacy. They're also involved in developing the aTLS standard.
}>
aTLS is best seen as the TLS-specific standardization of that broader channel-binding pattern.
# Conclusion
aTLS solves a real problem: binding remote attestation to an encrypted channel.
Our path was not spec-first. We were migrating T+ from a forked networking stack to `msg-rs` and had to preserve an existing TEE-to-TEE attestation requirement during that transition, so we implemented a practical hook-based solution first and then studied the standard in detail.
That second step gave us a clear map of which guarantees we already had, which ones we did not, and why we might want each additional mechanism.
}>
The right question is not "Did we implement every RFC mechanism?". The right question is "Which attacks matter for our deployment, and which mechanism closes that gap?".
That framing turns aTLS from an all-or-nothing standard into a practical design toolbox for secure systems engineering.
# Blob Notaries: a distributed blob publishing design to scale DA
import { Step, Steps } from 'fumadocs-ui/components/steps';
Or, how to separate blob *data dissemination* from *consensus* in Ethereum.
**TL;DR:** This proposal stems from the recent discussions on sharding the blob
mempool to reach the medium-term scaling goals set out by the Ethereum roadmap.
It can be summarized as:
* Delegating a subset of Ethereum nodes to attest to having seen specific blobs
* Delaying DA checks until the next slot, to prevent DoS and allow more time for
propagation
* Replacing blob transaction sidecars in the EL mempool with lightweight DA
certificates
## Introduction
Ethereum’s DA scaling roadmap has been primarily concerned with scaling DA on
the consensus layer through DAS
([PeerDAS](https://eips.ethereum.org/EIPS/eip-7594), Proto-Danksharding). The
core idea here is that data is securely sharded and dispersed across the
validator set, with some mechanisms to ensure that the data in question is truly
available without having to download all of it. This will allow us to scale DA
to potentially hundreds of blobs per block.
But as shown in a
[recent post](https://ethresear.ch/t/on-the-future-of-the-blob-mempool/22613) by
Mike Neuder, the EL blob mempool (*blobpool*) has not been given the same
attention. In fact, in the blobpool today, every validator still downloads the
full type 3 transaction (including blobs) through a lazy-pull mechanism. This
asymmetry, even if not a problem today, could definitely become a bottleneck in
the future. Furthermore, scaling DA on the CL will eventually be bottlenecked by
the uplink of the (local) block builder, as the proposer is required to upload
the entire blob bundle. We’ll see later that our proposal for fixing the
blobpool unlocks distributed blob publishing as well.
## Existing Proposals
Many solutions have been proposed for the asymmetry pointed out above. Here is a
brief overview of these, along with their limitations:
* **Deprecate the blobpool**: remove the blobpool from the EL and rely on
builders to offer private
[blob inclusion endpoints](https://docs.titanbuilder.xyz/api/eth_sendblobs).
The problems here are the following:
* we *want* a public blobpool for censorship resistance (CR), because without
them, type 3 transactions could never be part of inclusion lists (ILs). CR
would overall be much worse.
* builders become responsible for propagating *all the blobs* as fast as
possible, or they risk their block not becoming canonical. The blobpool
today also serves as a pre-proposal blob distribution mechanism. For local
block builders, this might force them to not include blobs at all, because
they can’t deal with the required upload bandwidth. More on that in
[this post](https://ethresear.ch/t/is-data-available-in-the-el-mempool/22329).
* **Vertically shard the blobpool**: we can just mirror the structure on the CL,
and require blob senders to vertically shard their blob before propagating it.
The main concern here is DoS: malicious participants can flood the network
with invalid shards, i.e. shards that don’t belong to a valid, fee-paying
transaction, or just incomplete data. Researchers have proposed solutions
including a
[data-driven approach](https://notes.ethereum.org/@dankrad/BkJMU8d0R#Vertically-sharded-mempool)
(i.e. gate the ability to send blobs based on some heuristics to make sure
that senders won’t spam the network) and a
[market-driven one](https://ethresear.ch/t/on-the-future-of-the-blob-mempool/22613)
(i.e. use an in-protocol auction to gate access to the blobpool).
* **Horizontally shard the blobpool**: blob transactions are still broadcast in
full, like today, but they are propagated in different mempools or *subnets*
based on some predicate (like sender address or transaction hash). The main
advantage here is that it’s simple, and DoS resistant. But it would still
require (local) block builders to download and propagate all blobs in order to
propose them, as Dankrad pointed out
[here](https://notes.ethereum.org/@dankrad/BkJMU8d0R#Vertically-sharded-mempool).
The
[most recent proposal](https://ethresear.ch/t/on-the-future-of-the-blob-mempool/22613)
builds on vertical sharding, but includes a market mechanism to determine who
can write to the blobpool. This ensures that there’s always an upfront cost to
publishing blobs that can only be recovered when the full blob is included, and
thus remediates the DoS issues with the original proposal. The main downsides we
see are the following:
* As a blob publisher:
* The delay between having a blob to include and when it’s actually included.
* The execution gas cost of placing a bid.
* As a protocol:
* Running an auction on the L1 can introduce significant complexity
* Using Ethereum blockspace and gas for auction tickets might not be very
efficient
## Proposal: Blob Committee
We propose to introduce a new committee (the *Blob Committee*) that is a random
subset of the validator set with the following responsibilities:
* Receiving type 3 transactions (carrying blobs) and validating them
* Attesting to the availability of the blob data by producing *blob
certificates* (BLS signatures over the blob commitments)
* Sharding and publishing the columns on the CL eventually
We call these committee members ***Blob Notaries***. Blob notaries are able to
isolate initial blob and transaction validation to protect against network-wide
DoS. Additionally, they can be relied on as a temporary DA oracle during block
production, which can therefore remove download bottlenecks from the hot-path of
proposal, both with local building and PBS. Finally, they can increase the
throughput of blob dissemination by distributing the work between themselves,
unlocking *distributed blob publishing*.
Note that this committee could eventually evolve into a separate validator role,
skewed towards higher bandwidth usage, in the spirit of
[rainbow staking](https://ethresear.ch/t/unbundling-staking-towards-rainbow-staking/18683).
### New Blob Transaction Flows
We will focus on the proposal happy path to avoid cluttering the article with
complexity. However, some preliminary questions and answers can be found in
Implementation Notes below.
#### Step 1: Acquire Blob Quorum Certificates (QCs)
* The blob sender gossips the type 3 transaction (including the blob sidecar) to
all blob notaries for that slot.
* The blob notaries will validate the transaction on the pending state, and
respond with their signature over the blob commitment. We call this signature
a *Blob Certificate*.
* The blob sender aggregates these BLS signatures into a 2/3 majority, obtaining
a blob *Quorum Certificate (QC)*.
This step ensures that, between committee members, enough honest members custody
the blob data and attest that the transaction is valid. The QC is a reflection
of that. However, the existence of a QC does not guarantee DA at network-level,
as the attesters haven’t voted yet. They can be thought of as a “credible
signal”, but not guaranteed availability.
#### Step 2: Broadcast to the EL Mempool
* Once the QC is obtained, the blob sender can send the type 3 transaction
envelope with QC on the EL mempool instead of the type 3 transaction with blob
sidecar.
* Regular nodes only need to propagate the transactions with lightweight
certificates.
* Validating QC requires some level of communication with the CL. The design
space is quite large, but we haven’t found a convincing design yet. See the
*Open Questions* section for more details.
#### Step 3: Block Proposal and Attestation
* The proposer includes type 3 transactions in its block as usual. However,
instead of adding the blob bundle to the beacon block envelope, it will add
the QCs of the blobs. This consists in a new field `blob_quorum_certificates`
on the
[`BeaconBlockBody`](https://github.com/ethereum/consensus-specs/blob/dev/specs/deneb/beacon-chain.md#beaconblockbody)
container.
* The beacon block envelope *does not contain the full blob bundle here
anymore*.
* When the attesters receive the new block, they won’t validate the availability
of its QCs *yet*, but instead they will *only validate the execution of the
block* (more on this later).
* If the block receives enough valid votes in time, it will become canonical, as
usual.
#### Step 4: Blob Shard Propagation
* Now that the attesters have voted and added the new block to their canonical
chain, the blob notaries can start disseminating the blob data to the rest of
the network via vertical shards.
* Any node is able to immediately verify that these shards are valid, because
they belong to blobs whose commitment were included in the chain’s head block,
with valid QCs. This prevents the DoS vector identified in the *Existing
Proposals* above.
* Blob notaries have already signed the blobs in the canonical block, so they
are fully incentivized to share the data before the *data availability
attestation deadline.*
#### Step 5: DA Attestation
* The next slot starts, and attesters are asked to vote for two things:
1. Valid execution of the block proposed in that slot (block N)
2. Data availability of the blobs proposed in the previous slot (block N-1)
* Logically these are conceived as two attestation events, but in practice we
can think of the attestation as accommodating a new, more sophisticated
fork-choice rule:
* block `N` *execution* must be *valid*, AND block `N-1` *data* must be
*available*.
* This mechanism allows for a full-slot worth of time to propagate the blob
shards: approximately from the previous slot’s attestation deadline to the
current slot’s attestation deadline.
### Benefits of This Design
#### 1. High DA Throughput
Blob throughput in Ethereum is bottlenecked by the following factors:
* validators (including solo-stakers) need to upload the entire blob bundle when
proposing
* after PeerDAS, proposers will still have limited time to propagate blob shards
in the CL
Our proposal tries to address these limitations with traditional scaling
methodologies:
* Horizontal scaling: by having many blob notaries instead of just one proposer,
the network’s cumulative blob uplink can be orders of magnitude higher.
* Vertical scaling: by designating a specific network role (*à la rainbow
staking*) for blob notaries, we can extend their hardware/bandwidth
requirements without compromising on the nice properties of a credibly
decentralized attester set, such as censorship and collusion resistance.
#### 2. No blob shard spam on the CL
If CL nodes were to receive a shard that is not part of any recent blob, they
can reject it and apply the necessary reputation penalties to the peer that
shared it, minimizing the DoS potential.
As a side benefit, delaying the DA checks to the next slot will also maximize
the likelihood of data being available in time, because blob notaries now have a
full slot (roughly from the previous slot’s attestation deadline to the
current’s slot attestation deadline) to propagate shards.
#### 3. Cheaper blob transaction replacements in the EL
Currently, replacing a type 3 transaction is very expensive (e.g.
[Geth requires a 100% fee bump](https://github.com/ethereum/go-ethereum/blob/b47e4d5b38b34c045cb10af6c0b5603c285310cd/core/txpool/blobpool/blobpool.go#L1142-L1179)).
This is mainly because the network has to incur the cost of propagating the full
blob sidecar in the EL mempool. With blob committees, transactions in the EL
mempool would only carry lightweight QCs, making transaction replacements as
cheap as other transaction types.
#### 4. Lower “cost of latency” for including blobs
Blob transactions in PBS today need to pay an indirect latency cost. This is
because blocks with more blobs
[need to compete with more lightweight blocks in the PBS auction](https://ethresear.ch/t/blobs-reorgs-and-the-role-of-mev-boost/19783).
With this proposal, blocks with more blobs would only carry a marginal size
increase for their certificate.
#### 5. Trivial support for type 3 IL transactions
Since the full blob is replaced by a lightweight DA certificate in the EL
mempool, type 3 transactions become much smaller and can be supported by
[FOCIL](https://eips.ethereum.org/EIPS/eip-7805). Validating the QC would be
part of the
[CL P2P validation rules](https://eips.ethereum.org/EIPS/eip-7805#cl-p2p-validation-rules)
for validating ILs.
## Compatibility with ePBS
In this section we discuss the compatibility with some of the slot restructuring
proposal headliners for [Glamsterdam](https://forkcast.org/upgrade/glamsterdam).
### ePBS ([EIP-7732](https://eips.ethereum.org/EIPS/eip-7732))
The current ePBS spec makes it possible to shift the execution and DA
attestations into the same slot, allowing for a much simpler fork-choice rule.
However, since there is no natural "commit" phase for the QCs anymore, nodes
must rely on weaker guarantees to counter DoS.
Essentially, blob notaries would need to wait for the payload release before
they can start propagating blob shards. This way, recipient nodes would be able
to verify that the data they're receiving is actually part of the payload they
just received from peers. Any shard not part of the payload would be discarded,
and the sender would accrue negative reputation. To address data races, nodes
could even wait for a payload before judging the validity of recently received
shards.
Here is the slot structure, following the recent
[double deadline PTC vote](https://notes.ethereum.org/@anderselowsson/Dual-deadlinePTCvote)
ePBS design:
Blob shards received before the red X would be cached by attesters for a short
while, and once the payload arrives, they would be validated. If they don’t
match any of the blob commitments included in the payload, then they can be
discarded and the sending peer penalized. If they are valid, then they can be
broadcast to other peers. This way, regular attesters will only participate in
fan-out once they know they are propagating valid data, which is desired.
## Open questions
1. The blob notaries will most likely require higher bandwidth than regular
attesters. How should the network deal with low-performing notaries? Should
they simply miss the rewards or get slashed? How does the network detect
(poor) performance reliably?
2. Should blob notaries be rewarded fairly for their job in-protocol? If so,
how?
3. How to verify QCs in the EL mempool? Any solution would require some
communication of CL data to the EL; here are some possible options we’ve
thought of:
1. We could store and update the blob notaries pubkeys in a system contract
which gets regularly updated, and broadcast on the EL an aggregation
bitlist in the new EIP-4844 variant along with the QCs (sort of similar to
EIP-4788).
2. The existing engine API could be extended by sending tuples of
`(tx_hash, quorum_certificates)` periodically via an endpoint
`engine_newQuorumCertificates`. When a type 3 transaction announcement is
received on devp2p, it is pulled only if the transaction hash has been
already heard from the CL.
3. There is no validation of QCs in devp2p, but when the next proposer
creates a block from EL data, in the `engine_forkchoiceUpdated`
`PayloadAttributes` we also send a list of QCs already validated by the CL
client. While this can still ensure proposal of valid transactions, it
might open some DoS concerns in the EL mempool.
4. How would private blob submission to block builders work under this model?
1. Perhaps it would be possible to keep the current blob transaction pipeline
as a fallback, which can then be used by builders to include blobs.
Essentially, this means it would be possible to include a blob by either
providing its entire contents in the proposal OR a valid signed QC. In
case the beacon block envelope carries some full blobs, the block builder
and proposer for that slot carry the extra risk of distributing the data
in time for the DA attestation. This added risk could simply be absorbed
by the PBS market by making private blobs more expensive to send on
average, which also aligns with the desiderata of not straining the
regular attesters with added bandwidth requirements.
5. Failure cases for attestation, missing DA, blob notary rewards, etc are still
mostly TBD!
## Implementation Notes
**On EIP-4844 transaction envelope and certificates**
We introduce a new EIP-4844 transaction **variant** which carries a *quorum
certificate* instead of the blob sidecar. The signed transaction envelope won’t
change, as it already doesn’t include the sidecar today, but rather the QC will
be treated as a new consensus layer item.
Here is a short overview of how we imagine blobs could exist in the protocol:
* With full blob sidecar: on the EL, at the RPC layer only (necessary for
ingestion from users)
* With a QC: in the EL mempool when the transaction can be included
Note: we are not proposing to create a new transaction type. The EL spec already
involves different representations of how EIP-4844 transactions can be seen: one
with and one without the blob sidecar. We propose to add a new variant: without
the blob sidecar but *with the QC*.
**A malicious user might send type 3 transactions to the blob notaries, but then
never submit the transactions in the mempool, wasting the resources of the blob
notaries**
* To avoid this, we can leverage the role of *aggregators* in the beacon chain
to provide redundancy, so that any of them can submit the valid transaction in
the EL mempool in place of the blob originator; this would look similar to the
current attestation subnet aggregators, randomly selected between the
committee members.
* Blob aggregators perform their duty in step 1) of the proposed flow. In step
2\), aggregators can send the blob transaction in the mempool for added
redundancy.
**Why separate the two attestations (execution and DA)?**
* The idea is that with this distinction the shards of blobs that are still
pending wouldn’t be allowed to enter the CL P2P network, minimizing DoS
vectors overall.
* The added benefit is a *much* larger time frame for disseminating data, which
scales well with throughput increases.
## Future Work
* We plan to study compatibility with other EIPs currently planned for the
[Glamsterdam](https://forkcast.org/upgrade/glamsterdam) hard-fork, such as
[https://eips.ethereum.org/EIPS/eip-7886](https://eips.ethereum.org/EIPS/eip-7886), and six-second slot times as proposed
in [https://eips.ethereum.org/EIPS/eip-7782](https://eips.ethereum.org/EIPS/eip-7782).
* We’d like to come up with theoretical benchmarks for throughput allowed by
this technique, based on existing p2p data and the new slot deadlines.
* The requirements and economics of blob notaries are also an interesting topic
that is out of scope of the technical spec but would be nice to explore.
## References
* [https://ethresear.ch/t/on-the-future-of-the-blob-mempool/22613](https://ethresear.ch/t/on-the-future-of-the-blob-mempool/22613)
* [https://ethresear.ch/t/payload-timeliness-committee-ptc-an-epbs-design/16054](https://ethresear.ch/t/payload-timeliness-committee-ptc-an-epbs-design/16054)
* [https://ethresear.ch/t/estimating-validator-decentralization-using-p2p-data/19920#long-lived-subnets-node-metadata-9](https://ethresear.ch/t/estimating-validator-decentralization-using-p2p-data/19920#long-lived-subnets-node-metadata-9)
* [https://ethresear.ch/t/decoupling-throughput-from-local-building/22004](https://ethresear.ch/t/decoupling-throughput-from-local-building/22004)
* [https://www.paradigm.xyz/2023/04/mev-boost-ethereum-consensus](https://www.paradigm.xyz/2023/04/mev-boost-ethereum-consensus)
* [https://eips.ethereum.org/EIPS/eip-7886](https://eips.ethereum.org/EIPS/eip-7886)
* [https://notes.ethereum.org/@dankrad/BkJMU8d0R](https://notes.ethereum.org/@dankrad/BkJMU8d0R)
# FlowProxy: Approaching Optimality
import { Accordion, Accordions } from 'fumadocs-ui/components/accordion';
import { GlobalMap } from '@/components/diagrams/map';
## Background
In
[**Introducing FlowProxy**](https://collective.flashbots.net/t/introducing-flowproxy/5341),
[Chainbound](https://chainbound.io/) collaborated with Flashbots to deprecate
its [original orderflow proxy](https://github.com/flashbots/buildernet-orderflow-proxy) implementation
in Go in favor of a new one built in Rust, created with the goal of reducing
end-to-end latency, improving efficiency, and increasing observability in
[BuilderNet](https://buildernet.org/)'s networking layer. This first
collaboration aimed at bringing it to production-ready quality and ready for
deployment, already observing good improvements in networking and processing
latency.
This second collaboration expanded on the *Next Steps* section outlined in the
previous report, with success. It includes a quantitative analysis of order
losses and latencies across BuilderNet, and extensive work to improve the
transport layer to reduce both CPU and memory usage. We encountered a couple of
non-obvious learnings that we outline below, and we hope this will be useful to
the community.
## Network Analysis
FlowProxy features a Clickhouse integration which makes possible to record
*bundle receipts.* A bundle receipt consists of a short summary that includes
the timestamp of when the bundle was sent and received by instances, along with
its raw size in bytes (useful for tracking size → latency correlation). We used
this data to better understand FlowProxy’s network performance in its entirety,
and to discover what improvements to focus on.
### Bundle Loss
We started with analyzing bundle loss between instances. The table below reports
the percentage of bundle loss on individual links, during a small time range of
congestion:
| src | dst | lost | total | loss\_rate\_pct |
| ---------------------- | ---------------------- | ---- | ------ | --------------- |
| beaver\_eastus\_07 | nethermind\_we\_08 | 7857 | 35848 | 21.92% |
| beaver\_eastus\_07 | flashbots\_we\_09 | 1969 | 96597 | 2.04% |
| beaver\_eastus\_07 | beaver\_we\_08 | 1354 | 94000 | 1.44% |
| beaver\_eastus\_07 | flashbots\_eastus\_10 | 2 | 140801 | 0.00% |
| beaver\_eastus\_07 | nethermind\_eastus\_07 | 1 | 141534 | 0.00% |
| nethermind\_eastus\_07 | flashbots\_eastus\_10 | 10 | 35315 | 0.03% |
| nethermind\_eastus\_07 | beaver\_eastus\_07 | 10 | 35377 | 0.03% |
| nethermind\_eastus\_07 | flashbots\_we\_09 | 6 | 34560 | 0.02% |
| nethermind\_eastus\_07 | beaver\_we\_08 | 6 | 35216 | 0.02% |
| nethermind\_eastus\_07 | nethermind\_we\_08 | 4 | 34490 | 0.01% |
| flashbots\_eastus\_10 | beaver\_we\_08 | 4 | 34877 | 0.01% |
| flashbots\_eastus\_10 | flashbots\_we\_09 | 4 | 34266 | 0.01% |
| flashbots\_eastus\_10 | nethermind\_we\_08 | 3 | 34156 | 0.01% |
| flashbots\_eastus\_10 | beaver\_eastus\_07 | 1 | 35054 | 0.00% |
| src | dst | lost | total | loss rate |
| ---------------------- | ---------------------- | ---- | ------ | --------- |
| beaver\_eastus\_07 | nethermind\_we\_08 | 7857 | 35848 | 21.92% |
| beaver\_eastus\_07 | flashbots\_we\_09 | 1969 | 96597 | 2.04% |
| beaver\_eastus\_07 | beaver\_we\_08 | 1354 | 94000 | 1.44% |
| nethermind\_eastus\_07 | flashbots\_mkosi\_1 | 58 | 31205 | 0.19% |
| beaver\_eastus\_07 | flashbots\_mkosi\_1 | 13 | 125199 | 0.01% |
| flashbots\_eastus\_10 | flashbots\_mkosi\_1 | 11 | 31415 | 0.04% |
| nethermind\_eastus\_07 | flashbots\_eastus\_10 | 10 | 35315 | 0.03% |
| nethermind\_eastus\_07 | beaver\_eastus\_07 | 10 | 35377 | 0.03% |
| nethermind\_eastus\_07 | flashbots\_we\_09 | 6 | 34560 | 0.02% |
| nethermind\_eastus\_07 | beaver\_we\_08 | 6 | 35216 | 0.02% |
| flashbots\_eastus\_10 | beaver\_we\_08 | 4 | 34877 | 0.01% |
| nethermind\_eastus\_07 | nethermind\_we\_08 | 4 | 34490 | 0.01% |
| flashbots\_eastus\_10 | flashbots\_we\_09 | 4 | 34266 | 0.01% |
| nethermind\_we\_08 | flashbots\_eastus\_10 | 3 | 113560 | 0.00% |
| nethermind\_we\_08 | beaver\_eastus\_07 | 3 | 116296 | 0.00% |
| nethermind\_we\_08 | nethermind\_eastus\_07 | 3 | 113862 | 0.00% |
| flashbots\_eastus\_10 | nethermind\_we\_08 | 3 | 34156 | 0.01% |
| beaver\_eastus\_07 | flashbots\_eastus\_10 | 2 | 140801 | 0.00% |
| flashbots\_mkosi\_1 | beaver\_we\_08 | 1 | 95552 | 0.00% |
| flashbots\_mkosi\_1 | nethermind\_eastus\_07 | 1 | 95172 | 0.00% |
| nethermind\_we\_08 | beaver\_we\_08 | 1 | 104890 | 0.00% |
| flashbots\_mkosi\_1 | beaver\_eastus\_07 | 1 | 96537 | 0.00% |
| flashbots\_mkosi\_1 | flashbots\_we\_09 | 1 | 95015 | 0.00% |
| flashbots\_eastus\_10 | beaver\_eastus\_07 | 1 | 35054 | 0.00% |
| beaver\_eastus\_07 | nethermind\_eastus\_07 | 1 | 141534 | 0.00% |
| flashbots\_mkosi\_1 | flashbots\_eastus\_10 | 1 | 93635 | 0.00% |
| nethermind\_we\_08 | flashbots\_we\_09 | 1 | 103316 | 0.00% |
Or visualized:
Europe (northern route)
{
from: 'beaver_eastus',
to: 'nethermind_we',
metrics: { bundleLoss: 21.92 },
waypoints: [
[-42, 47],
[-18, 49],
],
},
{
from: 'beaver_eastus',
to: 'flashbots_we',
metrics: { bundleLoss: 2.04 },
waypoints: [
[-40, 48],
[-15, 50],
],
},
{
from: 'beaver_eastus',
to: 'beaver_we',
metrics: { bundleLoss: 1.44 },
waypoints: [
[-42, 44],
[-18, 46],
],
},
// US East local
{
from: 'beaver_eastus',
to: 'flashbots_eastus',
metrics: { bundleLoss: 0.0 },
},
{
from: 'beaver_eastus',
to: 'nethermind_eastus',
metrics: { bundleLoss: 0.0 },
},
{
from: 'nethermind_eastus',
to: 'flashbots_eastus',
metrics: { bundleLoss: 0.03 },
},
{
from: 'nethermind_eastus',
to: 'beaver_eastus',
metrics: { bundleLoss: 0.03 },
},
// Transatlantic: Nethermind EUS -> Europe (middle routes)
{
from: 'nethermind_eastus',
to: 'flashbots_we',
metrics: { bundleLoss: 0.02 },
waypoints: [
[-38, 46],
[-12, 48],
],
},
{
from: 'nethermind_eastus',
to: 'beaver_we',
metrics: { bundleLoss: 0.02 },
waypoints: [
[-35, 43],
[-10, 45],
],
},
{
from: 'nethermind_eastus',
to: 'nethermind_we',
metrics: { bundleLoss: 0.01 },
waypoints: [
[-32, 50],
[-8, 52],
],
},
// Transatlantic: Flashbots EUS -> Europe (southern routes)
{
from: 'flashbots_eastus',
to: 'beaver_we',
metrics: { bundleLoss: 0.01 },
waypoints: [
[-30, 41],
[-5, 44],
],
},
{
from: 'flashbots_eastus',
to: 'flashbots_we',
metrics: { bundleLoss: 0.01 },
waypoints: [
[-28, 44],
[-3, 47],
],
},
{
from: 'flashbots_eastus',
to: 'nethermind_we',
metrics: { bundleLoss: 0.01 },
waypoints: [
[-25, 47],
[0, 50],
],
},
{
from: 'flashbots_eastus',
to: 'beaver_eastus',
metrics: { bundleLoss: 0.0 },
},
// European interconnections
{ from: 'beaver_we', to: 'nethermind_we', metrics: { bundleLoss: 0.01 } },
{ from: 'beaver_we', to: 'flashbots_we', metrics: { bundleLoss: 0.01 } },
{
from: 'nethermind_we',
to: 'flashbots_we',
metrics: { bundleLoss: 0.01 },
},
]}
metricKey="bundleLoss"
metricRange={[0, 25]}
height={350}
center={[-35, 45]}
scale={350}
showLegend={true}
/>
This map represents a simplified view of the BuilderNet topology, it's not
geographically accurate.
We can see in particular that inter-continental links are mostly affected by
bundle loss, especially the instance `beaver_eastus_07` which receives the most
flow in that region.
The main reason bundles are marked as “lost” is when they time out, or when
buffers overflow and backpressure is applied. This confirms what we
[explored](https://collective.flashbots.net/t/introducing-flowproxy/5341#p-10727-http-connection-pools-11)
in the last post: that HTTP/1.1 and its incapacity to multiplex requests makes
it highly inadequate to deal with BuilderNet’s workload.
### Latency
From the table below we can see latency between individual BuilderNet links
during high congestion.
| src | dst | p50\_ms | p99\_ms | corr\_payload\_size | observations |
| ---------------------- | ---------------------- | ------- | -------- | ------------------- | ------------ |
| beaver\_eastus\_07 | nethermind\_we\_08 | 39.619 | 1947.045 | -0.06 | 27991 |
| beaver\_eastus\_07 | flashbots\_we\_09 | 38.022 | 1789.051 | -0.06 | 94628 |
| beaver\_eastus\_07 | beaver\_we\_08 | 39.440 | 1521.599 | -0.04 | 92646 |
| beaver\_eastus\_07 | nethermind\_eastus\_07 | 62.639 | 292.451 | -0.02 | 141533 |
| nethermind\_eastus\_07 | beaver\_we\_08 | 33.381 | 194.660 | 0.59 | 35210 |
| nethermind\_eastus\_07 | nethermind\_we\_08 | 31.541 | 193.107 | 0.58 | 34486 |
| nethermind\_eastus\_07 | flashbots\_we\_09 | 30.992 | 192.134 | 0.60 | 34554 |
| nethermind\_eastus\_07 | flashbots\_eastus\_10 | -0.662 | 58.648 | 0.06 | 35305 |
| nethermind\_eastus\_07 | beaver\_eastus\_07 | -4.514 | 49.629 | 0.02 | 35367 |
| flashbots\_eastus\_10 | beaver\_we\_08 | 40.197 | 201.535 | 0.60 | 34873 |
| flashbots\_eastus\_10 | nethermind\_we\_08 | 38.467 | 200.267 | 0.59 | 34153 |
| flashbots\_eastus\_10 | flashbots\_we\_09 | 37.945 | 199.814 | 0.61 | 34262 |
| flashbots\_eastus\_10 | beaver\_eastus\_07 | 2.506 | 56.186 | 0.02 | 35053 |
| flashbots\_eastus\_10 | nethermind\_eastus\_07 | -7.746 | 26.184 | 0.04 | 35048 |
| src | dst | p50\_ms | p90\_ms | p99\_ms | p999\_ms | corr\_payload\_size | observations |
| ---------------------- | ---------------------- | ------- | -------- | -------- | -------- | ------------------- | ------------ |
| beaver\_eastus\_07 | nethermind\_we\_08 | 39.619 | 1668.253 | 1947.045 | 2005.370 | -0.06 | 27991 |
| beaver\_eastus\_07 | flashbots\_we\_09 | 38.022 | 1414.608 | 1789.051 | 1978.707 | -0.06 | 94628 |
| beaver\_eastus\_07 | beaver\_we\_08 | 39.440 | 852.713 | 1521.599 | 1970.474 | -0.04 | 92646 |
| beaver\_eastus\_07 | flashbots\_mkosi\_1 | 39.591 | 419.213 | 1099.179 | 1221.009 | -0.03 | 125186 |
| nethermind\_we\_08 | flashbots\_eastus\_10 | 44.564 | 126.509 | 294.049 | 502.033 | -0.21 | 113557 |
| beaver\_eastus\_07 | nethermind\_eastus\_07 | 62.639 | 61.099 | 292.451 | 476.941 | -0.02 | 141533 |
| flashbots\_mkosi\_1 | flashbots\_eastus\_10 | 43.261 | 84.517 | 281.969 | 560.069 | 0.16 | 93634 |
| flashbots\_mkosi\_1 | beaver\_eastus\_07 | 44.489 | 66.412 | 257.931 | 519.216 | 0.24 | 96536 |
| nethermind\_we\_08 | beaver\_eastus\_07 | 46.272 | 97.637 | 227.358 | 323.382 | 0.32 | 116293 |
| nethermind\_we\_08 | nethermind\_eastus\_07 | 51.281 | 101.732 | 224.596 | 324.134 | 0.32 | 113859 |
| nethermind\_we\_08 | beaver\_we\_08 | 25.860 | 5.459 | 216.067 | 324.788 | -0.02 | 104889 |
| flashbots\_eastus\_10 | flashbots\_mkosi\_1 | 40.740 | 45.731 | 202.758 | 286.997 | 0.59 | 31404 |
| flashbots\_mkosi\_1 | nethermind\_eastus\_07 | 49.122 | 61.092 | 202.012 | 273.676 | 0.37 | 95171 |
| flashbots\_eastus\_10 | beaver\_we\_08 | 40.197 | 46.283 | 201.535 | 284.387 | 0.60 | 34873 |
| flashbots\_eastus\_10 | nethermind\_we\_08 | 38.467 | 46.728 | 200.267 | 283.487 | 0.59 | 34153 |
| flashbots\_eastus\_10 | flashbots\_we\_09 | 37.945 | 44.632 | 199.814 | 281.712 | 0.61 | 34262 |
| nethermind\_eastus\_07 | flashbots\_mkosi\_1 | 33.733 | 37.802 | 195.461 | 204.237 | 0.59 | 31147 |
| nethermind\_eastus\_07 | beaver\_we\_08 | 33.381 | 38.238 | 194.660 | 201.383 | 0.59 | 35210 |
| nethermind\_eastus\_07 | nethermind\_we\_08 | 31.541 | 39.041 | 193.107 | 208.030 | 0.58 | 34486 |
| nethermind\_eastus\_07 | flashbots\_we\_09 | 30.992 | 36.886 | 192.134 | 203.846 | 0.60 | 34554 |
| flashbots\_we\_09 | flashbots\_eastus\_10 | 44.702 | 52.008 | 151.492 | 352.513 | 0.45 | 23666 |
| nethermind\_we\_08 | flashbots\_mkosi\_1 | 3.024 | 5.885 | 149.235 | 237.358 | -0.02 | 128776 |
| flashbots\_we\_09 | nethermind\_eastus\_07 | 51.533 | 53.694 | 135.308 | 214.133 | 0.65 | 23682 |
| flashbots\_we\_09 | beaver\_eastus\_07 | 46.286 | 51.180 | 131.223 | 213.616 | 0.58 | 23676 |
| beaver\_we\_08 | flashbots\_eastus\_10 | 42.202 | 45.092 | 124.000 | 320.903 | 0.46 | 21596 |
| beaver\_we\_08 | beaver\_eastus\_07 | 43.786 | 46.276 | 110.435 | 222.164 | 0.54 | 21630 |
| nethermind\_we\_08 | flashbots\_we\_09 | 0.300 | 4.165 | 88.781 | 154.436 | -0.01 | 105315 |
| flashbots\_mkosi\_1 | nethermind\_we\_08 | -1.529 | 1.878 | 84.565 | 172.061 | -0.02 | 109297 |
| beaver\_we\_08 | nethermind\_eastus\_07 | 49.078 | 50.298 | 62.551 | 215.596 | 0.82 | 21608 |
| nethermind\_eastus\_07 | flashbots\_eastus\_10 | -0.662 | -2.668 | 58.648 | 220.645 | 0.06 | 35305 |
| flashbots\_mkosi\_1 | beaver\_we\_08 | -0.075 | 2.209 | 57.603 | 115.534 | -0.02 | 95551 |
| flashbots\_eastus\_10 | beaver\_eastus\_07 | 2.506 | 5.522 | 56.186 | 188.124 | 0.02 | 35053 |
| flashbots\_mkosi\_1 | flashbots\_we\_09 | 2.188 | 9.008 | 52.778 | 91.397 | -0.00 | 95014 |
| nethermind\_eastus\_07 | beaver\_eastus\_07 | -4.514 | -1.465 | 49.629 | 194.338 | 0.02 | 35367 |
| flashbots\_eastus\_10 | nethermind\_eastus\_07 | -7.746 | -9.508 | 26.184 | 101.051 | 0.04 | 35048 |
| flashbots\_we\_09 | nethermind\_we\_08 | -1.307 | 3.241 | 17.573 | 83.503 | 0.02 | 23585 |
| beaver\_we\_08 | nethermind\_we\_08 | 1.076 | 3.533 | 14.321 | 98.753 | 0.03 | 21147 |
| flashbots\_we\_09 | flashbots\_mkosi\_1 | 3.017 | 4.058 | 11.831 | 43.950 | 0.06 | 23651 |
| beaver\_we\_08 | flashbots\_mkosi\_1 | 3.506 | 4.566 | 9.304 | 22.996 | 0.10 | 23289 |
| beaver\_we\_08 | flashbots\_we\_09 | 1.137 | 1.804 | 8.416 | 25.661 | 0.10 | 20771 |
| beaver\_we\_08 | flashbots\_we\_09 | -1.624 | -0.159 | 8.029 | 67.550 | 0.05 | 20979 |
The negative p50 latencies here are due to [clock
drift](https://en.wikipedia.org/wiki/Clock_drift).
Or, with P99 latencies visualized:
Europe (high latency routes)
{
from: 'beaver_eastus',
to: 'nethermind_we',
metrics: { latency: 1947.05 },
waypoints: [
[-42, 47],
[-18, 49],
],
},
{
from: 'beaver_eastus',
to: 'flashbots_we',
metrics: { latency: 1789.05 },
waypoints: [
[-40, 48],
[-15, 50],
],
},
{
from: 'beaver_eastus',
to: 'beaver_we',
metrics: { latency: 1521.6 },
waypoints: [
[-42, 44],
[-18, 46],
],
},
// US East local
{
from: 'beaver_eastus',
to: 'nethermind_eastus',
metrics: { latency: 292.45 },
},
{
from: 'nethermind_eastus',
to: 'flashbots_eastus',
metrics: { latency: 58.65 },
},
{
from: 'nethermind_eastus',
to: 'beaver_eastus',
metrics: { latency: 49.63 },
},
{
from: 'flashbots_eastus',
to: 'beaver_eastus',
metrics: { latency: 56.19 },
},
{
from: 'flashbots_eastus',
to: 'nethermind_eastus',
metrics: { latency: 26.18 },
},
// Transatlantic: Nethermind EUS -> Europe
{
from: 'nethermind_eastus',
to: 'beaver_we',
metrics: { latency: 194.66 },
waypoints: [
[-35, 43],
[-10, 45],
],
},
{
from: 'nethermind_eastus',
to: 'nethermind_we',
metrics: { latency: 193.11 },
waypoints: [
[-32, 50],
[-8, 52],
],
},
{
from: 'nethermind_eastus',
to: 'flashbots_we',
metrics: { latency: 192.13 },
waypoints: [
[-38, 46],
[-12, 48],
],
},
// Transatlantic: Flashbots EUS -> Europe
{
from: 'flashbots_eastus',
to: 'beaver_we',
metrics: { latency: 201.54 },
waypoints: [
[-30, 41],
[-5, 44],
],
},
{
from: 'flashbots_eastus',
to: 'nethermind_we',
metrics: { latency: 200.27 },
waypoints: [
[-25, 47],
[0, 50],
],
},
{
from: 'flashbots_eastus',
to: 'flashbots_we',
metrics: { latency: 199.81 },
waypoints: [
[-28, 44],
[-3, 47],
],
},
]}
metricKey="latency"
metricRange={[0, 2000]}
height={350}
center={[-35, 45]}
scale={350}
showLegend={true}
/>
As with the previous table, we can see the highest latency between the most
active inter-region links, peaking at nearly 2s in p99. Another interesting
result is the correlation with payload size (`corr_payload_size`): in normal
working conditions (low p99s), it is more pronounced. This could mean a couple
of things that will be useful to know later:
* We’re not able to send large payloads in a single transmission (related to
[BDP](https://en.wikipedia.org/wiki/Bandwidth-delay_product), which we’ll talk
about below).
* Preparing the order for transmission, or processing the order on the receiver
side, takes a noticeably longer time the bigger the message. Some correlation
is expected here, but it should be minimal. The main processing steps in the
hot path here are JSON encoding / decoding, and signing / signature
verification.
## Improvements
### Thread modelling
FlowProxy runs with the [Tokio](https://docs.rs/tokio/latest/tokio/)
asynchronous runtime. The initial implementation of the proxy indiscriminately
used Tokio tasks for all different kind of workloads, including CPU intensive
operations like signature recovery, signing and decoding transactions. This
approach is not ideal because the runtime and its tasks are fundamentally
optimized for non-blocking, I/O-bound work, and using it for other
[blocking or CPU-bound work](https://docs.rs/tokio/latest/tokio/index.html#cpu-bound-tasks-and-blocking-code)
*will* increase tail latencies.
Tokio schedules many lightweight tasks onto a small number of OS threads. If a
task performs CPU-heavy or blocking work, it can monopolise a worker thread,
preventing other tasks from making progress. We suspected this could partly be
causing some of the high tail latencies we were seeing.
The Tokio authors recommend using
[`tokio::task::spawn_blocking`](https://docs.rs/tokio/latest/tokio/task/fn.spawn_blocking.html),
which we tried initially. This will spawn (or reuse) a thread managed by the
runtime that is purely used for CPU-bound and blocking operations. However, this
resulted in many more threads being spawned than we knew was necessary, and also
had more overhead than expected. We didn’t dive into this too much, but
intuitively it seemed like the blocking thread scheduler was not reusing threads
effectively ([Github issue](https://github.com/BuilderNet/FlowProxy/pull/141)).
To mitigate this, we introduced a configurable pool of specialised
[rayon](https://docs.rs/rayon/latest/rayon/) threads for compute-heavy
operations, and tweaked the number of Tokio worker threads to match production
environment requirements. This setup allows also to have a healthy environment
where we can control how much resources a certain service is using, since the
same machine would also run other processes like the block builder.
**Side Note**
While experimenting with the parameters, we observed how moving *some* specific
compute-heavy operations resulted in a worse order processing latency. An
interesting learning was that on a busy machine with many threads, sending tasks
off to a different thread than the currently executing one *can* increase
latency significantly. The cost of scheduling a new thread (context switching,
waiting) should be taken into account, and is very context dependent!
### HTTP/2
HTTP/2 was designed to address various performance limitations of HTTP/1.1 while
keeping the same semantics. Among various improvements, the most impactful for
FlowProxy is **multiplexing**: with HTTP/1.1 only one request/response can be in
flight per TCP connection (called *head-of-line blocking),* while HTTP/2 allows
multiplexing multiple requests and responses over a single TCP connection, using
*streams*.
Streams are logical, bidirectional channels within one connection. They’re
managed by *windows*: credit-based flow control mechanisms to limit how much
data can be sent on a single stream, to ensure it doesn’t starve the connection.
This multiplexing allowed us to greatly reduce the number of open connections,
and improve connection reuse, which we already hinted was a source of message
loss.
Upgrading to HTTP/2 was the first improvement we rolled out, because of its
complete backwards compatibility: communication between and towards instances
running on a previous version of FlowProxy would simply fallback to HTTP/1.1.
Below, you can see how the number of failures (read: lost messages) have been
essentially reduced to 0 after its deployment.
HTTP failures after deployment of HTTP/2
While request failures dropped, latency didn’t significantly improve. In
particular, we’ve observed some improvement over small requests (with body size
less than 32KiB) over inter-regional links, as we can see below. However, for
same-region requests and bigger messages the situation remained identical or
slightly worsened.
RPC call duration latency (p99) before and after the deployment of HTTP/2.
This was a very different result compared to our staging environment, consisting
of four nodes distributed between East US and West Europe. We think the main
culprit is an overall different topology and network load compared to the
production environment, which would be hard to completely emulate. After this
result, we started looking into tuning configurations.
FlowProxy instances operate with a reverse [HAProxy](https://www.haproxy.org/)
that sits before the user and system endpoint, with the latter used for internal
orderflow sharing. The proxy exposes some
[HTTP/2 tuning configurations](https://docs.haproxy.org/3.2/configuration.html#tune.h2.be.initial-window-size:~:text=%2D%20tune.h2,copy%2Dfwd%2Dsend)
that could help further reducing latency and spikiness.
The dimensions in which we could operate were:
* The number of maximum open streams;
* The size of the window buffers;
* Creating dedicated clients for small and big requests.
While tuning those resulted in marginal improvements, we were still working on
high-level abstractions, without much control over the metal. Moreover, HTTP/2
windows play a similar role to TCP congestion control / window scaling,
resulting in some overhead and confusion about how the two interoperate. Because
of this, we decided to pause HTTP/2 tuning efforts, and focus on a full
migration to raw TCP (with TLS) with the
[msg-rs](https://github.com/chainbound/msg-rs) messaging library.
**Bundle loss after HTTP/2**
After the deployment of this improvement, we analyzed bundle loss once again
(read a full analysis
[**here**](https://www.notion.so/PUBLIC-FlowProxy-Bundle-Receipts-Analysis-v2-2a45abfafc1980a29bb9fda91b3dd16d?pvs=21)).
The table below contains a day worth of data, that includes both periods of low
activity and high activity. We can see that bundle loss has essentially
disappeared. Sample for 2025-11-14:
| src | dst | lost | total |
| ---------------------- | ---------------------- | ---- | -------- |
| nethermind\_eastus\_07 | beaver\_we\_08 | 28 | 8894080 |
| beaver\_eastus\_07 | flashbots\_eastus\_10 | 8 | 3001624 |
| beaver\_eastus\_07 | flashbots\_we\_09 | 8 | 11168780 |
| beaver\_eastus\_07 | nethermind\_eastus\_07 | 8 | 30111251 |
| beaver\_eastus\_07 | beaver\_we\_08 | 8 | 10864297 |
| beaver\_eastus\_07 | nethermind\_we\_08 | 8 | 10069603 |
| nethermind\_eastus\_07 | flashbots\_we\_09 | 10 | 9028245 |
| nethermind\_eastus\_07 | nethermind\_we\_08 | 8 | 9052880 |
| nethermind\_eastus\_07 | beaver\_eastus\_07 | 6 | 10309013 |
| nethermind\_eastus\_07 | flashbots\_eastus\_10 | 5 | 10302474 |
| flashbots\_eastus\_10 | beaver\_we\_08 | 14 | 9024652 |
| flashbots\_eastus\_10 | nethermind\_eastus\_07 | 6 | 10443797 |
| flashbots\_eastus\_10 | beaver\_eastus\_07 | 6 | 10458639 |
| flashbots\_eastus\_10 | nethermind\_we\_08 | 4 | 9087691 |
| flashbots\_eastus\_10 | flashbots\_we\_09 | 2 | 9115212 |
| src | dst | count | total | loss pctg |
| --------------------------------------------- | --------------------------------------------- | ----- | -------- | --------- |
| flashbots\_test\_1 | beaver\_eastus\_07 | 47 | 5297796 | 0% |
| buildernet\_nethermind\_azure\_eastus\_07 | buildernet\_beaver\_azure\_westeurope\_08 | 28 | 8894080 | 0% |
| buildernet\_beaver\_azure\_westeurope\_08 | buildernet\_beaver\_azure\_eastus\_07 | 24 | 5565633 | 0% |
| buildernet\_flashbots\_azure\_eastus\_10 | buildernet\_flashbots\_mkosi\_test\_1 | 18 | 9283988 | 0% |
| buildernet\_beaver\_azure\_westeurope\_08 | buildernet\_nethermind\_azure\_eastus\_07 | 16 | 5547538 | 0% |
| buildernet\_beaver\_azure\_eastus\_07 | buildernet\_flashbots\_mkosi\_test\_1 | 16 | 10189003 | 0% |
| buildernet\_flashbots\_azure\_eastus\_10 | buildernet\_beaver\_azure\_westeurope\_08 | 14 | 9024652 | 0% |
| buildernet\_beaver\_azure\_westeurope\_08 | buildernet\_flashbots\_azure\_eastus\_10 | 10 | 5578403 | 0% |
| buildernet\_nethermind\_azure\_eastus\_07 | buildernet\_flashbots\_azure\_westeurope\_09 | 10 | 9028245 | 0% |
| buildernet\_flashbots\_mkosi\_test\_1 | buildernet\_flashbots\_azure\_eastus\_10 | 9 | 6146894 | 0% |
| buildernet\_flashbots\_mkosi\_test\_1 | buildernet\_nethermind\_azure\_eastus\_07 | 9 | 6222701 | 0% |
| buildernet\_nethermind\_azure\_eastus\_07 | buildernet\_nethermind\_azure\_westeurope\_08 | 8 | 9052880 | 0% |
| buildernet\_beaver\_azure\_eastus\_07 | buildernet\_flashbots\_azure\_eastus\_10 | 8 | 3001624 | 0% |
| buildernet\_beaver\_azure\_eastus\_07 | buildernet\_flashbots\_azure\_westeurope\_09 | 8 | 11168780 | 0% |
| buildernet\_beaver\_azure\_eastus\_07 | buildernet\_nethermind\_azure\_eastus\_07 | 8 | 30111251 | 0% |
| buildernet\_beaver\_azure\_eastus\_07 | buildernet\_beaver\_azure\_westeurope\_08 | 8 | 10864297 | 0% |
| buildernet\_beaver\_azure\_eastus\_07 | buildernet\_nethermind\_azure\_westeurope\_08 | 8 | 10069603 | 0% |
| buildernet\_nethermind\_azure\_westeurope\_08 | buildernet\_flashbots\_azure\_eastus\_10 | 7 | 6573168 | 0% |
| buildernet\_flashbots\_azure\_westeurope\_09 | buildernet\_beaver\_azure\_eastus\_07 | 7 | 6078754 | 0% |
| buildernet\_nethermind\_azure\_eastus\_07 | buildernet\_beaver\_azure\_eastus\_07 | 6 | 10309013 | 0% |
| buildernet\_flashbots\_azure\_eastus\_10 | buildernet\_nethermind\_azure\_eastus\_07 | 6 | 10443797 | 0% |
| buildernet\_nethermind\_azure\_eastus\_07 | buildernet\_flashbots\_mkosi\_test\_1 | 6 | 8967508 | 0% |
| buildernet\_flashbots\_azure\_eastus\_10 | buildernet\_beaver\_azure\_eastus\_07 | 6 | 10458639 | 0% |
| buildernet\_nethermind\_azure\_westeurope\_08 | buildernet\_nethermind\_azure\_eastus\_07 | 5 | 6513729 | 0% |
| buildernet\_nethermind\_azure\_westeurope\_08 | buildernet\_beaver\_azure\_eastus\_07 | 5 | 6194898 | 0% |
| buildernet\_nethermind\_azure\_eastus\_07 | buildernet\_flashbots\_azure\_eastus\_10 | 5 | 10302474 | 0% |
| buildernet\_flashbots\_azure\_eastus\_10 | buildernet\_nethermind\_azure\_westeurope\_08 | 4 | 9087691 | 0% |
| buildernet\_flashbots\_azure\_westeurope\_09 | buildernet\_flashbots\_azure\_eastus\_10 | 3 | 6117425 | 0% |
| buildernet\_flashbots\_azure\_eastus\_10 | buildernet\_flashbots\_azure\_westeurope\_09 | 2 | 9115212 | 0% |
| buildernet\_flashbots\_azure\_westeurope\_09 | buildernet\_flashbots\_mkosi\_test\_1 | 1 | 7617802 | 0% |
| buildernet\_flashbots\_mkosi\_test\_1 | buildernet\_beaver\_azure\_westeurope\_08 | 1 | 18836193 | 0% |
| buildernet\_flashbots\_mkosi\_test\_1 | buildernet\_flashbots\_azure\_westeurope\_09 | 1 | 18043957 | 0% |
| buildernet\_nethermind\_azure\_westeurope\_08 | buildernet\_flashbots\_mkosi\_test\_1 | 1 | 9123905 | 0% |
| buildernet\_nethermind\_azure\_westeurope\_08 | buildernet\_beaver\_azure\_westeurope\_08 | 1 | 13247037 | 0% |
| buildernet\_flashbots\_azure\_westeurope\_09 | buildernet\_nethermind\_azure\_eastus\_07 | 1 | 6076706 | 0% |
| buildernet\_flashbots\_mkosi\_test\_1 | buildernet\_nethermind\_azure\_westeurope\_08 | 1 | 7608331 | 0% |
**HTTP/2 latency**
Below we can see a more complete before-after of latency between links. We can
see that after HTTP/2 we see more bounded p99 and p999, while p50 and p90 stayed
almost the same. While picking a single day from both deployments may not be
completely indicative, the behaviour remained quite consistent during the next
days. Latency comparison of P99 latencies befor, each with 24 hours worth of
datae and after HTTP/2 for a day's worth of data (all numbers in milliseconds):
| src | dst | p50\_a | p50\_b | Δp50 | p99\_a | p99\_b | Δp99 |
| ---------------------- | --------------------- | ------ | ------ | ---- | ------ | ------ | ---- |
| nethermind\_we\_08 | beaver\_eastus\_07 | 42 | 42 | \~0 | 458 | 155 | -303 |
| nethermind\_we\_08 | flashbots\_eastus\_10 | 42 | 41 | -1 | 455 | 142 | -313 |
| beaver\_eastus\_07 | nethermind\_we\_08 | 44 | 41 | -3 | 357 | 125 | -232 |
| beaver\_eastus\_07 | flashbots\_we\_09 | 45 | 46 | +1 | 297 | 122 | -175 |
| nethermind\_eastus\_07 | flashbots\_we\_09 | 48 | 45 | -3 | 299 | 120 | -179 |
| flashbots\_eastus\_10 | nethermind\_we\_08 | 42 | 43 | +1 | 206 | 119 | -87 |
| flashbots\_eastus\_10 | flashbots\_we\_09 | 43 | 44 | +1 | 206 | 106 | -100 |
| flashbots\_eastus\_10 | beaver\_eastus\_07 | 43 | 44 | +1 | 284 | 131 | -153 |
| nethermind\_eastus\_07 | beaver\_eastus\_07 | 39 | 41 | +2 | 200 | 96 | -104 |
| beaver\_eastus\_07 | flashbots\_eastus\_10 | 40 | 41 | +1 | 197 | 103 | -94 |
### TCP + `bitcode`
To preserve backwards compatibility, we offered a new TCP-only version of system
API along with the existing HTTP/2 one. Using a new temporary endpoint allowed
us to experiment with binary encoding format, so we could finally remove JSON
which other than offering high decoding latency, resulted in \~50% bigger
payloads than necessary due to encoding hex data as strings.
First, we tried to look into `serde` compatible binary encoding formats, like
`bincode` and MessagePack, but both of them do not have full feature
compatibility with `serde_derive`. Because of that, we opted out from `serde`
compatibility and went with
[`bitcode`](https://github.com/SoftbearStudios/bitcode) instead, which currently
sits at the top of the
[Rust Serialization Benchmark](https://david.kolo.ski/rust_serialization_benchmark/)
leaderboard.
This was the result of both a switch to TCP (without any additional tuning yet),
and migration from JSON to bitcode:
However, there’s a big caveat here: we only saw these levels of improvements on
very specific links. The plot above depicts latency from a specifically
broadcast-heavy instance: one that ingests and forwards a lot of orders. We
barely saw any improvements on other instances, which led us down the rabbit
hole of TCP kernel parameters.
### Kernel Parameter Tuning
In order to get the most of raw TCP sockets, it has been fundamental to tune
kernel parameters to achieve optimised connections. To understand better the
changes we’ve done, let’s brush up on some preliminaries.
The *bandwidth-delay product*, or BDP, is a property of a network path, and is
computed as the product of bandwidth, expressed in bytes per second, and
round-trip time (delay), expressed in seconds. In the context of TCP, it
represents how much in-flight data the connection can hold before the sender
must wait for acknowledgments from the receiver.
Let’s make a concrete example: a node in FlowProxy has a upload bandwidth of 1
Gbps, and an Azure link between East US and West Europe is around 85ms
([source](https://learn.microsoft.com/en-us/azure/networking/azure-network-latency?tabs=Americas%2CEastUS)).
This results in a BDP of `1 Gbps × 85 ms = 85 Mbit = 10.625 MB`, meaning that is
the theoretical maximum of data we can have unacknowledged in TCP, assuming
ideal network conditions.
Related to BDP is the *congestion window* (referred as `cwnd` in code). It is a
core TCP mechanism that controls how much data a sender is allowed to have “in
flight” (sent, but not yet acknowledged) at any given time. On the receiver
side, there’s `rwnd` (the receive window), which determines how much
unacknowledged data can accumulate at the receiver side before packets are
dropped. The minimum of `cwnd` and `rwnd` determines your maximum throughput.
One direct consequence of BDP is the following: if
`min(cwnd, rwnd) < size(message)`, that message will need an **additional round
trip to fully transmit!**
By definition, the theoretical maximum size of the congestion window for
communicating over a link matches the BDP. That would mean there is no
congestion at all! However, the network might not be always stable, and packet
loss might happen and RTT may vary over time. As such, the congestion window
dynamically adapts to the appropriate amount of unacknowledged data to not waste
any resources. These mechanisms are called *TCP congestion control algorithms.*
All of these parameters can be modified in the Linux kernel with `sysctl`. The
following are the most important:
* `net.ipv4.tcp_congestion_control`: the default (`cubic`) worked for us.
* `net.ipv4.tcp_window_scaling`: ensure this is turned on.
* `net.ipv4.tcp_rmem`: sets the bounds for `rwnd`. Default and max had to be
significantly increased, some multiple of your expected max message size is a
good guideline. Linux will take care of autoscaling this if window scaling is
turned on.
Other than congestion algorithms, there are other settings that may impact the
congestion window. One of them is called “slow start after idle”, which `sysctl`
setting is `net.ipv4.tcp_slow_start_after_idle` . A connection is considered
idle if a packet hasn’t been sent for a certain amount of time, computed as a
function of the RTT, but with a default minimum of 200ms that we would be used
in case of a 85ms RTT. In case of idleness, the congestion window is greatly
reduced, diminishing throughput and increasing latency.
Given connection between FlowProxy instances are long lived, and can be bursty
in period of high-traffic, this setting should be disabled. In the picture below
you can see the tremendous impact of the switch on low volume instances:
Call duration latency (p99) before and after the
`net.ipv4.tcp_slow_start_after_idle` has been disabled.
This setting was exactly why the high message volume instances displayed really
good latency after the switch to TCP, and the others didn’t: their TCP
connections were never idle, so the congestion windows were never reset.
The general takeaway here is that Linux TCP settings are very conservative and
always try to save on resources, and should almost always be changed depending
on your workload.
### Latency Comparison
After all these modifications, we were finally approaching optimality in terms
of order propagation: variance for big and small messages, over long and short
links was reduced significantly. Check out the full comparison of P99 latencies
below, each with 24 hours worth of data:
**Before**
Europe (high latency routes)
{
from: 'beaver_eastus',
to: 'nethermind_we',
metrics: { latency: 1947.05 },
waypoints: [
[-42, 47],
[-18, 49],
],
},
{
from: 'beaver_eastus',
to: 'flashbots_we',
metrics: { latency: 1789.05 },
waypoints: [
[-40, 48],
[-15, 50],
],
},
{
from: 'beaver_eastus',
to: 'beaver_we',
metrics: { latency: 1521.6 },
waypoints: [
[-42, 44],
[-18, 46],
],
},
// US East local
{
from: 'beaver_eastus',
to: 'nethermind_eastus',
metrics: { latency: 292.45 },
},
{
from: 'nethermind_eastus',
to: 'flashbots_eastus',
metrics: { latency: 58.65 },
},
{
from: 'nethermind_eastus',
to: 'beaver_eastus',
metrics: { latency: 49.63 },
},
{
from: 'flashbots_eastus',
to: 'beaver_eastus',
metrics: { latency: 56.19 },
},
{
from: 'flashbots_eastus',
to: 'nethermind_eastus',
metrics: { latency: 26.18 },
},
// Transatlantic: Nethermind EUS -> Europe
{
from: 'nethermind_eastus',
to: 'beaver_we',
metrics: { latency: 194.66 },
waypoints: [
[-35, 43],
[-10, 45],
],
},
{
from: 'nethermind_eastus',
to: 'nethermind_we',
metrics: { latency: 193.11 },
waypoints: [
[-32, 50],
[-8, 52],
],
},
{
from: 'nethermind_eastus',
to: 'flashbots_we',
metrics: { latency: 192.13 },
waypoints: [
[-38, 46],
[-12, 48],
],
},
// Transatlantic: Flashbots EUS -> Europe
{
from: 'flashbots_eastus',
to: 'beaver_we',
metrics: { latency: 201.54 },
waypoints: [
[-30, 41],
[-5, 44],
],
},
{
from: 'flashbots_eastus',
to: 'nethermind_we',
metrics: { latency: 200.27 },
waypoints: [
[-25, 47],
[0, 50],
],
},
{
from: 'flashbots_eastus',
to: 'flashbots_we',
metrics: { latency: 199.81 },
waypoints: [
[-28, 44],
[-3, 47],
],
},
]}
metricKey="latency"
metricRange={[0, 2000]}
height={350}
center={[-35, 45]}
scale={350}
showLegend={true}
/>
The maximum P99 here is ~1950ms.
**After**
Europe
{
from: 'beaver_eastus',
to: 'nethermind_we',
metrics: { latency: 50.06 },
waypoints: [
[-42, 47],
[-18, 49],
],
},
{
from: 'beaver_eastus',
to: 'flashbots_we',
metrics: { latency: 43.51 },
waypoints: [
[-40, 48],
[-15, 50],
],
},
{
from: 'beaver_eastus',
to: 'beaver_we',
metrics: { latency: 43.82 },
waypoints: [
[-42, 44],
[-18, 46],
],
},
// US East local
{
from: 'beaver_eastus',
to: 'nethermind_eastus',
metrics: { latency: 7.55 },
},
{
from: 'nethermind_eastus',
to: 'flashbots_eastus',
metrics: { latency: 9.0 },
},
{
from: 'nethermind_eastus',
to: 'beaver_eastus',
metrics: { latency: 7.55 },
},
{
from: 'flashbots_eastus',
to: 'beaver_eastus',
metrics: { latency: 4.4 },
},
{
from: 'flashbots_eastus',
to: 'nethermind_eastus',
metrics: { latency: 2.25 },
},
// Transatlantic: Nethermind EUS -> Europe
{
from: 'nethermind_eastus',
to: 'beaver_we',
metrics: { latency: 47.88 },
waypoints: [
[-35, 43],
[-10, 45],
],
},
{
from: 'nethermind_eastus',
to: 'nethermind_we',
metrics: { latency: 46.47 },
waypoints: [
[-32, 50],
[-8, 52],
],
},
{
from: 'nethermind_eastus',
to: 'flashbots_we',
metrics: { latency: 48.87 },
waypoints: [
[-38, 46],
[-12, 48],
],
},
// Transatlantic: Flashbots EUS -> Europe
{
from: 'flashbots_eastus',
to: 'beaver_we',
metrics: { latency: 42.32 },
waypoints: [
[-30, 41],
[-5, 44],
],
},
{
from: 'flashbots_eastus',
to: 'nethermind_we',
metrics: { latency: 45.89 },
waypoints: [
[-25, 47],
[0, 50],
],
},
{
from: 'flashbots_eastus',
to: 'flashbots_we',
metrics: { latency: 44.28 },
waypoints: [
[-28, 44],
[-3, 47],
],
},
]}
metricKey="latency"
metricRange={[0, 2000]}
height={350}
center={[-35, 45]}
scale={350}
showLegend={true}
/>
The maximum P99 here is ~50ms.
There was one more change that we knew would be impactful, not on latency but on
CPU usage.
### mTLS
The last improvement we’ve been working on during this collaboration has been
[mutual TLS](https://www.cloudflare.com/learning/access-management/what-is-mutual-tls/)
(mTLS). If required by the server, the client must present a valid X509 TLS
certificate to authenticate itself. This works, because all instances know each
other's TLS certificates through a central service registry.
Currently, FlowProxy authenticates other instances on a message-by-message
basis: every message it gets needs to be signed by a known ECDSA public key.
This is secure, but wasteful: every message needs to be a) signed by the sender,
and b) verified by the receiver. mTLS would change authentication on a message
basis, to authentication on a connection basis. Authentication would occur
during connection setup, and after a successful authentication, all messages
sent on that connection are transitively authenticated too.
As of December 22, 2025, this feature hasn’t been deployed yet on a production
environment, but from the picture below we can already appreciate its effects on
the staging environment.
Global CPU usage and thread usage after deployment of mTLS on a staging
environment.
From this preliminary result we can see a 50% reduction of CPU and thread usage,
lowering total CPU usage of FlowProxy from \~10% to 5%. While this may seem like
a minor improvement, freeing up CPU to be used by other critical services such
as rbuilder can make the difference between winning and losing a MEV-Boost
auction.
## Next Steps
At the moment we’re pretty satisfied with the performance of FlowProxy, but
there is still room for improvement. In particular, we haven’t yet worked on the
communication between a FlowProxy instance and its local builder, which still
uses JSON-RPC over HTTP, although via localhost. Assuming these two services
keep running on the same machine, we could implement a shared memory based
transport. While this would be clearly a benefit over the status quo, we’re
coming closer to a point of diminishing returns compared to other improvements.
As such, next steps could move away from FlowProxy and concentrate over a
different part of the stack of BuilderNet and its architecture.
# FlowProxy: Approaching Optimality
import { Accordion, Accordions } from 'fumadocs-ui/components/accordion';
import { GlobalMap } from '@/components/diagrams/map';
## Background
In
[**Introducing FlowProxy**](https://collective.flashbots.net/t/introducing-flowproxy/5341),
[Chainbound](https://chainbound.io/) collaborated with Flashbots to deprecate
its [original orderflow proxy](https://github.com/flashbots/buildernet-orderflow-proxy) implementation
in Go in favor of a new one built in Rust, created with the goal of reducing
end-to-end latency, improving efficiency, and increasing observability in
[BuilderNet](https://buildernet.org/)'s networking layer. This first
collaboration aimed at bringing it to production-ready quality and ready for
deployment, already observing good improvements in networking and processing
latency.
This second collaboration expanded on the *Next Steps* section outlined in the
previous report, with success. It includes a quantitative analysis of order
losses and latencies across BuilderNet, and extensive work to improve the
transport layer to reduce both CPU and memory usage. We encountered a couple of
non-obvious learnings that we outline below, and we hope this will be useful to
the community.
## Network Analysis
FlowProxy features a Clickhouse integration which makes possible to record
*bundle receipts.* A bundle receipt consists of a short summary that includes
the timestamp of when the bundle was sent and received by instances, along with
its raw size in bytes (useful for tracking size → latency correlation). We used
this data to better understand FlowProxy’s network performance in its entirety,
and to discover what improvements to focus on.
### Bundle Loss
We started with analyzing bundle loss between instances. The table below reports
the percentage of bundle loss on individual links, during a small time range of
congestion:
| src | dst | lost | total | loss\_rate\_pct |
| ---------------------- | ---------------------- | ---- | ------ | --------------- |
| beaver\_eastus\_07 | nethermind\_we\_08 | 7857 | 35848 | 21.92% |
| beaver\_eastus\_07 | flashbots\_we\_09 | 1969 | 96597 | 2.04% |
| beaver\_eastus\_07 | beaver\_we\_08 | 1354 | 94000 | 1.44% |
| beaver\_eastus\_07 | flashbots\_eastus\_10 | 2 | 140801 | 0.00% |
| beaver\_eastus\_07 | nethermind\_eastus\_07 | 1 | 141534 | 0.00% |
| nethermind\_eastus\_07 | flashbots\_eastus\_10 | 10 | 35315 | 0.03% |
| nethermind\_eastus\_07 | beaver\_eastus\_07 | 10 | 35377 | 0.03% |
| nethermind\_eastus\_07 | flashbots\_we\_09 | 6 | 34560 | 0.02% |
| nethermind\_eastus\_07 | beaver\_we\_08 | 6 | 35216 | 0.02% |
| nethermind\_eastus\_07 | nethermind\_we\_08 | 4 | 34490 | 0.01% |
| flashbots\_eastus\_10 | beaver\_we\_08 | 4 | 34877 | 0.01% |
| flashbots\_eastus\_10 | flashbots\_we\_09 | 4 | 34266 | 0.01% |
| flashbots\_eastus\_10 | nethermind\_we\_08 | 3 | 34156 | 0.01% |
| flashbots\_eastus\_10 | beaver\_eastus\_07 | 1 | 35054 | 0.00% |
| src | dst | lost | total | loss rate |
| ---------------------- | ---------------------- | ---- | ------ | --------- |
| beaver\_eastus\_07 | nethermind\_we\_08 | 7857 | 35848 | 21.92% |
| beaver\_eastus\_07 | flashbots\_we\_09 | 1969 | 96597 | 2.04% |
| beaver\_eastus\_07 | beaver\_we\_08 | 1354 | 94000 | 1.44% |
| nethermind\_eastus\_07 | flashbots\_mkosi\_1 | 58 | 31205 | 0.19% |
| beaver\_eastus\_07 | flashbots\_mkosi\_1 | 13 | 125199 | 0.01% |
| flashbots\_eastus\_10 | flashbots\_mkosi\_1 | 11 | 31415 | 0.04% |
| nethermind\_eastus\_07 | flashbots\_eastus\_10 | 10 | 35315 | 0.03% |
| nethermind\_eastus\_07 | beaver\_eastus\_07 | 10 | 35377 | 0.03% |
| nethermind\_eastus\_07 | flashbots\_we\_09 | 6 | 34560 | 0.02% |
| nethermind\_eastus\_07 | beaver\_we\_08 | 6 | 35216 | 0.02% |
| flashbots\_eastus\_10 | beaver\_we\_08 | 4 | 34877 | 0.01% |
| nethermind\_eastus\_07 | nethermind\_we\_08 | 4 | 34490 | 0.01% |
| flashbots\_eastus\_10 | flashbots\_we\_09 | 4 | 34266 | 0.01% |
| nethermind\_we\_08 | flashbots\_eastus\_10 | 3 | 113560 | 0.00% |
| nethermind\_we\_08 | beaver\_eastus\_07 | 3 | 116296 | 0.00% |
| nethermind\_we\_08 | nethermind\_eastus\_07 | 3 | 113862 | 0.00% |
| flashbots\_eastus\_10 | nethermind\_we\_08 | 3 | 34156 | 0.01% |
| beaver\_eastus\_07 | flashbots\_eastus\_10 | 2 | 140801 | 0.00% |
| flashbots\_mkosi\_1 | beaver\_we\_08 | 1 | 95552 | 0.00% |
| flashbots\_mkosi\_1 | nethermind\_eastus\_07 | 1 | 95172 | 0.00% |
| nethermind\_we\_08 | beaver\_we\_08 | 1 | 104890 | 0.00% |
| flashbots\_mkosi\_1 | beaver\_eastus\_07 | 1 | 96537 | 0.00% |
| flashbots\_mkosi\_1 | flashbots\_we\_09 | 1 | 95015 | 0.00% |
| flashbots\_eastus\_10 | beaver\_eastus\_07 | 1 | 35054 | 0.00% |
| beaver\_eastus\_07 | nethermind\_eastus\_07 | 1 | 141534 | 0.00% |
| flashbots\_mkosi\_1 | flashbots\_eastus\_10 | 1 | 93635 | 0.00% |
| nethermind\_we\_08 | flashbots\_we\_09 | 1 | 103316 | 0.00% |
Or visualized:
Europe (northern route)
{
from: 'beaver_eastus',
to: 'nethermind_we',
metrics: { bundleLoss: 21.92 },
waypoints: [
[-42, 47],
[-18, 49],
],
},
{
from: 'beaver_eastus',
to: 'flashbots_we',
metrics: { bundleLoss: 2.04 },
waypoints: [
[-40, 48],
[-15, 50],
],
},
{
from: 'beaver_eastus',
to: 'beaver_we',
metrics: { bundleLoss: 1.44 },
waypoints: [
[-42, 44],
[-18, 46],
],
},
// US East local
{
from: 'beaver_eastus',
to: 'flashbots_eastus',
metrics: { bundleLoss: 0.0 },
},
{
from: 'beaver_eastus',
to: 'nethermind_eastus',
metrics: { bundleLoss: 0.0 },
},
{
from: 'nethermind_eastus',
to: 'flashbots_eastus',
metrics: { bundleLoss: 0.03 },
},
{
from: 'nethermind_eastus',
to: 'beaver_eastus',
metrics: { bundleLoss: 0.03 },
},
// Transatlantic: Nethermind EUS -> Europe (middle routes)
{
from: 'nethermind_eastus',
to: 'flashbots_we',
metrics: { bundleLoss: 0.02 },
waypoints: [
[-38, 46],
[-12, 48],
],
},
{
from: 'nethermind_eastus',
to: 'beaver_we',
metrics: { bundleLoss: 0.02 },
waypoints: [
[-35, 43],
[-10, 45],
],
},
{
from: 'nethermind_eastus',
to: 'nethermind_we',
metrics: { bundleLoss: 0.01 },
waypoints: [
[-32, 50],
[-8, 52],
],
},
// Transatlantic: Flashbots EUS -> Europe (southern routes)
{
from: 'flashbots_eastus',
to: 'beaver_we',
metrics: { bundleLoss: 0.01 },
waypoints: [
[-30, 41],
[-5, 44],
],
},
{
from: 'flashbots_eastus',
to: 'flashbots_we',
metrics: { bundleLoss: 0.01 },
waypoints: [
[-28, 44],
[-3, 47],
],
},
{
from: 'flashbots_eastus',
to: 'nethermind_we',
metrics: { bundleLoss: 0.01 },
waypoints: [
[-25, 47],
[0, 50],
],
},
{
from: 'flashbots_eastus',
to: 'beaver_eastus',
metrics: { bundleLoss: 0.0 },
},
// European interconnections
{ from: 'beaver_we', to: 'nethermind_we', metrics: { bundleLoss: 0.01 } },
{ from: 'beaver_we', to: 'flashbots_we', metrics: { bundleLoss: 0.01 } },
{
from: 'nethermind_we',
to: 'flashbots_we',
metrics: { bundleLoss: 0.01 },
},
]}
metricKey="bundleLoss"
metricRange={[0, 25]}
height={350}
center={[-35, 45]}
scale={350}
showLegend={true}
/>
This map represents a simplified view of the BuilderNet topology, it's not
geographically accurate.
We can see in particular that inter-continental links are mostly affected by
bundle loss, especially the instance `beaver_eastus_07` which receives the most
flow in that region.
The main reason bundles are marked as “lost” is when they time out, or when
buffers overflow and backpressure is applied. This confirms what we
[explored](https://collective.flashbots.net/t/introducing-flowproxy/5341#p-10727-http-connection-pools-11)
in the last post: that HTTP/1.1 and its incapacity to multiplex requests makes
it highly inadequate to deal with BuilderNet’s workload.
### Latency
From the table below we can see latency between individual BuilderNet links
during high congestion.
| src | dst | p50\_ms | p99\_ms | corr\_payload\_size | observations |
| ---------------------- | ---------------------- | ------- | -------- | ------------------- | ------------ |
| beaver\_eastus\_07 | nethermind\_we\_08 | 39.619 | 1947.045 | -0.06 | 27991 |
| beaver\_eastus\_07 | flashbots\_we\_09 | 38.022 | 1789.051 | -0.06 | 94628 |
| beaver\_eastus\_07 | beaver\_we\_08 | 39.440 | 1521.599 | -0.04 | 92646 |
| beaver\_eastus\_07 | nethermind\_eastus\_07 | 62.639 | 292.451 | -0.02 | 141533 |
| nethermind\_eastus\_07 | beaver\_we\_08 | 33.381 | 194.660 | 0.59 | 35210 |
| nethermind\_eastus\_07 | nethermind\_we\_08 | 31.541 | 193.107 | 0.58 | 34486 |
| nethermind\_eastus\_07 | flashbots\_we\_09 | 30.992 | 192.134 | 0.60 | 34554 |
| nethermind\_eastus\_07 | flashbots\_eastus\_10 | -0.662 | 58.648 | 0.06 | 35305 |
| nethermind\_eastus\_07 | beaver\_eastus\_07 | -4.514 | 49.629 | 0.02 | 35367 |
| flashbots\_eastus\_10 | beaver\_we\_08 | 40.197 | 201.535 | 0.60 | 34873 |
| flashbots\_eastus\_10 | nethermind\_we\_08 | 38.467 | 200.267 | 0.59 | 34153 |
| flashbots\_eastus\_10 | flashbots\_we\_09 | 37.945 | 199.814 | 0.61 | 34262 |
| flashbots\_eastus\_10 | beaver\_eastus\_07 | 2.506 | 56.186 | 0.02 | 35053 |
| flashbots\_eastus\_10 | nethermind\_eastus\_07 | -7.746 | 26.184 | 0.04 | 35048 |
| src | dst | p50\_ms | p90\_ms | p99\_ms | p999\_ms | corr\_payload\_size | observations |
| ---------------------- | ---------------------- | ------- | -------- | -------- | -------- | ------------------- | ------------ |
| beaver\_eastus\_07 | nethermind\_we\_08 | 39.619 | 1668.253 | 1947.045 | 2005.370 | -0.06 | 27991 |
| beaver\_eastus\_07 | flashbots\_we\_09 | 38.022 | 1414.608 | 1789.051 | 1978.707 | -0.06 | 94628 |
| beaver\_eastus\_07 | beaver\_we\_08 | 39.440 | 852.713 | 1521.599 | 1970.474 | -0.04 | 92646 |
| beaver\_eastus\_07 | flashbots\_mkosi\_1 | 39.591 | 419.213 | 1099.179 | 1221.009 | -0.03 | 125186 |
| nethermind\_we\_08 | flashbots\_eastus\_10 | 44.564 | 126.509 | 294.049 | 502.033 | -0.21 | 113557 |
| beaver\_eastus\_07 | nethermind\_eastus\_07 | 62.639 | 61.099 | 292.451 | 476.941 | -0.02 | 141533 |
| flashbots\_mkosi\_1 | flashbots\_eastus\_10 | 43.261 | 84.517 | 281.969 | 560.069 | 0.16 | 93634 |
| flashbots\_mkosi\_1 | beaver\_eastus\_07 | 44.489 | 66.412 | 257.931 | 519.216 | 0.24 | 96536 |
| nethermind\_we\_08 | beaver\_eastus\_07 | 46.272 | 97.637 | 227.358 | 323.382 | 0.32 | 116293 |
| nethermind\_we\_08 | nethermind\_eastus\_07 | 51.281 | 101.732 | 224.596 | 324.134 | 0.32 | 113859 |
| nethermind\_we\_08 | beaver\_we\_08 | 25.860 | 5.459 | 216.067 | 324.788 | -0.02 | 104889 |
| flashbots\_eastus\_10 | flashbots\_mkosi\_1 | 40.740 | 45.731 | 202.758 | 286.997 | 0.59 | 31404 |
| flashbots\_mkosi\_1 | nethermind\_eastus\_07 | 49.122 | 61.092 | 202.012 | 273.676 | 0.37 | 95171 |
| flashbots\_eastus\_10 | beaver\_we\_08 | 40.197 | 46.283 | 201.535 | 284.387 | 0.60 | 34873 |
| flashbots\_eastus\_10 | nethermind\_we\_08 | 38.467 | 46.728 | 200.267 | 283.487 | 0.59 | 34153 |
| flashbots\_eastus\_10 | flashbots\_we\_09 | 37.945 | 44.632 | 199.814 | 281.712 | 0.61 | 34262 |
| nethermind\_eastus\_07 | flashbots\_mkosi\_1 | 33.733 | 37.802 | 195.461 | 204.237 | 0.59 | 31147 |
| nethermind\_eastus\_07 | beaver\_we\_08 | 33.381 | 38.238 | 194.660 | 201.383 | 0.59 | 35210 |
| nethermind\_eastus\_07 | nethermind\_we\_08 | 31.541 | 39.041 | 193.107 | 208.030 | 0.58 | 34486 |
| nethermind\_eastus\_07 | flashbots\_we\_09 | 30.992 | 36.886 | 192.134 | 203.846 | 0.60 | 34554 |
| flashbots\_we\_09 | flashbots\_eastus\_10 | 44.702 | 52.008 | 151.492 | 352.513 | 0.45 | 23666 |
| nethermind\_we\_08 | flashbots\_mkosi\_1 | 3.024 | 5.885 | 149.235 | 237.358 | -0.02 | 128776 |
| flashbots\_we\_09 | nethermind\_eastus\_07 | 51.533 | 53.694 | 135.308 | 214.133 | 0.65 | 23682 |
| flashbots\_we\_09 | beaver\_eastus\_07 | 46.286 | 51.180 | 131.223 | 213.616 | 0.58 | 23676 |
| beaver\_we\_08 | flashbots\_eastus\_10 | 42.202 | 45.092 | 124.000 | 320.903 | 0.46 | 21596 |
| beaver\_we\_08 | beaver\_eastus\_07 | 43.786 | 46.276 | 110.435 | 222.164 | 0.54 | 21630 |
| nethermind\_we\_08 | flashbots\_we\_09 | 0.300 | 4.165 | 88.781 | 154.436 | -0.01 | 105315 |
| flashbots\_mkosi\_1 | nethermind\_we\_08 | -1.529 | 1.878 | 84.565 | 172.061 | -0.02 | 109297 |
| beaver\_we\_08 | nethermind\_eastus\_07 | 49.078 | 50.298 | 62.551 | 215.596 | 0.82 | 21608 |
| nethermind\_eastus\_07 | flashbots\_eastus\_10 | -0.662 | -2.668 | 58.648 | 220.645 | 0.06 | 35305 |
| flashbots\_mkosi\_1 | beaver\_we\_08 | -0.075 | 2.209 | 57.603 | 115.534 | -0.02 | 95551 |
| flashbots\_eastus\_10 | beaver\_eastus\_07 | 2.506 | 5.522 | 56.186 | 188.124 | 0.02 | 35053 |
| flashbots\_mkosi\_1 | flashbots\_we\_09 | 2.188 | 9.008 | 52.778 | 91.397 | -0.00 | 95014 |
| nethermind\_eastus\_07 | beaver\_eastus\_07 | -4.514 | -1.465 | 49.629 | 194.338 | 0.02 | 35367 |
| flashbots\_eastus\_10 | nethermind\_eastus\_07 | -7.746 | -9.508 | 26.184 | 101.051 | 0.04 | 35048 |
| flashbots\_we\_09 | nethermind\_we\_08 | -1.307 | 3.241 | 17.573 | 83.503 | 0.02 | 23585 |
| beaver\_we\_08 | nethermind\_we\_08 | 1.076 | 3.533 | 14.321 | 98.753 | 0.03 | 21147 |
| flashbots\_we\_09 | flashbots\_mkosi\_1 | 3.017 | 4.058 | 11.831 | 43.950 | 0.06 | 23651 |
| beaver\_we\_08 | flashbots\_mkosi\_1 | 3.506 | 4.566 | 9.304 | 22.996 | 0.10 | 23289 |
| beaver\_we\_08 | flashbots\_we\_09 | 1.137 | 1.804 | 8.416 | 25.661 | 0.10 | 20771 |
| beaver\_we\_08 | flashbots\_we\_09 | -1.624 | -0.159 | 8.029 | 67.550 | 0.05 | 20979 |
The negative p50 latencies here are due to [clock
drift](https://en.wikipedia.org/wiki/Clock_drift).
Or, with P99 latencies visualized:
Europe (high latency routes)
{
from: 'beaver_eastus',
to: 'nethermind_we',
metrics: { latency: 1947.05 },
waypoints: [
[-42, 47],
[-18, 49],
],
},
{
from: 'beaver_eastus',
to: 'flashbots_we',
metrics: { latency: 1789.05 },
waypoints: [
[-40, 48],
[-15, 50],
],
},
{
from: 'beaver_eastus',
to: 'beaver_we',
metrics: { latency: 1521.6 },
waypoints: [
[-42, 44],
[-18, 46],
],
},
// US East local
{
from: 'beaver_eastus',
to: 'nethermind_eastus',
metrics: { latency: 292.45 },
},
{
from: 'nethermind_eastus',
to: 'flashbots_eastus',
metrics: { latency: 58.65 },
},
{
from: 'nethermind_eastus',
to: 'beaver_eastus',
metrics: { latency: 49.63 },
},
{
from: 'flashbots_eastus',
to: 'beaver_eastus',
metrics: { latency: 56.19 },
},
{
from: 'flashbots_eastus',
to: 'nethermind_eastus',
metrics: { latency: 26.18 },
},
// Transatlantic: Nethermind EUS -> Europe
{
from: 'nethermind_eastus',
to: 'beaver_we',
metrics: { latency: 194.66 },
waypoints: [
[-35, 43],
[-10, 45],
],
},
{
from: 'nethermind_eastus',
to: 'nethermind_we',
metrics: { latency: 193.11 },
waypoints: [
[-32, 50],
[-8, 52],
],
},
{
from: 'nethermind_eastus',
to: 'flashbots_we',
metrics: { latency: 192.13 },
waypoints: [
[-38, 46],
[-12, 48],
],
},
// Transatlantic: Flashbots EUS -> Europe
{
from: 'flashbots_eastus',
to: 'beaver_we',
metrics: { latency: 201.54 },
waypoints: [
[-30, 41],
[-5, 44],
],
},
{
from: 'flashbots_eastus',
to: 'nethermind_we',
metrics: { latency: 200.27 },
waypoints: [
[-25, 47],
[0, 50],
],
},
{
from: 'flashbots_eastus',
to: 'flashbots_we',
metrics: { latency: 199.81 },
waypoints: [
[-28, 44],
[-3, 47],
],
},
]}
metricKey="latency"
metricRange={[0, 2000]}
height={350}
center={[-35, 45]}
scale={350}
showLegend={true}
/>
As with the previous table, we can see the highest latency between the most
active inter-region links, peaking at nearly 2s in p99. Another interesting
result is the correlation with payload size (`corr_payload_size`): in normal
working conditions (low p99s), it is more pronounced. This could mean a couple
of things that will be useful to know later:
* We’re not able to send large payloads in a single transmission (related to
[BDP](https://en.wikipedia.org/wiki/Bandwidth-delay_product), which we’ll talk
about below).
* Preparing the order for transmission, or processing the order on the receiver
side, takes a noticeably longer time the bigger the message. Some correlation
is expected here, but it should be minimal. The main processing steps in the
hot path here are JSON encoding / decoding, and signing / signature
verification.
## Improvements
### Thread modelling
FlowProxy runs with the [Tokio](https://docs.rs/tokio/latest/tokio/)
asynchronous runtime. The initial implementation of the proxy indiscriminately
used Tokio tasks for all different kind of workloads, including CPU intensive
operations like signature recovery, signing and decoding transactions. This
approach is not ideal because the runtime and its tasks are fundamentally
optimized for non-blocking, I/O-bound work, and using it for other
[blocking or CPU-bound work](https://docs.rs/tokio/latest/tokio/index.html#cpu-bound-tasks-and-blocking-code)
*will* increase tail latencies.
Tokio schedules many lightweight tasks onto a small number of OS threads. If a
task performs CPU-heavy or blocking work, it can monopolise a worker thread,
preventing other tasks from making progress. We suspected this could partly be
causing some of the high tail latencies we were seeing.
The Tokio authors recommend using
[`tokio::task::spawn_blocking`](https://docs.rs/tokio/latest/tokio/task/fn.spawn_blocking.html),
which we tried initially. This will spawn (or reuse) a thread managed by the
runtime that is purely used for CPU-bound and blocking operations. However, this
resulted in many more threads being spawned than we knew was necessary, and also
had more overhead than expected. We didn’t dive into this too much, but
intuitively it seemed like the blocking thread scheduler was not reusing threads
effectively ([Github issue](https://github.com/BuilderNet/FlowProxy/pull/141)).
To mitigate this, we introduced a configurable pool of specialised
[rayon](https://docs.rs/rayon/latest/rayon/) threads for compute-heavy
operations, and tweaked the number of Tokio worker threads to match production
environment requirements. This setup allows also to have a healthy environment
where we can control how much resources a certain service is using, since the
same machine would also run other processes like the block builder.
**Side Note**
While experimenting with the parameters, we observed how moving *some* specific
compute-heavy operations resulted in a worse order processing latency. An
interesting learning was that on a busy machine with many threads, sending tasks
off to a different thread than the currently executing one *can* increase
latency significantly. The cost of scheduling a new thread (context switching,
waiting) should be taken into account, and is very context dependent!
### HTTP/2
HTTP/2 was designed to address various performance limitations of HTTP/1.1 while
keeping the same semantics. Among various improvements, the most impactful for
FlowProxy is **multiplexing**: with HTTP/1.1 only one request/response can be in
flight per TCP connection (called *head-of-line blocking),* while HTTP/2 allows
multiplexing multiple requests and responses over a single TCP connection, using
*streams*.
Streams are logical, bidirectional channels within one connection. They’re
managed by *windows*: credit-based flow control mechanisms to limit how much
data can be sent on a single stream, to ensure it doesn’t starve the connection.
This multiplexing allowed us to greatly reduce the number of open connections,
and improve connection reuse, which we already hinted was a source of message
loss.
Upgrading to HTTP/2 was the first improvement we rolled out, because of its
complete backwards compatibility: communication between and towards instances
running on a previous version of FlowProxy would simply fallback to HTTP/1.1.
Below, you can see how the number of failures (read: lost messages) have been
essentially reduced to 0 after its deployment.
HTTP failures after deployment of HTTP/2
While request failures dropped, latency didn’t significantly improve. In
particular, we’ve observed some improvement over small requests (with body size
less than 32KiB) over inter-regional links, as we can see below. However, for
same-region requests and bigger messages the situation remained identical or
slightly worsened.
RPC call duration latency (p99) before and after the deployment of HTTP/2.
This was a very different result compared to our staging environment, consisting
of four nodes distributed between East US and West Europe. We think the main
culprit is an overall different topology and network load compared to the
production environment, which would be hard to completely emulate. After this
result, we started looking into tuning configurations.
FlowProxy instances operate with a reverse [HAProxy](https://www.haproxy.org/)
that sits before the user and system endpoint, with the latter used for internal
orderflow sharing. The proxy exposes some
[HTTP/2 tuning configurations](https://docs.haproxy.org/3.2/configuration.html#tune.h2.be.initial-window-size:~:text=%2D%20tune.h2,copy%2Dfwd%2Dsend)
that could help further reducing latency and spikiness.
The dimensions in which we could operate were:
* The number of maximum open streams;
* The size of the window buffers;
* Creating dedicated clients for small and big requests.
While tuning those resulted in marginal improvements, we were still working on
high-level abstractions, without much control over the metal. Moreover, HTTP/2
windows play a similar role to TCP congestion control / window scaling,
resulting in some overhead and confusion about how the two interoperate. Because
of this, we decided to pause HTTP/2 tuning efforts, and focus on a full
migration to raw TCP (with TLS) with the
[msg-rs](https://github.com/chainbound/msg-rs) messaging library.
**Bundle loss after HTTP/2**
After the deployment of this improvement, we analyzed bundle loss once again
(read a full analysis
[**here**](https://www.notion.so/PUBLIC-FlowProxy-Bundle-Receipts-Analysis-v2-2a45abfafc1980a29bb9fda91b3dd16d?pvs=21)).
The table below contains a day worth of data, that includes both periods of low
activity and high activity. We can see that bundle loss has essentially
disappeared. Sample for 2025-11-14:
| src | dst | lost | total |
| ---------------------- | ---------------------- | ---- | -------- |
| nethermind\_eastus\_07 | beaver\_we\_08 | 28 | 8894080 |
| beaver\_eastus\_07 | flashbots\_eastus\_10 | 8 | 3001624 |
| beaver\_eastus\_07 | flashbots\_we\_09 | 8 | 11168780 |
| beaver\_eastus\_07 | nethermind\_eastus\_07 | 8 | 30111251 |
| beaver\_eastus\_07 | beaver\_we\_08 | 8 | 10864297 |
| beaver\_eastus\_07 | nethermind\_we\_08 | 8 | 10069603 |
| nethermind\_eastus\_07 | flashbots\_we\_09 | 10 | 9028245 |
| nethermind\_eastus\_07 | nethermind\_we\_08 | 8 | 9052880 |
| nethermind\_eastus\_07 | beaver\_eastus\_07 | 6 | 10309013 |
| nethermind\_eastus\_07 | flashbots\_eastus\_10 | 5 | 10302474 |
| flashbots\_eastus\_10 | beaver\_we\_08 | 14 | 9024652 |
| flashbots\_eastus\_10 | nethermind\_eastus\_07 | 6 | 10443797 |
| flashbots\_eastus\_10 | beaver\_eastus\_07 | 6 | 10458639 |
| flashbots\_eastus\_10 | nethermind\_we\_08 | 4 | 9087691 |
| flashbots\_eastus\_10 | flashbots\_we\_09 | 2 | 9115212 |
| src | dst | count | total | loss pctg |
| --------------------------------------------- | --------------------------------------------- | ----- | -------- | --------- |
| flashbots\_test\_1 | beaver\_eastus\_07 | 47 | 5297796 | 0% |
| buildernet\_nethermind\_azure\_eastus\_07 | buildernet\_beaver\_azure\_westeurope\_08 | 28 | 8894080 | 0% |
| buildernet\_beaver\_azure\_westeurope\_08 | buildernet\_beaver\_azure\_eastus\_07 | 24 | 5565633 | 0% |
| buildernet\_flashbots\_azure\_eastus\_10 | buildernet\_flashbots\_mkosi\_test\_1 | 18 | 9283988 | 0% |
| buildernet\_beaver\_azure\_westeurope\_08 | buildernet\_nethermind\_azure\_eastus\_07 | 16 | 5547538 | 0% |
| buildernet\_beaver\_azure\_eastus\_07 | buildernet\_flashbots\_mkosi\_test\_1 | 16 | 10189003 | 0% |
| buildernet\_flashbots\_azure\_eastus\_10 | buildernet\_beaver\_azure\_westeurope\_08 | 14 | 9024652 | 0% |
| buildernet\_beaver\_azure\_westeurope\_08 | buildernet\_flashbots\_azure\_eastus\_10 | 10 | 5578403 | 0% |
| buildernet\_nethermind\_azure\_eastus\_07 | buildernet\_flashbots\_azure\_westeurope\_09 | 10 | 9028245 | 0% |
| buildernet\_flashbots\_mkosi\_test\_1 | buildernet\_flashbots\_azure\_eastus\_10 | 9 | 6146894 | 0% |
| buildernet\_flashbots\_mkosi\_test\_1 | buildernet\_nethermind\_azure\_eastus\_07 | 9 | 6222701 | 0% |
| buildernet\_nethermind\_azure\_eastus\_07 | buildernet\_nethermind\_azure\_westeurope\_08 | 8 | 9052880 | 0% |
| buildernet\_beaver\_azure\_eastus\_07 | buildernet\_flashbots\_azure\_eastus\_10 | 8 | 3001624 | 0% |
| buildernet\_beaver\_azure\_eastus\_07 | buildernet\_flashbots\_azure\_westeurope\_09 | 8 | 11168780 | 0% |
| buildernet\_beaver\_azure\_eastus\_07 | buildernet\_nethermind\_azure\_eastus\_07 | 8 | 30111251 | 0% |
| buildernet\_beaver\_azure\_eastus\_07 | buildernet\_beaver\_azure\_westeurope\_08 | 8 | 10864297 | 0% |
| buildernet\_beaver\_azure\_eastus\_07 | buildernet\_nethermind\_azure\_westeurope\_08 | 8 | 10069603 | 0% |
| buildernet\_nethermind\_azure\_westeurope\_08 | buildernet\_flashbots\_azure\_eastus\_10 | 7 | 6573168 | 0% |
| buildernet\_flashbots\_azure\_westeurope\_09 | buildernet\_beaver\_azure\_eastus\_07 | 7 | 6078754 | 0% |
| buildernet\_nethermind\_azure\_eastus\_07 | buildernet\_beaver\_azure\_eastus\_07 | 6 | 10309013 | 0% |
| buildernet\_flashbots\_azure\_eastus\_10 | buildernet\_nethermind\_azure\_eastus\_07 | 6 | 10443797 | 0% |
| buildernet\_nethermind\_azure\_eastus\_07 | buildernet\_flashbots\_mkosi\_test\_1 | 6 | 8967508 | 0% |
| buildernet\_flashbots\_azure\_eastus\_10 | buildernet\_beaver\_azure\_eastus\_07 | 6 | 10458639 | 0% |
| buildernet\_nethermind\_azure\_westeurope\_08 | buildernet\_nethermind\_azure\_eastus\_07 | 5 | 6513729 | 0% |
| buildernet\_nethermind\_azure\_westeurope\_08 | buildernet\_beaver\_azure\_eastus\_07 | 5 | 6194898 | 0% |
| buildernet\_nethermind\_azure\_eastus\_07 | buildernet\_flashbots\_azure\_eastus\_10 | 5 | 10302474 | 0% |
| buildernet\_flashbots\_azure\_eastus\_10 | buildernet\_nethermind\_azure\_westeurope\_08 | 4 | 9087691 | 0% |
| buildernet\_flashbots\_azure\_westeurope\_09 | buildernet\_flashbots\_azure\_eastus\_10 | 3 | 6117425 | 0% |
| buildernet\_flashbots\_azure\_eastus\_10 | buildernet\_flashbots\_azure\_westeurope\_09 | 2 | 9115212 | 0% |
| buildernet\_flashbots\_azure\_westeurope\_09 | buildernet\_flashbots\_mkosi\_test\_1 | 1 | 7617802 | 0% |
| buildernet\_flashbots\_mkosi\_test\_1 | buildernet\_beaver\_azure\_westeurope\_08 | 1 | 18836193 | 0% |
| buildernet\_flashbots\_mkosi\_test\_1 | buildernet\_flashbots\_azure\_westeurope\_09 | 1 | 18043957 | 0% |
| buildernet\_nethermind\_azure\_westeurope\_08 | buildernet\_flashbots\_mkosi\_test\_1 | 1 | 9123905 | 0% |
| buildernet\_nethermind\_azure\_westeurope\_08 | buildernet\_beaver\_azure\_westeurope\_08 | 1 | 13247037 | 0% |
| buildernet\_flashbots\_azure\_westeurope\_09 | buildernet\_nethermind\_azure\_eastus\_07 | 1 | 6076706 | 0% |
| buildernet\_flashbots\_mkosi\_test\_1 | buildernet\_nethermind\_azure\_westeurope\_08 | 1 | 7608331 | 0% |
**HTTP/2 latency**
Below we can see a more complete before-after of latency between links. We can
see that after HTTP/2 we see more bounded p99 and p999, while p50 and p90 stayed
almost the same. While picking a single day from both deployments may not be
completely indicative, the behaviour remained quite consistent during the next
days. Latency comparison of P99 latencies befor, each with 24 hours worth of
datae and after HTTP/2 for a day's worth of data (all numbers in milliseconds):
| src | dst | p50\_a | p50\_b | Δp50 | p99\_a | p99\_b | Δp99 |
| ---------------------- | --------------------- | ------ | ------ | ---- | ------ | ------ | ---- |
| nethermind\_we\_08 | beaver\_eastus\_07 | 42 | 42 | \~0 | 458 | 155 | -303 |
| nethermind\_we\_08 | flashbots\_eastus\_10 | 42 | 41 | -1 | 455 | 142 | -313 |
| beaver\_eastus\_07 | nethermind\_we\_08 | 44 | 41 | -3 | 357 | 125 | -232 |
| beaver\_eastus\_07 | flashbots\_we\_09 | 45 | 46 | +1 | 297 | 122 | -175 |
| nethermind\_eastus\_07 | flashbots\_we\_09 | 48 | 45 | -3 | 299 | 120 | -179 |
| flashbots\_eastus\_10 | nethermind\_we\_08 | 42 | 43 | +1 | 206 | 119 | -87 |
| flashbots\_eastus\_10 | flashbots\_we\_09 | 43 | 44 | +1 | 206 | 106 | -100 |
| flashbots\_eastus\_10 | beaver\_eastus\_07 | 43 | 44 | +1 | 284 | 131 | -153 |
| nethermind\_eastus\_07 | beaver\_eastus\_07 | 39 | 41 | +2 | 200 | 96 | -104 |
| beaver\_eastus\_07 | flashbots\_eastus\_10 | 40 | 41 | +1 | 197 | 103 | -94 |
### TCP + `bitcode`
To preserve backwards compatibility, we offered a new TCP-only version of system
API along with the existing HTTP/2 one. Using a new temporary endpoint allowed
us to experiment with binary encoding format, so we could finally remove JSON
which other than offering high decoding latency, resulted in \~50% bigger
payloads than necessary due to encoding hex data as strings.
First, we tried to look into `serde` compatible binary encoding formats, like
`bincode` and MessagePack, but both of them do not have full feature
compatibility with `serde_derive`. Because of that, we opted out from `serde`
compatibility and went with
[`bitcode`](https://github.com/SoftbearStudios/bitcode) instead, which currently
sits at the top of the
[Rust Serialization Benchmark](https://david.kolo.ski/rust_serialization_benchmark/)
leaderboard.
This was the result of both a switch to TCP (without any additional tuning yet),
and migration from JSON to bitcode:
However, there’s a big caveat here: we only saw these levels of improvements on
very specific links. The plot above depicts latency from a specifically
broadcast-heavy instance: one that ingests and forwards a lot of orders. We
barely saw any improvements on other instances, which led us down the rabbit
hole of TCP kernel parameters.
### Kernel Parameter Tuning
In order to get the most of raw TCP sockets, it has been fundamental to tune
kernel parameters to achieve optimised connections. To understand better the
changes we’ve done, let’s brush up on some preliminaries.
The *bandwidth-delay product*, or BDP, is a property of a network path, and is
computed as the product of bandwidth, expressed in bytes per second, and
round-trip time (delay), expressed in seconds. In the context of TCP, it
represents how much in-flight data the connection can hold before the sender
must wait for acknowledgments from the receiver.
Let’s make a concrete example: a node in FlowProxy has a upload bandwidth of 1
Gbps, and an Azure link between East US and West Europe is around 85ms
([source](https://learn.microsoft.com/en-us/azure/networking/azure-network-latency?tabs=Americas%2CEastUS)).
This results in a BDP of `1 Gbps × 85 ms = 85 Mbit = 10.625 MB`, meaning that is
the theoretical maximum of data we can have unacknowledged in TCP, assuming
ideal network conditions.
Related to BDP is the *congestion window* (referred as `cwnd` in code). It is a
core TCP mechanism that controls how much data a sender is allowed to have “in
flight” (sent, but not yet acknowledged) at any given time. On the receiver
side, there’s `rwnd` (the receive window), which determines how much
unacknowledged data can accumulate at the receiver side before packets are
dropped. The minimum of `cwnd` and `rwnd` determines your maximum throughput.
One direct consequence of BDP is the following: if
`min(cwnd, rwnd) < size(message)`, that message will need an **additional round
trip to fully transmit!**
By definition, the theoretical maximum size of the congestion window for
communicating over a link matches the BDP. That would mean there is no
congestion at all! However, the network might not be always stable, and packet
loss might happen and RTT may vary over time. As such, the congestion window
dynamically adapts to the appropriate amount of unacknowledged data to not waste
any resources. These mechanisms are called *TCP congestion control algorithms.*
All of these parameters can be modified in the Linux kernel with `sysctl`. The
following are the most important:
* `net.ipv4.tcp_congestion_control`: the default (`cubic`) worked for us.
* `net.ipv4.tcp_window_scaling`: ensure this is turned on.
* `net.ipv4.tcp_rmem`: sets the bounds for `rwnd`. Default and max had to be
significantly increased, some multiple of your expected max message size is a
good guideline. Linux will take care of autoscaling this if window scaling is
turned on.
Other than congestion algorithms, there are other settings that may impact the
congestion window. One of them is called “slow start after idle”, which `sysctl`
setting is `net.ipv4.tcp_slow_start_after_idle` . A connection is considered
idle if a packet hasn’t been sent for a certain amount of time, computed as a
function of the RTT, but with a default minimum of 200ms that we would be used
in case of a 85ms RTT. In case of idleness, the congestion window is greatly
reduced, diminishing throughput and increasing latency.
Given connection between FlowProxy instances are long lived, and can be bursty
in period of high-traffic, this setting should be disabled. In the picture below
you can see the tremendous impact of the switch on low volume instances:
Call duration latency (p99) before and after the
`net.ipv4.tcp_slow_start_after_idle` has been disabled.
This setting was exactly why the high message volume instances displayed really
good latency after the switch to TCP, and the others didn’t: their TCP
connections were never idle, so the congestion windows were never reset.
The general takeaway here is that Linux TCP settings are very conservative and
always try to save on resources, and should almost always be changed depending
on your workload.
### Latency Comparison
After all these modifications, we were finally approaching optimality in terms
of order propagation: variance for big and small messages, over long and short
links was reduced significantly. Check out the full comparison of P99 latencies
below, each with 24 hours worth of data:
**Before**
Europe (high latency routes)
{
from: 'beaver_eastus',
to: 'nethermind_we',
metrics: { latency: 1947.05 },
waypoints: [
[-42, 47],
[-18, 49],
],
},
{
from: 'beaver_eastus',
to: 'flashbots_we',
metrics: { latency: 1789.05 },
waypoints: [
[-40, 48],
[-15, 50],
],
},
{
from: 'beaver_eastus',
to: 'beaver_we',
metrics: { latency: 1521.6 },
waypoints: [
[-42, 44],
[-18, 46],
],
},
// US East local
{
from: 'beaver_eastus',
to: 'nethermind_eastus',
metrics: { latency: 292.45 },
},
{
from: 'nethermind_eastus',
to: 'flashbots_eastus',
metrics: { latency: 58.65 },
},
{
from: 'nethermind_eastus',
to: 'beaver_eastus',
metrics: { latency: 49.63 },
},
{
from: 'flashbots_eastus',
to: 'beaver_eastus',
metrics: { latency: 56.19 },
},
{
from: 'flashbots_eastus',
to: 'nethermind_eastus',
metrics: { latency: 26.18 },
},
// Transatlantic: Nethermind EUS -> Europe
{
from: 'nethermind_eastus',
to: 'beaver_we',
metrics: { latency: 194.66 },
waypoints: [
[-35, 43],
[-10, 45],
],
},
{
from: 'nethermind_eastus',
to: 'nethermind_we',
metrics: { latency: 193.11 },
waypoints: [
[-32, 50],
[-8, 52],
],
},
{
from: 'nethermind_eastus',
to: 'flashbots_we',
metrics: { latency: 192.13 },
waypoints: [
[-38, 46],
[-12, 48],
],
},
// Transatlantic: Flashbots EUS -> Europe
{
from: 'flashbots_eastus',
to: 'beaver_we',
metrics: { latency: 201.54 },
waypoints: [
[-30, 41],
[-5, 44],
],
},
{
from: 'flashbots_eastus',
to: 'nethermind_we',
metrics: { latency: 200.27 },
waypoints: [
[-25, 47],
[0, 50],
],
},
{
from: 'flashbots_eastus',
to: 'flashbots_we',
metrics: { latency: 199.81 },
waypoints: [
[-28, 44],
[-3, 47],
],
},
]}
metricKey="latency"
metricRange={[0, 2000]}
height={350}
center={[-35, 45]}
scale={350}
showLegend={true}
/>
The maximum P99 here is ~1950ms.
**After**
Europe
{
from: 'beaver_eastus',
to: 'nethermind_we',
metrics: { latency: 50.06 },
waypoints: [
[-42, 47],
[-18, 49],
],
},
{
from: 'beaver_eastus',
to: 'flashbots_we',
metrics: { latency: 43.51 },
waypoints: [
[-40, 48],
[-15, 50],
],
},
{
from: 'beaver_eastus',
to: 'beaver_we',
metrics: { latency: 43.82 },
waypoints: [
[-42, 44],
[-18, 46],
],
},
// US East local
{
from: 'beaver_eastus',
to: 'nethermind_eastus',
metrics: { latency: 7.55 },
},
{
from: 'nethermind_eastus',
to: 'flashbots_eastus',
metrics: { latency: 9.0 },
},
{
from: 'nethermind_eastus',
to: 'beaver_eastus',
metrics: { latency: 7.55 },
},
{
from: 'flashbots_eastus',
to: 'beaver_eastus',
metrics: { latency: 4.4 },
},
{
from: 'flashbots_eastus',
to: 'nethermind_eastus',
metrics: { latency: 2.25 },
},
// Transatlantic: Nethermind EUS -> Europe
{
from: 'nethermind_eastus',
to: 'beaver_we',
metrics: { latency: 47.88 },
waypoints: [
[-35, 43],
[-10, 45],
],
},
{
from: 'nethermind_eastus',
to: 'nethermind_we',
metrics: { latency: 46.47 },
waypoints: [
[-32, 50],
[-8, 52],
],
},
{
from: 'nethermind_eastus',
to: 'flashbots_we',
metrics: { latency: 48.87 },
waypoints: [
[-38, 46],
[-12, 48],
],
},
// Transatlantic: Flashbots EUS -> Europe
{
from: 'flashbots_eastus',
to: 'beaver_we',
metrics: { latency: 42.32 },
waypoints: [
[-30, 41],
[-5, 44],
],
},
{
from: 'flashbots_eastus',
to: 'nethermind_we',
metrics: { latency: 45.89 },
waypoints: [
[-25, 47],
[0, 50],
],
},
{
from: 'flashbots_eastus',
to: 'flashbots_we',
metrics: { latency: 44.28 },
waypoints: [
[-28, 44],
[-3, 47],
],
},
]}
metricKey="latency"
metricRange={[0, 2000]}
height={350}
center={[-35, 45]}
scale={350}
showLegend={true}
/>
The maximum P99 here is ~50ms.
There was one more change that we knew would be impactful, not on latency but on
CPU usage.
### mTLS
The last improvement we’ve been working on during this collaboration has been
[mutual TLS](https://www.cloudflare.com/learning/access-management/what-is-mutual-tls/)
(mTLS). If required by the server, the client must present a valid X509 TLS
certificate to authenticate itself. This works, because all instances know each
other's TLS certificates through a central service registry.
Currently, FlowProxy authenticates other instances on a message-by-message
basis: every message it gets needs to be signed by a known ECDSA public key.
This is secure, but wasteful: every message needs to be a) signed by the sender,
and b) verified by the receiver. mTLS would change authentication on a message
basis, to authentication on a connection basis. Authentication would occur
during connection setup, and after a successful authentication, all messages
sent on that connection are transitively authenticated too.
As of December 22, 2025, this feature hasn’t been deployed yet on a production
environment, but from the picture below we can already appreciate its effects on
the staging environment.
Global CPU usage and thread usage after deployment of mTLS on a staging
environment.
From this preliminary result we can see a 50% reduction of CPU and thread usage,
lowering total CPU usage of FlowProxy from \~10% to 5%. While this may seem like
a minor improvement, freeing up CPU to be used by other critical services such
as rbuilder can make the difference between winning and losing a MEV-Boost
auction.
## Next Steps
At the moment we’re pretty satisfied with the performance of FlowProxy, but
there is still room for improvement. In particular, we haven’t yet worked on the
communication between a FlowProxy instance and its local builder, which still
uses JSON-RPC over HTTP, although via localhost. Assuming these two services
keep running on the same machine, we could implement a shared memory based
transport. While this would be clearly a benefit over the status quo, we’re
coming closer to a point of diminishing returns compared to other improvements.
As such, next steps could move away from FlowProxy and concentrate over a
different part of the stack of BuilderNet and its architecture.
# Glamsterdam Headliners
The following document contains Chainbound’s take on what the headliners
should be on the upcoming Glamsterdam hard-fork of Ethereum, according to the
format specified in this [ethereum-magicians
post](https://ethereum-magicians.org/t/soliciting-stakeholder-feedback-on-glamsterdam-headliners/24885).
## Priorities
**Question: What do you view as the top priority theme in this fork & why?**
*e.g. censorship resistance, scaling the L1, improving UX, etc.*
We see scaling the L1 in terms of throughput and lowering fees as the main theme
for this fork. If done properly, without introducing new negative externalities
and by lowering down the technical debt of the protocol, it has downstream
effects in both the rollup roadmap and for improving UX for a wide variety of
users. This process makes Ethereum as a global platform more attractive for
everyone, fostering its growth in multiple directions.
We see also censorship resistance as an equally important part of the roadmap,
however we’re not completely satisfied with current proposals, as motived below.
## Headliners
**Question: Which EIP(s) do you favor as a headliner for Glamsterdam?**
The process is aiming for one EIP each for the consensus and execution layers.
Execution Layer: EIP-7928 — Block Level Access list. This is clearly a
low-hanging fruit improvement that is both future-proof (e.g. non-contentious
with EVM upgrades) and with basically no tradeoffs. It benefits a wide variety
of users: both users of L1 dApps because of cheaper usage of read/writes on
storage, and users on L2s, as carefully explained by @donnoh.
Consensus Layer: EIP-7732 — enshrined Proposer-Bulider Separation. This EIP
creates a framework to better spreads tasks and resources over the slot,
favoring both home-stakers and future improvements like APS and ETs. We
basically agree with what’s mentioned in this
[post](https://ethereum-magicians.org/t/eip-7732-the-case-for-inclusion-in-glamsterdam/24306#p-59218-detailed-justification-3).
We acknowledge the following open questions of the proposal:
* The existence of the
“[Free Option Problem](https://collective.flashbots.net/t/the-free-option-problem-in-epbs/5115)”
— Thanks to some comments (see
[here](https://collective.flashbots.net/t/the-free-option-problem-in-epbs/5115/2)
and [here](https://x.com/hasufl/status/1949093915950186547)), we consider it
more an independent problem of the already existing PBS market structure. In
our opinion, this EIP more concerned with efficient slot restructuring rather
than enshrining a specific market structure. For example, by requiring a
validator signature over the transaction list in the execution header, we
could keep the pipelining benefits while killing the PBS market structure.
* DA attestation deadline — The current proposal states that the payload and the
blobs must be observed by the Payload-Timeliness-Committee at the same time.
As such, moving the PTC deadline to favour execution time would result in less
blob propagation time. To contrast this, we’re fully supportive of the
[Dual-deadline PTC vote](https://notes.ethereum.org/@anderselowsson/Dual-deadlinePTCvote)
proposal.
Lastly, we mark EIP-7805 (FOCIL) as the second best candidate for the
Glamsterdam CL headliner. However, we have minor questions regarding its
implementation, expressed in the “concern” section.
## Concerns
**Question: The leading headliners among client teams are described in
a [series of blog posts listed here](https://forkcast.org/upgrade/glamsterdam#client-team-perspectives).
Do you have any concerns about any particular proposal?**
As mentioned above, we consider FOCIL as the 2nd best option for the CL
headliner in Glamsterdam. However, we have some minor concerns about its
implementation:
1. It doesn’t support blob transactions, so it means support for it must come in
another hard-fork that might change how blob transactions are propagated.
While some proposals do exist (for example, our own
[blob notaries design](https://ethresear.ch/t/blob-notaries-a-distributed-blob-publishing-design-to-scale-da/22709)),
there isn’t a concrete roadmap to make it happen as far as we know. With
Ethereum doubling down in increasing its blob count, supporting them becomes
more and more relevant.
2. For rollups, if censored, settling becomes much harder due to the 8KiB as max
IL size. Moreover, recent updates such as
[EIP-7623](https://eips.ethereum.org/EIPS/eip-7623) make using large amount
of calldata more expensive than before.
3. The current proposal of FOCIL states that
[the inclusion list building logic is up to the consensus clients implementation](https://eips.ethereum.org/EIPS/eip-7805#il-building).
In our opinion, assessing which transactions are being censored is a very
hard task, unless explicitly part of the OFAC list. This would lead to a risk
of wasting a small yet extremely precious resource. Examples include:
1. Picking at random from the mempool can lead to not choosing a censored
transaction for some slots in a row, degrading eventual censorship
resistance UX.
2. Picking the highest paying transactions can lead to IL members picking the
same small subset of mempool transactions that fit in a 8KiB IL.
3. Picking transactions that have been in the mempool for a while can
incentivise non-censored users flooding with low-priority transactions
(e.g. 1 wei priority fee).
While we acknowledge the difficulty of this problem, we see more suited a
design where the user explicitly ask for its transaction to be force-included
in the next blocks in exchange for a tip. A tip is justified to avoid spam
and because the service done by the protocol is converting from probabilistic
inclusion to deterministic inclusion.
# Introducing FlowProxy
**TLDR**; [Chainbound](https://chainbound.io) collaborated with Flashbots to
deprecate its
[original orderflow proxy](https://github.com/flashbots/buildernet-orderflow-proxy)
implementation in Go in favor of a new one built in Rust, created with the goal
of reducing E2E latency and increasing observability in
[BuilderNet](https://buildernet.org/). We’re calling it
[FlowProxy](https://github.com/BuilderNet/FlowProxy). Over a 1-month period, we
picked up the work on the new Rust implementation to get it production ready.
Along the way, we carefully profiled and benchmarked any changes to ensure they
met BuilderNet performance requirements. We write about this journey below.
FlowProxy has been live on all BuilderNet instances since October 17th.
## Background
The [orderflow proxy](https://github.com/flashbots/buildernet-orderflow-proxy)
is a central component in BuilderNet. It’s responsible for receiving bundles and
transactions from users, and ensuring those orders are shared with all other
instances in BuilderNet, as well as to the local
[rbuilder](https://github.com/flashbots/rbuilder). Performance is an important
aspect here: the faster these proxies can share orders, the more effective the
builders will be able to collaborate, and the more blocks BuilderNet as a whole
will win. If a proxy goes down for even a couple of seconds, that could mean
1000s of lost orders, so reliability is crucial too.
Simplified diagram of BuilderNet topology
For all of these reasons, it was decided that the successor to the legacy proxy
should be written in Rust. We were tasked with taking over the initial work and
making it production ready. One of our goals was to ensure this new version
would outperform v1, without any drastic changes yet. In this post, we’ll go
over our methodology, findings and challenges.
## Methodology
The first goal of this project was to create a realistic testbed for integration
tests, e2e tests, compatibility tests, profiling CPU and measuring network
throughput and latencies. These last 3 measurements would help us assess, quite
quickly, if we’re meeting the performance requirements as we continued the
development.
When talking about performance in this context, we’re primarily talking about
various types of latencies or delays. Examples of manifestations that are
relevant here include **processing latencies**, where the CPU is busy doing some
computations (in the critical path), and **network latencies** between instances
determined by their physical distance and link capacities.
We can define the critical path as follows: the “wire-to-wire” latency between
the proxy receiving an order, and the order leaving the program to be sent to
the next destination. This primarily includes a bunch of order validation
routines and some cryptographic operations such as signature verification.
### Stage 1: Network simulation & profiling
To replicate a realistic testbed we turned to
[Shadow](https://shadow.github.io/docs/guide/), a discrete-event network
simulator that directly executes your binaries as-is. It allows you to specify
network topologies in
[scenario files](https://github.com/flashbots/buildernet-orderflow-proxy-v2/blob/main/simulation/scenarios/network-graph-2-zones.yaml)
that will result in modified traffic patterns to follow these configured
bandwidths and latencies. Shadow directly intercepts syscalls coming from your
binary, and in the case of intercepted network packets, will route them over its
simulated, deterministic networking stack.
We created a scenario that mirrored a real-world BuilderNet deployment, and then
replayed live traffic captured previously at various rates. We then captured
packets, logged the timestamps for order receipts at all the proxies, and
profiled the application with perf and flamegraph. With some additional tooling,
we could very quickly iterate on changes and see the direct impact in the form
of outputs like (printout of `./sim.sh process`):
```
Processing results/bundle_receipts_2025-10-03-08-57-50_runtime-5m_scale-5.parquet, results/proxy1_eth0_2025-10-03-08-57-50_runtime-5m_scale-5_summary.csv and results/proxy2_eth0_2025-10-03-08-57-50_runtime-5m_scale-5_summary.csv...
Number of rows:
┌─count(bundle_hash)─┐
1. │ 177985 │
└────────────────────┘
Aggregated statistics:
┌─────────────avg_us─┬─p50_us─┬─p90_us─┬─p99_us─┬─p999_us─┬─min_us─┬─max_us─┬────────────corr_tp─┬─────────avg_size─┬─p50_size─┬─p90_size─┬─p99_size─┐
1. │ 44447.021855774365 │ 44004 │ 44008 │ 48994 │ 132006 │ 44003 │ 220050 │ 0.1461330024788131 │ 6894.85476304183 │ 2966 │ 19376 │ 40390 │
└────────────────────┴────────┴────────┴────────┴─────────┴────────┴────────┴────────────────────┴──────────────────┴──────────┴──────────┴──────────┘
Bandwidth usage data:
┌─host─────┬─upload_total_MB─┬────upload_avg_Mbps─┬─upload_peak_Mbps─┬─download_total_MB─┬──download_avg_Mbps─┬─download_peak_Mbps─┐
1. │ 10.0.0.3 │ 2692.634864 │ 104.56834423300971 │ 930.272048 │ 127.1639 │ 4.938403883495146 │ 23.797824 │
2. │ 10.0.0.4 │ 127.1639 │ 4.938403883495146 │ 23.797824 │ 2692.634864 │ 104.56834423300971 │ 930.272048 │
└──────────┴─────────────────┴────────────────────┴──────────────────┴───────────────────┴────────────────────┴────────────────────┘
```
Or flamegraphs like:
We won’t go into all the information that is displayed here, but it allowed us
to identify processing bottlenecks, bandwidth usage, and network latencies. It
also offers a robust e2e testing environment that can be very realistic, and it
helped us uncover a lot of bugs and implementation differences with the v1
proxy. However, there are some important caveats to Shadow that we’ve outlined
in a section below.
### Stage 2: Live testing
Once we got to feature parity with the v1 proxy, and understood the initial CPU
profiles and networking results, it was decided to move to live testing.
Simulation can only get you so far. Because of some Shadow limitations that we
discuss below, and the fact that real orderflow proxies run inside of
[TDX](https://buildernet.org/docs/orderflow-sharing-confidentiality), realistic
measurements would need to be obtained in live environments.
Two existing BuilderNet instances were upgraded to use FlowProxy. The rest of
the analysis was all done using real metrics from the live deployment.
## Findings
Below we’ll talk about some of the initial findings encountered. Note that
performance was not the main goal of this project, but there were some low
hanging fruits that we’ll talk about below.
### Signature Verification
In stage 1, looking at some profiling results in the form of flamegraphs, it was
immediately obvious that ECDSA signature verification was claiming the bulk of
CPU cycles. This was true for both ingress proxies (who directly receive orders
from users), and receiving proxies (who get forwarded these orders by the
ingress proxies). In addition, ingress proxies that had to forward data also
spent quite some time on hashing and signing messages.
Signature verification happens in roughly 3 places:
1. Verifying the `X-Flashbots-Signature` header when receiving orders from users
(on the user endpoint);
2. Verifying the signature on the transactions present in bundles as part of the
bundle validation process → Greatly amortised with a cache, see below;
3. Verifying the `X-Flashbots-Signature` header when receiving orders from other
proxies (on the system endpoint).
CPU profile on an ingress-only proxy: blue box is `X-Flashbots-Signature`
header verification, purple box is transaction signature verification.
Together they account for roughly 40% of CPU time spent.
Verifying the signature headers on user requests is absolutely necessary to
authenticate and score users, but could we decrease the other types of
verification? Turns out, when looking at real data, that most bundles contain
duplicate transactions (up to 90% in some deployed instances). As such, we can
store the result of transaction signature recovery (the signer) in a small cache
that maps transaction hashes to signer addresses, and do a very cheap lookup on
any incoming transactions to reuse the verification result. By doing this **we
have observed instances on Mainnet showing a transaction signature recovery
cache hit ratio of up to 70%**, meaning that *time spent validating transaction
signatures in bundles has been decreased by the same amount*.
Proxy v2 running with transaction signature cache. For this instance we have
that 68% of the requests received contain already processed transactions for
which we can skip signature recovery. TTL applied is 36s.
Verifying system signatures (point 3 above) is another issue that we haven’t
addressed yet. We’ll revisit this in the next steps section below.
### Freebies by Rust
Some metrics improved quite drastically by just switching to Rust. We’ll focus
on some of the network-related latencies in the graphs below:
**Cross-region RPC**
These 2 graphs show the P99 of cross-regional RPC call durations for the v1 and
v2 proxies. The first pair is small requests, and the second pair is for big
requests (> 50KB).
Proxy v1, small requests (< 50KB). Range over 2 days:
**137ms**
**→ 3.6sec**
Proxy v2, small requests (< 50KB). Range over 2 days:
**85ms → 2sec**
Proxy v1, big requests (> 50KB). Range over 2 days:
**260ms → 2.8sec**
Proxy v2, big requests (> 50KB). Range over 2 days:
**85ms → 2sec**
## Lessons Learned
### Shadow & Simulation
Since Shadow is a discrete-event *simulator*, it has a notion of time that is
purely tied to network events. If no network events are sent, time (from the
perspective of your app, since Shadow intercepts timings syscalls as well) does
not progress. That means you can’t use Shadow to accurately measure *processing
latencies*, since Shadow won’t progress the time your app sees on compute only.
Additionally, micro-benchmarking a single connection by, for example, comparing
different transport protocols such as HTTP or gRPC, will also not give you the
most accurate results. This is because the simulated networking stack is also
completely written from scratch to be deterministic, and the way TCP is
implemented is not exactly the same as it would be in the Linux kernel, for
example. One very real example here is TCP send & receive buffer sizes, or
congestion control protocols, both of which can have major impacts. In Shadow,
these are not as configurable as in the Linux kernel.
If we were to do this again, we would most likely use something like
[`tc netem`](https://www.man7.org/linux/man-pages/man8/tc-netem.8.html) on Linux
to *emulate* networking (instead of *simulating*), which reuses the kernel’s
networking stack. In fact, the ease of setting up and running scenarios with
Shadow, combined with the realistic emulation of `tc netem` would be a great
project if anyone is interested.
### **HTTP Connection Pools**
The v1 proxies use JSON-RPC over HTTP to forward orders to each other, which is
something v2 also does for backwards compatibility reasons. When initially
running the Go proxy in Shadow, we noticed that we couldn’t saturate even 1% of
the capacity of the simulated link (1 Gbps). After looking into it a bit
further, the culprit was a disabled HTTP connection pool, which resulted in the
proxy only using 1 underlying TCP connection to send data over. Since HTTP 1.1
(which is what’s currently used) does not pipeline or multiplex, it supports
only 1 inflight request per TCP connection! This obviously throttles throughput
enormously, especially on long RTT links (which was the case at 88ms RTT). After
enabling a connection pool of 300 connections, we were able to almost saturate
the bandwidth of the link, and the issue was solved.
This was something we kept in mind for the Rust proxy, where the HTTP client
library we used was [reqwest](https://github.com/seanmonstar/reqwest). Reqwest,
by default, does not allow a lot of granular control over underlying connection
pools, and in fact will open 1000s of connections if it needs to, exhausting the
system of underlying ephemeral ports. This happened occasionally when incoming
requests (and thus forwarding requests) spiked, and we had to introduce
functionality to put a hard cap on the amount of available connections.
However, it was still noticed that there were a *lot* of connection attempts on
the server side of any proxy, many more than before. Additionally, the number of
timeouts increased drastically, which would result in lost orders if not dealt
with.
Connection attempts from other proxies on the server side. Spot the upgrade.
Connection attempts from other proxies on the server side. Spot the upgrade.
After some digging, this turned out to be an issue with the connection pool, and
specifically, how connections are reused. Connection pool strategies determine
which idle connection is picked up to handle a new request, and for practical
purposes can be reduced to either FIFO (oldest idle connections are reused
first), or LIFO (most recent connections are reused first). We knew the Go HTTP
library used was FIFO. Reqwest, on the other hand (and the `hyper` connection
pool under the hood), uses LIFO.
To recap, this means that new requests are routed to connections that are still
“warm”, i.e. have recently been used. In theory, this is fine, and might even
result in better performance in some cases due to better CPU cache utilization.
*However*, in a bursty environment, with a slightly misconfigured client, this
can result in a lot of cascading failures. With LIFO, a parameter that becomes
very important is the **idle timeout**, an option that can be set on both HTTP
server and client. If the server observes a connection that has been idle for
more than `X` seconds (10, in our case), it will close the HTTP session. The
problem is that if the client doesn’t close the connection at the same time or
earlier, it can only *know* the session is closed by sending a request to the
server!
Imagine the following scenario:
1. OF proxy is in a state where it's happily routing requests over N \<\< 512
(the limit) TCP connections. These are the "warm" connections that
continually get re-used.
2. A burst of requests come in, and the HTTP client has to start re-using some
of the "cold" connections that were opened previously, because current
connections are all unavailable.
3. *However*, because these have become stale (i.e. closed by the server’s
10-second idle timeout), our client quickly bursts requests over all of these
stale connections, that in a cascading way get closed for new connections to
open, which takes time and may result in timeouts and connection bursts.
This issue was fixed by increasing the idle timeout on the server to 30s, and
ensuring every HTTP client is configured with an idle timeout slightly less than
that. The results can be seen on this server panel:
In addition, the HTTP timeouts for long RTT links also decreased since then:
As well as p99 round-trip latencies on those links:
Y-axis is logarithmic, p99 dropped from ~125ms to ~95ms.
The p99 improvement can be attributed to the decrease in these cascading
connection failures, which always resulted in a higher RPC latency (since
connections needed to be tried, failed, and re-established).
## Next Steps
The main goal of this project was to get FlowProxy production ready and
deployed. We believe there are still a lot of improvements that can be made to
reduce both processing and networking latencies, but this will require extensive
profiling and benchmarking in live environments, where we have the realistic
overhead of TDX and data center network links. Two potential high-impact
improvements come to mind:
### mTLS
In the flamegraphs above, we talked about a lot of time being spent doing
authentication on the system API, which is used for intra-BuilderNet
communication, through signature generation *and* verification. This is needed
because the system API is a public endpoint, which requires authentication. Note
that HTTP clients inside of the proxies already authenticate servers (other
proxies) through publicly known TLS certificates. If we could use that same
mechanism to authenticate clients to servers too, we wouldn’t need
signature-based authentication at all on the system API!
Mutual TLS (mTLS) is a standard that solves exactly this. It allows the client
to attach a certificate to their connection requests, which the server can then
authenticate to allow for mutual authentication. Once authenticated, anything
sent over this connection is authenticated by default, which means we can get
rid of the signing and signature verification. Since system API authentication
takes up a large chunk of CPU time in the proxy, this would drastically improve
efficiency and tail latencies.
### Replace or Upgrade HTTP
As we saw above, HTTP can cause a lot of headaches, especially in high-load
environments like this. The fact that HTTP1.1 does not allow pipelining makes it
extremely wasteful with connections, so the first step would be changing to
something that allows pipelining / multiplexing. HTTP2 / HTTP3 are options here.
However, we think that messaging libraries that are even more lightweight, and
tuneable for different connection types (long links, short links, high
bandwidth) are more suitable (request multiplexing is still a requirement).
[Zeromq](https://github.com/zeromq/zmq.rs),
[Nanomsg-nng](https://github.com/nanomsg/nng), or our own
[msg-rs](https://github.com/chainbound/msg-rs). Switching to a streaming
communication pattern (instead of request / reply in HTTP) seems more suitable
here.
***
On a final note, we want to thank Flashbots engineers for helping us debug and
monitor live deployments. We hope to report back soon with the results of
upcoming improvements.
# Launching Taikoscope
Today, we are launching Taikoscope, a monitoring and analytics platform for
Taiko Alethia, the first based rollup built on Ethereum.
## Why Taikoscope?
Sequencers are currently whitelisted, and Taiko can only partially trust the
operators to meet network standards, so independent monitoring is essential.
Without visibility, users, the sequencer teams, or Taiko cannot verify
performance, diagnose incidents, or quantify economic risk. Taikoscope is a
unified dashboard that turns L1 and L2 data into objective alerts and signals
across network resilience, security, and value flows in the Taiko ecosystem.
Taikoscope gathers live data from Ethereum and Taiko. This data is processed,
aggregated, and presented in intuitive dashboards. With Taikoscope, users can:
* Monitor network uptime
* Compare sequencer performance
* Analyze sequencers economics
Taikoscope is part of our work on based rollups with two goals: improving
observability and analytics, and enhancing usability through better UX and
expanded features.
## What Does Taikoscope Track?
### Uptime Monitoring
Uptime monitoring tracks critical operational metrics to ensure the network
remains responsive and reliable. Incidents are created when metrics breach
thresholds and shown at [status.taiko.xyz](http://status.taiko.xyz/)
**Key Metrics:**
* **Transaction sequencing:** L2 blocks are produced regularly
* **Batch submissions:** L2 blocks are batched and posted as blobs
* **Proof submission:** Proofs are generated
* **Proof verification:** Proofs are verified
* **Public API uptime:** Availability of the official public RPC
**Why It Matters:** Monitoring uptime metrics ensures network reliability,
enables rapid incident response, and maintains community trust through
transparent, measurable performance.
### Sequencer Economics
Economic metrics analyze revenue and costs for batch submissions and sequencer
operations.
**Key Metrics:**
* **Revenue from base and priority fee:** Income from user transactions on Taiko
* **L1 posting costs:** Expenses incurred when posting data to Ethereum
* **Network net profit/loss:** Economic viability of sequencers
* **Sequencer profit ranking:** Detailed information and comparison of sequencer
profits
**Why it matters:** Monitoring sequencer economics informs strategic
decision-making, helps maintain profitability, and ensures long-term
sustainability.
### Sequencer Performance
Performance metrics offer detailed, comparative insights into the efficiency and
reliability of sequencers across the network.
**Key Metrics:**
* **Transactions per second (TPS):** Real-time transaction processing rate
* **L2 block cadence:** How often Taiko blocks are created
* **Batch posting cadence:** How often batches of L2 blocks are posted to L1
* **Gas usage:** Efficiency in gas consumption per transaction/block
* **Tx count per L2 block:** How many transactions are included in each Taiko
block
**Why It Matters:** Analyzing sequencer performance helps identify bottlenecks,
optimize sequencer usage, and incentivize improved network efficiency.
### Network Health
**Key Metrics:**
* **Prove time:** Time it takes to prove batches posted on L1
* **Verify time:** Time it takes to verify batches posted on L1
* **L2 reorg frequency and depth:** Stability and consistency of blockchain data
* **Failed proposals:** If a sequencer creates an L2 block but is unable to
include it in a batch and another sequencer has to pick up the work
* **Slashing events:** If a sequencer fails to honor their commitments
* **Forced inclusions:** Transactions that have been forced directly on L1
**Why It Matters:** Monitoring network health is essential for maintaining
stability and community confidence. Tracking these metrics helps quickly detect
and mitigate issues, ensuring robust and trustworthy infrastructure.
## Architecture
Taikoscope subscribes to events from both L1 and L2 RPCs, and on each L1 head,
and performs the needed contract calls. Data is inserted into ClickHouse and
queried from the Dashboard via an API.
Taikoscope is developed entirely in Rust, aligning with Chainbound’s
performance-oriented engineering ethos.
🔗 **Dashboard**: [taikoscope.xyz](https://taikoscope.xyz/)
🧑💻 **Code**:
[github.com/chainbound/taikoscope](https://github.com/chainbound/taikoscope)
Taikoscope is funded through Taiko’s grant program and MIT licensed.
Contributions are welcome!
# Linkem: Building a Rust Network Emulator from Scratch
## Introduction
When we started working on [`msg-rs`](https://github.com/chainbound/msg-rs) we
ran into the necessity of emulating real networks with latency, jitter, packet
loss, and bandwidth constraints. We needed a way to test under realistic
conditions without setting up actual distributed infrastructure.
**Linkem** is what we ended up building. It's a Rust library that creates
isolated network peers using Linux namespaces and injects network impairments
between them, fully leveraging the kernel TCP/IP stack for modelling traffic.
This article documents how it works and the decisions we made along the way.
## Existing Tools and the Gap
Before building something new, we looked at what already existed. Network
emulation is a solved problem in many contexts, depending on your requirements.
**tc** is the foundation everything else builds on. The Linux kernel's traffic
control subsystem can add delay, loss, bandwidth limits, and more to any network
interface. The problem is that it operates at the system level. You run `tc`
commands to modify interfaces, and those changes affect everything using that
interface. There's no isolation between tests, no easy way to create multi-peer
topologies, and integrating it into a Rust test suite means shelling out to
external commands.
[**Mininet**](https://github.com/mininet/mininet) and its fork
[**Containernet**](https://containernet.github.io/) are Python-based network
emulators popular in SDN research. They can create complex topologies with
switches, routers, and hosts. Containernet extends this to use Docker containers
as hosts. These are powerful tools, but they require defining your topology
upfront in Python, spinning up the emulated network, and then running your tests
inside it. The workflow is "create topology first, then run code inside" — the
opposite of what we wanted for Rust integration tests.
[**Toxiproxy**](https://github.com/Shopify/toxiproxy) from Shopify takes a
different approach: it's a TCP proxy that sits between your application and its
dependencies. You configure "toxics" (latency, timeouts, bandwidth limits) via a
REST API. It's great for testing how your app handles a flaky database
connection, but it's not network emulation — it's application-layer proxying.
Your code has to connect through the proxy, which means changing connection
strings. It also can't simulate things like packet loss at the IP level or test
UDP-based protocols.
[**Estimator**](https://github.com/commonwarexyz/monorepo/tree/main/examples/estimator)
from Commonware is a great tool for testing prototypes of consensus protocols,
simulating peers distributed across specified AWS regions with real-world
latency/jitter in virtual time. It features a handy DSL to create different
scenarios and replay them deterministically. Its main downside is the limited
scope: it's designed for consensus protocols, not general-purpose network
testing.
[**Pumba**](https://github.com/alexei-led/pumba) is a chaos testing tool for
Docker containers. It can kill containers, pause them, or use `tc` to inject
network faults. But it operates on running containers from the outside — you
define your Docker Compose setup, start it, then point Pumba at containers to
disrupt. Like Containernet, the topology exists before your test code runs.
[**netsim**](https://github.com/canndrew/netsim) is the closest to what we
needed. It's a Rust library that uses Linux namespaces to isolate network stacks
and lets you run async code inside them. However, it doesn't provide the
per-destination impairment control we needed. When Peer A sends to Peer B versus
Peer C, we wanted different latency and loss characteristics — netsim's model
didn't support that out of the box.
### Using the Kernel’s TCP/IP stack
There's another issue with most of these tools: they operate at different layers
of the stack and aren't true kernel-level network emulation. This matters when
you want to test things like:
* *TCP buffer tuning* (`tcp_rmem`, `tcp_wmem`) — how do different buffer sizes
affect throughput on high-latency links?
* *Window scaling* — is your system correctly configured for leveraging TCP
window scaling on high bandwidth-delay product links?
* *Congestion control algorithms* — how does BBR compare to CUBIC under packet
loss?
You can't answer these questions with application-layer proxies. The traffic
never goes through the real kernel networking stack in a meaningful way. What we
wanted was an environment as realistic as possible, where the kernel's TCP
implementation, buffer management, and congestion control are all part of the
test.
To recap, existing tools either they require external setup (Docker, Python
scripts, CLI wrappers), they model a single degraded link rather than a
topology with per-peer-pair impairments, or they don't operate at the kernel
level. For e2e tests and benchmarks where you want real kernel networking
behavior, there wasn't an obvious solution.
## Usage
The core abstraction is simple: create a network, add peers, define impairments
between them, and run async code inside each peer's isolated namespace.
Add the library to your project:
```bash
cargo add linkem
```
And then model your network with it!
```rust
use linkem::*;
use std::net::{IpAddr, Ipv4Addr, SocketAddr};
#[tokio::main]
async fn main() -> eyre::Result<()> {
let subnet = Subnet::new(IpAddr::V4(Ipv4Addr::new(10, 100, 0, 0)), 16);
let mut network = Network::new(subnet).await?;
// Add peers, each gets its own network namespace and IP
let frankfurt = network.add_peer().await?;
let tokyo = network.add_peer().await?;
let new_york = network.add_peer().await?;
// Define realistic cross-region conditions
network.apply_impairment(
Link(frankfurt, tokyo),
LinkImpairment::new()
.latency_ms(120)
.jitter_ms(5)
.loss_percent(0.1)
).await?;
network.apply_impairment(
Link(frankfurt, new_york),
LinkImpairment::new()
.latency_ms(40)
.bandwidth_mbit(100.0)
).await?;
// Run your distributed system
network.run_in_namespace(frankfurt, || async {
// This code sees frankfurt's network stack.
// Connections to tokyo experience 120ms latency.
// Connections to new_york experience 40ms latency.
start_consensus_node(config).await
}).await?;
// ... Start other nodes as well
}
```
Impairments are directional and per-link. Frankfurt to Tokyo can have different
characteristics than Tokyo to Frankfurt. You can model asymmetric links,
regional variations, or specific failure scenarios.
### Testing Distributed Systems
You can reproduce specific scenarios from production by replaying with
comparable request volumes and network conditions, then iterate quickly on
improvements. You can also verify message delivery guarantees before deploying
to a live environment:
* *At-least-once delivery*: ensure consumers are idempotent and can safely
handle duplicates.
* *At-most-once delivery*: confirm messages are not retried unexpectedly under
transient failures.
* *Ordering guarantees*: observe how your system behaves when messages arrive
late or out of order.
* *Eventual consistency*: verify convergence after partitions heal or delayed
messages are delivered.
### Chaos Testing
Impairments are dynamic, as you can change them at runtime without recreating
the network:
```rust
// Start with good conditions
network.apply_impairment(Link(a, b), LinkImpairment::default()).await?;
// Degrade the link mid-test
tokio::time::sleep(Duration::from_secs(10)).await;
network.apply_impairment(
Link(a, b),
LinkImpairment::new().latency_ms(500).loss_percent(10.0)
).await?;
// Restore it
tokio::time::sleep(Duration::from_secs(10)).await;
network.apply_impairment(Link(a, b), LinkImpairment::default()).await?;
```
This lets you test how your system responds to transient network issues,
degradation during operation, or recovery from failures.
### Case Study: Bandwidth-Delay Product and TCP Tuning
One example that shows why kernel-level emulation matters: testing TCP
throughput on high-latency links.
TCP throughput is fundamentally limited by the
[bandwidth-delay product](https://en.wikipedia.org/wiki/Bandwidth-delay_product)
(BDP), which we’ve grappled with in a
[previous post](https://engineering.chainbound.io/flowproxy-approaching-optimality#kernel-parameter-tuning).
On a 10 Mbit/s link with 40ms RTT, the BDP is about 50 KB. If TCP's receive
window is smaller than the BDP, you can't fill the pipe — you'll get less
throughput than the link can handle.
The advertised default receive window starts small and grows dynamically via TCP
receive buffer autotuning, bounded by `tcp_rmem` and `rmem_max`, and may exceed
64 KB only if window scaling is enabled. This is controlled by kernel
parameters: `tcp_window_scaling` enables the feature, and `tcp_rmem` sets the
buffer sizes.
With `linkem`, we can actually play around with these settings. Set up a 10
Mbit/s link with 40ms RTT, then:
1. **Disable window scaling, use 64 KB max buffer**: Transfer throughput is
limited — TCP can't keep enough data in flight to fill the pipe.
```rust
network
.run_in_namespace(receiver, |_| {
Box::pin(async {
// Disable window scaling
std::fs::write("/proc/sys/net/ipv4/tcp_window_scaling", "0").unwrap();
// max 64KB
std::fs::write("/proc/sys/net/ipv4/tcp_rmem", "4096 16384 65535").unwrap();
})
})
.await?
.await?;
```
2. **Enable window scaling, use 4 MB max buffer**: Throughput jumps
significantly, approaching the link's capacity.
```rust
network
.run_in_namespace(receiver, |_| {
Box::pin(async {
// Enable window scaling
std::fs::write("/proc/sys/net/ipv4/tcp_window_scaling", "1").unwrap();
// max 4MB
std::fs::write("/proc/sys/net/ipv4/tcp_rmem", "4096 262144 4194304").unwrap();
})
})
.await?
.await?;
```
[In this example](https://github.com/chainbound/msg-rs/blob/main/linkem/examples/bdp_throughput.rs),
you can see the difference in measured throughput. And the test runs against the
real Linux TCP stack, with real kernel buffer management. You can tune
`tcp_rmem` in one namespace without affecting others, given each namespace has
isolated `sysctl` parameters.
```
Running `/Users/birb/oss/msg-rs/target/debug/examples/bdp_throughput`
=== BDP Throughput Demo ===
Link: 10 Mbit/s, 40 ms RTT, BDP = 50 KB
Transfer: 20 messages × 256 KB = 5 MB
Test 1: Window scaling OFF, max rwnd = 64 KB
Transfer elapsed: 6.715359154s
Throughput: 6.2 Mbit/s (62%)
Test 2: Window scaling ON, max rwnd = 4 MB
Transfer elapsed: 4.609455782s
Throughput: 9.1 Mbit/s (91%)
Window scaling + larger buffers improved throughput by 46%!
```
## How It Works
Under the hood, Linkem creates a hub-and-spoke network topology using Linux
namespaces:
Each peer lives in its own network namespace with a virtual ethernet pair
connecting it to a central bridge. Traffic control rules on each peer's
interface apply impairments based on destination IP — so Peer 1 can have
different latency to Peer 2 versus Peer 3.
The `tc` configuration uses a hierarchy of queue disciplines (qdiscs):
* *DRR (Deficit Round Robin)*: the root qdisc that classifies packets by
destination IP, routing each flow to its own class
* *TBF (Token Bucket Filter)*: enforces bandwidth limits using a token bucket
algorithm
* *netem*: adds delay, jitter, packet loss, and duplication
This is all managed through direct netlink socket communication — no shelling
out to `tc` commands.
### Previous implementation
The current `linkem` is actually a rewrite. The first version was a wrapper
around `tc` and `ip` shell commands — it could create namespaces and apply
impairments, but the implementation was brittle. Shelling out meant parsing text
output and debugging failures through command-line error messages. The code was
hard to extend and the developer experience suffered.
That version also supported macOS via `pfctl` and `dnctl` (the BSD packet filter
and dummynet). While cross-platform support sounds nice, maintaining two
completely different implementations with different capabilities split our
focus. Neither platform got the attention it needed.
For the rewrite, we made two key decisions: Linux-only, and direct netlink
communication. Dropping macOS let us focus on one platform and ship something
more polished. Using netlink instead of shell commands gave us programmatic
control over the kernel's networking stack. We build on the
[`rtnetlink`](https://docs.rs/rtnetlink) crate for standard operations and
construct custom netlink messages where needed. The result is more modular,
easier to debug, and a software for which we have more awareness and control.
## Impairment Options
Each link can be configured with:
| Parameter | Unit | Description |
| ----------- | --------- | -------------------------------------- |
| `latency` | ms | Base propagation delay |
| `jitter` | ms | Random variation added to latency |
| `loss` | % (0-100) | Packet loss percentage |
| `duplicate` | % (0-100) | Packet duplication percentage |
| `bandwidth` | Mbit/s | Rate limit |
| `burst` | KiB | Burst allowance for bandwidth limiting |
Latency and jitter model propagation delay i.e., the time it takes packets to
travel the link. Bandwidth limiting models link capacity with a token bucket
filter. These can be combined to simulate various network conditions: a
satellite link (high latency, moderate bandwidth), a congested datacenter link
(low latency, bandwidth constrained), or a flaky mobile connection (variable
latency, packet loss).
## Limitations
* **Linux only.** The implementation uses namespaces, netlink, and tc. No macOS
or Windows support planned for now.
* **Root required.** Creating and mounting namespaces `CAP_NET_ADMIN` and
`CAP_SYS_ADMIN`
## Closing Notes
`linkem` is still in alpha — the API and ergonomics are evolving as we use it
ourselves and learn what works best. If you're building distributed systems in
Rust and need realistic network testing, we'd love for you to try it out.
We're especially interested in feedback on:
* **API ergonomics**: Is the interface intuitive? What would make it easier to
use?
* **Compatibility**: We've tested on a limited set of kernel versions and
distributions. If you run into issues on your setup, let us know: edge cases
with different kernels, or minimal Linux environments are exactly what we need
to hear about.
Check out the [API documentation](https://docs.rs/linkem) or open an issue on
the [GitHub repo](https://github.com/chainbound/msg-rs). Suggestions, bug
reports, and contributions are all welcome.
# Payflow: An Exploration of Agentic Commerce
**TLDR**; We’re releasing Payflow, a toolkit and an exploration of agentic
commerce through an MCP server powered by
[Cryo](https://github.com/paradigmxyz/cryo) and
[Reth](https://github.com/paradigmxyz/reth), with paid tools using
[x402](https://www.x402.org/) micropayments.
Check out the demo [here](https://cryo-mcp.fly.dev/) and the code
[here](https://github.com/chainbound/payflow).
## Introduction
At Chainbound, we’ve been using agentic tools for engineering and research work
for a while now. Our AI arsenal includes integrated tools like Claude Code,
Codex, Cursor and Deep Research, but more and more it’s also starting to include
various [MCP](https://modelcontextprotocol.io/introduction) servers.
[Exa](https://docs.exa.ai/examples/exa-mcp) is a good example: it can
supercharge the web search capabilities of your agents by including specialized
tools like `research_paper_search`**,** `company_research` and `github_search`.
One annoyance right now, and potentially serious bottleneck in the future, is
that each of these different MCP servers requires you to register for an account
(using your human credentials), and then either sign up for a monthly
subscription or load up on credits. Imagine your AI agent needs to analyze a
competitor's GitHub repositories (using Exa's GitHub search), fetch real-time
market data (via a financial data MCP), and then cross-reference patent filings
(through a legal research MCP). Today, this requires you to manually sign up for
three services, manage three sets of credentials, and probably pay three monthly
fees.
Looking at the future, we believe a more scalable solution will need to come
into place, as tens of thousands of useful MCP servers will be supplying your
agent with different tools and resources, or potentially access to *other AI
agents.* [Smithery](https://smithery.ai/?q=is%3Aremote), an MCP registry, shows
us we’re actually not that far from that situation, with hundreds of remote MCP
servers. MCP is [claimed to tackle](https://humanloop.com/blog/mcp) what’s known
as the “M×N integration problem”, which refers to the exponential complexity
involved in connecting **M agents** to **N integrations** if they all follow a
different protocol. The standardization that MCP brings makes this true on a
technical level, but if each MCP integration requires human involvement, *are we
reaching the full capacity of autonomous agents?* \[1]
In that world, there are a couple of things that don’t work in the flow we
described earlier, and they are closely interrelated:
1. **Identity**: using traditional authentication, authorization, and
potentially KYC methods, which are *always* tailored towards humans and
require humans to be in the loop. This will also be a problem on the
receiving side of the payments. For example, try setting up a Stripe merchant
account as an AI agent, and you will certainly run into problems.
2. **Payments**: human-focused, clunky and potentially inefficient payment
methods such as prepaid credits or monthly subscriptions. On top of that,
small, pay-per-use payments are
[too](https://www.reddit.com/r/stripe/comments/1bno93e/comment/kwn2hxi/?utm_source=share\&utm_medium=web3x\&utm_name=web3xcss\&utm_term=1\&utm_content=share_button)
expensive using traditional rails.
Interestingly, cheap blockchains allow you to tackle both at the same time.
Blockchains allow anyone (including machines) to create and own a non-custodial
wallet, that gives them a unique global address on which to receive transactions
as an (autonomous) merchant. As an AI consumer, the corresponding private key
can be used to authorize transactions from that wallet. An address becomes a
universal API key. \[2]
With smart contract wallets, developers can encode arbitrarily complex spending
rules & authorization methods for their agents, enforced by the contract itself.
The agent could be assigned a passkey to sign transactions with, that only allow
it to spend a certain amount through various rate limits.
Additionally, stablecoin-based micropayment protocols like Coinbase
[x402](https://www.x402.org/) make it much easier to create and accept
stablecoin payments without any human involvement. This is why we felt the
timing was right to explore this paradigm further.
## What is Payflow?
Payflow is our interpretation and example of agentic commerce. It includes 3
things:
* An SDK for building paid MCP servers
* A local MCP server for your agent to create payments
* A demo Payflow-enabled MCP server that puts the 2 together
### `payflow-sdk`
A TypeScript SDK that seamlessly extends
[`@modelcontextprotocol/sdk`](https://github.com/modelcontextprotocol/typescript-sdk)
with the ability to register paid tools using the x402 payment schema. Check out
the code and documentation on Github:
[`@chainbound/payflow-sdk`](https://github.com/chainbound/payflow/tree/main/packages/payflow-sdk).
Adding a paid tool using the `payflow-sdk` is as easy as:
```typescript
import { PayflowMcpServer } from '@chainbound/payflow-sdk';
import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
import { z } from 'zod';
// Create a server
const server = new PayflowMcpServer(
{
name: 'my-paid-server',
version: '1.0.0',
},
{
// Configure x402
x402: {
version: 1,
keyId: process.env.CDP_API_KEY_ID,
keySecret: process.env.CDP_API_KEY_SECRET,
},
},
);
// Simple paid tool without parameters
server.paidTool(
'hello_world',
'Says hello to the world',
{
price: 0.01, // Price in USDC
recipient: '0x1234...', // Ethereum address to receive payment
},
async () => {
return {
content: [
{
type: 'text',
text: 'Hello, World!',
},
],
};
},
);
// Connect transport
const transport = new StdioServerTransport();
await server.connect(transport);
```
This will mark `hello_world` as a paid tool, and the Payflow SDK will add the
necessary description, arguments and return types to ensure the agent using this
tool understands the requirements. The payments are currently settled with
Coinbase’s mainnet facilitator on Base.
This library is still alpha and we're looking for feedback and feature
requests.
Some potential improvements we’re thinking about:
* Support for
[Agent Commerce Kit](https://www.agentcommercekit.com/overview/introduction)
ACK-PAY, with multiple different settlement methods and currencies.
* Support for custom x402 facilitators.
* Extending the library to support paid resources on top of paid tools.
### `payflow-mcp`
Giving an agent access to payment credentials directly in its context window is
not a good idea. In Payflow, the way we deal with this is a fully private, local
MCP server called
[`payflow-mcp`](https://github.com/chainbound/payflow/tree/main/packages/payflow-mcp)
that specifies one important tool (simplified):
```jsx
create_payment(amount, recipient);
```
Only the MCP server holds the credentials, and can be configured with hard
limits that make sure the agent doesn’t bankrupt any connected wallets. MCP
hosts like Claude always request permission to call a certain tool (unless
turning it off), so if required, a human in the loop can still authorize the
payments after checking the details.
Configuring it in Claude is as easy as putting the following into your
`claude_desktop_config.json` file:
```json title="claude_desktop_config.json"
{
"mcpServers": {
"payflow": {
"command": "npx",
"args": ["@chainbound/payflow-mcp"],
"env": {
// The private key of the agent's wallet
"PRIVATE_KEY": "",
// Hard payment limit per query
"MAX_PAYMENT_AMOUNT_USDC": "10"
}
}
}
}
```
### `Cryo MCP`
An [example Payflow MCP server](https://github.com/chainbound/cryo-mcp) that
wraps around [Cryo](https://github.com/paradigmxyz/cryo) and is powered by a
Reth archive node. Learn more about setting it up here:
[https://cryo-mcp.fly.dev/](https://cryo-mcp.fly.dev/). Because we wanted this to be a realistic display of
agentic commerce, we’ve tried to make the MCP server as useful as possible, and
are excited about its capabilities. You can find some examples of what it can do
in the section [here](https://cryo-mcp.fly.dev/#examples).
## Conclusion
Agentic commerce is in its infancy. It’s still unclear what payment rails agents
would use, or how authentication & authorization would work. But crypto payments
are a serious contender, and new protocols like [x402](https://www.x402.org/)
and [Agent Commerce Kit](https://www.agentcommercekit.com/overview/introduction)
are showing a lot of promise. In a world where agents autonomously access
thousands of tools and resources, and pay other agents for services, one thing
is clear: existing mechanisms are insufficient, and cheap blockchains offer an
alternative.
Payflow attempts to display this on a smaller scale, and we offer some tools and
libraries that can help other developers to the same. If you’re interested in
iterating on this experiment with us, reach out to
[explorations@chainbound.io](mailto:explorations@chainbound.io)!
## Footnotes
1. There’s a good argument to be made that human involvement will still be
necessary to give the green light on an MCP integration based on trust,
because malicious MCP servers can do a lot of damage. Check out
[these posts](https://blog.trailofbits.com/categories/mcp/) by Trail of Bits
to get an idea. One way to address this argument would be a verifiable
discovery and reputation layer for MCP services and other agents, but that is
outside of the scope of this article.
2. Compliance might become more challenging in this paradigm. Compliance
workflows might shift from KYC to KYT (Know Your Transaction) or KYA (Know
Your Agent), and different identity and authorization schemes will be
important. Take a look at
[Agent Commerce Kit](https://www.agentcommercekit.com/overview/introduction)
for a promising approach to this
[problem](https://catenalabs.com/blog/ai-and-money-why-legacy-financial-systems-fail-for-ai-agents),
as well as [this article](https://ernstberger.xyz/posts/03_delegate/) from
Jens Ernstberger for more context.
# Block & Blob Propagation with PeerDAS
import { Accordion, Accordions } from 'fumadocs-ui/components/accordion';
TLDR; This article analyzes various key Ethereum network metrics to measure the
impact of PeerDAS and increasing blob counts. These metrics include block + blob
validation latency, attestation rates, and orphan rates. We find a clear
correlation between higher blob counts with PeerDAS, and a negative change in
said metrics ([link to results summary](#summary-of-results)), and argue that
especially with the competitive and latency-sensitive dynamics of PBS, this may
impose artificial blob limits that are below any protocol-defined limits. In
response to these issues, we outline a design for a network of supernodes that
we want to build to help alleviate some of them.
| Does blob count impact: | Block latency? | Attestation rate? | Orphan block rate? |
| ------------------------------ | ----------------------------------------------------------------- | -------------------------------------------------------------------- | ----------------------------------------------------------- |
| Mainnet w/o PeerDAS (baseline) | [**Yes** across the board](#before-peerdas) | [**Yes** but only for p99](#mainnet-before-peerdas-attestations) | [**Inconclusive** results](#mainnet-before-peerdas-orphans) |
| Mainnet w/ PeerDAS-BPO1 | [**Yes** across the board](#with-peerdas-bpo1) | [**Yes** across the board](#mainnet-with-peerdas-bpo1-attestations) | [Not enough data](#mainnet-with-peerdas-bpo1-orphans) |
| Hoodi w/ PeerDAS-BPO2 | [**Yes** for blob count ≥ 9](#latency-on-hoodi-with-peerdas-bpo2) | [**Yes** for blob count ≥ 14](#hoodi-with-peerdas-bpo2-attestations) | [**No**](#hoodi-with-peerdas-bpo2-orphans) |
## Introduction
**PeerDAS**. Ethereum recently deployed Fusaka, an important milestone of its
rollup-centric roadmap, which introduces
[PeerDAS](https://eips.ethereum.org/EIPS/eip-7594) as its first set of features
enabling data availability sampling (DAS). The concrete goal of PeerDAS is to
help the network sustain a much higher number of blobs per block. The network
supported a blob (target, max) of (6, 9) before Fusaka’s deployment and is
expected to support up to (48, 72) hopefully
[within 2026](https://stokes.io/blob-scaling-fyi). PeerDAS heavily modifies blob
mechanics; it changes how blobs are created, referenced, disseminated/custodied,
and verified. PeerDAS therefore naturally raises trade-offs.
**The limitations of DAS**. While validators no longer have to download and
verify full blobs thanks to PeerDAS, proposers do however have to perform more
computation and have to upload more data. With PeerDAS, blobs are now encoded
using Reed-Solomon, resulting in encoded blobs that are twice the size of
pre-PeerDAS blobs (128 kB → 256 kB). The block proposer organizes the encoded
blobs of a block as a 2D matrix where each blob is a row of the matrix. The
matrix is split into 128 columns by the proposer who computes a
[KZG commitment](https://github.com/ethereum/consensus-specs/blob/master/specs/fulu/polynomial-commitments-sampling.md)
per column and disseminates each column separately to each of the 128
sub-networks of validators. Proposers do have to compute and upload more data
per block compared to before PeerDAS, mostly because of Reed-Solomon encoding.
This problem will be exacerbated as more blobs are added per block with the
rollout of BPOs. Some [notable proposals](https://ethresear.ch/t/22298) for the
future of DAS, including [FullDAS](https://ethresear.ch/t/19529) and
[FullDASv2](https://ethresear.ch/t/22477), are aiming for 2D encoding which
further doubles the overhead due to encoding, hence further worsen the bandwidth
cost for builders, proposers, and relays and its impact on dissemination latency
for all validators.
**Empirical analysis**. This document mainly includes an analysis of the impact
of blob count per block on Ethereum thanks to the observations recorded in
ethPandaOps’ [Xatu database](https://ethpandaops.io/data/xatu/). Namely, the
analysis studies the impact of blob count on (1) **block & blob reception
latency for validators**, (2) **successful attestation rate by validators**, and
(3) **the probability of a block to not be finalized and to be orphaned
instead**. The measurements are made on
1. Mainnet pre-PeerDAS deployment to serve as a baseline with a blob count of
`(6, 9)`
2. Mainnet post-PeerDAS and BPO1 hence with a blob count of `(10, 15)`
3. Hoodi post-PeerDAS and BPO2 with blob count of `(14, 21)`
**Results \[[summary table](#summary-of-results)]**. **Increasing the blob count
incurs a clear worsening of both block latency and attestation rate** on almost
all observed networks, both before and after PeerDAS alike. Additionally, we
observed a worse tail latency after PeerDAS compared to before, and the tail
latency increasingly worsens as the blob count increases. However, increasing
the blob count has no noticeable impact on the orphan rate; some results are
inconclusive and overall more data is required to have a meaningful conclusion
regarding orphan rate.
**Proposal:** 🌱 **Seeding network**. Considering the results of the analysis,
we propose to add a network of specialized supernodes into the Ethereum network
with the goal of speeding up the propagation of blocks, blobs, and blob columns
from block proposers (the relays in PBS) to the rest of the network. This added
service will act like a training-wheels network, ensuring specialized actors in
the PBS supply chain don’t start artificially limiting blob counts due to the
risk of including them, as discussed in
[this presentation](https://www.youtube.com/watch?v=gmu2222iQjc).
## Blob Count Empirical Analysis
The impact of blobs on some of these metrics can already be observed in the
Ethereum networks today. The following analysis highlights our main observations
related to the number of blobs present per block and shows that there is still
room for improvement after PeerDAS’ deployment.
### Analysis Scope
This analysis focuses on the following questions. Does increasing the blob count
impact:
1. The time required for validators to receive blocks and required blob columns?
2. The ability for validators to attest to blocks by the 4s deadline?
3. The probability of a block to end up orphaned?
### Analysis Structure
We study four combinations of networks and forks with four different blob
counts:
1. Mainnet under the Pectra fork with a stable network of
[20,000 nodes](https://arxiv.org/abs/2511.15388) and supporting a
`(target, max)` of `(6, 9)` blobs. This dataset serves as a pre-PeerDAS
baseline.
2. Mainnet under the Fusaka fork with PeerDAS and BPO1 activated to support a
blob count of `(10, 15)`. We observed a noticeable reduction in noise after
BPO1 activation so we opted to not include the data before it, i.e., the
first week of Fusaka data. This dataset serves as a stable post-PeerDAS case
study.
3. The Hoodi testnet under the Fusaka fork, with PeerDAS, run by
[2,000 nodes](https://arxiv.org/abs/2511.15388) with BPO2 activated to
support a blob count of `(14, 21)`. This network is less stable and smaller
than Mainnet but already has BPO2 activated. As a testnet, Hoodi’s
architecture is closest to Mainnet which makes it preferable to study than
Sepolia. This dataset serves as a case study with PeerDAS and BPO2.
### Setup
**Sources**: All the results are extracted from the Xatu database maintained by
ethPandaOps. The exact queries used to obtain the plots are publicly accessible
for reproduction in [https://github.com/chainbound/blob-seeder-data](https://github.com/chainbound/blob-seeder-data).
**Plots**: Some plots use a box-and-whisker representation to depict the
distribution of results. The boxes are classic: the ends of the boxes represent
the 25th and 75th percentiles (noted p25 and p75) and the bar inside each box
represents the median (p50). The following plots include two pairs of whiskers
to better depict tail latency behaviors; the whiskers represent p1 and p5 on one
end and p95 and p99 on the other. Outliers are half-transparent colored circles
outside of the whiskers’ range. The box-and-whisker parameters are reminded in
the top right corner of the plots.
### Summary of Results
| Does blob count impact: | Block latency? | Attestation rate? | Orphan block rate? |
| ------------------------------ | ----------------------------------------------------------------- | -------------------------------------------------------------------- | ----------------------------------------------------------- |
| Mainnet w/o PeerDAS (baseline) | [**Yes** across the board](#before-peerdas) | [**Yes** but only for p99](#mainnet-before-peerdas-attestations) | [**Inconclusive** results](#mainnet-before-peerdas-orphans) |
| Mainnet w/ PeerDAS-BPO1 | [**Yes** across the board](#with-peerdas-bpo1) | [**Yes** across the board](#mainnet-with-peerdas-bpo1-attestations) | [Not enough data](#mainnet-with-peerdas-bpo1-orphans) |
| Hoodi w/ PeerDAS-BPO2 | [**Yes** for blob count ≥ 9](#latency-on-hoodi-with-peerdas-bpo2) | [**Yes** for blob count ≥ 14](#hoodi-with-peerdas-bpo2-attestations) | [**No**](#hoodi-with-peerdas-bpo2-orphans) |
* **Latency: Clear impact of blob count on latency.**
* When comparing latency before and after PeerDAS for blocks with the same
blob count, the data shows that, even for low blob counts of 3 to 9,
**PeerDAS has improved validation latency for 95% of validators on Mainnet
compared to Pectra but worsened it for the remaining minority. However,
blocks are disseminated faster with PeerDAS than without, which indicates
that the dissemination of blob data necessary for validation is actually
slower for the last 5% of validators with PeerDAS than without.**
* Mainnet without PeerDAS: The latency increases linearly with the number of
blobs. This trend covers the entire distribution and is visible from p5 to
p99 values. Additionally, p99s are at or above the 4 s deadline for all blob
counts; p99 for 9 blobs is even greater than 5 s.
* Mainnet with PeerDAS-BPO1: The linearly increasing trend persists with
PeerDAS and is visible across the plot from top to bottom for all values
from p5 to p99. The latency is slightly improved for most nodes with PeerDAS
compared to without since most p50 and p75 values are improved by 100-200
ms. However, the unluckiest 1% of validators experience worse block latency
with PeerDAS since the p99 values are all worse for blob count ≥ 3. The p99
values are all greater than 5 s for blob count ≥ 9, greater than 6 s for 14
blobs, and it even reaches 12.4 s for the max blob count of 15.
* Hoodi with PeerDAS-BPO2: There is a clear increasing trend for blob count ≥
9 values that appear mostly linear for all p1 to p99. This indicates that
the trend may likely remain linearly increasing on Mainnet once BPO2 is
deployed.
* **Attestation rate: Clear impact of blob count on attestation rate.**
* Mainnet without PeerDAS: A higher blob count increases p99 of failed
attestation rate for blob count ≥ 4. The blob count barely impacts
attestation rate otherwise.
* Mainnet with PeerDAS-BPO1: Failed attestation rate is overall significantly
worse with PeerDAS than without. p75 before PeerDAS is stable at 0.6% across
all blob counts but it increases from 0.9% to 1.8% as the blob count
increases after PeerDAS. Similarly, p95 increases from 2.4% for 1 blob to 7%
for 15 blobs.
* Hoodi with PeerDAS-BPO2: The failed attestation rate increases linearly with
the number of blobs for blob count ≥ 14. The median rate of validators that
do not attest finalized blocks goes from 4% for 14 blobs to 5% for 21 blobs,
and p99 goes from 11% for 14 blobs to 37% for 21 blobs.
* **Orphan block rate: Little to no impact of blob count on orphan block rate.**
* Mainnet without PeerDAS: The results are inconclusive. One plot computing
the orphan rate based on the total number of slots (the absolute rate) shows
patterns indicating a correlation between blob count increases and orphan
rate increases. However, the second plot that computes the orphan rate based
on the number of finalized blocks containing the same number of blobs —
i.e., the proportional rate — does not show any pattern.
* Mainnet with PeerDAS-BPO1: Orphans are rare, and there is not enough data
yet.
* Hoodi with PeerDAS-BPO2: There does not appear to be a correlation between
blob count and rate of orphan blocks on Hoodi. The plot showing absolute
rate mostly highlights two outlier values, while the plot showing
proportional rate mostly highlights the pattern seen on block latency
between 6 and 13 blobs.
### Mainnet Latency Comparison
**Description**. This plot shows the evolution of the fraction of validators
that have received enough data to validate a block on Mainnet with Pectra versus
with Fusaka rules when there are 1, 5, and 9 blobs in a block. More precisely,
it depicts the cumulative distribution function (CDF) for the block validation
latency that we define as the earliest time at which a validator is able to
validate a block according to the consensus rules. In order to validate a block,
a validator in Pectra must have received the block and all its blob sidecars,
while it must only receive the block and at least 8 blob columns, including all
the blob columns it must custody, under Fusaka rules. The details of the Pectra
and Fusaka datasets — Mainnet before PeerDAS and Mainnet with PeerDAS-BPO1,
respectively — are explained further in the description of their respective
latency plots.
**Takeaways**. All three pairs of lines, when paired by blob count, exhibit a
similar pattern: most validators can verify blocks earlier in Fusaka compared to
Pectra, i.e., the dashed line is mostly above the solid line. Additionally,
adding blobs in a block naturally adds latency. We observe a slight slump in the
dissemination around 2.3-2.5 s that may be due to cross-continental latencies
between dense clusters of validators.
**Description**. This plot is a zoomed version of the above plot, with lines for
3 and 7 blobs added, to highlight the beginning of the tail of the
disseminations, past p95.
**Takeaways**. The behavior of the pair of lines for 1 blob remains consistent
with the above plot: more than 99% of the validators are able to verify blocks
faster under Fusaka than under Pectra when the blob count is that low.
**However, the behavior is reversed for higher blob counts**: the unluckiest
4-5% validators actually require more time to receive the validation data with
Fusaka than with Pectra. The trend worsens as the blob count increases as shown
by the gap between the solid and dashed lines of the same color. These two plots
show that PeerDAS has improved latency for 95% of the validators but worsened
latency for the remaining minority.
**Description**. These two plots mimic the above plots but focus on the
latencies for validators to receive enough blob data to validate blocks, i.e.,
all blobs for each block in Pectra compared to only the required blob columns in
Fusaka. These plots do not consider block latencies, only blob column latencies.
**Takeaways**. These two plots confirm the previous deduction that most nodes
receive blob data faster in Fusaka than in Pectra, that it is the opposite at
the tail, for the unluckiest 5%, and that the trend worsens as the blob count
increases.
#### Before PeerDAS
**Description**. This plot shows the distribution of block discovery &
validation latencies on Mainnet under Pectra (before PeerDAS) depending on the
number of blobs referenced in each block. As mentioned above, Pectra requires
validators to download all blob sidecars referenced in a block in order to
validate it. That latency is visible in the difference between discovery and
validation latencies. Each data point of the plot is a unique tuple: (receiving
node, finalized block). Block latencies higher than 30s have been filtered out
to remove obvious outliers. The dataset is aggregated over the whole month of
November 2025. The black vertical line at the 4s mark indicates the block
attestation deadline.
**Takeaways**. There is a clearly visible increasing trend for all metrics from
p25 up to p99, indicating that a higher blob count leads to a higher block
validation latency. The p99s are already at the 4s deadline for 0-1 blobs and
are above that deadline for all blobs ≥ 2, yet the validators are expected to
not only receive but also process the block and its associated blobs by that
deadline.
#### With PeerDAS-BPO1
**Description**. This plot replicates the previous latency plot on Mainnet but
after PeerDAS and BPO1 have been deployed. In addition, it shows the *block
discovery latency*, as the first box for every level. This makes the added
latency purely from blob column propagation more clear.
As mentioned above, the validation rules for Fusaka are different and require a
validator to not download full blobs but instead at least 8 blob columns,
including all the blob columns it requires to custody. The 11-day sampling
period ranges from 2025-12-10 00:00 UTC, soon after the activation of BPO1, to
2025-12-21 23:59 UTC.
**Takeaways**. The trend appears similar pre- and post-PeerDAS from p5 to p99.
Interestingly, p50 values are overall improved by 100-200ms post-PeerDAS, but
p99 values are worse for blob count ≥ 3, implying a lower quality of service
experienced by the unluckiest 1% of validators. The p99 values are all greater
than 5s for blob count ≥ 9, reaching 6.1 s for 14 blobs and even 12.4 s for the
max blob count of 15.
### Latency on Hoodi with PeerDAS-BPO2
**Description**. This plot depicts block validation latency on Hoodi once
PeerDAS and BPO2 have been deployed. BPO2 has been deployed on 2025-11-12 on
Hoodi and raised the blob count to (target, max) of `(14, 21)`. The data used
for this plot has been sampled over 7 days, from 2025-11-28 00:00:00 UTC to
2025-12-04 00:00:00 UTC, compared to a month for Mainnet because of the much
higher volatility observed on the Hoodi testnet.
**Takeaways**. The impact of blob count on block latency is not as clear on
Hoodi as it is on Mainnet. There is a clear increasing trend that appears mostly
linear for blob count ≥ 9 for all p1 to p99 values. We observed that a lot of
nodes were not receiving enough blob columns to verify blocks, hence
artificially reducing the overall latency because only successful validations
are taken into account in the plot.
### Attestation Rates
#### Mainnet before PeerDAS
**Description**. This plot shows the fraction of validators that never attested
to a block depending on the number of blobs in that block. Each data point
corresponds to a finalized block. For instance, a p99 of 30% means that 1% of
the blocks are attested by only 70% of the validators. The data is aggregated
over the whole month of November 2025 on Mainnet. The x-axis of the plot is
split into two linear axes of different scales to improve the readability of
both boxes and whiskers.
**Takeaways**. There is a very light increasing trend on the median and a more
visible trend on p99 for blob count ≥ 4. The p75 values are all close to or
below 0.6% of failed attestation rate, which indicates a stable network even for
a blob count of 9.
#### Mainnet with PeerDAS-BPO1
**Description**. This plot replicates the above attestation rate on Mainnet but
after PeerDAS is activated and up until BPO1 activation. The plot is similarly
split in two for readability.
**Takeaways**. Most importantly, the attestation rate overall worsens: p75
before PeerDAS hovers around 0.6% of failed attestation rate while it goes from
0.9% to 1.8% as the blob count increases once PeerDAS activates. All
measurements from p50 up to p99 depict a similar trend: blocks have worse
attestation rates as the blob count increases. The attestation rate is much more
impacted by the blob count in this plot, with PeerDAS, than in the previous
plot, without PeerDAS.
#### Hoodi with PeerDAS-BPO2
**Description**. This plot replicates the above plot on attestation rate but on
Hoodi with BPO2 activated. The data has only been sampled over a week because of
the much higher volatility observed Hoodi compared to Mainnet; in some cases,
increasing the sample size increases the presence of outliers.
**Takeaways**. There is a clear increasing trend starting at 12 blobs where all
measures above p25 visibly worsen as blob count increases. Additionally, p99
worsen across the board as the blob count increases. Compared to Mainnet before
PeerDAS and its p75 below 1%, the p75 is much worse on Hoodi with PeerDAS and
ranges between 5% and 7% instead.
### Orphan Block Rates
#### Mainnet before PeerDAS
**Absolute**
**Description**. This plot depicts the orphan block rate categorized by the
number of blobs referenced in these blocks. These orphan blocks have been
created by the expected proposer of their respective slots but, for unspecified
reasons, did not end in the finalized chain of blocks. These rates are computed
proportional to the total number of slots in the sampling period: 216,000 slots
in November 2025.
**Takeaways**. The overall orphan rate is low at 584 / 216000 = 0.27%, which
indicates an “uptime” of 99.73% for Ethereum. We observe two patterns in this
plot. First, a pattern appears to repeat every 3 rows where a high value for a
row is followed by two lower values: 63 orphans for 0 blobs followed by 11 and 1
orphans for the next two rows, 71 orphans for 3 blobs followed by 30 and 22
orphans for the next rows, and similarly for 6-9 blobs. The second pattern shows
that increasing the blob count increases the orphan rate but only once we
consider the first pattern. The increasing trend is clear when only observing
blob counts of 0, 3, 6, and 9. The confidence in this second interpretation is
lowered due to our inability to explain the first pattern.
**Proportional**
**Description**. This plot replicates the above absolute orphan rate plot but
changes the divisors used in the rate computation. This plot shows
“proportional” rates in that they are based on the number of finalized blocks
that have the same blob count as the orphan blocks.
**Takeaways**. We mainly observe a diverse distribution of blob count in
finalized blocks (the divisors in the fraction next to the bars) and two
outliers at 2 and 9 blobs. With outliers ignored, the blob count does not appear
to impact the orphan rate, unlike in the plot showing absolute orphan rate.
#### Mainnet with PeerDAS-BPO1
It would take several weeks to gather enough data to generate a plot for Mainnet
post-Fusaka. For comparison, the above plot showing orphan blocks on Mainnet
before PeerDAS depicts a small amount of orphan rate despite being computed over
a whole month. The data below shows a very small number of observations for the
higher blob counts, so is not indicative. The 15 blob outliers do stand out with
an orphan rate of >5%.
#### Hoodi with PeerDAS-BPO2
**Absolute**
**Description**. This plot depicts the rate of orphan blocks on Hoodi as a
fraction of the total number of slots in the sampling period: 50,400 slots in 7
days.
**Takeaways**. The overall orphan rate over the sampling period is low at 319 /
50400 (orphans / total slots) = 0.63%, which is marginally higher than the 0.27%
of Mainnet. There are two clear outliers, for 0 blobs and 21 blobs, and there
does not appear to be any clear trend indicating an impact of the blob count on
the rate of orphan blocks in Hoodi.
**Proportional**
**Description**. This plot shows the rate of orphan blocks on Hoodi over a week
as a fraction of the number of finalized blocks that have the same blob count.
**Takeaways**. There is a notable surge of the orphan rate between 6 and 13
blobs; we have no explanation for this event. The value for 0 blobs is a clear
outlier, and the value for 21 blobs can be treated as an outlier due to very
high divisor, between 4 and 46x larger than for other blob counts. Once the last
row of 21 blobs is discarded as an outlier, there is no clear trend indicating a
correlation between blob count and rate of orphan blocks.
## Related Studies
* 2024-11: Pre-PeerDAS data showing a trend between block+blob sizes and
delivery latencies
[\[post\]](https://ethresear.ch/t/block-arrivals-home-stakers-bumping-the-blob-count/21096)
* 2025-09: **Ethpandas BPO report** on safety/timely delivery of blobs on Fusaka
devnet 5 with up to 30-50 blobs but with a small dataset
[\[post\]](https://ethpandaops.io/posts/fusaka-devnet-5-bpo-analysis/)
* 2025-09: **EF’s detailed attestation timings analysis** in the post studying
the feasibility of 6-second slots
[\[post\]](https://ethresear.ch/t/an-analysis-of-attestation-timings-in-a-6-s-slot/23016#p-56048-factors-impacting-late-attestations-7)
* 2025-10: Ethpandas report on up/down bandwidth consumption for various types
of nodes on Fusaka devnet 5
[\[post\]](https://ethpandaops.io/posts/fusaka-bandwidth-estimation/)
* Lots of reports with analyses from Sunnyside Labs
[\[reports\]](https://www.notion.so/21d8fc57f546803cb2afd760727c2ff6?pvs=21)
* 2025-10-10
[\[report\]](https://www.notion.so/2858fc57f54680d2a700d4ee1f735587?pvs=21)
that complements the Ethpandas BPO report
[\[post\]](https://ethpandaops.io/posts/fusaka-devnet-5-bpo-analysis/): The
number of missed attestation deadlines (the opposite of the “head
correctness” metric in the reports) goes up to more than 50% for full nodes
as the blob count increases from 10 to 40. Super nodes are mostly fine. The
p75 latency for 50-60 blobs per block reaches 3 s so the tail is much higher
(unclear in reports). Sampled columns are hard to get with 60+ blobs per
block, likely due to network contention: too much bandwidth is required for
the execution client and the consensus client struggles to respond to column
requests.
* 2025-09-30
[\[report\]](https://www.notion.so/27e8fc57f5468093b562f7161d3464f0?pvs=21):
Fusaka devnet 5 with 1,700 nodes (1/8 of mainnet): The bottleneck for high
blob count is the full node uplink due to the high number of blobs in the
mempool (execution client), while the bandwidth for super nodes is mostly in
sampled columns (consensus client).
* 2025-07-14
[\[report\]](https://www.notion.so/2308fc57f5468054b18fcbff31cc032c?pvs=21):
Appendix B for list of Grafana dashboards, Appendix D for some simple
avg+max bandwidth on nodes when increasing blob count.
## 🌱 Proposal: Seeding network
As highlighted in the analysis, PeerDAS negatively impacts a number of
measurements related to consensus stability, especially at the tail. On its own,
for small bumps in blob counts with upcoming BPO’s, this shouldn’t be cause for
alarm. However, in the PBS supply chain, milliseconds matter. Relays offer
services for validators to delay committing to a block for as long as possible,
to maximize MEV (timing games). This works because as of today, the latency
penalty for including blobs is relatively minimal. As we’ve seen above however,
that’s already changing, which may lead some entities in the PBS supply chain to
impose artificial ceilings on the blob limit that are way below the *actual*
limit. This would counteract the DA scaling benefits that are PeerDAS’ main
raison d'être.
**What.** Because of this, we propose to build a global network of supernodes
whose only task is to accelerate the propagation of data (blocks and blobs) from
the block originator (relay in PBS) to the rest of the network. We expect the
seeding network to reliably reduce and stabilize the latency experienced by
validators by reducing and bounding the number of communication hops required to
disseminate data to most validators. It will additionally improve blob usage
efficiency by ensuring that builders can reliably include the economically
optimal number of blobs without any additional latency penalties.
The concept of seeding network is aligned with the
[Rainbow staking framework](https://ethresear.ch/t/18683) (albeit out of
protocol) as it relies on more powerful supernodes to contribute more to the
network than regular nodes, ultimately improving the quality of service for
everyone. These supernodes extend the idea of *DAS providers* as described in
the [original PeerDAS post](https://ethresear.ch/t/16541#das-providers-13). A
[couple](https://notes.ethereum.org/NgKvvYiAQ7WdxPECCVdGUQ) of
[designs](https://ethresear.ch/t/21758) have proposed for PBS relays to become
DAS providers and to offer an RPC service that validators can query to obtain
samples. We propose to extend these designs with an additional support for
GossipSub in order for supernodes to proactively accelerate data dissemination
instead of only being passive actors waiting for client queries.
**How**. This seeding network will be composed of highly performant and highly
connected supernodes that will act as network hubs in order to deliver as much
data to as many nodes as possible. Supernodes will contribute to the
dissemination of block and blob columns by subscribing to the GossipSub topics
used for blocks (`beacon_block`) and for the 128 blob column subnets
(`data_column_sidecar_[0-127]`). Additionally, supernodes will reply to the
relevant RPC requests for blocks, blobs and blob columns, e.g.,
`BeaconBlocksByRoot`, `BlobSidecarsByRoot`, `DataColumnSidecarsByRange`.
The inter-supernode connections will be implemented the battle-tested, highly
performant [https://github.com/chainbound/msg-rs](https://github.com/chainbound/msg-rs) that powered our low-latency
mempool service
[Fiber](https://research.chainbound.io/fiber-vs-bloxroute-the-standoff). The
geographical placement of each supernode and the topology of the network will be
designed to minimize overall dissemination and request latencies for the
validators, ensuring proper distribution in hotspots such as Virginia, Frankfurt
and Tokyo.
### Expected Metric Improvements
We expect the seeding network to have clear improvements on the Ethereum network
in terms of:
1. **Block and blob reception latency**: A small network of high-performance
supernodes enables dissemination with fewer hops than a larger and more
heterogeneous network does.
2. **Attestation rate and orphan block rate**: Thanks to a faster dissemination
of both blocks and blob columns, validators will be able to attest blocks
faster, hence reducing the probability that blocks fail to be attested in
time and end up as unproductive orphans instead of being part of the
finalized chain.
3. **Reduction of strategic blob limiting in PBS**: Because of the above 2
improvements, the benefits of strategic (lower) blob inclusion in the PBS
supply chain are nullified.
### ePBS (Glamsterdam)
**Source**:
[\[high-level description\]](https://ethereum-magicians.org/t/eip-7732-the-case-for-inclusion-in-glamsterdam/24306)
**Problem**: A 12 s slot may be too small to do all the downloading and
verification that validators do before attesting a block.
**Proposal**: Minimize what information is downloaded and verified (i.e., only
the consensus block) during the slot that the block is proposed in, and tolerate
a delay of up to an additional slot to perform the rest of the download and
verification (e.g., blobs, execution payloads).
**Expected result**: Better attestation rate, particularly for blocks with high
blob count.
**Readiness**: Headliner for Glamsterdam.
**Compatibility with seeding network**: Compatible but greatly reduces the need
to reduce block and blob delivery latency. However, ePBS does not impact the
motivation regarding bandwidth or reduced slot duration.
### Sparse blobpool (Maybe Glamsterdam?)
**Source**:
[\[EIP draft\]](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-8070.md)
[\[older ethresearch post with similar ideas\]](https://ethresear.ch/t/is-data-available-in-the-el-mempool/22329)
**Problem**: The network bandwidth of the execution client overshadows that of
the consensus client when the blob count increases towards high dozens, e.g.,
68% of the bandwidth is used by the execution client for blob (target, max) of
(48, 72).
**Proposal**: Perform blob sampling on the execution client as well, in addition
to PeerDAS doing sampling on the consensus client. The sampling done by the two
clients is aligned: the execution client queries the network for the columns
needed for the consensus client. The probability of sampling a blob could
theoretically be set independently by each full node, but the authors propose a
single network-wide parameter to ensure enough nodes have full blobs.
**Expected result**: By only downloading full blobs 15% of the time and sampling
85% of the time, the execution client reduces by 75% its bandwidth consumption.
**Readiness**: Proposed for integration into Glamsterdam
[\[list update\]](https://github.com/ethereum/EIPs/pull/10651).
**Compatibility with seed network**: Good. Sparse blobpools reduce remote
queries for full blobs so a seed network would be less useful but not useless
since a lot of blob data is still downloaded.
### Sharded blob mempool
**Source**: [\[ethresearch post\]](https://ethresear.ch/t/22537)
**Problem**: (1) Blobs downloaded by execution clients incur a high network
bandwidth, (2) gossiped data related to blobs received by both consensus and
execution clients may be redundant, and (3) consensus clients download complete
columns while the execution client may already have a subset of them.
**Proposal**: Better synergy between consensus and execution clients by: (1)
Sharding the blob mempool such that execution clients only download a subset of
it (e.g., 16 in the proposal) and it must be the same rows custodied by the
consensus client, (2) having only the execution client disseminate the blob
rows, (3) enabling partial column queries to avoid having consensus clients
downloading what they already have. The idea behind sharded blob mempool is
similar to sparse blobpool in the sense that it makes consensus and execution
clients work better together.
**Expected result**: Reduced bandwidth and latency required to validate blocks.
**Readiness**: High-level description post.
**Compatibility with seed network**: Good. The proposal reduces the overall
network bandwidth so a seed network would be less useful but not useless.
### Future of DAS
**Main source:**
[\[Ethresearch summary of various combinations of commitments and coding\]](https://ethresear.ch/t/22298)
**Other sources**: [\[FullDAS ethresearch post\]](https://ethresear.ch/t/19529)
[\[FullDASv2 ethresearch post\]](https://ethresear.ch/t/22477)
[\[a16z post on using Pederson commitments with IPA instead of KZG\]](https://a16zcrypto.com/posts/article/an-overview-of-danksharding-and-a-proposal-for-improvement-of-das/)
**Problem**: PeerDAS applies Reed-Solomon coding separately per blob, in what is
called 1D coding and sampling, and a future goal is to move towards 2D coding
and sampling for greater resilience to blob data loss. The problem is how to
best design this second dimension of coding.
**Proposal**: FullDAS and FullDASv2 are similar to PeerDAS in that
Reed-Solomon + KZG are used for the second dimension as well.
The summary post explores how to do cell-level DAS several combinations of
commitment schemes — namely Merkle trees, KZG commitments, and Pedersen
commitments — and coding schemes — namely Reed-Solomon, random linear codings
(RLCs), and random linear network codings (RLNCs). The summary concludes that
using KZG commitments may not be the best option for the second dimension due to
the high computational cost incurred on every node that forwards the samples,
which would reduce propagation at every step. Merkle trees may be the most
viable for fast propagation.
**Expected result**: Reduce the amount of data required to validate blocks.
**Readiness**: Descriptive posts.
**Compatibility with seed network**: Perfect. 2D coding and sampling mean more
data to transmit and an even slower seeding, making the seed network even more
relevant.
### RLNC-DAS by Optimum
**Source**: [\[arXiv\]](https://arxiv.org/abs/2509.21586)
**Problem**: PeerDAS has a small CPU footprint on the proposers that must
perform additional computation and a big networking footprint for proposers and
supernodes.
**Proposal**: Replace PeerDAS’ mechanisms based on KZG commitments and
Reed-Solomon coding that are computed once at block building time with Pederson
commitments generated once at block building time and later random linear
network coding (RLNC) that is done on the fly by full/supernodes as clients
query them for blob samples. RLNC codes are therefore not broadcast on the
network; there is no network overhead during broadcast since only the payload of
blobs is broadcast.
**Expected result**: Much better network bandwidth footprint for sampling
clients (10x better than PeerDAS) and zero storage overhead. The main downside
is the increased CPU and network footprint for full nodes, which appears to
scale linearly with the number of querying clients but is unfortunately not
evaluated.
**Readiness**: Prototype with unreleased code and public tech report.
**Compatibility with seeding network**: Incompatible. RLNC-DAS is completely
different from PeerDAS. It is designed to minimize the amount of broadcast data
while the proposed seeding network strives for scenarios with a lot of broadcast
data. It is compatible with supernodes since the CPU footprint grows linearly on
a node with the number of clients that query blob samples.
## Acknowledgments
We thank the ethPandaOps team for giving us access to and helping us with the
Xatu databases, as well as the Xatu contributors for populating Xatu with data
for us to analyze. We also thank the authors of the “Multiple Sides of 36 Coins”
blockchain network analysis published at SIGMETRICS 2026
[\[arXiv version\]](https://arxiv.org/abs/2511.15388), and particularly Lucianna
Kiffer, for sharing updated numbers on the sizes of various Ethereum networks.
# Private API Credits: A Simple Construction
## Introduction
Privacy is back on the public radar. Over the past year, multiple prominent
players within the Ethereum ecosystem have been
[unveiling](https://ethereum-magicians.org/t/a-maximally-simple-l1-privacy-roadmap/23459)
[their](https://ethresear.ch/t/ethereum-privacy-the-road-to-self-sovereignty/22115)
[privacy roadmaps](https://pse.dev/blog/pse-roadmap-2025). More recently, the
Privacy & Scaling Explorations ([PSE](https://pse.dev/)) team have undergone a
metamorphosis: they are now the Privacy Stewards of Ethereum. The price of Zcash
increased 9-fold over the last year, and shielding protocols like
[Railgun](https://www.railgun.org/) have
[continued growing in deposited liquidity](https://dune.com/railgun_project/railgun).
Despite this rapid growth and renewed attention, a recent
[report](https://pse.dev/blog/privacy-experience-report) by PSE shows that
overall adoption remains low, with many pain points persisting for more privacy
conscious users.
One concern of the report stood out to us in particular: a user setting up
[Privacy Pools](https://privacypools.com/) locally was confused that they had to
configure an Alchemy API key to get the front-end running locally.
> There are so many leaks if I’m using Alchemy… what is the point?
They have a point. An API key is linked to an account on a certain platform,
which contains personally identifiable information (PII). It also surfaces the
full chain of requests made by that single account. Having an account is clearly
needed for purchasing API credits, as well as potentially any compliance checks.
But is the link between an API key and the PII really necessary? Additionally,
do we really need a persistent identifier across *all* requests? Can it be
avoided so users can remain safe from honest-but-curious providers or data
breaches? To answer this question, we’ll need to take a detour and look at a
fairly annoying but ubiquitous aspect of browsing the internet: CAPTCHAs.
## CAPTCHAs & Privacy Pass
CAPTCHAs were introduced to identify requests coming from humans instead of
bots, and are used in website protection services like Cloudflare. This was a
necessary service to protect websites from DoS attacks, but it introduced a lot
of friction for users. Millions of collective hours were wasted in solving
semi-useless puzzles as a proxy for personhood.
When [Privacy Pass](https://privacypass.github.io/) was introduced in 2017,
Cloudflare was one of its
[first adopters](https://blog.cloudflare.com/cloudflare-supports-privacy-pass/).
Why? Because Privacy Pass provided a way to drastically reduce the amount of
CAPTCHAs users had to solve online, all while maintaining user privacy. This
last part is important. Cloudflare could technically issue a persistent
identifier to users solving a CAPTCHA, that can be redeemed every time a user is
presented another challenge. But that would allow Cloudflare to trivially track
the complete browsing history of a user. Not great.
The Privacy Pass protocol solves this by leveraging smart cryptography. Instead
of a persistent identifier, successful solvers of a CAPTCHA get a set of blinded
tokens that are signed by Cloudflare. The magic happens when the user receives
these signed, blinded tokens: they can unblind them into a form that is
cryptographically unlinkable to the original blinded token, all while the
cryptographic signature representing the stamp of approval remains valid! These
unblinded tokens are what the user (the Privacy Pass extension to be exact)
redeems when they are presented with a challenge, and Cloudflare allows them to
bypass the challenge because they see their valid signature on it. To
Cloudflare, every token looks completely random, so they can’t track users with
this mechanism.
### Privacy Pass for General Authentication
In the [Privacy Pass Architecture RFC](https://www.ietf.org/rfc/rfc9576.html),
the issuance of tokens happens after a process they call attestation. In this
context, an attestation just signifies the fact that you have satisfied the
attester. But an attestation can be anything: with CAPTCHAs, your attestation is
that you solved a puzzle correctly. With Apple’s
[Private Access Tokens](https://developer.apple.com/videos/play/wwdc2022/10077/)
(their extension of Privacy Pass), what’s being attested to is the fact that the
user is using an Apple device, is logged in to an Apple ID, and is not rate
limited. Aside from the lock-in concerns, this is neat because it’s all private.
If we generalize the Privacy Pass mechanism, it can be used for *any* form of
private authentication, not just CAPTCHA puzzles. The magical property here is
the *unlinkability* between the *issuance* of tokens, which happens after a
successful attestation, and *redeeming* those tokens. This mechanism is a
powerful and private alternative to persistent session identifiers.
[Kagi](https://kagi.com/), a subscription-based search engine, realized this as
well.
[They use Privacy Pass](https://help.kagi.com/kagi/privacy/privacy-pass.html)
for exactly this mechanism: allowing authenticated users to execute private,
unlinkable searches.
An astute reader may have realized where we’re going with this: **this protocol
could easily be adapted to retrofit RPC providers with private RPC requests**.
It could completely sever the link between signing up and buying credits, and
then spending those credits in subsequent requests. RPC providers can still work
with their prepaid API credits model, they just won’t be able to correlate any
of the requests with the original accounts, or track a single user over time.
This would (almost) solve the problem our user was having! We can even design it
in a way that users of client libraries barely have to change anything in their
workflows, but more on that later.
Key takeaway: the same mechanism that replaces repeated CAPTCHAs can replace
persistent API keys.
## The Protocol
So how does Privacy Pass work? We won’t get into the actual cryptography here,
but rather give you a high level overview of the interactions to help you build
an intuition. The first iteration of Privacy Pass based on blind signatures is
the easiest to understand, so we’ll start there.
Blind signatures are usually conceptually explained with
[carbon paper](https://en.wikipedia.org/wiki/Carbon_paper). Imagine a voting
center where a local community goes to vote. Before shipping off the ballots to
a central area where the votes are counted, the local voting center certifier is
required to sign (certify) each ballot. Certifying in this case consists of
ensuring that the ballot came from a person that was present, and that it’s
their only ballot (no double voting). The problem is that voters are not
comfortable sharing their ballots with the certifier, because they don’t trust
him not to look at the vote while signing.
Someone comes up with a system to address this:
1. Put a piece of carbon paper on top of the ballot, wrap it in an envelope, and
seal the envelope.
2. Give the envelope to the certifier who will authenticate you and then sign
the envelope. Because of the carbon paper, the signature will be transferred
onto the ballot.
3. When votes are tallied, the counter can unseal the envelope and find the
certifier’s signature on the ballot, thereby knowing the vote was valid.
This system ensures that the certifier can authenticate a ballot without seeing
the actual votes, and the content of the ballot can then be verified by a
different party who doesn’t know the voter!
### Blind RSA Signatures
In the digital realm, this system can be replicated with cryptographic
primitives like RSA blind signatures. In the issuance process, clients can
generate a bunch of random nonces $n$, and then *blind* them with a blinding
factor $r$. The blinding process works such that any signature on the blinded
nonce will also be valid on the unblinded nonce, but the signer cannot recover
the unblinded nonce because it doesn’t know what $r$ is.
In practice, Privacy Pass has moved toward VOPRF-based constructions, but blind
RSA is easier to explain and shares many of the same properties.
We’ve skipped over some details of RSA blind signing here, but this should give
you a bit of an intuition of how it works.
## Architecture
As a thought exercise, what could a UX-friendly private API credits architecture
look like? Let’s define the components first:
* **Provider**: hosts a **portal** for users to sign up and purchase credits,
and hosts protected **resources** that users want to access privately. In the
case of RPC providers, the resources would be the nodes they host, accessible
over JSON-RPC.
* **User**: user of the system, signs up for an account and purchases credits
through a portal. Interacts with the resources through a **client**.
* **Client**: software operated by the user that interacts with the provider and
its **resources**, and abstracts away the complexities of dealing with private
credits.
The logic of both the **Provider** and the **Client** will have to be modified
to replace regular API keys with private credits. However, for ease of use, we
can still use a key that uniquely identifies a user, and allows the client to
programmatically request private credits from the provider, and the provider to
keep track of balances. This is not a problem, because all this information is
contained to the initial authentication and issuance context. This key will
**not** be used when making actual API requests. We’ll call this key the
**Account Key**.
Schematic overview of the architecture.
### Client
We’ll start with the client. One of our goals here is to minimize the friction
introduced by private credits. Therefore, the client should encapsulate all of
the following logic:
* Producing random nonces and blinding them
* Interacting with the provider to receive blinded, private API credits
* Attaching those credits to RPC requests to authenticate them
As mentioned before, we can make use of the account key. This is a key that
uniquely identifies the user and has an associated balance that it can mint in
private API credits. All the user has to do is purchase credits on the portal
and instantiate the client library with the account key. It will take care of
the rest.
### Provider
We can explain the functionality that would need to be added to the provider in
the form of a hypothetical Private Credits SDK. This SDK would be initialized
with a keypair (the **Issuance Key**) of which the public key is known to all
clients. The corresponding private key will be used to sign private credits
with.
The portal would manage user billing and account keys, and upon issuance it will
use the SDK to blind sign credits.
One key requirement of the SDK is that it should contain some nullifier set of
already spent credits to prevent double-spending. This means that on credit
redemption, the unblinded nonces provided with API requests should be added to a
cache to indicate that they’ve been spent.
## Practical Considerations
### Nullifier State
Edge services need to redeem and validate private credits, which includes
invalidation by adding the nonce to a nullifier set. To limit state growth, the
first requirement is the use of epochs. These are fixed intervals of time that
set bounds on the validity of a private credit (i.e., they expire after 1
epoch). This alleviates the state growth problem by being able to purge the
nullifier set after every epoch.
**Data Structure**
The second consideration is that of which data structure to use, which can also
have a big impact on state growth. We outline 2 options here:
* Small but probabilistic
* Large but deterministic
The first option contains data structures like Bloom filters that are extremely
space-efficient. They can be used for membership testing just like a hash set,
but they can render occasional false positives. In this case, this could mean
that a certain credit could be wrongly marked as spent! Fortunately, because
epochs limit the amount of possible items in a filter, it’s possible to
configure the size of the Bloom filter to make these events sufficiently rare.
Alternatively, newer constructions like
[cascading Bloom filters](https://philthompson.me/misc/cascading-bloom-filters/)
could be used as well.
The second option uses sets. These are deterministic (no false positives), but
grow linearly with the number of elements in the set, and are thus not very
space-efficient.
**To Share Or Not To Share?**
In a highly-available setup, many API endpoints may exist to service requests.
This means that each of them should have access to the latest nullifier set.
Sharing this state directly between services with strong consistency is not an
option, because it would require consensus between potentially distributed
services.
What we would recommend instead is to have a dedicated service for redeeming
credits that manages the nullifier state. One could make this highly available,
but probably only within a region to ensure low latency.
### Compute Units Accounting
Since RPC providers use compute units for accounting, private credits should be
denominated in CUs. For example, 1 credit could represent 10 CUs. This works for
all models where the cost is known upfront. However, more research is needed to
make this work for models with dynamic API costs, like LLM APIs with token
accounting.
### Latency
Modern Rust-based privacy pass libraries like the one used in the Brave Browser
are quite fast
([link](https://github.com/brave-intl/challenge-bypass-ristretto)). Signing a
batch of 30 tokens takes \~1.2ms, while redeeming 30 tokens takes \~850µs. Check
out the benchmark here:
[https://github.com/brave-intl/challenge-bypass-ristretto/pull/77](https://github.com/brave-intl/challenge-bypass-ristretto/pull/77).
Interacting with the nullifier service is also in the hot-path, so should be
accounted for as well.
## Improvements
The [Privacy Pass Architecture RFC](https://www.rfc-editor.org/rfc/rfc9576.html)
defined 3 logical roles: the Issuer, Attester, and Origin. In the previously
discussed design, all of these roles were played by the Provider. The portal
authenticates users and verifies balances (attestation), and then issues credits
(issuance). The resources here are also exposed by the provider (origin). The
RFC actually
[mentions](https://www.rfc-editor.org/rfc/rfc9576.html#name-shared-origin-attester-issu)
that there *could* be cases where this presents a risk.
All of these risks boil down to the simple observation that privacy loves
company. If your anonymity set is small, because this feature is exposed as a
separate paid plan that initially does not have a lot of users, you have a
chicken-and-egg problem on your hands. With only a small set of users, the
anonymity guarantees are quite low. But this would be mitigated as soon as there
are more users using the feature, and the more users, the stronger the privacy
guarantees. On the other hand, if this feature is introduced as a default, the
problem would not exist.
Another risk the RFC mentions is the following:
> *Origin-Client, Issuer-Client, and Attester-Origin unlinkability requires that
> issuance and redemption events be separated over time \[…], or that they be
> separated over space, such as through the use of an anonymizing service when
> connecting to the Origin.*
Separation over time is useful because of timing correlation analysis: if a user
gets issued credits, and immediately after that starts redeeming them, it could
be possible to correlate the 2 events. Note that this is once again tied to the
size of the anonymity set: if there are many users, that are constantly issuing
and redeeming, this analysis becomes sufficiently hard. But to be safe, the
client should implement this separation in time.
Separation in space is related to network metadata: if the same IP is used in
the issuance process, and later on when redeeming credits, unlinkability is also
lost. It is therefore necessary to anonymize either of the 2 interactions
through a third-party service. Another IETF standard that could be leveraged
here is
[Oblivious HTTP (RFC 9458)](https://www.rfc-editor.org/rfc/rfc9458.html).
Oblivious HTTP introduces third-party relay in the request path that hides
network metadata from the origin, while not being able to deduce anything about
the request itself other than its source and destination. From a trust
perspective, it’s similar to
[iCloud Private Relay](https://blog.cloudflare.com/icloud-private-relay/): it
works unless the 2 parties collude.
## Use Cases
We’ve explored private API credits from the perspective of private RPC requests.
As mentioned before though, the construction is general enough to work in any
deployment that uses API keys for authentication. It is especially relevant in
scenario’s where decoupling account information (PII), with API usage is
desirable. One particularly interesting case is **private LLM provider APIs**.
With private credits, prompts and conversations will be unlinkable to PII
(unless you include PII in the prompts). We think that this should be an option
that LLM users have if they don’t want the providers to be able to construct a
complete picture around their identity. There are some interesting open problems
here though:
* Conversations are by their nature a chain of messages, so achieving
unlinkability between requests is not really possible. Maybe this is less of
an issue as long as it can’t all be linked back to a certain identity.
* The exact cost of an API call in tokens is not possible to know in advance, so
a variant of this that works with dynamic costs should be investigated.
## Conclusion
All of the primitives for replacing the API key with private credits are there,
and on top of that, they are standardized by the IETF. Multiple companies have
open-source, audited implementations of the cryptography. In conclusion, all of
the building blocks exist, and are waiting to be put together. If you are
interested in collaborating with us on this, please reach out to
[dev@chainbound.io](mailto:dev@chainbound.io) or ping
[@mempirate](https://x.com/mempirate) on X.
# Estimating Validator Decentralization Using P2P Data
This research was funded by the Robust Incentives Group at the Ethereum
Foundation. This work is specifically related to ROP-8. Additional information
can be found
[here](https://www.notion.so/bad7233658cc41f38b26e7b4f6cf6e8b?pvs=21).
We want to thank [soispoke](https://x.com/soispoke), the
[EF DevOps team](https://x.com/EthPandaOps), [MigaLabs](https://migalabs.io/)
and [ProbeLab](https://probelab.io/) for their advice and contributions!
## Introduction
The geographical distribution of a validator set is
[one of the most critical factors](https://collective.flashbots.net/t/decentralized-crypto-needs-you-to-be-a-geographical-decentralization-maxi/1385)
in determining a blockchain's level of decentralization. Validator
decentralization is vital for Ethereum. It enhances network security,
resilience, and censorship resistance by distributing control and minimizing the
risk of single points of failure or malicious attacks.
It is well known that Ethereum has a
[very large](https://beaconcha.in/charts/validators) validator set, but **is
this validator set geographically distributed?** Ethereum has a substantial
amount of beacon nodes running on the consensus layer network, with current
estimates at around \~12,000 active nodes ([source](https://nodewatch.io/)). A
beacon node serves as a *potential* entrypoint into the network for validators,
but it is not representative of the actual validator distribution.
Probably not.
## Anatomy of a validator
An Ethereum validator is a virtual entity that consists of a balance, public key
and other properties on the beacon chain. They are roughly responsible for 4
things:
1. Proposing new blocks
2. Voting on other block proposals (attesting)
3. Aggregating attestations
4. Slashing other validators in case they commit faults
A *validator client* is the piece of software that executes these
responsibilities for each of its registered validator keys (which can be many).
But a validator client on its own cannot connect to the P2P beacon network to
talk directly to other validators. Instead, it connects to an entity known as a
*beacon node*, which is a standalone client that maintains the beacon chain and
communicates with other beacon nodes.
Schematic of validator clients and a beacon node.
Beacon nodes can have a number of validators attached to them that ranges from
zero to thousands. In fact,
[it’s been reported](https://medium.com/@grandine/grandine-0-4-1-released-fb98daef6d60)
that in some Ethereum testnets client developers have been running upwards of
50k validators on a single machine. This separation of concerns makes our
investigation somewhat harder: a simple crawl of the P2P network might give us a
good overview of the set of online beacon nodes in real time, but this is not
representative of the overall validator client distribution at all.
Before we address this problem, we’ll take a closer look at validator duties and
their footprint on the network.
## Attestation duties and committees
As mentioned above, one of the main responsibilities of a validator is voting on
blocks by broadcasting *attestations*. These attestations express the view of a
validator about which chain they think is correct. In more detail, they actually
cast 2 different votes: one to express their view of the current head block, and
one to help finalize past blocks. This is because Ethereum’s consensus is a
combination of [2 subprotocols](https://arxiv.org/pdf/2003.03052): LMD Ghost, a
fork-choice rule, and a finality gadget called Casper FFG.
These duties are assigned randomly every epoch (with some
[lookahead](https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/validator.md#lookahead))
with RANDAO as the source of randomness. Validators get assigned to one slot per
epoch at which they have to cast their attestation, which is just a message with
the votes that is signed over with the validator BLS private key. These votes
are then to be packed and stored in the next beacon block. However, if
[all 1 million validators](https://beaconcha.in/charts/validators) were to
attest for every block, the network would be flooded with messages, and the
proposer that is supposed to pack these attestations into their block would have
trouble verifying all of those signatures in time. This would make Ethereum’s
design goal of low resource validation unfeasible.
To address these issues, the beacon network is subdivided into *committees*,
which are subsets of the active validator set that distribute the overall
workload. Committees have a minimum size of $128$ validators, and there are
$64$ committees that are assigned per slot. But how is this achieved in
practice? What network primitives do we require to enable such a logical
separation?
## Attestation subnets
The Ethereum consensus P2P network is built with
[GossipSub](https://github.com/libp2p/specs/tree/master/pubsub/gossipsub), a
scalable pubsub protocol running on libp2p. Being a pubsub protocol, GossipSub
supports publish/subscribe patterns and the segmentation of networks into
logical components called *topics* (aka P2P overlays)*.* These are the
networking primitives that underpin beacon committees.
One example of a topic is the
[`beacon_block`](https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/p2p-interface.md#beacon_block)
topic, which is a *global topic* on which new beacon blocks are broadcast. Every
validator must subscribe to this topic in order to update their local view of
the chain and perform their duties.
The attestation overlays look quite a bit different. For each committee, we
derive a subnet ID based on the committee index (0-64). The topic for the
respective subnets then becomes
[`beacon_attestation_{subnet_id}`](https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/p2p-interface.md#beacon_attestation_subnet_id).
Every validator knows their upcoming attestation duties at least 1 epoch ahead
of time and can join the correct subnet in advance. When they have to make an
attestation, they broadcast it on this subnet.
As mentioned before, these attestations are eventually supposed to make it into
a beacon block. But since upcoming proposers might not be subscribed to these
subnets, how does that work? This is where *attestation aggregators* come in.
These are a subset of the beacon committees that are responsible for
*aggregating* all of the attestations they see and broadcasting the aggregate
attestations on the global
[`beacon_aggregate_and_proof`](https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/p2p-interface.md#beacon_attestation_subnet_id)
topic. This topic is again a mandatory global topic that all validators will be
subscribed to, thus providing a way for local unaggregated attestations to make
it into the global view of the network. Per committee, there’s a target number
of aggregators of $16$.
### Subnet types
These attestation subnets described above are ephemeral and directly tied to the
validator duties. We call these **short-lived** attestation subnets. The problem
with these ephemeral subnets is that they are not very robust, and could result
in lost messages. To deal with this issue, the notion of a
“[subnet backbone](https://github.com/ethereum/consensus-specs/issues/2749)” was
introduced.
This backbone consists of **long-lived**, persistent subnet subscriptions that
are not tied to validator duties but rather a
[deterministic function](https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/p2p-interface.md#attestation-subnet-subscription)
of the beacon node’s unique ID and the current epoch. These long-lived subnets
are maintained for $256$ epochs, or around 27 hours, and each beacon node has to
subscribe to 2 of them. They are also advertised on the discovery layer, making
it easier for beacon nodes with certain duties to find peers on the relevant
subnets.
## Validator footprints
Returning to the separation of the beacon node and validator clients, there’s
now a clear footprint that validators leave on the beacon node’s network
identity: their short-lived subnet subscriptions. This will be the core of our
methodology.
## Methodology
Generally, the beacon network consists of 3 domains:
* The discovery domain
* The Req/Resp domain
* The gossip domain
Each of these domains provides some information about a beacon node.
### Long-lived subnets & node metadata
At the **discovery layer**
([discv5](https://github.com/ethereum/devp2p/blob/master/discv5/discv5.md)), a
beacon node’s identity consists of an
[ENR](https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/p2p-interface.md#enr-structure)
with some additional metadata. This metadata can roughly be represented as the
following object:
```js
{
(peer_id,
ip,
tcp_port,
udp_port,
attnets,
fork_digest,
next_fork_version,
next_fork_epoch);
}
```
This metadata helps other peers connect to peers that are relevant to them,
indeed, one of the extra metadata fields are the (long-lived) attestation
subnets that this node is subscribed to!
The **Req/Resp domain** is where the actual handshake happens. This is where
nodes exchange `Status` messages that look like the following in order to
establish a connection:
```js
(
fork_digest: ForkDigest
finalized_root: Root
finalized_epoch: Epoch
head_root: Root
head_slot: Slot
)
```
The underlying protocol used for the Req/Resp domain is (again) libp2p. On the
lower levels, additional information like `client_version` is also exchanged
when connections are set up.
It is at this level that peers can also exchange `MetaData` objects to identify
each other’s most up to date long-lived subnet subscriptions. The
[`MetaData`](https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/p2p-interface.md#metadata)
object looks like this:
```js
(
seq_number: uint64
attnets: Bitvector[ATTESTATION_SUBNET_COUNT]
...
)
```
### Short-lived subnets
So far, we’ve only seen how nodes exchange metadata and their long-lived subnet
subscriptions, which tell us nothing about potential validators. For that, we
need the short-lived subnets, which we can only collect on the gossip domain.
Our initial strategy was doing just that:
1. Listen to incoming topic subscription requests
2. Save and index them
However, on an initial review of the data, we saw way too many beacon nodes that
didn’t subscribe to any additional subnets besides their long-lived, mandatory
subscriptions.
Our assumption was that in order to publish data on a gossipsub topic, one
needed to be subscribed to it. It turns out that this is not the case, and many
clients have different behaviour to minimize bandwidth and CPU usage. Rather
than subscribing to the subnet directly, the peer finds other peers that are
subscribed to the required subnet beforehand and shares the attestation with
them. The subscribed peers make sure to verify and forward these attestations.
Remember that in theory, only attestation aggregators need to be listening to
all incoming attestations in order to do their jobs. This is exactly what was
happening, and explains why we had so little short-lived subnet observations.
With this understanding, we could now tune our assumptions:
* For each subnet, there’s a target of `TARGET_AGGREGATORS_PER_COMMITTEE=16`
aggregators per committee
* This means that on average, there will only be $16$ validators per committee
that will be subscribed to an additional short-lived subnet for the duration
of an epoch
* This results in a maximum of $16 * 32 * 64 = 32768$ useful observations per
epoch
With these assumptions in mind, we can start estimating validator counts.
### Estimating validator counts
For each observation, we subtract the number of long-lived subnets $S_l$ from
all subscribed subnets $S_{all}$ to arrive at the number of short-lived
subnets $S_s$:
```math
S_s = S_{all} - S_l
```
Since we know aggregators are subscribed to one additional subnet per epoch,
$S_s$ will result in an estimated validator count for a certain beacon node in
this epoch. Note that just one observation will not be enough to get an accurate
estimate, because of the following reasons:
* It could be that a validator is not an aggregator for this epoch, and thus
won’t subscribe to any subnets
* There could be overlap between the long-lived and short-lived subnets
Due to this reason, we continuously try to collect observations for each known
beacon node per epoch, and save the maximum estimated validator counts. Note
also that the ceiling for validator estimations is at $64 - 2$, because that’s
the maximum amount of short-lived subnets we can record. This is important! It
means that for beacon nodes with more than $62$ validators, we can not estimate
how many there are, and just record the ceiling. We want to highlight again that
this is just an estimation and won’t be a very accurate representation of the
total number of validators.
## Architecture
In this section we’ll dive a bit deeper into the architecture. All the code for
this is open source and can be found in this repository:
[https://github.com/chainbound/valtrack](https://github.com/chainbound/valtrack).
A lot of the crawler code is based on projects like
[Hermes](https://github.com/probe-lab/hermes) and
[Armiarma](https://github.com/migalabs/armiarma/). An overview can be seen here:
### Crawler
The crawler is the core component of the system. It will crawl the discv5
discovery DHT, find nodes that are on the correct network by looking at the
metadata in their ENRs, and then try to connect with them. It will keep a local
cache of known peers and try to reconnect every epoch to get updated
observations.
We outline 2 types of events (observations): `PeerDiscoveryEvent` and
`MetadataReceivedEvent`. The second one is most relevant and contains the
following fields:
```go
type MetadataReceivedEvent struct {
ENR string `json:"enr"`
ID string `json:"id"`
Multiaddr string `json:"multiaddr"`
Epoch int `json:"epoch"`
MetaData *eth.MetaDataV1 `json:"metadata"`
SubscribedSubnets []int64 `json:"subscribed_subnets"`
ClientVersion string `json:"client_version"`
CrawlerID string `json:"crawler_id"`
CrawlerLoc string `json:"crawler_location"`
Timestamp int64 `json:"timestamp"` // Timestamp in UNIX milliseconds
}
```
Along with some metadata, this contains all of the fields required to apply the
previously described methodology: `SubscribedSubnets` contains the actually
subscribed subnets, obtained by listening on the GossipSub domain, and
`MetaData` contains the peer’s long-lived subnets.
All of these events are then sent to a persistent message queue, where they are
stored until they’re read by the consumer.
### Consumer
The consumer turns the event logs into a stateful view of the network by
implementing the methodology described above. It parses the short-lived subnets
from the metadata events to get the estimated validator counts, and updates any
existing entries in its stateful view. This stateful view is saved in a local
sqlite database, which we expose over an API. The table schema roughly looks
like this:
```sql
validator_tracker (
peer_id TEXT PRIMARY KEY,
enr TEXT,
multiaddr TEXT,
ip TEXT,
port INTEGER,
last_seen INTEGER,
last_epoch INTEGER,
client_version TEXT,
possible_validator BOOLEAN,
max_validator_count INTEGER,
num_observations INTEGER,
hostname TEXT,
city TEXT,
region TEXT,
country TEXT,
latitude REAL,
longitude REAL,
postal_code TEXT,
asn TEXT,
asn_organization TEXT,
asn_type TEXT
)
```
We then join this data together with an IP location dataset to provide more
information about geographical distribution.
## Results
[Chainbound](https://www.chainbound.io/) runs a
[https://github.com/chainbound/valtrack](https://github.com/chainbound/valtrack) deployment that pushes all data to Dune
every 24 hours.
*This data has been stripped of sensitive information such as IP addresses and
exact coordinates. However, it retains information like city, coordinates with a
precision of a 10km radius, and ASN information.*
**An example dashboard leveraging this information can be seen
[here](https://chainbound.grafana.net/d/fdry7answ53b4a/ulisse-v2?orgId=1\&from=now-6h\&to=now\&timezone=browser)**.
## Limitations
* The maximum number of validators we can estimate with this methodology per
beacon node is 62, due to that being the maximum amount of short-lived subnet
subscriptions. This will result in a significantly underreported total number
of validators, but should still be able to provide a reasonable estimation of
the geographical distribution.
* We failed to gather any meaningful data on Teku nodes over the 30-day period,
which could signify an error in our P2P implementation and impact the results.
* These results will be skewed towards validators attached to beacon nodes that
have opened P2P networking ports in their firewall, which will mostly be
beacon nodes running on cloud providers. The reason for this is that our
crawler can more easily connect to nodes that have exposed ports.
## References
* [https://eth2book.info/capella/](https://eth2book.info/capella/)
* [https://hackmd.io/@dmarz/ethereum\_overlays](https://hackmd.io/@dmarz/ethereum_overlays)
* [https://github.com/ethereum/consensus-specs/tree/dev](https://github.com/ethereum/consensus-specs/tree/dev)
# Exploring Verifiable Continuous Sequencing with Delay Functions
*Thanks to Conor, Lin and Swapnil from the Switchboard team, Cecilia and Brecht
from the Taiko team, Alex Obadia, Justin Drake, Artem Kotelskiy and the
Chainbound team for review.*
## Abstract
Agreeing on time in a decentralized setting can be challenging: wall clocks may
drift between machines, agents can lie about their local times, and it is
generally hard to distinguish between malicious intent and just unsynchronized
clocks or network latencies.
Ethereum can be thought of as a global clock that ticks at a rate of 1 tick per
\~12 seconds. This tick rate is soft-enforced by the consensus protocol: blocks
and attestations produced too early or too late will not be considered valid.
But what should we do in order to achieve a granularity lower than 12 seconds?
Do we always require a consensus protocol to keep track of time?
We want to explore these questions in the context of untrusted L2 sequencers,
who don't have any incentive to follow the L2 block schedule that is currently
maintained by trusted L2 sequencers, and will likely play various forms of
timing games in order to maximize their revenue.
In this article, we introduce mechanisms to enforce the timeliness, safety and
non-extractive ordering of sequencers in a decentralized rollup featuring a
**rotating leader mechanism**, without relying on additional consensus, honest
majority assumptions or altruism. To do so, we use three key primitives:
1. Client-side ordering preferences,
2. Ethereum as a global 12s-tick clock,
3. Verifiable Delay Functions.
Lastly, we show the case study of MR-MEV-Boost, a modification of MEV-Boost that
enables a variation of based preconfirmations, where the same construction
explored can be applied to reduce the timing games of the proposer.
## Rationale
Rollup sequencers are entities responsible for ordering (and in most cases,
executing) L2 transactions and occasionally updating the L2 state root on the
L1. Currently, centralized sequencers benefit from the reputational collateral
of the teams building them to maintain five properties:
* **Responsiveness**: responding to user transactions with soft commitments /
preconfirmations in a *timely* manner. We want to highlight that this
definition includes the timely broadcast of unsafe heads on the rollup
peer-to-peer network.
* **Non-equivocation (safety)**: adhering to preconfirmation promises when
submitting the ordered batch on the L1, which is what will ultimately
determine the total ordering of transactions.
* **Non-extractive ordering**: not extracting MEV from users by front-running or
sandwiching, or by accepting bribes for front-running privileges.
* **Liveness**: posting batches to L1 and updating the canonical rollup state
regularly.
* **Censorship-resistance:** ensuring that no valid transactions are
deliberately excluded by the sequencer regardless of the sender, content, or
any external factors.
In this piece we are concerned with how the first four properties can be
maintained in a permissionless, untrusted setting. Note that
censorship-resistance is ensured by construction: by introducing multiple
organizationally distinct sequencers in different geographies and jurisdictions
we have a strong guarantee that any transaction will be accepted eventually.
Consider a decentralized sequencer set $S := \{S_1,\dots,S_n\}$ with a
predictable leader rotation mechanism and a sequencing window corresponding to a
known amount of L1 slots. For simplicity, let’s assume $S_{i}$ is the current
leader and $S_{i+1}$ is the next one. At any point in time, only one sequencer
is active and has a lock over the rollup state.
Here are two strategies that sequencer $S_i$ can explore to maximize its
expected value:
**1. Delaying the inclusion of transactions**
Suppose a user sends a transaction to $S_i$ at a certain L2 slot. Then, the
sequencer could wait some time before inserting the transaction into a block in
order to extract more MEV with sandwich attacks in collaboration with searchers
or by directly front-running the user. In particular,
[since MEV grows superlinearly with time](https://www.youtube.com/watch?v=01dnINiLhAk\&t=287s),
it’s not in the sequencer’s best interest to commit early to a transaction. The
worst case scenario would be the sequencer delaying inclusion until the
sequencer rotation$^1$.
**2. Not publishing unsafe heads in the rollup peer-to-peer network**
In this setting the sequencer has low incentives to publish the unsafe heads in
the rollup network: since L2 blocks are signed by the sequencer (e.g. in
[Optimism](https://docs.optimism.io/builders/node-operators/configuration/consensus-config#p2psequencerkey)),
they act as a binding commitment which can be used by users to slash it in case
of equivocations.
This has a major downstream consequence on the UX of the rollup: both the next
sequencer and users need to wait until a batch is included to see the latest
transactions. For users it means they won't know the status of their
transactions in a timely manner, while the next sequencers risks building blocks
on invalid state.
We will now explore mechanisms to mitigate these behaviours and introduce
slashing conditions for sequencers.
## Primitive 1: Transaction Deadlines
We introduce a new EIP-2718 transaction type with an additional field:
* `deadline` - `uint256` indicating the last L2 block number for which the
transaction is considered valid.
This idea is not entirely new. For instance, the
[LimeChain](https://limechain.tech/) team has explored this in their
[Vanilla Based Sequencing](https://github.com/LimeChain/based-preconfirmations-research/blob/main/docs/preconfirmations-for-vanilla-based-rollups.md#preconfirmation-deadline)
article. However, in our variant the `deadline` field is signed as part of the
transaction payload and it is not expressed in L1 slots.
The reasoning behind it is that the sequencer cannot tamper with either the
`deadline` field or `block.number` (because it is a monotonically increasing
counter), and therefore it is easy to modify the L2 derivation pipeline to
attribute a fault in case the sequencer inserts the user transaction in a block
where `block.number > deadline`.
This approach mitigates problem #1. However, it does not in any way solve the
*responsiveness* issue, since sequencers can still delay proposing the block in
order to extract more MEV.
## Primitive 2: Ethereum as a Global Clock
A simple rotating sequencer design would be one where $S_i$ loses the power to
settle batches after the end of its sequencing window $W_i$, which is dictated
by an L1 smart contract. However, the sequencer still needs some time to post
the batch with the latest L2 blocks. We therefore introduce an *inclusion
window* that is shifted $n \geq 1$ slots ahead of $W_i$, where $S_i$ still
has time to land rollup batches on L1 with the last L2 blocks, even if the
responsibility of sequencing has shifted to $S_{i+1}$.
In case of any safety fault, the sequencer should be slashed. If the sequencer
has not managed to post all their assigned L2 blocks by the end of its inclusion
window, it will forego all associated rewards. Optionally, there could also be
penalties for liveness faults. This also helps with the problem of collaboration
with the next sequencer, by ensuring that the latest blocks will be known to it
within $n\cdot12$ seconds. Ideally, we’d like to keep $n$ as small as
possible with a value of $1$.
There are still some potential issues here: getting a transaction included on
Ethereum is probabilistic, meaning that you can’t be sure that a transaction you
send will actually be included in time. In this context it means that the last
batch sent by an honest leader may not be included in the L1 by the end of its
inclusion window. This can be helped with two approaches:
* A “based” setup, where the sequencer is also the L1 block proposer and can
include any transactions right up to the point they have to propose, or
* Using proposer commitments with a protocol like
[Bolt](https://boltprotocol.xyz). We expand more on this in the *”Further
work”* section below.
Note that we assume there is a registry smart contract that can be consulted for
the currently active sequencer, i.e. it implements some leader election
mechanism and takes care of sequencer bonds along with rewards and penalties. It
is up to the rollup governance to decide whether the registry can be fully
permissionless or if it should use an allowlist. In case of any misbehaviour,
governance would be used to temporarily or permanently remove the sequencer from
the allowlist.
## Primitive 3: Verifiable Delay Functions
[Verifiable Delay Functions](https://medium.com/iovlabs-innovation-stories/verifiable-delay-functions-8eb6390c5f4)
(VDFs henceforth) are a cryptographic primitive that allows a prover to show a
verifier that a certain amount of time was spent running a function, and do it
in a way that the verifier can check the result quickly.
For instance, consider a cryptographic hash function $h$ and define the
application
```math
H(n,s) := (h \circ \underset{n\ times}\dots \circ h)(s),
```
where $s$ is a byte array and $n$ is a natural number.
Composing (or chaining) hash functions like SHA-256 cannot be trivially sped up
using parallel computations, but the solution lacks efficient
verification$^2$, as the only way to verify the result is to recompute the
composition of functions. This solution appeared as a naïve VDF in
[Boneh's paper](https://eprint.iacr.org/2018/601.pdf), and for this reason it is
referred to as *weak*.
Another example of VDF is
[iterated squaring over a group of hidden order](https://people.csail.mit.edu/rivest/pubs/RSW96.pdf),
with which it is possible to construct time-lock puzzles. We’ll explore the
usage of the latter in the next sections.
### Why VDFs tho?
VDFs are very useful in the context of sequencing because they can act as a
*proof of elapsed time* for the duration of the block (specifically `block_time`
/ `max_adversary_speedup`, see *"Security Considerations"*). Consider the
following algorithm for the block production pipeline:
1. At the beginning of L2 block $N$, the sequencer starts computing a VDF that
takes an L2 block time (or slightly less) to compute for honest players,
using the previous block hash as its input.
2. After the end of the L2 slot the sequencer builds a block $B_N$ where the
header contains the result of the VDF, denoted $V_N$. We call this
*sealing* a block. This means the block hash digest contains $V_N$.
This algorithm has the nice property of creating a chain of VDF computations, in
some sense analogous to
[Solana’s Proof of History](https://solana.com/news/proof-of-history) from which
we inherit the security guarantees. What does this give us in the sequencer
context? If we remember that a sequencer has a certain deadline by which it has
to post batches set by the L1 slot schedule, we can have the L1 enforce that *at
least* some number of L2 blocks need to be settled. This has two downstream
results:
* The sequencer *must* start producing and sealing blocks as soon as their
sequencing window starts. Pairing this with the transaction deadline property
results in an upper bound of time for when a transaction can be confirmed. If
they don’t follow the block schedule set by the VDF and the L1, they risk not
being able to post *any* batch.
* We mitigate problem #2 by taking away the incentive to withhold data (not
considering pure griefing attacks): this is because the sequencer cannot
tamper with an existing VDF chain, which would require recomputing all the
subsequent VDFs and result in an invalid batch.
In general, for the sake of this post we will consider a generic VDF, provided
as a “black box” while keeping the hash chain example in mind which currently
has stronger guarantees against ad-hoc hardware such ASICs. See *“Security
Considerations”* below for more insights.
### Proving correct VDFs
If a sequencer provides an invalid VDF in an L2 block header it should be
slashed, and ideally we’d like to ensure this at settlement time. However,
recalculating a long hash chain on the EVM is simply unfeasible due to gas
costs.
How to show then that the number of iterations of the VDF is invalid? One way
could be to enforce it optimistically (or at settlement, in case of ZK-rollups)
by requiring a valid VDF chain output in the derivation pipeline of the rollup.
In case of equivocation in an optimistic rollup the sequencer can be challenged
using fraud proofs.
### Hardware requirements
Since by definition VDFs cannot be sped up using parallelism, it follows that
computing a VDF can be done by only using a single core of a CPU, and so it does
in our block production algorithm.
This makes it different and way more lightweight compared to most Proof-of-Work
consensus algorithms such as Bitcoin’s which requires scanning for a value such
that, when hashed with SHA-256, the hash begins with a certain number of zero
bits.
It’s also worth to note that modern CPUs are optimized to compute the SHA-256
hash function. Since 2016 Intel, starting with the *Goldmount* family of chips,
is offering
[SHA Extensions](https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sha-extensions.html)
in the *Core* and *Xeon* line-ups on selected models which introduces three new
instructions specialized in computing different steps of the hash function
algorithm more efficiently.
Lastly,
[single-core performance has stagnated over the years](https://www.man.com/single-core-stagnation-and-the-cloud)
indicating that there is a minor benefit in investing in the latest generation
of CPUs, thus lowering down the requirements of the system.
## Case Study: MR-MEV-Boost
[Multi-Round-MEV-Boost](https://ethresear.ch/t/based-preconfirmations-with-multi-round-mev-boost/20091),
is a modification of MEV-Boost that enables based preconfirmations by running
multiple rounds of MEV-Boost auctions within a single L1 slot. The usage of this
primitive is to output after each round a based rollup block built by L2 block
builders. As shown in the article, this approach inherits the L1 PBS pipeline
and mitigates some of the negative externalities of based preconfirmations as a
result.
Like MEV-Boost, this fork relies on the opted-in proposer to be an auctioneer
which ends the sealed auction by calling the
[`getHeader`](https://ethereum.github.io/builder-specs/#/Builder/getHeader)
endpoint of the relays. After having signed the sealed bid, the
[`getPayload`](https://ethereum.github.io/builder-specs/#/Builder/submitBlindedBlock)
is called by the proposer to receive the actual content of the winning bid and
to publish the block in the based rollup network.
In the original protocol, the end of the auction usually coincides with the end
of the L1 slot (more precisely,
[near one second after it](https://mevboost.pics/)); delaying it results in a
high risk of not being able to broadcast the block in time to gather all the
needed attestation and forgo all its associated rewards. As such, a block time
is proposed every twelve seconds with consistency, enforced by Ethereum
consensus.
In contrast, given it consists of multiple rounds happening *during* the slot,
in MR-MEV-Boost an *untrusted proposer is incentivized to end the auction
seconds later or earlier*$^{3}$ *according to the incoming bids,* in order to
extract more more MEV. In the worst case, MR-MEV-Boost will reflect L1 block
times. Another consequence of this is an inconsistent slot time for the based
rollup. This can be seen as a much more serious form of timing games.
In the article, the discussed possible solutions to this problem are the
following:
1. Introduce user incentives: if users determine that a proposer is misbehaving,
they stop sending transactions to said proposer.
2. Introduce a committee (consensus) to attest to timeliness and maintain slot
durations.
We now argue that a trustless solution that strongly limits the proposer without
requiring actions from the user does exist, and it leverages the same
construction we used for the VDF-powered block production algorithm in the
context of decentralized sequencing.
The construction is fairly simple and consists of computing a VDF that lasts
$x := 12/r$ seconds, where $r$ is the number of rounds in an L1 slot (the L2
block time). The proposer must calculate this VDF using the previous based
rollup block hash as public input and, at the end of the round, sending it along
with the body of a modified `getPayload` call. The output of the VDF is then
stored in the rollup block header and if invalid can result in slashing the
proposer after a successful fraud proof.
With this approach the amount of time a proposer can delay the end of a round is
limited: for instance if the first auction ended one second later then during
the last round it won’t be able to provide three seconds of computation for the
VDF but two, resulting in an invalid block and consequent slashing$^4$. This is
because in order to start computing a valid VDF, it requires the previous block
hash as its input, implying a sealed block.
## Security Considerations
**Are VDFs really safe for this purpose?** Suppose an adversary owns hardware
which is capable of computing the VDF faster compared to the baseline of honest
players *without getting noticed* (otherwise the number of iterations for the
VDF is adjusted by the protocol). Then, the faster the attacker
(`max_adversary_speedup`), the less our construction would constrain the space
of its possible actions. In particular, the sequencer would be able to commit a
bit later to blocks and be able to re-organize some of them for extracting more
value.
However, given we don’t need the “fast proving” property, hash-chains have
proven to be robust with Solana’s Proof of History and will continue to be at
least in the short-term. Also, our security requirements will not be as strict
as something that
[needs to be enshrined in Ethereum](https://ethresear.ch/t/statement-regarding-the-public-report-on-the-analysis-of-minroot/16670)
forever.
Some solutions and directions to get stronger safety guarantees can be found in
the *”Further work”* section below.
## Current limitations
**Sequencer credibility**
As with many new services which leverage (re)staking, the credibility of the
sequencer has an upper bound which is the amount it has staked: if a MEV
opportunity exceeds that, then a rational untrusted actor would prefer to get
slashed and take the MEV reward.
**Leader rotation can be a critical moment**
As discussed in the batcher and registry smart contract section, the inclusion
window is shifted of one slot forward at minimum compared to the sequencing
window. This is needed because of the time required to settle the last batch
before rotating leader, but leaves an additional slot time of at least 12
seconds in which the sequencer has room to re-organize the last L2 blocks before
publishing them on the rollup peer-to-peer network. As a consequence, liveness
is harmed temporarily because $S_{i+1}$ might be building blocks on invalid
state if it starts to sequence immediately.
Lastly, one additional slot might not be enough to settle a batch according to
recent data on
[slot inclusion rates for blobs](https://ethresear.ch/t/slot-inclusion-rates-and-blob-market-combinatorics/19817).
This can be mitigated by leveraging new inclusion preconfirmation protocols, as
explained below.
**Sequencer last-look**
Our construction makes very difficult for a sequencer to reorg a block after it
has been committed to, however it doesn’t solve front-running in its entirety.
In particular, the sequencer may extract value from users transactions while
building the block with associated `deadline` field. A possible solution along
with its limitations is explored in the section below.
## Conclusion
In this article, we explored mechanisms to enforce the timeliness, safety, and
non-extractive ordering of untrusted L2 sequencers in a decentralized rollup
environment. The primitives discussed ensure that sequencers can act more
predictably and fairly, mitigating issues such as transaction delays and data
withholding. Moreover, these techniques can reduce trust assumptions for
existing single-sequencer rollups, aligning with the concept of rollups
functioning as
[“servers with blockchain scaffolding”](https://vitalik.eth.limo/general/2024/06/30/epochslot.html#what-should-l2s-do).
These findings provide a robust framework for the future development of
decentralized, secure rollup architectures.
## Further work
**Trusted Execution Environments (TEEs) to ensure the sequencer is not running
an ASIC**
A
[Trusted Execution Environment](https://en.wikipedia.org/wiki/Trusted_execution_environment)
is a secure area of a CPU, often called *enclave*, that helps the code and data
loaded inside it be protected with respect to confidentiality and integrity. Its
usage in blockchain protocols is an active area of research, with the main
concerns being trusting the hardware manufacturer and the
[various vulnerabilities found in the past](https://en.wikipedia.org/wiki/Software_Guard_Extensions)
of some implementations (here’s the
[latest](https://x.com/_markel___/status/1828112469010596347)). Depending on the
use case these trust assumptions and vulnerabilities might be a deal-breaker.
However, in our setting we just need a guarantee that the sequencer is not using
specialized hardware for computing the VDF, without caring about possible
leakage of confidential data from the enclave or manipulation of the wall clock
/ monotonic clock.
**Adapt existing anti-ASICs Proof-of-Work algorithms**
The [Monero](https://www.getmonero.org/resources/about/) blockchain, launched in
2014 as a privacy and untraceable-focused alternative to Bitcoin, uses an
ASIC-resistant Proof-of-Work algorithm called
[RandomX](https://github.com/tevador/RandomX). Quoting their `README`:
> RandomX is a proof-of-work (PoW) algorithm that is optimized for
> general-purpose CPUs. RandomX uses random code execution (hence the name)
> together with several memory-hard techniques to minimize the efficiency
> advantage of specialized hardware.
The algorithm however leverages
[some degree of parallelism](https://github.com/tevador/RandomX/blob/102f8acf90a7649ada410de5499a7ec62e49e1da/README.md#cpu-performance);
it is an interesting research direction whether it can adapted into a
single-core version, leading to a new weak-VDF. This approach, while orthogonal
to using a TEE, can potentially achieve the same result which is having a
guarantee that the sequencer is not using sophisticated hardware.
**Time-lock puzzles to prevent front-running**
As mentioned in the *“Current limitations”* section, our construction doesn’t
limit the problem of sequencer front-running the users. Luckily, this can be
solved by requiring users to encrypt sensitive transactions using
[time-lock puzzles](https://people.csail.mit.edu/rivest/pubs/RSW96.pdf), as we
will show in more detail in a separate piece. However, this solution doesn’t
come free: encrypted transactions or encrypted mempools can incentive spamming
and statistical arbitrage,
[especially when the protocol fees are not very high](https://collective.flashbots.net/t/it-s-time-to-talk-about-l2-mev/3593).
**Inclusion Preconfirmations and Data Availability layers**
Batch submissions to an L1 contract could be made more efficient by leveraging
some of the new preconfirmations protocol like [Bolt](https://boltprotocol.xyz)
by Chainbound or
[MEV-Commit](https://docs.primev.xyz/concepts/what-is-mev-commit) by Primev to
have guaranteed inclusion in the same slot. In particular, sequencing windows
should end precisely in the slot before one where the proposer is running the
aforementioned protocols in order to leverage inclusion commitments.
Additionally, the batch could be posted into an efficient and lightweight Data
Availability layer run by proposers to enforce a deadline of a configurable
amount of seconds in the beginning of the slot, otherwise the sequencer would be
slashed.
## Footnotes
1. More precisely, if an operator controls multiple subsequent sequencers it
could delay inclusion until the last sequencer rotation.
2. In Solana, the verification of a SHA-256 chain is actually parallelised but
requires dividing a block associated to a \~400ms computation into 32 shreds
which are forwarded to the rest of the validators as soon as they’re
computed. As such, verification is sped up by computing the intermediate
steps of the hash chain in parallel.
3. In general, the proposer will end some rounds earlier as a side effect of
delaying other rounds. For example, it could force a longer last round to
leverage possible L1 \<> L2 arbitrage opportunities.
4. There is an edge case where the proposer might not be able to compute all the
VDFs even if honest, and it is due to the rotation mechanism: since the
public input of the VDF must be the previous rollup block hash, during
rotation the next leader will need some time before hearing the block from
the rollup network, potentially more than 1s. This could lead the next
proposer to be late in computing the VDFs. To reduce this risk, the next
proposer could rely on various parties to receive this information such as
streaming services and/or trusted relays.
# Vixy: Vibe-coding an Ethereum Node Proxy
At Chainbound we regularly run internal experiments. Since Claude launched Opus
4.5, it has been especially fun to push it beyond toy examples. In this
experiment, we tried to one-shot an Ethereum Execution Layer and Consensus Layer
JSON-RPC proxy.
We applied modern engineering practices as guardrails and wanted to see whether
Claude can ship a production-ready service quickly and without too much
hand-holding.
## Why build a proxy service from scratch?
We operate many execution and consensus clients as part of our infrastructure.
When nodes fall out of sync or stop responding, we need to detect that
automatically and fail over to a backup. This is different from load balancing:
we always want a single active node, and we want to switch only when something
is wrong.
We initially considered existing tools like
[blutgang](https://github.com/rainshowerLabs/blutgang) and
[sōzu](https://github.com/sozu-proxy/sozu). After some planning, it became clear
that they solved a much broader problem than we needed. We would have depended
on two external systems and used a small fraction of their features.
So we asked a different question: *what if we built a single binary that did
exactly what we needed, for both EL and CL, and nothing more?*
## Phase 1: Specification
We started by writing a precise spec. The goal was to remove ambiguity before
involving an LLM.
The service needed to:
1. Read a `config.toml` defining primary and backup nodes.
2. Continuously evaluate node health.
3. Expose HTTP and WebSocket proxy endpoints that always route to a healthy
node.
Health rules were explicit:
* **Execution Layer** Call `eth_getBlockNumber`, compare block heights across
nodes, track the highest one as the chain head, and compute lag. A node is
unhealthy if it lags more than a configurable number of blocks.
* **Consensus Layer** `/eth/v1/node/health` must return HTTP 200, and
`/eth/v1/beacon/headers/head` must expose a slot at
`/data/header/message/slot`. Track the highest slot and mark nodes unhealthy
if their lag exceeds a configurable threshold.
## Phase 2: Prompting the agent
The initial prompt lives in
[this PR](https://github.com/chainbound/vixy/pull/1).
Instead of asking Claude to immediately write code, we asked it to behave like
an engineer and produce a plan first. We created an `AGENT.md` file that
instructed the agent to think through the system before touching implementation.
One trick that worked particularly well was asking Claude to draw an
architecture diagram up front. This mirrors how we usually write internal design
docs: start with the big picture, then fill in details. The diagram gave us a
quick way to validate the overall approach and fix mistakes early, before the
agent committed to a direction.
Once the architecture made sense, we asked Claude to explain the system end to
end and break the work into phases. It produced a clear plan covering project
setup, configuration, health checks, proxying, metrics, and testing, with
explicit acceptance criteria for each step. From that point on, the agent could
work largely autonomously without drifting.
**This is where experience matters**. LLMs are good at writing code, but they
need constraints. Modern engineering practices act as guardrails. Without them,
things get messy quickly.
### Agent-oriented Test Driven Development
The core instruction was straightforward: write the tests first, then make them
pass.
Tests were treated as the source of truth. When Claude claimed something worked,
the only thing that mattered was whether the tests passed. When code was
refactored, correctness was defined entirely by green tests.
By the end, Claude had written 85 unit tests. Every feature started red, then
went green, then got cleaned up. This caught real bugs: in one case, unreachable
nodes were incorrectly marked healthy due to an edge case in the lag
calculation. The failing test exposed it immediately, the logic was fixed, and
we moved on.
We also used BDD (behavior-driven development) with
[Cucumber](https://github.com/cucumber-rs/cucumber). The scenarios read like
plain English specs: “Given an EL node at block 1000, when the health check
runs, then it should be marked healthy.” There was no ambiguity, so Claude
consistently implemented exactly what the scenario described. We ended up with
33 scenarios that function as both executable tests and up-to-date
documentation.
Unit tests alone are not enough, though. Eventually you have to hit real
infrastructure.
For that, we used [Kurtosis](https://github.com/ethpandaops/ethereum-package) to
spin up a local Ethereum network with four EL nodes and four CL nodes. We ran 16
integration tests against it. This surfaced issues unit tests could not catch,
such as missing `Content-Type` header forwarding, which caused Geth to return
HTTP 415. Another annoying bug with a trivial fix, caught long before
production.
### Context rot and the diary pattern
LLMs do not have durable memory. As context grows, earlier details get
compressed or lost. To deal with this, we asked Claude to maintain a `DIARY.md`
file throughout development.
Each entry followed a simple structure:
* What was done
* Challenges encountered
* How they were solved
* Key takeaways
* Mood
The diary turned out to be surprisingly effective. When the Content-Type bug
appeared during integration testing, the diary captured the failure, the
debugging process, and the fix. When we implemented WebSocket reconnection
logic, it documented the design decisions, tests, and type system issues
involved.
When Claude's context inevitably got compacted after hours of work, it could
reread the diary and immediately regain situational awareness. Without it, the
agent would sometimes suggest approaches we had already tried and rejected. It
became a knowledge base.
“Why did we choose approach A over B?” The answer was always in the diary.
### Break work down, commit constantly
We also enforced small, concrete tasks. “Build the proxy server” was decomposed
into steps like parsing requests, extracting JSON-RPC methods, and selecting a
healthy node. Claude always knew what it was working on next.
Finally, we required a commit after every completed phase. CI ran on every
commit: formatting, Clippy, unit tests, and BDD scenarios. If anything failed,
work stopped until it was fixed. This prevented the classic “I'll clean it up
later” failure mode. By the end, we had more than 60 commits, each passing CI
and usable as a rollback point.
## Phase 3: Implementation
With the plan and guardrails in place, we mostly stayed out of the way.
Claude wrote tests, implemented features, ran CI, and committed. We stepped in
to clarify requirements or review decisions, but the agent did most of the work.
**It felt less like using a tool and more like pairing with someone who is
extremely fast at execution but relies on you for judgment calls.**
We still hit issues like health check edge cases, Axum 0.8 route changes,
WebSocket type mismatches and more, but the difference was that none of them
lingered. Tests failed, the agent fixed them, and work continued.
## The result
We shipped [Vixy](https://github.com/chainbound/vixy), a production-ready
Ethereum proxy that monitors node health, handles automatic failover, proxies
both HTTP and WebSocket traffic, and exposes status and metrics endpoints. Here
are some numbers from the experiment:
* 85 unit tests
* 33 BDD scenarios (147 steps)
* 17 integration tests against real Ethereum nodes
* 60+ CI-passing commits
* \~4,400 lines of Rust
## What we learned
Three takeaways stood out:
1. LLMs are force multipliers, not replacements. We made the architectural
decisions and set quality bars. Claude handled the execution. Think of it as
a very fast engineer who needs clear direction and a plan.
2. Guardrails are mandatory. TDD, BDD, integration tests, small tasks, frequent
commits, and written context are what made this work. Without them, results
would have been unpredictable.
3. Good specs unlock autonomy. A clear architecture diagram and phased plan let
the agent work productively for hours without constant steering.
***
**Repository:** [https://github.com/chainbound/vixy](https://github.com/chainbound/vixy)
**License:** MIT / Apache-2.0
## P.S.
Special shoutout to [Mert](https://x.com/mert) for the architecture diagram tip,
and [Lwastuargo](https://x.com/lwastuargo) for reinforcing that guardrails are
everything.
## Background
In
[**Introducing FlowProxy**](https://collective.flashbots.net/t/introducing-flowproxy/5341),
[Chainbound](https://chainbound.io/) collaborated with Flashbots to deprecate
its [original orderflow proxy](https://github.com/flashbots/buildernet-orderflow-proxy) implementation
in Go in favor of a new one built in Rust, created with the goal of reducing
end-to-end latency, improving efficiency, and increasing observability in
[BuilderNet](https://buildernet.org/)'s networking layer. This first
collaboration aimed at bringing it to production-ready quality and ready for
deployment, already observing good improvements in networking and processing
latency.
This second collaboration expanded on the *Next Steps* section outlined in the
previous report, with success. It includes a quantitative analysis of order
losses and latencies across BuilderNet, and extensive work to improve the
transport layer to reduce both CPU and memory usage. We encountered a couple of
non-obvious learnings that we outline below, and we hope this will be useful to
the community.
## Network Analysis
FlowProxy features a Clickhouse integration which makes possible to record
*bundle receipts.* A bundle receipt consists of a short summary that includes
the timestamp of when the bundle was sent and received by instances, along with
its raw size in bytes (useful for tracking size → latency correlation). We used
this data to better understand FlowProxy’s network performance in its entirety,
and to discover what improvements to focus on.
### Bundle Loss
We started with analyzing bundle loss between instances. The table below reports
the percentage of bundle loss on individual links, during a small time range of
congestion:
| src | dst | lost | total | loss\_rate\_pct |
| ---------------------- | ---------------------- | ---- | ------ | --------------- |
| beaver\_eastus\_07 | nethermind\_we\_08 | 7857 | 35848 | 21.92% |
| beaver\_eastus\_07 | flashbots\_we\_09 | 1969 | 96597 | 2.04% |
| beaver\_eastus\_07 | beaver\_we\_08 | 1354 | 94000 | 1.44% |
| beaver\_eastus\_07 | flashbots\_eastus\_10 | 2 | 140801 | 0.00% |
| beaver\_eastus\_07 | nethermind\_eastus\_07 | 1 | 141534 | 0.00% |
| nethermind\_eastus\_07 | flashbots\_eastus\_10 | 10 | 35315 | 0.03% |
| nethermind\_eastus\_07 | beaver\_eastus\_07 | 10 | 35377 | 0.03% |
| nethermind\_eastus\_07 | flashbots\_we\_09 | 6 | 34560 | 0.02% |
| nethermind\_eastus\_07 | beaver\_we\_08 | 6 | 35216 | 0.02% |
| nethermind\_eastus\_07 | nethermind\_we\_08 | 4 | 34490 | 0.01% |
| flashbots\_eastus\_10 | beaver\_we\_08 | 4 | 34877 | 0.01% |
| flashbots\_eastus\_10 | flashbots\_we\_09 | 4 | 34266 | 0.01% |
| flashbots\_eastus\_10 | nethermind\_we\_08 | 3 | 34156 | 0.01% |
| flashbots\_eastus\_10 | beaver\_eastus\_07 | 1 | 35054 | 0.00% |
## Background
The [orderflow proxy](https://github.com/flashbots/buildernet-orderflow-proxy)
is a central component in BuilderNet. It’s responsible for receiving bundles and
transactions from users, and ensuring those orders are shared with all other
instances in BuilderNet, as well as to the local
[rbuilder](https://github.com/flashbots/rbuilder). Performance is an important
aspect here: the faster these proxies can share orders, the more effective the
builders will be able to collaborate, and the more blocks BuilderNet as a whole
will win. If a proxy goes down for even a couple of seconds, that could mean
1000s of lost orders, so reliability is crucial too.
Taikoscope subscribes to events from both L1 and L2 RPCs, and on each L1 head,
and performs the needed contract calls. Data is inserted into ClickHouse and
queried from the Dashboard via an API.
Taikoscope is developed entirely in Rust, aligning with Chainbound’s
performance-oriented engineering ethos.
🔗 **Dashboard**: [taikoscope.xyz](https://taikoscope.xyz/)
🧑💻 **Code**:
[github.com/chainbound/taikoscope](https://github.com/chainbound/taikoscope)
Taikoscope is funded through Taiko’s grant program and MIT licensed.
Contributions are welcome!
# Linkem: Building a Rust Network Emulator from Scratch
## Introduction
When we started working on [`msg-rs`](https://github.com/chainbound/msg-rs) we
ran into the necessity of emulating real networks with latency, jitter, packet
loss, and bandwidth constraints. We needed a way to test under realistic
conditions without setting up actual distributed infrastructure.
**Linkem** is what we ended up building. It's a Rust library that creates
isolated network peers using Linux namespaces and injects network impairments
between them, fully leveraging the kernel TCP/IP stack for modelling traffic.
This article documents how it works and the decisions we made along the way.
## Existing Tools and the Gap
Before building something new, we looked at what already existed. Network
emulation is a solved problem in many contexts, depending on your requirements.
**tc** is the foundation everything else builds on. The Linux kernel's traffic
control subsystem can add delay, loss, bandwidth limits, and more to any network
interface. The problem is that it operates at the system level. You run `tc`
commands to modify interfaces, and those changes affect everything using that
interface. There's no isolation between tests, no easy way to create multi-peer
topologies, and integrating it into a Rust test suite means shelling out to
external commands.
[**Mininet**](https://github.com/mininet/mininet) and its fork
[**Containernet**](https://containernet.github.io/) are Python-based network
emulators popular in SDN research. They can create complex topologies with
switches, routers, and hosts. Containernet extends this to use Docker containers
as hosts. These are powerful tools, but they require defining your topology
upfront in Python, spinning up the emulated network, and then running your tests
inside it. The workflow is "create topology first, then run code inside" — the
opposite of what we wanted for Rust integration tests.
[**Toxiproxy**](https://github.com/Shopify/toxiproxy) from Shopify takes a
different approach: it's a TCP proxy that sits between your application and its
dependencies. You configure "toxics" (latency, timeouts, bandwidth limits) via a
REST API. It's great for testing how your app handles a flaky database
connection, but it's not network emulation — it's application-layer proxying.
Your code has to connect through the proxy, which means changing connection
strings. It also can't simulate things like packet loss at the IP level or test
UDP-based protocols.
[**Estimator**](https://github.com/commonwarexyz/monorepo/tree/main/examples/estimator)
from Commonware is a great tool for testing prototypes of consensus protocols,
simulating peers distributed across specified AWS regions with real-world
latency/jitter in virtual time. It features a handy DSL to create different
scenarios and replay them deterministically. Its main downside is the limited
scope: it's designed for consensus protocols, not general-purpose network
testing.
[**Pumba**](https://github.com/alexei-led/pumba) is a chaos testing tool for
Docker containers. It can kill containers, pause them, or use `tc` to inject
network faults. But it operates on running containers from the outside — you
define your Docker Compose setup, start it, then point Pumba at containers to
disrupt. Like Containernet, the topology exists before your test code runs.
[**netsim**](https://github.com/canndrew/netsim) is the closest to what we
needed. It's a Rust library that uses Linux namespaces to isolate network stacks
and lets you run async code inside them. However, it doesn't provide the
per-destination impairment control we needed. When Peer A sends to Peer B versus
Peer C, we wanted different latency and loss characteristics — netsim's model
didn't support that out of the box.
### Using the Kernel’s TCP/IP stack
There's another issue with most of these tools: they operate at different layers
of the stack and aren't true kernel-level network emulation. This matters when
you want to test things like:
* *TCP buffer tuning* (`tcp_rmem`, `tcp_wmem`) — how do different buffer sizes
affect throughput on high-latency links?
* *Window scaling* — is your system correctly configured for leveraging TCP
window scaling on high bandwidth-delay product links?
* *Congestion control algorithms* — how does BBR compare to CUBIC under packet
loss?
You can't answer these questions with application-layer proxies. The traffic
never goes through the real kernel networking stack in a meaningful way. What we
wanted was an environment as realistic as possible, where the kernel's TCP
implementation, buffer management, and congestion control are all part of the
test.