Storage legos: an update on Filecoin deal auctions

filecoin • Jul 26, 2021

TL;DR: Auctions are live and adding efficiency to data storage workflows. We started governance calls to invite the community into the design process of auctions. We are adding new features and onboarding new use-cases every week.

Filecoin auctions aim to drive better deal success, speed, and features through economics. Unlike the default deal-making flow on Filecoin, auctions bring miners to the deal stream and encourage them to compete, in real-time, for storage deals. In an emerging market like Filecoin, auctions make it easy for anyone who needs to store data to get great results.

Keep it simple: five parameters and five steps

Auctions are produced by clients using only five required parameters (current data location, payload CID, piece CID, byte count, and replication factor). Including preparation of that payload, the whole process can be broken down into five easy steps:

A client prepares and submits an auction payload.
The auction is published to the miner pool.
Miners select deals matching their criteria and submit bids.
The auctioneer notifies the winning miners.
Miners pull the data and an offline deal is finalized.

What is really cool is that everything after preparation (Step 1; see the first of many tool releases to help with this step) is automated for both the clients and the miners. On the client-side, this decouples the client from any difficulties with network price evaluation, miner selection, or deal negotiation. Clients just create deals and get great storage for a great price.

Miners also benefit from the setup. The bidbot tool can be setup and left to run along-side their mining operation. The bidbot will submit bids automatically and process deals that in wins. Early feedback has been overwhelmingly positive and helpful (thanks miners 👋).

The bidbot process is extremely convenient and low touch. Unlike an online deal or regular deals that take a lot more care and feeding... It's awfully convenient to just come in and see a bunch of stuff sealing because it made the auction. —f01278

Early results: all systems go

System metrics have been promising. At the time of writing, we've successfully stored 49 deals with miners with an average price of 0 FIL and a median price of 0 FIL 😋. No auction has had an outright failure and in a handful of cases, miners have not managed to get a deal on chain before the agreed-on start-time. In those rare cases, the replica is immediately distributed to miners via a new auction.

The early pool of miners are all actively submitting bids as fast as we can generate deals. This level of real-time competition ensures that data is stored as cheaply as possible. Clockwise from top left: "New auctions rate" shows a rapid up-tick as one of our direct-to-auction users starts firing new auctions; "Finalized auction rate" sees an uptick that matches new auctions within about three seconds (time to finalize an auction); "Peering miners" shows some of the heartbeats on the auction pubsub channel; "Daily accepted bids by miner" shows how many valid bids are coming through the system; "Daily bids by miner" are the total bids sent, the higher number indicates that some miners are submitting bids outside the accepted range (e.g. proposing sealing times that are too long) or multiple bids for the same auction (possibly changing their price); "New bids rate" just shows the rate at which we are receiving bids from all miners.

The Auctioneer (technically called the "Dealer" at this stage of the system) watches the progress of deals post-auction. Fifteen won auctions are still awaiting miners to confirm successful data transfer. Another eight are going through the steps to finalize the deal on-chain.

Development: here's what we have on deck

Winner algorithm. Right now, the winning algorithm is simple, best price wins. In cases of ties, the first received wins. This was the major topic of our last Governance Call (more below) and we'll be improving that algorithm pretty quickly.
Auction metrics. The above data is too cool! We're working to make it all available over open APIs shortly.
Remote client signing. The live auctioneer handles all deal signing for clients but we have scoped and will be moving toward an API that allows remote Filecoin address owners to submit auctions and sign their own deals.
We'll also continue to open source the layers of the stack as they become ready for others to test out.

The future: auction governance and you

Auctions are already solving a handful of issues for clients, miners, and the network at large. We believe that auctions will be an important building block on Filecoin for a long-time to come and are excited to broaden our development to capture more use-cases and workflows. With both impact and relevance in mind, we've launched a bi-weekly Auction Governance call that all are free to join. Read the notes from our last call, add future calls to your calendar, and see future agenda topics all here.

Action items: start using auctions in your data storage plan

While we haven't launched a public interface to auctions yet, we are incubating a small number of projects that are already using them to store their data. If you'd like to be on the list of next projects we onboard, here is what we are looking for:

Greater than 10TiB of data to store.
Public and liberally licensed datasets.

If that's you, DM us on Twitter or on Slack.

Each week that goes by working on storage auctions has gotten us more and more excited about connecting the pipelines of data to the Filecoin storage network. We can't wait to see where this goes!