Introducing Storage Auctions on Filecoin: First Releases from Textile’s Upcoming Auctioneer Stack

Textile is excited to share our work to make Filecoin adoption easier, cheaper, and faster for new storage users. Today, we are open-sourcing two new libraries in a stack to enable Filecoin storage deal auctions. Storage auctions, or auctions for short, help move high-throughput streams of data to Filecoin miners. Auctioneers contain several unique innovations that enhance the user experience. At the heart of the auction system is a new model for storage auctions, where miners compete to store the deal feed from an auctioneer, assuring low prices and eager miners. Let's take a look.

Default storage mode on Filecoin

The default flow for creating storage on the Filecoin network is what is called an online deal. Online deals are storage on demand. When a client decides to store some data, they pick a miner, negotiate a deal, send the data, and get on-chain confirmation.

Online deals pose a few challenges to large-scale Filecoin clients and the miners they choose to make deals with. Clients themselves need to identify the miners they decide to send their deals. This means that Miners are potentially subject to spikes in demand from clients, and don't have many means to increase or decrease that deal flow to match their hardware constraints.

To help solve the challenge of creating high-throughput, high-success, high-value deals on Filecoin, we are releasing a new format for deal-making on the network, auctions.

Introduction to auctions

Textile has been developing a new model of creating deals on Filecoin through auctions. Filecoin auctions allow miners to bid on open storage requests in real-time. The miner-driven bidding has multiple direct benefits, including giving them the power to adjust bidding frequency and aggressiveness (e.g., price) based on their current infrastructure load. They also can be more selective on the storage requests that match their general capabilities. For example, in online deals, the speed at which a miner can prove a deal on-chain may lead to failed deals that never would be selected by a miner in the auction setup.

For clients, there are a few main benefits of using auctions for storing their data. The most obvious is cost. With miners more directly competing for every deal based on cost and reputation, the client will get better final prices for storing data (currently often $0 due to block rewards). We also think that auctions can provide a higher-quality user experience for heavy storage users by helping to organize miners in direct competition for their data.

Who's using Filecoin auctions?

To date, we have two primary groups of users. First are external blockchains. We originally devised the auctions to help scale the storage requirements originating in smart contracts on NEAR, Ethereum, and other blockchains (more on that in tomorrow's blog post). The second group of users is what we call direct-to-auction users. These users, primarily large-scale data storage users, are already storing data on Filecoin and are looking for more efficient ways to move data to miners.

How auctions work

The auction process is similar in either case.

1. Filecoin CAR files are packaged and prepared for storage on the network.
2. That CAR file is wrapped, with metadata, in an auction payload.
3. The auction payload is published over a pubsub channel containing miners awaiting new auctions.
4. Miners submit bids to compete for the deal contained in the auction.
5. After the auction completes, the winning miners are selected from the available list of bids. The full lists of bids, winning miners, and selection criteria (e.g. location filters, reputation penalties, etc) are published publicly. Later, we will begin anchoring the auction feed to the Filecoin chain itself. The auction algorithm itself is likely to change as we learn how miners organize and compete in the system.
6. The final storage deal parameters are proposed and accepted by both parties.
7. The miner pulls the data and finalizes the deal on-chain.

There are several exciting technologies developed to make this process smooth for clients and miners alike. Today, we are open-sourcing the first of them.

Release: A lightweight file packaging utility

Obtaining high-throughput, cheap storage on Filecoin depends a little bit on how you package your data for storage. The smallest files alone won't be picked up by miners, they aren't currently incentivized to store files under a specific size. Medium-size files are also less desirable than files that fill most or all of a sector on Filecoin. When possible, clients are packing their data into files that fill most or all of a sector, ensuring that a high number of miners are interested and incentivized to store the data. Today, we are releasing a new tool in Powergate that allows you to generate and prepare these batched CAR files quickly and with minimal dependencies (i.e., no need to run IPFS). Read more how to use the library here: https://docs.textile.io/powergate/offline/#filecoin-batching.

Release: A miner utility to automatically compete in auctions

Filecoin nodes don't currently contain a method for miners to connect or bid on auctions. For that, we've implemented a daemon that miners are running on their infrastructure called bidbot. The bidbot daemon is designed to run on a different machine than the miner's Filecoin node. Isolation away from the Filecoin node is important to miners for scaling and security reasons. To setup bidbot, a miner first signs a message from their miner address that confirms a bidbot is under their control. After the initial proof, the bidbot can make signed bids from the known miner identity (without needing private keys). Removing access to private keys is an important security implementation. Finally, bidbot simplifies the handling of won auctions. By running the daemon, miners quickly download and begin processing the data to report on-chain. Explore the code here: https://github.com/textileio/bidbot.

Using Storage Auctions

We'll continue to release layers of the Storage Auction Stack over the summer. We are testing it now with high-throughput clients. If you are a Filecoin user that would like to test out the system to see how it can help your storage pipeline, get in touch here.

We will have more exciting announcements and releases tomorrow, so follow us on Twitter to stay up-to-date. You can also find us on Slack.

We're hiring! See our open positions here.