Since the launch of Filecoin Storage Auctions, the Textile team has been continuously collecting feedback and use-cases from our growing community of Data Producers and Storage Providers. This feedback has helped us to keep iterating on the Auctions stack, as well as our bidbot system.

bidbot, and the Storage Auction system that it interfaces with, is driven by the idea that by tightly coupling Filecoin economics with the deal making process, we can generate a cheaper and more efficient storage pipeline for everyone.

For those not already familiar, bidbot is a sidecar daemon that Storage Providers can run to allow them to watch for Auctions (via libp2p), and bid on them. Storage Providers need bidbot running to participate in Auctions. Once a Storage Provider has bidbot running and connected to their storage infrastructure, it automatically bids on Auctions (based on pre-configured parameters), and upon winning (and a storage proposal being accepted by the markets node) bidbot downloads the CAR file from the source and imports it into their Lotus miner. You can learn all about bidbot and how it works in the project README.

Today, we are excited to announce a series of new features and optimizations that will make running bidbot and interfacing with the Auctions system even easier and more powerful.

Controlling throughput

Running a Filecoin Mining operation is no easy task. A Storage Provider's infrastructure needs to be stable enough that it can deal with constant fluctuations to their incoming deal pipeline. With this in mind, we developed a series of options to help with controlling throughout.

• The --running-bytes-limit flag allows bidbot users to govern the total bytes from the Auction system that will be processed, from initial CAR file download right up until they finish importing the deal.
  The check for the limit happens at the point when bidbot wins a bid, which means a Miner will still bid in the Auctions, but they'll reject the winning bid if it would exceed the limit, without penalty. This design avoids missing out on potential deals while still maintaining the limit.
• While the above flag only takes the deals from auction into account, the --sealing-sectors-limit flag is aware of all deals, offline or online. If the Miner's node has more than X sectors, bidbot stops bidding to avoid exceeding sealing capacity.
  Bear in mind that it takes some time from accepting a deal to the point of sealing, so it's advisable to use --running-bytes-limit to handle surges from bidbot, while also using this flag to manage overall load.
• In some cases, importing one or two files at the same time is pretty fast but importing more can take like forever. If you see that happening, you can limit the import concurrency using the  --concurrent-imports-limit flag.

CID Gravity integration

Even with the above configuration flags in place, Storage Providers might still prefer intuitive Web UIs over config files to adjust pricing rules, or they might even want the ability to manage their operation while sitting on the beach 🏖️! That's where our CID gravity integration comes into play. You asked, and we delivered, and now managing bidbot with CID gravity is as easy as clicking "connect" on the CID Gravity console, coping the generated key as the ---cid-gravity-keyflag when running bidbot, and you are all set! Optionally, you can use ---cid-gravity-strict to control the fallback behavior for if/when the CID Gravity server is temporarily unreachable.

And one more thing... if you have CID gravity integration enabled, the deal proposal rate you configure in CID Gravity gets automatically applied to bidbot, giving you one more way to control throughput!

Automation

Many Storage Providers like to create scripts and bots to help fine tune their mining operation in real-time. So in an effort to provider greater flexibility to bidbot users, we've added a few useful command for controlling bidbot behavior. Use whichever scripting tool you have at hand, whether it's shell, python or PHP, you can just call bidbot pause when you see more than expected load, and bidbot resume when the load comes back down to normal. Easy!

Run the daemon like a demon

While commands like nohup make it easy to keep bidbot running even after you log out, many Storage Providers would prefer to run bitbot along with their system tools. While crafting a systemd service file is pretty straightforward, sudo bidbot install-service makes it even easier, by setting up a basic service that you can tweak!

If you prefer to run bidbot as a Docker container, pull the latest image and configure it with environment variables. For any command line flags, you can simply specify them in ALL_CAPS (replacing the hyphens with underscores), and prepend BIDBOT_. For example, to configure --storage-provider-id=f011 as an environement variable, you would use BIDBOT_STORAGE_PROVIDER_ID=f011.

Community contributions

Have some ideas for making bidbot even better or easier to use? Want to improve the documentation? We welcome contributions! Huge thanks to cerblue who submitted a fix for separated markets and mining nodes, and another big improvement to avoid downloading files for deals already exceeding the start epoch.

Winner selection

Winner selection logic is core to the Auction system, which is why we carefully iterate the winner selection algorithm based on community feedback. Anyone can join the monthly Auction Governance meeting to contribute. Currently it is a multi-track system. For example, if the storage request requires three replicas, then...

• The winner for the first replica is chosen from the top 1/5 Storage Providers with the lowest deal failure rate over the proceeding week.
• The second replica goes to the 1/5 of Storage Providers who have won the least number of bids in the last week.
  This means that a newly joined Storage Provider has a higher chance of winning a few Auctions early on, allowing them to move into the top 1/5 candidate pool to grab those first replica deals — assuming they are able to confirm their deals on chain in time!
• The rest of replicas are chosen at random.

The logic described above is subject to change, but any future changes will be decided and built out in the open, and you can see exactly why each winner is selected via the public dashboard.

Speaking of winners

The current season of development at Textile ends in mid-November, and by that time we will have stored over 200 TiB and made over 5000 deals on the Filecoin network. Powering that throughput is a dedicated group of Filecoin Storage Providers that run bidbot to connect to the Auctions network. To celebrate those Providers, we are offering up some cold hard Filecoin as part of an end-of-season competition. Let's see who is providing the highest quality storage service! All Filecoin Storage Providers participating in bidbot are eligible! To learn more and get involved, join our dedicated Discord channel. Good luck!

Interested in running bidbot? It's easy to get started!