Filecoin tools progress update: 8 April

filecoin • Apr 08, 2020

We are half way between the time Textile began our work on Filecoin tooling back in January and Mainnet launch this summer. In this short period of time, we've built a lot of very cool things and learned a lot about the internals of the Filecoin network. I thought it would be fun to depart from the normal format of our updates today, and share with you instead of rundown everything we've built to date. If you are sensitive to vertigo, you may want to read one of our other blog posts first 🚀.

A rapid tour of the Powergate features

Let's start with some of the core features in the Powergate and how they are organized and what they are capable of doing. We'll focus on three main features:

  • how the Powergate handles deal creation with/for users;
  • how the Powergate helps users manage long-term deal health;
  • and how the Powergate splits storage into persistant storage (cold) on Filecoin and network available storage (hot) on IPFS.

Storing New Data

Storage management and deal making are core-competencies of the Powergate. Without getting into too much detail here, the Powergate allows a system to provision new wallet addresses and configure settings per address for how to create deals and how to manage IPFS availability. A configuration would include: default deal duration, replication factor, blocked miners and more (we'll cover more details below). Default configuations can be created that will be used for all deals, or new configurations can be created for any single deal.

Here's a rundown of some of the deal management features in the Powergate:

  • Storage Deals
    • Interactive deals
      • Ability to create interactive deals with a set of miners.
      • Filter & select miners manually.
      • Address based configs for matching miner attributes, setting replication factors, etc.
      • Integration with Reputation indices for selection.
    • Automated deals
      • Ability to initiate deals over API.
      • Ability to track progress of deals through the stages.
      • Use config for matching miner attributes, setting replication factors, etc.
    • Storage config (hot and cold)
      • Cold storage
        • Enabling.
        • Excluded Miners.
        • Replication Factor.
        • Country settings.
        • Duration.
        • Renewal setting.
        • Unfreeze (Filecoin → IPFS data flow)
      • Hot storage (IPFS) config.
        • Enabling.
        • Add timeout.
      • Hot and Cold storage are extensible interfaces.
    • Retrieval available per CID.

Long-term deal management

The next interesting feature area of the Powergate is long-term deal management. This work is handled by a system called, the Scheduler. The Scheduler uses the configuration attached to the deal (see above) to know how the deal owner want a deal to be monitored and maintained. The Scheduler will then continuously monitor deals for upcoming expiration (and start renewal) or slashing (and repair based on replication). There is so much cool stuff here... too much to unpack here today, but here's a sample:

  • Repair & Monitoring
    • Scheduler implemented. Enforces Cid configurations (aka: desired state of Cid storage).
    • Deal renewal is handled by the Scheduler and can be changed using the deal config.
    • Renew
      • Blocked while the logic is in place and runs on master, if you attempt to make two deals with the same miner, it fails. (testnet/3, see here)
    • Repair is also handled by the Scheduler, and have two flavors:
      • Triggered: When a new Cid configuration is pushed, it will do whatever work is necessary to enforce that state considering the current state. e.g: if the replication factor is increased from 1 to 3, it will make two new deals.
      • Automatic: Pending, but will be the same logic of the Triggered flavor just re-pushing the last known Cid configuration.

All of this and more can be specified in a declarative way using the config setup described below. Updates can even be made a pushed from the command line or API after the fact!

Filecoin storage, IPFS availability

One of the main reasons people will use the Powergate instead of running only a Lotus node (one of the main Filecoin implementations) is for the Powergate's ability to store data on Filecoin and make it available on the IPFS network. The internal systems refer to these two layers as the cold storage (Filecoin network) and hot storage (IPFS network), and deals created in the Filecoin can leverage the two in different ways depending on needs. For example, data owners can create (and update) configuration of cold storage unfreezing settings and hot storage expiration settings. Takign a look at the config options will give you a great sense for how powerful this system is going to be. See Deal Config and be sure to drill down into Fil and IPFS configs. Remember, these configs can be set per deal!

How people will use the Powergate

Powergate Devops Tools 🎛️

The powergate is meant to build with. So we started shipping the code with some helpful devops tools for any admin to monitor and understand the system better.

All of that ships as part of a docker-compose setup. So a simple docker-compose up will get users Powergate, Lotus, and all the above goodies connected together. While we are still a few weeks away from encouraging you to jump in and run your own, it's not far away 💪.

  • Automatic configuration the the same docker-compose setup

CLI and APIs 🍹

The Powergate has some really nice features for developer experience. All the APIs for example, are exposed as gRPC services. This will allow clients to be quickly developed and upgrades very smooth. The first client, the command-line tool, gives the Powergate owner a handy way to get their hands dirty with minimal setup.

Public APIs & Data 🌈

Every Powergate can also contribute to globally available, off-chain, APIs. Data served over these APIs can help users make better decisions about deals, monitor network health, or simplify network onboarding. Some example indices the Powergate publishes include: slashing, miners, asks. Also, since the beginning of the project, we've been publishing nightly snapshots of the chain to help other nodes bootstrap more quickly: https://lotus-archives.textile.io/.

Documentation for users and system integrators is still to come, but if you are itching to explore what the system has to offer internally, the technical documentation is 🤩: https://godoc.org/github.com/textileio/powergate. There is additional bonus material in the project Readme and in some of the design docs.

Broader Network Contributions

We've been on the forefront of development for using Filecoin storage. When we've found challenges, we try to solve them in ways that will help more projects ramp up faster. A cool example of this is in the Lotus Devnet: a contanerized Lotus devnet using a mocked sectorbuilder. This code allows us to launch the entire system (Powergate, Lotus, etc) on a self-contained network that will skip some of the hard steps in deal storage. The devnet allows us to launch the Powergate in a fully functional network that is resource thrifty where we can run tests, poke at APIs, or prepare integration. There are a couple of other big pieces we'll polish up and share in our future reports.

Using the Powergate

We hope that by this point you are getting very excited about the Powergate. We have a few really exciting things just hovering on our backlog that could land any minute. If you couldn't tell, we're moving fast, so stay tuned to our blog and on Twitter where we'll share links when they are ready.

Finally, in case you missed any of our previous updates, here they are all together.

Contact

Andrew Hill

Along with Ignacio Hagopian, Carson Farmer

Recommended for you

No results for your search, please try with something else.

Great! You've successfully subscribed.
Great! Next, complete checkout for full access.
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.