We're excited to announce a new major version of Powergate. This release comes with many new features and tools. Since the breadth of changes is quite broad, we will focus on particular points in our release notes.

Before jumping into the exciting new features, we want to stress that in Powergate development, we are following a strict semantic-versioning strategy. Whenever a release has breaking APIs or assumptions, we'll bump the major version. This major version bump isn't only about breaking APIs though, it also contains interesting new features we want to share with you.

TL;DR: We added a deal-importing feature which makes it possible to request arbitrary retrievals from the Filecoin network. We also improved our APIs in general and included a set of new dev-tools to make the operator's life easier. Want to know more of the nitty gritty details? Keep reading!

💥 API Changes

We keep a close eye on the #powergate-users channel in the Filecoin slack workspace. It's where most Powergate users meet, sharing their questions and experiences. This information is invaluable to us since we can understand their pain-points and keep evolving the project in the right direction.

We think software sometimes needs to have breaking APIs at some point in order to evolve in a healthy way — particularly in the early stages of the project. In Powergate, we always try to avoid breaking APIs since this produces friction to existing software but keeping that as a strict rule would cause issues in the long run.

The first breaking API of this release is just avoiding some method name stuttering, for example, to transform Client.StorageInfo.StorageInfo(cid string) calls into Client.StorageInfo.Get(cid string). This avoids some redundant naming in the Go client library where method names can be understood within the scope of where they are being called from. Similarly, other APIs were renamed; you can inspect them in more detail by looking at the commit diff.

The second improvement and biggest breaking change is in the StorageJobs APIs. Individual methods for getting queued, executing, and final storage jobs have been replaced with a single List() method. This method takes query options that can be configured to change the query behavior and filter the results to jobs in the queued, executing, or final states.

Let's see some code on how this API works:

const (
	// All lists all StorageJobs and is the default.
	All ListSelect = iota
	// Queued lists queued StorageJobs.
	Queued
	// Executing lists executing StorageJobs.
	Executing
	// Final lists final StorageJobs.
	Final
)

// ListConfig controls the behavior for listing StorageJobs.
type ListConfig struct {
	// CidFilter filters StorageJobs list to the specified cid. Defaults to no filter.
	CidFilter string
	// Limit limits the number of StorageJobs returned. Defaults to no limit.
	Limit uint64
	// Ascending returns the StorageJobs ascending by time. Defaults to false, descending.
	Ascending bool
	// Select specifies to return StorageJobs in the specified state.
	Select ListSelect
	// NextPageToken sets the slug from which to start building the next page of results.
	NextPageToken string
}

func (j *StorageJobs) List(ctx context.Context, config ListConfig) (*userPb.ListStorageJobsResponse, error)

This change intends to narrow the surface area of the APIs and allow future improvements in query-options that won't produce breaking changes.

You might notice something interesting things in ListConfig: we now support paging! This need came from listening to heavy-users in #powergate-users. The number of storage-jobs in their instances was too big to only have get-all-data style APIs. This same List() API allows to limit the number of returned storage-jobs per call and handle pages using the NextPageToken field. 🙌

📑 Deal importing

This release includes a new feature for importing Filecoin deals into a Powergate StorageInfo. This means that you can bootstrap or augment the known deals for a Cid even if your Powergate daemon didn't make those deals in the first place! Maybe you're not used to Powergate terminology, so let's unpack that a bit.

👩‍🎓A refresher on Powergate terminology

Whenever you're using Powergate, you can define your desired state for storing a Cid. This declarative information is called a StorageConfig. A StorageConfig has information such as the desired replication factor, trusted miners to make deals, maximum price to pay per GiB/epoch, and others. Similarly, each Cid has a StorageInfo, which contains the latest storage state information, such as existing Filecoin deals for this data.

Whenever you apply a StorageConfig to a Cid, a Job will check if this configuration should make new deals for this data, and if that's the case, it will do proper work to enforce it whenever new deals are made. The final storage state of the Cid is saved in a corresponding StorageInfo.

🔎 What happens exactly when importing?

Starting in this release, you can run pow config apply [cid] --import-deals [deal-ids...] to import Filecoin deals into the StorageInfo, even if Powergate didn't create those deals initially. Recall that each Deal ID is a globally unique deal identifier in the Filecoin network. You can use some existing tools such as filecoin.tools to explore existing deals in the network — discovery services are a nascent idea!

When the importing happens, Powergate does some important checks to ensure those deal identifiers match the Cid. This turns out to be a bit "delicate" since Cids of stored data aren't in Filecoin on-chain data. Some miners might include the Cid as metadata information, but that isn't mandatory. This happens because when you store a Cid in Filecoin, the DAG is transformed to a CAR file with some extra padding. This result, PieceCid, is the practical stored reference that lives in the miner's sectors and is the data that is cryptographically verified automatically by the network.

If you're pushing a StorageConfig for a Cid where the data is only in Filecoin you are in an interesting situation. If you also import external Deal IDs, there's no 100% safe way to check that those Deal IDs correspond to the Cid you're targeting. To verify that the corresponding on-chain  PieceCid for those deal corresponds to that Cid, you need the Cid data to do the transformation and verify if the PieceCid output matches! The only thing Powergate can do in this scenario is to check that the PieceCid of all imported deals matches each other, so at least they're for the same data.

If the Cid StorageInfo already has known deals made by Powergate, we can rely on their PieceCid of the Cid because Powergate had calculated the PieceCid when the deal was made. In this scenario, we can do a stronger check and compare the deals PieceCid with known deals in StorageInfo to see if they match. In a nutshell, if deals are known, we check that the imported deals' PieceCid matches all known PieceCid for the Cid.

Most of the above checks are important to avoid user errors when importing deals that don't correspond to the same data. You don't need to understand all this, but we wanted to share how much Powergate tries to take care of so that nothing bad happens. 👨‍⚕️

Apart from the above, every imported deal is checked if it is still active on-chain, so you know instantly if the deals you're importing have expired or have been slashed. Deals in that situation won't be present on-chain. This sounds reasonable since importing unretrievable deals is useless and might alert you to some breaking assumptions in your work flow.

🤔 Why would I be interested in importing external deals?

Importing deals opens the door to let external deals serve other existing features in Powergate, one example being Filecoin retrievals. Before this release, Powergate couldn't retrieve Filecoin deal data were not originally created/stored by Powergate, but now you can! We can imagine many powerful application features built on top of Filecoin retrievals.

Recall that if you enable hot-storage in your StorageConfig, Powergate will try to find that data and store it in its go-ipfs node. First, it makes a lookup in the IPFS network to see if it's available just to save spending Filecoin if possible. If the data isn't available, it will use known deals stored in StorageInfo to do a Filecoin retrieval. The more Deal IDs that are known in StorageInfo, the more options Powergate has to make a successful retrieval with the lowest possible price.

🔥 Demo of import and retrieval

Let's see the whole thing in action! The video is pretty self-explanatory, but basically we make a deal in the local-net, and we create a fresh user to import the deal and make the retrieval (see in 4k quality for the full effect):

🛠️ Tooling bonus!

In this release we include two new tools that makes an operator's or developer's life easier... and safer!

The first one is powcfg, a dev-tool with all the wiring boilerplate to make Powergate-wide StorageConfig transformations. For example, you might want to change all your StorageConfigs with an updated Filecoin.MaxPrice. You could do that with the CLI, but that might be cumbersome.

Additionally, some changes involve complex logic that might only change some configurations but not others. A developer can leverage this tool and only focus on programming its needed logic and forgetting about the extra boilerplate of connecting to the database, serialize the changed configuration, and save in the database at the correct keys. If you're curious, you can see here some examples of transformations you can create; quite easy!

The second tool is migrtest, which helps to have easier and safer migration testing. Every evolving software should deal with database migrations at some point, and this process can be quite hairy. Although we have CI tests for testing migrations, it's always interesting to note that the tests might be too synthetic, and real data can produce corner-case scenarios that might crash migrations in ugly ways.

This tool connects to a MongoDB backed go-datastore, which corresponds to a Powergate instance, it then copies the data (locally or in the same MongoDB cluster), and runs the appropriate migrations there. You can test exactly what the migration output would look like in your data before running the real migration in your Powergate upgrade. This tool doesn't connect to a Badger backed go-datastore, so if you're interested in using this tool in that scenario, let us know in an issue! We can quickly help you with adding support for that. We're planning to keep evolving this tool to provide further features such as generating a diff to double-check that changed keys are expected and other handy things!

🥗 Other changes

This release also includes some important changes that might break some current assumptions in your setup.

💽 The default miner-selector is reputation

The default miner-selector is reputation again, like in the early days of Powergate before the SpaceRace 2 competition.

While this competition was running, we created a new miner-selector that selects miners on-the-fly, referring to an external URL file instead of generated scores in Powergates indices. Special curators paid cautious attention to only include reliable and affordable miners for clients to use in the competition. For this reason, when Powergate runs (until v2.0.0) using sr2 miner-selector, it wouldn't pay attention to your TrustedMiners filter in the StorageConfig.

Of course, Powergate users always had the freedom to use the reputation miner-selector via flags/env configuration. But now, since the initial round of the competition is over and the official miner list is no longer maintained, we decided to make reputation the default miner-selector. This allows avoiding confusions regarding Why Powergate isn't reacting to my TrustedMiners list? Saying it differently, Powergate by default will now do 100% what you describe in your StorageConfig.

The reputation miner selector tries to select miners using indices built with on-chain information and market prices. Coming to an automatic score system that is 100% reliable is a very hard problem, mostly because this reputation still doesn't include other important network-wide information about miner's reliability, such as successful deals and retrieval history. So, with great power also come great responsibility! Please set your TrustedMiners and MaxPrice with reasonable values so you can optimize deal-success and minimization of risk.

We're working towards an improved reputation miner-selector that can leverage network-wide knowledge about reliable miners. Stay tuned for updates on that front, you might hear news soon! 😉

So, how does this change matter from an operational point of view? The default value of --ffsminerselector is now reputation, not sr2. If you were intentionally using the --ffsminerselctor default value before, you should explicitly set --ffsminerselector=sr2. Also, if you have set --ffsminerselectorparams to customize the previous default URL, then there's a 99% chance you were relying on --ffsminerselector=sr2, so you need to make that flag explicit now.

It turns out that, despite the fact that the official SR2 competition is done and the list is no longer maintained, some Powergate users liked the sr2 way of controlling on-the-fly deal-making decisions and created their own remote lists. For example, our friends at Deplatformr had configured --ffsminerselectorparams to override the miner's list to their customized list. We feel that this miner-selector still makes sense, and we decided to keep it for some more time, but just not being the default.

🛑 A new --no-exec flag for applying storage-config changes

This new flag comes to allow operators to skip Job execution while pushing a new StorageConfig.

Before v2.0.0, every pow config apply action to push a modified StorageConfig would trigger a Job that would enforce that desired state. That's not something that you always want to happen. When a Job runs, Powergate will do whatever it can to make your current storage state be the same as your desired one, and that may involve making deals.

For example, if you want to update the Filecoin MaxPrice for future use, say in an upcoming storage renewal, but without trying to enforce the current StorageConfig immediately, you can now provide the flag/option NoExec which would only save the new StorageConfig without executing a job. You can have better flexibility now in how you manage StorageConfig updates without being tangled with job execution.

🙌 Conclusions

We hope you're excited as we are with this release! To refer to the full set of changes, take a look at the v2.0.0 release notes. Feel free to join our community at #powergate-users in the Filecoin slack, where you can receive help from us and the community. 🍻

By the way! If you're interested running your own hosted Powergate with zero-operational burden, checkout https://textile.io/#managed-infra.