Integrating Powergate: Intro to the Powergate JavaScript Client

Textile built Powergate as a way to bridge the gap between IPFS and Filecoin storage. You get flexible data storage configurations optimized for long term, cryptographically verified, affordable storage on Filecoin, and high availability, fast, distributed storage on IPFS, or both simultaneously. According to the Powergate docs:

Powergate is a multi-tiered file storage API built on Filecoin and IPFS, and and index builder for Filecoin data. It's designed to be modular and extensible.

Powergate is meant to be integrated into other systems and applications. It's the easiest way to leverage IPFS and Filecoin, and by using Powergate to orchestrate how they work together, you get the best of both worlds plus the added smarts of Powergate's powerful data storage configuration mechanism.

If you'd like more information about Powergate in general, please see the repo's README and FFS design documentation.

Options for Integration

Today, there are a few tools we provide for integrating Powergate into your system or application. If you're building in Go, there's the Go Powergate client. If you're writing shell scripts or want to quickly experiment with Powergate APIs, the CLI would be a great tool (Installable executables coming soon -- for now, follow instructions in the README about how to install from source). The newest addition is our TypeScript/JavaScript Powergate client, and that's what we'll focus on in this post.

As a side note, all of the clients I just mentioned are built using gRPC services exposed by Powergate, so in theory any type of client could be built in any language supported by gRPC. Browse around the repo to see the various .proto files defining these services. For example, the service definition for the Deals module can be found here.

The Example Project

The project I'll work through in this post is a simple Node.js web app built with Express as a server, router, and middleware and it uses Pug templates for rendering HTML. You can find the finished product in the examples folder of the Powergate JavaScript client repo. I'll use TypeScript because I like it, but you don't have to. The app will display some basic information about the Powergate instance it's connected to and allow users to authenticate using GitHub OAuth. Once authenticated, the web app will display additional information about the user's own FFS instance.

Initial Project Setup

If you have an existing project you'd like to use as we walk through this example, you'll just need to install @textile/powergate-client and then adapt further code examples I provide to whatever tools your project uses.

npm install @textile/powergate-client

For everyone else, I created a minimal application you can clone or fork that includes all the dependencies we'll need, plus some basic plumbing for user authentication, and UI. To get started with it, run the following:

git clone https://github.com/textileio/node-starter.git
cd node-starter
npm install

Setup a GitHub OAuth App

To get GitHub authentication working in the example app, you'll need to log into your GitHub account and create a new OAuth App. You'll find this under Settings > Developer Settings > OAuth Apps. Create a new one, you can use http://127.0.0.1:3000 as the Homepage URL and http://127.0.0.1:3000/auth/github/callback as the Authorization callback URL. Take note of the generated client id and secret provided after you submit the form.

Back in our project, copy .env.example to .env and open it for editing. Update GITHUB_CLIENT and GITHUB_CLIENT_SECRET with the values from your new OAuth app.

Important: Don't check your .env file into any public repository since it contains application secrets. The .gitignore in our starter project is already set up to do the right thing here.

Initial Run

You should now be able to run the app in debug mode:

npm run debug

And you should see the default application home page at http://localhost:3000.

Application Anatomy

Our starter app already has some basics that we'll take advantage of. Here's an overview of the important pieces and how we'll use them when integrating Powergate.

src/server.ts is the main entry point to our application.

src/util/env.ts gives us access to variables defined in our .env file. We'll use those to specify where to connect to our Powergate instance as well as access other application secrets.

src/config/passport.ts configures our Passport GitHub authentication strategy. We'll allocate Powergate resources for authenticated users.

src/models/user.ts contains our User data model and provides SQLite-based persistence of user data. We'll extend this model to associate Powergate information with users.

src/app.ts is where our Express application lives. All routes and route-handling middleware are defined here. We'll use middleware to interact with the Powergate client, querying for data to display and creating Powergate resources for authenticated users.

views/ contains Pug templates that we'll use to render information about Powergate and authenticated user Powergate resources.

Powergate currently uses two categories of API access; Unauthenticated requests to access general information about the Powergate and Lotus node health, network, miners and more, and authenticated requests to access to individual FFS instances. A single FFS instance provides partitioned access to and management of data storage on IPFS and Filecoin.

Our goal with this example app is to display some general information about Powergate using the unauthenticated APIs, and then create a FFS instance for  each authenticated user. The same FFS instance should be used by a user that logs out and then returns to the app at a later time. We'll display a bit of information about the authenticated user's FFS instance.

Running Powergate

Of course, in order to integrate Powergate into our app, we'll need an instance of Powergate running. Textile is considering offering hosted Powergate services, so if that's something you'd be interested in, please get in touch. For now, there are a few options to get Powergate running yourself.

1. BYO - Powergate depends on connecting to IPFS and Lotus nodes. You are free to run your own and then run the Powergate server powd yourself, configuring it with the addresses of your IPFS and Lotus nodes.
2. Docker + testnet - We provide a Docker compose configuration that will spin up Powergate, IPFS, and Lotus, plus some extra metrics tracking and visualization tools. The resulting setup runs on the Filecoin testnet (and mainnet once it launches). This is good for a production setup, but will feel slow for development purposes.
3. Docker + localnet - We created a special configuration of Lotus that runs a filecoin network locally at high speed, and it works great for testing and development. It's easy to use via a localnet Docker compose configuration. This is how we'll be running Powergate and its dependencies for this example app.

The Powergate releases page includes bundled up Docker Compose files that use appropriate Docker image versions of Powergate, Louts, and IPFS. Let's check our package-lock.json to see the appropriate release to download and run. Look for @textile/grpc-powergate-client. Its version number corresponds to the version of Powergate the underlying gRPC bindings were created from.

"@textile/grpc-powergate-client": {
    "version": "0.0.1-beta.13",
    ...
},

Download the powergate-docker-<version>.zip from the appropriate release, in this case, version 0.0.1-beta.13. Assuming you have Docker Desktop installed, start up Powergate using the provided Makefile:

unzip powergate-docker-v0.0.1-beta.13.zip
cd powergate-docker-v0.0.1-beta.13
BIGSECTORS=true make localnet

Important note on BIGSECTORS: When running the localnet setup, the Lotus node is configured with a mocked sector builder, using either "small" or "big" sector sizes. The practical effects of this configuration are on the size of files you can store in the localnet and how quickly the storage deals will complete. Using BIGSECTORS=false will limit you to storing files of around 700 bytes and deals will complete in 30-60 seconds. Using BIGSECTORS=true will allow you to store files anywhere from 1Mb to 400Mb, but deals will complete in 3-4 minutes. Be sure to choose the value that makes sense for your development scenario.

Connecting to Powergate

First, we'll configure the project to connect to our Powergate instance and then create the client we use to make calls to Powergate. Open .env for editing and add a new variable for the Powergate API host. The below value is correct if you're running our Docker compose setup, but adjust it as needed if you're running Powergate elsewhere:

POW_HOST=http://0.0.0.0:6002

Then, let's add an the exported value of this variable to src/util/env.ts so we can easily read it's value from elsewhere in the application:

export const POW_HOST = process.env["POW_HOST"]

Now we can create our instance of the Powergate client, connecting to the configured host. In src/app.ts we'll import our POW_HOST variable and the Powergate client factory function createPow, and finally create our Powergate client we'll call pow:

import { createPow } from "@textile/powergate-client"
import { EXPRESS_PORT, POW_HOST, SESSION_SECRET } from "./util/env"

const pow = createPow({ host: POW_HOST })

Unauthenticated Requests + Display

Now that our Powergate client is ready to go, we can use it to query some data from Powergate, and then we'll display that data on the home page of the web app. In src/app.ts, we'll update our / route handler to call some Powergate APIs and pass the resulting data to the render function so it's available in our HTML template. Most Powergate functions return Promises since they are making async calls over a network. For that reason, we also need to label our route handler function as async, use try/catch as usual with Promises, and call the handler's next parameter in case of an error. Here's the updated version:

app.get("/", async (_, res, next) => {
  try {
    const [respPeers, respAddr, respHealth, respMiners] = await Promise.all([
      pow.net.peers(),
      pow.net.listenAddr(),
      pow.health.check(),
      pow.miners.get(),
    ])
    res.render("home", {
      title: "Home",
      peers: respPeers.peersList,
      listenAddr: respAddr.addrInfo,
      health: respHealth,
      miners: respMiners.index,
    })
  } catch (e) {
    next(e)
  }
})

We've now passed the results from our calls for peers, listen address, health status, and Filecoin miners into our render function, so we can update views/home.pug to render that information. I updated the header text, and to avoid needing to make more UI, I'm just stringifying the data and printing it inside <pre> tags:

extends layout

block content
  h1 Node Info
  p.lead A sample of generally available data, no auth required.
  hr
  .row
    - const jsonPeers = JSON.stringify(peers, null, 4)
    - const jsonAddr = JSON.stringify(listenAddr, null, 4)
    - const jsonHealth = JSON.stringify(health, null, 4)
    - const jsonMiners = JSON.stringify(miners, null, 4)
    .col-sm-6
      h2 Node Health
      pre= jsonHealth
    .col-sm-6
      h2 Listen Address
      pre= jsonAddr
    .col-sm-6
      h2 Peers
      pre= jsonPeers
    .col-sm-6
      h2 Miners
      pre= jsonMiners

Refresh, and you should see:

User Model Update

We need to update our User model to represent the FFS instance associated with each user. We'll add a ffsToken?: string property to the User type in src/models/user.ts. It will hold the token returned when we create a FFS instance for a user:

export type User = {
  gitHubId: string
  email: string
  ffsToken?: string
}

Next, update the User references and SQL statements in src/models/user.ts for the User CRUD operations, taking into account the new ffsToken property. For example, findByGitHubId becomes (notice we update the SQL statement with a new field and the response object with a new field):

export const findByGithubId = async function (gitHubId: string): Promise<User | undefined> {
  const q = "SELECT gitHubId, email, ffsToken FROM users WHERE gitHubId = ?"
  const row = await db.get(q, gitHubId)
  if (!row) {
    return undefined
  } else {
    const user: User = {
      gitHubId: row.gitHubId,
      email: row.email,
      ffsToken: row.ffsToken,
    }
    return user
  }
}

This is a little tedious, so you may want to just copy the final version from the example app repo.

FFS Creation Middleware

In src/app.ts, the route /auth/github/callback is called when GitHub redirects the user back to our application as part of the OAuth process. It currently finishes the authorization process and then redirects the user to /user. We'll add another handler to that flow, after completing the authorization process, but before redirect, to create a FFS instance if needed for the now-authenticated user then update and save the User instance:

import { save, User } from "./models/user"

app.get(
  "/auth/github/callback",
  passport.authenticate("github", { failureRedirect: "/" }),
  async (req, _, next) => {
    if (req.user) {
      const user = req.user as User
      if (user.ffsToken) {
        pow.setToken(user.ffsToken)
        return next()
      } else {
        try {
          const createResp = await pow.ffs.create()
          user.ffsToken = createResp.token
          await save(user)
          pow.setToken(user.ffsToken)
          next()
        } catch (e) {
          next(e)
        }
      }
    } else {
      next(new Error("no user found in session"))
    }
  },
  (_, res) => {
    res.redirect("/user")
  },
)

Our handler is async since it calls the Powergate client's Promise-based ffs.create() and the User save() functions.

We first make sure we can access our Express user object (which we know is our User type). If not, this is an error.

Then we check if the user already has a ffsToken. If so, this is a returning user and we're done. We call next() so the request handler chain simply continues.

If the user doesn't have a ffsToken, we create a new FFS instance, set the user's ffsToken property with the returned token and save the updated User.

The last step is calling pow.setToken(user.ffsToken). This passes the user's FFS token into our Powergate client, and the client will now send that token along with each request to Powergate so it knows which FFS instance the user is interacting with.

Important note: The communication with Powergate is currently unencrypted http so the FFS token is sent in plain text. We plan on providing tooling to support encrypted https in an upcoming release.

Authenticated Request + Display

In src/app.ts, we already have a route, /user, thar requires authentication (enforced by the passportConfig.isAuthenticated middleware). Having an authenticated user at this point implies that we've already created a FFS instance for that user, and that our pow client is configured to use that FFS instance's token. So let's now make an authenticated call to Powergate. We'll do this by updating the handler for the /user route:

app.get("/user", passportConfig.isAuthenticated, async (_, res, next) => {
  try {
    const info = await pow.ffs.info()
    res.render("user", {
      title: "User",
      info: info.info,
    })
  } catch (e) {
    next(e)
  }
})

Here, we call ffs.info() to which returns some information about the user's FFS instance. We pass the result into our render function so we can display it in the updated user.pug template:

extends layout

block content
  h1 FFS (#{info.id})
  p.lead Data for your FFS instance.
  hr
  .row
    - const jsonInfo = JSON.stringify(info, null, 4)
    .col-sm-6
      h2 Instance Info
      pre= jsonInfo

If you refresh and make sure you're logged in, you should see:

If you log out and log back in, you should see that the FFS instance id remains the same, showing that the same user will always be interacting with their unique FFS instance.

Wrap Up

In this example, we've retrofitted a very simple, but typical, Node.js web app with Powergate capabilities. With this simple change, we can view information about our Powergate, IPFS, and Lotus nodes, and users now have access to powerful data storage on IPFS and Filecoin through their own FFS instance. I hope that seeing one way to map Powergate concepts onto an existing application gives you ideas about how you might integrate Powergate into your own systems.

Next Steps

The Powergate JavaScript client is brand new and improving every day. The most commonly used Powergate APIs are available already, but we'll be adding more and keeping the client up to date with the core Powergate APIs. Be sure to star or subscribe to the repo to stay updated.

Creating FFS instances for users is really just scratching the surface of what's possible with Powergate. FFS is the main API you'll be interacting with, and I'd encourage you to extend this example to use more of its functions for adding and retrieving data to and from IPFS and/or Filecoin. In fact, that may be a great follow up example app and blog post... Stay tuned!