Textile Buckets: Dynamic data storage and management for IPFS

Update: Buckets have gotten a major update. CLI examples below have been changed. To get the latest, check out the new Buckets documentation.

Today we are excited to add Buckets, simple IPFS dynamic folder management, and remote pinning to the Textile suite of tools. Textile Buckets provide a simple way for you to publish and manage your website, blog, or data. Additionally, Buckets give you team- and project-level management of folders on IPFS. With it, developers or teams can easily manage remotely pinned datasets, publish those datasets over IPFS or HTTP, and make changes to them easily. Here are a few features that make Buckets magical:

1. Buckets are dynamic folders on IPFS (built on Textile's Threads).
2. Each Bucket can be updated and managed by multiple members on a team.
3. Every Bucket has a web URL that never changes, but will always display the latest contents of the folder!
4. Every Bucket has a web URL that never changes but will always render your Bucket as a website (as long as it contains an index.html).
5. Developers can easily track the latest IPFS CID of their Bucket "head", making it simple to use Cloudflare to render your Bucket on your custom domain.
6. Buckets are amazing for publishing websites right from GitHub Repos or local static site builds! More below.

Buckets – built with IPFS and Threads, together – provides users with simple to use, interoperable, and dynamic data folders shared over IPFS. Here are a few ways you can start using Buckets today:

1. Download the Textile CLI to create your own Buckets or invite collaborators to Buckets.
2. Use our example CI scripts to automatically publish your website, blog, or code repository to IPFS.
3. User our example CI scripts to automatically run your website on IPFS using a Textile Bucket.

Getting Started

Access to Textile Buckets is available to anyone using the Textile command-line too. Grab the latest release of Textile here. Textile offers free, password-less login for you and your team.

Next, go ahead and try pushing a Bucket. Here are the steps to create your first Bucket.

cd <a project folder with an image, file, or website to publish>
textile login
...enter your email
textile team add <make a team name>
textile switch
...select your new team
textile project init <choose a project name>
textile bucket push <a file, directory, or *> <choose a bucket name>

Next, you can visit your content at:

https://cloud.textile.io/dashboard/<project name>/<bucket name>

Or, if you pushed a website, visit it live at https://<bucket name>.textile.cafe

Use Buckets in your CI

The easiest way to pin data to IPFS right from your GitHub repo.

We've published a few easy to use GitHub Actions that allow you to set up simple workflows for updating your Buckets just by pushing code to your repository. You can use these Actions to:

1. Publish your website or blog post to IPFS.
2. Create ephemeral Buckets/URLs to view builds of your website before release.
3. Publish and update your website using IPFS.
4. Mirror your source-code on IPFS.

The Bucket Push Action returns two variables that you can chain with additional Action, or just use for your own projects. The first is the CID. This is the Content ID of your data/blog/file/etc on IPFS. You can request it over any IPFS gateway or pin it to your own node to improve speed and redudancy (you can even use an external pinning service). The Action also returns the URL of your content on Textile's cloud gateway.

CI Examples: Gatsby websites and JavaScript web apps.

We've put together two simple examples for using Buckets with CI on GitHub to help you get started.

Gatsby Blog

In the first, we create a simple blog using Gatsby. The CI setup allows you to build your blog using ephemeral buckets whenever you create or update a Pull Request. Then, whenever those changes are merged to master, it updates a primary Bucket. In each case, the Bucket URLs are live for you to review or connect to your own domain.

The setup is very handy. This specific demo website publishes to a bucket that can always be viewed at https://gatsby-ipfs-blog.textile.cafe/. The URL never changes but always reflects the content pinned in the latest updates to the Bucket. Every Pull Request opened (such as this one) gets an ephemeral URL for viewing the changes: https://gatsby-ipfs-blog-377515556.textile.cafe/.

JavaScript (Typescript) Web App

In the second, we use the GitHub Actions to build a web app (and PWA) written in Typescript to the final web app (including HTML and JavaScript). Next, it pushes that build folder to a different Textile Bucket depending on whether it is a Pull Request or Master. The setup is the same as the Gatsby site above, except uses the specific npm build steps to get the right output. Read the setup tutorial on the GitHub site for details!

Hugo Blog

Adapted and pushed by one of our community members, this example includes the build step for Hugo based websites and blogs.

What's next?

Buckets are still an experiment and we are learning how best to organize them and expose the right experience around them. If you try them out and have feedback, please let us know. We already have a few things we know will change, so keep an eye out for updates and progress.

The Textile team is already using Buckets and the corresponding tools on several of our production repos, and we're constantly finding new and interesting ways to combine web2 tech like GitHub Actions, with web3 innovations, like IPFS. If you'd like to give this a try, please dig in, ask us questions over in our Slack channel, start playing around with our Actions in your own repos, or just drop us a line to tell us what cool stuff you're building! We're excited about the prospect of combining good tools with the features of decentralized technology, and we think you'll find the experience is getting better every day.