Quick Look: Textile Files and Schemas

out of date Nov 28, 2018

Update: some of the methods below are out of date. For a good overview of the latest and greatest methods, be sure to read the Tour of Textile.

A video and transcript introducing Textile’s new IPFS file loader

Textile command line foo…

Earlier this week, we posted the following video of Sander introducing Textile’s brand new IPFS files API. We are extremely excited to be sharing these initial ‘quick looks’ with the community, so to make the content even more accessible, today we’re releasing a ‘transcription’ of Sander’s video. With both the video and this tutorial in hand, you should be able to follow along with Sander, and actually start playing with the Textile command line tools to see what’s going on under the hood. So let’s get started!

Initial setup

Before we jump into the content in the video, you’ll need to have a working Textile Go development environment going. This is easier than ever to get setup, and just requires a few developer tools and some initial build steps.

First, you’ll need Go installed. Some of our developer tools assume you have a pretty recent version (we’re on go1.11), so just be aware. While you’re at it, make sure your Go environment variables are setup (they should be by default). You might need to set GOROOT and GOPATH and update your PATH to reflect where you’re go binary is installed. Here are some good instructions on getting this all setup.

Developer tools

We’ll be using dep for Golang dependency management, and git for fetching packages. You’ll also need gx for ipfs/libp2p dependencies (we’ve written about gx before). Installation instructions for these various tools will depend on your OS, but both have good instructions. For example, on Mac they’re both just a simple brew install away. For gx, you’ll just do:

go get -u github.com/whyrusleeping/gx
go get -u github.com/whyrusleeping/gx-go

Installing textile-go

This part is easy! Just grab the package from our GitHub repo and get installing:

go get github.com/textileio/textile-go
cd ~/go/src/github.com/textileio/textile-go/
make setup

While you’re at it, you might as well build and install the command line tools:

make build && make install

Following Along

The video moves pretty quickly through the demos, so it will be useful to start and stop along the way, or simply follow along the steps here and play the video separately. We’ll start with the initial setup (first thing Sander does in the video), and then move on to the queries and examples.

Initial setup

We only really need to do this once, but what we’re doing is creating a new Textile Wallet, initializing a new peer with the default Wallet account, and then firing up the Textile daemon which will allow us to query and interact with our peer. We’ll be writing up these steps in more detail in a future post, but for the time being, just think of these as one-time initialization steps.

textile wallet init

Copy the output to a safe/secure place. You’ll need SECRET SEED to initialize a new wallet ID:

textile init --seed=blahblahblah

This will create a new peer/repo in ~/.textile. We can look at the config (~/.textile/repo/textile) of that peer, and edit it if we need to (e.g., change ports etc), though for the purposes of this tutorial we can leave everything as their defaults.

If you want to tail the logs during the demo, so you can see what’s going on, open up a new terminal, and do so:

tail -f ~/.textile/repo/logs/textile.log

Now, we just fire up the daemon and we’re ready to go:

textile daemon

If you ever get ‘stuck’ along the way, or want to learn more about a command or tool that Sander is using, you can always call textile --help, and any of the textile sub-commands also have their own help entry. For example, Sander shows us the help message for the threads command via textile threads --help.

Threads

Speaking of threads, you can check out our previous post where we cover in detail how threads work, and some of their underlying tech. In the mean time, to list existing threads available via your peer, you can use the threads ls command, which will return a JSON array of objects with details about each thread. By default, there is already an account thread, which we’ll talk about more in a figure video/post.

textile threads ls

To do anything interesting, you’ll want to create a new thread. In the video, Sander creates a thread devoted to horses for demonstration purposes (and because who doesn’t like horses?!):

textile threads new horses --open --photos

Using the --open and --photos flags, we’ve just created a new thread that is open (allows any member to add content), and uses Textile’s built-in photos schema (which we use for Textile Photos).

Schemas

Now that we have a thread created, take a quick look at its schema (which you get as part of the JSON output when you create the thread). Sander also goes through schemas more generally, comparing them to Transloadit schemas. Essentially, schemas describe what the node should do with incoming files. Skip to the video at 2:15 to get a nice explanation from Sander about these concepts.

Adding content

Switching back now to the command line, let’s add some files. Sander adds a directory of horse photos to his horse thread, though of course, you are welcome to add any folder of image files you like for the purposes of the example. We take the thread_id from the previous thread ls output, and the folder should be something existing on your local hard-drive. You can specify relative paths, which is what Sander has done here (horses is a local directory). Essentially, the following command is saying “add the horses folder to thread <thread_id> with a caption of howdy” (for an explanation of what’s going on, skip to 4:45 in the video):

textile add horses -t <thread_id> --caption "howdy"

Exploring content

Once those files are added, you can explore their structure via the peer’s local gateway. This is very similar to an IPFS gateway, though the Textile gateway supports on the fly decryption and the Textile schemas. To follow along with Sander, with your daemon running, you can append a file hash from the add command output onto your (localhost) gateway’s ipfs endpoint: http://127.0.0.1:5050/ipfs/Qmblahblah. From here, Sander explores (skipe to 5:10 in the video) the DAG structure of the added folder, explaining the structure.

Since all the content is encrypted by default, in order to view it in plaintext, you’ll need the keys for the given DAG structure:

textile keys Qmblahblah # Group above add

Which you can then copy-paste as a query string parameter (?key=<key>) onto your gateway url to decrypt the file on the fly.

Groups

What if you wanted to create photo ‘albums’ or groups of photos as a single object? There’s a command (flag) for that as well! Simply add the --group flag to your add command, et voilà, you have a group of photo objects, each setup to fulfill the provided schema. This is great for keeping whole directories of files organized in a Textile thread:

textile add horses -t <thread_id> --caption "howdy" --group
textile keys Qmblahblah # From above group add

Listing/paginating

At 8:50 in the video, Sander lists the threads (textile threads ls) on his peer, copies the thread id of his ‘horses’ thread, and then uses this id to list and paginate the files in that thread:

textile ls -t <thread_id>

Think of this like scrolling through your social media feed, from the command line (which is arguably way cooler). Speaking of which, these commands are all available on all platforms, so if you’re building a decentralized social media app, then this is exactly how you’d paginate through posts or media feeds using our mobile frameworks! Get in touch if this is something you’re interested in.

Inviting and Joining

In Sander’s last set of examples (skip to 9:20 in the video), he works through how to invite a new peer to an existing thread, and how to have that peer accept that invite and start interacting and sharing files directly via a peer-to-peer (p2p) connection. This requires you to spin up another Textile peer, and configure it to run on a different local port. We won’t go over this in today’s tutorial, but you should be able to get the idea from Sander’s video demo. The long and the short here is that, when two peers are interacting in a thread, they get each others’ updates in real-time! Very nice.

Stay Tuned

Thanks for following along with us! We hope you enjoyed the video and accompanying tutorial. We’re hoping to publish more of these as we continue to roll out our file loader API (slick name forthcoming 😉), so let us know what you think! You can reach out over Twitter or Slack, or pull us aside the next time you see us at a conference or event. We’re happy to provide background, thoughts, and details on where we’re taking this work. In the mean time, don’t forget to check out our GitHub repos for code and PRs that showcase our current and old implementations. We try to make sure all our development happens out in the open, so you can see things as they develop. Additionally, don’t miss out on signing up for our waitlist, where you can get early access to Textile Photos, the beautiful mobile interface to Textile’s Threads and underlying APIs.

Great! You've successfully subscribed.
Great! Next, complete checkout for full access.
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.