Quick Look: Textile Files and Schemas
Update: some of the methods below are out of date. For a good overview of the latest and greatest methods, be sure to read the Tour of Textile.
A video and transcript introducing Textile’s new IPFS file loader
Earlier this week, we posted the following video of Sander introducing Textile’s brand new IPFS files API. We are extremely excited to be sharing these initial ‘quick looks’ with the community, so to make the content even more accessible, today we’re releasing a ‘transcription’ of Sander’s video. With both the video and this tutorial in hand, you should be able to follow along with Sander, and actually start playing with the Textile command line tools to see what’s going on under the hood. So let’s get started!
Before we jump into the content in the video, you’ll need to have a working Textile Go development environment going. This is easier than ever to get setup, and just requires a few developer tools and some initial build steps.
First, you’ll need Go installed. Some of our developer tools assume you have a pretty recent version (we’re on go1.11), so just be aware. While you’re at it, make sure your Go environment variables are setup (they should be by default). You might need to set GOROOT and GOPATH and update your PATH to reflect where you’re go binary is installed. Here are some good instructions on getting this all setup.
We’ll be using dep for Golang dependency management, and git for fetching packages. You’ll also need gx for ipfs/libp2p dependencies (we’ve written about gx before). Installation instructions for these various tools will depend on your OS, but both have good instructions. For example, on Mac they’re both just a simple brew install away. For gx, you’ll just do:
go get -u github.com/whyrusleeping/gx go get -u github.com/whyrusleeping/gx-go
This part is easy! Just grab the package from our GitHub repo and get installing:
go get github.com/textileio/textile-go cd ~/go/src/github.com/textileio/textile-go/ make setup
While you’re at it, you might as well build and install the command line tools:
make build && make install
The video moves pretty quickly through the demos, so it will be useful to start and stop along the way, or simply follow along the steps here and play the video separately. We’ll start with the initial setup (first thing Sander does in the video), and then move on to the queries and examples.
We only really need to do this once, but what we’re doing is creating a new Textile Wallet, initializing a new peer with the default Wallet account, and then firing up the Textile daemon which will allow us to query and interact with our peer. We’ll be writing up these steps in more detail in a future post, but for the time being, just think of these as one-time initialization steps.
textile wallet init
Copy the output to a safe/secure place. You’ll need SECRET SEED to initialize a new wallet ID:
textile init --seed=blahblahblah
This will create a new peer/repo in ~/.textile. We can look at the config (~/.textile/repo/textile) of that peer, and edit it if we need to (e.g., change ports etc), though for the purposes of this tutorial we can leave everything as their defaults.
If you want to tail the logs during the demo, so you can see what’s going on, open up a new terminal, and do so:
tail -f ~/.textile/repo/logs/textile.log
Now, we just fire up the daemon and we’re ready to go:
If you ever get ‘stuck’ along the way, or want to learn more about a command or tool that Sander is using, you can always call textile --help, and any of the textile sub-commands also have their own help entry. For example, Sander shows us the help message for the threads command via textile threads --help.
Speaking of threads, you can check out our previous post where we cover in detail how threads work, and some of their underlying tech. In the mean time, to list existing threads available via your peer, you can use the threads ls command, which will return a JSON array of objects with details about each thread. By default, there is already an account thread, which we’ll talk about more in a figure video/post.
textile threads ls
To do anything interesting, you’ll want to create a new thread. In the video, Sander creates a thread devoted to horses for demonstration purposes (and because who doesn’t like horses?!):
textile threads new horses --open --photos
Using the --open and --photos flags, we’ve just created a new thread that is open (allows any member to add content), and uses Textile’s built-in photos schema (which we use for Textile Photos).
Now that we have a thread created, take a quick look at its schema (which you get as part of the JSON output when you create the thread). Sander also goes through schemas more generally, comparing them to Transloadit schemas. Essentially, schemas describe what the node should do with incoming files. Skip to the video at 2:15 to get a nice explanation from Sander about these concepts.
Switching back now to the command line, let’s add some files. Sander adds a directory of horse photos to his horse thread, though of course, you are welcome to add any folder of image files you like for the purposes of the example. We take the thread_id from the previous thread ls output, and the folder should be something existing on your local hard-drive. You can specify relative paths, which is what Sander has done here (horses is a local directory). Essentially, the following command is saying “add the horses folder to thread <thread_id> with a caption of howdy” (for an explanation of what’s going on, skip to 4:45 in the video):
textile add horses -t <thread_id> --caption "howdy"
Once those files are added, you can explore their structure via the peer’s local gateway. This is very similar to an IPFS gateway, though the Textile gateway supports on the fly decryption and the Textile schemas. To follow along with Sander, with your daemon running, you can append a file hash from the add command output onto your (localhost) gateway’s ipfs endpoint: http://127.0.0.1:5050/ipfs/Qmblahblah. From here, Sander explores (skipe to 5:10 in the video) the DAG structure of the added folder, explaining the structure.
Since all the content is encrypted by default, in order to view it in plaintext, you’ll need the keys for the given DAG structure:
textile keys Qmblahblah # Group above add
Which you can then copy-paste as a query string parameter (?key=<key>) onto your gateway url to decrypt the file on the fly.
What if you wanted to create photo ‘albums’ or groups of photos as a single object? There’s a command (flag) for that as well! Simply add the --group flag to your add command, et voilà, you have a group of photo objects, each setup to fulfill the provided schema. This is great for keeping whole directories of files organized in a Textile thread:
textile add horses -t <thread_id> --caption "howdy" --group textile keys Qmblahblah # From above group add
At 8:50 in the video, Sander lists the threads (textile threads ls) on his peer, copies the thread id of his ‘horses’ thread, and then uses this id to list and paginate the files in that thread:
textile ls -t <thread_id>
Think of this like scrolling through your social media feed, from the command line (which is arguably way cooler). Speaking of which, these commands are all available on all platforms, so if you’re building a decentralized social media app, then this is exactly how you’d paginate through posts or media feeds using our mobile frameworks! Get in touch if this is something you’re interested in.
Inviting and Joining
In Sander’s last set of examples (skip to 9:20 in the video), he works through how to invite a new peer to an existing thread, and how to have that peer accept that invite and start interacting and sharing files directly via a peer-to-peer (p2p) connection. This requires you to spin up another Textile peer, and configure it to run on a different local port. We won’t go over this in today’s tutorial, but you should be able to get the idea from Sander’s video demo. The long and the short here is that, when two peers are interacting in a thread, they get each others’ updates in real-time! Very nice.
Thanks for following along with us! We hope you enjoyed the video and accompanying tutorial. We’re hoping to publish more of these as we continue to roll out our file loader API (slick name forthcoming 😉), so let us know what you think! You can reach out over Twitter or Slack, or pull us aside the next time you see us at a conference or event. We’re happy to provide background, thoughts, and details on where we’re taking this work. In the mean time, don’t forget to check out our GitHub repos for code and PRs that showcase our current and old implementations. We try to make sure all our development happens out in the open, so you can see things as they develop. Additionally, don’t miss out on signing up for our waitlist, where you can get early access to Textile Photos, the beautiful mobile interface to Textile’s Threads and underlying APIs.