The pioneer’s guide to GX — decentralized dependency management on IPFS

Decentralized package management is an exciting area of tool development for today’s developers. Decentralized package managers have the potential to remove centralized points of failure from the interconnected network of modern software libraries. They give development teams a straight-forward way to ensure their internal networks of code are always available, while at the same time ensuring redundancy across the entire Internet. Decentralized package managers can make our code, one of modern society’s most valuable outputs, more resistant to tampering, censorship, and manipulation.

That said, decentralized package management is still incomplete. There are many issues to sort out when publishing our code into networks built around an immutability of digital information where unlimited copies of that information may exist at any time. Without central authorities, they need new solutions for discovery, information removal (e.g. content not owned by the publisher), disputes, and potential bad actors. These issues are shared not just by decentralized package managers, but by decentralized networks in general, and so some solutions have already been proposed, and many others are actively being developed. That said, before using any decentralized package manager, it’s important you understand your risks and responsibilities.

I’ve included a more complete discussion of the promise (and some of the pitfalls) we face when decentralizing our source code and network dependencies. If you aren’t familiar with the topic, I highly recommend you read it before working through this tutorial. You can find that here, Decentralized code distribution for the future of open source.

So with that, here’s a brief introduction to using the decentralized package manager GX (specifically the Golang implementation, gx-go) to distribute your code and manage your code dependencies over IPFS. One last thing before we start, here is what the GX developers say in their own readme,

gx is Alpha Quality. It’s not perfect yet, but it’s proven dependable enough for managing dependencies in go-ipfs and ready for pioneering developers and early users to try out and explore.

An introduction to GX and gx-go

The GX library core functionality that is used by both the gx-go and gx-js package managers. The basic principles that they all share are,

1. Code versions you publish are referenced by an immutable hash of the code contained.
2. Dependencies you link into your code link to those immutable hashes both to find and retrieve the code and to ensure you are using the version you expect.
3. The immutable hashes are the addresses where the published code should be available on IPFS.

What does all that mean? Well, to start, instead of relying on something like semantic versioning as a global reference to a specific release of code, it uses the fingerprint (the hash) of the code itself. GX also enables a secondary mapping of the hash to a semantic version, this is mostly to improve human interaction with code (e.g. readability). These hashes become part of the content addressing system used in IPFS.

If you aren’t familiar with IPFS, there are a few great overviews to explain what the system is, how it works, and using it outside of GX. The thing to understand is that it is a set of technologies and protocols that enable any user to distribute media or data over a decentralized network of computers. In the case of a GX user, their code. As long as someone is distributing the media you are looking for, you can use IPFS to retrieve that data over the network.

For GX, this distribution model means that if you want to publish your code over the system, you need to start by making it available on the IPFS network. While you might be the first to host that data, once other people start using it, it will likely become available from other nodes pretty quickly. Textile and other GX users often mirror the code they use, adding redundancy and speed to the network. In the Textile Photos Go library for example, we have a team IPFS node where we pin all the dependencies we rely on from GX.

Okay, that’s the basic mechanics. Let’s try using it.

Installing GX and gx-go

This easy step is required for both of the next sections (distribution and dependency management). Get GX and gx-go running on your system, simply run the following two commands.

go get -u github.com/whyrusleeping/gx
go get -u github.com/whyrusleeping/gx-go

As long as you added your Go root and /bin correctly to your path the above should make the GX command line tools available. If the previous line doesn’t make sense to you, there are a few tutorials on getting started with Go that could be useful before continuing here.

Sharing code on GX

We more or less stumbled upon GX because it’s the package manager used in IPFS related projects. But using it got us really thinking about all the benefits we mentioned above. GX was originally written to support gx-go, but there is also gx-js, and the original GX code is ready to be forked for package managers in any language. It’s extensible enough that it got us thinking about what the future of an mobile app store on IPFS could look like… We’ll save that for another day though. In the following blog post, let’s walk through the basics of using gx-go in your own projects.

Setting up GX in your Go project

From inside the folder of your Go project, you’ll simply run the following in your terminal,

gx init --lang=go

Running the above is going to add a package.json file to your folder containing a few necessary bits to work with GX. If your project already has a package.json file for some other reason: the important bits will be added. Here is an overview of what information GX stores in your package.json file.

Your package.json will list the GX version you are using in this project:

"gxVersion": "0.12.1",

Also from the package.json, GX adds the import rewrite paths. This will specify how to rewrite import statements in a user’s code (as opposed to showing them non-human readable hashes all the time). Here is what mine looks like, automatically detected from my Go path.

"gx": {
  "dvcsimport": "github.com/textileio/gx-demo"
}

Next, the package.json contains the helper command that will later tell GX how to handle new releases for you.

"releaseCmd": "git commit -a -m \"gx publish $VERSION\""

That’s it. Now, your code is ready to start publishing on GX. Let’s try publishing the version version of your code on GX. To do that, we can just commit our code using GX, which will run the releaseCmd script above.

gx release 0.0.1-dev

Error: ipfs daemon isn't running

Now… you may have hit the first gotcha. Like I said in the introduciton, GX uses IPFS to distribute your code, which means you need to have an IPFS daemon running on your system. I’m not going to walk through the installation steps here or the one-liner you need to get the daemon up and running. But once you’ve done that, you should be able to run gx release 0.0.1-dev again and get an output like,

gx release 0.0.1-dev 

package gx-demo published with hash: QmZFb51F1rJcHy9UUWWgPvwJAMorMgdzw2a52y2xgjVZJu

Now, anyone should be able to access your code through GX or directly through IPFS (or even a gateway).

You may have noticed that when your ran gx release a new file was created, .gx/lastpubver. This file contains a mapping of semantic version number to the IPFS hash. You should include this folder in your git history as it can link the code to the releases at any point in the future. Unlike a hash, a semantic version isn’t guaranteed to be immutable. Even in GX, you could republish the same semantic version with a totally new set of code, and therefor a new hash.

Publishing a new release on GX

Okay, so you’ve gone through and changed your code, now you are ready to release a new version. You now have a couple of options. GX doesn’t have many rules here so technically you could republish the same version number again just by running,gx release 0.0.1-dev. But more likely you are going to want to release the next version, so just run,

gx release 0.0.1

package gx-demo published with hash: QmcUaXVES69qZEhV8tdUbhYtVBkbGU7b9h7bpvAo6xnvC9
[master 6f1a98f] gx publish 0.0.1
 2 files changed, 2 insertions(+), 2 deletions(-)

You’ll notice that your .gx/lastpubver has been updated and you can now share your new hash with the web.

Installing dependencies from GX

If you want to write source code that is resistant to centralized services going offline, changing their roadmap, or being blocked, then you may want to use GX for managing your dependencies. You don’t necessarily have to publish your project on GX in order to use GX this way and not every library is available on IPFS, but hopefully more and more will get there in the months to come.

So let’s take a look.

The first thing you can do is import a new dependency. You’ll need to have a target hash for GX to be able to do anything. Okay, so here is an open source project we could use https://github.com/mr-tron/base58. It’s a straight-forward library that can convert strings in and out of base58in Go. So, if you know what you’re looking for you’ll notice that the repo doesn’t contain any .gx/pubver file. So we have a couple options:

• We can use the steps in the above section to add GX to the project and get the hash that way. After that, we can use and share this dependency and we can help out the network by pinning the source on our IPFS node.
• We can convince the owner to do it themselves.

Making it easy on us, someone has already taken the first route with this project. Forking the original base58 project and publishing the fork with GX version here, https://github.com/gxed/base58. So now, we can just use the published hash to attempt and install the import.

gx import QmWFAMPqsEyUX7gDUsRVmMWz59FxSpJ1b2v6bJ1yYzo7jY

update imports of github.com/mr-tron/base58 to the newly imported package? [y/N]

By answering yes to the prompt, you’ll prompt GX to rewrite the imports of the library to use the GX provided library. So in this case import github.com/mr-tron/base58 would look more like, import gx/ipfs/QmWFAMPqsEyUX7gDUsRVmMWz59FxSpJ1b2v6bJ1yYzo7jY/go-base58-fast/base58.It is recommended that you not publish your projects to GX with their imports rewritten. Instead, let that be dealt with by GX downstream.

GX will use your package.json file to track these dependencies and handle future rewrites / un-rewrites. You can find the list of your GX managed dependencies in the gxDependencies section, which looks like,

"gxDependencies": [
    {
      "author": "mr-tron",
      "hash": "QmWFAMPqsEyUX7gDUsRVmMWz59FxSpJ1b2v6bJ1yYzo7jY",
      "name": "go-base58-fast",
      "version": "0.1.1"
    }
  ]

Start using GX today!

That’s it, just a few commands and you are up and running… but… oh wait. You might hit some snags around project availability. At Textile, we collaborate on code and each can update the dependencies. This means, if I add a dependency and am the first IPFS peer to pin the source code, any of my collaborators might hit snags when they try and install it and the code isn’t available. GX does its best, but sometimes still has issues. To solve this on our team, we’ve been pinning all our GX dependencies to our shared IPFS server.

By pinning the code our work depends on we are also making the redundancy and availability of the network better. You should join the effort :)

If you are interested in more of our development work, check out our textile-go library or our textile-mobile library on GitHub. If you are interested in other thoughts on the IPFS network, check our our plans to add the next 1,000,000 peers to the network. And of course, if you just want to have IPFS running on your phone, storing and preserving your photos, or enabling peer-to-peer sharing, get on the list to try out our mobile app.