Textile's primer on libp2p, part 2 (or why we really ❤️ libp2p)

decentralized Dec 12, 2019
Photo by Jordan Harrison on Unsplash

In part 1 of this series, we started with the question of “how to approach your first p2p application?”. After some quick initial thoughts, we quickly realize not relying on a centralized server, and focusing on making the app peer-centric comes with multiple additional complexities. The two main groups of “issues” are application state and infrastructure / protocol diversity. Luckily, we found out we don’t need to reinvent the wheel by (re)solving a lot of these infrastructural challenges, and instead can leverage the amazing libp2p p2p networking stack/library.

In today’s post, we’ll take things a little bit further, and present a “toy”-app to get a feel for actually developing something with libp2p, and hopefully, motivate you to build your own p2p app. Seriously, you’ll be amazed at how easy it is!

The App

First off, our application today is going to be written in Go, using the go-libp2p library. If you aren’t familiar with Go, we highly recommend you check it out. It is really good for applications that rely on currency and networking (for example, like handling multiple p2p connections!), and most of the IPFS/libp2p libraries have their reference implementations written in Go. For a great introduction to Go, check out the golang.org tour.

Our app is going to be a simple Ping-Pong application, with some added tweaks to make it more interesting compared to the usual naive examples. Here are some points about our app (don’ worry, we’ll include details about these points later):

  • The app binds to a free TCP port by default.
  • If the "quic" flag is provided, it will also bind to a QUIC listening port, which will be the preferred address of the peer to play Ping-Pong.
  • The peer will use an mDNS discovery service to discover new peers on the LAN network.
  • On every newly discovered peer (e.g., Peer A), our app will run a custom sayMyAddr protocol (that we’ll implement), which asks Peer A to tell us its preferred address for playing Ping-Pong.
  • We connect to Peer A using its preferred address and start the Ping-Pong “dance”. Putting it differently, we’ll run another custom protocol in which we send a Ping message, and Peer A replies with a Pong message. Cool!

Even with this simple back-and-worth setup, there are several decisions we need to make while developing our p2p app. We can identify the following initial concerns:

  • Which underlying transport protocols (e.g., TCP, QUIC) to use?
  • Which peer discovery mechanism (e.g., mDNS) to use? — i.e., how we get to know about other peers using our app?
  • How will our custom protocols (Streams) work? — i.e., how will we support bidirectional communication with other peers?

All these app responsibilities are independent from one other, and luckily the modularity of  libp2p forces us to actually avoid coupling them together. Score one for good library design!

Jump into Code!

Now we recommend you jump right in and start playing with the app code. It is heavily commented to help guide you along in our journey of understanding! Despite that, we’ll also sketch out a general overview here, which is a great preamble for reading the code in detail. Feel free to clone the repo locally so you can get your hands dirty:

git clone [email protected]:textileio/go-libp2p-primer-article.git
cd go-libp2p-primer-article
code . // We like VSCode, but you do you ;)

Next, you should start with main.go where you can see how a libp2p host is bootstrapped. Additionally, here we specify which underlying network transports our host will support. Notice that if the -quic flag is set to true, we add a new binding for the QUIC transport. Adding further transports is as easy as adding additional options to the host constructor! Notice also that we register here all the handlers of our custom protocols: RegisterSayPreferedAddr and RegisterPingPong. Finally, we register the built-in mDNS service.

Jumping now to discover.go, we find the logic of the mDNS setup. Here we should basically setup the frequency of the mDNS broadcasts and an identifier which, in our case, is an empty one. The last step is registering a discovery.Notifee, which will be called whenever mDNS triggers a peer discovery, providing us with their info. In our case, this will trigger:

  1. If we already knew about this peer, do nothing; we’ve already played Ping Pong. Otherwise…
  2. Open a Stream of our SayPreferedAddr protocol to ask the discovered peer which address (addr) on which it prefers to play Ping Pong. Finally…
  3. If that isn’t the case, we add their addr to our address-book, close the current connection with them, and run our second protocol PingPong, which will reconnect with the peer using their preferred addr (which we added in the last step).

Finally, in pingpong.go we can see the RegisterPingPong method we mentioned earlier called from main.go, and two more methods:

  • Handler: This method will be called when an external peer call us to run the PingPong protocol. You can think of this as an HTTP REST handler. In this handler, we receive a Stream, which implements io.ReadWriteCloser from which we can run our protocol to send and receive information to do something useful.
  • playPingPong: This is the other side of the coin; the client starts a new Stream to an external peer to run the PingPong protocol.

As you can see, defining protocols is quite easy and completely abstracted away from other infrastructure tasks. The only thing we need to care about is writing the code that is useful for our app. Consider that adding a new protocol, such as in saymyaddr.go, is very similar to pingpong.go.

If you’re interested in the fine-grained details of the code, you can look in the comments, which points out some important things you should probably have in mind while using libp2p.

To actually run this sample app, you can open two terminals and just run: go run *.go , go run *.go -quic, or combinations of these. Here you can see a demo with two terminals running with the -quic flag:

Notice how the lower terminal peer discovers the top terminal peer as soon as runs, since mDNS will immediately discover existing peers. Then it jumps straight into playing PingPong. Due to the 5 second delay we configured for our mDNS setup, the top peer will eventually also discover the lower peer by its own means; where it will trigger a new PingPong game.

Finally, notice that when each party sends or replies to/with a PingPong message, it gives detailed information about the multiaddr it is talking to, where we can see that it is using the QUIC protocol. Try running this example without the -quic flag for both peers and see how this affects the output!

Notice if you run one terminal with the -quic flag, and one without, the latter peer can’t play PingPong with the former, because it doesn’t have QUIC support enabled. In a more realistic scenario, you would leverage all remote peer addresses you have available to have more chances to speak within an underlying transport protocol that both understand. Neat right?!

Next Steps?

An important part of an application is its maintainability. In p2p applications, communication is at the heart of the application logic. An example of this is having powerful and simple ways to manage protocol changes. If someday in the future we want to augment our PingPong protocol with extra functionality or features, we need to consider that old peer versions will still run the old protocol version! This sounds like a nightmare, but fear not, we’ve got it covered. Here’s where you should note the following code snippet from pingpong.go:

const (
    protoPingPong = "/pingpong/1.0.0"
func RegisterPingPong(h host.Host) {
    pp := &pingPong{host: h}
    // Here is where we register our _pingpong_ protocol.
    // In the future when you add features/fixes to your protocol
    // you can make the current one backwards compatible, or
    // you'll need to register a new handler with the new major
    // version. If you want, you can use semver logic too, see
    // here: http://bit.ly/2YaJsJr
    h.SetStreamHandler(protoPingPong, pp.Handler)

The comments pretty much cover it!

Another important example is related to our peer-discovery mechanism, mDNS. This protocol serves our purposes on LANs, but what about discovering peers on the wider Internet? You can later add a Kademlia DHT,  or use some type of pubsub mechanism to find out about new peers as well.

What’s important to note here is how all these future features are things you can add to your application, and doesn’t force you to change existing code or behavior. Making an app easy to change is a sign of good design, which means that libp2p is also helping us to structure our code using good practices. Thanks for that libp2p devs!

Closing Words

libp2p has many built-in tools to solve most of the hard problems you can face in a p2p system. We recommend you to take a look at the implementations section of the official libp2p webpage to get a grasp of what’s available, and on the way. Things move fast, so its a good idea to stay up-to-date with the latest and greatest stuff.

Caveat: Please also keep in mind that if you’re using libp2p with Go Modules enabled, it is mandatory that you should go get a specific tag, since relying on default behavior might fetch an unwanted or at least unexpected tagged version. You can find more info in the Usage section of the go-libp2p readme.

We hope you enjoyed taking a look at this toy-app, and we hope it makes you feel more confident that writing p2p applications isn’t as hard as you might think! In fact, it could be quite fun and empowering! If you like this stuff, join us on our public Slack channel to talk p2p protocols, or follow us on Twitter to learn about the latest and greatest Textile stuff. At the very least, stick around and read some of our other articles and demos to get a feel for what is possible in this exciting world of p2p apps!