Taking your first steps towards building a decentralized application is hard. Changing the centralized design mindset isn't easy since it breaks most of the assumptions baked into our brains (and the software tools we use). To help illustrate how our team thinks about peer-to-peer communication, we're releasing a two-part series dedicated to the libp2p protocol stack. In this first article, we reflect on some of the complexities related to building decentralized applications and exploring now a network protocol layer abstraction such as libp2p can help us. In the next article (coming soon), we'll use these same concepts to create a simple Go example to understand how the various components we talk about here today fit together.

Skyrocketing complexity

Integrating peer-to-peer (p2p) communication in any app is hard. After just a few minutes of imagining how your app should work, things start getting complicated, very quickly. Consider two main problems for p2p applications: application-state, and communication infrastructure. Application state management isn't trivial. There isn't a central authority that defines the current state of the system. The system state is the convolution of multiple peer states, which has exploding complexity in unreliable networks and complex protocols. On the communication infrastructure side, your application needs to talk to multiple peers, so you'll carry a heavy bag of challenges. Think about the following insights about remote peers:

• They have unreliable hardware and network.
• They have uncertain computing power and storage.
• Firewalls may block them or live in NATed networks.
• They're running old application versions.

Before you start writing your first line of code, you might end up just freezing up due to the overwhelming amount of work ahead of you. Even if you manage to have your first application release, how easy would it be to change something without breaking things? Wouldn't it be nice to have some library that makes this task easier?!

Libp2p to the rescue!

Libp2p is a library that emerged from Protocol Labs' work on IPFS. If you've been following this blog even a little bit, you know we're big fans! When you tackle writing a production-grade p2p application from bare-bones, you happen to realize you're not only building the app but its infrastructure – and you're going to very quickly find you have to reinvent a lot of wheels. Not fun. On the other hand, libp2p lets you stand on the shoulders of some pretty huge giants, to decrease infrastructure complexity so that you can concentrate on business logic. Very fun! Sure libp2p isn't a silver bullet for tackling all our p2p monsters, but it will certianly alleviate the communication infrastructure burden.

Libp2p core concepts

At the core of libp2p is a Host, which models our local peer in the p2p network. A high-level description of its components:

• An identity to be identifiable to other peers.
• A set of local addresses where we can be dialed.
• Books to store other peer's information, such as their ids, keys, addresses, etc.
• A Network interface to manage connections to other peers.
• A Muxer which allows using single connections with multiple protocols (more on this later!).

The next big concept in libp2p are Streams. A Stream is a one-to-one communication channel with another peer. It's crucial to understand the difference between a Stream and a raw network protocol such as TCP or UDP to grasp the power of libp2p: you can think of a raw network protocol as an opinionated way to send bytes through the wire. If you need high reliability on packet delivery, you can use TCP, or if that isn't the case, UDP might be better. Now, raw network protocols don't have any semantics on what data is being sent; they're just bytes.

A Stream on the other hand, is a stream-based communication channel between two peers that has defined semantics. They're not just bytes, but bytes that respect a developer-defined protocol. On that note, a protocol in this case is identified by an id such as /sumtwointegers/v1.0.0. A Stream is conceptually a conversation within this protocol. For example, one side sends two integers, and the other replies the sum of it. That's what we mean by byte stream with semantics.

Streams run over raw network protocols such as TCP or UDP – in fact, the idea is to actually decouple the p2p communication from the network protocols. We just need to think about having a stream-based channel to send meaningful information with the flexibility of running on whatever raw network protocol might be available between the peers. This is really nice, and really powerful. It also means things can be optimized at every layer of the stack. Better network protocol handling? Great, leave it to libp2p. A more semantically meaningful p2p protocol design? Cool that's do that too!

On top of this, decoupling the semantics from raw network protocols allows libp2p to go further, and multiplex Streams in the same network protocol. We may use the same TCP connection for many Streams. We can run a multiply-two-integers protocol in the same connection where our sum-two-integers protocols were running. But this isn't something that Streams need to handle on their own. Libp2p actually relies on Muxers to do this magic. It's the muxer's job to separate bytes to the corresponding streams from the same raw stream. Here's a quick diagram to help explain this a bit.

As we saw in the above bullet points, our Host has a Muxer, which multiplexes multiple streams with a peer in the same connection. You can think of what it is doing as prefixing messages of different flows with some identifier, so the other end can identify bytes from different streams. Of course, the underlying raw network protocol used might limit some things about multiplexing, for example, head-of-line blocking. Or the complete opposite might be the case, and the transport itself has a full multiplexing solution (such as QUIC). But that's another conversation...

Taking a step back

Let's try to separate what we should be thinking about while writing the application, from what libp2p is doing for us. From our side (the developer), we want a clear way to think about things, trying to separate our application logic as much as possible from infrastructure concepts. What we should think about is dialing peers, not addresses. Then just designing our communication at the Stream level abstraction.

In the background, libp2p is doing the heavy lifting of dialing the remote peer using the information stored in the addresses book (second point in the bullet-list above). It tries to resolve which network protocol we both understand to establish a network connection. Slick!

It may realize it isn't necessary to dial because we already had an open connection. When we run a new Stream with a peer, the muxer will do the heavy lifting of mixing it with other existing Streams in the existing connection.

Why so much insistence on re-using a single connection? Turns out, our p2p apps must live in very constrained network environments where firewalls, NATs, connection limits exist. After we achieve establishing a connection, we should take advantage of it as much as we can! There many unknown unknowns that you can find when switching to a p2p mindset, so using libp2p helps you focus on good design, even if you aren’t an expert.

But there's more!

Libp2p was designed to be modular. This means making it easy to switch implementations of components without affecting others.  I think the rationale behind this decision – besides a good engineer practice – is acknowledging diversity in the environments where our app will run. And you can bet this happens in p2p applications. Our app will run in very different runtime environments, with different capabilities or network restrictions. Also, evolving your app while maintaining backward compatibility with older app versions isn't trivial.

For this reason, when two libp2p apps talk with each other for the first time, most of their work is resolving which overlapping compatibility they have to make progress. This includes which network transports both talk (TCP, UDP, QUIC, Websockets), protocol versions we can handle (/myprotocol/ v1.0.0, /myprotocol/ v1.1.0), which muxer implementations we can both leverage, security negotation, etc. And if that wasn't enough, libp2p has built-in protocols to resolve everyday needs of p2p applications, such as:

• NAT traversal: a pain in p2p applications
• Peer discovery: how do you actually discover new peers without a central authority?
• Pubsub: having a publish-subscribe mechanism to send messages in our application without having to know all existing peers or flooding the network
• And much, much more!

A simple bonus/caveat: Libp2p is growing very fast, so you can expect that new powerful features will be added that you can leverage with little coding work. Remember that p2p applications are about diversity, so libp2p characteristics should be considered different between language implementations. That's to say, some muxer implementations might not be available in JS but in Go, or Rust.

Conclusion

Now there's hope that we won't be overwhelmed by re-inventing the wheel in our application. We can focus on creating value in our application and let libp2p do the infrastructure heavy-lifting. Moreover, future improvements in libp2p will make our app better without having an impact on our main business-logic.

So that's pretty much it for now. Stay tuned for our next 'episode', where we will translate some of these concepts into code through a hands-on example. Also, if you're interested in this type of stuff, or want to integrate p2p communication into your own app or project, get in touch, and let's experiment together! You could also give one of your 'lite' libraries a go if you want an easy way to use a libp2p peer in the browser, iOS, Android, and/or desktop. Finally, if you like these types of overview article, please let us know, or make requests for additional topics... in the mean time, happy coding 👋👨‍💻👩‍💻.