HomeLog InSource

notes.public

[View] [Short] [Hash] [Raw]

2017-02-12

What’s been going on lately? Well, progress has been slow.

I mentioned[#] my research into consensus algorithms… almost a month ago? Well, on the one hand, I’ve figured out what I’m trying to do. On the other, I haven’t really been able to do it.

At this point, my goal has moved from “write a consensus algorithm” to “write an abstraction layer for consensus algorithms.” The idea is to write a minimal single-file library that has one universal consensus API, plus a variety of modular back-ends including Paxos, Raft and Bitcoin.

In theory, the interface of a consensus algorithm is extremely simple. You put in local events and get out global events in a uniform order. However, the details are always devilish. You have to worry about peer discovery, membership changes, networking, persistent storage, and weird back-end quirks like Bitcoin being implemented as RPC to a daemon process. All of the existing libraries I’ve seen have obtuse APIs of their own.

For libkvstore, I was able to use the LMDB API almost as-is, just simplifying and generalizing it a bit. For libconsensus, I’m trying to design an API from scratch, in a field where I am obviously far from an expert. It’s tough.

The good news is that the libkvstore distributed back-end is mostly done (barring a few tweaks), which means that once we have a consensus algorithm to plug into it, it’ll be pretty much ready to serve. And the SQLite port is at least functional, albeit buggy. So aside from the hard part, this might be the fastest ever development of a distributed database!

The path to get here: StrongLink -> libkvstore -> libconsensus.

It’s been a long time coming and there’s just one last Everest to summit.