Lately I’ve been coming across some Notational Velocity stuff again.
The first time I heard about NV, I was like probably 14. People were talking about it like it was the best thing ever, so I checked it out, but I didn’t get it at all. Of course.
I came up with the idea of making the search field huge at the top of the page, like on a Google results page or maybe even bigger. Several reasons, like search being a major feature of our system, but a major one is people are supposed to put in hash URIs, which are often long, so they need a big search field to do it.
I felt like a genius when I came up with this idea, but I had forgotten that NV had this user interface for ages before us. So that convinced me of it’s extreme relevance and now I’m looking into it again.
An interview with Notational Velocity developer Zachary Schneirov
Wow, remember this?
All existing apps either tried to place a multi-document drawer on TextEdit with a slew of buttons, made you file information into discrete fields or categories, had very imprecise searching, […]
This comment reflects the strength of NV’s underlying model or ideology. EarthFS has a slightly different model, which IMHO is even stronger (as you’d hope for a project 10 years later). We consider notes to be just files, and we have the best form of linking (I’ve come a long way to be able to assert that now), and we dropped the concept of “modeless,” which produced a very nice UI but didn’t really pull its weight. We added Dave Winer’s concept of River of News, which I think fits very well with search.
[…] or weren’t secure enough for storing passwords (the encrypted database was mandatory in the first version).
NV seems to have moved away from this over time, and EarthFS doesn’t try to support it at all. Either use a dedicated password manager, or encrypt individual files in EarthFS at the application layer.
I guess given some bloggers’ early promotion and consequent direction of its development (I’m very grateful to Giles Turnbull as well as Merlin), it makes sense that they would continue to see value in NV as a general-purpose writing app. So the fact that a note-taking program / password manager also happened to appeal to bloggers almost certainly helped spread the word.
EarthFS fits this niche very well and actually goes further by turning notetaking+sync and blogging into the same problem.
I’d love to see a program like one of those that accepts tons of filetypes, but still keeps everything organized in folders, not some giant database.
- Syndie and other apps built on I2P
- CouchDB (Currently used on the desktop as part of Ubuntu One)
Why isn’t EarthFS built on any of these? I guess I have to take a look at all of them.
EarthFS isn’t built on any of them because content addressing is simple and fundamental, and an application that implements it should be simple and fundamental too.
Apple’s SearchKit and Spotlight frameworks have high latency, search on per-word boundaries, and are unreliable at returning and ranking results. Unfortunately most mainstream note-taking applications seem to rely on frameworks like these for finding information across notes. NV, however, has its own incremental filtering algorithm which never searches the same part of a note more than once and can find text at any location relative to a word. And its brute-force approach makes it quite predictable — there’s no “intelligence” to second-guess the user. I tuned it for faster than realtime performance with 1000 notes on a 500MHz G3 and it hasn’t been a problem since.
This is very interesting given we just finally got our completely custom search system basically working and extremely fast. We’re still limited to word boundaries (no prefix/suffix matching, although we have stemming). Phrase search is hopefully coming.
I’m curious about what “incremental filtering” and “never searches the same part of a note more than once” mean.
I wonder how long his average note is? He keeps them small, so maybe 100 bytes? So 1000 notes is 100KB, which is trivial to keep in memory and probably no trouble for linear search.
When I was reading this interview originally, I was at 5000 notes. Now we’re up to 8000. Slowing down a bit, but that’ll change when we start storing other files and (hopefully) pulling from other people.
I should check how many files are in my collection of saved web pages.
$ find . -maxdepth 2 -type f | wc -l
$ find . | wc -l
That’s just the pages on my laptop, going back to 2014-06-02, so three months. Only around 1000 pages per month seems like less than I expected, actually.
Still, dumping 50,000 files on EarthFS is going to be fun. Hopefully we’re up for the task.
Remember though, we currently have around 16,000 files when counting meta-files. And most of the files in a web page wouldn’t need meta-files. So it’s not actually as bad as it sounds.
Maintaining an encrypted database of notes was one of the main reasons I built Notational Velocity. And though CoreData makes it incredibly easy to persist an “object-graph” to disk, to this day there’s no way add a layer of encryption beneath it. So NV serializes all note-data to memory, compresses it, and then encrypts it before writing it all out in a single atomic operation that’s protected by the HFS+ metadata journal. And to handle incremental updates (i.e., auto-saving every few seconds), it uses its own incrementally compressed, encrypted write-ahead log.
I wonder if SQLite could’ve worked.
Yes, it could’ve, although the encrypted SQLite VFS isn’t free.
NV is open source, so we can examine its database format if we care enough. It sounds interesting at least.
Though I’ve tried to clarify the app’s purpose and design as much as possible on the web site, I seem to have managed to cultivate quite a varied user-base. So I now find myself spending most of my NV-time maintaining features that just aren’t as relevant to me personally. And those things I do want to work on are continually getting pushed back.
I think we’re in a better position on this. We have a broader and more practical core purpose, and we’re leaving a lot of room for extensibility.
I overviewed all of the notetaking projects I could find on GitHub too… That was a really good idea. A ton of them were inspired by NV. But the main thing I got out of it was that content addressing for notetaking is badly needed.