Categories
Software

The Sad State of Personal Knowledgebases

A Personal Knowledgebase (PK) is any system that you use to store and retrieve general information. The key here is that the system should be capable of storing a large amount information on any number of topics and, ideally, provide some type of way to view relationships between the information.

Historically this has been the realm of pen and paper. More recently, people have used their computers to create folders of documents that they then search when they need to find something they stored. With the dawn of the Internet, the wiki came along to take advantage of hyperlinks. For the most part, that’s where progress stopped.

Today most people don’t use a PK but they do, it’s almost certain to be Evernote or OneNote or something along those lines, basically a flat list of notes that’s easily searchable and taggable or folderable. Power users might use a personal wiki. 

To me, all of these seem comparable to using a roll of toilet paper to write a book. You can do it, but there are better ways. Some of better options exist now, but I think that we’re still far from having a great personal knowledgebase.

In a perfect world a PK would have the following features:

  • Good search.
  • Unlimited size. Since it will be used to store just about everything you want to save for your whole life it needs to handle getting big well.
  • Simple to use. It should have zero learning curve for someone who just wants to dump a bunch of notes in it and a fast learning curve for anyone wanting to use more powerful features.
  • Convenient and fast. It should be available online or offline on your phone or tablet or laptop or wherever else you might want to use it. Adding content to it should be as close to effortless as possible and accessible from within other apps.
  • Structured. It should work fine without any organization but should allow for very flexible relationships between notes and, now that basic AI is becoming more viable, it should suggest relationships intelligently.

Surprisingly, no software with all those features exists yet. There are some interesting options though:

  • TheBrain – I’ve used this one for quite a few years. It’s stagnating somewhat these days but from what I hear, a full rewrite is underway and will be launched sometime in 2016. I’m curious to see what they come up with. Jerry’s brain is the canonical example of TheBrain in use.
  • Kumu is a new and interesting take on the idea of a PK. It’s mostly geared toward network visualization now, but I think there’s a lot of potential there.
  • Inforapid is new to me, but has been around for some time now. It’s Windows only, so I haven’t tried it, and the UI and website seem quite dated but otherwise it seems interesting.
  • Faqt seems to be going for a kanban / SCRUM style card-based layout.
  • PiggyDB is a somewhat complicated PK. It’s not for me, but it has a community of people who love it. It seems to focus on structure over visualization.
  • Curio and Tinderbox are two others that have been around for awhile. They’re more project-based, but are close enough to being PK’s to make the list.

The key point here is that years after “Web 2.0” we’re long overdue for a very good personal knowledgebase. I don’t have a solution, but the problem is worth bringing to light. If you build this or can point me to it, I’d be happy to pay for it!

beaded formal dresses

51 replies on “The Sad State of Personal Knowledgebases”

I’ve been a long time user of DEVONthink: http://www.devontechnologies.com. It can be many things depending on the features used, including file organizer, email organizer, document folders and tagging, etc. My favorite feature is the Magic Hat, which will suggest folders with similar documents.

Yep, Devonthink was pretty good. But cross-platform is a requirement these days… especially mobile. I have gone back to plaintext notes… which is so painful since it won’t work with attachments.

There are some open-source clones of Evernote… but they lack mobile apps for capture on the go. That and Markdown.

Gonna give Faqt a spin. Thanks for the reccos Marcus!

I have used Tiddlywiki some and for what it is, it’s really cool. It doesn’t work as a general purpose knowledgebase for me though because everything is stored in one file and it gets unwieldy quickly. After several years use, my current knowledgebase has 10k+ notes in it and is over 1gb when you include attachments.

When you got to that size was it general slowness for things like searching or do you mean organization wise you had to sift through too many items?

At the time I was using it it was front-end only. The file size got big and it started to slow the entire browser window down. I just read in another comment that it can be node-backed, so that should largely be mitigated.

Thanks! That’s a cool one that I’d never seen. I like outlines, but I feel like there are so many better ways to visualize and structure information that they miss out on. TheBrain comes closer to how my brain things about information organization.

Linking from one note to another one that is deeply nested elsewhere is difficult with outlines.

It takes some time to get used to Workflowy, but there’s great value in its simplicity. What I especially appreciate are the powerful keyboard shortcuts.

Came here to say this. Workflowy feels natural and simple. I use it for my consulting business, for personal use, to share lists with people.

I use Trello, I create boards for different domains (blog, house fixing, consulting time sheets etc..) and columns for subject breakdown and sometimes flow/pipeline representation

I agree, something needs to be done. To the features you’ve mentioned I’d add:

– easy to type fast math formulas

– ability to compute within the system (the more the better)

Actually those two, convinced me to write my notes in Mathematica, which at first looks like a bizarre choice, but I’m very happy with it (got already about 100 notebooks on various topics, mostly connected with science), plus I’ve customized my stylesheet and bridged it to Anki for remembering stuff, without creating flashcards on my own (I generate them from notebooks). Here are the project files if anyone’s interested https://github.com/masteranza/MathematicaAnki (OSX only for now)

Marcus, I use Evernote for this very purpose. It stores all kinds of information and a flat list is not so bad: the search and categorization more than make up for it. The only problem I see with it (and it’s a small one) it’s that Evernote is not good at content hierarchies (i.e. you cannot link notes to one another).

Vladimir, I also use Evernote but I find that when I put stuff in, I almost never see it again. Sometimes it shows up in the “related notes”, but mostly it’s just a black hole unless I specifically search for something.

The ability to add and visualize relationships between notes is the biggest thing that Evernote and OneNote are missing. You can put stuff in and search for it, but options for structuring information are very limited.

I find the simple grouping it offers to be adequate and I regularly go through past entries. Even relationships between entries is no stand-in for regularly re-reading the things I wrote.

I do, however, miss certain things I’ve taken for granted from other systems. It’s why I started coding an alternative. Something open source that should give people the ability to add plugins the same way we do with WordPress etc.

Great article. I would add, that your feature list misses one important feature – sharing the knowledge with others (typically teams, or other organization units). If for personal use, you can adopt existing software (more or less) I never seen good KB for teams.

Co-founder of Faqt, thanks for the mention! The lack of good options
is what drove us to create our own solution, with our primary/religious
focus being on simplicity. At the risk of self-promotion, we have been
quietly beta testing with several organizations a collaborative/team
version. If you’d like to try it, there’s a free signup at https://app.faqt.co/team/signu…

Let me know if anyone has any question: hello@faqt.co

Co-founder of Faqt, thanks for the mention! The lack of good options is what drove us to create our own solution, with our primary/religious focus being on simplicity. At the risk of self-promotion, we have been quietly beta testing with several organizations a collaborative/team version. If you’d like to try it, there’s a free signup at https://app.faqt.co/team/signup

Let me know if anyone has any question: hello@faqt.co

I think Tinderbox is worth a look here; its learning curve should not be formidable for the sort of users that visit this page regularly! Note that Tinderbox, DEVONthink Pro Office, and a bunch of other artisanal tools are on sale this week for the Artisanal Software Festival: http://www.artisanalSoftwareFestival.com/

I apologize in advance for this… I’ve used PersonalBrain for many years, and have been increasingly frustrated with ‘the sad state of personal knowledgebases’ for about half of that time (around when the company started moving towards complexity to justify ‘pro’ features). And it doesn’t help that the entire space has been fragmented for so long that there are factions of people who are firmly committed to the way their fragment works.

I apologize, because I think we’re right on the edge of some really revolutionary knowledgebases, and I’m going to share some things that are out of reach for personal use.

Has everyone here seen Palantir’s software? This is just one example: https://www.youtube.com/watch?v=dVsx4I8gkKk and another: http://www.bloomberg.com/bw/articles/2012-11-27/from-wartime-technology-comes-hurricane-relief – it’s HTML5 now, and priced for governments.

How about Lumify? http://lumify.io It’s open source, but built on such a massive tech stack that they have difficulty keeping the demo up and the discussions threads are littered with posts where people work for days to get it running to no effect.

Most of the solutions we’re talking about are based on ‘graph theory’, and we have a number of graph databases (that is to say; databases that are wickedly optimized for connections). Let’s take the top frontrunner as an example: Neo4j ( http://neo4j.com ). It’s open-source, easy to install and run, has already powered some huge projects, and is poised to tackle many more. Unfortunately, since it’s a back-end tool, there is no available personal knowledgebase based on Neo4j.

There are, however, projects like http://the-codex.net : https://www.youtube.com/watch?v=_R0ESfLBuHo
the-codex’s code is not publicly available, but it uses a relatively simple tech stack on top of Neo4j in order to do all it’s magic (I’m not being flippant here; the-codex.net feels like magic).

The technology exists today to capture every interesting fragment of data, give it permissions, connect it to any other bit of data (even as deep as connecting a building to a scene from a movie, then zooming out and seeing where else on a map that movie was filmed, where actors were on the street during each scene, what else was happening around them at the time, and so on.), give *those connections* permissions, make everything securely accessible to any ‘smart’ device, or even to other people’s interfaces (collaborative family history, anyone?), and make it so easy that people find themselves logging in just to offload and clear their mind or discover a new story about something they already thought they knew.

Why don’t we currently have personal knowledgebases that live up to this? Good question.

AFAIK people tend to think of a single piece of software when they come across the cognitive input “personal knowledge base”. AFAIK IMHO there is no such “single” thing. It is a personal process that lets you maintain your personal knowledge base using any whichever way that works for you, individual leeway and deviation allowed, not too far from normal.

I use CINTANOTES which is made by a Russian software company and it works out for me very well. I used to use Evernote but I found it too bloated. Cintanotes box upon box concept and search as I type works out really cool and I am not in the payroll of Cintanotes developers even.

You could even use Notepad and a very large containing folder with many hundreds of text files to achieve the same purpose. I also use TREEPAD LITE freeware, sometimes that kind of UI is required.

Personally, to anyone looking to optimize their personal process of knowledge bases I would recommend CINTANOTES and TREEPAD LITE together with a homegrown process as in “connecting software with people while accounting for differences and deviations in individual working habits”. YMMV what works for one may be suboptimal productivity for another!

Thx
L.Mohan Arun
“The speedy emailer”/Full Stack Technology Consultant, Architect and Developer/Copyeditor
Blog: http://www.mohanarun.com/wordpress
Facebook: http://www.facebook.com/seekspecial
Incoming Texts/SMS only: 209-231-8759 (San Francisco number)
Newsletter: http://www.tinyletter.com/mohanarun

I’d never heard of Cintanotes, it looks pretty interesting. It seems like it has good search and an interesting presentation, but lacks a good visualization of the relationships between notes and like you mentioned, is unfortunately Windows only.

All notes are one below the other. It works well for me on Windows platform.
To get visualization of notes is a far call that will probably add to bloat…
You want to quickly jot down something on your mind, just as I do and
be able to refer to it later!!
The typical use case does dismiss the idea of “good visualization of the
relationships between notes”… it is probably a feature used by less number of ppl.
Address of Sitex OOO: from Russia they make this quick and handy software app for a price. Its not free, and it feeds the Russian development team as well as the testers.
Thx
L.Mohan Arun

I think visualization is something that you don’t know you’ll miss until you have it. That said, it sounds like Cintanotes works great for you and if that’s the case, stick with it!

I have a bad habit of putting data into spreadsheets. I come across a list in some article, or a table whose format I don’t like, and I’ll copy, paste, search and replace, and stuff it into a spreadsheet in a format that I like. Some of these I post on my blog. Some of them just get stuffed into a folder and forgotten.

I collect pictures off the internet and stuff them into a folders. I might post some of them on my blog, but mostly they just go into the folders and are forgotten. I’ve been thinking I need a database to organize them, but that would take a certain amount of effort. And could anything I create possibly compete with Google? Whenever I need a new picture, where do I go? Google or my collection? Google, because I’m looking for something I don’t have.

Do I even have enough personal data that I need my own knowledge base? Most of the information I have collected is more like stuff that should be public.

Private information? Google’s gmail handles my contacts well enough, the bank keeps track of my money. The only thing I can think of that would require having my own copy is music, and that is a hassle anyway you slice it.

A personal knowledgebase shouldn’t be about hoarding information or redundantly saving stuff that you can easily find online. It’s more about keeping a record of what you’ve come across and how it’s relevant to you, your work and things you’re interested in.

If I add a Wikipedia article to my PK, it’s because I want to relate the article to something else I’m interested in. Once there’s enough stored in a PK, patterns start to emerge and ideas for new projects, stories, articles, etc. start to form.

the ability to extract patterns as ideas for new projects would be the most important feature of PK; otherwise evernote is good enough…

Great article Marcus. An automated tool for capturing and sharing knowledge will truly make the world a
better place. Learning can be faster , reducing the cost of education. People across the globe can connect based on topics of interest and knowledge, yielding more economic and job opportunities.

A personal knowledge base is far more valuable than what it might seem at first glance. Very happy to see you have articulated clearly and made people realize it. A PKB has great potential for knowledge transfer within organization or even at research labs for new people to familiarize with literature. Many people want to capture their thoughts for writing a book someday, but current tools aren’t good enough for this. Sadly, a person’s social graph is well captured today rather than their knowledge graph.

With these thoughts, I started implementing Hyperbook – http://www.getbook.co few months back. Currently it is in beta and really working hard to improve it every day. We got good feedback and reviews from thought leaders in content curation and knowledge management industry. Thanks for bringing the problem for discussion. It gives me motivation that is very much a worthy problem to solve.

I’ve been using WikidPad (http://wikidpad.sourceforge.net/) for years. Sure, it leaves room for improvement, especially when seeing the possibilities of web apps these days.
BUT:
* everything stored in open formats, TXT + SQLite. I want to keep my notes accessible even when the software itself would stop working or existing.
* fast. Any delay will cause me taking less notes throughout the day.
* open source, easily extendible to suit your own work- or mindflow. Because there are no two persons thinking in the same way.
* markup-based. I don’t want to waste time formatting my notes, I just want to put the information there and get on with the job at hand.

Whatever site is your favorite, use this smartphone/smartwatch/web app to access your content by voice command. Use cases: 1) quick access to your to-do list from your smartwatch by saying “show me my to do list”, 2) knowledge retrieval from a wiki by asking “how do I calculate xyz”, 3) photo retrieval by voice and more. Send your use cases: support@treycent.com

As the size of a database grows, the amount of work needed to maintain it grows exponentially. In today’s world, there is an expectation of zero learning curve, as you’ve expressed yourself. What is a learning curve but time spent? If you need to spend a lot of time maintaining these connections, why prioritize for a fast learning curve in the first place?

I think there’s a tendency to like new products (I’m one of the many in these comments to have switched from Evernote to Workflowy) simply because they’re clean and new and manageable. As they get larger, they become disliked and people will find a newer and shinier, but more importantly, emptier storage system and claim that it’s easy to organize simply because it doesn’t need to organize very much. Rinse and repeat. Now, that may not be true of this readership, who are willing to put the time and effort into this organization, but it’s true of most users.

At the cost of being self promotional, I’ll share my solution, which is just to remember everything. I built a Chrome extension called Hibou that allows you to store everything you come across that’s interesting. You highlight what you want to remember and it uses spaced repetition to remind you at optimal intervals when to reread what you’ve highlighted. You can organize your content with tags, but you don’t have to because it surfaces old content right before you’re going to forget it. Then when you review it you can make mental connections with new and old material that might not have dawned on you before. This kind of passive review makes it really easy to add content and review content without ever having to manage it.

Willlma, I see what you’re saying with the expectation of no learning curve and the tendency to jump to whatever is new and shiny only to be faced with the inevitable difficulty of maintenance once it grows big, whatever “it” is.

I don’t think low learning curve and powerful maintenance tools are mutually exclusive, TheBrain is actually pretty successful in that regard, but most other tools don’t even try. Either way though, it does require effort and you have a point that most people probably aren’t invested in their tools enough to put in the effort.

I’ve been using Hibou for a couple days, a friend recommended it to me after seeing an early version of this article. I haven’t used it enough to say for sure that I’ll stick with it, but my initial impressions are very positive. I use Anki for learning languages and I really like how Hibou is essentially a no-effort card creation system for everything else. I’m excited to see if it leads to the type of results you describe. I’d love to see it work inside of Marvin or Hyphen on iOS and I bet the devs of either of those two apps would be willing to do it.

Quite timely, I’ve just been evolving my strategy for doing this over the last couple of days. What I started with 10 years ago I called my “Personal Search Engine”. It was essentially a key/value pair system that allowed you to associate a list of keywords (quickly) with each key. I used it for website passwords, people’s addresses, phone numbers, etc. I might search for “dad work number” and the top result, usually, would be exactly what I wanted. When I worked at my first software company after university, I created something called “kbdocs”, which was similar, except that each “value” was a wiki document. Keywords sound old and crusty and “yesterday”, but believe me, they can be incredibly powerful for quickly accessing information in a personal knowledge base. For the last 5 years, I’ve worked for a company called Wolfram Research, and they use something called “Notebooks” in their flagship application. I’ve been without a good way to string these things into a good knowledge base for the last few years, but in the last year I’ve started to move in that direction. But the realization in the last few days, as I’ve started to struggle more with my “memory” as an aged 35 year old, is that I need to have an object-oriented notebook system. The point is this: You have a notebook for every “thing” in your life that you want to associate information with. If you read an academic paper, you have a notebook designated to that paper. If you have a person that you’ve met, you have a notebook for them. If you have a project you’re working on, you have a notebook for that. If you have learned a new term like “Partially Ordered Set”, you have a notebook for that. Everything gets a notebook. Step #2, for me, is to allow all of those things to be accessed via a “natural language” interface that allows you to type something in, press ENTER, and like the “I’m Feeling Lucky” of Google, it immediately opens up that information. The point is this: Search is cool, but 95% of the time, it should be unambiguous what you’re referring to when you type something in, and it’s electrifying to be able to immediately jump to your document via typing something in. How does this work? For me, I define linguistics for every notebook I create. Take a person. I might define the linguistic: OR[“gerry”, “gerry lewis”] For a term I’ve learned, I might say OR[“poset”, “partially ordered set”]. The system also allows “synonyms” to be defined and re-used all over the place. These notebooks, when used for projects, can also function as a project management system. Because the state of the project is part of that “state” that you want to associate with the thing. In that case, I organize my activities around the notion of a “stack”. I have a “Stack” section of the notebook and I add “new things to do” at the top of the stack. (which is a list of bullet points) When I start work on something, I create a new section of the notebook above the “Stack” section which is the name of that task, and I then create a new notebook for that task, linked to from the parent notebook in the relevant section. This again emphasis: Every thing, every object, every task, gets a notebook. And finally… once you get the “natural language” interface working well, it can evolve into a full fledged virtual assistant, allowing you to type in “commands”. They might be things like “email Meredith” which knows to compose an email using GMail since it’s a personal email, or “email Bob”, which composes an email using your work email program. It might be “stopwatch” which adds a stopwatch to your notebook to keep track of how long you’ve spent on something, or “create library function StringPositionToLineNumber[string, stringPosition] ‘returns the line number in the string that the given position occurs in'”, which defines a new function in a source code file for you, opens that file in your editor, creates a new notebook for working on that function, and arranges things on your screen appropriately. The point I’m making is that a “personal knowledge base” is actually a subset of a good natural language based personal knowledge base. I’ve constructed one in the last year and I’d never go back to working without it. It’s all about minimizing the friction between having an intention in your head and actualizing that intention. (the whole story line of “tools” basically, right?) Personal knowledge bases are an important part of that equation — although more focused on the “I want to remember this” part of that spectrum. Powerful stuff.

Very interesting. This sounds a lot like how I use TheBrain in terms of organizing stuff, but TheBrain completely missing the natural language aspects of your system. It seems a little tedious to have to define linguistics for every notebook, couldn’t it almost always be inferred based on the title? That seems like something Wolfram Language could probably do out of the box.

Also, someone linked to this video in the comments on Hacker News. It may give you additional ideas, I know it’s gotten me thinking: https://www.youtube.com/watch?v=VZQoAKJPbh8

Very cool system you have there, it’s hard to imagine exactly how it works without seeing it, but it sounds pretty effective.

Knowen.org . After reading your page this afternoon, Mr Marcus Vorwaller, I found the link above. I too have been searching for a knowledge base that I could use. Knowen uses directed acyclic graphs to organize content. There is an option to run your own server. Knowen is web based.

Is there a free alternative to TheBrain? It looks like a more advanced form of free mindmap softwares.
Mindmap softwares I used showed every node in a mindmap file. TheBrain shows only immediate nodes.

“Is there a free alternative to TheBrain?” Compendium is free open-source software that has been used for a variety of different purposes, and can serve well for some kinds of personal knowledge bases. Before using Compendium you will want to learn about issue-based information system (IBIS) notation; Compendium is optimized to build knowledge bases in IBIS notation.

I use an older version of Compendium, version 2.0 beta 1-NE, and it works great for me. My primary personal knowledge base is a large set of tagged files that I create and access in a variety of ways, but when I’ve worked on issues that need more rigorous and fine-grained exploration using IBIS, Compendium has worked well.

Leave a Reply