Building an open source personal knowledge base

Once we have made sense of our thoughts, there is often the desire to present these thoughts in a well-formed and easily navigable format for later reference. A plethora of tools exist today for taking these kinds of notes. In my opinion, none of them are satisfactory.

Cross-referencing and organization is already solved with the advent of the wiki. The wiki makes information easily searchable and provides a way to organize and connect information together. There are many wiki systems both for public use and for personal or small-scale use.

The personal knowledge base (PKB) is basically a wiki system for personal use. Unlike a public wiki which most often focuses on representing facts, the PKB can contain subjective material relevant to the person or persons using the PKB. Therefore the desired properties of a PKB are equivalent to those of the personal wiki.

Representing structured information is a solved problem on the internet, proved by Wikipedia and other information wiki’s. However, the PKB is lagging behind due to the difficulty of providing an easy to use user experience for data input. The challenge at hand is to find the most effective process to go from unstructured information (for example on paper) to structured information in a PKB for later reference.

I propose that every useful PKB system should have the following characteristics:

• A powerful organization scheme
• Linking information should be central
• Minimal effort for input
• Semi-automation of organization
• Provide powerful visualizations
• Support all media types
• Easily navigable
• Sharing
• Data export
• Open source
• Cross-platform

Below I will expand on these characteristics.

A powerful organization scheme

This is the characteristic which distinguishes a PKB from a note-taking application. The organization scheme should not be limited to a hierarchical schema or just a tagging schema. Instead, it should be powerful enough to link nodes together (I will refer to an entity of input as a node) and tag them separately. These two concepts together (linking and tagging) are very powerful and complement each other to provide extra information about the nodes and their relations (see the picture below for an example with three tags/collections).

Linking information should be central

The main means of organization should be linking to other nodes. Linking nodes should be easy and intuitive. Because of its importance, linking should be one of the main focuses of the UI.

Minimal effort for input

This is probably the most essential characteristic to get right. There is no point in focusing on the output of the PKB if getting things in there is a big hassle. We want input to be very easy for all types of media.

Semi-automation of organization

To allow the system to grow with as little effort as possible, it should take over some of the heavy lifting on the input side. Of course this process can’t totally be automated but the system could for example provide suggestions on which collections the current node is likely to belong to and/or provide suggestions on relations between nodes.

Provide powerful visualizations

This is one of the main advantages of using the organization scheme as described above. It allows the user to visualize his knowledge in a multitude of ways. The user could follow links in his PKB or visualize his data using the collections he assigned to the different nodes.

Easily navigable

We want to be able to easily navigate through our PKB and get to the things we care about as quickly as possible. This can be done by providing a good search capability and providing means of exploring the knowledge base.

Sharing

One of the main advantages of creating a web-based PKB is the possibility to share your work with other people. This provides for better collaboration abilities due to the ability to easily share your state of knowledge in a certain subject.

Data export

The system should provide some sort of data export in a generic format so that the user is not completely reliant on this specific PKB solution but can migrate to other PKB solutions if a better solution comes along.

Cross-platform

The PKB should be accessible on any platform (PC, Mac, smartphone) to make sure the information can be accessed from anywhere.

Open source

We don’t want to work with a knowledge base system for years just to hear that the vendor is discontinuing its services and that we are losing all of the information we have built up over the years. This also allows people with sufficient technical skills to run the software on their own servers if there are any privacy concerns for the user.

What is out there

There are a lot of applications out there that have at least a few of the desirable characteristics of a PKB as described. However, there don’t seem to be any (that I know of) that have all of these characteristics.

In my search for the right PKB solution for me, I have looked at several applications that implement some subset of the desired PKB characteristics I proposed above. Below I will briefly go through them along with what I think can be learned from them.

Note-taking applications (Evernote, OneNote, Google Note, …)

“Note-taking systems” are by far the most used today for PKB purposes. These note-taking systems usually provide a way of assigning categories or tags to flat notes and make them searchable.

The focus of these applications is on rapid input of any media type. In doing so, note-taking applications often sacrifice on organization features by making them purely hierarchical or limiting them in other ways. There is no focus on connections anywhere in the system which severely limits the organization capabilities.

Note-taking software is therefore not suitable as a PKB but has more of a loose note use-case where the notes themselves are not interconnected in any sophisticated way.

Personal wiki software (MediaWiki, DocuWiki, …)

The “personal” Wiki software in use today is usually run by companies wishing to provide company documentation and a handful of power users that put a lot of time into maintaining their personal wikis. These software packages have pretty much all characteristics of a proper PKB, but lack the ability for easy input and discovery. Also, the software does not tend to provide powerful visualizations or exploration tools which makes the output less useful.

Outliners (Workflowy, Org-mode, …)

Outliners usually focuses on easy input for hierarchical data. For a single document a good outliner is usually one of the fastest ways to create lists and checkbox lists with different levels. However, outliner software suffers from the same problem as note-taking software. The focus is only on quick and easy input which means the software sacrifices in its organization scheme and is therefore not sufficient to use as a PKB.

Mind mapping software

It is good to take a look at mind mapping software to see the potential for the visualizations that can result from a PKB. Mind maps are frequently used for mapping out quick ideas. However, it seems to be a powerful UI for PKB systems in general. Mind maps provide the user with a good overview of their knowledge. However, mind mapping software is usually focused on short-term input. There does not seem to be any mind mapping software that has the characteristics of a wiki.

Devonthink Pro

Devonthink has one very interesting feature that distinguishes it from all other PKB software. Devonthink makes use of an “inbox” for newly created notes in the system. Later the user can click on the note and Devonthink uses an algorithm for providing the user with suggestions for which collections the note belongs to and which notes it might be related to.

This satisfies the semi-automation requirement for a PKB. Unfortunately, Devonthink has a limited organization scheme (just folders basically) and is only available for mac. Devonthink is the only PKB software that I know of that attempts some form of semi-automation.

Putting it all together

To get a proper PKB we would like to take all the good parts of what is out there and somehow combine them into a system that satisfies all PKB requirements.

We want the easy input of note-taking software, the powerful output of a personal wiki, the powerful editing mechanism of an outliner, the visualization and UI possibilities of mind mapping software, and as much semi-automation in the PKB as possible. I believe that if someone is able to combine the good parts of all these different types of knowledge management applications, one ends up with a killer app.

My attempt at a proper PKB

About one year ago I started working part-time on my vision of what a PKB should be. Throughout the process I have kept the principles outlined above in mind and tried to combine them in a way that is optimal for my learning process. The project is still in active development.

I find that I am currently successfully using the application in my studying and will continue to do so and in the meantime improve on the software.

Views

Example view of my collection “Physics”

In the image above you can see an example of the “collection detail view”. In this view you see all the nodes in the collection “physics” along with their interconnections.

All of the collections in my PKB along with their connections

In the image above you see my “collection overview” view, which shows all of the collections in the PKB and their interconnection. A link from a collection A to a collection B indicates that whenever a node is added to collection A, it is also automatically added to collection B.

In the image above you see the “node detail” view where you can edit the contents of the node using a rich text editor. You can add different kinds of media from different sources, link to another node by selecting text, along with all the other features you expect from a rich text editor.

The project is named Geist and you can have a look at an early version here. The source code can be found on Github.

My main focus at the moment is improving on the UI to make the user experience more intuitive.

Get involved

If you are interested in the project and want to stay up to date on the latest updates and changes or have questions for me, you can reach me via twitter or sign up for the mailing list below.

Resources

• http://marcusvorwaller.com/blog/2015/12/14/personal-knowledgebases/
• http://www.acuriousmix.com/2014/09/03/designing-a-personal-knowledgebase/