Implementing Hierarchies/Namespaces in org-roam (V2)

Development
1

Hello everyone,

After taking a look at other tool called Dendron I was hooked with the concept of hierarchies, because they fits perfectly with my workflow and the type of content I write, so I want to show you what I’ve been working lately.

Why?

In my experience, pure zettelkasten is awesome for academic information, taking notes on books, and articles that are full of abstract concepts that don’t belong to any category but are inter-linked with other ideas. I use that a lot.

However since I work in the IT industry, I take a lot of notes on software and software architecture. This kind of information has a clear structure, a structure that repeats between information that belongs to an specific context. And here zettelkasten in my opinion is not the perfect fit.

I ended up having lots of notes like: data types in Java, data types in c… Repeating information in the title to be able to find specific notes.

This namespaces helps grouping and branching notes that are strongly connected but leaves the concept of atomic note intact, a note that you can still link with others that are related. Also, makes finding this references faster among all your notes.

You can find more on this rationale in the blog of Dendron’s creator, and some of the conversation I had about this in Org roam’s Slack

Disclaimer

I’m not writing this to suggest a new feature set to include in org roam, I don’t even know if this is aligned with jethro’s plans for org roam, I just wanted to share what I’ve been working on.

I was thinking on this more like a plugin for me and my friends who like the approach. Something that could work with the base project without disturbing the development

Currently this is just a bunch of functions in my config file that take advantage of org-roam code. I’m just learning elisp, so probably the code is not perfect, any suggestions or recommendations are more than welcome!

Features

Hierarchies are just filenames separated by dots “.”, each subsequent hierarchy is child from the former. Something like:
org.refile => refile is child of org
org.refile.cfg => cfg is child of refile

Each child is a note, so we can have a folder-like structure, but much less rigid.

Finding and creating notes

org-roam-node-find shows hierarchies in the minibuffer. If the note does not exists use the input to populate the hierarchy (embedded in the file name) and the title.

org-roam-find

spaces are changed for hyphens in the file name. You can also create notes without any hierarchy, like org roam normaly do.

Refactor namespaces

Useful to move group of files to another namespace name:

org-roam-refactor

Of course if you want just change the hierarchy of the current file you can do it with another function.

Time notes

This hierarchies help to create what I call Time notes, which is probably a bad name, let me know if you find something better :rofl:

Time notes are sub-hierarchies of other notes that indicates the time, you might want this notes under different other notes, journal notes are a great example. In this case it is integrated with org-roam-dailies, so we can still using the current functions:

journal.daily.2021.04.21
journal.weekly.2021.04.40

Or hierarchy where you capture meetings:

project.customer1.meet.2021.04.21.123020
project.customer2.meet.2021.04.21.183020

This makes notes easy to sort and easy to find. You can create time notes under any other note as well

Scratch notes are other way of looking at time notes. This type of notes are just a dump bucket where you want to capture something on the fly. It is nothing more than a top-level hierarchy with time:

scratch.2021.04.21.183020

Take a look at the example:

Dailies:
dailies

Regular time notes:
time-note

And scratch notes:

scratch

This notes are possible using org-capture through org-roam new capture functions, so it is easy to creae new types of time notes.

For me this works perfectly and let me find the information I need really quickly. This is just a basic feature set borrowed from dendron, I plan to add more of it but for the moment this has all what I most use.

Here is the gist with the code:

Let me know what do you think!

PS: I would say this is kind of related to the find node UI thread.

7 Likes
2

I don’t really have time ATM to dig into this, but with tags, which in org can have hierarchy, why the need for other hierarchy?

https://orgmode.org/manual/Tag-Hierarchy.html

3

Yes, tags in org are awesome, indeed that’s what I was using before for this kind of purpose.
However, at least in my case it’s not quite the same thing, I still use tags but for other type of relationship between notes.

This hierarchies (the name is inheritance of dendron) are namespaces, It reminds me of code packages. Here each sibling is actually a note and not a group of categories that can be in a lot of notes. It’s more like outline levels inside a file.

For me this has much less friction in searching, grouping and re-arranging notes. Even when I can’t use emacs but I want to read my notes.

Anyway this is something I personally find useful, but I understand this could not be for everyone and that can even be a bit redundant with other org features. I just wanted to share it in part because I think this is a good place to discuss these things, and think it can add value, and in part because it is my first elisp code and for me it is exciting haha

4

This looks super cool!
I’m also a programmer and was also considering Dendron before committing to org-roam.
Especially their example with “How does feature X work in language A, B and C” is something that I encounter frequently in slightly different forms.
I’m wondering if a directory structure would also work for creating the hierarchy? I know the Dendron developer was considering this at one point, but don’t know the downsides and feasibility.

1 Like
5

I’m sure this can be done with folders as well, where each folder has a note with the same name.

However I didn’t want to mess with folders, if eventually I want a folder structure I think it can be easily done from de file name with an script

I would say the downside with folders is a bit more of friction, and I think the refactor thing would be much more complicated

6

@spicy , I feel that it is a fruitful avenue to pursue :slight_smile: Thank you for sharing.

I just quickly read through Dendron’s documentation (excellent, isn’t it?) and your dendroam.el (I am learning a lot about object implementation within Elisp from it; thank you).

My quick summary of what the “Dendron paradigm” would add to Org-roam is:

  1. Create a hierarchy with file names (the dot syntax in the form of “root.domain.category.topic.org”)
  2. Re-organize the hierarchy with ease – it’s sort of an enhanced file re-namer to “refactor” the hierarchy
  3. Search in the hierarchy and quickly get to the note of your interest

As the hierarchy is based on filenames, I feel that Emacs already can do a lot about point 3 (search) quickly and easily (e.g. completion with regex, and even Dired)

My current filename convention is not compatible with Dendron. It looks like I should review it.

Very stimulating. Thank you.

2 Likes
7

Thanks @nobiot I’m glad you find it interesting!

To give the credit where the credit is due, the object implementation in elisp is thanks to @jethro’s code, I didn’t even knew that it was possible in elisp (or lisp languages in general). It so happened that I could understand it a bit and exploited it as much as I could :joy:.

I think you did a great summary of what this can add to org-roam (at least for me of course), and I’m agree that emacs core functionalities make really convenient to work with file names. The great thing is it can fit a lot of workflows.

For example I have a customer. top level hierarchy where I keep all the notes I take from customer meetings and all the planning or research I do for them. All my ideas and abstract thoughts are in my thought. hierarchy or my concept. one. There I just keep a 1 level namespace for all the notes, like a mini slipbox inside my slipbox. Thanks to this and emacs powerful searching I narrow results really fast.

I would love to know more Ideas you can came up with. For now this has all I need but I want to add support for the dendron’s concept of schemas as well, not sure how to approach it though. Eventually I would like to add support for headings as part of the hierarchy for searching and creation. Headings are already part of hierarchies, since they are just nodes, and they look like it were files, but my idea is more like having something like this in the minibuffer:
root.domain.category:Topic:Heading1
root.domain.category:Topic:Heading2

Where the first colon marks the title (this is how it is working now) and the second colon marks the title of a heading with an ID within the file

1 Like
8

@spicy

For a “schema” concept, I have tried a different approach as a proof of concept. I took it to be a list of categories in a hierarchy.

Sometimes it might be useful to get outside the Org paradigm a little – this time, I create a new file create function without using org-capture or org-roam-capture. Resultant files are ID’ed and part of Org-roam (they are nodes).

image
image1300×1179 44 KB

Searching with stock standard find-file

2021-04-25-dendroam
2021-04-25-dendroam1328×1218 285 KB

Creating a new note with a custom function (source on Gist).

You can create a new category in the hierarchy (Emacs prompts for confirmation). This implementation does not persist the hierarchy in a file, so the history resets when you quit Emacs.

I haven’t used Dendron so the UX might not be as good.

And I have to admit I don’t know how histories actually work in completion functions I used (homework for me, I think).

Thoughts?

9

I’ve also been looking at dendron, but I decided to use roam_tags for my hierarchy. This lets a file live in several hierarchy, changing the hierarchy does not require renaming the file, and org-roam-find-file lets me filter on the tags in addition to the title.

10

@nobiot Great! I like the approach. I’m not sure how histories work either but in your POC you have sort of the “prefix” part of the schema, that gives me ideas. Thank you for that :grinning:

Ideally schemas are static (so they don’t change unless you manually change them)
and are tied to an specific hierarchy to create like a template for siblings. For example, you have lots of projects and all of them has the same childs, because you want to follow a consistent pattern:

proj.project1.tasks
proj.project1.notes
proj.project1.resources

When you create another project under proj. you want emacs to understand that you want the same childs, all projects will match this:

proj.*.tasks
proj.*.notes
proj.*.resources

That something you want to be able to know when you are creating notes on that hierarchy, so all of them are the same and easily searchable because you recognize the pattern.
What I was thinking is to add this potential siblings of your note in the minibuffer as a candidate, with a marker indicating that the note doesn’t exists but that the system is expecting it. Something like:

proj.project3.tasks         => this one exists and is part of the schema
proj.project3.notes [s]     => this one doesn't exists but is part of the schema ([s])

But this shouldn’t stop you from creating a note outside of your schema if you need it, and the minibuffer can mark it as well

proj.project3.foo [?]      => Tells you that the note is not part of the schema

I think this is useful for groups of notes that will always follow a pattern. I don’t want over time to have e.g. projects with .todo, .tasks, .actions that made searching difficult.

I hope this makes sense :sweat_smile: I’m just trying to land this concept into org roam + dendroam context.

With your POC I have a great starting point, now it’s time to think about all these details

1 Like
11

That was actually the way I was approaching this before in v1. I explained about it a little bit in this post:

But for me at least in reality is not that convenient, and makes difficult to re arrange group of notes, when you try it you realize is not the same than having tags with subtags.
As well I like to have my notes grouped when I check them in a filesystem that is not emacs

12

I have created a repo to work better on this and explaining a bit what are my expectations for this project.

Feel free to take a look and open discussions, suggest code, features things or whatever you want.

4 Likes
13

I have not been able to find a good YAML parser in Elisp (or one that works with Emacs).

14

Yeah, looks like there’s none. I’m thinking on using dir-locals.el to define schemas there so they are always with the notes and are portable

1 Like
15

Hi!

I think having hierarchies is an important feature, especially I think graphical or textual displays of notes could benefit from such hierarchies; not only for computer science, but also in academia. Thanks for your ideas and implementation!

But using dots in filenames feels a bit weird, and refiling using special command does not seem natural. Two more natural ways to organize hierarchies, for me, would be:

  • Using folders/subfolders to organize nodes. Then, refiling is just moving files around.
  • Using dedicated roam notes, that are tagged with a special tag (e.g. “structure”), and that contains a list of sub-nodes. In these “structure” notes, the standard org headings could also be used to build a hierachy. Refiling can then be done using the standard org-refiling commands.

One benefit of these schemes is that a note can appear in several hierarchies, which can be important.

What do you think about these ideas?

1 Like
16

If the question is whether this is technically possible in Org-roam, then the answer is yes.

Org-roam recursively parses org-roam-directory, so you can group your notes into (nested) subdirectories. You can have different templates for different directories, or even create them on the fly, like when using ORB. I personally achieve this for bibliography notes as follows:

...
:target (file+head "references/${entry-type}/${citekey}.org"
                              "#+TITLE: ${title}\n")
...

Bibliography notes go into references subdirectory of org-roam-directory and then into another nested subdirectory depending on the entry type, e.g. article, book, collection, etc.

Using dedicated structure notes has also been extensively discussed. There is no universal solution for everyone, if hierarchy works for you - good, use it. All technical prerequisites are there.

If you are asking whether Org-roam should implement special logic to handle hierarchy, then the answer is no. It already supports nested directories, tags, backlinks and so on. These can be used to create all sorts of “physical” and “logical” hierarchies, perhaps with the aid of external tools like file manager or Org-refile, and enforcing a specific workflow for hierarchies would actually limit the current flexibility of Org-roam.

17

Blockquote If you are asking whether Org-roam should implement special logic to handle hierarchy, then the answer is no

Why not? Using structure notes (and an index) to retrieve notes is an integral part of the Zettelcasten method, and org-mode files are inherently hierarchical (being based on outline). Just by tagging some files as being structure notes and one as being the index, we could allow quick hierarchical navigation by following the paths from the index to the target node that would only “go through” structure nodes, juste like org-goto does in standalone org files. A hierarchy could also allow display of a graph to be more readable. People who do not want a hierarchy would just not use it (I, for myself, would not want to stick to a tree hierarchy which is too strict; but a DAG hierarchy, where cycles are avoided, is useful).

1 Like
18

This sounds excellent. You get most of the discoverability of infinitely nested knowledge graphs like remnote, but all the ease and power of pure file based techniques like org. Definitely something I’d like to see.
I wonder if you could Simply do it by making an org file and a folder with the same name, making it so when you capture you can either capture normally or capture a child, which goes into the folder and either adds a heading or just automatically links to the parent

1 Like
19

I came across org-brain, whose visualization seems to do something like this;in essence, one only has to categorize all links between nodes as either “parent/child” or “friend/sibling”.

20

Just came across this after reading about Dendron. Interesting concept, although maybe you are reinventing the one-big-org-file method here :slight_smile:

I wonder if this would be possible to use to inherit tags from parent nodes for example for the org-agenda. In the simpler form, at least you can filter the agenda on file names up in the hierarchy to only get child notes.