How easy would it be to extend org-roam with backlinks to other types of items?

Development
1

I have recently played with a simple implementation of backlinks for emails (when I open an email in notmuch I get a view of all org entries that link to that email). Today I thought about doing the same for git commits and branches. Magit adds support for linking to branches and commits among other things. It would be pretty cool to get which org entries link to this particular branch, which could be useful for notes or task management.

This sounds like what an extension of org-roam already does. Iā€™m guessing such functionality is beyond the scope of the project, but maybe itā€™s possible for a third party (i.e. me) to implement this? How easy would that be?

2

Iā€™d say that it lies on the outskirts of org-roam. If you consider the model to be links between nodes, and those nodes to be primarily notes (be they files or headlines), any node that is not a note could be considered a pseudo-node. Links could lead to that pseudo-node as if they were regular nodes, and I donā€™t see why we couldnā€™t handle that.

However, I believe the opening of those links should probably be deferred to an ol.el extension, i.e., the part of org-mode's core which handles the links.

Obviously, for now, this is quite vague; but Iā€™m confident that this is a topic we could broach upon in the coming months.

1 Like
3

Yes, that sounds pretty sensible, especially since most of the relevant links are probably already implemented by some ol- extension.

I have only a vague understanding about how v2 works, but itā€™s entirely ID based right? So to implement an extension, say for the notmuch email program, youā€™d only need to store the link target (e.g. notmuch:id:foo123 ) and possibly the link type in the database. Then you only need to tell org-roam how to fetch the ID of the current buffer (in this example, the id of the current email).

Anyways, Iā€™m just throwing ideas out there. It would be fun to hack on this soon, but I first need to migrate to v2.

4

Yes, thatā€™s the idea. Itā€™s a little too early to tell how this would be handled in our schema (link vs node), but weā€™ll play around with the concept. All I can tell you right now is that Iā€™m on board with your idea.

1 Like
5

Sharing experience from building V2 compatible extension: Md-roam for v2 (not compatible with the latest commit yet; Iā€™ll need to fix it for the latest DB schema).

@zaeph , you can take this as feedback from one of the first penguins :wink:

It is relatively easy if your implementation is done to be compatible with Org-roam in the following way (I try to be exhaustive based on what I have seen, but Iā€™m writing this at my lunch time. Apologies if I miss important elements).

If you want to re-use what Org-roam offers as much as you can, I would assume you wanted to use backlink-section in the Org-roam buffer. To do this:

  1. Both source and target are text files with a file extension and are located in org-roam-directory
  2. Both source and target are recognized by org-id-get

Let me explain.

1. File and extension

Org-roam looks only at files within org-roam-directory, and it checks if a given file is an Org-roam file. It does so by checking each fileā€™s extension defined in user option org-roam-file-extensions (inspect the function org-roam--org-file-p). The user option is a list so it can be more than one (In my case, I need to add md to org in the list).

For the idea of the email or Git branch to work, with the current Org-roam implementation, an email or Git branch would need to be stored as a file within org-roam-directory. You could use symbolic link, so the location might not be an issue; however, I am not sure if emails and Git branches are stored as files with an extension. If not, you would need tweak this part of Org-roam, probably in rather significant a way.

2. Nodes and org-id-get

V2 looks at nodes only. This means both source and target files need to be nodes.

For your idea of an email, as an example, you open an email file, and it needs to be a node as well as the Org-files to be displayed as backlinks. Technically, you can inspect the function org-roam-buffer. It immediately checks if the current buffer (a buffer for an email, for example) is a node with org-roam-node-at-point; if the point is not on an org-id, it first looks for the next org headline and then to the beginning of the buffer to check for the fileā€™s ID.

Thusā€¦ you can see that the current implementation has a specific definition of nodes: nodes are Org headlines or files recognized by Org-ID and only Org-ID. Org-ID does not care about the file extension, but (I believe) the file needs to be a text file (not a binary) and conform to the Org syntax to define a file or headline ID (within a property drawer).

In my case, I advise an Org-ID function so that an ID within YAML frontmatter can be recognized as an Org-ID, thus an Markdown file can be a node within Org-roam.

Emails might be OK (arenā€™t they stored as text files?). For text files with different syntax, advising works like I did for Markdown files. But Iā€™m not sure of Git branches and commits (are they somehow compressed binary to reduce the size?)

EDIT: One last thing before I forget. If you go down the path of adding a file to the database, note how org-roam-db-update-file is implemented. It uses org-roam-with-file macro, which turns off the major mode and on org-mode ā€“ this took me some time to debug and understand. This does not fit in the two points above and implementation detail, but could save you some time.

2 Likes
6

I donā€™t think this ought to be a necessity. In our v1 model, we were able to relate refs without needing for the notes to be explicitly created, which was close to the concept of pseudo-node Iā€™ve highlighted above. I think something similar could be devised for emails or repos, provided that we come up with a good way to differentiate those links within the files and the database.

Taking into account the points Iā€™ve just mentioned, we could just prepend the init of our org-roam-buffer to check if the buffer is accounted for in our slip-box.

Obviously, weā€™re only drafting ideas now, but Iā€™ve been thinking about many of those points during my time off. Iā€™m looking forward to writing down a design document that would synthetise some of the points broached upon on Discord along our own ideas.

7

@zaeph,

Thank you. Itā€™s good to know that there is a different way by using ref-section ā€“ is this meant to be a future idea?

What I have outlined is what I needed to do for backlink-section, in implementing V2 update Md-roam; so itā€™s not an idea (for me, itā€™s how it has been already done). Itā€™s a stepping stone for others :wink:

8

Disclaimer: Iā€™m speaking as somebody who has no idea how the internal implementation works.

These ā€œspecial nodesā€ (e.g. emails, git branches etc.) are a special case because they canā€™t have links, only be linked to. Because of this it should be sufficient to store a {notmuch ID, git commit hash}-to-org-ID mapping in the database. Then for displaying the backlinks in an email buffer you query the database for the current emailā€™s ID and display information of all related org entries. I imagine this process is similar to how org-roam does it now, fetch the id of the current org buffer and query all the nodes that link to it. I donā€™t know how org-roam actually works, but maybe this ID-fetching-function could be a variable that could be set by the user or plugin.

I think theyā€™re stored as files, but they are probably moved around. At least notmuch has an unique ID that is used for org links. I think the same is true for mu4e too.

9

This is similar to what I have outlined, yes. Implementation will dictate whether it ought to be a property of the link or a property of a (pseudo-)node, but weā€™ll only be able to tackle that question when we move to the implementation phase. For now, letā€™s keep our options open.

10

Thank youā€¦

My quick summary of what this discussion thread has been saying:

  1. @Zetagon asks how easy it would be to implement a backlink feature for ā€œspecial nodesā€ as an extension to Org-roam:

  1. I am saying it would go against the grain if you wanted to implement it now because it would not meet the "easyā€ conditions I outlined above (both source and target are expected to be files or part of one)

  2. @zaeph is saying the Org-roam project could consider implementing the ā€œpseudo-nodeā€ concept in some way in the future, which would make this type of extension easier to develop

Fair summary?

I will be happy to be corrected for my summary and factual understanding how V2 currently works.

11

To me, it is. ā€˜Futureā€™ is a vague term, but considering weā€™re merely at the designing stage, itā€™s easy to omit some crucial data-points that will reveal themselves when we get to the prototypes. Iā€™ll stress again that this is something that I believe to be beneficial to the project.

1 Like
12

Shouldnā€™t be very difficult, but currently unsupported.

Weā€™ll basically need to modify the Org-roam buffer such that the entire window context is passed over, rather than just the node at point. From then on, itā€™s just a matter of implementing the widget.

FWIW because notmuch/magit supports linking already, you can simply make use of ROAM_REFS, which isnā€™t limited to org-ref.

2 Likes
13

So, apparently after 7 months I decided to try to implement this with the org-roam2 and it was indeed pretty easy. Iā€™m not sure if I should upstream this to org-roam or if I should extract this with advices in my own config. Should I send a pull request?

Edit: I opened a draft PR for people who want to see the code: Support backlinks in notmuch email messages by Zetagon Ā· Pull Request #2008 Ā· org-roam/org-roam Ā· GitHub

1 Like