Custom Roam-style link

[alan]

Summary

As an excuse to make myself learn some elisp, org, and dig into org-roam’s source, I set out to create an org-link that would emulate Roam’s a bit more (inspired by https://github.com/jethrokuan/org-roam/issues/510).
Some goals:

• Mimic Roam brackets [ [
• Otherwise hide link prefix/syntax
• Easy to write / change
• Lazy creation of notes
• Auto-complete from notes in buffer without creating them first
• Integrate with org-roam’s existing functionality
• Allow links in normal org files

I’m very happy just using the result myself but I was wondering if anyone else would be interested in adding some of this functionality in org-roam.

Functionality

Quickly write

Insert links as you write using typical org syntax: [[roam:_]]. But the link face hides the roam: prefix, and colors the brackets and link path to stand out. I bound M-[ to simply insert the roam pattern and place the point where the _ is, ready to write, but of course the shortcut is arbitrary. I have M-r bound to the same construct, but with an ivy completion prompt for more difficult titles. The link is color-coded to indicate whether the note already exists or not.

Follow links like normal

The notes use TITLE/ALIAS as their point of reference for matching files. Clicking through the roam-links opens their corresponding file, or pulls up org-capture to create a new note if it doesn’t exist, just like org-roam-find-file. After using org-capture, the link face updates to show that it exists in the org-roam db.

Easy change and completion

Because the links use TITLEs, they are incredibly easy to change. They also offer full completion support for both existing org-roam notes and as-yet uncreated notes in the current buffer. This is only functional in the roam-link construct, so company and org-roam still insert file-links everywhere else.

Integrate with backlink buffer

Obviously they must be recognized as backlinks! That is the whole point after all. The roam-links are inserted into the db as if they were regular file links, and so they mix together in the backlinks buffer without problem. The roam-links that don’t yet have files are ignored.

Links are interchangeable

For the case where writing with roam-links might feel easier, but file-links are a preferable long-term storage solution, the types are easily converted between each other. This function could easily run on an entire buffer before saving (or convert from file to roam on load – it’s your world).

Non-existent links can be automatically created

At any time all of the links without corresponding notes in a buffer can be created with a custom ‘automatic’ org-template. It is also possible to loop through and select different templates for different notes, but this is very buggy right now Using a single automatic template for all of them works without issue.

Extras

Minibuffer shows filename (or not) and raw-link

Uses shortened name relative to org-roam-directory. This should work for sub-dirs also? I don’t have any to test on right now.

Use descriptions

The current org-roam file-links use the description to show a note’s TITLE, but since roam-link uses the title as the path, a link’s description is free to shorten a title or provide local context. It is easy to see the underlying note title, whereas with an altered file-link you might take a whole FIVE seconds to navigate to a new buffer, see the title, and navigate back. Ugh! /s
The description form also uses one less bracket pair and a different face to indicate that it is not a TITLE.

Case insensitive!!!

AFAIK Roam doesn’t even allow this yet. To roam-link [[roam:unicorn]] is the same as [[roam:UNICORN]] is the same as [[roam:uNiCoRn]]. I personally never use capitalization to indicate meaning, so case sensitivity drives me nuts. It would be possible to switch it on/off, if someone is the opposite of me and thrives on capital letters…

Link in non-org-roam org files

I didn’t limit link insertion strictly to org-roam buffers because there’s no explicit reason to block their use in other org files – they just don’t create backlinks and aren’t part of the org-roam environment. I don’t anticipate using this functionality much, but it’s nice to have the option for easy linking wherever I may want it.

Future additions

• Function to search/replace a link by an ALIAS
• Mimic org-roam’s file re-name advice for TITLE or ALIAS changes
• Scan org-roam files for links to current buffer TITLE and update db (in case of roam-links that were saved without ever creating the note).
• Turn word/selection into roam-link
• Nested automatic creation in org-capture

Let me know what you think! I have to have a project to really learn any new programming, so this is a jumble of code in my config file I plan to refine for myself, but if there’s interest in any of it I’d be happy to contribute something.

[tecosaur]

I think this looks great and would love to see this! Great job

A few comments from me:

• I don’t see the point in showing square brackets [[link]] when fontification exists. I find the colour alone sufficient, leaving the brackets as unnecessary visual clutter (what can I say, I’m a minimalist).
• Lazy creation of notes, in particular, seems like a great feature. Really looking forward to that
• Did I mention I’m also quite particular? Re: the --> arrow you show in the mini buffer, how about → or ⟶ ?
• Case insensitive seems sensible.
• I completion

Overall this looks marvellous! Please don’t stop

[khinsen]

Lots of good ideas there!

One thing to keep in mind is existing note databases. All org-roam note collections in existence today use Org-Mode’s file links internally. If you change that to distinct roam: links, you will have to think about a way to update existing notes.

Whether or not a special roam link is better or worse than standard file links is a different question. I don’t have a firm opinion about that.

[jethro]

This looks fantastic , would love to see this merged into core so it’s available for others to easily use.

Regarding the special roam: link syntax, as much as I’d rather not have it, off the top of my head it seems unavoidable for the kind of streamlined workflow @alan has demonstrated in his GIFs. There is, however, an easy solution to those who prefer file links where possible. @alan has provided a function to convert between roam: links and file: links, and we can have that called after some of the interactive commands shown above (e.g. when creating the file for the template, or after the ivy completion prompt). The downside to this is that the link is no longer as easy to change, but that’s just the tradeoff of using file: links.

I think the two can coexist together, at the very least.

[bruce]

I agree with this list of suggestions.

And also, great job @alan! Yes, I’d love to have it included in OR, with the changes suggested by @tecosaur.

[bruce]

@tecosaur

Did I mention I’m also quite particular? Re: the --> arrow you show in the mini buffer, how about → or ⟶ ?

On this, I’d suggest the latter, I think.

[Memex]

This is great. Please continue to work on this.

I’m worried if this would conflict with native org-mode functions like org-export and org-publish.
Maybe we can have something like org-fragtog?

[tecosaur]

While in most default system fonts “→” looks bad, I think you’ll find that in monospace fonts it actually looks decent, here’s a more true-to-how-it-would-look comparison.

[bruce]

Yes, now that I see; I agree.

[rsperko]

I am really excited about this. I think this is better than my desire for nested capture. I don’t see a PR for it, will watch closely for it. Thank you.

[bmg]

Really great work and writeup! Agree with @tecosaur and @bruce, on the UI cleanup .

For the case of multi note auto completion, maybe a buffer (ibuffer-esque) or some means show the recently created files / buffers, if it is a longer file where there are a mix of existing OR links and new ones, keeping track of which ones need to be addressed could become challenging. Of course ibuffer will list them and I should close some occasionally to declutter it

[alan]

Wow, thanks for the encouragement!! I thought everyone might see this as a lot of syntactic sugar I’m just now learning how to really use git and github (I’m slightly ashamed the amount of programming I’ve done without it), so I’ll be a little slow in putting together a PR. Plus I have to actually do some analysis work for my lab instead of just playing with org But I’ll start putting something together this week.

@tecosaur

I don’t see the point in showing square brackets [[link]] when fontification exists. I find the colour alone sufficient, leaving the brackets as unnecessary visual clutter

The brackets are purely personal choice. The starting goal was to mimic Roam’s syntax, which means brackets, but I also like the gestalt of them – font color alone differentiating file/roam links bothers me for some reason, I like the two color pop, and I hate underlining.

But they’re handled as individual groups in fontification, which means they can be hidden or shown depending on user choice. Just set a var in config and they’ll be hidden/shown (description links use the single bracket pair for instance – also easy to change).

@tecosaur

Did I mention I’m also quite particular? Re: the --> arrow you show in the mini buffer, how about → or ⟶ ?

I figured people would have opinions about that It was just a quick solution since I didn’t want to dig through different unicode arrows. Completely open to a consensus character!

@khinsen

One thing to keep in mind is existing note databases. All org-roam note collections in existence today use Org-Mode’s file links internally. If you change that to distinct roam: links, you will have to think about a way to update existing notes.

The file-to-roam or vice-versa conversion shown on the single link in the gif can easily work on an entire buffer or a loop through all org-roam files at once. It would also be possible to do something like on-load convert org-roam file-links to roam-links, and on-save convert them back to file-links. They can also be completely mixed in the document and database (they’re identical in the database) forever without any problems (if you moved your documents out of org-roam you would just need to run conversion first).

@jethro

Regarding the special roam: link syntax, as much as I’d rather not have it, off the top of my head it seems unavoidable for the kind of streamlined workflow

Yes I think this is unavoidable. There are other hacky options in org that can achieve some of these things, but they would all be somewhat fragile and/or abuse org a bit too much for my taste. Using a custom link allows full flexibility like what org-ref can do with cite: links.

@Memex

I’m worried if this would conflict with native org-mode functions like org-export and org-publish .
Maybe we can have something like org-fragtog ?

roam: uses org’s built-in custom link toolset, so there is no conflict with any native org functionality. I haven’t done anything with it, but you can set a custom :export function for the link to handle any kind of pre-processing necessary depending on your output choice. And if your export should already work with standard file-links, then it is easy to convert back-and-forth as needed.

@bmg

For the case of multi note auto completion, maybe a buffer (ibuffer-esque) or some means show the recently created files / buffers, if it is a longer file where there are a mix of existing OR links and new ones, keeping track of which ones need to be addressed could become challenging. Of course ibuffer will list them and I should close some occasionally to declutter it

I like it!

Thanks for the input everyone! I’ll try to have something up by the weekend.

[codygman]

This is amazing and I look forward to trying it out after work.

If all goes well, I can see how some friends that aren’t emacs users get along using it.

[tecosaur]

@alan

The file-to-roam or vice-versa conversion shown on the single link in the gif can easily work on an entire buffer or a loop through all org-roam files at onc

I may be missing something but to me this seems like a prime opportunity for caching! We have an SQLite DB, and we can use that to trade computation for storage, and hopefully improve scalability by going from O(n) to O(1).

At the moment there seem to be 4 small tables:
All fields not null, unless labelled as “nullable”

• files
  • file : text : unique
  • hash : text
  • last_modified : text
• links
  • from : text
  • to : text
  • type : text
  • properties : text
• refs
  • ref : text, unique
  • file : text
• titles
  • file : text
  • tites : nullable

I suspect this could be both made more efficient, tidier, and more versatile, with a structure making use of the date as a UUID (if someone makes two notes within the same second, I’d just pretend they made the second note one second later).

• notes
  • example file, 20200429235659-zettelkasten-method.org
  • created : integer : unique (e.g. 20200429235659)
  • file : text : unique (e.g. “zettelkasten-method. org”)
  • title : text : unique (e.g. “The Zettelkasten Method”)
  • hash : text
  • ref : text : null or unique
• links
  • from : number (e.g. 20200429235659)
  • to : number (e.g. 20200429231326)
  • type : text (e.g. “roam”)
  • properties : text

Regarding the notes table:
Since we refer to files by their [[title]], in order to have unique link targets, we must have unique titles. As the file suffix (name) is derived from the title, it must also be unique. This allows for hashmap style searches across the title field, which seems advantageous.
NB: I’m not sure what hash and ref do, so I wasn’t able to take their purpose into consideration.

Regarding the links table:
By using the created field as a UUID, we can refer to links by this, and the link is automatically immune to note renaming. This seems desirable.

Overall, this structure should make translating a roam:title link to a file as trivial as using the title provided, converted to lower case, with whitespace replaced with -, which I shall refer to by the variable title-lower-case-dashed.

SELECT created FROM notes WHERE file IS title-lower-case-dashed.org

Then created-title-lower-case-dashed.org gives you the file name, and with just some simple string processing, and a single DB lookup.

I also suspect this may make renaming more robust, but I haven’t looked into that.

@jethro please let me know what your thoughts are.

[technovangelist]

this looks incredible

[alan]

@tecosaur

Overall, this structure should make translating a roam:title link to a file as trivial as using the title provided, converted to lower case, with whitespace replaced with - , which I shall refer to by the variable title-lower-case-dashed .

SELECT created FROM notes WHERE file IS title-lower-case-dashed.org

Then created-title-lower-case-dashed.org gives you the file name, and with just some simple string processing, and a single DB lookup.

FWIW I completely avoided adding any changes to the db structure so that functionality would be seamlessly integrated with the rest of core, and usage of roam: would be optional. I’m sure there is a schema improvement that can be made if everyone likes this style enough.

I’m not sure I understand what you’re suggesting, though. The UUID would make the backlinks buffer immune to renaming, but the contents in files themselves would still be broken since they don’t have direct access to the UUID: If a TITLE is changed, a roam: link will miss it in the db and think it doesn’t exist. If a filename is changed, a file: link will miss it in the db and think it doesn’t exist. Both of these situations would require a hook on filename/TITLE change to update the affected files (the file hook already exists, and a TITLE hook is on my todo).

The roam: link is immune to filename changes because it already uses the fact that TITLEs should be unique to just do a single db lookup (if they’re not it just returns the first one it finds). Additionally ALIASes are stored in an s-expression with TITLE – it will match against any of those for a single filename lookup.

It also sounds like you’re dynamically building a file name from the TITLE every lookup? I know some users like to customize their filenames away from the default.

Am I misrepresenting your suggestion? I almost never work with SQL so I don’t know if I’m missing a key idea here.

[tecosaur]

I suspect it’s a mix of both of us not quite getting the other’s idea

When I saw

loop through all org-roam files

I assumed this was per-link file processing. If it’s just a DB query, then my point is moot.

It also sounds like you’re dynamically building a file name from the TITLE every lookup? I know some users like to customize their filenames away from the default.

Yea, that was what I was thinking. Considering that, as you stated, some users customise the format: it sounds like my idea would need amending. I’d imagine replacing file with the full file name and doing the lookup based on title instead would be sufficient.

FWIW I completely avoided adding any changes to the DB structure so that functionality would be seamlessly integrated with the rest of core, and usage of roam: would be optional.

Seems sensible, I just thought it could be worth talking about the DB structure, considering that it seems to tie into this.

The UUID would make the backlinks buffer immune to renaming, but the contents in files themselves would still be broken since they don’t have direct access to the UUID: If a TITLE is changed, a roam: link will miss it in the db and think it doesn’t exist. If a filename is changed, a file: link will miss it in the db and think it doesn’t exist. Both of these situations would require a hook on filename/TITLE change to update the affected files (the file hook already exists, and a TITLE hook is on my todo).

Well, in this suggestion the UUID is the creation timestamp. I think this is the most sensible, as it allows for title/filename changes with minimal issue. However, a roam:20200429235659 link is quite uninformative, especially if viewed outside of Emacs (rare, but possible, and I think it would be good to retain how well org-mode file view as plain-text.

Hence, the best compromise I can think of is having roam:title links, and using the org-roam rename function to edit all references when needed.

At the end of the day, my idea is just my current best thought on how to have

• A static UUID associated with each note, for robust lookups and references
• Speedy determination of file from the title

Please let me know if this helps, and any more thoughts you may have

[alan]

@tecosaur

When I saw

loop through all org-roam files

I assumed this was per-link file processing. If it’s just a DB query, then my point is moot.

Ah yes, the link-conversion is a single db query. The loop through all org-roam files is in reference to the extreme case that someone has hundreds of notes filled with file: links and they want to batch replace all of them with roam: links (or vice versa). Since the conversion is just a db query, I was saying conversions scales easily if someone wants it to.

@tecosaur

However, a roam:20200429235659 link is quite uninformative, especially if viewed outside of Emacs (rare, but possible, and I think it would be good to retain how well org-mode file view as plain-text.

Yes this is exactly what I am trying to avoid Using the title/alias as the link’s path means you can read and change the link semantically at any time without obscure filenames or ID’s. It also means you can create new notes lazily as you write.

Otherwise I like the idea of a UUID. I had been planning to put them into my templates as a personal use case, but perhaps they shouldn’t be a default pattern, since they still require filename / title for look-up in the db. I’m not sure what they’ll add to the schema. Someone else with more experience can weigh in on this.

[zaeph]

This is stellar work, @alan, and I’m glad to see you with a working prototype.

There are interesting ideas being developed in this thread, and this is exactly the type of discussions that @jethro and I wanted to encourage by creating this Discourse. Thank you all for participating!

@alan: I can only encourage you to submit this as a PR on the tracker as soon as you’ve familiarised yourself with Git, even if all you have is a semi-clean prototype. That would allow us to look at your code, figure out what works, what doesn’t, and iron out the kinks.
I’m so glad to see you’ve put your money where your mouth is, so congratulations!

@tecosaur: As interesting as your proposition seems to be, I would heartily recommend you to move on to the implementation. A prototype would go a long way in showing the validity of the changes you’re suggesting. I’ll refrain on making any comment on them since it isn’t my area of expertise on the package.

If any of you need help with Git, don’t hesitate to ask us questions about best practices. If you feel a little intimidated by the process, know that it’ll pass quickly. You’ll run through fuck-ups, as we all have, but you’ll learn through them.

Thank you all again for you work, and I look forward to reviewing your PRs on the tracker!

[tecosaur]

@zaeph

As interesting as your proposition seems to be, I would heartily recommend you to move on to the implementation. A prototype would go a long way in showing the validity of the changes you’re suggesting. I’ll refrain on making any comment on them since it isn’t my area of expertise on the package.

I can say it’s something I have a lot of expertise with, but it does strike me that having a UUID for the notes could be beneficial, the only other change I made was combining the tables.