Find node UI possibilities for v2

[bruce]

Introduction

Theoretically, the number of candidates could increase a fair bit with sub-file nodes, but the current search UI only allows you to search on, and see, titles, and to see tags in the annotations.

How should this UI ideally work?

Completing-read and annotation/affixation possibilties

I’ve been working on a project to adapt bibtex-completion for completing-read, and so have been deep in the details of that, some of which might relate to org-roam v2.

The use case there is you have one or more bibtex files and you need to find and act on some candidates within.

In all of the front-end implementations (mine, as well as ivy-bibtex and helm-bibtex), the basic approach is the same:

You have a basic distinction between the candidate search string, and the display string.

EDIT: see comment below for how I concluded this was the wrong approach; but the change was trivial.

In completing-read, the latter can be designated using the display property.

The search string itself will include more data than the display string, and is actually configurable. So there’s an “additional-fields” defvar that allows you to specify which data to include in the string.

It strikes me this would make sense possibly in org-roam. It could be nice to be able to filter on content not actually displayed.

Finally, annotation and (introduced in Emacs 28) affixation.

In my implementation, I exploit this to allow the display string to be more compact, and put different types of metadata in different parts of the UI.

So here you can see the prefix contains symbols that indicate presence of associated PDFs or notes (including org-roam notes, BTW), and the suffix configurable secondary metadata.

The suffix, BTW, is defined using the affixation-function for Emacs 28, and the annotation function for pre-28, but uses the same content. Emacs will prefer the affixation function if present on 28, and ignore it on earlier versions.

Caveats

The above examples are designed for vertically-oriented completion UIs. I don’t think we want that restriction for OR?

Still, some ideas could be implemented in an ancillary “annotation” package.

What does this mean for OR v2?

I think, at minimum, the UI should clearly distinguish file nodes and sub-file nodes, and on the latter, include the context of that node. If I have a section in a file called “Introduction”, just having access to the node title is not enough.

So include the title of the containing file-node in the annotation.

Aside: sometimes titles are really long. Perhaps use truncate-string here?

Not sure whether there’s need for a prefix, and so affixation?

Anyway; I just thought I’d open this up for discussion. Thoughts?

[nobiot]

Your usage of embark-collect-snapshot looks fascinating. I have wanted to try Embark properly ever since Prot introduced his way of using Embark and completion with Embark. I have a need for querying my personal notes separately from work ones in Org-roam. I think I can pick ideas from your implementation…

[bruce]

Yes, this collection of decoupled-but-collaboratively-developed tools is pretty awesome.

FYI, I got help on it from the embark and consult developers.

[jethro]

Thanks, this makes for great reference implementation. I gave it a quick shot, tags are now searchable:

@bruce

I think, at minimum, the UI should clearly distinguish file nodes and sub-file nodes, and on the latter, include the context of that node. If I have a section in a file called “Introduction”, just having access to the node title is not enough.

I’ll have to think about this, I don’t think I want to make a distinction between file and sub-file node: these should be abstracted away, so the user only needs to think about “node”.

[bruce]

@jethro

I’ll have to think about this, I don’t think I want to make a distinction between file and sub-file node: these should be abstracted away, so the user only needs to think about “node”.

I can see that.

Well, you have the example; sections with generic headings (introduction and such).

I guess a generic way to think about this is context within a hierarchy; where the file (or maybe the roam directory) is the top of the hierarchy?

[bruce]

PS - I realized today for my case that annotations and affixation suffix actually weren’t the best approach; instead, I am now putting that content in the main candidate, using a different face for the “suffix”.

That allows that content to be searched on, and the matches highlighted.

But the affixation-function is still there to render the prefix.

[spicy]

I’m not sure if I understand completely how V2 works but IMHO, I don’t think the in-file headings should be search candidates for org-roam-find-node (I guess I got the name correctly). That will just increase drastically the number of things to search and finding what you want will be much more difficult with a big amount of notes.

I think headings as a candidates only make sense within the file (it belongs) context, I can have many notes with the same set of headings so having them in the search list just creates overhead.

So from my point of view, a solution could be allowing to search just files and when you have the file then selecting the heading. Something like what you can do with org refile for example, a two phase search.

[jethro]

@spicy

I don’t think the in-file headings should be search candidates for org-roam-find-node (I guess I got the name correctly). That will just increase drastically the number of things to search and finding what you want will be much more difficult with a big amount of notes.

Only headings with IDs are searchable: these are by definition nodes. Generic headings are generally not deserving of being a node.

[spicy]

Thanks for the answer jethro. So I guess ID generation is automatic for files (e.g. capturing) and manual for headings? If so that looks great to me, actually I don’t rely too much on heading links and if I do, I use just references of a heading in the same file (which if I’m not wrong is something that org mode can do out of the box)

[midas]

hmmm… I will offer an alternative view, supported by my experience:

A target for linking ought to come to our attention precisely when we are in the process of seeking to make a link to an idea or concept with keywords in mind. As we start typing these keywords after invoking org-roam-find-node, a list of possibilities populates (including — and especially — ones we haven’t tagged with an ID yet) from which we can choose the desired target. This approach makes possible one of the goals of zettelkasten: during our active process of creating new links, candidate node-targets that presently lack an org-ID will surface and could become anointed with an org-ID for the first time. One might ask: this sounds good in theory, but in practice, is it fast enough (computationally) and effective (does not present an unmanageable list of link-targets)?

I can say from experience that having all nodes (including those without org-IDs) be a part of search-targets hasn’t in any way slowed down or complicated the process of finding link targets. I do it presently with my “regular” org files using: (org-id-get-with-outline-path-completion org-refile-targets). My org-refile-targets are all my agenda files. The list narrows very quickly as I start to type keywords. My org-refile is set to search 6 levels deep — a non-trivial number of org nodes Performance has never been an issue, and finding relevant nodes has been super effective.

Performance has not been an issue with regular org files. But, with the org-roam structure of many small org files, I imagine org-refile is going to bog down. I was hoping org-roam-db would include all nodes so that the user can then set (based on his/her preferences) whether org-roam-find-node includes all nodes and how many layers deep of nodes should be included. I think this more flexible approach would cater better to a broad audience of org-roam users.

[nobiot]

@midas, It seems to me that the current design caters to either way you prefer (manual ID to select headings or automatic ID to all headings as nodes).

It would be easy to add a function to create IDs for all the headings in the current buffer; then you can add it to the save-hook before the db-sync runs when you save the current buffer. This way, you can save all headings as nodes, if that’s your preference.

You can experiment both and see which way is for you.

As for me, I have almost no headings in my notes, so I’m fine to have only file-level IDs so far.

It’s comforting to know it would be an easy switch and there is someone in the community honing their practice on the other side

[bruce]

@nobiot

It seems to me that the current design caters to either way you prefer (manual ID to select headings or automatic ID to all headings as nodes).

Yes, but per my original post (though I’ve not tested more recent changes), I’m not sure the current UI is well-suited to ids on all headings, for situations like mine, where many of the headings are generic.

[nobiot]

oh yes, sorry my reply was off topic for the UI discussion. I agree with you re the current UI when you have many headings with generic titles. I have the impression that you are on a good track with annotations, etc. (I’m on Emacs 27, so i can’t use affixation). I might experiment with completion UI a bit more.

As an example, the DB has “level” as a field so that you can differentiate file (level = 0) vs headings (level > 0). You can either have a different query for file vs headings, or theoretically use this distinction to alter the display on the UI.

[bruce]

@nobiot

oh yes, sorry my reply was off topic for the UI discussion. I agree with you re the current UI when you have many headings with generic titles. I have the impression that you are on a good track with annotations, etc. (I’m on Emacs 27, so i can’t use affixation). I might experiment with completion UI a bit more.

Actually, I ditched affixation/annotation. I just use a different face for the “suffix”, so that content can searched against and matches higlighted.

When I was using that approach, however, I realized that affixation is just a generalization of annotation, so you can easily support both.

@nobiot

As an example, the DB has “level” as a field so that you can differentiate file (level = 0) vs headings (level > 0). You can either have a different query for file vs headings, or theoretically use this distinction to alter the display on the UI.

Yes, this is something like I had in mind (but didn’t know the specifics of the db).

[nobiot]

@bruce

Just sharing my quick-and-dirty prototype.

I have taken advantage of some nice configuration potentials Jethro has left.

See the image below. It’s my minibuffer completion for org-roam-node-find.

It has 5 columns:

1. Title of the node (standard)
2. Tags, searchable with “#” (standard)
3. Title of the file that node belongs to (custom)
4. Tags, annotation and not searchable (standard)
5. Number of backlinks for the node (custom – my earlier custom; not described in this post)

As you can see, the first two nodes belong to the same file (“Org File with http link”). You see the first node in the main buffer above the minibuffer.

At this stage, I haven’t worked out a good “query” set up for this completion; I cannot look for a file title first and then a node title within it. I’m using the Emacs default completion with Vertico and Orderless for this demo – the query is not quite what I want, but I believe this is one step closer to what you would like.

See the custom script below. Evaluating the whole thing should enable the third column.

org-roam-node-display-template is something Jethro recently added (I believe, with the “#tag” notation). By default the file field is hidden.

I have taken it and used its value to query the file’s title. That’s what my/org-roam-get-file-title does.

I have then overridden the standard org-roam-node--format-entry to insert the file’s title to the “file” field instead of the file’s relative path – you could redefine the function, but I am using advice to override the standard function.

(setq org-roam-node-display-template
  "${title:48}   ${tags:10}  ${file:48}")

(defun my/org-roam-get-file-title (filename)
  "Return the title of the file node for FILENAME."
  (caar (org-roam-db-query
        [:select [title] :from nodes :where (and (= level 0)(= file $s1))] filename)))

(defun my/org-roam-node--format-entry (node width)
  "Formats NODE for display in the results list.
WIDTH is the width of the results list.
nobit has modified one line of this function (see the source comment) to get title of the file."
  (let ((format (org-roam--process-display-format org-roam-node-display-template)))
    (s-format (car format)
              (lambda (field)
                (let* ((field (split-string field ":"))
                       (field-name (car field))
                       (field-width (cadr field))
                       (getter (intern (concat "org-roam-node-" field-name)))
                       (field-value (or (funcall getter node) "")))
                  (when (and (equal field-name "tags")
                             field-value)
                    (setq field-value (org-roam--tags-to-str field-value)))
                  (when (and (equal field-name "file")
                             field-value)
                    (setq field-value (my/org-roam-get-file-title field-value))) ;; << Changed by nobiot
                  (if (not field-width)
                      field-value
                    (setq field-width (string-to-number field-width))
                    (truncate-string-to-width
                     field-value
                     (if (> field-width 0)
                         field-width
                       (- width (cdr format)))
                     0 ?\s)))))))

(advice-add #'org-roam-node--format-entry :override #'my/org-roam-node--format-entry)

[midas]

@nobiot

It would be easy to add a function to create IDs for all the headings in the current buffer; then you can add it to the save-hook before the db-sync runs when you save the current buffer. This way, you can save all headings as nodes, if that’s your preference.

good idea! I hope I remember this when I eventually transition to V2.

A potential downside, however, is clutter. Org-IDs will be generated for all nodes when we cannot yet know an ID will be needed or not. An alternative is to have an adjustable variable (org-roam-find-nodes-without-ID). If nil (default), present only nodes that have ID. If t, present all nodes to choose from, and if a node without ID is chosen, generate ID.

EDIT: The clutter of excess IDs will not only be in the user’s org-roam files, but also in org-id-locations that keeps a log of all org-IDs.

[public_image_limited]

Having updated to V2 today, I have the same problem that I would like to have some extra information (e.g. hierarchy). I’ll try the snippet above. But in your proposal above, what do you meanby “present all nodes … without ID”? I thought that a node is defined by having an ID. Actually, the solution to somehow include all /headings/ without an ID in the completion interface seems to be the perfect solution. Thus, one could just add an ID for linking, and else the heading is still discoverable through this interface.

[midas]

Yes, I think we are on the same page. I was using node to refer to any header, with or without ID.

[fleimgruber]

From the helm buffer, how can we jump to the location of the candidate? For other helm completions, e.g. for jumping to org heading, the key to jump to the location in other window is TAB.