Using consult-ripgrep with org-roam for searching notes

I find deft pretty slow, and even notdeft isn’t as fast as I would like (and is harder to set up). ripgrep is very fast though, and the consult package includes a very nice consult-ripgrep function.

Assuming you have consult installed, here’s a useful function:

(defun bms/org-roam-rg-search ()
  "Search org-roam directory using consult-ripgrep. With live-preview."
  (interactive)
  (let ((consult-ripgrep-command "rg --null --ignore-case --type org --line-buffered --color=always --max-columns=500 --no-heading --line-number . -e ARG OPTS"))
    (consult-ripgrep org-roam-directory)))
(global-set-key (kbd "C-c rr") 'bms/org-roam-rg-search)

brief demo:

(ripgrep does have a --files-with-matches option, which in theory would produce output more similar to what deft/notdeft returns, but I haven’t figured out how to use it via consult-ripgrep yet.)

8 Likes

Very nice! Thank you. Will try it

Thanks for posting this. I installed the consult package and ran your function, but it didn’t return any matches. I also tried just running consult-ripgrep but that didn’t return anything either. Do I need to do anything to initialize consult-ripgrep?

Do you have ripgrep installed on your system? (it’s external to emacs)

I do. I can get consult-ripgrep to work on vanilla Emacs, but not on my Doom config (or a fresh Doom config). I’ve logged an issue.

Any alternatives? I love the idea of ripgrepping my Org/roam folder and then having previews of the files as I scroll through the results.

@emacsomancer Thanks again for this post. It will be useful for my set up.

I wanted to make ripgrep search to case-insensitive globally (otherwise I won’t find what I should find), so I did a very small customization to add --ignore-case like this.

Just wanted to share it in case someone else wants to the same thing.

;; --ignore-case is added, not part of the default
(consult-ripgrep-command
   "rg --ignore-case --null --line-buffered --color=always --max-columns=500   --no-heading --line-number . -e ARG OPTS")

You might also look at the rg.el package, though you’d have to think about how get previews on scroll, I think.

For using consult-ripgrep, my guess is that Doom probably uses some other completion framework by default (Helm, Ivy) that’s interfering.

You may want to alter the function too: I have the --smart-case option enabled there (which ignores case as long as you don’t type any CAPS), you might want to replace that with --ignore-case, if you prefer that because my function over-writes the global consult-ripgrep options for its search. (I’m not sure which option I prefer - I usually just search in all lowercase so it probably doesn’t matter).

Oh i see. Thank you. I was happy with the ignore-case option, but i’ll look up the smart-case one to see which i prefer :slight_smile:

revised code to be a bit cleaner, and changed to ignore-case rather than smart-case as the former is probably more transparent.

[edited original post, but also reproduced here:

(defun bms/org-roam-rg-search ()
  "Search org-roam directory using consult-ripgrep. With live-preview."
  (interactive)
  (let ((consult-ripgrep-command "rg --null --ignore-case --type org --line-buffered --color=always --max-columns=500 --no-heading --line-number . -e ARG OPTS"))
    (consult-ripgrep org-roam-directory)))
(global-set-key (kbd "C-c rr") 'bms/org-roam-rg-search)

]

And here’s my initial attempt at a more deft-like version, which just returns one file match per term no matter how many hits are in the file. It doesn’t have the nice live-preview of the above function though:

(defun bms/consult-ripgrep-files-with-matches (&optional dir initial)
  "Use consult-find style to return matches with \"rg --file-with-matches \". No live preview."
  (interactive "P")
  (let ((consult-find-command "rg --ignore-case --type org --files-with-matches . -e ARG OPTS"))
    (consult-find dir initial)))

(defun bms/org-roam-rg-file-search ()
  "Search org-roam directory using consult-find with \"rg --file-with-matches \". No live preview."
  (interactive)
  (bms/consult-ripgrep-files-with-matches org-roam-directory))
(global-set-key (kbd "C-c rf") 'bms/org-roam-rg-file-search)
3 Likes

FYI, there’s work (that jethro started actually) going on to add selectrum and consult as an alternative to ivy and helm.

EDIT: this has since been merged as the “vertico” completion module, which is now default.

2 Likes

I ended up using deadgrep instead.
Instead of calling rg directly, I have a bash wrapup. This allow me to remove id string in the link. Make reading easier. It’s not pretty but very useful

rg -i --color always "$@" | sed "s/\[id:[a-z0-9]\+-[a-z0-9]\+-[a-z0-9]\+-[a-z0-9]\+-[a-z0-9]\+\]//g"
1 Like

I use counsel and ag, with this function.

(defun jmb/counsel-ag-roam ()
 "Do counsel-ag on the org roam directory"
 (interactive)
 (counsel-ag nil org-roam-directory))

It works like a charm. :slight_smile:

1 Like

This is great! It is going to be a huge boost to finding notes. Relying on org-roam-find-file was very precarious since the title of a note does not adequately represent keywords by which future-me will look for a file.

The combination of selectrum-prescient-consult-ripgrep and your function is a key inflection point in my willingness to adopt org-roam as a core in my workflow. I am very much reassured that I can find any needle in a haystack of org-roam files.

The preview is a great touch. And, the speed is incredible.

I also use consult-ripgrep as provided by Doom Emacs and it works great.

One thing I haven’t been able to figure out is how to search for multiple terms, i.e. to let it return all files that contain the terms foo and bar, for example. Do you know how to do this?

This seems to be possible in several ways using ripgrep: Proper way to search for term1 AND term2 · Discussion #1845 · BurntSushi/ripgrep · GitHub.

Using # as a separator?

1 Like

Oh wow, how could I miss that, so easy! Thanks! :slightly_smiling_face:

The thing is that this operates on lines though, i.e. returning only those files in which foo and bar appear in the same line. To be really useful, I would like it to return all notes where there’s a match in the same file.

Is this possible, too? I found that you can use (rip)grep like that using the -U flag ^1 and that works fine for me from the terminal:

rg -Ul '(?s)foo.*?\n.*?bar|bar.*?\n.*?foo'

However, I haven’t managed to get this done within Emacs using consult-ripgrep. I don’t know how you can pass flags to the command. I tried modifying your code from above and added the --multiline flag, but it doesn’t work:

(defun bms/org-roam-rg-search ()
  "Search org-roam directory using consult-ripgrep. With live-preview."
  (interactive)
  (let ((consult-ripgrep-command "rg --multiline --null --ignore-case --type org --line-buffered --color=always --max-columns=500 --no-heading --line-number . -e ARG OPTS"))
    (consult-ripgrep org-roam-directory)))
1 Like

Hmm, that’s a good point. I’m not sure. I’ll try to poke around at it and see if I can figure out anything further.

Cool, let me know …

My first impression was that one would need to pass flags to (consult-)ripgrep to make it match on the file- rather than line-level.

However, I think I read somewhere that the # splitting the search string signifies that the first part is provided by ripgrep, the second part is a second filter step done within Emacs. If that’s true, it would be more complicated.

So far, I got it working on the shell for two terms and I can use that to output an org table of file names now:

#+name: get-backlinks-a-b
#+begin_src sh :var termone="Emacs" termtwo="science"
rg -Ul "(?s)$termone.*?\n.*?$termtwo|$termtwo.*?\n.*?$termone" ~/roam --type org
#+end_src

#+call: get-backlinks-a-b(termone="Emacs", termtwo="science")

#+RESULTS:
| /Users/quirin/roam/20200615102826-qw.org       |
| /Users/quirin/roam/Johnson2021EmacsTool.org    |
| /Users/quirin/roam/20211027001104-ripgrep.org  |
| /Users/quirin/roam/20200516141142-org_roam.org |
| /Users/quirin/roam/journal/2020-07-22.org      |

It would be much better to have it output actual file links so that they are clickable, of course, but I don’t have too much elisp-fu. Much better yet would still be interactive usage though and having the output in a nice vertico buffer like with consult-ripgrep.

1 Like

How about passing the --multiline flag to ripgrep, e.g. redefining like this

(defun bms/org-roam-rg-search ()
  "Search org-roam directory using consult-ripgrep. With live-preview."
  (interactive)
  (let ((consult-ripgrep "rg --null --multiline --ignore-case --type org --line-buffered --color=always --max-columns=500 --no-heading --line-number . -e ARG OPTS"))
    (consult-ripgrep org-roam-directory)))

Does that get closer to the desired behaviour?