I have many org-roam v2 files and want to link subheadings from each other. Currently, the approach I am following is-
Use org-store-link to generate ID for the (sub) heading in file A.
Use org-roam-node-insert to insert the previously generated ID in file B.
I wish to automate these 2 steps into 1. There should be a way in which I can select all the subheading in all org roam files and just select it to insert a link to it. If there was no ID property to it, then create it, else, use it.
I have found counsel-org-link to work similarly according to my requirement, but, it works for only a single file. I want to work with multiple files. Please help.
You’d need a basic understanding of Elisp and how marker works (easy), and familiarity with libraries org-element and completing-read (will take time…).
It is not difficult, but time-consuming to get right (this is as far as I can do).
Function org-roam-list-files gives you a list of all the files in a list
For each .org file in the list, get all the headline elements (you should be able to use the built-in org-element library to parse each .org file and get the headlines with the marker (file and position) and headline title
Create a minibuffer selection. This would depend on your completion framework (especially Ivy/Counsel), but for Vertico/Consult, it would be completing-read (you can see how completing-read does this)
Based on the user’s selection of the headline from the minibuffer, you can visit the file and headline, and get the existing ID or create an new one (eg org-id-get-create)
EDIT: Sorry, you mention org-store-link – I think it will work too, assuming your configuration is correctly set to create an ID when it does not exist
At this stage, the new ID is not in the org-roam-db, so you’d need construct the ID link (not difficult because you have the ID and the headline title)
EDIT: If you use org-store-link, then you should be able to use org-insert-link for the ID
I like the idea - I already have a way to parse all the headings of a file - I think I can code this.
But we have to solve a few problems. Firstly if we search for all the headings in all the files the process might be cpu intensive - can you work with first selecting the file then getting a list of the subheadings and headings?
or do you want to arbitarily search for all the subheadings and headings? We can cook something up. Lets work on it - let me know how you conceive of the workflow.
An alternative is to use regex for the headers in the files in org-roam-dir. This is probably better for performance than my first suggestion to use org-element to parse the files.
;; beach (variables)
(defvar 0/org-headlines-hash-table (make-hash-table :test 'equal)
"Hash table containing headlines (keys),
their corresponding positions in the file (value 1), and the file names (value 2).
A Hash table is a list-like structure containing pairs: Keys and Values")
(defvar 0/cache-directory org-roam-directory
"Directory to cache Org files for headline extraction.")
(defvar 0/cache-directory--exclude '()
"Name of directories to exclude from headline extraction.
Subdirectories will be excluded too.")
;; user functions
(defun 0/org-goto-headline ()
"Navigate to the position of a headline selected from the 0/org-headlines-hash-table."
(interactive)
(let ((headline (completing-read "Choose headline: " (hash-table-keys 0/org-headlines-hash-table) nil t)))
(when headline
(let ((position (gethash headline 0/org-headlines-hash-table)))
(if position
(progn
(find-file-other-window (cadr position))
(goto-char (car position))
(message "Navigated to %s in %s" headline (cadr position)))
(message "Headline '%s' not found in hash table" headline))))))
(defun 0/insert-org-link-to-headline ()
"Create an Org link to a headline selected from the 0/org-headlines-hash-table and insert it into the current buffer."
(interactive)
(let ((current-buffer (current-buffer))
(headline (completing-read "Choose headline: " (hash-table-keys 0/org-headlines-hash-table) nil t)))
(when headline
(let ((position (gethash headline 0/org-headlines-hash-table)))
(if position
(save-excursion
(let* ((org-file (cadr position))
(org-buffer (find-file-noselect org-file)))
(with-current-buffer org-buffer
(goto-char (car position))
(let* ((headline-text (org-get-heading t t t t))
(id (org-id-get-create)))
(save-buffer)
(switch-to-buffer current-buffer)
(insert (format "[[id:%s][%s]]" id headline-text))
(message "Inserted Org link to headline '%s' with ID '%s' in current buffer" headline-text id)))))
(message "Headline '%s' not found in hash table" headline))))))
(defun 0/org-extract-headlines-from-directory (&optional clear-cache debug)
"Recursively extract headlines from all Org files in the cache directory, excluding directories specified in `0/cache-directory--exclude`."
(interactive "P")
(when clear-cache
(0/org-headlines-hashtable-clear))
(let ((files-to-process (0/org-filter-excluded-directories))) ; refer to abyss below
(dolist (file files-to-process)
(when debug
(message "Processing file: %s" file))
(0/org-extract-headlines file))))
;;anchors (and hooks)
;; Define a function to update the hash table upon saving a buffer
(defun 0/update-hash-table-on-save ()
"Update the hash table upon saving the current buffer."
(when (and (eq major-mode 'org-mode)
buffer-file-name)
(let ((org-file (buffer-file-name)))
(when org-file
(when (0/org-headlines-file-member-p org-file)
(0/org-delete-keys-for-file org-file)
(0/org-extract-headlines org-file))))))
;; Define a global minor mode
;;;###autoload
(define-minor-mode 0/org-auto-update-hash-table-mode
"Global minor mode to automatically update the hash table upon saving Org files."
:global t
:init-value nil
:lighter " 0/rHT"
(if 0/org-auto-update-hash-table-mode
(progn
;; Set up after-save hook to update the hash table
(add-hook 'after-save-hook #'0/update-hash-table-on-save)
;; Clear the hash table
(setq 0/org-headlines-hash-table (make-hash-table :test 'equal))
;; Refresh the hash table
(0/org-extract-headlines-from-directory))
;; Remove hook if mode is deactivated
(remove-hook 'after-save-hook #'0/update-hash-table-on-save)))
;; seabed (underlying functions)
(defun 0/org-extract-headlines (org-file &optional clear-cache debug)
"Extract and store all headlines from ORG-FILE in 0/org-headlines-hash-table.
If CLEAR-CACHE is non-nil, clear the existing cache before extraction.
If DEBUG is non-nil, display a message after populating the hash table."
(interactive "fEnter Org file: \nP")
(when clear-cache
(setq 0/org-headlines-hash-table (make-hash-table :test 'equal))) ; reset the cache when requested
(with-temp-buffer
(insert-file-contents org-file)
(goto-char (point-min))
(while (re-search-forward "^\\*+\\s-+\\(.*?\\)$" nil t)
(let ((headline (match-string 1)))
(puthash headline (list (line-beginning-position) org-file) 0/org-headlines-hash-table))))
(when debug
(message "Populated hashtable with contents from %s" org-file)))
(defun 0/org-delete-keys-for-file (org-file &optional debug)
"Delete all keys associated with the specified Org file from the 0/org-headlines-hash-table."
(interactive "fEnter Org file: ")
(let ((keys-to-delete '()))
(maphash (lambda (key value)
(let ((file-path (cadr value)))
(when (file-equal-p (expand-file-name file-path) (expand-file-name org-file))
(push key keys-to-delete))))
0/org-headlines-hash-table)
(dolist (key keys-to-delete)
(remhash key 0/org-headlines-hash-table)))
(when debug
(message "Deleted keys associated with %s from the hash table." org-file)))
;;abyss (helper functions)
(defun 0/org-filter-excluded-directories (&optional debug)
"Return a list of files in the cache directory excluding directories listed in `0/cache-directory--exclude` and their subdirectories.
When DEBUG is non-nil print verbose output."
(let ((exclude-dirs 0/cache-directory--exclude)
(cache-directory 0/cache-directory)
(filtered-files '()))
(let ((excluded-dirs (mapcar (lambda (dir)
(file-name-as-directory (expand-file-name dir cache-directory)))
exclude-dirs)))
(dolist (file (directory-files-recursively cache-directory "\\.org$"))
(let ((file-directory (file-name-directory file)))
(if (or (string-prefix-p "." (file-name-nondirectory file))
(cl-some (lambda (excluded-dir)
(string-prefix-p excluded-dir file-directory))
excluded-dirs))
(when debug
(message "Filtered out file: %s (Excluded directory)" file))
(progn
(when debug
(message "Keeping file: %s" file))
(push file filtered-files)))))
(nreverse filtered-files))))
(defun 0/org-headlines-file-member-p (org-file &optional debug)
"Check if ORG-FILE is a member of the filtered org files in the cache directory."
(let* ((filtered-files (0/org-filter-excluded-directories))
(org-file-abs (expand-file-name org-file))) ; Get the absolute path
(when debug
(message "Filtered files: %s" filtered-files))
(if (member org-file-abs filtered-files)
t
nil)))
;; debug
(defun 0/org-headlines-show-entries ()
"Read and display all key-value pairs in the 0/org-headlines-hash-table."
(message "Contents of org-headlines-hash-table:")
(maphash (lambda (key value)
(message "[%s] (%s %s)" key (car value) (cadr value)))
0/org-headlines-hash-table))
(defun 0/org-headlines-hashtable-clear ()
(interactive)
(setq 0/org-headlines-hash-table (make-hash-table :test 'equal))
)
I have not been able to come up with some good names to define things – but it works from my side,
currently the limitation is that there cannot be two headlines with the same exact names – this is a limitation because I use the headlines as unique identifiers – tags could be made to separate different headlines if the need is so dire.
Please provide feedback if possible – thanks.
Updates
Update 1
Fixed minor mode protocols
++ Fixed an issue where the minor mode did not respect user choices defined by inclusion and exclusions
++ streamlined the refresh process upon save while minor mode is true (reverted)
Update 2
Fixed an issue where the minor mode was not getting set correctly (still might need some fix)
Fixed an issue where .dotfiles were being parsed for headlines
Fixed an issue where the extraction process left behind all buffers open
Update 3
Made all debug outputs as optional to stop it from spamming the messages buffer since functions have reached stability and reliability in testing.
Update 4
Final Update: Fixed an issue where inserted links were improperly formatted.
