How to dynamically create ID to other files?

I have many org-roam v2 files and want to link subheadings from each other. Currently, the approach I am following is-

  1. Use org-store-link to generate ID for the (sub) heading in file A.
  2. Use org-roam-node-insert to insert the previously generated ID in file B.

I wish to automate these 2 steps into 1. There should be a way in which I can select all the subheading in all org roam files and just select it to insert a link to it. If there was no ID property to it, then create it, else, use it.

I have found counsel-org-link to work similarly according to my requirement, but, it works for only a single file. I want to work with multiple files. Please help.

1 Like

I think you’d need to create your own code.

Below is an outline of the program I’d make.

You’d need a basic understanding of Elisp and how marker works (easy), and familiarity with libraries org-element and completing-read (will take time…).

It is not difficult, but time-consuming to get right (this is as far as I can do).

  • Function org-roam-list-files gives you a list of all the files in a list
  • For each .org file in the list, get all the headline elements (you should be able to use the built-in org-element library to parse each .org file and get the headlines with the marker (file and position) and headline title
  • Create a minibuffer selection. This would depend on your completion framework (especially Ivy/Counsel), but for Vertico/Consult, it would be completing-read (you can see how completing-read does this)
  • Based on the user’s selection of the headline from the minibuffer, you can visit the file and headline, and get the existing ID or create an new one (eg org-id-get-create)
    EDIT: Sorry, you mention org-store-link – I think it will work too, assuming your configuration is correctly set to create an ID when it does not exist
  • At this stage, the new ID is not in the org-roam-db, so you’d need construct the ID link (not difficult because you have the ID and the headline title)
    EDIT: If you use org-store-link, then you should be able to use org-insert-link for the ID
1 Like

I like the idea - I already have a way to parse all the headings of a file - I think I can code this.
But we have to solve a few problems. Firstly if we search for all the headings in all the files the process might be cpu intensive - can you work with first selecting the file then getting a list of the subheadings and headings?

or do you want to arbitarily search for all the subheadings and headings? We can cook something up. Lets work on it - let me know how you conceive of the workflow.

An alternative is to use regex for the headers in the files in org-roam-dir. This is probably better for performance than my first suggestion to use org-element to parse the files.

1 Like

I have tried to use hashtables to do this -


;; beach (variables)
(defvar 0/org-headlines-hash-table (make-hash-table :test 'equal)
  "Hash table containing headlines (keys),
 their corresponding positions in the file (value 1), and the file names (value 2).

A Hash table is a list-like structure containing pairs: Keys and Values")

(defvar 0/cache-directory org-roam-directory
  "Directory to cache Org files for headline extraction.")

(defvar 0/cache-directory--exclude '()
  "Name of directories to exclude from headline extraction.
Subdirectories will be excluded too.")

;; user functions
(defun 0/org-goto-headline ()
  "Navigate to the position of a headline selected from the 0/org-headlines-hash-table."
  (interactive)
  (let ((headline (completing-read "Choose headline: " (hash-table-keys 0/org-headlines-hash-table) nil t)))
    (when headline
      (let ((position (gethash headline 0/org-headlines-hash-table)))
        (if position
            (progn
              (find-file-other-window (cadr position)) 
              (goto-char (car position))
              (message "Navigated to %s in %s" headline (cadr position)))
          (message "Headline '%s' not found in hash table" headline))))))

(defun 0/insert-org-link-to-headline ()
  "Create an Org link to a headline selected from the 0/org-headlines-hash-table and insert it into the current buffer."
  (interactive)
  (let ((current-buffer (current-buffer))
        (headline (completing-read "Choose headline: " (hash-table-keys 0/org-headlines-hash-table) nil t)))
    (when headline
      (let ((position (gethash headline 0/org-headlines-hash-table)))
        (if position
            (save-excursion
              (let* ((org-file (cadr position))
                     (org-buffer (find-file-noselect org-file)))
                (with-current-buffer org-buffer
                  (goto-char (car position))
                  (let* ((headline-text (org-get-heading t t t t))
                         (id (org-id-get-create)))
                    (save-buffer)
                    (switch-to-buffer current-buffer)
                    (insert (format "[[id:%s][%s]]" id headline-text))
                    (message "Inserted Org link to headline '%s' with ID '%s' in current buffer" headline-text id)))))
          (message "Headline '%s' not found in hash table" headline))))))

(defun 0/org-extract-headlines-from-directory (&optional clear-cache debug)
  "Recursively extract headlines from all Org files in the cache directory, excluding directories specified in `0/cache-directory--exclude`."
  (interactive "P")
  (when clear-cache
    (0/org-headlines-hashtable-clear))
  (let ((files-to-process (0/org-filter-excluded-directories)))  ; refer to abyss below
    (dolist (file files-to-process)
      (when debug
	(message "Processing file: %s" file))
      (0/org-extract-headlines file))))

;;anchors (and hooks)
;; Define a function to update the hash table upon saving a buffer
(defun 0/update-hash-table-on-save ()
  "Update the hash table upon saving the current buffer."
  (when (and (eq major-mode 'org-mode)
             buffer-file-name)
    (let ((org-file (buffer-file-name)))
      (when org-file
        (when (0/org-headlines-file-member-p org-file)
	  (0/org-delete-keys-for-file org-file)
          (0/org-extract-headlines org-file))))))

;; Define a global minor mode
;;;###autoload
(define-minor-mode 0/org-auto-update-hash-table-mode
  "Global minor mode to automatically update the hash table upon saving Org files."
  :global t
  :init-value nil
  :lighter " 0/rHT"
  (if 0/org-auto-update-hash-table-mode
      (progn
        ;; Set up after-save hook to update the hash table
        (add-hook 'after-save-hook #'0/update-hash-table-on-save)
        ;; Clear the hash table
        (setq 0/org-headlines-hash-table (make-hash-table :test 'equal))
        ;; Refresh the hash table
        (0/org-extract-headlines-from-directory))
    ;; Remove hook if mode is deactivated
    (remove-hook 'after-save-hook #'0/update-hash-table-on-save)))

;; seabed (underlying functions)
(defun 0/org-extract-headlines (org-file &optional clear-cache debug)
  "Extract and store all headlines from ORG-FILE in 0/org-headlines-hash-table.
If CLEAR-CACHE is non-nil, clear the existing cache before extraction.
If DEBUG is non-nil, display a message after populating the hash table."
  (interactive "fEnter Org file: \nP")
  (when clear-cache
    (setq 0/org-headlines-hash-table (make-hash-table :test 'equal))) ; reset the cache when requested
  (with-temp-buffer
    (insert-file-contents org-file)
    (goto-char (point-min))
    (while (re-search-forward "^\\*+\\s-+\\(.*?\\)$" nil t)
      (let ((headline (match-string 1)))
        (puthash headline (list (line-beginning-position) org-file) 0/org-headlines-hash-table))))
  (when debug
    (message "Populated hashtable with contents from %s" org-file)))

(defun 0/org-delete-keys-for-file (org-file &optional debug)
  "Delete all keys associated with the specified Org file from the 0/org-headlines-hash-table."
  (interactive "fEnter Org file: ")
  (let ((keys-to-delete '()))
    (maphash (lambda (key value)
               (let ((file-path (cadr value)))
                 (when (file-equal-p (expand-file-name file-path) (expand-file-name org-file))
                   (push key keys-to-delete))))
             0/org-headlines-hash-table)
    (dolist (key keys-to-delete)
      (remhash key 0/org-headlines-hash-table)))
  (when debug
    (message "Deleted keys associated with %s from the hash table." org-file)))

;;abyss (helper functions)
(defun 0/org-filter-excluded-directories (&optional debug)
  "Return a list of files in the cache directory excluding directories listed in `0/cache-directory--exclude` and their subdirectories.
When DEBUG is non-nil print verbose output."
  (let ((exclude-dirs 0/cache-directory--exclude)
        (cache-directory 0/cache-directory)
        (filtered-files '()))
    (let ((excluded-dirs (mapcar (lambda (dir)
                                   (file-name-as-directory (expand-file-name dir cache-directory)))
                                 exclude-dirs)))
      (dolist (file (directory-files-recursively cache-directory "\\.org$"))
	(let ((file-directory (file-name-directory file)))
          (if (or (string-prefix-p "." (file-name-nondirectory file))
                  (cl-some (lambda (excluded-dir)
                             (string-prefix-p excluded-dir file-directory))
                           excluded-dirs))
              (when debug
                (message "Filtered out file: %s (Excluded directory)" file))
            (progn
              (when debug
                (message "Keeping file: %s" file))
              (push file filtered-files)))))
      (nreverse filtered-files))))

(defun 0/org-headlines-file-member-p (org-file &optional debug)
  "Check if ORG-FILE is a member of the filtered org files in the cache directory."
  (let* ((filtered-files (0/org-filter-excluded-directories))
         (org-file-abs (expand-file-name org-file))) ; Get the absolute path
    (when debug
      (message "Filtered files: %s" filtered-files))
    (if (member org-file-abs filtered-files)
        t
      nil)))

;; debug
(defun 0/org-headlines-show-entries ()
  "Read and display all key-value pairs in the 0/org-headlines-hash-table."
  (message "Contents of org-headlines-hash-table:")
  (maphash (lambda (key value)
             (message "[%s] (%s %s)" key (car value) (cadr value)))
           0/org-headlines-hash-table))

(defun 0/org-headlines-hashtable-clear ()
  (interactive)
  (setq 0/org-headlines-hash-table (make-hash-table :test 'equal))
  )



I have not been able to come up with some good names to define things – but it works from my side,

currently the limitation is that there cannot be two headlines with the same exact names – this is a limitation because I use the headlines as unique identifiers – tags could be made to separate different headlines if the need is so dire.

Please provide feedback if possible – thanks.

Updates

Update 1

  • Fixed minor mode protocols

++ Fixed an issue where the minor mode did not respect user choices defined by inclusion and exclusions

++ streamlined the refresh process upon save while minor mode is true (reverted)

Update 2

  • Fixed an issue where the minor mode was not getting set correctly (still might need some fix)
  • Fixed an issue where .dotfiles were being parsed for headlines
  • Fixed an issue where the extraction process left behind all buffers open

Update 3

  • Made all debug outputs as optional to stop it from spamming the messages buffer since functions have reached stability and reliability in testing.

Update 4

  • Final Update: Fixed an issue where inserted links were improperly formatted.

Cleaned up naming scheme to org-traverse and moved to gist
leave feedback if you find it useful, thanks :slight_smile: