Interoperability between Org Roam and regular Org

How To
#1

Hi, I’ve built a little graph in Org Roam:

Screenshot from 2020-09-05 13-57-19
Screenshot from 2020-09-05 13-57-191664×509 77.4 KB

I’d like to “linearize” this and work on it inside of one (big) regular Org Mode file, including the ability to export to downstream formats like PDF. I’d also like to save changes back into the Org Roam notes.

The obstacle that comes to mind is links like [[file:20200810132653-top.org][Top]]. Inside of the “downstream” Org file, this should presumably look like [[*Top]] (assuming that Org Roam notes get mapped to top-level sections). All well and good, but then where does the file:20200810132653-top.org information go, so that it can be re-inserted later?

I’m sure several solutions are possible. Curious to know if there’s a “standard” way to do this yet!

#2

Hi, that of course should go into a property under the respective headline. I’m curious, however, whether you’ve already succeeded in linearising your Org roam files in the first place?

#3

that of course should go into a property under the respective headline.

Yeah, that makes sense, though the script to restore things back to Org Roam would have to know to look there, and would need to reinflate things appropriately.

Anyway, here’s what I came up with so far, dealing with the Org Roam to Org direction. This seems to work fine though it may not be the ideal solution yet.

roam2org.sh

#! /bin/bash

emacs --batch -l ~/bin/downsample-org-roam.el --eval "(combine-files)" "$@"

and,

downsample-org-roam.el

(defun downsample ()
  ;; node title becomes a top-level section
  (if (looking-at "^#\\+TITLE:")
      (replace-match "*"))
  (forward-line 1)
  ;; roam tags become org properties
  (if (looking-at "^#\\+roam_tags:\\(.*\\)")
      (replace-match ":PROPERTIES:
:tag:\\1
:END:"))
  ;; All subsections go down in depth
  (while (re-search-forward "^\\*" nil t)
    (replace-match "**"))
  (goto-char (point-min))
  ;; Links to files are replaced with internal links to sections
  (while (re-search-forward "\\[\\[file:\\([^]]*\\)\\]\\[\\([^]]*\\)\\]\\]" nil t)
    (replace-match "[[*\\2][\\2]]"))
  (concat "# " (buffer-name) "\n" (buffer-substring-no-properties (point-min) (point-max))))

;; `command-line-args' corresponds roughly to $@ in the shell
(defun combine-files ()
(princ
(apply #'concat
(mapcar (lambda (file) (progn (find-file file)
                           (downsample)))
        (nthcdr 5 command-line-args)))))

Then just run:

roam2org.sh *.org

(Edited after I improved the script somewhat!)

1 Like
#4

Does it actually alter your original Org roam files? Usually one would do something like

(let ((contents (buffer-substring-no-properties (point-min) (point-max))))
    (with-temp-buffer
       (insert contents)
       ;; ... do the processing
  ))

For writing changes back into an Org roam file, I would recommend you to look at the built-in Org parser, the org-element library.

Maybe also the forward parsing could be done by means of org-element and other standard tools, because your approach, for example, will work only when org-odd-levels-only is set to nil, that is each nested headline gets only one additional star. Many people, however, use org-odd-levels-only, in which case each nested headline gets two stars.

* Parent
*** Child 1
***** Child 2

It can be painful to handle this and possibly other situations without re-inventing parts of the built-in Org functionality. For promoting subtrees, see the org-promote-subtree function.

#5

Yeah, I rewrote it with with-temp-buffer for direct use from within Emacs. Since the version I wrote above ran in a script and didn’t ever “save” the contents, it could be a bit more hacky.

I also ended up using the org-element library for subsequent downstream processing of the outputs (because I wanted to nest elements that were not tagged as “high level”), viz., here’s what I needed:

(org-map-entries (lambda ()
                   ;; don’t demote the top level items and their sub-items
                   (if (string= (org-entry-get nil "tag") "HL")
                       (progn (org-end-of-subtree)
                              (setq org-map-continue-from (point)))
                     ;; demote everything else
                     (org-do-demote)))
                 nil 'file)

It certainly makes sense to use a similar process to set up the nesting in the initial pass as well. Probably I can clean this up and get it into a state where it can go into a “contrib” commit. (I’m not sure what the protocol is for that, though it looks like you discussed a potential contrib workflow with Jethro at some point!)

#6

Well, thanks to @zaeph and @jethro, that contrib code had quickly grown into a separate package, org-roam-bibtex, of which I was initially sceptical, and which eventually incorporated more functionality than was intended in the beginning. This is a plausible way to bring your ideas and their realisations into Org Roam’s infrastructure.

I actually envisioned something like this for org-roam-bibtex as a means of producing annotated bibliography for printing out. Moreover, such an annotated bibliography was one of the earliest features requested by others, as were the requests to provide a means for migrating from one file-style notes to Org Roam. So, both one way and round trip general Org Roam–Org export functionality with some perks like filtering, sorting and so on, will find its users, and implementing it is a worthy project.

#8

I’d like to go the other way; I have a giant org file I’d like to port:

OneGiantOrgFile --> org-roam.

Has anyone written something doing that? My big org file has the usual org-mode outline hierarchy, so I’d love to see how other people have mapped an outline into small org-roam notes.

But in addition, my big org file has a bunch of manually created links. For these, the thing being linked to is always a dedicated target with some unique, usually meaningful name, either in text:

Clicking on a link brings you here: <<targetName>>

or on a headline like so:

** <<targetName>>

or like so:

** Some text <<targetName>>

The referencing side of the link is an internal link, either on a headline or in ordinary text:

** [[targetName]]
** another headline
   This is a reference with a description: [[targetName][targetDescription]]

When the <<targetName>> is in text, the text around it could be turned into a separate org-roam note, possibly including the headline above it (if it also doesn’t contain a dedicated target) and any headlines underneath it. If the stuff underneath it contains a <<targetName>>, then this doesn’t have to be in a separate note, but could be somehow linked to from other notes.

When the <<targetName>> is on a headline, the headline and everything underneath it could become a separate org-roam note.

Any suggestions for where to start?

#9

Here is a basic exporter for Org→Org Roam. It doesn’t do anything with the <<targetName>> things you described. So, I guess this would be a start! The conditional behaviour you are asking about could draw on these patterns and some (while (re-search-forward ...)) testing.

;; Here’s some code to turn a large Org mode file into sub-files
;; ready for Org Roam

(defun org-get-headline ()
  (org-back-to-heading t)
  (looking-at org-complex-heading-regexp)
  (match-string-no-properties 4))

(defun org-get-body ()
  (save-excursion
    (save-restriction
      (widen)
      (let* ((elt (org-element-at-point))
             (title (org-element-property :title elt))
             (beg (save-excursion (org-end-of-meta-data t) (point)))
             (end (save-excursion (org-end-of-subtree) (point))))
        (buffer-substring-no-properties beg end)))))

(defun zoom-in-and-enhance ()
  "Process an Org Mode buffer into multiple Org Roam-style files.
Changes headers to titles."
  (org-map-entries
   (lambda ()
     (let* ((title (org-get-headline))
            (contents (org-get-body)))
       (with-temp-buffer
         (insert "#+TITLE: " title "\n"
                 contents)
         (org-mode)
         (org-map-entries #'org-promote-subtree)
         (write-file (concat
                      (funcall
                       org-roam-title-to-slug-function
                       title) ".org")))))
   "LEVEL=1"))
#10

Thanks very much for the code, @holtzermann17, that is indeed a start. I’ll see what I can do to fill in the details.