[TIPS] dynamically add Org-roam files to your agenda file

How To Tips
#1

Dear folks:

An org-roam user (lld2001) post a very useful function to dynamically
add Org-roam files that contain TODO to your agenda files.

(defvar dynamic-agenda-files nil
  "dynamic generate agenda files list when changing org state")

(defun update-dynamic-agenda-hook ()
  (let ((done (or (not org-state) ;; nil when no TODO list
                  (member org-state org-done-keywords)))
        (file (buffer-file-name))
        (agenda (funcall (ad-get-orig-definition 'org-agenda-files)) ))
    (unless (member file agenda)
      (if done
          (save-excursion
            (goto-char (point-min))
            ;; Delete file from dynamic files when all TODO entry changed to DONE
            (unless (search-forward-regexp org-not-done-heading-regexp nil t)
              (customize-save-variable
               'dynamic-agenda-files
               (cl-delete-if (lambda (k) (string= k file))
                             dynamic-agenda-files))))
        ;; Add this file to dynamic agenda files
        (unless (member file dynamic-agenda-files)
          (customize-save-variable 'dynamic-agenda-files
                                   (add-to-list 'dynamic-agenda-files file)))))))

(defun dynamic-agenda-files-advice (orig-val)
  (union orig-val dynamic-agenda-files :test #'equal))

(advice-add 'org-agenda-files :filter-return #'dynamic-agenda-files-advice)
(add-to-list 'org-after-todo-state-change-hook 'update-dynamic-agenda-hook t)

I find it is useful and hope it can be useful to you as well.

2 Likes
Org-roam-dailies: move previous day's TODOs
#2

Thank you for sharing it. The problem with this approach is that (a) you can’t plug it into existing pile of notes and (b) messing up with your customize file leads to broken agenda.

This post motivated me to share how I approach this problem that I am using for a month already and super happy about it’s performance. You can read detailed description on my blog. The code is available as a GitHub Gist.

TL;DR - in the before-save-hook current buffer is traversed for existence of any TODO entires and Project tag is either added or removed from #+ROAM_TAGS property. Now, org-agenda is adviced before to set the value of org-agenda-files to the list of files with Project tag (quieried from db, which is enourmously fast).

There is also a snipped to use for migration. This way all required information is always there, in your files and can be easily reconstructed. That being said, proposed solution can be easily adapted to dynamic-agenda-files approach.

Hope someone finds it useful :slight_smile:

5 Likes
#3

This code snippet for collecting files with todos has greatly impacted my workflow! Thanks for sharing.

I am just trying to figure out how this would work with org-roam V2. ROAM_TAGS have been dropped and from the code it seems that FILETAGS will only get stored in the database if there are headline nodes with ID that inherit the tag. And even if this would work, every todo in the agenda woud be tagged with “Project”, adding visual clutter.

Have you adapted this for the new version already? Would it make sense to use a note that is being linked to by every file that contains todos and then query the database for backlinks?

1 Like
#4

Hi @S_Fabris

I cannot contribute to @cool_ran’s script, but wanted to respond to two points you commented about V2.

I have been using V2 every day for work for a couple of weeks now; FILETAG just works without headline nodes.
Most of my org files do not have headlines. I use FILETAG as a category, like Meeting, Reference, etc. No issue.

I think this is largely related to how you use/customise agenda (I do not show any tags in my agenda view; I just use them as search keys)… You could also look to control how file-level tags get inherited by the looks of this Org documentation.

To limit tag inheritance to specific tags, or to turn it off entirely, use the variables org-use-tag-inheritance and org-tags-exclude-from-inheritance .

I don’t use these variables so I cannot be 100% certain, but it’s probably worth looking into.

1 Like
#5

Thanks for the hint to org-use-tag-inheritance, this keeps the agenda view clean as I do not use tags for other purposes than marking files with todos inside.

#6

After some time testing, this does not work for me. Filetags are somehow not consistently added to the database. It worked for the some daily files with tasks and for some the caching function does not add the filetags to the db entry.

However, the modified code from @d12frosted works flawlessly in adding the tags and retrieving them from the db.

Has anyone else used filetags successfully with V2?

#7

This issue you experienced might be caused by this technical issue, which has been resolved.

1 Like
#8

That’s it! Everything working now!

1 Like
#9

This is a very interesting idea for workflow that might work for me well.
I’m currently trying to make the code work at least partially with org-roam v2.

What I want is to add to org-agenda-files files with “Project” tag. This is implemented by those two functions:

(defun vulpea-project-files ()
  "Return a list of note files containing Project tag."
  (seq-map
   #'car
   (org-roam-db-query
    [:select file
     :from tags
     :where (like tags (quote "%\"Project\"%"))])))

(defun vulpea-agenda-files-update (&rest _)
  "Update the value of `org-agenda-files'."
  (setq org-agenda-files (vulpea-project-files)))

When I run vulpea-project-files function, I receive the following error:
EmacSQL had an unhandled condition: "no such column: file"

I tried to investigate the structure of org-roam.db with .schema command to understand which column name should I use. Here is what I got:

2021-05-08-174636_1920x1034_scrot
2021-05-08-174636_1920x1034_scrot1920Ă—400 80.2 KB

Given this structure, shouldn’t it be working as expected? (I haven’t used sql in my life, but it looks that there is a column with name file).

Thank you

#10

I will upgrade that for V2 (as well as other posts).

1 Like
#11

The problem here is that scheme was changed in V2. But the good thing, I’ve updated my post. In short the query should look like this:

(defun vulpea-project-files ()
  "Return a list of note files containing 'project' tag." ;
  (seq-uniq
   (seq-map
    #'car
    (org-roam-db-query
     [:select [nodes:file]
      :from tags
      :left-join nodes
      :on (= tags:node-id nodes:id)
      :where (like tag (quote "%\"project\"%"))]))))

Please let me know if that works for you!

1 Like
#12

Hey @d12frosted,
your code works flawlessly!

Thank you so much for your effort in sharing your workflow. This is valuable.

1 Like
#13

Slight mod that I had to figure out to keep specific files in the agenda. As the functions does’t append, it overwrites. Please Educate me of a better way to do this…

(setq  marty/org-agenda-files (list
       (concat org-directory "Tasks.org")
       (concat org-directory "Habits.org")
       (concat org-directory "Calendar.org")
       (concat org-directory "contacts.org")
       (concat org-directory "Someday.org")
       (concat org-directory "0mobile.org")
       "~/.cache/calendar/google.org"
       "~/.cache/calendar/personal.org"))

  (setq org-agenda-files marty/org-agenda-files)


(defun vulpea-agenda-files-update (&rest _)
  "Update the value of `org-agenda-files'."
  (setq org-agenda-files
        (append marty/org-agenda-files (vulpea-project-files))))
#14

Sure, the method I described simply overrides whatever you had in org-agenda-files. If you wish it to always contain some values you have two options:

  1. A method you described with a separate list of agenda files that is concatenated with project files. Works well even for notes outside org-roam-directory.
  2. Tag these files with an appropriate tag, so that vulpea-project-files result contains them. Doesn’t work if files in question are outside of org-roam-directory (like your calendar files).
#15

woo i’m 1/8th step past Noob… cool. only took 3 years…

1 Like
#16

Is there a way to hook this to something or advise a function so that it runs every time the database updates?

#17

Sure, you can hook into org-roam-db-update-file and org-roam-db-sync via advice-add, e.g.

(advice-add 'org-roam-db-update-file :after #'vulpea-agenda-files-update)
(advice-add 'org-roam-db-sync :after #'vulpea-agenda-files-update)

I think these two functions should be enough. Though I must ask - why do you want to do it? :slight_smile: DB gets updated much more often than your list of agenda files gets modified. At least in my experience. So I would get unnecessary reads from DB. And this might lead to performance degradation (though this is mere speculation).

1 Like
#18

Thanks. It is indeed automation overkill - just interested how it would work.

#19

The code is fantastic—thanks d12frosted for sharing it.

I’d like to supplement it with a function that checks whether each file in the org-roam directory was modified in the last n days, so that org-agenda lists both files with a TODO element and files modified recently. Unfortunately, I am not able to create such a function. The shell command find "path-to-org-roam-directory" -mtime -n -ls finds the files modified in the past n days, and perhaps one could pass its output to shell-command-to-string to generate a list of the relevant directories, but I’m not sure how to do that, or whether a different approach is more appropriate. Any help would be appreciated.

#20

Okay, it looks that I was just missing split-string. The following function generates the list of files I wanted:

(defun ps/org-roam-recent (days)
  "Return list of files modified in the last DAYS."
  (let ((mins (round (* 60 24 days))))
    (split-string
     (shell-command-to-string
      (format
       "find %s -name \"*.org\" -mmin -%s"
       org-roam-directory mins)))))
1 Like