Now I have been using org-roam for six months, and I would like to mention one observation I have.
Before coming to org-roam, I briefly had used Logseq. Logseq is another program that lets you have zettelkasten-style notes on your computer. In Logseq, when you start the program, you are dropped into your day’s file. Each day you get a new day’s file. This is the same as in org-roam’s dailies pages.
Since I came to using org-roam from Logseq, I immediately liked the familiarity of org-roam dailies. And, I also center my org-roam usage around my dailies pages. I have specific org-roam pages I insert using C-c n i, but these are mainly empty files, just for aiding me in linking and finding links between org-roam subheaders scattered around different dailies (I use C-c n l to find references and thus find my scattered notes around a Book’s Title, for example).
Because of that, everything, including my todo items are scattered around the many daily files I have (I currently have 191 of them).
So, one thing I noticed is that when I access my org-agenda, org-mode has to scan all those 191 org-roam dailies files and check for TODO keywords. In the past, I mentioned that here:
Back then, it was taking org-agenda to complete this scanning 10 seconds or so. Now, with 191 daily files containing TODO keywords, I noticed that it is taking around 15 to 20 seconds.
This doesn’t render org-agenda unusable whatsoever. The 20 seconds wait time occurs only for the first time it builds the org-agenda view, and the subsequent calls to org-agenda seems to use the cached data and I don’t wait another 20 seconds. This is still acceptable and I will keep using it that way.
Well if you want someday to reduce the time taken by not scanning DONE items - it is trivial to do this, by leveraging the sqlite database that roam already implements - infact org-roam already does store the TODO status - org-agenda is not made aware of it by default.
This will disable org-agenda mechanism in favour of leveraging org-roam’s infrastructure.
Tags in agenda would be broken nevertheless.
But if you inisist on not leveraging org roam here. It would be still beneficial to not create a new file node for everyday. Why not simple add a headline to your monthly or yearly register? That way agenda doesnt have to go around the world opening every org file reading from top to bottom then to the next to the next. The bottleneck is surely the sheer amount of file io operations. If it had to open 1-2 files no matter how many entries in them it would be lightning fast.
The user is always put in control in emacs. If the system is slow, it is because of operator error 98% of the time. 2% is because of using inefficient protocols and code, but the fault is the users too for they have allowed this inefficiency.
Such a system that takes 15 sec to load everytime is a nightmare if you do elisp coding and have to restart over and over to test it from point 0. Youdnt find it a problem if you dont do elisp coding much, since youd have to deal with it maybe max 1-2 times a day. But if in the future you do, youdnt find it sustainable.
I also use dailies (and project and area files) for todo items.
This is what I do:
I have a file tag called ‘dmgtodo’. In a hook when saving the file, the file is scanned, and if there is a todo item (TODO, NEXT, etc). the tag is added to the file. If there is none, it is removed.
Before I run agenda, I read the org-roam database for files that have this tag. From this result I dynamically create the org-agenda-files.
This way org-agenda only scans the files that actually have items (around 10% of the files)
Try the gist I posted here from vulpea - it doesn’t do this. You do not need to make an id for every todo. @/d12frosted is a good programmer - they wouldnt approve of suboptimal workflows
The problem I see with this approach from reading the code is that it opens up a buffer with the contents of ripgrep as text - org-agenda is interactive and uses its own mode org-agenda-mode. So we lose all interactivity in here -
I use ripgrep for many things - but it should be weighed against using the database circumstance wise. I feel using ripgrep is non efficient for the task at hand.
org-agenda-mode uses the symbol org-agenda-files to get a list of files that it should use to create the agenda. Via the hook, the function is called before org-agenda-mode - and ripgrep is so fast that there is little to no delay in generating that list (even with a few thousand org files). The interactivity is not lost. I have been using this little hack for more than a year and it works flawlessly (for me, that is).
org-agenda itself is the bottleneck. If there are more than a hundred org-files to be considered, building the org-agenda starts to take time.
(In fact, rg is so fast and I had so many issues with the database of org-roam v2 that I wrote my own note-taking system based on rg and some of the ideas of org-roam v1 (see here Orgrr - org-roam-ripgrep) Still using it daily.)
Okay I understand - I didn’t know the timing of the hook - so as i get it ripgrep retrieves a list of files - then org-agenda takes over - it parses each of these files then creates it own thing,
The problem as you stated is in the number of files - in the processing thereof - maybe a good solution – and i am just throwing ideas – I have not seen the functions of org-agenda very closely because I do not keep my TODOs in waiting so all in all I never have more than 1-2 files with todos at a particular time, – if there exists a symbol that stores the list of processed items required by agenda - such as position and text of a todo item - we can create a small database that already precaches all these things - preferably work without utilising org-ids. I think it could be done subject to we can tap in between the process – but I don’t know much about the problem at hand in detail.
The alternative solution for me has been to become a grown-up and to limit the number of (project) files with todos to be less than 100 (see also your solution). Org was designed to deal with a few (potentially very long) files, not with many small ones.
