I’ve been using org-roam for nearly a year and recently upgraded to v2 as part of migrating to doom on a new laptop.
I experienced no problems for a week or so, but then last night noticed that org-roam couldn’t sync with the db. Emacs hung (emacs-sqlite process using 0% of CPU, but emacs itself using 100%) and the minibuffer read “Processing modified files…”.
I moved the db to a backup location, restarted emacs, and ran org-roam-db-sync. This seemed promising as I now began to see “Processing modified files…1%”, “Processing modified files…12%”, and so on. The process then hung again at “Processing modified files…61%”. I left it overnight and returned to find it in exactly the same place. I’ve repeated this process multiple times: after a round of doom clean && doom sync -u and after running org-id-update-id-locations and org-roam-update-id-locations (wondering if a duplicate id was causing an issue). Most recently, the sync made it to just 41% before hanging.
I have 528 notes with about 34000 lines of text total.
Here’s my diagnostic information:
Emacs: GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.30, cairo version 1.16.0)
Framework: Doom
Org: Org mode version 9.6 (9.6-??-2e99997 @ /home/rule/.config/doom/straight/build-29.0.50/org/)
Org-roam: 3e47f19
Can you help me get my db back up and running? Thanks!
If it started happening at certain point in recent time, I’d try two things, both done with db deleted (back it up in case):
Divide org files into two buckets: updated before and after the point in time
Move the files in the “after bucket” and try db-sync. If there is no problem with only the before bucket, then you will search which file(s) are the cause in the after bucket. You may have mis categorised before and after; try with only the after bucket as well. If you can’t find the cause in files, then it may be somewhere in a program
Remember the updates to packages, or newly installed ones. Revert them and try db-sync
If you can’t do this, I would suggest to remove Doom, and try only with Org-roam and its dependencies. You’ll find if the issue is somewhere in the packages
If it’s not files nor packages… then it’s going to be a lot harder to find the cause, but probably it’s either 1 or 2.
I removed the updates I’ve made in the last week and rebuilt the database in ~30s with no difficulties. I then restored the updates from this past week and again synced the database with no errors. Everything seems to be working well again.
I don’t understand the org-roam internals very well. Is there any reason to expect this sort of behavior? Why would attempting to build a database with all the files fail but building it in two chunks would succeed?
This is an unexpectedly good result; congratulations!
The two-bucket way should have made no difference to db-update.
My hypothesis was that some part of the data was causing the problem. The two-bucket approach was just an easy way to narrow down the file(s) in question.
You could try db-update in one go and see if the problem occurs again, if you are curious. If it does, you may be finding a hitherto unknown issue. But my hunch is that somehow the problem is gone.
One possibility may be that you sound like you use Git for your notes (“restore updates”). If so, there may be something that Git does to the data in Org-roam-directory that influences db-update process. Only a speculation…
Your hunch is correct. I just removed the working db and regnerated it in one go without issue.
I do use git, but I don’t have any commit hooks, etc. Perhaps simply saving all the files and committing them rectified some issue, but I don’t know exactly what it would have been.
It looks like the problem is a race condition with the database. A DB transaction is made for each file. But each of this transactions has lots of inserts to the same table.
org-roam uses emacsql-with-transaction which has a HUGE warning about side effects. The code in org-roam has lots of side effects.
One more think to consider when the processing for a file is redo to avoid calling an insert for each item in the current file.
my instance gets stuck in org-roam-db-insert-link.
I erased the database and it was able to recreate it without problems.