I’m a year into Emacs, a few months into Doom Emacs, and two months into mostly happy use of org-roam. After a lot of reading here and elsewhere and several hours of experimenting, I still cannot get the nodes in an org-roam graph running in my browser (Chrome or Safari) to link back to their corresponding notes. When I click on a node I get the following system (Mac Catalina) error message:
Chrome: “There is no application set to open the URL org-protocol://roam-file?” followed by the path to the file in question in my org-roam directory.
Safari: “Safari can’t open :org-protocol://roam-file?” followed by the path to the file in question in my org-roam directory "because macOS doesn’t recongize internet addresses starting with “org-protocol:”.
Except for my simple org-roam-capture template, I believe I’m using the Doom Emacs private configuration for org-roam that I found in the org-roam author’s .doom.d config.el, init.el and packages.el files. I have created the the desktop app for emacsclient following carefully the steps in the org-roam documentation. I’m happy to share any or all of the above configuration or anything else that would help get to the bottom of this. What would be useful for me to share given my challenge?
For MacOS, this section of Org-roam documentation suggests, I believe, two options: use application called Platypus, or use Emacs Mac Port.
Which options did you use?
You created a desktop app for emacsclient, but I am not sure if this is the right way for MacOS – my reading of the documentation is that this option is for Linux, not MacOS.
I am not a Mac user; I invite Mac users to jump in.
Thank you, nobiot. I think you may be focused on the main problem for me, the “enabling of org-roam’s protocol extensions” (quotes here are all from section 14.2 of the org-roam documentation, the Installation section, https://www.orgroam.com/manual/Installation-_00281_0029.html#Installation-_00281_0029).
Before the OS-specific installation instructions begin, the introduction to that section says “we will also need to create a desktop application for emacsclient.” I believe I followed (several times, in case I was making a mistake in previous attempts) the instructions for MacOS, using Platypus to create the desktop app for emacsclient (I also disabled the “confirm” prompt in Chrome using the shell command provided.). I’m not using Emacs Mac Port. I use the Doom Emacs distribution. I have followed precisely, I believe, the org-roam specific configuration settings and code, except for my org-roam-capture template, as I find it in the Doom Emacs configuration files found in the org-roam author’s “dots” folder on GitHub (https://github.com/jethrokuan/dots/search?q=doom.d).
Clearly I’m doing something incorrectly. As you indicate, it might be how I’m setting up org-protocol/org-roam-protocol, but I’m at a loss as to what I should do differently. Before the detailed instructions for creating the app with Platypus, I read that Platypus is “one solution,” indicating to me that there might be others. If anyone knows of other Mac OS solutions for creating this app, other than Platypus, maybe I could make work one of them.
BTW, I love the org-roam graph. It’s is full of useful information about my org-roam files. I’m really looking forward to taking advantage of that information by clicking on the nodes to see those files and thus take full advantage of that terrific visual information.
Is Emacs server up? M-x server-start in Emacs.
I wrote a detailed guide on Windows. The specifics would be different, but fundamental ideas should be applicable.
Can you call the emacsclient app from within Terminal, like
emacsclient text.txt
Does this open a correct buffer in Emacs (while Emacs server is active)?
Thanks again, nobiot, for the help.
Yes, Emacs server is running, and I can call emacsclient from the terminal.
I don’t know Windows, but your org-roam guide seems fantastic. If, as you describe for Windows, I should be able to display a note on a Mac running Doom Emacs by running the shell command with the path to emacsclient then “org-protocol://roam-file?file=” followed by the path to my org-roam note, then this may be where my problem is, or part of it anyway. When I run that command in my shell, the system tells me it’s “Waiting for Emacs…” and then times out and returns to the command line.
I have what I think is the equivalent of the
(require 'org-protocol)
(require 'org-roam-protocol)
you mention in your guide in my config.el file. This is what I have:
"(use-package! org-protocol)
(use-package! org-roam-protocol
:after org-protocol)"
(The org-roam-protocol use-package! lines are in the org-roam author’s /.doom.d/config.el, and I’ve had it in my corresponding file for most of my effort to get this to work. I just added the use-package! line for org-protocol after reading your guide and before performing the tests you suggest.)
Can you discern from the above the mistake I’m making?
And/or could it be that I’ve not created my org-protocol app correctly?
If it would help for me to upload more of my config.el file or screenshots of my Platypus settings for the desktop app, I’m happy to do so.
I think you should be able to. By my reading, Platypus works on top of this Terminal part working. It looks to be an equivalent of the Windows registry for Mac – it connects the URL scheme org-protocol and the script, thus becoming the glue between browser and Emacs. The connection points should be like this:
Browser -> URL -> Platypus -> script -> emacsclient
So… Before checking Platypus, I think you still need to check the emacsclient and script.
You checked this:
… the shell command with the path to emacsclient then “org-protocol://roam-file?file=” followed by the path to my org-roam note …
When I run that command in my shell, the system tells me it’s “Waiting for Emacs…”
Is the shell command your version of the script called launch_emacs.sh in the manual?
#!/usr/bin/env bash
/usr/local/bin/emacsclient --no-wait $1
Question 1. Does the shell work if you call it directly in Terminal like this?
(I think the file name should be surrounded by double quotes “”, but Mac may not need it… You could try with and without).
path/to/launch_emasc.sh "test.txt"
Question 2. What happens when you call emacsclient directly in Terminal with one argument "test.txt"
path/to/emacslient "test.txt"
Question 3.
What happens when you call emacsclient directly in Terminal with one argument for org-protocol "org-protocol://roam-file?file=path/to/your/note.org"
path/to/emacslient "org-protocol://roam-file?file=path/to/your/note.org"
Answers should tell you where in the chain of commands there is a breakdown. At the moment, it’s not certain whether the problem is in the script or emacsclient, before Platypus.
I’m not a Mac person but I’ve debugged a fair share of little scripts here and there and I think these useful tricks will help for debugging shell scripts and should work whereever.
-
Try to run the command from the terminal to make sure it works by hand.
-
Put in a line at the top that does something like this, here
$0prints out the name this script is running as and$*will print out all the arguments to the script one after the other with a space between them (here>>is append and it directs the output toshell-script-arguments.login your home directory so every time it runs it will output the time its happening and all the arguments it was given).
echo "$(date) '$0': '$*'" >> $HOME/shell-script-arguments.log
- Think about quoting your arguments just in case they’re getting mangled due to spaces and things
/usr/local/bin/emacsclient --no-wait "$1"
- Think about catching the error output, something like this should run your command and append all the error output into a file called “shell-script-errors.log” (error output is
2and>>is append)
/usr/local/bin/emacsclient --no-wait "$1" 2>> $HOME/shell-script-errors.log
Hello again, nobiot, and thank you for your patient, well organized help.
Yes, my shell command is my version of the script, launch_emacs.sh, as described in the manual. In fact, my version is the same as the manual version. My env file and emacsclient file are located in the same places in my directories.
Question 1: No it does not. When I run mypath/to/launch_emacs.sh (mypath = ~/launch_emacs.sh, in case that makes any difference) followed by “test.txt,” the terminal window flickers for less than a second (seems odd, but that’s what happens) and performs nothing. This happens with or without the script and/or command changes suggested below by twitchy-ears. When I include the script debugging suggestions, the error log is empty and the arguments log records them exactly as I gave them.
Question 2: When I call emacsclient directly in my terminal (thusly: /usr/local/bin/emacsclient “test.txt” – with or without the debugging measures suggested by twitchy-ears below) an emacs buffer, text.txt, opens in my terminal. I can edit and save the file, delete the buffer, and find the file as saved.
Question 3: Exactly the same answer as for Question 1.
Is it fair to conclude from these tests that emacsclient works and one or both of my script, launch_emacs.sh, and org-protocol are not working properly in my setup? What might I do to fix my script (which is exactly what the manual specifies)? And/or, what should I look into with regard to my org-protocol setup?
Unfortunately and so you know how screwed up I am with this, I managed to take a step backward somehow since I started this topic. Whereas I was able to display the graph of my org-roam notes in a browser (just not click on a node and go to the note), I can’t even display the graph now. Org-roam-server runs with the red and blue buttons at the top and bottom of the browser window, but my graph no longer displays. I am not sure what I changed that caused this step backward.
Thank you, twitchy-ears, for the help debugging and monitoring my script. I’m new to scripts, and it is good to know now how to cache errors and a record of arguments.
Do not worry about the issue of graph not displaying. It’s most likely a separately issue. If you have changed nothing, then it is likely to be changes in Doom (did you upgrade Doom in the mean time?) or org-roam-server, or org-roam itself (did you upgrade them?).
I realise it’s frustrating and confusing. You are looking at many separate issues simultaneously in front of you. You can solve them one at a time – this is the only way, I think.
Back to the org-roam-protocol issue…
Yes, but in a twisted way. We cannot say anything about the script yet.
Edit: i think you are right. The script is not working either. Q1 and Q2 should yield the same result if the script worked.
Your test result for Q2 tells us that Emacs server is working. But the one for Q3 tells us that it is NOT working. This is a contradiction.
It’s not org-protocol or org-roam-protocol not working; the terminal command seems to have failed to reach Emacs server (not 100% sure though…). If it’s an issue of org-protocol not working, then you expect an empty buffer to open in Emacs with the buffer with name of some chunks of org-protocol://roam-file?file=your/note.org. This is because Emacs server take the whole org-protocol://roam-file?file=your/note.org as one argument (in the same way as “test.txt” that you confirmed to work). What org-protocol does is to intercept this inside Emacs and parse the string with the use of org-protocol part. org-roam-protocol piggyback off this process, and then does some additional stuff based on roam-file part. Your test result for Q3 seems to tell us none of these happened, and not even a buffer opening… It’s weird.
How did you install Emacs?
I am now suspecting that you might have two different installations of Emacs. /usr/local/bin/emacsclient may reach out to another installation of Emacs, different from the one you use daily?
If you are sure that you have only one installation of Emacs, and then… Maybe the terminal command for Q3 did reach Emacs server, but failed to do anything (I wonder how this happens, though). This can be confirmed by looking at runtime debugger via edebug in some of the functions… Not sure if you are up for looking at code on runtime. If you are up for it, I can guide you there, too…
I borrowed my girlfriend’s MacBook and tried what I said should happen:
On the left is Emacs in Terminal. Started Emacs with emacs -q and started server by evaluating (server-start). Then in another Terminal window on the right, I type the command you see. You can see that the buffer name is roam-file?file=test.org (Something stripped off org-protocol:// for me).
Edit: I even misspelt “org-protocol”. But as you can see, something chops off “://“ and Emacs takes the remainder as one argument. This should happen when org-protocol is not loaded with require, (or your use-package! macro not there).
I guess… The “Waiting for Emacs” that you see is a sign it is working correctly.
One thing I noticed in this setup is that the Emacs does not come to your focus (the window does not pop up in front of other windows). Maybe your setup is working, but you need to manually focus on your Emacs?
It’s an Emacs that comes with Mac, I think. emacs -version command in Terminal tells me that it is 22.1.1.
This launch_emacs.sh script works too.
Note that I have an empty line after $1; don’t think it does anything.
No bash env thing. Don’t know if it is necessary.
I also needed to chmod for the file, but that’s probably not your problem (because if there is a permission problem with the script file itself, Terminal should tell you that).
emacsclient --no-wait $1
Nobiot, you are keeping me in the game! Thank you.
Given your concern about my Emacs installation (well placed, I’m sure), I
– uninstalled emacs,
– scoured my system for any artifacts and deleted them,
– reinstalled (emacs-plus@27),
– cloned and reinstalled Doom Emacs,
– copied my org-roam settings (mostly stolen from the .doom.d files of an org-roam author) from my old config.el/init.el Doom files,
– ran “doom sync,” ran “doom doctor” and installed a couple dependencies “doom doctor” said I needed,
– made sure emacsclient and launch_emacs.sh are still in the locations they occupied before, and
– tried the three tests you outline above.
– Sadly, I have the same results. I pass Q2 test but fail Q1 and Q2 test.
I’m very willing to debug if you’re willing to see what true ignorance looks like. I’d have to have help going from zero to debugging.
Hi @Sub,
Thanks for all the detail.
I now tried emacs-plus@27 and Doom on my girfriend’s MacBook (!).
– installed (emacs-plus@27)
– checked that Emacs version that is open is 27.1 (not 22.1.1)
– cloned and installed Doom Emacs
– No additional configuration added (I didn’t even open .doom.d and .emacs.d)
The script just works.
As it was yesterday, and I changed it to this, adding the rest, and it works too (you see my nano command used in the first screen shot).
#!/usr/bin/env bash
/usr/local/bin/emacsclient --no-wait $1
The only notable difference between yours and mine is the config (I’m disregarding MacOS’s version at the moment).
Before trying debugging, would you like to remove your config (just set it aside somewhere so that you can put it back in), and try the tests?
If they work like my tests on MacBook, then there is something in your config that needs to be changed.
org-protocol seems to be working by default with Doom with no extra configuration (but not org-roam-protocol)
Once the script is working, the rest seems to be smooth sailing. I also downloaded Platypus via brew, followed Org-roam documentation, and created a desktop app (you can see the icon on the desktop on the top right corner). In Safari’s address bar, typed “org-protocol://roam-file?file=test.org” (without the double quotations), the same argument for the script we called within Terminal. There are a couple of confirmation dialog, etc. but in the end, you get the same result in Emacs (on the left).
So… I think it will work once the script works for org-protocol.
By the way, when you call your script (you said it’s in your home directory), do you also add ./ in the beginning? I think it is necessary to explicitly specify that it is in the current working folder (home). See my example images; they all have ./ as in ./launch_emacs.sh.
nobiot, I’m pleased to report some progress as a result of your last suggestions.
I tucked away all of my Doom Emacs private configuration and used what comes with the clone/install. As a result I can pass all three of your tests. As you say there were a couple of dialog boxes to check, (including having to let emacs create an empty “org-protocol/” folder in “~/Applications/OrgProtocol.app/Contents/Resources/” which seemed odd to me, but I’m not complaining), but then all went as planned, at least for the 3 tests.
I still am not able to display the graph in a browser. The interface shows up (now without fill colors for the buttons), but no graph. That was the case while still using default Doom Emacs config files, which didn’t surprise me. I slowly layered in some of my old config file settings, each time checking to see if I was passing the tests, which with a few edits here and there, I continued to do. Eventually, I got had all of my old org-roam configuration settings, mostly lifted from the org-roam author’s .doom.d files (mostly from his config.el here https://github.com/jethrokuan/dots/blob/master/.doom.d/config.el). I also have the org-roam and related packages set up as he does in my packages.el file.
In case it helps in getting from here to an interactive org-roam graph, here are what I think are the relevant settings in my current .doom.d/config.el:
(use-package! org-roam
:commands (org-roam-insert org-roam-find-file org-roam-switch-to-buffer org-roam)
:hook
(after-init . org-roam-mode)
:init
(map! :leader
:prefix “n”
:desc “org-roam” “l” #'org-roam
:desc “org-roam-insert” “i” #'org-roam-insert
:desc “org-roam-switch-to-buffer” “b” #'org-roam-switch-to-buffer
:desc “org-roam-find-file” “f” #'org-roam-find-file
:desc “org-roam-show-graph” “g” #'org-roam-show-graph
:desc “org-roam-insert” “i” #'org-roam-insert
:desc “org-roam-capture” “c” #'org-roam-capture)
(setq org-roam-directory (file-truename “~/Dropbox/org/120notes/”)
org-roam-db-gc-threshold most-positive-fixnum
;; org-roam-graph-exclude-matcher “private”
;; org-roam-tag-sources '(prop last-directory)
org-id-link-to-org-use-id t)
:config
(setq org-roam-capture-templates
'((“d” “default” entry (function org-roam–capture-get-point)
“%?”
:file-name “{slug}"
:head "#+title: {title}\n”
:unnarrowed t)))
(set-company-backend! 'org-mode '(company-capf)))
(use-package! org-roam-server)
(use-package! org-roam-protocol
:after org-protocol)
It feels good to have gotten this far.
Hi @Sub
Do you know what was different?
In any case, two observations:
1. Org-roam-directory variable:
Just for a style, I’d change this part to (no change to what the code does, but this way, it’s easier to remove/add one variable at a time):
(setq org-roam-directory (file-truename “~/Dropbox/org/120notes/”))
(setq org-roam-db-gc-threshold most-positive-fixnum)
(setq org-id-link-to-org-use-id t)
And… This (setq org-roam-directory (file-truename “~/Dropbox/org/120notes/”)) looks odd to me.
Is your Org-roam itself working?
I have a feeling you would need to remove (file-truename) – Jethro might have it, but he knows what he is doing
(setq org-roam-directory “~/Dropbox/org/120notes/”) – I don’t have the trailing slash, so it would be “~/Dropbox/org/120notes” on my machine, but this might not affect anything. I’d suggest try both if one does not work.
Does this change anything?
If not, also try upgrading org-roam and org-roam-server packages (I assume you know how this is done with Doom).
2. Org-roam protocol:
(use-package! org-roam-protocol :after org-protocol)
Now you have this one in your config, Q3 path/to/emacslient "org-protocol://roam-file?file=path/to/your/note.org" should try to open a file in your org-roam-directory if you specify an exisitnig note in this part: file=path/to/your/note.org.
You should also be able to type “org-protocol://roam-file?file=path/to/your/note.org” (without quotations) in your browser, and get the same result (like my girlfriend’s MacBook above).
In addition…
You start the server via M-x org-roam-server-mode RET manually, right?
For org-roam-server, the installation part of its README has more variables than just a one-liner… Not sure if that is relevant in the current version.
Translating to the Doom’s use-package! macro, it would be something like this (I would have setq for each variable but that is more or less personal style, I guess):
Why not try this, if the one-liner (use-package! org-roam-server) still does not work…
(use-package! org-roam-server
:config
(setq org-roam-server-host "127.0.0.1"
org-roam-server-port 8080
org-roam-server-authenticate nil
org-roam-server-export-inline-images t
org-roam-server-serve-files nil
org-roam-server-served-file-extensions '("pdf" "mp4" "ogv")
org-roam-server-network-poll t
org-roam-server-network-arrows nil
org-roam-server-network-label-truncate t
org-roam-server-network-label-truncate-length 60
org-roam-server-network-label-wrap-length 20))
Houston, we have a clickable graph in a browser!
I added the additional org-roam config you pointed to, nobiot. I’d had it in at one point, but that was before I reinstalled everything as you also suggested. I think those two things together did the trick.
Many thanks for seeing me through this. It’s great to be working with my notes this way.
My database appears to be somewhat “unstable” because some backlinks are not displaying in the graph. The node is there, but it’s not connected to what it’s supposed to be connected to. And when I click on it, it opens a new note, not the existing note. But this is kind of a good thing, because I’m able to clean up some things about the notes that I need to clean up anyhow. In any case, that would be another topic. This one is closed!
Thanks again!
Dear Sub,
Did you rebuild your data base cache ?
We need to run [org-roam-db-build-cache] after we make new note in our Roam folder.
I asked this short-cut for this User Forum, but it is not messy using [ M-x org-roam-db-build-cache ] in Emacs’s mini buffer, some times.
Bonne chance !! Thx and regards, WAKAMATSU