Problem with org-roam-protocol

Hello, this is my first post, and I am a fairly new user of OrgRoam, which I REALLY like :slight_smile:

I have set up org-roam-protocol following the manual (Org-roam User Manual), but for some reason it is not working for me, and I need some help, please.

Details:

  • I use Mac OS Sonoma 14.4.1 (23E224)
  • I have GNU Emacs 29.3 (build 1, aarch64-apple-darwin21.6.0, NS appkit-2113.60 Version 12.6.6 (Build 21G646)) of 2024-03-24 - installed with homebrew
  • I started emacs from command line using emacs --daemon

Following the manual, I tested my installation using
open org-protocol://roam-ref\?template=r\&ref=test\&title=this
When I do this a new emacs window starts, but it just shows the standard “Welcome to GNU Emacs” page but no capture template. If I already have an emacs window open, then it becomes active, but again no capture template.

I have also tried with the bookmarklet from Firefox, and it has the same effect: a new emacs windows opens showing the standard “Welcome to GNU Emacs” page but no capture template. Also in this case, if I already have an emacs window open, then it becomes active, but again no capture template.

I have the following in my .emacs

(require 'org-roam-protocol)
(use-package org-roam
  :ensure t
  :init
  :custom
  (org-roam-directory (file-truename "~/my_org_roam_directory"))
  (org-roam-dailies-directory "journal/")
  (org-roam-completion-everywhere t)
  (org-roam-capture-templates
   '(
     ("d" "default" plain "%?"
      :target (file+head "%<%Y%m%d%H%M%S>-${slug}.org" "#+title: ${title}\n")
      :unnarrowed t)
     ;; several more tmeplates defined here     
     )
   )
  (org-roam-capture-ref-templates
   '(
     ("r" "ref" plain "%?" :target
      (file+head "private/from_web/%<%Y%m%d%H%M%S>-${slug}.org" "#+title: ${title}\n#+date: %U\n\n${body}")
      :unnarrowed t)
     )
   )
  (org-roam-dailies-capture-templates
   '(
     ("d" "default" entry "* %?"
      :if-new (file+datetree "%<%Y>.org" year))      
     )
   )  
  :bind (("C-c n l" . org-roam-buffer-toggle)
         ("C-c n f" . org-roam-node-find)
         ("C-c n i" . org-roam-node-insert))
  :config
  (org-roam-setup)
  (org-roam-db-autosync-mode))

Any help is more than welcome! :slight_smile:

1 Like

Hi I am a new user too – so we have a situation of blind leading the blind – I did not use to use org-protocol for capturing through the ref protocol but I tested it by following the implicit guide in your code

It is working for me – can you try seeing for debug that the org-protocol://roam-node?node=node-id is atleast working before we try to debug the ref protocol more specifically ?

Tangentially – in your setup

(org-roam-setup)
(org-roam-db-autosync-mode))

is doing the same thing - the first one is obsolete I suppose. Use anyone since running both is a waste on your cpu cycle probably.

Hi @akashp, and thank you for your reply.

I have tried open org-protocol://roam-node\?node=node-id replacing node-id with the id of a node I had already created (I hope this was the intended usage): nothing happens: emacs does not even starts (and if it is open then it does not react to the command).

Thank you for pointing out the useless lines in my configuration. I will remove org-roam-setup.

Which points to us that the org-protocol is probably failing.

To set up org-protocol we need configuration in a 3-tier system.

First is your OS - it needs to be notified of the protocol - in Linux we do this by setting up a mimetype
The second is emacs itself – the configuration of which seems alright to me in first glance

The third is for the medium from which we would interact - in normal circumstances this could be the browser –

since we are using open from the terminal - we can keep the third step out of our debug process for now.

Can you confirm that the OS in informed of the protocol through mimetype ?

also as to the intended format of the node protocol it is this

org-protocol://roam-node?node=9a86a3fc-89c5-456c-afaa-b19ba70e295d
etc..

OK, here is some update:

  1. I did not know that I had to use the node id in the open command that you suggested. I have checked the node id (a33e3f37-7bea-4780-b62e-c6683f984d46) of an existing Org Roam file on my system, and I have tried again with open org-protocol://roam-node\?node=a33e3f37-7bea-4780-b62e-c6683f984d46: this opens emacs, but does not open the org file, it just starts emacs displaying the standard “Welcome to GNU Emacs” page.

  2. I do not have the mimetype command on Mac OS, but following this suggestion macos - Where is the location of the mime plugin files in mac and linux? - Stack Overflow I did file --mime /path/to/a/org_roam_file.org and the result is /path/to/a/org_roam_file.org text/plain; charset=us-ascii

Does this shows if OS is informed of the protocol?

Hi I will admit - I have zero idea about running it in Apple Mac

The process seems a bit involved to start using org-protocol in Mac

https://orgmode.org/worg/org-contrib/org-protocol.html#orgd331b5e

Can we debug on your linux system if possible?

Org Mode uses the emacsclient command-line program for capturing data as discussed on the org-protocol.el webpage. For Org Mode capture to work correctly, you must integrate the emacsclient executable with Mac OS X’s URL dispatch mechanism. This is a non-trivial task.

From : GitHub - neil-smithline-elisp/EmacsClient.app: Use emacsclient to load URLs.

Please undo this however its done.
It is probably not doing what you want.

I have tried to follow the instructions here GitHub - neil-smithline-elisp/EmacsClient.app: Use emacsclient to load URLs., but it does not work. In fact there is an open issue (#3) which makes this not-usable any more :frowning:

I have also found this other set of instructions GitHub - xuchunyang/setup-org-protocol-on-mac: Setting Up org-protocol on Mac, and tried that too, but did not work for me :frowning:

I guess the instructions reported in the org-roam manual (section 13.1.1, sub-section Mac OS) are the most updated. I have tried again with these instructions, and although it is not working yet, I have noticed that when I use the bookmarklet from Firefox, then Emacs open (it just displays the standard Welcome page), but it also create a file in the directory specified by org-roam-capture-ref-template, but that file is a broken symbolic link.

I am stuck at the moment, and would really appreciate some help.

How do you know it’s a “broken symbolic link”?
How do you know “it also create a file in the directory specified by org-roam-capture-ref-template”?

You are very close. The whole flow is technically “working” if some data is actually written to file – it’s just not receiving a correct data. The only remaining thing to do is identify what in this flow causes this incorrect data.

To find it out, it would be typically useful if you inspect (and share) the following:

  1. The name of the file created and symoblic ink
  2. The content of the “broken symobilc link” if it points to a file (how do you know a file is created if the symbolic link is broken?)
  3. The exact JavaScript string that you put in Firefox Bookmarklet.
    You may say it’s identical with what’s in the instruction; I suggest you verify it carefully, and perhaps paste it here, too.

Alternatively, and, if you know how, the quickest is probably to use edebug on the function org-roam-protocol-open-ref to see how the program is actually called with what data is passed to it.

1 Like

In addition to what I suggested above, where are you sending the body?

If you aren’t, it is possible that the template errors because of missing body data (you may see an error message in the message buffer in the emacs frame that opens). I suggest you remove this part of the template.

Hi @nobiot, and thank you for your messages. I decided to delete bookmarklet and app that I created and to redo everything from scratch documenting all the steps. I apologise in advance for the long post, but I documented everything I did (sorry I do not know how to use edebug. In the steps below you also find the answers to your questions. I hope this helps in giving you a clue why configuration/set-up is not working. Thank you!

  1. From command line I run emacs —daemon
  2. From command line I do which emacsclient, and get /opt/homebrew/bin/emacsclient
  3. I launch the Script Editor and paste in the following script:
on open location this_URL
    set EC to "/opt/homebrew/bin/emacsclient --no-wait "
    set filePath to quoted form of this_URL
    do shell script EC & filePath
    tell application "Emacs" to activate
end open location
  1. I save that script as /Applications/OrgProtocolClient.app … I notice that saving this causes emacs to start up with the default Welcome text.
  2. I add the following XML snippet in /Applications/OrgProtocolClient.app/Contents/Info.plist, before the last tag
<key>CFBundleURLTypes</key>
<array>
  <dict>
    <key>CFBundleURLName</key>
    <string>org-protocol handler</string>
    <key>CFBundleURLSchemes</key>
    <array>
      <string>org-protocol</string>
    </array>
  </dict>
</array>
  1. I save the XML and double click on OrgProtocolClient.app to run it and register the protocol
  2. I restart my computer
  3. I run again emacs —daemon
  4. I test org-protocol with open org-protocol://roam-ref\?template=r\&ref=test\&title=this: this starts the OrgProtocolClient.app (I can see it among my running apps when I do Command+TAB, but nothing else happens … I think this is already means that something is wrong. Interestingly, at this stage if I do the following from the command line emacsclient -e '(kill-emacs)’then emacs start up with the usual Welcome page. No file is ever created in the path specified by my template inorg-roam-capture-ref-templates(see.emacs` file posted above).
  5. I restart emacs —daemon
  6. I try open org-protocol://roam-node\?node=a33e3f37-7bea-4780-b62e-c6683f984d46 (where a33e3f37-7bea-4780-b62e-c6683f984d46 is the node-id of an existing org-roam note in my computer): an emacs window with the default Welcome message appears, but my note is not loaded.
  7. I kill again emacs daemon with emacsclient -e '(kill-emacs)’, and restart it with emacs —daemon
  8. I create a bookmarklet in Firefox: Bookmarks → Manage Bookmarks, right-click on “Bookmarks Toolbar, and “Add bookmark”: Name ‘OrgRoam’, URL: ’javascript:location.href = 'org-protocol://roam-ref?template=r&ref=' + encodeURIComponent(location.href) + '&title=' + encodeURIComponent(document.title) + '&body=' + encodeURIComponent(window.getSelection()), and then I save it.
  9. I go to Org-roam User Manual and click on OrgRoam: emacs window opens with Welcome message. In the directory specified by my template in org-roam-capture-ref-templates I see the following:
ll -lart                                                                                                                            
total 0
drwxr-xr-x@ 7 marco  staff   224B 16 May 14:49 ../
lrwxr-xr-x@ 1 marco  staff    35B 20 May 09:56 .#20240520095652-org_roam_user_manual.org@ -> marco@Marcos-MacBook-Pro.local.2914
drwxr-xr-x@ 3 marco  staff    96B 20 May 09:56 ./

Also, I try to check the symbolic link with file and this is what I get:

file .#20240520095652-org_roam_user_manual.org                                                                                       
.#20240520095652-org_roam_user_manual.org: broken symbolic link to marco@Marcos-MacBook-Pro.local.2914

I also try cat:

cat .#20240520095652-org_roam_user_manual.org                                                                                       
cat: .#20240520095652-org_roam_user_manual.org: No such file or directory
  1. I do emacsclient -e '(kill-emacs)’ and then check again the directory specified by my template in org-roam-capture-ref-templates I see the following:
ll -lart                                                                                                                            
total 8
drwxr-xr-x@ 7 marco  staff   224B 16 May 14:49 ../
-rw-r--r--@ 1 marco  staff   418B 20 May 10:00 #20240520095652-org_roam_user_manual.org#
drwxr-xr-x@ 3 marco  staff    96B 20 May 10:00 ./

Now the file is not a symbolic link and there is no dot at the beginning.
I try cat and this is what I see:

cat \#20240520095652-org_roam_user_manual.org\# 
:PROPERTIES:
:ID:       6aeb62ad-6fbf-4068-b9e5-a4fbdfcfb2e9
:ROAM_REFS: https://www.orgroam.com/manual.html#Mac-OS
:END:
#+title: Org-roam User Manual
#+date: [2024-05-20 Mon 09:56]

javascript:location.href =
    'org-protocol://roam-ref?template=r&ref='
    + encodeURIComponent(location.href)
    + '&title='
    + encodeURIComponent(document.title)
    + '&body='
    + encodeURIComponent(window.getSelection())

  1. I kill emacs daemon and restart it and try removing the `${body} part of the template as suggested, so my template is now:
     ("r" "ref" plain "%?" :target
      (file+head "private/from_web/%<%Y%m%d%H%M%S>-${slug}.org" "#+title: ${title}\n#+date: %U")
      :unnarrowed t) 

I repeat step 14, and get another symbolic link .#20240520100759-org_roam_user_manual.org@ -> marco@Marcos-MacBook-Pro.local.3359.
I repeat step 15, and I get a file with the following content:

cat \#20240520100759-org_roam_user_manual.org\#                                                                                      (base)
:PROPERTIES:
:ID:       79cca87a-9ae7-488d-adc8-b92102925f9e
:ROAM_REFS: https://www.orgroam.com/manual.html#Mac-OS
:END:
#+title: Org-roam User Manual
#+date: [2024-05-20 Mon 10:07]

That is an autosave file.
Q) What about your buffer list – any open buffers you spot? Bring up the active buffer list.

When the Welcome message comes up, can you press “q” and quit it? I think you will see the buffer you want is actually created behind it.

If you still don’t see the one you expect, then as @akashp, check the buffer list and see if you see it.

@akashp @nobiot I cannot see the buffer when emacs open. If I do C-x C-b I get the following:

As you can see the only buffers are “GNU Emacs”, “scratch”, and “Messages”.

Which Emacs is this screen from?

I suspect you have multiple instances, and this screen is not the one that creates #20240520095652-org_roam_user_manual.org# when you do emacsclient -e '(kill-emacs)’

The question I have is, what creates this file?

The theory I have is as follows:

  • At step 14, a buffer is created and it creates a lock file (see this).
  • At step 15, you kill the emacsclient, which kills the emacs without prompting to save the buffer
  • However, some auto-save process creates the file with #

So the configuration you created is working.

One thing I would also verify is if this Apple Script from the user manual is correct:

on open location this_URL
    set EC to "/opt/homebrew/bin/emacsclient --no-wait "
    set filePath to quoted form of this_URL
    do shell script EC & filePath
    tell application "Emacs" to activate
end open location

Somehow, it seems to “activate” the normal Emacs, in addition to calling the shell script for emacsclient.
It may be doubling up emacsclient and emacs, and you may be seeing the latter instead of the former. (Or… tell application "Emacs" to activate brings the Emacs app to the front and selected…).

I cannot verify this because I don’t use Mac… But reading this instruction from Org manual seems to be aligned with my understanding. I’d start with getting org-protocol first…

One more thing. You should also look at what message buffer says

I finally have org-roam-protocol working :slight_smile: :slight_smile: :slight_smile:
I want to thank @akashp and @nobiot for their help and support in the past few days.
After a lot of experiments and a lot of reading, I discovered that there was nothing wrong my configuration (described above) … except that I was missing a little bit in my .emacs file:

(server-start)

Silly me! I thought that was not required when starting emacs with emacs --daemon, but I was very wrong! (best succinct explanation I found is here emacsclient - How are a daemon and a server different in Emacs? - Emacs Stack Exchange).

I would suggest to add a sentence in the Org-roam User Manual to clarify this, perhaps in section 13.1.1 Installation (Org-roam User Manual)

[[
To enable Org-roam’s protocol extensions, simply add the following to your init file:

(server-start)
(require 'org-roam-protocol)

]]

Well, I hope all these messages posted here may be useful for others trying to use org-roam-protocol. Thank you!

1 Like

Good to know :slight_smile:

But… Then, this from the Emacs manual does not make sense to me (my emphasis)

  • Run Emacs as a daemon, using one of the ‘–daemon’ command-line options. See Initial Options. When Emacs is started this way, it calls server-start after initialization and does not open an initial frame. It then waits for edit requests from clients.

Do you have any thought about this?

In fact, the Stack Exchange reply turns off the server after starting Emacs as daemon:

you can start Emacs server without daemon or daemon without Emacs server, however, if you start daemon without Emacs server, the daemon is useless, e.g.,

$ emacs --daemon
$ emacsclient --eval '(server-mode -1)'

Just replying to myself.

Adding server-start would be a friendly update to the user manual – @marcos Would you be willing to create a PR for it?

(server-start)
(require 'org-roam-protocol)

It looks like it might have worked if -nw was added (to open the daemoned Emacs session to open in the Terminal – it may require the Terminal to be already open. Can’t tell without a Mac machine).

Or -c option added to create a new frame.

Basically, daemon has been working as an Emacs server quietly in the background without a new frame to interact with the user… So the Stack Exchange response is not entirely correct. You cannot start daemon without an emacs server, because daemon always starts an emacs server. But you can turn it off, which renders daemon useless as the post says.

on open location this_URL
    set EC to "/opt/homebrew/bin/emacsclient -c --no-wait "
    set filePath to quoted form of this_URL
    do shell script EC & filePath
    tell application "Emacs" to activate
end open location

The daemoned Emacs does not seem to create a new frame automatically through the protocol. So it looks like adding server-start in the init for the usual Emacs session would be the easiest way…