Citar/org roam bibtex/bibtex and pdf associations

tim_uni · January 11, 2022, 3:29pm

Dear all, I am using doom emacs and recently the biblio module was added there. It is using citar instead of helm-bibtex and now I can not figure out how to associate pdfs with bibtex entries. Usually I use doi-insert-bibtex to add new references. Then I download the pdf and associate it with the bibtex entry (unfortunately, I don’t remember the command, it is no longer available since the new biblio module no longer uses helm-bibtex I think. Then I imported the bibtex entries into org roam using a special capture template (this still works). Now the pdf field in my org files is empty for new bibtex entries and I have no idea how to restore the pdf → bibtex entry mapping (tbh I have no clue how it worked before, I was just very happy it did). Any ideas on how to restore this functionality? Thanks a lot!

mshevchuk · January 11, 2022, 4:10pm

Take a look at your older BibTeX entries you created with doi-insert-bibtex and doi-utils-get-bibtex-entry-pdf (I assume you used the latter to download PDFs). Do they contain the BibTeX field file? It should hold the path to your PDF associated with the BibTeX entry. If yes, then you have at least two options that readily come into my mind:

install back the doi-utils library, which is a part of Org-ref not Helm-bibtex. So basically install Org-ref. You don’t need to use Org-ref citations or whatever but you need the doi-utils library. You should now be able to proceed with your usual workflow.
download a PDF manually. Open your BibTeX file and find the BibTeX entry in question. Add the file field and set it to the path of the downloaded PDF. You may want to put the PDF in the same directory you used to keep your PDFs earlier, e.g. check the older entries and note the paths stored in the file field, but that’s up to you:

@article{someEntry,
	author = ...,
	date = ...,
	doi = ...,
	file = {/path/to/your/file.pdf},
	journaltitle = ...,
    pages = ...,

If no, well, you used a different mechanism. Helm-bibtex indeed supports different mechanisms, you should figure which one.

Then I imported the bibtex entries into org roam using a special capture template (this still works).

Yes. This is the only part Org-roam-bibtex is responsible for.

Now the pdf field in my org files is empty for new bibtex entries

Yes. Because Org-roam-bibtex cannot retrieve this information because it is likely absent from new entries.

bruce · January 11, 2022, 7:39pm

The README includes a section on this; let me know if something is unclear or missing.

tim_uni · January 12, 2022, 8:01am

Thank you both for getting back to me so quickly! This is probably more a doom config specific question so thanks for still taking your time and giving me hints!

@mshevchuk: my bibtex entries do not contain a file field and never did, I still wonder how/where the mapping was stored before… I tried installing back the doi-utils library by including org-ref in my packages.el, this brought back all the functions I used earlier (doi-utils-get-bibtex-entry-pdf and especially org-ref-bibtex-assoc-pdf-with-entry), so thanks for the hint! Now I am only looking for a way to create another embark-action in citar to bind those two functions, but I read something about this in the documentation. The doi-utils-get-bibtex-entry-pdf does set the file field now, I still have no idea how this mapping was created before but probably something similar to the citar KEY.FILEENDING just implicit without the corresponding bibtex entry… Sometimes it is hard to keep track of which modules/functions and how they are used in doom.

@bruce: The README is very concise (in a good way!) and does include a section about file associations which I read, I was just struggling with “how to download/move/rename pdfs automatically so citar does find them” but thanks to @mshevchuk I consider this solved.

Thanks again! Should I mark this question as solved somehow?

mshevchuk · January 12, 2022, 8:33am

No, it is not. The respective Doom modules do little tweaking in this specific case apart from just loading the packages. It’s more about interplay between different packages, so your question is of general interest.

You are right. So by looking at the source code of org-ref-bibtex-assoc-pdf-with-entry, what this function does is it lets you select a downloaded file in the prompt, reads the BibTeX entry at point to get the citation key, renames the selected file to CITEKEY.ext, and copies/moves it to the location specified in bibtex-completion-library-path. Then whenever you want to open the file with Org-ref or Helm-bibtex, the function bibtex-completion-find-pdf-in-library just looks for the file CITEKEY.ext in that location.

So with respect to ORB and Org-roam template expansion, if you want to keep your old workflow you again have two options:

Keep using org-ref-bibtex-assoc-pdf-with-entry
If you want to uninstall Org-ref, you’ll need to manually rename new PDFs and move them to bibtex-completion-library-path.

ORB actually uses bibtex-completion-find-pdf under the hood, so if you have a file CITEKEY.ext in bibtex-completion-library-path it will eagerly pick it up during a template expansion no matter how this file ended up in that location.

tim_uni · January 12, 2022, 10:20am

You are right, and the interplay of different packages is sometimes hard to grasp (at least for me).

I decided to stick to option 1., since automating this process was my goal in the beginning.

Unfortunately, I now face another Problem: my old notes are no longer recognized (all entries are still displayed as having notes indicated by the icon to the left) and whenever I want to open the notes of a bibtex entry (citar-open-notes), orb (I assume) is prompting for capture template selection, even if I created that note file previously (with the new system), any ideas what could be going wrong here? Thanks!

Edit: I also noticed the created note org files no longer contain a ROAM_REFS entry in the property drawer, which is likely the cause of this issue I assume.

Edit2: while going through the called functions, I noticed that orb-edit-note (which is called because I set (setq citar-open-note-function 'orb-citar-edit-note) first tries to call org-roam-find-ref which I only found as org-roam-ref-find, but the org-roam-node-find function should be able to fetch the correct node.

mshevchuk · January 12, 2022, 11:34am

Currently, ORB does not provide support for note indicator in Citar. This will be added later. Normally, you shouldn’t see a note indicator. Maybe Citar is actually not configured to use ORB?

Edit: I also noticed the created note org files no longer contain a ROAM_REFS entry in the property drawer, which is likely the cause of this issue I assume.

This is an indication that either Citar is not calling ORB or that you do not have org-roam-bibtex-mode running.

while going through the called functions, I noticed that orb-edit-note (which is called because I set (setq citar-open-note-function 'orb-citar-edit-note) first tries to call org-roam-find-ref which I only found as org-roam-ref-find , but the org-roam-node-find function should be able to fetch the correct node.

orb-edit-note does not call org-roam-ref-find anymore. It’s just an outdated docstring, I’ll fix it now.

any ideas what could be going wrong here?

Not sure what’s wrong. Do you have org-roam-bibtex-mode turned on? Or maybe it is some loading issues? Try restarting Emacs or reinstalling the packages… Also, you do not need to set citar-open-note-function, just activate org-roam-bibtex-mode.

tim_uni · January 12, 2022, 11:58am

Maybe Citar is actually not configured to use ORB?

How would I make sure that citar is configured to use orb? I have (setq citar-open-note-function 'orb-citar-edit-note) in my config but nothing else (to my knowledge) connects citar and orb.

This is an indication that either Citar is not calling ORB or that you do not have org-roam-bibtex-mode running.

I did not have it running :), forgot to comment it back in since it caused issues with an earlier configuration state.

With that fixed, it is now working (mostly), thanks a lot! The only issue left is: when I am in an org roam file and try to open the notes of a reference I am still prompted with the node creation template selection dialog, even if org-roam-bibtex-mode is running, it does work in regular org files and everywhere else… So I consider this fixed, good enough for me

Edit: org-roam-db-sync did the trick for the last part…

bruce · January 12, 2022, 12:17pm

We’ve discussed a bit possibly adding a related command:

github.com/emacs-citar/citar

Add citar-add-library-file

opened 08:14PM - 11 Jan 22 UTC

closed 05:26PM - 15 May 22 UTC

bdarcus

I added a modified, more general, version of this to the wiki. I may add it back… to citar; I just wasn't sure if anyone used this ;-) https://github.com/bdarcus/citar/wiki/Example-functions#add-file-to-library There's still a little bug in it that I'll fix. It uses `completing-read` rather than `completing-read-multiple`, and works on any file. Note: recent changes in many of these commands are designed to make them compatible with `embark-act-all`. I'm not sure the UI on the current version of this command actually is, seems it seems to prompt you to rename the file in one case. Worth testing this. It also occurs to me that we might want to include an option to add from a DOI, which maybe does that without prompting if the entry has a DOI? _Originally posted by @bdarcus in https://github.com/bdarcus/citar/discussions/522#discussioncomment-1924933_

Not following this closely, but the citar note function will add the ROAM_REFS property if org-roam is present. Not sure what’s going on, but orb isn’t required for this.

github.com

emacs-citar/citar/blob/b24b5b94b7d5f6e7e96331b48a22e977e0013de1/citar-org.el#L270


      
          
          With optional argument FORCE, force the creation of a new ID."
            (let ((point (point-marker)))
              (set-marker-insertion-type point t)
              (unwind-protect
                  (org-id-get-create force)
                (goto-char point)
                (set-marker point nil))))
          
          ;;;###autoload
          (defun citar-org-roam-make-preamble (key)
            "Add a preamble to org-roam note, with KEY."
            (when (and (derived-mode-p 'org-mode)
                       (fboundp 'org-roam-buffer-p)
                       (org-roam-buffer-p))
              (ignore-errors (citar-org-id-get-create))
              (ignore-errors (org-roam-ref-add (concat "@" key)))))
          
          ;;;###autoload
          (defun citar-org-format-note-default (key entry filepath)
            "Format a note FILEPATH from KEY and ENTRY."

mshevchuk · January 12, 2022, 12:45pm

That’s about it.

That unfortunately is very vague a description for me. And an almost impossible one since Org-roam files are just regular Org files. What exactly do you do? Are these the Org-cite [cite:@citekey] references, which you are opening with embark-act or something else?

tim_uni · January 12, 2022, 1:23pm

Sorry, let’s try this: I am editing an arbitrary org roam file and calling citar-open, select a reference and then SPC-; n to open its notes. Then I am prompted for a node creation template every time I do this. This works (I am not prompted for a node creation template but taken to the correct org roam file) in any other buffer and after updating the roam database. I don’t know why those two things influence each other, if you need more information from my side, please let me know.

mshevchuk · January 12, 2022, 2:05pm

Can you try M-x citar-open-notes? It should be basically the same in effect just omitting the SPC-; n step.

mshevchuk · January 12, 2022, 2:09pm

This is also strange. Do you have org-roam-db-auto-sync-mode enabled?

tim_uni · January 12, 2022, 2:19pm

I do have org-roam-db-auto-sync-mode enabled, and without going through the embark act part the result is still the same (I like having a single point of entry for all library related stuff, thats why I use it this way).

bruce · January 12, 2022, 2:27pm

FWIW, we recently expanded citar-open to include notes as well. I don’t think I’ve yet updated the doom biblio module to include that though. But I’ll likely do that soon-ish.

mshevchuk · January 12, 2022, 2:39pm

It seems Doom’s biblio module is pinned to a fairly old Citar commit. Skimming through commit headlines I didn’t find anything that could relate to your issue, but who knows. Try (unpin! citar) in your packages.el.

bruce · January 12, 2022, 2:55pm

Actually, I just went to bump the biblio module, and realized henrik already bumped tools yesterday.

tim_uni · January 13, 2022, 8:45am

I upgraded doom emacs but the problem persists. But this is good enough for me

mshevchuk · January 13, 2022, 9:33am

Well, I won’t insist on fixing it then . It’s just itching me I can’t reproduce any of your issues .

tim_uni · January 13, 2022, 10:34am

I updated doom emacs today (29536a22), using emacs master branch with native compilation (77c3d41adc), here is my config and init, maybe that helps

Topic		Replies	Views
Automated workflow for pulling PDFs into org-ref + org-roam-bibtex Requests	1	1872	August 9, 2020
Minimal setup for helm-bibtex, org-roam v2, org-roam-bibtex Requests	7	2833	November 26, 2022
Guide: Bibliography system with org-roam-bibtex and org-noter integration Guides	13	1606	February 22, 2024
Opening notes associated to .pdf with org-roam-bibtex Requests	6	1322	September 8, 2022
Org-roam-bibtex for Org-Roam v2 Development	113	8058	February 4, 2022

Citar/org roam bibtex/bibtex and pdf associations

Related Topics