What do you think about org-ref V3?

maikol · October 18, 2021, 7:13pm

Hi all!

Today, the new org-ref version 3 was released (GitHub - jkitchin/org-ref: org-mode modules for citations, cross-references, bibliographies in org-mode and useful bibtex tools to go with it.).

Besides some changes in the configurations, the most prominent is the use of & instead of @ (used by org-cite) to cite references.

Here it is a comparative of the formats

org-ref => [[cite:&my-key]]
org-cite => [cite:@my-key].

My daily work is based in org-roam and org-roam-bibtex thus I’m a little confused if I should use org-ref or change to org-cite.

For the moment, I pinned org-ref to version 2, to avoid breaking something with ORB.

What do you think about those changes?
What do you think org-roam and org-roam-bibtex should use in the long term?

Thanks.

bruce · October 18, 2021, 8:16pm

OR3 basically reimplements the (better) org-cite citation model in a new and non-standard syntax, and adds it as a preferred option alongside the earlier org-ref link citation syntax.

You can see his explanation for why he did this, and my reply, here:

github.com/jkitchin/org-ref

Transition from org-ref version 2 to org-ref version 3 - roadmap

opened 08:46PM - 25 Aug 21 UTC

closed 01:01PM - 18 Oct 21 UTC

jkitchin

I originally thought that the new citation syntax in org-mode was going to make …the citation links in org-ref obsolete. Now that I have worked my way through org-ref-cite (https://github.com/jkitchin/org-ref-cite), I no longer think that is true. It is more clear to me these two projects had different goals, and those goals are not fully compatible. It comes down to org-ref using bibtex/biblatex as the predominant citation processor, and org-cite using CSL. These two processors have different syntaxes, and I don't think it is possible to have a single approach that works for both of them without making compromises in capability. I use org-ref professionally, and I don't want to make compromises on its functionality. In org-ref, I used the link type to indicate the LaTeX export command explicitly, so it looks like what you would read about in LaTeX documentation. That is a feature IMO, so you don't have to figure out how to translate the cite style. Of course, that is only a benefit if you know LaTeX, and are aiming to generate LaTeX for scientific publications. The disadvantage of this approach is trying to define the export behavior for other backends, which has been limited in org-ref. org-cite, on the other hand, uses a generalized citation style like cite/style/variant which is the same across export backends. I found it was not easy to map these onto the full set of natbib/biblatex citation commands, and not easy to remember that mapping while writing. Admittedly, the original syntax used in org-ref provided minimal support for pre/post notes in a clunky way that used the link description. The org-cite syntax supports this better, allowing common pre/postnotes, as well as individual pre/postnotes on multiple references. It turns out that org-ref can support this syntax natively in the existing citation links, which provides a path to exporting org-ref links to other backends using citeproc to render them. Finally, org-cite and org itself do not provide adequate cross-referencing (or index/glossary functions) for scientific publishing, so it is clear that the rest of org-ref also still has a place in this ecosystem. Soooo.... I have rewritten a lot of org-ref to support all these issues. There is a new "Version 3" in https://github.com/jkitchin/org-ref/tree/org-ref-3. I really struggled to decide this was the right thing for me to do; a lot of people have invested time in org-cite over the past years. In the end I decided to do it for these reasons: 1. This is the tool I need and use professionally on a daily basis. It has to do what I need the way I need it. 2. I (and others) have *a lot* of legacy documents that use org-ref, and it is clear I have to be able to support those for a long time 3. I thought it was worth pushing the link syntax to support this. The link behavior in org 9.0 was designed to support this kind of applications. I wish I would have figured out much earlier that this could all be done this way. 4. There is room for both approaches. They have different goals and applications. 5. I learned emacs-lisp to write org-ref, and the early code shows that. I am a better programmer now, and now the code-base is better than it was. Version 3 supports the legacy link syntax, and a new link syntax that is identical to what is in org-cite. I took this opportunity to clean up ~8 years of buildup. It now only supports bibtex-completion by @tmalsburg for candidates, opening notes, pdfs, light formatting, etc. The vanilla org-ref library is completion backend agnostic, so if you don't want ivy or helm, you won't get it, but you do have to wire in the completion backend you prefer (e.g. Selectrum, ido, etc.). I have removed the ivy and helm dependencies so you won't get them if you don't want them, although you will have to install them if you want them. If you do like helm or ivy, you can load `org-ref-ivy` which depends on `ivy-bibtex` or `org-ref-helm` which depends on `helm-bibtex`. Good news on this front, these libraries will no longer mess with the order of actions in them; there are dedicated insert commands that look like those functions, but aren't them. I removed all the `org-ref-*` variables that were just shadowing `bibtex-completion-*` variables to keep things simpler. I also took out all the key-bindings, in case you have different ideas. Those have to be set in your own init files now. Apologies in advance to anyone who contributed code that is now gone. The code is still in version 2, and it can probably be added back, let me know what it is, and I can work it back in. There are still some minor issues of over-active fontification between org-ref and org-cite that have not been resolved, but they only affect the cite: link. I am going to use "Version 3" myself for a few more weeks to get it as bug-free as I can before I merge it on the master branch, at which point it will become the standard version available in MELPA. It is pretty functional at this point, and I only anticipate fixing some small corner case bugs in the coming weeks. I still need to test it more thoroughly in the new version of org-mode with org-cite enabled. The version in MELPA is still what I call "Version 2" with the old syntax everyone is used to. There is a tagged version at 2.0.0 in MELPA Stable, and the Version 2 branch can be found at https://github.com/jkitchin/org-ref/tree/org-ref-2, for anyone who wants to keep the old behavior. It is conceivable I might finally be able to split out two new packages for org-ref-ivy and org-ref-helm that do properly install their dependencies, but I am going to wait until it is ready to merge to worry about that. Until then, if you are the bleeding edge type, give Version 3 a spin, and let me know of any issues.

I think it will lead to years of needless confusion and fragmentation for users and developers.

So, for example, the OR devs will now have to figure out if they want to support three different citation syntaxes, and which to prioritize.

My strong preference is org-cite, since it’s very well-designed, is highly-modular, and is built-in.

But I’ve never really used org-ref much. If you have a ton of legacy org-ref documents, and you rely heavily on its features beyond citations (cross-references and such), then you might as well continue to use it, and maybe experiment with org-cite?

I’m personally hoping to see a more flexible and modular ecosytem of org-cite libraries develop over the next year or two.

maikol · October 18, 2021, 10:11pm

Thanks @bruce for your response.

I read the discussion in the org-ref issue tracker, and I agree with you that it will be a lot of confusion between the 3 formats.

I’m the regular Joe Doe user, and in terms of usability and stability, I prefer something consistent across the time.

At least for me, I think that I will keep using org-ref V2 for a while and try to convert all my references to org-cite (I hope your biblio module in doom gets merged soon).

Just one last question. In what cases, org-ref is superior to the built-in tools in org-mode for cross-references?

Thanks.

bruce · October 18, 2021, 10:28pm

You can play with the built-in tools yourself; for example [[tab-1]] is a cross-reference, that will export to latex, etc.

But unlike org-ref, there’s no easy insertion (for example nothing like M-x org-crossref-insert), or other functionality attached to those links.

This new package does allow one to just use the org-ref cross-reference features without loading the rest. Just wish it were distributed as a separate melpa package.

maikol · October 19, 2021, 2:14am

Thanks again @bruce. Your reply was very informative.

If I understood correctly, the only missing piece to make ORB completely functional with org-cite is to solve this issue Add org-cite support · Issue #191 · org-roam/org-roam-bibtex · GitHub, isn’t ?

bruce · October 19, 2021, 10:09am

Depends what you mean by “completely functional”.

OR supports OC now.

If you use citar you get pretty robust functionality in org, and so also in OR.

That works out-of-box with OR notes without need for ORB. Or, per that post, you can use the ORB note function instead.

I’m not really sure what is missing at this point actually. What else do you need?

maikol · October 19, 2021, 12:00pm

For example, with org-ref if I have a citation, by just pressing “ENTER” I get a prompt to select “Add notes” and it triggers the org-roam capture template for this reference

For org-cite if I press “ENTER” over the cite, I got an error in the minibuffer

Following your PR in doom, I copied this in my config:

(after! org
  (require 'oc)
  (setq org-cite-global-bibliography '("~/Dropbox/home/documents/org/roam/_library.bib"))
  (map! :map org-mode-map
        :localleader
        :desc "Insert citation" "@" #'org-cite-insert)

  (setq org-cite-export-processors
        '((latex biblatex)
          (t csl))))


(use-package! oc-bibtex-actions
  :when (featurep! :completion vertico)
  :after oc
  :demand t
  :config
  (setq org-cite-insert-processor 'oc-bibtex-actions
        org-cite-follow-processor 'oc-bibtex-actions
        org-cite-activate-processor 'oc-bibtex-actions
        ;; The activate processor relies on shift-select, so we set to t.
        org-support-shift-select t))

And now with SPC a I can trigger embark-act and get this menu

But if I select “open notes” another error appears

For the moment, I can’t figure it out how to trigger the org-roam’s template system to create a note with my citation. The idea is this new note has the property :ROAM_REFS: @Hastie1990.

For sure, I am missing something in my config, but I don’t know.

Thanks.

bruce · October 19, 2021, 12:18pm

Did you do set bibtex-actions-file-open-note-function, or leave it default?

By default, it should (does on my system) open the new or existing note; here’s what a new one looks like:

It doesn’t include :ROAM_REFS: line ATM, though I’ll figure that out (probably add a dedicate OR function).

If you want to use ORB there instead, you need to do:

(setq bibtex-actions-file-open-note-function 'orb-bibtex-actions-edit-note)

If this doesn’t help, you might post an issue on the bibtex-actions tracker?

maikol · October 19, 2021, 2:20pm

I tried bibtex-actions-file-open-note-function with the default value bibtex-actions-file-open-notes-default-org and with orb-bibtex-actions-edit-note and nothing works for me.

I will fill the issue in the ORB github.

Thanks for all the help.

bruce · October 19, 2021, 3:19pm

Sorry; I meant the bibtex-actions repo.

mshevchuk · October 19, 2021, 3:20pm

@maikol I haven’t had a chance to start using new org-cite citations yet but I just tried it and it works well with Bibtex-actions and ORB. Both opening the existing notes and creating new ones works as expected with ORB properly expanding Org-roam templates. Note that :ROAM_REFS: will still use the old Org-ref cite:link syntax though.

This error looks to me as what we discussed with @bruce some time ago when the internal data structures used in Bibtex-actions had changed their signatures. ORB has introduced the dedicated function orb-bibtex-actions-edit-note to be used with bibtex-actions-file-open-note-function to play nice with Bibtex-actions. Typically you don’t have to do anything, just enable org-roam-bibtex-mode as usual. I believe you just have to upgrade either Bibtex-actions or ORB or both to the latest state, otherwise the mismatching versions will cause this error.

@bruce I noticed that bibtex-actions-open-notes would refuse to open a note if bibtex-actions-notes-paths is set to nil throwing the following warning:

bibtex-actions-open-notes: You must set ’bibtex-actions-notes-paths’ to open notes

Conceptually, I would expect bibtex-actions-open-notes to be less strict. Since an external function for note processing such as orb-bibtex-actions-edit-note may take care of locating the note and would not necessarily rely on bibtex-actions-notes-paths. But this is a minor nuance hardly relevant here.

bruce · October 19, 2021, 3:33pm

Yes, that makes sense. I’ll take a look at it.

Thanks.

maikol · October 19, 2021, 5:11pm

Thanks, @mshevchuk. The problem was that I had an old version of bibtex-actions.

This config works and SPC a work for me,

(use-package! oc-bibtex-actions
  :when (featurep! :completion vertico)
  :after oc
  :demand t
  :config

  (setq org-cite-global-bibliography '("~/Dropbox/home/documents/org/roam/_library.bib"))
  (setq bibtex-actions-bibliography org-cite-global-bibliography)
  (setq bibtex-completion-bibliography org-cite-global-bibliography)
  (setq bibtex-actions-notes-paths '("~/Dropbox/home/documents/org/roam/literature"))
  (setq org-cite-insert-processor 'oc-bibtex-actions
        org-cite-follow-processor 'oc-bibtex-actions
        org-cite-activate-processor 'oc-bibtex-actions
        ;; The activate processor relies on shift-select, so we set to t.
        org-support-shift-select t))

For me is ok to leave it like cite:my-cite for the ROAM_REFS: as a standard way to access the citations. The users can choose whatever they like to insert them in their documents.

Just curious about the future, do you plan to add @my-cite or &my-cite formats for this property, or leave it like now?

Best.

bruce · October 19, 2021, 6:34pm

Org itself doesn’t allow citations in the property drawer (I asked).

Edit: I just merged a commit to fix the too aggressive error, and also better support org-roam with the default function.

maikol · October 19, 2021, 8:06pm

Awesome! Thanks for all.

mshevchuk · October 20, 2021, 8:52am

This is exactly how I view it.

As far as I know, Org-roam recognizes Org-cite’s @my-cite citations as a :ROAM_REFS: keys and treats them identically to Org-ref’s cite:links. If you have a note with such a key, it should be indexed into the database and would pop up in the backlinks section of the Org-roam buffer. It may also partially work with ORB in that you would be able to open such a note from Helm-bibtex, Bibtex-actions or Org-ref but I haven’t checked it. What definitely does not work is creating a new note with ORB - you’ll always get an Org-ref cite:link in the properties drawer.

So the Github issue you mentioned earlier is about bringing a user option to allow them to choose which format they prefer.

I however have little incentive currently to address this due to a lack of time and because there is no added value to Org-cite @citations in :ROAM_REFS. As @bruce has already mentioned, this type of structures in property drawers is not recognized by Org and is simply a non-functional piece of plain text. Unlike with Org-ref cite:links, you won’t be able to interact with Org-cite @citations in property drawers, e.g. open a PDF file or the BibTeX entry, as some users find it useful with cite:links.

The issue is there though so I’ll address it sooner or later but it’s not on the top of the priority list.

bruce · October 20, 2021, 12:16pm

I agree; it’s really not a significant practical problem.

If a user wants access to additional actions from an org citation, they can just include the citation(s) in the body of the note, which is what they should be doing anyway for serious academic writing (Ahrens talks about this in his discussion of the topic, and I emphasize it to my students).

mshevchuk · October 20, 2021, 12:37pm

Right, also the ORB’s orb-note-actions functionality was actually designed for such additional actions based on the citation key. It acts on the note level so the point does not have to be on the :ROAM_REFS: citation key. It’s also easily extensible.

maikol · October 20, 2021, 12:51pm

Thanks @bruce and @mshevchuk for this enlightening conversation.

I agree with you two, and I think the way org-roam, org-roam-bibtex and org-cite interact is stable enough to make any changes.

As a side note, today I finally could adjust org-cite and bibtex-actions to behave similar to the old org-ref. I’ll keep using org-ref V3 only for the cross-references. Best of the two worlds.

Best.

prashantnag93 · October 20, 2021, 1:03pm

@maikol can I get your .doom.d configuration so that I can also setup the same in my doom emacs.

Topic		Replies	Views
Request: bibtex / org-roam-bibtex for morons Requests	3	661	July 14, 2023
An idea for a roam-based bibliographic manager Development	11	1004	September 3, 2021
All about the upcoming Org-cite feature Development	28	2774	November 22, 2021
What are cite backlinks How To	7	2092	August 4, 2020
Org-roam-bibtex for Org-Roam v2 Development	113	8064	February 4, 2022

What do you think about org-ref V3?

Related topics