# An idea for a roam-based bibliographic manager **URL:** \ **Category:** Development\ **Created:** [August 31, 2021, 11:32pm UTC](https://org-roam.discourse.group/t/an-idea-for-a-roam-based-bibliographic-manager/1997 "2021-08-31T23:32:17Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![jonathan](https://avatars.discourse-cdn.com/v4/letter/j/aeb1de/32.png) [@jonathan](https://org-roam.discourse.group/u/jonathan)\ **Post date:** [August 31, 2021, 11:32pm UTC](https://org-roam.discourse.group/t/an-idea-for-a-roam-based-bibliographic-manager/1997/1 "2021-08-31T23:32:18Z")
I can’t stop thinking about using org-roam for full-stack bibliographic management, so I thought I’d share it. Partly because I don’t think I have all the skills necessary to do this myself, and I’m hoping someone would want to pick up this idea, and/or collaborate with me on it. The way I see it, maintaining org-roam bibliographic notes can be a bit kludgey at times, especially if you’re like me, and use org-roam, org-roam-bibtex, org-ref, helm-bibtex, bibtex-completion, and more. Here are some problems with that stack, as I see them: ## Problems 1. That’s a big software stack, with lots of moving parts to keep track of. 2. Helm-bibtex and its siblings need to parse all your bibtex files each time you try to insert a citation. That can get pretty time-intensive, if you have a large bibtex file, like I do. 3. Org-roam notes reference citations (`cite:stanley2021`), which point to entries in bibtex files (`@article{stanley2021...`), which are what ultimately contain the bibliographic metadata. But this means that you’re maintaining notes for bibliographic entries in two places, ultimately. This can get messy. 4. Org-ref is slowly being replaced with org-cite, the built-in citation mechanism for Org-mode. ## Proposed Solution What if, instead of trying to Frankenstein together a bibliographic manager, by combining org-roam, org-ref, org-roam-bibtex, helm-bibtex, and others, what if you could just use org-roam as your bibliographic manager? That way, not only would you simplify your software stack, but you could simplify your notes, too, by avoiding BibTeX altogether. The pieces are all almost there. [Org-bibtex](http://gewhere.github.io/org-bibtex), which is already included in Org, provides templates for org-based bibliographic metadata storage, using `PROPERTIES` drawers. Here’s an example: ```org-mode * A multi-language computing environment for literate programming and reproducible research :babel: :PROPERTIES: :BTYPE: article :AUTHOR: Eric Schulte and Dan Davidson and Tom Dye and Carsten Dominik :JOURNAL: Journal of Statistical Software :VOLUME: 46 :NUMBER: 3 :YEAR: 2012 :MONTH: January :CUSTOM_ID: schulte2012babel :END: Some annotation about babel. ``` This isn’t too far off from what an org-roam node looks like, as generated by org-roam-bibtex. If this were your canonical format for bibliographic metadata, it would allow you to do some sophisticated org-ql queries for finding, say, all entries from 2021. It would also allow you to leverage hierarchical org-roam nodes to represent articles in edited collections–the collection itself could have the parent heading, and the collection’s articles could be subheadings. ## What would be needed What would be needed for something like this is, I imagine: - Importer functions that could take ISBNs or DOIs, or even plain-text queries, and convert them to org-roam nodes, containing bibliographic metadata gleaned from some REST API. It could just be a light rewrite of [org-ref’s isbn-to-bibtex](https://github.com/jkitchin/org-ref/blob/3ca9beb744621f007d932deb8a4197467012c23a/org-ref-isbn.el#L131) function, and other functions like that. - Functions to cache bibliographic metadata in org-roam’s database, whenever it scans for changes. - Functions for helm, ivy, or whatever other system, to select bibliographic items from org-roam nodes, based on that cached metadata, like titles, authors, years, and so on. - A backend for org-cite’s citeproc, that can look up bibliographic data from the org-roam database, rather than looking for it in a bibtex file. That way, when exporting an org file containing citations, it can generate the Works Cited page automagically, using org-roam data. Let me know what you think, and whether you think this might deserve further thought and effort. ---
**Author:** ![bruce](https://avatars.discourse-cdn.com/v4/letter/b/ccd318/32.png) [@bruce](https://org-roam.discourse.group/u/bruce)\ **Post date:** [September 1, 2021, 11:55am UTC](https://org-roam.discourse.group/t/an-idea-for-a-roam-based-bibliographic-manager/1997/2 "2021-09-01T11:55:53Z")
Org-roam v2 and org–cite both reflect a move towards more focused, and modular, tools. Org-cite, in particular, is incredibly modular. I think the future is likely best served with that direction: equally focused and modular tools that integrate these pieces well. The recent addition of org-cite indexing in org-roam, which includes a new `citations` db table, seems to provide a good foundation for just this. I maintain bibtex-actions, and most of its code is independent of org. And none of it is specific to org-roam; not even the “open notes” embark menu items you see below. But there are a few key places that integrate it with org-cite, so you end up with things like this within org/org-roam. ![image](https://global.discourse-cdn.com/free1/uploads/orgroam/original/1X/06f390fdeb4aa15258af6a179d8dd3178ce6b020.png) Or here’s embark at-point integration menu. ![image](https://global.discourse-cdn.com/free1/uploads/orgroam/original/1X/ceee2f283f5aa76b6df4b31479b51ef1a11c9d8c.png) So I think there’s a lot of opportunity for doing cool stuff with org-roam v2 and org-cite. BTW, this issue seems related to your point on org-bibtex. > > > It should also be possible to add bibliographic data inline, i.e. without a sepe…rate bibfile. This is useful for shorter documents. With pandoc, bibliographic data can be added in a special metadate field, see \. Something similar should be possible with citeproc-el/citeproc-org. ---
**Author:** ![danderzei](https://yyz2.discourse-cdn.com/free1/user_avatar/org-roam.discourse.group/danderzei/32/59_2.png) [@danderzei](https://org-roam.discourse.group/u/danderzei)\ **Post date:** [September 1, 2021, 9:32pm UTC](https://org-roam.discourse.group/t/an-idea-for-a-roam-based-bibliographic-manager/1997/3 "2021-09-01T21:32:20Z")
How would an Org-based literature database work with exporting citations to pdf?Would you then still need to maintain the BibTeX file? ---
**Author:** ![danderzei](https://yyz2.discourse-cdn.com/free1/user_avatar/org-roam.discourse.group/danderzei/32/59_2.png) [@danderzei](https://org-roam.discourse.group/u/danderzei)\ **Post date:** [September 2, 2021, 12:04am UTC](https://org-roam.discourse.group/t/an-idea-for-a-roam-based-bibliographic-manager/1997/4 "2021-09-02T00:04:15Z")
Researching my own question, i stumbled on this interesting discussion by the Org-ref developer John Kitchin: > > > I originally thought that the new citation syntax in org-mode was going to make …the citation links in org-ref obsolete. Now that I have worked my way through org-ref-cite (https://github.com/jkitchin/org-ref-cite), I no longer think that is true. It is more clear to me these two projects had different goals, and those goals are not fully compatible. It comes down to org-ref using bibtex/biblatex as the predominant citation processor, and org-cite using CSL. These two processors have different syntaxes, and I don't think it is possible to have a single approach that works for both of them without making compromises in capability. I use org-ref professionally, and I don't want to make compromises on its functionality. > > In org-ref, I used the link type to indicate the LaTeX export command explicitly, so it looks like what you would read about in LaTeX documentation. That is a feature IMO, so you don't have to figure out how to translate the cite style. Of course, that is only a benefit if you know LaTeX, and are aiming to generate LaTeX for scientific publications. The disadvantage of this approach is trying to define the export behavior for other backends, which has been limited in org-ref. org-cite, on the other hand, uses a generalized citation style like cite/style/variant which is the same across export backends. I found it was not easy to map these onto the full set of natbib/biblatex citation commands, and not easy to remember that mapping while writing. > > Admittedly, the original syntax used in org-ref provided minimal support for pre/post notes in a clunky way that used the link description. The org-cite syntax supports this better, allowing common pre/postnotes, as well as individual pre/postnotes on multiple references. It turns out that org-ref can support this syntax natively in the existing citation links, which provides a path to exporting org-ref links to other backends using citeproc to render them. > > Finally, org-cite and org itself do not provide adequate cross-referencing (or index/glossary functions) for scientific publishing, so it is clear that the rest of org-ref also still has a place in this ecosystem. > > Soooo.... I have rewritten a lot of org-ref to support all these issues. There is a new "Version 3" in https://github.com/jkitchin/org-ref/tree/org-ref-3. I really struggled to decide this was the right thing for me to do; a lot of people have invested time in org-cite over the past years. In the end I decided to do it for these reasons: > > 1. This is the tool I need and use professionally on a daily basis. It has to do what I need the way I need it. > 2. I (and others) have \*a lot\* of legacy documents that use org-ref, and it is clear I have to be able to support those for a long time > 3. I thought it was worth pushing the link syntax to support this. The link behavior in org 9.0 was designed to support this kind of applications. I wish I would have figured out much earlier that this could all be done this way. > 4. There is room for both approaches. They have different goals and applications. > 5. I learned emacs-lisp to write org-ref, and the early code shows that. I am a better programmer now, and now the code-base is better than it was. > > Version 3 supports the legacy link syntax, and a new link syntax that is identical to what is in org-cite. > > I took this opportunity to clean up ~8 years of buildup. It now only supports bibtex-completion by @tmalsburg for candidates, opening notes, pdfs, light formatting, etc. The vanilla org-ref library is completion backend agnostic, so if you don't want ivy or helm, you won't get it, but you do have to wire in the completion backend you prefer (e.g. Selectrum, ido, etc.). I have removed the ivy and helm dependencies so you won't get them if you don't want them, although you will have to install them if you want them. > > If you do like helm or ivy, you can load \`org-ref-ivy\` which depends on \`ivy-bibtex\` or \`org-ref-helm\` which depends on \`helm-bibtex\`. Good news on this front, these libraries will no longer mess with the order of actions in them; there are dedicated insert commands that look like those functions, but aren't them. > > I removed all the \`org-ref-\*\` variables that were just shadowing \`bibtex-completion-\*\` variables to keep things simpler. I also took out all the key-bindings, in case you have different ideas. Those have to be set in your own init files now. > > Apologies in advance to anyone who contributed code that is now gone. The code is still in version 2, and it can probably be added back, let me know what it is, and I can work it back in. > > There are still some minor issues of over-active fontification between org-ref and org-cite that have not been resolved, but they only affect the cite: link. > > I am going to use "Version 3" myself for a few more weeks to get it as bug-free as I can before I merge it on the master branch, at which point it will become the standard version available in MELPA. It is pretty functional at this point, and I only anticipate fixing some small corner case bugs in the coming weeks. I still need to test it more thoroughly in the new version of org-mode with org-cite enabled. > > The version in MELPA is still what I call "Version 2" with the old syntax everyone is used to. There is a tagged version at 2.0.0 in MELPA Stable, and the Version 2 branch can be found at https://github.com/jkitchin/org-ref/tree/org-ref-2, for anyone who wants to keep the old behavior. > > It is conceivable I might finally be able to split out two new packages for org-ref-ivy and org-ref-helm that do properly install their dependencies, but I am going to wait until it is ready to merge to worry about that. > > Until then, if you are the bleeding edge type, give Version 3 a spin, and let me know of any issues. ---
**Author:** ![mshevchuk](https://avatars.discourse-cdn.com/v4/letter/m/35a633/32.png) [@mshevchuk](https://org-roam.discourse.group/u/mshevchuk)\ **Post date:** [September 2, 2021, 7:19am UTC](https://org-roam.discourse.group/t/an-idea-for-a-roam-based-bibliographic-manager/1997/5 "2021-09-02T07:19:54Z")
> Let me know what you think, and whether you think this might deserve further thought and effort. It definitely deserves further thought and evaluation. I would be interested to see a working prototype of this, but personally wouldn’t invest my time into it right now and that’s why: 1. It’s technically more challenging than it may seem. And we already have a reference manager that indexes bibliographic data into an SQL database. It’s called Zotero. 2. Emacs is more about text data and BibTeX with all its cons, cons and cons perfectly fits into this scheme. The BibTeX format is mature and widely used. It’s a native format for JabRef and BibDesk and a few minor scale programs and is supported to various extend by all reference managers, by all the publishing houses and so on. 3. An Org-base reference manager is ultimately limited to Emacs in the foreseeable future. I played many years ago with Org-bibtex and couldn’t figure out how to fit it into my workflow. > 1. That’s a big software stack, with lots of moving parts to keep track of. In fact no. Helm-bibtex is an interface to bibtex-completion and is not independent of it. Org-roam-bibtex is an extension to Org-roam and is not independent of it. You have two packages for two different tasks. It’s a sort of Unix way within Emacs. > 1. Helm-bibtex and its siblings need to parse all your bibtex files each time you try to insert a citation. That can get pretty time-intensive, if you have a large bibtex file, like I do. Maybe something’s wrong with your config? Hem-bibtex and similar re-parses the BibTeX file only if it has changed. It stores all the data in an internal cache, which is pretty instantaneous. For me the initial (re-)parsing takes a bearable amount of time (~2 s) on ~2000 BibTeX entries and it’s done only once in a while when I update the BibTeX file. As I mostly work with BibTeX in an external program (BibDesk), I often don’t notice that. > 1. Org-roam notes reference citations ( `cite:stanley2021` ), which point to entries in bibtex files ( `@article{stanley2021...` ), which are what ultimately contain the bibliographic metadata. But this means that you’re maintaining notes for bibliographic entries in two places, ultimately. This can get messy. I wouldn’t agree with that. You store bibliographic data in a proven and future-proof format. You however don’t store your PDF files in that format, do you? You just keep links to them. Neither do you store your bibliographic data in PDF format, although most modern PDF files in fact keep such data internally. So why would it be messy to store a conceptually rich-text note in a separate file? > 1. Org-ref is slowly being replaced with org-cite, the built-in citation mechanism for Org-mode. No. Org-ref adopts to the new mechanism, but it’s not going to be replaced by it. All that said, I would encourage you to pursue your ideas. They have a potential. They may not result in what you originally anticipate, but actually in something even more useful. ---
**Author:** ![bruce](https://avatars.discourse-cdn.com/v4/letter/b/ccd318/32.png) [@bruce](https://org-roam.discourse.group/u/bruce)\ **Post date:** [September 3, 2021, 1:41pm UTC](https://org-roam.discourse.group/t/an-idea-for-a-roam-based-bibliographic-manager/1997/6 "2021-09-03T13:41:25Z")
> [@mshevchuk](#): > > Org-ref adopts to the new mechanism, but it’s not going to be replaced by it. Org-cite alone will not replace org-ref, given the latter’s scope is much wider (it includes cross-referencing, glossaries, etc.). But org-cite + smaller and more focused packages, along with possible tweaks to org, could. Hopefully org-ref itself evolves in that direction, or if not, other developers and packages provide that. ---
**Author:** ![mshevchuk](https://avatars.discourse-cdn.com/v4/letter/m/35a633/32.png) [@mshevchuk](https://org-roam.discourse.group/u/mshevchuk)\ **Post date:** [September 3, 2021, 9:05pm UTC](https://org-roam.discourse.group/t/an-idea-for-a-roam-based-bibliographic-manager/1997/7 "2021-09-03T21:05:04Z")
@bruce By the way, do you have any idea why Org-cite’s parsing is so slow? I didn’t measure it precisely, but I managed to count to 100 while waiting for parsing initiated by `org-cite-insert` to finish. Bibtex-actions and Bibtex-completion parse the same library of 1954 entries within 2-3 seconds. ---
**Author:** ![bruce](https://avatars.discourse-cdn.com/v4/letter/b/ccd318/32.png) [@bruce](https://org-roam.discourse.group/u/bruce)\ **Post date:** [September 3, 2021, 9:35pm UTC](https://org-roam.discourse.group/t/an-idea-for-a-roam-based-bibliographic-manager/1997/8 "2021-09-03T21:35:21Z")
You mean the oc-basic insert processor? I noticed that too. Does it not use parsebib? ---
**Author:** ![mshevchuk](https://avatars.discourse-cdn.com/v4/letter/m/35a633/32.png) [@mshevchuk](https://org-roam.discourse.group/u/mshevchuk)\ **Post date:** [September 3, 2021, 9:54pm UTC](https://org-roam.discourse.group/t/an-idea-for-a-roam-based-bibliographic-manager/1997/9 "2021-09-03T21:54:08Z")
Yes, I’m currently using the basic processor. I now have more time for a little bit of Elisp, so will try to dig into it, thanks for the tip. ---
**Author:** ![bruce](https://avatars.discourse-cdn.com/v4/letter/b/ccd318/32.png) [@bruce](https://org-roam.discourse.group/u/bruce)\ **Post date:** [September 3, 2021, 10:47pm UTC](https://org-roam.discourse.group/t/an-idea-for-a-roam-based-bibliographic-manager/1997/10 "2021-09-03T22:47:43Z")
You know about oc-bibtex-actions? There’s a insert processor there (config example [here](https://github.com/bdarcus/bibtex-actions#org-cite)). Also: > > > Sorry; because of the massive recent merges, and my shaky git skills, I had to r…edo #5212 to get the git correct. > > \## Summary > > This adds support for native org-mode \`org-cite\` citations. Specifically it: > > 1. configures the native \`org-cite\` processors > 2. adds \`oc-bibtex-actions\` "follow" and "insert" processors for the vertico module > 3. ~adds and configures the new \`org-ref\` OC rewrite: https://github.com/jkitchin/org-ref-cite~ (removed because of unclear future) > 3. installs \[citeproc-el\](https://github.com/andras-simonyi/citeproc-el), which is required for the \`oc-csl\` export processor, and will be needed also for forthcoming preview functionality > > \## TODO > > Beyond addressing the issues below, the key feature I want look into: > > 1. A dedicated embark map that includes some of the functions in the \`org-ref-cite\` hydra, so that ui and the embark-act which-key one are more comparable. > 2. Finish README. > > \## Status > > This is ready for testing, with the below caveats. If you can help with any of this, please do. > > The \`org-cite\` ecosystem has picked up steam quickly, so my plan is to keep this in draft for a bit to give it a chance to mature: \`org-cite\` bugs to be fixed and new features added, new third-party functionality ready to go, etc. > > \## Issues and Questions > > \### What to do with #2888? > > 1. Drop \`org-ref\` from there. > 2. Merge this. > 3. That can rebase on this and just focus on \`org-roam-bibtex\`. > > \### Keybindings? > > This is tricky. > > We need to bind \*\*\`org-cite-insert\`\*\* and \*\*\`org-open-at-point\`\*\*, as these are the two entry points for the org-cite module. > > But how, and where? > > But \`org-open-at-point\` is also more general, so probably should be in more than one place. > > A biblio submenu would also be helpful given the above, and that \`bibtex-actions\` (for \`vertico\`) has multiple entry points, and a keymap to access them (which is also used by \`embark\`). ---
**Author:** ![mlemerre](https://avatars.discourse-cdn.com/v4/letter/m/e480ec/32.png) [@mlemerre](https://org-roam.discourse.group/u/mlemerre)\ **Post date:** [September 3, 2021, 11:10pm UTC](https://org-roam.discourse.group/t/an-idea-for-a-roam-based-bibliographic-manager/1997/11 "2021-09-03T23:10:02Z")
I have a setup which is quite close to the one that you propose, except that all of my bibliography (and notes, and links to PDFS) is in a large [biblio.org](http://biblio.org) file. I have commands that parse bibtex that I copy-and-pasted and insert it at the right location in the file (which is sorted by the reference id). Everytime i add such an entry, org-bibtex just regenerate the .bib file; and I use org-ref to insert citations easily, go to the node or get the PDF. My big problem for a while was a workflow for capturing. Now I use this: I open my pdf in emacs. If the paper interests me, I copy the title, and I have one function that uses biblio.el to lookup entries in a database, retrieve the bibtex, then put it in my .org file at a correct location, copy the PDF at a suitable place, and insert a link to it from the [biblio.org](http://biblio.org) file. If that is interesting, I can put my code online. I would be interested in splitting this file into several org-roam notes, as it is becoming quite large. ---
**Author:** ![mlemerre](https://avatars.discourse-cdn.com/v4/letter/m/e480ec/32.png) [@mlemerre](https://org-roam.discourse.group/u/mlemerre)\ **Post date:** [September 3, 2021, 11:11pm UTC](https://org-roam.discourse.group/t/an-idea-for-a-roam-based-bibliographic-manager/1997/12 "2021-09-03T23:11:21Z")
(But as in org v2 notes can be outline in a file, I think my setup already works quite well with org-roam)