Merging Org-roam and Org

Development
1

The longer I use Org-roam, the more I get convinced - Org-mode has to change its conceptual model. The file system base model is no longer sufficient. The optimized and efficient indexed DB-driven model used by Org-Roam is the future of Org-mode. It is faster, and it is more straightforward. Easier to query, etc.

I think we are only delaying inevitable. Even if we consider the subject of migrating Org-Roam into Org-mode completely as a controversial and highly debatable matter, we definitely should start talking about making at least some parts of Org-Roam to be in Org-mode.

Perhaps the DB thingy is an excellent starting point.

I don’t know if that’s happening already, but I haven’t seen any discussions in the emacs-orgmode@gnu.org mailing list.

And we need to start talking about this and perhaps even get vocal about it. Things in the Emacs world move at a very slow and steady pace. And Org-mode is a giant heavy-weight machine. The sooner we start discussing these things among ourselves, and sooner we invite core Org-contributors into the discussion, the sooner we get done something useful for the entire community of Org-mode users.

I realize that this all sounds like I am speaking sitting on a high horse, and I understand the complexity and the sheer amount of work of this endeavor. Unfortunately, I am not offering concrete solutions here because I don’t know where to start with this. But I am positively confident we should at least start digging in that direction.

4 Likes
2

I can see a lot of disadvantage to doing that. What would be the advantage?

2 Likes
3

Well, let me list the pros of indexing the org files and migrating the roam-db feature into Org-mode:

  • This model is more scalable. Currently, Org-mode is optimal to work with a few files that contain lots of headings. It should also work nicely with lots of files with fewer headings;
  • It is faster. Integrating the indexing in Org-mode, would make Org-Agenda more responsive, it would load faster, regardless of how big is the number of entries;
  • It will improve the re-filing and moving around the headings ;
  • It would have a better interface for searching, aggregating, and grouping data;
  • It would bring a simple way for analyzing the data. Can you today easily build a clock report to find out how much time you spent working on work-related things vs. open-source contributions for a given month?
  • It would open possibilities for integrating proper full-text search using NLP - stemming algorithms, etc.;
  • It would open possibilities for synchronizing the data between different systems;
  • It would make collaborative work possible. You don’t have to keep track of whose changing the given note in the Org file itself;
  • Having a unified model of storing data would be great for Org-roam and would reduce the fragmentation of different useful features

Now let’s see the apparent disadvantages:

  • This is not a small work;
  • Keeping Org-mode backward compatible would be a challenge;
  • It would not be easy to persuade people who’ve been using Org-mode for years and don’t care about Org-Roam at all;

I’d say the pros easily outweigh the cons. Certain features of Org-Roam have already proven to be a natural next step in the evolution of Org-mode. Yes, evolution is always risky. Sometimes the entire species can get wiped out. But evolution still better than the alternative - which is stagnation.

4

FWIW Bastien has reached out before, asking what changes to Org-mode can be made so that Org-roam can be developed easier. I don’t really have a good answer to that either.

I don’t think caching should be a part of Org-mode. There are too many questions to answer here: what to cache, when to cache, how to cache. Besides, this introduces an additional dependency (sqlite3) and data duplication.

To this end I think Org-roam is already in a good place. An optional extension to Org-mode with opinions on the above questions, to serve a singular purpose.

I do intend to implement stuff like agenda on top of the cached data, maybe what Org could do is to instead expose an API to build the interface from a different data source, but that seems a lot of effort.

6 Likes
5

I am curious, could you please share them?

6

I think, this is where controversity comes. Because all previous points are achievable by building db of metadata (like org-roam does now), meaning that your files are source of thruth. But collaborative work requires files to be a view and db to be a source of truth (because you need to work in terms of operations). And it opens a huge class of questions and issues, which I think are out of org-mode scope.

  • It would open possibilities for synchronizing the data between different systems;

It is possible already. It’s easier to synchronise non-binary files. DBs are on the other hand are not that simple.

And it makes me wonder. What is org-mode? In a nutshell, it’s a language and a simple outliner that does not impose any complexity on the users. But here we are, users imposing so much complexity on org-mode that are relevant not uniquely to org-mode. So I think that org-mode should focus on small things with good API, so bigger stuff can be built on top of that core.

This is what I believe should happen to org-roam, it should provide a core for managing metadata (including links) with APIs to build on top of that (like org-roam-buffer, dailies, agenda and other stuff) which could be shared either in the org-roam repository or separately.

3 Likes
7

What’s your idea about that ?

8

This project has been in heavy development until now, and is slated for a major breaking change.

EDIT: I somehow missed that jethro had weighed in on this before I posted it. But will leave here.

The main one I was thinking of was related to this.

I guess one way to find out if there’s any interest in this is to ask jethro, and if he’s open, then ask the org list.

I think that project is pretty conservative with change, so am skeptical.

9

@agibragimov: I’m glad you bring this up, as I have independently been having a lot of similar thoughts.

I think part of the confusion here is that org-roam is two things at once: an opinionated workflow for knowledge management built on org-mode, and a bunch of enhancements to org-mode that facilitate that workflow. Those enhancements themselves can be divided into those that are specific to the org-roam workflow and those that are general improvements.

In particular, I think it’s pretty hard to deny that org-mode is slow; For instance (to pick a little on @d12frosted who’s already in this thread), this post on building a “Dynamic and fast agenda” starts with a 50 second agenda build time (!!) but brings it back to less than 1 second. Memacs FAQ mentions a 4-second org-agenda latency. The internet is full of users complaining about poor org performance only to be told to:

I hope this isn’t controversial, and I don’t mean to sound ungrateful, but: it’s 2021, there’s no reason parsing some text files that fit in memory 10000x over should take longer than a packet of data going from me to a data center 500 miles away that searches the entirety of text content humans have produced in the past 50 years for something matching the string “doughnut shops houston.”

I think part of the reason that org is slow is that it’s very flexible, has hooks for everything, etc., and part of the reason is just that Emacs isn’t a great platform for building snappy applications, but a big part is that it wasn’t ever engineered for performance. I think you could make org-mode a lot faster without any changes to the user-facing behavior.

So IMO the future is not parsing plain-text files in elisp; it’s building on the core of org-mode as it currently exists, maintaining its current flexibility. In my dream world, we:

  1. Bring even more enhancements into org-roam to facilitate ever-better workflows (I’m really excited to hear that @jethro wants to build something agenda-like). As I understand the org-roam v2 improvements, they’re a step in the right direction here.
  2. Backport only the general ones to org-mode in a non-breaking, non-opinionated way. For instance:
    • (Here’s where I differ with @jethro): I think some kind of standardized DB format isn’t a terrible idea – the org agenda is really slow right now, and there’s a subset of things that seem obvious to not reparse every time (tag inheritance is good! I don’t want to turn it off). I think there’s a reasonable model that keeps the files as a source-of-truth and uses the DB just for performance enhancements (i.e. moving)
    • Elisp is not a good language for parsing in: can we pull a ripgrep here? I’m imagining: moving the core of org-mode parsing into a low-level language, optimize the heck out of the implementation, running a daemon that inotify-watches a directory. Not unrelated to e.g. an org-mode LSP server. This could be totally optional (you could imagine a variable controlling a range from pure-elisp, to running this program synchronously, to the full daemonized experience, to not having text files anymore) and would have no impact on semantics.
    • I don’t want to separate org and Emacs, but I think modularization wouldn’t be terrible – the phone apps for org are pretty rough right now (for very understandable reasons!), and they’d benefit.
    • The org code is super tightly coupled – it mixes concerns left and right. Displaying menus, buffer management, etc. happens in the same functions as parsing. I think having org-roam be a first-class kinda-separate downstream user will help untangle a lot of this.
  3. This would hopefully result in improvements like live agenda updates, as-you-type search, being able to have an org-file that’s more than 10K lines long without Emacs falling over, etc. And a better org-roam experience too!

So I think that org-mode should focus on small things with good API, so bigger stuff can be built on top of that core.

I agree totally; however, I think that the API should be really good and the implementation should be really good.

Of course, this is a lot of work and very difficult to get right, and I’m in the peanut gallery right now. I also want to be crystal clear that problems always seem easier to solve from the outside, and I have enormous respect for the work and decisions of the org-mode (and org-roam :slight_smile: ) maintainers so far. I think the conservative approach is the right one until we can compellingly demonstrate (1) huge benefit (2) without breakage (3) or imposing unrealistic costs on the org maintainers. All that said: I hope to put my money where my mouth is and start building some proofs-of-concept soon, and @agibragimov (and others) if you’re interested in the broader project of org/org-roam generally I’d be happy to brainstorm. I think it’s totally possible to build something that’s so good here it’s relatively uncontroversial to pitch backporting (I think org-roam is so good it’s crazy to use emacs without it! Can we make other people see that?)

7 Likes
10

These kinds of threads seem to be quite common around any software project of moderate age and popularity. People are always calling for grand refactors and reimplementations, and not for bad reasons. Org-mode certainly can be improved. But I would caution against calling for building around a totally new model like a DB format. Many people who use Emacs and Org use it because everything is in plain text. A database implementation being open-source does not ensure access to your files 20 years from now. People trust Emacs and Org because they have stood the test of time and remain maintained. Like you’ve mentioned, it’s easier for developers on the outside to declare bankruptcy on an aging codebase and start fresh. But rewrites inevitably fragment the user base and very often never supersede the original—Guile Emacs and Remacs are good ideas, but sadly they aren’t going to replace Emacs and Elisp anytime soon.

I think the current course of action isn’t as bad as it seems (certainly not stagnant). Org-roam is growing and improving, and org-mode is growing and improving, and a bunch of org-roam users without a history in the Org community flooding the Org mailing list calling for large refactors is only going to breed animosity. It’s easy to call for a big sweeping rewrite or merging of packages, but it’s more sustainable to dig in and improve things incrementally, as Jethro, Bastien, and more are doing now.

I think the communities would be better served if less time was spent discussing ways to move forward in favor of more time spent actually moving forward, if in less spectacular fashion.

PS: I don’t think Elisp is as bad as it seems, either. Some of the greatest advantages of Emacs come from its mostly-Elisp ecosystem, and hooking into external packages creates some limitations too. A good number of bullet point advantages of the database aren’t dependent on it. crdt.el and org-ql are improving collaboration and data analysis in Org already.

5 Likes
11

I have made comments in Org-roam Slack, I’ll repeat them here again:

First of all, let’s emphasize the notion of “optional”. I’m talking about the optional indexing layer, which is never intended and never will be “the source of truth.”
Second of all, Org-mode is not just syntax; it’s a holistic system - a project management tool and organizer, reproducible research instrument, blog publishing app, a document preparation system, and many other things.

It would be a bit awkward to say “let’s add an index caching layer to markdown.”

It would be awkward indeed, but again: Org-mode is not Markdown, it is much, much bigger than Markdown.Imagine, for example, if it was possible to build a system that would scan any given codebase, extract comments into Org-mode structures, and then would build a “knowledge graph” based on relations, extracted after applying natural language analysis of some sort.

which is why we have nice things like org-roam

My main concern is fragmentation. Org-roam already has parts that are now incompatible with Org-mode, e.g., Org-capture vs. Org-roam-capture.
And we still have to make so many changes to make Org-Roam work better. With the Agenda; With refiling; With sparse trees and search in general; With clock-reports; etc.
These new additions to Org-Roam will always be at risk of breaking due to upstream changes in Org-mode itself.

So I’m just thinking, can we design a higher-level abstraction, the fundamental indexing layer built-in into Org-mode? Something that would be useful for applications that are orthogonal to Org-Roam goals. Sure I’m not the only one seeing the potential, versatility, and extreme utility of a feature like that?

4 Likes
12

@liam makes some good points:

a bunch of org-roam users without a history in the Org community flooding the Org mailing list calling for large refactors is only going to breed animosity

Absolutely, I think we should avoid that. However, I think a lot of what would be required for an effort like this is smaller-scale, incremental changes that are a definite win regardless of whether we ultimately adopt any of the bigger ideas. For instance, to do such a thing would require more separation of concerns, a more thorough test suite, etc.

However, I’m also looking to other projects that do have a formal RFC procedure and the good, larger-scale changes that come out of that – in contrast with the quite incremental approach taken so far. A formal proposal makes no sense coming from someone without a history of contribution. But it doesn’t mean we shouldn’t dream big in the long term!

I don’t think Elisp is as bad as it seems, either.

I can definitely get over the ergonomics, but Elisp seems to encourage a highly untestable, stateful approach to programming that makes modularity a lot harder. That’s my big complaint (also, just that it’s never going to be as fast as C for some tasks).

That said, there are some incredible Elisp programmers out there doing very impressive (and useful) things.

On a different note, I’ll echo @agibragimov here:

First of all, let’s emphasize the notion of “optional”. I’m talking about the optional indexing layer, which is never intended and never will be “the source of truth.”

I think this is the key thing that would keep these changes backwards compatible. One way to look at it is that there have been multiple independent inventions of a DB caching layer for org, which suggests that perhaps we want one semi-official one. (Or at least an interface with org core that unifies a lot of the common tasks.)

Perhaps the right first steps would involve:

  • More prototyping (Jethro and the rest of the org-roam contributors are doing an excellent job of this; I have some orthogonal ideas)
  • Refactoring work to make the org-roam / org boundary very nice, clean, and modular, and at the right abstraction level (including lots of testing work).
  • Can we look at org-db, org-ql, and org-roam to see what they have in common? The old software adage is to not try to make something general until you’ve done it 3 times. If there are nice, shared abstractions, perhaps bringing something into org contrib would make sense (but definitely not at the cost of agility downstream)

So I guess where I come down is: yes, the skepticism is entirely warranted (and healthy/helpful; I appreciate pushback). But I think we can make some unambiguous, incremental improvements to org, perhaps with an eye on some longer-term, ambitious goals.

And a final (quite relevant) objection: talk is cheap, where’s the code? I’m guilty here (my org contributions have been minimal, though in my defense I’ve had patches get ignored a couple of times on the mailing list), and so my first steps in a project like this will look a lot like trying to get more involved in both these projects on a normal contributor basis, while simultaneously trying out crazy, orthogonal experiments. It’s going to be longer-term because I have a day job. So I guess you should consider this me volunteering to do chores for org-roam :slight_smile: I have the same username on GitHub. Feel free to tag me in, but I don’t want to impose management on the maintainers so I’ll probably pop in on the issue tracker as soon as I get a grip on the org-roam rewrite.

And one final, pretty out-there thought: I get the impression that org-mode contributors are doing most of the work in their spare time (which is heroic and I thank them). But there are some important institutions that depend on org-mode. I can think of at least one multi-billion (USD) trading company, not to mention lots of scientists. So there might be some grant/contract funding in the world (from an organization that supports scientific infrastructure, like the Simons Foundation) to support someone working part/full-time on these things (pending lots of paperwork, networking, etc.).

2 Likes
13

If you can come up with a good proposal/desired implementation, I’m certain Bastien is willing to peruse it. An RFC is especially necessary if there are going to be any backward-incompatible changes, but an optional caching layer seems okay, at least within org-contrib.

A big part of this is that you deal largely with stateful objects in Emacs anyway (e.g. buffers). When you’re not dealing with stateful objects it’s perfectly fine to go functional (data in, data out).

Nice :slight_smile: i’m always looking for alpha testers of the v2 branch. I’m not aware of anyone using it other than myself.

This is where @zaeph can chime in, he’s in the best position to speak about how development on org-roam or org can be sped up with monetary grants, and how to go about raising funding. Org-roam moves a lot slower than the other note-taking apps because my free time and mental capacity is rather limited: more code contributors taking on sub-projects (e.g. dailies, org-roam-agenda) will definitely help.

3 Likes
14

I dislike DB-based note-taking apps like the plague. I want all my notes to be under version control, and usable (and editable) by common tools, not locked into a binary blob. This was the reason I chose org-mode in the first place.

1 Like
15

And I’m not too fond of stupid sitcoms, plagiarism, spam calls, heavy traffic, and many other things. The intrinsic value of my complaints is at the same level of relevance to the topic as your comment.

It looks like you either didn’t read the post and the comments, or perhaps you don’t understand the mechanics of Org-roam and its DB. Nothing is “locked in a binary blob”, and no one is even suggesting anything like that.

If you want to get emotional, at least, please keep it relevant.

1 Like
16

LOL

17

FWIW, I, too, am nervous about losing the usability of org-mode’s plain text, and I am similarly concerned about getting locked into binary-ossification. On the other hand, while I can’t speak for @NightMachinary I do certainly fall into the category of “don’t understand the mechanics of Org-roam and its DB.”

Given that, I found both NightMachinary’s comment and your reply perfectly relevant. Well, to be precise, I found the fourth sentence of your reply to be relevant, since it gives at least a hint of reassurance that my fears are groundless. That said, I’d have found yours to be even more relevant had you gone beyond mere “proof by assertion”.

As to the emotional content of the two comments? Well, NightMachinary’s barely moved the needle for me, although maybe that’s because the phrase “like the plague” has lost most of its force because of COVID. :slight_smile:

Ironically, yours, on the other hand, in the form of everything but the substantive sentence four, actually managed to reach the very bottom of the “Mildly Hissy Fit” region on the dial.

18

Have any proposals been made in regards to merging org-roam with org? I understand that such a proposal would be met with resistance but is there technically a compromise between the two by creating a virtual filesystem from the org-roam database and would that scale well?

19

No, neither have I seen any strong push for this. I think this would make more sense only when no external dependencies are required (e.g. sqlite is fully integrated in Emacs), since some version of Org ships with Emacs.

20

For this reason, I think a better proposal would be to turn org-roam-db into a more general purpose separate package, org-db, if one does not exist already. org-sql exists but I’m not sure if it or any of its forks fit org-roam’s requirements.

By more general purpose, what I mean is that users should be able to set their own schemas for databases, create database instances for separate applications or share a database between applications.

As much as I like the idea of org-roam merging with org, I don’t see it happening but separating org-roam-db from org-roam is a first step IMO.