[RFC] Interactive Visualization and Explorer Tool

[Memex]

Few weeks ago I started to build an interactive graph visualizer and integrated note browser (see a demo here). At first it was just a personal project and curiosity. However, the community showed appreciation and interest in the tool, and graphing capabilities of org-roam became a central discussion topic with many ingenious proposals. Right now I think it is a good idea to consider building it at full scale, and that’s why I need your comments on a few things.

Architecture

There are a few possible ways to organize the codebase.

At its core, what I am doing is just taking a JSON file (relationships between nodes and edges) and some HTML files (org-publish of the entire org-roam directory) and displaying them in a webpage using a http-server.

Now, I think this much should be a package separated from org-roam (primarily because this is not written in Elisp).

However, I am not sure what to do about the conversion of the SQL database to a JSON file part. This can be done in any language (and to do it incrementally, I think we should do it from Emacs, but I am not sure about scalability and performance). If we are doing it in Emacs using Elisp, should this be part of org-roam codebase? If not using Elisp, what other option do we have?

Also, as I am displaying backlinks below each file in the note browser, org-publish needs to be force called on file X each time I reference file X in any other file. I am not sure if this will cause performance issues since AFAIK org-publish is synchronous in nature. Would love to know if someone has a better idea.

Modularity and Other Graphing Features

Clustering and centrality measures seem to be the most sought after features. I think it is possible to achieve these using something like igraph where we add extra information in the JSON file. However, I am not sure if this can be done incrementally, as it would most possibly require computation using the entire database (which is compute-expensive). What are our options then?

Packaging

Right now, I am using a shell script that generates the JSON database and starts an http-server. This is complicated to setup cause you have to manually modify some parts of your Emacs dotfile as well. What would be the most user-friendly option for packaging this entire thing?

I welcome all of your comments, questions, and suggestions.

[jethro]

@Memex

Now, I think this much should be a package separated from org-roam (primarily because this is not written in Elisp).

This is fine, we’ll be rearchitecting Org-roam in a way that should help with integration with other graphing backends.

@Memex

However, I am not sure what to do about the conversion of the SQL database to a JSON file part. This can be done in any language (and to do it incrementally, I think we should do it from Emacs, but I am not sure about scalability and performance). If we are doing it in Emacs using Elisp, should this be part of org-roam codebase? If not using Elisp, what other option do we have?

We should have JSON output from within Org-roam, it’s not too big an issue, will at least be as performant as the current graphviz output.

@Memex

Also, as I am displaying backlinks below each file in the note browser, org-publish needs to be force called on file X each time I reference file X in any other file. I am not sure if this will cause performance issues since AFAIK org-publish is synchronous in nature. Would love to know if someone has a better idea.

This is probably not a good idea. You can use Emacs to run a batch script to run org-publish instead, rather than blocking the main Emacs instance.

By the way, there’s a graphing server already open-sourced in the wild: GitHub - org-roam/org-roam-server: A Web Application to Visualize the Org-Roam Database

I think an exchange of ideas/collaboration on this front could be fruitful.

[tecosaur]

A quick question from me, @Memex why couldn’t a standalone HTML file work? We can run scripts, fetch local files, etc.

[Memex]

@tecosaur I am not an expert in web development but seems like loading other HTML files in an iframe is much easier using a server.

@jethro Wow. This project makes me question whether I should continue with my efforts or not. While I can think of some reasons why even having two different implementations of graphing would be a good idea, I’m not sure if net productivity would be worth that.

[tecosaur]

@jethro

By the way, there’s a graphing server already open-sourced in the wild: GitHub - org-roam/org-roam-server: A Web Application to Visualize the Org-Roam Database

This looks great! I’m always a fan of extending a good existing solution (when one exists) over doing everything from scratch

[mpick]

Maybe this could go into #dev:graph ?

Just adding to the conversation : it seems like Obsidian beta uses G6 or similar. It’s beautiful, interactive, useful.
They make the node size proportional to the number of links to/from other nodes, and it seems to work well.

Clicking a node title simply opens it for now.

When the mouse hovers over a node title :

[mpick]

Edit : from a discussion in Obsidian’s Discord, it’s https://d3js.org/ not G6.