@nobiot , that’s exactly what I had also in mind! The primary reason I had suggested using the built-in bookmark functionality is to provide a built-in way to create such “pins”. But in general, it’s nothing but collecting lists. And those lists can be exported in all kind of ways, e.g., into transclusion buffers.
So I would suggest that I will focus on providing a rather open, flexible list UI for (a) collecting lists, (b) exporting lists, (c) store these lists persistently, (d) use search results as list (the latter only on lisp-level, until there is a ql). This would include a minor mode to “pin” the node at point, which in turn could be called by org-roam-ui.
Such a tool could also provide, say, a history of all items visited during the current session. Ideas begin to flow…
Delve based on v2 is now published: Delve. From the list in my latter post, it does (a) - collecting lists, (c) - storing them persistently; and some of (d) - built-in queries. Exporting to transclusions (b) will be easy to implement; since storing lists already is a kind of exporting.
I will focus now on gaining some stability and in particular cover it with tests. Seems to be stable overall, however. So feel free to test it.
@ThomasFKJorna Still interested in integrating that with org roam ui? Delve comes shipped with a minor mode which already allows to collect nodes from an org roam buffer, it should be trivial to adapt that to org roam ui. All we need is the ID of the node to add.
Awesome, I’ll try it out soon! I just added hooks for providing the id of a node after opening it from orui, look for org-roam-ui-before-open-node-hook or the same but “after”.
I’m also interested in adding a more tightly integrated node collection mode, where you can just click the nodes without opening them in Emacs, but this should do for now!
I finally had the time to install ORUI, and I must say I’m impressed! It looks nice and is actually informative. I don’t see, however, how you imagined the planned integration with Delve. The hook org-roam-ui-before-open-node-functions is called if I open a node inside ORUI, right? So is the idea that in this hook, we register the id and add it to a collection? I would actually prefer an explicit ‘collect’ command (right-click menu), but here I just want to understand how it is intended to use.
The other thing: How do you render the preview? Do you open the file in the background and copy it, as I do, or is there any trick? And how do you determine the bounds of the passage previewed? I grepped “preview” in the sources but was overwhelmed with the results…
Glad you got the chance to check it out, and thank you for the kind words!
To be quiet honest, I was also a bit confused on what you imagined with this collaboration, so I think we may have been talking past each other
The explicit “collect” command is actually what I had in mind at the start, but then I think I got confused along the way. I would be more than happy to implement this (makes much more sense to me as well), so how would you imagine this? I personally have not played around with Delve a lot (should do so though!), so I’m not sure how the “lists” function: is there one bookmark list, or are there multiple ones? In the latter case, I could imagine the right-click menu having multiple options for adding the node, something like
Add node to list > Thesis
______________ Draft
______________ something else
etc.
However, depending on the number of lists this might get kind of messy.
A simple collect function which just sends an id would work fine, of course, but I personally would prefer some tighter integration with Delve if possible (such that org-roam-ui would actually call the relevant delve function as well).
(bit of a side rant)
While this is arguably less flexible, I would like org-roam-ui to act as a way to discover and glue together some of the many org(-roam) packages out there, and I like Delve enough to give it “first class support” so to say. Emacs and Org-mode are fantastic, and being able to craft a perfect setup for yourself is amazing, but needing to make everything work together yourself is kind of a pain (I think). Having built in integration would be ideal I think. In the future I would like to accomplish this with plugins, but the codebase is not really in a state where that could be done easily (plus I’m not that proficient at building APIs yet).
/rant
I will split up this way too long message by topic, see the next one re: the preview.
Yes at the moment the message is basically copied (I use simple-httpd's insert-file-contents-literally, which just opens a temp buffer and serves the contents to http://localhost/id/<id>, from where I query it on the frontend. The file is then parsed using uniorg, but that’s not super relevant here.
At the moment I don’t grab the specific node, just the complete file. This is something I’ve been meaning to address, but have not gotten around to yet (mostly because I mostly use single-file notes myself).
If you have solved this problem yourself please let me know! I’ve been meaning to implement that for a while. If you haven’t, I will probably implement it on the frontend side by using the pos column from the org-roam-db rather than do it on the emacs side, as that’s probably faster.
Last thing I wanted to mention: I’m currently working on a query system for org-roam-ui in order to have very extensible filters, rather than just filtering based on tags and some predefined toggles.
Harking back to the “integrating well with other org-packages” above, I would like to make this querying system as interoperable as possible with other org-packages, and I think Delve would be a fit for such a system. You already have some sort of query system, although I could not really figure out how extensible it is, can you only set these using the delve-dashboard-tags variable?
I would really like to be able to query the org-roam-db, or, more accurately, get a list of nodes from that database, without using SQL, but something more like org-ql or like Roam Research query blocks. I would see this used in three situations:
In org-roam-ui, obviously, as I want to use it to create filters.
In Delve, to easily define a list of nodes to view/safe
In @nobiot 's org-translusion, to insert a (preview of a) list of nodes in any org-roam-buffer, to emulate Roam Research’s source blocks.
I think these are very feasible things to do, and I have already made some sort of system for org-roam-ui, but I would ideally like
a) us to use the same syntax
b) for these “blocks” to be as interoperable as possible
This would be such an asset to the org-roam ecosystem! Especially being able to view the nodes as filtered by date (plugging my own org-roam-timestamps) and links would be great, such as “give me all nodes which link to X and Y created after ZZZZ with tag W”. Delve would be such a fantastic way to visualize that (really great job on the ui side btw, love the icons).
The way I currently have the queries implemented is using the following syntax:
Obviously not very lispy, kind of ideosyncratic and just kind of cumbersome. The actual syntax does not matter that much to me. Roam’s syntax is something like {{query: {and: {property operator value}}}}, people use that all the time.
What is important is that we use the same syntax, otherwise there’s not really a point to having this shared.
What are you guys’ thoughts? I really want to implement this and would probably end up making some pull requests for org-transclusion and delve to add these features, but I don’t want to ideosyncratically decide this, nor do I think I understand your packages as well as you do and whether this would be feasible at all!
Here are some threads which have touched on this idea as well:
Cool, I think we actually had the same idea in mind. +1 for the submenu offering different collections (such as ‘Draft’, ‘Thesis’, etc.). This is already the common use case in Delve: When “collecting” something, the user is offered a menu with existing persistent stores and open buffers to which it can add the node in question.
How can we do this in an integrated way? The Delve submenu offers a menu of both buffers and files. The most simple way would be, for starters, to simply offer a collection of open Delve buffers (which can be visiting files, of course). We’d need a function which returns a list of buffer objects (which already exists: delve-buffer-list). Then, ORUI could first call this function to determine the buffer, and then call a collection function which adds the ID to the buffer object returned. Something like that latter function already exists, I would just have to refactor it so that it accepts an ID as an argument (or a list of IDs even).
I could do a PR for ORUI if you point out the places where these functions will be added. But I haven’t done any javascript or derivatives yet; no practice there.
Re: your “rant”: That’s not a rant, that’s a perfectly understandable vision! If you have any ideas how this little cooperation here could already lead to some useful abstraction, also on the Delve side, if necessary, that would be cool! Think of the selectrum/marginalia/consult stuff, they managed to write packages which work well together and still focus each on their own stuff.
And finally re: the ql: That would be the absolute killer feature!
I had toyed around a bit with a QL, too, but I struggled to understand how to map such a QL term to a SQL Query. That was above my limit of understanding both lisp and SQL. Or are you planning to offer the QL as a subset operating on a given list of nodes?
Let’s talk about how we can make both packages together well. I also agree that there should be some common “platform” so the entire ecosystem can flourish – I’m happy to see Delve-ORUI is already making their headway.
Integration with ORUI might need to be a separate “third-party” package outside ELPA, especially given ORUI is now in its own way to MELPA (I saw a PR discussion?) and Org-roam also in MELPA.
@nobiot I am looking forward to write an export module which exports a Zettel collection to an org buffer with transclusions. As to the “ecosystem”, I think the first and most basic step to further interoperability is to focus on the API. And here, the most natural move to me seems to always offer a function which accepts IDs as argument (instead of nodes, or links, etc.), and to make them publicly available (one dash, not two). Given that all packages work with node IDs, that will already facilitate things.
I think it would be doable to map to SQL, somewhat similar to this package:
This package only works with org-roam-v2, but should be mappable to v2. I think it’s doable because we basically just need to write some “where” queries.
However, the only way to make this work sensibly would be to restrict which kinds of queries can be made using the syntax and not just allow ALL sql (although we could allow users to just write sql if they want, but that should be implementation dependent). I am thinking of “allowing” the following keywords:
file
title
id
level
tag
todo state
directory (same as file, but useful to have it for clarity)
ctime (debatable as it’s not logged directly)
mtime
other properties (debatable, as this is not easily accessed using emacsql)
is linked to from (backlinks basically)
links to (forward links)
connection (forward/backlink agnostic, should be default)
cites
Additionally, I would like “bare” words in the query to be used to make ripgrep searches for those words. I think that is very useful, and ripgrep is so fast that I don’t think we should be worried about performance.
I would use the following modifiers
and
or
not
For keywords that have a numeric value (creation date, number of links etc) we should have a few modifiers, namely
<
<=
>
=>
=
Should be doable to create such a syntax, we just need a way to map the appropriate keywords/modifiers to SQL WHERE queries, most of which are not very difficult (for filenames, just :where (like node:file "%<filequery>%"). Tags and cites are somewhat more complex, but not much, and for full-text search we don’t need to make queries obviously.
As for the actual syntax, I think copying org-ql’s syntax is a good idea, might even be able to integrate with it.
And congrats about being added to ELPA! This org-roam-ql syntax for transclusion then might indeed work better as its own package (or, ideally, maybe be integrated with org-roam itself), as I think it’s good to keep org-transclusion only dependent on Org.
That sounds good. I’ll approach it the weekend, I hope. Just one thing: The “message”, in your alist, is just a marker that the data will be passed (or concerns) delve, right? So it makes sense to add a further symbol in that data list which defines the actual command, something like:
I guess my question was rather if the search string will produce an efficient query. Writing just some “where” queries actually is my approach, too, in Delve. I use a super query (thanks to @d12frosted) which is also used by org-roam, and which returns all data for a node. Then I wrap that around a SELECT statement and add some where clauses. I just thought that might not be the most efficient way to do it, but actually it works pretty fine unless you retrieve all nodes.
Said that, I fear that blanket querying all links (“connection” in your list) is too costly, since it would require lots of joins which are often not even used. I used, such a blanket query, including joins, in my first Delve version and it was really slow. Unless, of course, @d12frosted has a magic SQL query which also solves that problem
@public_image_limited Sorry I didn’t read the whole thread and my comment might be not very useful, but wanted to share something with you regarding efficiency of query operations.
Recently I added links to vulpea-note. And quickly noticed that performance degraded to the point where my day-to-day workflow started to suffer. So I decided to take a path of least resistance and provided some specialized queries (because they kind of solve the problem for my most used patterns). You can find more details in this discussion, but in short the idea was to narrow down the list of notes that I need to query using that horrific SQL with multiple joins, so that performance overhead is not that bad.
I just thought that might not be the most efficient way to do it, but actually it works pretty fine unless you retrieve all nodes.
Just as you say. It does not really scale. Limiting scope is kind of a hack. It turned out that it take 4 seconds to query ~9500 notes. What? 4 seconds??? So I decided to play with completely different approach - a view table, where all the data is already there. So no need to multiply several tables/matrices during query operation, instead build everything on sync. It reduced that time to 1 second! And it still enables Emacs Lisp filtering (e.g. it’s very powerful)! You can find more information in this PR. It is already merged and I am using it for few days and everything become really snappy. Insertion right now has penalty because I am kind of parsing the buffer for the second time together with org-roam, but that will be fixed in future iterations.
Unless, of course, @d12frosted has a magic SQL query which also solves that problem
