Org-transclusion -- now on ELPA!

[nobiot]

I have done a quick show & tell video on the work-in-progress version of Org-transclusion.
I will appreciate questions, feedback, and discussions.

Org-transclusion (alpha v0.2.1) Demo #9

v0.2.1 is merged with main today 2021-08-01

Video #8 featuring :only-contents and :exclude-elements properties per transclusion

Video #7 on live-sync edit feature with much more (2021-05-16)

2-minute video on how to install Org-transclusion from GitHub repo
Demo #6 2020-12-06

Source code is on GitHub. I included README to explain how it (should) work(s) currently.

GitHub - nobiot/org-transclusion: Emacs package to enable transclusion with...

Just very quickly, it’s my take on the idea John Kitchin outlined (link included in the GitHub README).

[zaeph]

Hi there.

This is amazing work, @nobiot. I’m liking a lot of things I’m seeing in your demo, and although I do have concerns about optimisation and recursion, this seems to be a solid proof of concept.

I’d love to look at your code to see how we could iterate on it. Could you ping me when you get around to it?

[nobiot]

Thank you! Yes, I’ll share the code. I hope one or two weekends away to clean things up and commenting to communicate the intentions, etc. more clearly for others.

Recursion is taken care of; as in, if you have another ortc: link within a transclusion overlay, the code skips it.

Optimization, as in performance, might be a valid concern. In my use case, though, I will probably have max 10s or 20s in one buffer; hoping that should not cause a big lag in the system…

[zaeph]

@nobiot

Optimization, as in performance, might be a valid concern. In my use case, though, I will probably have max 10s or 20s in one buffer; hoping that should not cause a big lag in the system…

It’s nothing that we wouldn’t be able to fix with some elbow grease. I had my own ideas about transclusions which I hoped to achieve with indirect buffers and narrowing, but your solution seems to work just as well.

At any rate, I’m starting to get more breathing space with work, and I’d love to have a look at your code in a couple of weeks, so take as much time as you need to review your code.

Thanks again!

[noorah]

This is exactly what I’ve been looking for. Thank you so much

[timquelch]

This looks pretty cool. Would love to see the source code.

[jethro]

Looks fantastic as is sharing this on Reddit.

[forkrul]

I haven’t been this excited about something in ages. Thanks for building this. This opens up a ton of opportunity for content creation, note, and book note embedding, and a bunch of other use cases.

Do you plan to have this to work for “turtles all the way down”? Example: all inclusions and sub-inclusions of other files would also work?

[YvesK]

Wonderful.

I think transclusion would be a very very good companion to org-roam. Atomic notes assembled together to form a complex thought path, it’s the goal of roam/zettlekast, isn’t it?

Using the special ortc custom link is already a great functionality, but an idea immediately pops to my mind: would’nt it be even better if it worked with the simple standard org-mode stuff (like org-roam do). What about using a :transclude: drawer property and standard org-mode links. These links would work normally when the transclude-mode is not activated.

Of course, even as I’m very very far from being an old elisp guru, I can guess that it would be quite harder to code. Would it be worth? Would it be interesting to keep that goal for later?

[nv-discourse]

@YvesK

Using the special ortc custom link is already a great functionality, but an idea immediately pops to my mind: would’nt it be even better if it worked with the simple standard org-mode stuff (like org-roam do). What about using a :transclude: drawer property and standard org-mode links. These links would work normally when the transclude-mode is not activated.

I would encourage development in this direction. Not necessarily implementing via a property, but implementing in a general way so that one can take advantage of the transclusion without using org-roam. It should be its own library and needn’t be “ortc” when it’s just general transclusion.

[mshevchuk]

Very cool work! Hope a package will appear on GitHub soon!

[timquelch]

It should be its own library and needn’t be “ortc” when it’s just general transclusion.

Agreed. There isn’t really anything that is specific to org-roam. It would be great to be able to use this in any org file

[mancub]

Great work @nobiot. This would be very helpful for academics as well, especially ones like me who use multiple papers to build an argument. I hope this makes it into the main org-roam package soon.

PS Allow me to buy you a beer to support your efforts.

[dangirsh]

Very excited for this!

In Roam, block embeddings are bi-directional. That is, there’s a way to find all places a given block has been transcluded. This can be very useful for e.g. finding all arguments that reference some source quote. If this feature is desired, we may favor an implementation with custom link types (like in the prototype) over a more generic transclusion mechanism.

[nobiot]

Everyone.
Thank you for your encouraging comments, thoughts, and suggestions.

I will share the source code soon, and respond to the comments. Just a little hectic this week at work and beyond (well, I knew this would be the case, and this is why I wanted to share the demo when I did).

It will be clear once you see the source code. Currently I have only about 300 lines of code (including comments!) to get the demo working. It’s my very humble attempt as a non-developer/programmer.

What I am saying is, I suppose, it’s an indication of both potential encouragement (300 lines of code, cobbled together by a non-programmer, and you get here), and potential disappointment (“what?! that’s it? That’s not going to cut it!”).

In any case, I will get back to this community. Stay tuned

[JeffBrown]

This is brilliant!

It makes me think about recursion, and homogeneity, and how to avoid getting lost.

It’s not obvious to me that using two separate kinds of links, transcluding and normal, is optimal. It seems more natural for a link to another file to appear as a heading, just like any other heading, except with some kind of indication that it’s not in the same file.

In that case the user would open it just like any other heading. They could moreover recurse, opening transcluded files within transcluded files, and the indentation would help them understand where they are.

Here’s an example. Suppose we have three files:

#+TITLE: a    #+TITLE: b          #+TITLE: c
* a1          * b1                * c1
** a11        ** b11              ** c11
** a12        ** <link to 'a'>    ** <link to 'b'>
* a2          * b2                * c2

We can then unfold the <link to ‘b’> header in file ‘c’ to reveal:

#+TITLE: c
* c1
** c11
** #+TITLE: b
   * b1
   ** b11
   ** <link to 'a'>
   * b2
* c2

and then recurse:

#+TITLE: c
* c1
** c11
** #+TITLE: b
   * b1
   ** b11
   ** #+TITLE: a
      * a1
      ** a11
      ** a12
      * a2
   * b2
* c2

That whitespace to the left of the transcluded files would ideally, I think, be unavailable – i.e. the cursor can’t go there.

If a lot of headers were unfolded at this point, it could be hard to see where you are. In fact this is a problem with org mode in general – the path to “home” might be well off screen. It would be cool if there was a command fcalled something like org-where-am-i that would display the header path to where you are. For instance, from header a2 in the last image, running org-where-am-i would display the following:

#+TITLE: c
* c1
** #+TITLE: b
   * b1
   ** b11
   ** #+TITLE: a
      * a2

[YvesK]

@JeffBrown

If a lot of headers were unfolded at this point, it could be hard to see where you are. In fact this is a problem with org mode in general – the path to “home” might be well off screen. It would be cool if there was a command fcalled something like org-where-am-i that would display the header path to where you are.

There is a package that does something like this:

org-sticky-header is an available package.
Status: Available from melpa – Install
Archive: melpa
Version: 20191117.549
Commit: 1053ebdeb3bd14fc8d4538643532efb86d18b73c
Summary: Show off-screen Org heading at top of window
Requires: emacs-24.4, org-8.3.5
Homepage: GitHub - alphapapa/org-sticky-header: Show off-screen Org heading at top of window

This package displays in the header-line the Org heading for the
node that’s at the top of the window. This way, if the heading for
the text at the top of the window is beyond the top of the window,
you don’t forget which heading the text belongs to.
…

[JeffBrown]

Nice!

It looks like it only displays the nearest header, though, not the entire chain of headers leading to where you are. Still, probably not hard to modify the package you found to show the whole path.

[jethro]

@nobiot not sure if you’re on Reddit but Prot has a comment for you regarding the faces for the transcluded section:

https://www.reddit.com/r/emacs/comments/j6k2j8/orgmode_transclusionblockreference_prototype/

The other comments have already been reflected in this thread.

[nobiot]

I’m just about to put it up on GitHub. I’m not really on Reddit, but I agree with Prot’s comment re face – it’s his theme I am using with slight adjustment Thanks for bringing it here for me to read.