Org-transclusion -- now on ELPA!

[nobiot]

GitHub up!

nobiot/org-transclusion

[zaeph]

Thank you for publishing it, @nobiot.

I’ll have a look as soon as I can.

[nobiot]

Big ask for moving forward

I have a big ask of the community, if you think the idea is worth pursuing.

Because transclusion modifies our notes, outcome of our hard intellectual work, I think the whole thing depends on the way it modifies and saves multiple buffers in the background. If this is not good enough, then the package should stay as a personal tool.

Please help validate the fundamental idea / architecture around modifying and saving the transclusion sources.

If this is solid enough to give confidence to academics, professional writers, and other people relying on writing (me included here), then I think this project has a chance of flying out of our incubation chamber.

I took the buffer-modifying logic from org-edit-src (-save and -exit more specifically). It looks like it works for my personal use. But it may not be the best of ideas for the type of applications we are dealing with.

I made the save-buffer part so that Emac’s backup facility will be turned on for the local-buffer when you have it turned off. This backup does not seem to be assuring enough; however, making multiple versions of backup files might clutter your notes repository.

I’d love to have your Ideas, issues, and PRs.

Responses

Recursive (or turtles of) transclusions

@forkrul

Do you plan to have this to work for “turtles all the way down”? Example: all inclusions and sub-inclusions of other files would also work?

I believe this is about recursively transcluding transclusions.

At the moment, recursion is specifically prevented (that is, if you have an ortc: link within a transcluded region, the code skips it to avoid recursion).

Two reasons:

1. My brain cannot handle this recursion.
   The program works procedurally from top to bottom; so it might be OK to remove this limit. I am still not super comfortable with the way current implementation modifies and saves multiple notes at the same time in the background. Can I really trust my minions doing the right things?

2. It’s not part of my writing workflow
   Since my brain really cannot wrap itself around the concept of recursion well… I just don’t feel like doing it in my notes. This can be just my personal limitation and style. For my writing workflow, I think I would prefer to have backlinks for my notes; use transclusion when I do long-form material.

Maybe recursion can be turned on in the future as an option (when we can be more confident with the modifying and saving multiple buffers). But… the efforts for testing can be intimidating, so practially not sure if I would like to try recursion.

Recursion and homogeneity; or merging different heading levels

@JeffBrown

about recursion, and homogeneity,

From the example, I took it to be about this question:

“How do we merge different heading levels into a note?”

Like:

** Heading 2

[ortc:note-2.org]

At the moment, there is no intelligent way of handling this in my prototype.
It’s simle copy & paste.

So… if note-2.org above starts with heading level 1, it will be simply like this:

** Headig 2

* Coming from note-2.org
Content of heading level 1 of note-2.org

** Heading level 2 in note-2.org

I’m OK with this simplicity at the moment.

Special link type, or bolt-on to the normal Org link types

@YvesK

would’nt it be even better if it worked with the simple standard org-mode stuff

@nv-discourse

Not necessarily implementing via a property, but implementing in a general way so that one can take advantage of the transclusion without using org-roam

@timquelch

Agreed. There isn’t really anything that is specific to org-roam. It would be great to be able to use this in any org file

Yes, as you can see now in the source, the current source does not rely on Org-roam; only some Org Mode functions. I think I slipped my tongue in the video show & tell mentioning Org-roam. So, it works with normal Org files now.

Just to add an side note, transclusion sources can be any (text) files, as tranclusion just copy-and-pate the content, and sticks an overlay on top of it to keep track of its provenance.

At this stage, I would prefer to have a special link type (it does not have to be ortc; John Kitchin seems to have used “transclusion”).

Three reasons:

1. My style. As I am writing, I want to explicitly make a link transclusion by myself.

2. Technical. I don’t know an easy way to change the link’s behaviour in other way

3. Technical. At the moment, the transclusion link should be at the beginning of a line
   I’m not checking this in the code. But you try to add transclusion on a link in the middle of a line, and you will see… The behaviour currently is confusing. So transclusion (as it is) should be defined distinctly from other link types, with latter of which users freely insert anywhere in a line of text.

But this disappearing and reappearing of transclusion links is meant to solve a problem. In my current design, you cannot separate the link from the transclusion overlay. One of the best ways to prevent separation is, it so occurred to me, to make the link disappear (so there would be no way for the user to manually separate it from the overlay). The link is “tucked in” to the overlay as a property. When you deactivate transclusion, the link will be back in the position it is supposed to be (the problem: it should be at the beginnig of line).

This has not been a big limitation, at least for me, so far. I want to transclude a whole section / element / buffer – it does not start in the middle of a line.

Bi-directionality

@dangirsh

block embeddings are bi-directional

I agree. I want it, too. I have this in my “List of things needed to be done” in the GitHub README. I am thinking of somehow adding a visual indication in the gutter… I need to see how this helps uses when there are multiple transcluding notes.

[nobiot]

All this social stuff is foreign to me, especially Reddit. I think I replied to you and in the “thread”. Not sure how all these things work. hmmm.

[nobiot]

@nobiot

I took the buffer-modifying logic from org-edit-src ( -save and -exit more specifically). It looks like it works for my personal use. But it may not be the best of ideas for the type of applications we are dealing with.

Just throwing out an idea for potential improvement. This Stack Exchange post has source code for a cloning text region with a modified version of text-clone-create. This might be an alternative direction for the “modify-n-sync” engine of transclusion.

Is there an emacs package which can mirror a region?

[yubrshen]

@nobiot It’s amazingly useable implementation. Coming from Roam Research user experience, I wish to also have the capability to refer interactively a text block (a paragraph) freely by the incremental substring of the paragraph, instead of the current uuid-link in the property of the to be transcluded. This will make transclusion much easier to have, less friction to one’s workflow. This is how Roam Research achieves.

[nobiot]

@nobiot

cloning text region with a modified version of text-clone-create . This might be an alternative direction for the “modify-n-sync” engine of transclusion.

Created a branch named feat/text-clone. Code is horrible at the moment, but the initial result is positive.

We may be onto something

[nobiot]

Thank you

Yes to friction-less writer experience (let me call it as user experience aimed for a specific group in mind).

Let me break down your feedback into two components:

1. Ability to find a paragraph (or a block) to reference by incremental substring search
2. Ability to reference a paragraph (or a block = unit of text we can reference)

For point 1 regarding incremental search, I think we will be relying on Emacs’s completion and completion-at-point framework(s).

As an example, I would love to somehow leverage what Org-roam has done for completion-at-point like this GIF below. It is an Org-roam example to look for a note first, and then incrementally look for an ID’ed headline after an asterisk.

I would love to get community’s advice on what else the standard Org mode offers for completion-at-point – I do not know much about it.

For point 2, it’s a question of what elements in the text we can reference.
Initially what I had in mind are Org mode elements that can be named or ID’ed, following the original John Kitchin’s idea – such as tables, source or quote blocks, ID’ed headlines, etc.

But… Emacs never fails to surprise us, does it?.. It can work well with paragraphs. We might be able to find a way to reliably reference a specific paragraph – I have not tried it yet, but I’m cautiously optimistic.

The current implementation uses Emacs’s “marker” to reference an ID’ed headline. It does not have to be ID’ed really. …So… Maybe, maybe, we can find a technical way to reference any paragraph in the text… Then the question becomes, how can we make it easy for writers to search the one they want incrementally while writing a body of text (now it’s point 1…).

[dangirsh]

Just for completeness, I’ll mention that Freex contains another implementation of transclusion in Emacs. Maybe it’s helpful.

gregdetre/emacs-freex

[nobiot]

Hey, @dangirsh, thank you for this. Yes, right, I haven’t looked at emacs-freex in detail, because it relies on Python. I don’t know what elements of the whole app uses Python.

Also… John Kitchin, whose idea my implementation is based on, mentions it as an inspiration – I thought he already took the “inspiration” for reason… All speculations on my part, without looking into detail of emacs-freex.

As a non-professional coder (I guess), I am challenged to understand its multiple files, but I might have a look…

[nobiot]

Actually, @dangirsh, can you point me in the specific location in the source code where it does “transclusion”, by any chance?

[nobiot]

It’s probably this file: https://github.com/gregdetre/emacs-freex/blob/master/freex-embed.el
Thanks again for your sharing. At a very quick glance, it sorta does a very similar thing with overlay that my implementation does. But the details need longer analysis – 1.3 k lines of code for this file alone

I will keep it as a source of wisdom from the past (like an ancient artifact of knowledge from our progenitor mysteriously extinct, but passed on through generations)

[nobiot]

@dangirsh, surprisingly to myself, I can read the code of emacs-freex!

It does something clever around before writing the current buffer to file.
This might be potentially useful for org-transclusion (!)

I will definitely explore this avenue in next iterations.
Thank you again.

[nobiot]

A new video is up. I talk about real-time sync (working) with two issues I recognise at the moment…

[gdetre]

@nobiot @dangirsh

Hi there,

It’s a real delight to see what you’re building here. It’s exactly what I had dreamed of when working on Freex, and you’re taking it much further.

I’m glad to hear that you’re able to make sense of the code, and I hope there’s something useful you can draw on. If there’s anything I can help with, I’ll do my best - though Per Sederberg wrote a lot of the embedding/transclusion code, and it has been a while

Thanks for introducing the Org-roam and Freex communities, @dangirsh

Yours,
Greg

P.S. I am working on a followup project, but outside Emacs this time. Drop me a line on greg@spideryarn.com if you’d like to hear more.

[nobiot]

Hi @gdetre

@gdetre

It’s a real delight to see what you’re building here. It’s exactly what I had dreamed of when working on Freex, and you’re taking it much further.

!!! Imagine a kid busking on the street, playing a guitar singing his own song. One day, Thom Yorke from Radiohead (sorry, I guess my generation) appears in front of you, pads on your shoulder, and says “you’re doing good” …

You are absolutely a hero in my book. Thank you for the encouraging and kind words.

Definitely interested in what your current project is.

Thanks again!

[nobiot]

Hi all,
Demo #3 video is up for your view.

I will appreciate your comments, questions, ideas and feedback of all types.
DM is also cool if you don’t want the whole world to see your commentaries.

@nv-discourse, @YvesK, @timquelch, I have taken your idea and implemented a support for a standard Org Mode link for [[file:path/to/file.org::*heading]]. It looks good and makes sense. Especially, it nicely integrates with the Org Mode UX we are familiar with (C-c C-l etc.)

[YvesK]

Hello Nobiot,

Thank you, what you do looks very interesting and very promising.

I don’t have much time until next week but I’ll try to give you some return now, fast.

I tried to look at your code, but I’m afraid this is a bit out of my league, I can manage to make some simple functions but not much more yet in elisp.

About the problem of multiple transclusions of the same data, paragraph, org heading/entry, file, not updating real time, the question I’m asking is, what would the the purpose? I mean it is a technical challenge to solve, do do we have real use cases where it would useful?

My personal use case of transclusion is to have simple atomic notes about one topic each (the zettle/roam principles) that get assebled together into a single document that woud be the notes you took from a conference a course or the other way around you yourself assdemble your notes into a conference/course… Starting or ending into a publication, html, printed or whatever, do we really need the same data several time?

Now there is also the problem of transclusion within transclusion and then if that would be possible, the problem of infinite transclusion loop.

Also, there is the problem of headings levels, should the transcluded item be a child of the current heading? If a higher level heading is transcluded above a lower level heading in the mother document, it becomes its parent heading, would that make any sense? Maybe one place to look for would be org-capture entry, I think it does that, make the captured data child of the target entry.

Even if it is not the idea of linux/emacs of powerfull tools that do one thing well and everything about it, my approach would be to keep it simple to make the job we want it to do, now I am no emacs guru, I guess that’s why.

Yves K

Oh and another idea/functionality that would be, I think, very useful: the reverse of what you called detach: that would be sopmething like “region refiled to already existing/new file and becoming a transclusion link in here” (big notes file becoming an index of small notes).

But I really must go now. Good day to you.

[wuqui]

I just want to share a small idea I had a while ago: would it be possible to make use of or build upon the org-babel-tangle and org-babel-detangle functionalities for transclusion? One advantage might be that these functions have a long history in Emacs (and literate programming), so they have apparently stood the test of time and should be quite robust.

Anyways, thanks a lot for your great work, I’ve been hoping to be able to use transclusion for quite a while, so I’m really looking forward to using your package!

[nobiot]

@YvesK

Also, there is the problem of headings levels, should the transcluded item be a child of the current heading? If a higher level heading is transcluded above a lower level heading in the mother document, it becomes its parent heading, would that make any sense? Maybe one place to look for would be org-capture entry, I think it does that, make the captured data child of the target entry.

There is an interesting suggestion on GitHub. I will look into it. I think your idea about org-capture is going in the same direction.

I also agree with the “do one thing well” approach. I’m thinking of an alternative way now.