Org-transclusion -- now on ELPA!

GitHub up!

1 Like

Thank you for publishing it, @nobiot.

Iā€™ll have a look as soon as I can.

1 Like

Big ask for moving forward

I have a big ask of the community, if you think the idea is worth pursuing.

Because transclusion modifies our notes, outcome of our hard intellectual work, I think the whole thing depends on the way it modifies and saves multiple buffers in the background. If this is not good enough, then the package should stay as a personal tool.

Please help validate the fundamental idea / architecture around modifying and saving the transclusion sources.

If this is solid enough to give confidence to academics, professional writers, and other people relying on writing (me included here), then I think this project has a chance of flying out of our incubation chamber.

I took the buffer-modifying logic from org-edit-src (-save and -exit more specifically). It looks like it works for my personal use. But it may not be the best of ideas for the type of applications we are dealing with.

I made the save-buffer part so that Emacā€™s backup facility will be turned on for the local-buffer when you have it turned off. This backup does not seem to be assuring enough; however, making multiple versions of backup files might clutter your notes repository.

Iā€™d love to have your Ideas, issues, and PRs.

Responses

Recursive (or turtles of) transclusions

I believe this is about recursively transcluding transclusions.

At the moment, recursion is specifically prevented (that is, if you have an ortc: link within a transcluded region, the code skips it to avoid recursion).

Two reasons:

  1. My brain cannot handle this recursion.
    The program works procedurally from top to bottom; so it might be OK to remove this limit. I am still not super comfortable with the way current implementation modifies and saves multiple notes at the same time in the background. Can I really trust my minions doing the right things?

  2. Itā€™s not part of my writing workflow
    Since my brain really cannot wrap itself around the concept of recursion wellā€¦ I just donā€™t feel like doing it in my notes. This can be just my personal limitation and style. For my writing workflow, I think I would prefer to have backlinks for my notes; use transclusion when I do long-form material.

Maybe recursion can be turned on in the future as an option (when we can be more confident with the modifying and saving multiple buffers). Butā€¦ the efforts for testing can be intimidating, so practially not sure if I would like to try recursion.

Recursion and homogeneity; or merging different heading levels

From the example, I took it to be about this question:

ā€œHow do we merge different heading levels into a note?ā€

Like:

** Heading 2

[ortc:note-2.org]


At the moment, there is no intelligent way of handling this in my prototype.
Itā€™s simle copy & paste.

Soā€¦ if note-2.org above starts with heading level 1, it will be simply like this:

** Headig 2

* Coming from note-2.org
Content of heading level 1 of note-2.org

** Heading level 2 in note-2.org

Iā€™m OK with this simplicity at the moment.

Special link type, or bolt-on to the normal Org link types

Yes, as you can see now in the source, the current source does not rely on Org-roam; only some Org Mode functions. I think I slipped my tongue in the video show & tell mentioning Org-roam. So, it works with normal Org files now.

Just to add an side note, transclusion sources can be any (text) files, as tranclusion just copy-and-pate the content, and sticks an overlay on top of it to keep track of its provenance.

At this stage, I would prefer to have a special link type (it does not have to be ortc; John Kitchin seems to have used ā€œtransclusionā€).

Three reasons:

  1. My style. As I am writing, I want to explicitly make a link transclusion by myself.

  2. Technical. I donā€™t know an easy way to change the linkā€™s behaviour in other way

  3. Technical. At the moment, the transclusion link should be at the beginning of a line
    Iā€™m not checking this in the code. But you try to add transclusion on a link in the middle of a line, and you will seeā€¦ The behaviour currently is confusing. So transclusion (as it is) should be defined distinctly from other link types, with latter of which users freely insert anywhere in a line of text.

But this disappearing and reappearing of transclusion links is meant to solve a problem. In my current design, you cannot separate the link from the transclusion overlay. One of the best ways to prevent separation is, it so occurred to me, to make the link disappear (so there would be no way for the user to manually separate it from the overlay). The link is ā€œtucked inā€ to the overlay as a property. When you deactivate transclusion, the link will be back in the position it is supposed to be (the problem: it should be at the beginnig of line).

This has not been a big limitation, at least for me, so far. I want to transclude a whole section / element / buffer ā€“ it does not start in the middle of a line.

Bi-directionality

I agree. I want it, too. I have this in my ā€œList of things needed to be doneā€ in the GitHub README. I am thinking of somehow adding a visual indication in the gutterā€¦ I need to see how this helps uses when there are multiple transcluding notes.

2 Likes

All this social stuff is foreign to me, especially Reddit. I think I replied to you and in the ā€œthreadā€. Not sure how all these things work. :expressionless: hmmm.

Just throwing out an idea for potential improvement. This Stack Exchange post has source code for a cloning text region with a modified version of text-clone-create. This might be an alternative direction for the ā€œmodify-n-syncā€ engine of transclusion.

2 Likes

@nobiot Itā€™s amazingly useable implementation. Coming from Roam Research user experience, I wish to also have the capability to refer interactively a text block (a paragraph) freely by the incremental substring of the paragraph, instead of the current uuid-link in the property of the to be transcluded. This will make transclusion much easier to have, less friction to oneā€™s workflow. This is how Roam Research achieves.

2 Likes

Created a branch named feat/text-clone. Code is horrible at the moment, but the initial result is positive.

We may be onto something :slight_smile:

Thank you :smiley:

Yes to friction-less writer experience (let me call it as user experience aimed for a specific group in mind).

Let me break down your feedback into two components:

  1. Ability to find a paragraph (or a block) to reference by incremental substring search
  2. Ability to reference a paragraph (or a block = unit of text we can reference)

For point 1 regarding incremental search, I think we will be relying on Emacsā€™s completion and completion-at-point framework(s).

As an example, I would love to somehow leverage what Org-roam has done for completion-at-point like this GIF below. It is an Org-roam example to look for a note first, and then incrementally look for an IDā€™ed headline after an asterisk.

I would love to get communityā€™s advice on what else the standard Org mode offers for completion-at-point ā€“ I do not know much about it.

For point 2, itā€™s a question of what elements in the text we can reference.
Initially what I had in mind are Org mode elements that can be named or IDā€™ed, following the original John Kitchinā€™s idea ā€“ such as tables, source or quote blocks, IDā€™ed headlines, etc.

Butā€¦ Emacs never fails to surprise us, does it?.. It can work well with paragraphs. We might be able to find a way to reliably reference a specific paragraph ā€“ I have not tried it yet, but Iā€™m cautiously optimistic.

The current implementation uses Emacsā€™s ā€œmarkerā€ to reference an IDā€™ed headline. It does not have to be IDā€™ed really. ā€¦Soā€¦ Maybe, maybe, we can find a technical way to reference any paragraph in the textā€¦ Then the question becomes, how can we make it easy for writers to search the one they want incrementally while writing a body of text (now itā€™s point 1ā€¦).

Just for completeness, Iā€™ll mention that Freex contains another implementation of transclusion in Emacs. Maybe itā€™s helpful.

Hey, @dangirsh, thank you for this. Yes, right, I havenā€™t looked at emacs-freex in detail, because it relies on Python. I donā€™t know what elements of the whole app uses Python.

Alsoā€¦ John Kitchin, whose idea my implementation is based on, mentions it as an inspiration ā€“ I thought he already took the ā€œinspirationā€ for reasonā€¦ All speculations on my part, without looking into detail of emacs-freex.

As a non-professional coder (I guess), I am challenged to understand its multiple files, but I might have a lookā€¦

Actually, @dangirsh, can you point me in the specific location in the source code where it does ā€œtransclusionā€, by any chance?

Itā€™s probably this file: https://github.com/gregdetre/emacs-freex/blob/master/freex-embed.el
Thanks again for your sharing. At a very quick glance, it sorta does a very similar thing with overlay that my implementation does. But the details need longer analysis ā€“ 1.3 k lines of code for this file alone :joy:

I will keep it as a source of wisdom from the past (like an ancient artifact of knowledge from our progenitor mysteriously extinct, but passed on through generations) :wink:

@dangirsh, surprisingly to myself, I can read the code of emacs-freex!

It does something clever around before writing the current buffer to file.
This might be potentially useful for org-transclusion (!)

I will definitely explore this avenue in next iterations.
Thank you again.

1 Like

A new video is up. I talk about real-time sync (working) with two issues I recognise at the momentā€¦

@nobiot @dangirsh

Hi there,

Itā€™s a real delight to see what youā€™re building here. Itā€™s exactly what I had dreamed of when working on Freex, and youā€™re taking it much further.

Iā€™m glad to hear that youā€™re able to make sense of the code, and I hope thereā€™s something useful you can draw on. If thereā€™s anything I can help with, Iā€™ll do my best - though Per Sederberg wrote a lot of the embedding/transclusion code, and it has been a while :slight_smile:

Thanks for introducing the Org-roam and Freex communities, @dangirsh

Yours,
Greg

P.S. I am working on a followup project, but outside Emacs this time. Drop me a line on greg@spideryarn.com if youā€™d like to hear more.

Hi @gdetre

!!! Imagine a kid busking on the street, playing a guitar singing his own song. One day, Thom Yorke from Radiohead (sorry, I guess my generation) appears in front of you, pads on your shoulder, and says ā€œyouā€™re doing goodā€ ā€¦

You are absolutely a hero in my book. Thank you for the encouraging and kind words.

Definitely interested in what your current project is.

Thanks again!

1 Like

Hi all,
Demo #3 video is up for your view.

I will appreciate your comments, questions, ideas and feedback of all types.
DM is also cool if you donā€™t want the whole world to see your commentaries.

@nv-discourse, @YvesK, @timquelch, I have taken your idea and implemented a support for a standard Org Mode link for [[file:path/to/file.org::*heading]]. It looks good and makes sense. Especially, it nicely integrates with the Org Mode UX we are familiar with (C-c C-l etc.) :smiley:

3 Likes

Hello Nobiot,

Thank you, what you do looks very interesting and very promising.

I donā€™t have much time until next week but Iā€™ll try to give you some return now, fast.

I tried to look at your code, but Iā€™m afraid this is a bit out of my league, I can manage to make some simple functions but not much more yet in elisp.

About the problem of multiple transclusions of the same data, paragraph, org heading/entry, file, not updating real time, the question Iā€™m asking is, what would the the purpose? I mean it is a technical challenge to solve, do do we have real use cases where it would useful?

My personal use case of transclusion is to have simple atomic notes about one topic each (the zettle/roam principles) that get assebled together into a single document that woud be the notes you took from a conference a course or the other way around you yourself assdemble your notes into a conference/courseā€¦ Starting or ending into a publication, html, printed or whatever, do we really need the same data several time?

Now there is also the problem of transclusion within transclusion and then if that would be possible, the problem of infinite transclusion loop.

Also, there is the problem of headings levels, should the transcluded item be a child of the current heading? If a higher level heading is transcluded above a lower level heading in the mother document, it becomes its parent heading, would that make any sense? Maybe one place to look for would be org-capture entry, I think it does that, make the captured data child of the target entry.

Even if it is not the idea of linux/emacs of powerfull tools that do one thing well and everything about it, my approach would be to keep it simple to make the job we want it to do, now I am no emacs guru, I guess thatā€™s why.

Yves K

Oh and another idea/functionality that would be, I think, very useful: the reverse of what you called detach: that would be sopmething like ā€œregion refiled to already existing/new file and becoming a transclusion link in hereā€ (big notes file becoming an index of small notes).

But I really must go now. Good day to you.

1 Like

I just want to share a small idea I had a while ago: would it be possible to make use of or build upon the org-babel-tangle and org-babel-detangle functionalities for transclusion? One advantage might be that these functions have a long history in Emacs (and literate programming), so they have apparently stood the test of time and should be quite robust.

Anyways, thanks a lot for your great work, Iā€™ve been hoping to be able to use transclusion for quite a while, so Iā€™m really looking forward to using your package!

1 Like

There is an interesting suggestion on GitHub. I will look into it. I think your idea about org-capture is going in the same direction.

I also agree with the ā€œdo one thing wellā€ approach. Iā€™m thinking of an alternative way now.