Org-transclusion -- now on ELPA!

@YvesK, by the way, the latest main branch has the org-transclude-paste-subtree in the source so no need for a separate file.
(in fact i should remove it to avoid confusion)

1 Like

Just to be sure… The first and main part of the demo is actually showing [[file:path/to/file.org::*heading][description]].

What I didn’t show and I know is not working properly at the moment is a link to an entire Org file (without the :: additional search part).

I will try to improve my demo in each iteration, so really appreciate your comments :slight_smile:

Regarding your point 3:

Having some links being transcluded and other not, I think too, that it is a good feature. [etc. etc.]

There has been very good discussion happening with Alphapapa (who, I learned only recently, is an author of org-super-agenda, and other cool packages) and others on Reddit. I am experimenting with some of the suggestions made there. Stay tuned :wink:

Very true, sorry I guess I didn’t pay enough attention (dazzled by all the other links possibilities, paragraphs, tables etc.) …

Also I tested org-refile on a transcluded entry, to see how it would break, and yes it does:

  • the entry stays (transcluded) in the file
  • a copy is created in the refile target file
  • this copy is read only

Probably it is not yet time to hunt problems for integration with all other org-mode functions/packages, though it seems a huge task to me.

1 Like

No worries.
To keep track of it, I create an issue on my GitHub repo.

It’s good that you are trying out Org-transclusion and find issues.
I have to be pragmatic about compatibility with Org functions, though, and have to admit that I cannot promise that I can fix all the problem.

What would be the purpose of refilling the transcluded content, rather than the original?
Perhaps it should be:

  1. Detach the transclusion to make the copied content ā€œnativeā€ to the buffer
  2. Then refile (detach is not currrently working, so it would have to be a future thing).

I will see what I can do with org-refile…

Thanks!

  1. The behavior I would expect would be to move the data, in that case refile the link, and, in the case the target file would get into org-transclusion-mode that link would naturally be transcluded again. Now that is not a showstopping problem on my side. I tried org-refile because it looked hard to integrate with transclusion…

  2. On the other hand, linking to full file is more important for the roam/zettlekast method since simple atomic notes are more likely to be simple paragraphs without any headings… A method to achieve file transclusion could be, say in a temp buffer:
    a. put a top level 1 heading with either the TITLE property or the filename
    b. translate other file # properties into properties of the level 1 heading
    c. increment all other heading levels within the file which become subheaders of the TITLE heading
    d. Now you can paste the whole subtree
    e. Now how do you make the logic of org-edit-src work? I don’t know. But we can as well edit the original file, no?

  3. Also I noticed a strange inconsistent behavior with heading levels
    the same transcluded link would get the heading level of the heading above
    unless there is a subheading under in which case it would get this last one.
    Okay, an example, because of the presence of heading 2, ā€œtranscluded stuffā€ gets a different heading level than if there was no heading 2:

* heading 1
* transcluded stuff
  lorem ipsum
* heading 1
**** transcluded stuff
     lorem ipsum
**** heading 2
1 Like

I this is how org-paste-subtree function works. It looks at a headline before and after, and uses the lower level of the two headlines.

In your examle, transcluded stuff looks at headling 1 (before) and heading 2 (after), and uses the headline level of heading 2 (lower).

1 Like

what is the default link a user should expacted to use?

I notice there are some of options here, but when it comes to
really scenario, the user has to determine the primary link type.

File link with ::*heading
File link with ::#custom-id
File link with ::name for blocks (e.g. blocked quotations) and tables
File link with ::dedicated-target; this is intended for linking to a paragraph. See below.
  ID link with   id:uuid
File link for non-org files 
      (tested with .txt and .md); for these, the whole buffer gets transcluded

For me, I tend to link file type of link, i.e. [[file:20201106144851-testing_example.org]],
which I believe is the original intention using org-roam.

Do you have any suggestions on which type of link a user should use by default?

Following the reddit thread, I want to comment on the (possible) design architecture of org-transclusion:

If I understand correctly, by transcluding links via indirect buffers, org-transclusion effectively delegates all the details of representation to emacs, instead of having to provide wrappers (as simple as they may be) if it were to create a new type of link, such as the otc: that was used at the beginning.

I came up with this feature of Notion embeds. Since we already have an emacs web browser (eww), we can transclude/embbed any kind of content that can be parsed by eww.

Similarly, you can also transclude everything that emacs knows how to open, such as pdf files, where for example you can transclude a certain page with a diagram on it.

I think this is worth mentioning.

1 Like

After wresting with org-transclusion, I find that the issue lies in

(defun org-transclusion--get-org-content-from-link (orgfn link &rest _arg)

Especially, let* ((el (org-element-context)), which returns the
actual content that is being transcluded.

I have tried to guide this org-element-context to include all the
content of a note, but I did not make it now.

If I manually set beg and end, I can let transcluded buffer to show
the content, but it is not able to change the content of its original
note.

(setq beg (org-element-property :begin el))
(setq end (org-element-property :end el))

After this, I reflect my initial thoughts about using
org-transclusion, especially the feature request I wrote in this issue.

If I want to insist on using file-based org-roam work flow, I would
like to do this:

For the notes I want to be transcluded elsewhere, I would add a
headline in that file with a name like this:

* TO BE TRANSCLUDED

The mature content.

I will put everything else out of this headline in that note. This
will:

  1. achieve partially transclusion
  2. transclude the previous content, which is in the headline
  3. encourage me to craft on the original note

@nobiot

I believe I’ve walked around the issue that I wrote to your
repository. I am really sorry about writing so much. Feel free to
close the issues.

Thanks for your amazing work.

1 Like

Thank you so much for spending time and efforts to using (still very early version of) the package, and logging the issues. Much appreciated (please don’t be sorry; your comments are helpful for me to see what/where others focus on).

I myself want links to entire files to work properly. Hopefully I can do the next show & tell soon (perhaps another weekend or two away)

@YvesK, it may not be as you suggest it would be, but I am working on transcluding a whole .org file :slight_smile:

Yes, I am flirting with that idea, too :). Not the main priority at the moment, as you have some excellent packages to copy the content to your file (some have been discussed in this forum, and Slacks, if I remember correctly). Let me have a look at Notion embeds, too.

Just a peak preview of the current state (available in feat/keyword branch on GH).
2020-11-08T130800-transclusion-v0.0.5

2 Likes

v0.0.5 is merged on GitHub and YT video uploaded.

@zaeph
This one is the latest – I’d appreciate your feedback/comments if you can have some time :slight_smile:

2 Likes

I’m scheduling a review for the second half of next-week.

Thank you for your work! :slight_smile:

2 Likes

Hello, I’m new to the forum and discovering this transclusion approach, and it looks great!

Regarding whole file transclusion, is it possible to make it work with header-less files? A lot of my file in my slip box look like this:

#+title: a title

- tags :: the_tags

summary of the file

- a first note about something
- another note

Is it possible to transclude such files?

1 Like

With the current version, no (i think; not really tested that). I think the program considers the content of such org files as part of the ā€œfirst-sectionā€ (everything before the first headline). I will review this in the upcoming versions, but for now you would need to have a headline to signal the start of the content to transclude.

Thanks for the encouragement :slight_smile:

@brab, I just tried this; good call. Thank you for asking.
It’s actually worse than ā€œit’s not workingā€. It gives you an error, and the link disappears (!).

I’m tracking this in issue #25 on GitHub.

Thanks again!

Added a new feature to transclude ā€œheadlinelessā€ Org files – you can toggle this customizing to include the ā€œfirst section (before the first headline)ā€ in transclusion.

It is toggled via cusotmizing variable org-transclusion-include-first-section. Its default value is nil. Set it to t to transclude the first section.

I only lightly tested it. Let me know if you (and others) find major problems… I might still need to revert this feature (my first glance was Okay).

I still need to work on the bug your question has revealed :wink:
Thanks again!

Well, thank you! I think this is going to be an amazing feature (the whole transclusion one).

1 Like

Hi again @Nobiot!

First of all, great job again! It’s very exciting to see the progress on this!

I wanted to comment on the ā€œbefore first headingā€ design decision. :slight_smile: Does the findings that you discussed with @brab mean that content before the first heading can be included also for files that have subsequent headings?

I’m asking since I’ve personally tried to push Org mode into considering content before the first heading in a similar manner as all the other content, I.e. as a part of the outline. Since Org 9.4 the outline level before the first heading should be regarded as level 0, and my (long term) hope is that level 0 some day will support everything that is supported within level-1 - level-n. That’s not the case yet, since it’s still difficult to assign a todo-keyword, tag, or schedule on outline-level 0. But conceptually in my opinion it should be possible. I’m digressing from the subject though. But thought to give some context to my question above. And to give reason to my wish for content before first heading to also be considered for transclusion. At least long term!

Best wishes

1 Like

Yes. As far as my light testing goes, the first section + headlines combo seems to work (the fear I have is duplicating the same content in one transclusion).

Thank you for the background and great encouragement :slight_smile:

1 Like

I just tried 0.0.5 and I find (setq org-transclusion-include-first-section nil) is a better idea to me. I can now just put the extra info before the first headline. Thanks for your effort.

PS: I will test the id issue later on and will report to GitHub issue.

1 Like