Package dedicated to workflows publishing digital garden on web

[reyman]

Hi roam-guys,

I first start a discussion based on my org-roam publishing needs on how-to section of this discourse (Advice on Roam workflow, how do you separate private and public part of your garden?), but after some reflexions and one enhancement issue open on github (https://github.com/org-roam/org-roam/issues/911) i think we could generalize the idea, starting from simple workflow to complex workflow for publishing digital garden on the web.

Lot of discussions about org / org-roam are note-taking-workflows or note-publishing-workflows discussions. As you know, there are probably more of these workflows than peoples on earth, so we don’t try to encapsulate all of them in this post.

I propose you to define some classic publishing note taking and note publishing workflow.

IMHO that could also help org beginer. It’s easier to start publishing note with a template in mind.

Like @jethro say, this is out of scope for org-roam core, and we probably need to create a specific org-roam-publish package.

Objectives of the package :

• generate a template based on simple or complex use-case already defined
• managing/preparing org-roam note for multiple front-end : ox-hugo, firn, etc.
• use filter rules to select which org notes are published (or not) for which front end
• …

Workflow :

• simple workflows :
  • one db, one website, all notes
• complex workflows :
  - one db, multiple website, filtering by taf and private/public note
  - multiple db, multiple website, filtering by tag and private/public note

Because i think it’s more easy to represent workflow using schema, i try to represent one of the complex publishing workflow in this schema. The svg is also shared, if you want to propose new things based on this first work.

Svg here : https://web.tresorit.com/l/xJeg3#Q8_u8EbdYmp6h2yX0sRKBQ

In this workflow :

• work & home could be defined by ROAM_TAGS context
• we have private (gpg) and public not in HOME and WORK context
• we have two publishing platform and three website
  • one website publish/contain only public note from WORK context
  • one website publish/contain only public note from HOME
  • one website publish/contain only private note from HOME

I think we could add one more level of filtering, because an user might want to publish only “note” marked as “ok-to-be-published” and not publish “draft”.

[jethro]

I guess I’ll describe my workflow a little here, just to give an idea of what’s already possible.

Some context:

• I don’t encrypt my files, my private files are just kept out of version control and sync.
• I use ox-hugo to publish my files. My public files have the necessary ox-hugo headers setup, based on the org-roam template. my private files do not, hence they should never get published.

The repo for my digital garden is at https://github.com/jethrokuan/braindump. You can see the org files in the org folder, and the generated md files in the content folder. Using my custom Hugo theme, it is able to detect backlinks without org-roam. This is dependent on the ox-hugo generating relref links.

Problems:

1. My public files may link to private files. While the content may be private, the links do leak some info. Not a dealbreaker for me, but might be a dealbreaker for some.
2. Severe duplication across org/md files. Ox-hugo was just really convenient.

My hope is that Org is able to produce static html/css without an intermediary. Org already has pretty good HTML export support. It just lacks a bunch of other things, like RSS and sitemaps.

[tees]

Hiya! @reyman shared this post with me and I thought I’d share a bit about a project I’ve been building called Firn.

Soon after discovering Org-Roam I wanted a tool that could generate a site from it, so I built a static site generator built for Org mode. Firn tries to solve some of the problems that I needed, similarly mentioned around here, including how to make work a mixture of private and public files. Here’s how I do things.

• I keep my wiki in a private repo.
• Private files state in a private folder, that Roam is able to index and Firn is able to ignore.
• Files can also be marked private in the in-buffer-setting using #+FIRN_PRIVATE.
• Public files may also link to private files, but I’m ok with that happening.

Firn also solves the problem of being able to render what I want from an org file. It provides a site-map, a basic “template engine” which includes the abilities to target exact headings within a file to render. This works for me, as the large majority of my notes mix private and public content; for example, my file french.org is for taking notes on learning french. The information that I want public is all under a headline called “notes”, which I call to render it in my layout.

Not sure if that’ll be of much use to anyone else, but that’s how things are working for now! I’ve got some things in the pipeline to build still, including site-wide tags that leverage org-roam tags as well as creating a link-graph / back links section based on the parsed org-files.

[reyman]

Thanks for replying @tees and @jethro, and sorry for the late answer due to hollidays.

We need the possibility to tag and collect “blogpost” anywhere in our digital garden for publication, like for example this tools made by karl-voit : https://github.com/novoid/lazyblorg , a tool cited in the critics of Zettel. workflow here :
https://karl-voit.at/2020/06/14/Zettelkasten-concerns/

I continue to list some ideas before sharing with you, but we already see on this forum that a possibility was to extend/adapt the org-publish existing tools to do some job between publishing engine (ox-hugo, firn, etc.)

Another idea that came to me was the possibility to publish part of garden (blog part) on fediverse (activityPub protocol) to easily cross publish and dissiminate words on any platform compatible with this protocol : https://github.com/Plume-org/Plume, mastodon, write.as, etc. It could be an interesting project.

[doubleloop]

@reyman

Another idea that came to me was the possibility to publish part of garden (blog part) on fediverse (activityPub protocol) to easily cross publish and dissiminate words on any platform compatible with this protocol : https://github.com/Plume-org/Plume, mastodon, write.as, etc. It could be an interesting project.

I like this idea too. I would like to combine my garden and stream into one space, and layering Fediverse or IndieWeb on top of org-roam could be the way to do it. I’ve toyed with Micropub before (https://github.com/ngm/micropub-layer) to publish to an IndieWeb-enabled stream from within Emacs.

[roi.holtzman]

I was looking at Firn and it seems awesome!
From looking at the docs, I see that it basically creates a html website locally on your machine.
I am not an expert on this - so excuse my ignorance - I was wondering if there is an easy way to share this notes-website with other people. Is there a section in the docs showing how this sharing can be done?

[tees]

Hi!

There are no plans for adding deployment docs; but if you google “how to deploy a static site” or “how to deploy a jekyll” site you will likely find some useful documents. On the firn side of things, you will just need to run firn build and then you’ll be on your way. Good luck!

[reyman]

Hi there, by coupling weblorg routing capacities (GitHub - emacs-love/weblorg: Static Site Generator for Emacs) and the great tool of alphapapa to filter content (org-ql), i’m thinking this is a the perfect tool to filter notes and contents to generate any website on top of org-roam : Using org-ql as powerfull filtering tool on top of org-roam to generate and publish any website · Issue #183 · alphapapa/org-ql · GitHub

Don’t hesitate to comment on issue.

[bbigras]

I’m very interested in this.

I saw the reply on the weblorg issue. Did you give it a try yet?

Are you on freenode or matrix?

[reyman]

Actually no tested, too much work on my job at this time, but i’m interested and i’m on matrix yup, you could find me on @reyman : matrix : org

[clintonboys]

Hi all. I wrote up my setup here: mt. solitary stack | mt. solitary. There’s some links to code snippets, mostly taken from other folks around the web, but hopefully someone can find it useful.

[florhizom]

heyhey,
I have made a longer journey trough the digital gardening wormhole now and while orgroam is probably the best “libre” notetaking backend, I find the publishing solutions not that enticing, namely since i am not a developer and rely on an existing community (and plugins) to add features to the site.
i find ikiwiki here interesting, since it’s already a wiki, but can generate a blog and has many useful functions/plugins: a blogfeed, graphplotting, search, comments/discussionpages, and, of course, the first criteria: an org plugin. There are other themes, or at least users with nicelooking pages. havent tried out though because the installation on windows seems whacky - transition to linux will happen sometime sooner though^^ Maybe someone will try it out and report

Right now I am myself looking into the possibilities of nikola (based on phython), which seems to be quite powerfull and moldable and has some existing org-connections and standalone plugins like above mentioned; themes seem to be transferable; and an org- plugin that calls on emacs for new posts/pages and integrates the org format.
from the org-side there are several existing solutions, too - but they’re pretty old and i’m not sure if they work.
GitHub - msnoigrs/ox-nikola: export Nikola blog posts in reStructuredText syntax from Org-Mode runs nikola from emacs
drymerisnothere / nikola-el · GitLab renders to rst in nikola
GitHub - xcodebuild/blog-admin: Write blog in emacs with hexo/org-page/nikola more general publishing to nikola/hexo/org-page
GitHub - olymk2/emacs-nikola: nikola plugin using magit-popup publishes to nikola via magit

well, i will see. I had the problem in jekyll that i couldn’t make blogposts work on the base of notes that this template introduced (it has some pretty strict rules in defining content) https://digital-garden-jekyll-template.netlify.app/, that’s why i’m looking for more moldable generators (not even sure if nikola is the answer here)

[jio32]

Thank you for sharing the setup. Going through I was able to setup garden.
I really liked your index page containing all notes. However going through code https://github.com/clintonboys/emacs.d, I couldn’t find any hint to how index is generated. Could you comment on it. Sorry if it sounds silly, but I don’t have much experience with web dev.
It would be great if there existed a hugo theme especially designed for digital garden.

[clintonboys]

Hi - the code that generates the index, recent updates section, stubs page and summary page is done with Hugo - I did initially generate it in org mode using org babel and bash but didn’t like that I had to remember to generate the Markdown every time I wanted to update it; doing it on the Hugo side solves this problem. The repository is still private but I’m planning to publish my fork to the hugo-book theme as a Digital Garden-specific theme for Hugo soon.

To generate the index for example, I created content/all-posts/_index.md which just contains front matter and a heading, and then at layouts/section/all-posts.html I have

{{ define "main" }}
{{.Content}}
Total notes: {{ len (.Site.RegularPages) }}
<p>
<ul.dashed class="all-posts">
{{range .Site.RegularPages.ByTitle}}
<li><a href="{{.Permalink}}">{{.Title}}</a> ({{.WordCount}} words, updated {{.Lastmod.Format "2006-   01-02"}})</li>
{{end}}
</ul>
{{ end }}

The other pages are generated similarly. Hope that helps!

[jio32]

Thank you for taking time to write. I was able to generate index page without any problems. Looking forward to your hugo theme.

[florhizom]

I actually have a WriteFreely account and there is an emacs package for publishing from org already. So that‘s pretty much possible, but I haven’t tried it out… one cool thing is that it will render the md–roam style tags #tags by custom.

[florhizom]

I decided to give Hugo a try, just to have something working for the Moment and be able to move.

Tried to just test your branch with the demo sites of the regular site, but it gives me errors bc it misses the mathjax.html partial…
other: The „version System“ implemented in Hugo–book actually sounds very cool.
I would like to add the usual taxonomies by tags/categories though. The more „entrances“ into the garden, the better I think

Here’s another Hugo theme, which is themed more like @jethro‘s approach

r/Zettelkasten - Check out my Zettelkästen Hugo Theme

The main thing I find interesting here in comparison is the transclusion, I don‘t really understand what is implemented to enable it (would like to style it in a blockquote at least, too, though.

[clintonboys]

Hi, yes I have a few small partials implemented outside of the fork:

• analytics.html which just wraps my Goatcounter script
• google-fonts.html which gets Google Fonts working based on this gist
• mathjax.html which just wraps the Mathjax script to support LaTeX rendering

These are then included in the header partial in the theme fork hugo-book/header.html at master · clintonboys/hugo-book · GitHub, so you can disable whichever you wish.

[florhizom]

did you have a look at org-publish?
I was trying to use it for publishing to hugo, but it will always give me “no publishing function chosen” when I envoke it with the single file command (for testing)…
my configuration is as minimal as possible right now…

(setq org-publish-project-alist
      '(
        ("digital-garden"
         :base-directory "~/org-roam/"
         :base-extension "org" "md"
         :publishing-directory "~/hugo/hugo-book/content/notes/"
         :publishing-function org-hugo-export-to-md))
) 

I would prefer using org-publish over copying the stuff that you and jethro do, without being really able to understand and customize it.

some stuff about the book-theme, maybe s.o can help or is inspired:
I found an easy way to keep the sidebar in hugo-book always toggleable which allows for more space. same is hoped for with the toc , that failed though until now. I would have liked it to be toggleable for a post on the right of the title, that i would use for longer articles/ blogs but maybe not for shorter notes, but have not really figured it out … .
i also want to have a “real header” where i can have some navigation elements, the toggle for the sidebar, the title of the site and the search.
Having the space on the sites generally empty on bigger screens is just cleaner and makes room for the implementation of side- and marginnotes for example, which i would hope for to accomplish somewhen (also transclusion, but these two methods only make sense when the site works in general and.
wanted to have a real header to put search, site name there but i’m struggling with implementing good spacing there.

a good thing i didn’t really recognize earlier is that categories and tags are implemented in the theme albeit not shown very prominently, but one can work with that :).

[zaeph]

A little update:

Publishing a slip-box online is something that we’ve been interested in developing (probably as an extension). We’re currently working on polishing v2 which, has you know, introduces a number of changes. Once we’ve polished the branch, merged it into main, and dealt with the ripples, we’ll probably be favouring the Emacs UX and the bigger extensions we have, most notably org-roam-server. As a result, the web-publishing is fairly low-priority, but it remains on our radar nonetheless. I’m working on a document that would specify how this publishing should work, and I’ll make sure to submit it to those interested for review.