V2 - Do we want Org-roam project to host an executable file?

I was having a little chat with @jethro and I wanted to open this up to the folks in the Org-roam user community. I would like to ask you to share your expertise and thoughts.

The topic affects Windows users in particular, but I think it would be great if macOS and Linux users among you chip in and share your knowledge.

Commit f8dd345 for V2 reintroduces emacsql-sqlite for Emacs to work with the SQLite database.

For Windows users, this means that your PC will need to be able to compile a piece of software from source to build an executable (.exe) file. The procedure is simple enough if you have done it once.

About a year ago when I wanted to start using Org-roam, this was an entry barrier for me on Windows. I gave up once. Now it has become trivial to me and I don’t even have to think about it (you will see a short description of the steps at the bottom of this post).

Do you think Org-roam should and can host a version of executable file as part of the project?

  1. Is it a good idea for the Org-roam project to host a binary executable?
  2. Is the project permitted to do so (license, etc.)?
  3. I will be happy to share a version I can produce (x86-64 only, not i686, I think) if my concerns are not really an issue:
  • Is there any technical concerns around doing so?
  • Am I permitted to do this?
  • I will not be able to support if anyone has an issue. Is this concern warranted? Or is it just my psychological barrier with no foundation (and I should get over it)?

Does anyone have any experience and knowledge in this matter?


Steps to compile emacsql-sqlite.exe from source

I have tested this to work on 2021-05-03.

  1. Install msys2*
  2. Within msys2, install gcc (via command pacman -S)
  3. Add msys2's bin directory to your environmental variable
  4. On Emacs, install emacsql-sqlite package and start Org-roam

Installing emacs-sqlite packcage can be before msys2. gcc needs to be accessible by Emacs before staring Org-roam (1–3 do just that). Emacs will start compiling the executable automatically.

*: You might ask why not scoop or chocolaty.

  1. The current version of scoop (as on 2021-05-03) has an unresolved issue in installing gcc . I have managed to install a previous version and confirmed that it works; but this procedure is not very beginner friendly
  2. Maybe chocolaty is great. I haven’t used it so cannot comment. It might be very easy to use instead of msys2

EDIT: My initial version had make as a requirement. This is incorrect and not needed. It does not do any harm (except for 1.4mb or so of storage). You only need gcc.

1 Like

I suggest that org-roam v2 should be as user-friendly as pdf-tools, which downloads and compiles the binaries it needs, almost by itself.

When I started on org-roam v1, I wasted an unreasonable amount of time trying to get sqlite to compile on Windows. But luckily, installing sqlite3 via chocolatey turned out to be nearly effortless.

Later, when I wanted to use pdf-tools, I ran into yet more Windows compile problems. But this time, I easily overcame them by… using chocolatey to install msys2. Then, I let the function pdf-tools-install (in the melpa pdf-tools emacs package) handle the rest. With the msys2 binaries in my path, pdf-tools-install downloaded the necessary msys2 packages and did the compilation and linking.

My config for pdf-tools is:

(defun sdo/find-exec(cmd_name &optional notFoundMsg)
  "Finds path to executable 'cmd_name' and returns it, or nil if not found."
  (let (cmd_path retpathstr)
    (setq cmd_path (executable-find cmd_name))
    (if (not cmd_path)
        (progn (if notFoundMsg
		   (warn "%s not found. %s" cmd_name notFoundMsg)
		 (warn  "%s not found." cmd_name))
	       (setq retpathstr nil))
      (progn (message "found %s at: %s" cmd_name cmd_path)
             (setq retpathstr cmd_path))))) ; there must be a better way...

(defun sdo/popdir (dir)
  "like unix popd.  Return parent directory of dir"
  (unless (equal "/" dir)
    (file-name-directory (directory-file-name dir))))

(if (setq pacbin (sdo/find-exec "pacman" "Need MSYS2 for pdf-tools & more"))
      (use-package pdf-tools
        :config
        ;; Ensure mingw64 libraries on front of PATH, not other tools' libs
        (setq msys2dir (sdo/popdir (sdo/popdir (sdo/popdir pacbin))))
        (setq msys2libdir (expand-file-name "mingw64\\bin" msys2dir))
        (setenv "PATH" (concat msys2libdir ";" (getenv "PATH"))))
        ;; initialise (also downloads and builds msys2 libs)
        (pdf-tools-install))

Internally, org-roam v2 could do something similar with sqlite.

1 Like

This is what emacs-sqlite does underneath as well: emacsql/emacsql-sqlite.el at master · skeeto/emacsql · GitHub

All you need is a C compiler, which normally isn’t too much to ask for: one of “cc”, “gcc”, or “clang” needs to be available, that is all.

@jethro Getting a C compiler ready is the biggest hurdle for ordinary Windows users (I struggled, @scotto struggled too). The struggle has two components:

  1. Finding out gcc is the requirements
    You would need to read the Elisp code you cited; I don’t find documentation
  2. Getting gcc on your Windows system to work
    cc and clang don’t seem to be readily available for Windows, so gcc is the compiler of choice. And there are many different ways that throw you into different rabbit holes – mingw64, msys2, scoop and chocolatey (now I know scoop and chocolatey are commandline wrappers for msys2 so there are two main options, really

Anyways, now the community has sufficient knowledge accumulated (through someone like me, @scotto, etc.) so hopefully Windows users among us will struggle less. The 4-step procedure I laid out above can be repeated and should cover most of Windows users for Org-roam.

@scotto

Thank you. I learned from you that chocolatey also relies on msys2 like scoop does (in different ways, I believe).

Is it correct to summarize your opinion to be this?

If Org-roam provides a way to automatically compile emacsql-sqlite.exe, as user-friendly as pdf-tools is, there is no need for the project to host a binary file.

If so… let me try to unpack what "as user-friendly as pdf-tools" means.

The procedure to compile emacsql-sqlite.exe is as follows. It’s automatic by Emacs as long as you have done all 1–3 as @jethro points out. What steps does pdf-tools automate in addition to this? Steps 2 and 3?

Honestly, it’s quite a lot to ask of the users to compile the program herself. Emacs, Org mode and Org roam is already obscure enough. Adding to it that one needs to compile her own program will make it even harder for anyone to even get close to this software.

I doubt that the benefit of the commit mentioned is big enough to alienate a whole infrastructure such as windows. It at least shouldn’t be taken lightly. It’s easy as a developer to see that it’s “just to install some programs and compile”. But the reality is that the regular user is lightyears away from that. Even if a crystal clear instruction was given on the Readme for the document. And putting such a requirement as is discussed here is really not a step in the right direction if this software wants to see increased mindshare and usage.

2c

/Gustav

Thanks, @Gustav. I take it that you are in favour of the Org-roam project hosting a binary for people to download. Or are you suggestion even more?

I ran into this same issue when I first started using org-roam (on Windows) so I’m a big +1 for including the binary. Also, I may be missing something obvious, but I don’t understand why Org-roam can’t use the official sqlite binary instead of someone from the project needing to precompile it.

Yeah, I suppose I suggest to not switch to emacs library that requires a custom sqlite binary in the first place. For the sake of making this tool a bit less difficult to install and use.

@jethro, maybe org-roam could install the C compiler too? This is what pdf-tools will do if necessary.

I’ve already forgotten what my problems were with compiling sqlite on Windows, but I remember being surprised at the messiness – I wrote in C professionally for a couple of years, used to compile my own Linux kernels, etc… I thought that compiling sqlite on Windows would surely be trivial. But it wasn’t, I was busy, and I dropped that project.

@nobiot, yes, I think that automatically compiling emascsql-sqlite.exe would be a good-enough solution.

If compilation is the way to go, the user must somehow make sure that gcc is available, and have that – and its libraries – in the path. But after that, org-roam could itself handle steps 3 and 4.

I work on a shifting set of machines, each of which I have to set up, so I’ve also simplified step 2 (installing gcc). Instead of assuming that gcc is installed, my config checks if msys2 is installed, and fails gracefully if it’s not there. But if it is there, it sets up paths and lets pdf-tools use msys2 to do whatever installs it needs to do. This extra config and the gcc install done by pdf-tools could also be in org-roam.

Admittedly, shipping org-roam with an sqlite binary would be simpler for the user, as it would remove the dependency on msys2.

I think it’s perfectly reasonable to vendor a binary. It would be nice to have the option to automatically build.

This seems a bit overkill (C compilers are huge). I think the best compromise is to vendor the binary, but I’m not sure what kind of troubles that brings either. Not too familiar with the intricacies of cross-compilation esp for other non-unix platforms.

It already does, if you have a C compiler available.

As a side-note I feel this sort of “improvements” can go directly into emacsql-sqlite, a number of other projects use it too (like magit’s forge), maybe they have better ideas.

I would opt with @Gustav that such additional requirements will raise the bar, and should be avoided. Now what is the reason to use that particular sql program instead of the old variant?

I would strongly discourage shipping a binary. For Linux, that would mean messing with all the different packaging systems, for Windows I don’t know, that might be an idea. But the ‘clean’ way would be to rely on the user to be able to install an additional package on her own. There are already external dependencies like that for org-roam, e.
g. ripgrep. So on this view, org-roam should avoid baking its own cake and just use a library which is widely available and can be installed easily via one’s systems packaging manager.

EDIT: I just recently installed magit’s forge and was surprised that it had to compile a binary. It worked, but it felt like too much. I immediately deinstalled it, since the features, for me, did not seem worth the trouble of having yet another binary installed somewhere by some elisp program. And I think that’s different from pdf-tools, which has no real choice. And pdf-tools, too, is annoying, since I have to recompile it each time a new poppler library drops in.

Chiming in with my 2c:

  • vendoring/hosting is fine (pending license concerns)
  • relying on normal, off-the-shelf binaries from a system’s package manager is preferred to custom binaries
  • it should be really easy (org-roam-install-foo) to get it installed, doing whatever you need to do under the hood
  • it’d be great if the binary was an optional dependency (e.g. org-roam worked without it, just missing features / slower)
  • but making org-roam even better is worth violating the above preferences