I was having a little chat with @jethro and I wanted to open this up to the folks in the Org-roam user community. I would like to ask you to share your expertise and thoughts.
The topic affects Windows users in particular, but I think it would be great if macOS and Linux users among you chip in and share your knowledge.
Commit f8dd345 for V2 reintroduces emacsql-sqlite for Emacs to work with the SQLite database.
For Windows users, this means that your PC will need to be able to compile a piece of software from source to build an executable (.exe) file. The procedure is simple enough if you have done it once.
About a year ago when I wanted to start using Org-roam, this was an entry barrier for me on Windows. I gave up once. Now it has become trivial to me and I donât even have to think about it (you will see a short description of the steps at the bottom of this post).
Do you think Org-roam should and can host a version of executable file as part of the project?
Is it a good idea for the Org-roam project to host a binary executable?
Is the project permitted to do so (license, etc.)?
I will be happy to share a version I can produce (x86-64 only, not i686, I think) if my concerns are not really an issue:
Is there any technical concerns around doing so?
Am I permitted to do this?
I will not be able to support if anyone has an issue. Is this concern warranted? Or is it just my psychological barrier with no foundation (and I should get over it)?
Does anyone have any experience and knowledge in this matter?
Add msys2's bin directory to your environmental variable
On Emacs, install emacsql-sqlite package and start Org-roam
Installing emacs-sqlite packcage can be before msys2. gcc needs to be accessible by Emacs before staring Org-roam (1â3 do just that). Emacs will start compiling the executable automatically.
*: You might ask why not scoop or chocolaty.
The current version of scoop (as on 2021-05-03) has an unresolved issue in installing gcc . I have managed to install a previous version and confirmed that it works; but this procedure is not very beginner friendly
Maybe chocolaty is great. I havenât used it so cannot comment. It might be very easy to use instead of msys2
EDIT: My initial version had make as a requirement. This is incorrect and not needed. It does not do any harm (except for 1.4mb or so of storage). You only need gcc.
I suggest that org-roam v2 should be as user-friendly as pdf-tools, which downloads and compiles the binaries it needs, almost by itself.
When I started on org-roam v1, I wasted an unreasonable amount of time trying to get sqlite to compile on Windows. But luckily, installing sqlite3 via chocolatey turned out to be nearly effortless.
Later, when I wanted to use pdf-tools, I ran into yet more Windows compile problems. But this time, I easily overcame them by⌠using chocolatey to install msys2. Then, I let the function pdf-tools-install (in the melpa pdf-tools emacs package) handle the rest. With the msys2 binaries in my path, pdf-tools-install downloaded the necessary msys2 packages and did the compilation and linking.
My config for pdf-tools is:
(defun sdo/find-exec(cmd_name &optional notFoundMsg)
"Finds path to executable 'cmd_name' and returns it, or nil if not found."
(let (cmd_path retpathstr)
(setq cmd_path (executable-find cmd_name))
(if (not cmd_path)
(progn (if notFoundMsg
(warn "%s not found. %s" cmd_name notFoundMsg)
(warn "%s not found." cmd_name))
(setq retpathstr nil))
(progn (message "found %s at: %s" cmd_name cmd_path)
(setq retpathstr cmd_path))))) ; there must be a better way...
(defun sdo/popdir (dir)
"like unix popd. Return parent directory of dir"
(unless (equal "/" dir)
(file-name-directory (directory-file-name dir))))
(if (setq pacbin (sdo/find-exec "pacman" "Need MSYS2 for pdf-tools & more"))
(use-package pdf-tools
:config
;; Ensure mingw64 libraries on front of PATH, not other tools' libs
(setq msys2dir (sdo/popdir (sdo/popdir (sdo/popdir pacbin))))
(setq msys2libdir (expand-file-name "mingw64\\bin" msys2dir))
(setenv "PATH" (concat msys2libdir ";" (getenv "PATH"))))
;; initialise (also downloads and builds msys2 libs)
(pdf-tools-install))
Internally, org-roam v2 could do something similar with sqlite.
All you need is a C compiler, which normally isnât too much to ask for: one of âccâ, âgccâ, or âclangâ needs to be available, that is all.
@jethro Getting a C compiler ready is the biggest hurdle for ordinary Windows users (I struggled, @scotto struggled too). The struggle has two components:
Finding out gcc is the requirements
You would need to read the Elisp code you cited; I donât find documentation
Getting gcc on your Windows system to work cc and clang donât seem to be readily available for Windows, so gcc is the compiler of choice. And there are many different ways that throw you into different rabbit holes â mingw64, msys2, scoop and chocolatey (now I know scoop and chocolatey are commandline wrappers for msys2 so there are two main options, really
Anyways, now the community has sufficient knowledge accumulated (through someone like me, @scotto, etc.) so hopefully Windows users among us will struggle less. The 4-step procedure I laid out above can be repeated and should cover most of Windows users for Org-roam.
Thank you. I learned from you that chocolatey also relies on msys2 like scoop does (in different ways, I believe).
Is it correct to summarize your opinion to be this?
If Org-roam provides a way to automatically compile emacsql-sqlite.exe, as user-friendly as pdf-tools is, there is no need for the project to host a binary file.
If so⌠let me try to unpack what âas user-friendly as pdf-toolsâ means.
The procedure to compile emacsql-sqlite.exe is as follows. Itâs automatic by Emacs as long as you have done all 1â3 as @jethro points out. What steps does pdf-tools automate in addition to this? Steps 2 and 3?
Honestly, itâs quite a lot to ask of the users to compile the program herself. Emacs, Org mode and Org roam is already obscure enough. Adding to it that one needs to compile her own program will make it even harder for anyone to even get close to this software.
I doubt that the benefit of the commit mentioned is big enough to alienate a whole infrastructure such as windows. It at least shouldnât be taken lightly. Itâs easy as a developer to see that itâs âjust to install some programs and compileâ. But the reality is that the regular user is lightyears away from that. Even if a crystal clear instruction was given on the Readme for the document. And putting such a requirement as is discussed here is really not a step in the right direction if this software wants to see increased mindshare and usage.
I ran into this same issue when I first started using org-roam (on Windows) so Iâm a big +1 for including the binary. Also, I may be missing something obvious, but I donât understand why Org-roam canât use the official sqlite binary instead of someone from the project needing to precompile it.
Yeah, I suppose I suggest to not switch to emacs library that requires a custom sqlite binary in the first place. For the sake of making this tool a bit less difficult to install and use.
@jethro, maybe org-roam could install the C compiler too? This is what pdf-tools will do if necessary.
Iâve already forgotten what my problems were with compiling sqlite on Windows, but I remember being surprised at the messiness â I wrote in C professionally for a couple of years, used to compile my own Linux kernels, etc⌠I thought that compiling sqlite on Windows would surely be trivial. But it wasnât, I was busy, and I dropped that project.
@nobiot, yes, I think that automatically compiling emascsql-sqlite.exe would be a good-enough solution.
If compilation is the way to go, the user must somehow make sure that gcc is available, and have that â and its libraries â in the path. But after that, org-roam could itself handle steps 3 and 4.
I work on a shifting set of machines, each of which I have to set up, so Iâve also simplified step 2 (installing gcc). Instead of assuming that gcc is installed, my config checks if msys2 is installed, and fails gracefully if itâs not there. But if it is there, it sets up paths and lets pdf-tools use msys2 to do whatever installs it needs to do. This extra config and the gcc install done by pdf-tools could also be in org-roam.
Admittedly, shipping org-roam with an sqlite binary would be simpler for the user, as it would remove the dependency on msys2.
This seems a bit overkill (C compilers are huge). I think the best compromise is to vendor the binary, but Iâm not sure what kind of troubles that brings either. Not too familiar with the intricacies of cross-compilation esp for other non-unix platforms.
It already does, if you have a C compiler available.
As a side-note I feel this sort of âimprovementsâ can go directly into emacsql-sqlite, a number of other projects use it too (like magitâs forge), maybe they have better ideas.
I would opt with @Gustav that such additional requirements will raise the bar, and should be avoided. Now what is the reason to use that particular sql program instead of the old variant?
I would strongly discourage shipping a binary. For Linux, that would mean messing with all the different packaging systems, for Windows I donât know, that might be an idea. But the âcleanâ way would be to rely on the user to be able to install an additional package on her own. There are already external dependencies like that for org-roam, e.
g. ripgrep. So on this view, org-roam should avoid baking its own cake and just use a library which is widely available and can be installed easily via oneâs systems packaging manager.
EDIT: I just recently installed magitâs forge and was surprised that it had to compile a binary. It worked, but it felt like too much. I immediately deinstalled it, since the features, for me, did not seem worth the trouble of having yet another binary installed somewhere by some elisp program. And I think thatâs different from pdf-tools, which has no real choice. And pdf-tools, too, is annoying, since I have to recompile it each time a new poppler library drops in.
Installation of SQLite on macOS and Linux is a breeze
Iâve tried running Emacs on Windows and it was hard; I gave up because I had a choice
I wouldnât like compiled blobs of code coming down with a package-install
pdf-tools breaks often for me and requires me to recompile; when it breaks the error message is not obvious but I can recognise some of the paths that appear
I took the libery of hosting a copy of the executable file I use on my Windows machine.
I have also included a brief instruction on how you might build one for yourself â recommended way is to build it yourself.
@jethro , Let me know if you see any issue with my doing this; I will happily remove the executable, leaving only the instruction (I am planning to update Zero to Org-roam and include the instruction in it).
If anyone finds issues around the license, please advise. It should be fine (see below):
I use org-roam heavily on Windows machines with no admin rights (at work). No choice as to OS or user permissions due to security policy. Iâm sure there are others in a similar position (e.g., government research labs).
So, installing additional tools and compiling source code would be a huge head ache with no admin access on Windows. Not sure if it will even be possible, but please let me know if Iâm incorrect. The current way for v1 works just fine on the other hand (e.g., instructions at Zero-to-Emacs-and-Org-roam/30.Set-up-Org-roam.md at main ¡ nobiot/Zero-to-Emacs-and-Org-roam ¡ GitHub do not need admin rights on windows). And explaining to the IT/security personnel why I need all of this seems rather futile⌠I would love to keep using org-roam, but may have to migrate to logseq or something else for these reasons when the ecosystem goes to v2.