lnikki.la

Use a global gitignore file, please!

2025-04-11T00:00:00Z

Here’s a quick tip that I find myself repeating, so I thought I’d write this up somewhere I can reliably link to.

Many Git repositories have a .gitignore file at the root of the repository. Generally this is used to exclude build outputs from ending up in source control, however often these files also accumulate file and directory names related to various editors (e.g. .vscode, .idea) and operating system caches (.DS_Store, Thumbs.db). The more people work on a repository, the longer these files tend to get.

It doesn’t have to be this way. Git has a global gitignore file where you should ignore files that relate to your tools. The default path of this file is $XDG_CONFIG_HOME/git/ignore, and following the XDG Base Directory Specification, this means the file is generally at ~/.config/git/ignore. You’ll need to create this file if it doesn’t exist.

To give you an example, currently my global gitignore file is:

 *~ .#* .DS_Store [Tt][Aa][Gg][Ss] /NOTES.org

In this file, I’m excluding:

common names of backup files
.DS_Store in case I’ve rsynced something from macOS
Ctags-like index files (case-insensitively)
the file I keep my notes related to a repository.

Using a global gitignore file, you free up .gitignore files in source control to list only those files that are related to the project (e.g. build outputs). Even better, you’ll only need to configure the global file once, since it’ll apply to all repositories on your machine.

Should you need one, there’s also a repository-specific gitignore file that doesn’t get tracked in source control at .git/info/exclude.

Using Lua on Nginx to fix Finder's WebDAV quirks

2024-02-04T00:00:00Z

This is a response to Rebecca Peck’s post about making WebDAV actually work on Nginx a few years back. Her post was incredibly helpful when I was moving my WebDAV server from lighttpd¹ to Nginx and trying to put this puzzle together on my own. I highly recommend you take a look at it first.

My short summary of the post is that Nginx’s ngx_http_dav_module lacks support for parts of the WebDAV specification that Finder depends on, but the third-party ngx_http_dav_ext_module adds support for those. Finder is also non-compliant and doesn’t follow the specification properly, which means you need to work around some quirks, like adding trailing slashes to all directory paths.

In her closing words, Rebecca mentions wanting to use Lua instead of various Nginx modules to jump through these hoops. I agreed; in particular I didn’t want to install the ngx_headers_more module just to rewrite some request headers.

I got invested enough to roll all of this configuration into a set of Lua scripts that you can install with relative ease on your own WebDAV server.

While working on this, I also discovered and fixed another Finder compatibility issue that Rebecca’s configuration didn’t address: Finder has trouble with certain kinds of Unicode paths.

But first, let’s look at how Nginx works with Lua, and what Unicode normalisation forms are.

Nginx and Lua

Lua support for Nginx is provided by the ngx_http_lua_module courtesy of the OpenResty project, who have their own Nginx distribution with additional Lua-based features. However, it’s entirely possible to use the module with the mainline version of Nginx as well, which is what I’m doing.

Loading the module is as simple as calling load_module:

load_module /usr/lib/nginx/modules/ngx_http_lua_module.so;

Setting the appropriate Lua load paths with lua_package_path and lua_package_cpath (for Lua files and compiled modules respectively) allows loading Lua libraries:

lua_package_path '/usr/share/lua/5.1/?.lua;;'; lua_package_cpath '/usr/lib/x86_64-linux-gnu/lua/5.1/?.so;;';

Once configured, you can call Lua code with the various configuration directives, e.g.:

location / { content_by_lua_block { ngx.say("Hello from Lua!") } }

Unicode normalisation forms

Here’s a quick example: my name (Leo Nikkilä) has multiple ways of being represented in Unicode. The most common is to use the precomposed code point U+00E4 LATIN SMALL LETTER A WITH DIAERESIS (ä). In UTF-8, this is:

 $ printf ä | od -An -tx1 -v c3 a4

However, you can also write the same letter by composing U+0061 LATIN SMALL LETTER A (a) and U+0308 COMBINING DIAERESIS (◌̈) to form ä which looks identical, but is different on the wire:

 $ printf ä | od -An -tx1 -v 61 cc 88

This makes it difficult to compare strings, especially arbitrary inputs that should match pre-existing values like path names, which is why Unicode defines various normalisation forms to deal with the issue. The most common forms are the ones above: NFC (“Normalization Form Canonical Composition”, preferring (pre)composed code points) and NFD (“Normalization Form Canonical Decomposition”, preferring decomposed code points)².

Unicode and file systems

Most file systems these days use NFC to encode Unicode path names³.

However, Apple’s (now legacy) HFS+ file system for Mac OS X used NFD to store path names. This caused problems with interoperability, giving rise to utilities (like Björn Jacke’s convmv) to rename files transferred over from Mac OS X.

This is still evident in HFS+’s successor APFS, which supports both NFC and NFD. Similar to how file systems provide case-insensitivity, APFS is able to recall a file using its original name, while preventing you from creating another file with the same name in another form.

Unfortunately Apple hasn’t completely moved away from NFD: as of macOS Sonoma 14.2.1, Finder’s WebDAV implementation still expects the server to encode paths in NFD! I’m guessing this is something Apple did to support WebDAV on macOS Server and HFS+, which was supported since Mac OS X Server 10.0.

Ultimately this means that if your WebDAV server uses NFC for path names (extremely likely since macOS Server is now discontinued) and a path’s NFC and NFD representations differ, Finder is unable to access that path through WebDAV.

The PROPFIND response body includes URL-encoded path names:

<?xml version="1.0" encoding="utf-8" ?> <D:multistatus xmlns:D="DAV:"> <D:response> <D:href>/Leo%20Nikkil%C3%A4</D:href> <D:propstat> <D:prop> <D:displayname>Leo Nikkilä</D:displayname> </D:prop> <D:status>HTTP/1.1 200 OK</D:status> </D:propstat> </D:response> </D:multistatus>

To work around this issue, we should mangle <D:href> elements from NFC to NFD. (The above should be /Leo%20Nikkila%CC%88.) We should also mangle request paths back from NFD to NFC when Finder tries to access them⁴.

Writing the configuration in Lua

Nginx can serve content from Lua, but we can also rewrite requests and responses from other Nginx modules, so that we can still serve responses using ngx_http_dav_module and ngx_http_dav_ext_module but tweak things where needed.

rewrite_by_lua_block allows modifying the request, e.g. rewriting the request URI and HTTP headers:

location / { rewrite_by_lua_block { ngx.req.set_uri("/new/path/") ngx.req.set_header("Destination", "/new/destination/") } }

header_filter_by_lua_block and body_filter_by_lua_block allow modifying the response, e.g. rewriting response headers and the response body:

location / { header_filter_by_lua_block { -- Reset Content-Length since we're changing the body length.  ngx.header.content_length = nil  }  body_filter_by_lua_block {  -- Append a comment to the response body. The response is chunked,  -- so we're processing every chunk separately here. let buf, eof = ngx.arg[1], ngx.arg[2] if eof then ngx.arg[1] = { buf, "<!-- Hello from Lua! -->" } end } }

These are enough to implement Rebecca’s WebDAV configuration and rewrite NFC to NFD and back where necessary.

Rewriting request URIs and Destination headers was the easy part. Rewriting NFC to NFD was more difficult. Since the PROPFIND response is XML, I needed to parse the XML in Lua using LuaExpat. I could then decide what to do with different parts of the response and rewrite the <D:href> elements only⁵.

To make things more difficult, I couldn’t find a widely available Lua library to transform NFC into NFD and back, although luautf8 recently gained support for isnfc and normalize_nfc thanks to Alex Dowad. Since the normalisation data is now included, the NFD functions would be a nice addition as well. I’ve been going through the C source code but unfortunately haven’t found the time to attempt this myself.

For now, I ended up stealing the little-known ustring module from Wikimedia’s Scribunto codebase, which implements functions for both transformations in pure Lua. The downside is that it isn’t available in any repositories, so I had to vendor in the whole thing.

Using this yourself

This took quite a bit of work, but I’m glad I didn’t have to change my name⁶ to access my files with Finder.

You’ll find the resulting Lua scripts and how to install them at https://git.sr.ht/~lnikkila/nginx-webdav-quirks. For discoverability, I’ve also made the repository available at a GitHub mirror.

Bonus: Disabling Finder’s Quick Look thumbnail generation

While working on this, I discovered that Finder requests some magic files from the server after connecting, e.g. /.ql_disablethumbnails and /.ql_disablecache.

I’ve been looking for a way to globally disable thumbnail generation while browsing remote files since this generates a lot of traffic. Creating empty files at these paths seems to work, but browsing will inexplicably get much slower.

Based on the access logs, Finder is downloading entire files from the server to determine which file types they are, whereas normally it only looks at the header/trailer of each file. This doesn’t make much sense, but I thought I’d document this here since the problem is being discussed but the magic files are rarely mentioned.

I ended up disabling this hack, but maybe someone is able to figure out something that works. Clearly Apple is aware of similar problems since they’ve also implemented the DSDontWriteNetworkStores flag.

Footnotes:

lighttpd is rarely mentioned when it comes to WebDAV, it has a very nice mod_webdav module that relies on SQLite for locking.

Some languages (like Vietnamese) have characters with multiple combining marks, which makes their ordering affect the on-the-wire representation as well. Normalisation forms also define a canonical order for ordering multiple combining characters.

For a comprehensive overview and all the things I glossed over, see the Unicode equivalence article on Wikipedia.

I have no proof to back this up, but I’m assuming this is because precomposed characters use less storage space.

⁴

Finder sends an easily sniffed User-Agent like WebDAVFS/3.0.0 (03008000) Darwin/23.2.0 (arm64) which helps us avoid mangling these path names unless we’re specifically talking to Finder.

⁵

LuaExpat doesn’t currently expose Expat’s XML_DefaultCurrent function, which would make it easier to output the original XML without having to format and escape it again. I had to write some extra code to deal with this.

⁶

Nor Kieran Hebden’s tracks in my music collection under his cheeky ⣎⡇ꉺლ༽இ•̛)ྀ◞ ༎ຶ ༽ৣৢ؞ৢ؞ؖ ꉺლ moniker.

Advanced HTML with Org mode to impress your friends

2023-09-02T00:00:00Z

This time, let’s indulge in the favourite pastime of every blog-building hacker out there: blogging about blogging.

This website replaced my decade-old and dormant site. Over time, the threshold to post got higher and higher. I wasn’t interested in re-learning how to build it just to write something.

I’ve since learned from my mistake. This site is built with Org mode using org-publish. I use Org daily, and there isn’t really anything to re-learn. Everything happens from Emacs, and there are zero additional dependencies to download.

Building the site

I’m not going to discuss how to set up org-publish, there are plenty of resources online. I’ve linked some posts that I found exceptionally helpful¹^,²^,³.

Essentially, I’ve set up a build-site.el script that runs Emacs with emacs -Q --script to invoke org-publish. This way the build can be fully automated with source control, behaving more like a usual static site generator, although you can still invoke it directly from the editor.

HTML export variables

org-html exposes a number of variables to customise the HTML output, for example:

org-html-doctype
org-html-head
org-html-head-extra
org-html-meta-tags
org-html-divs
org-html-preamble
org-html-postamble
org-html-scripts

For a full list, see M-x describe-variable ^org-html-. Most notable are org-html-{pre,post}amble, which people have used for all kinds of layouts.

However, all of these options are fairly limited. Conceptually simple changes, like placing the date and author information below the title, or changing the “Table of Contents” heading, are surprisingly tricky with these.

Now that I’ve learned about the better way to achieve these types of changes, I wish org-html didn’t have some of these variables at all. They make some narrow use cases easy, but at the expense of obscuring more powerful concepts behind org-export.

Demystifying org-export

org-export supports swappable backends for formats. These are defined using org-export-define-backend which takes a unique backend name and an alist of transcoders. Broadly speaking, transcoders are functions that take an Org element plus some metadata and return a string representing it in the output format.

Here’s an example of a minimal S-expression backend:⁴

(defun indent-string (string width) (org-ascii--indent-string (string-trim string) width)) (defun sexp-template (contents info) (format "(template\n%s)" (indent-string contents 1))) (defun sexp-inner-template (contents info) (format "(inner-template\n%s)" (indent-string contents 1))) (defun sexp-section (section contents info) (format "(section\n%s)" (indent-string contents 1))) (defun sexp-headline (headline contents info) (format "(headline %s\n%s)" (org-export-data (org-element-property :title headline) info) (indent-string contents 1))) (defun sexp-paragraph (paragraph contents info) (format "(paragraph %s)" contents)) (defun sexp-plain-text (text info) (format "\"%s\"" (string-replace "\n" "\\n" text))) (org-export-define-backend 'sexp '((template . sexp-template) (inner-template . sexp-inner-template) (section . sexp-section) (headline . sexp-headline) (paragraph . sexp-paragraph) (plain-text . sexp-plain-text)))

and its output for this simple document:

(with-temp-buffer (insert "Hello world!\n") (insert "* Here's a headline\n") (insert "Here's some more text\n") (org-export-as 'sexp))

 (template (inner-template (section (paragraph "Hello world!\n")) (headline "Here's a headline" (section (paragraph "Here's some more text\n")))))

Deriving the org-html backend

In addition to defining new backends, you can also derive an existing backend using org-export-define-derived-backend. Can you see where this is going?

Here’s how to derive the HTML backend and replace some transcoders:

(defun my-html-paragraph (paragraph contents info) (format "<p class=\"my-paragraph\">%s</p>" contents)) (org-export-define-derived-backend 'my-html 'html :translate-alist '((paragraph . my-html-paragraph)))

yielding HTML with a modified <p> tag:

 <p class="my-paragraph">Hello world!</p>

What can you customise with this? Everything⁵.

Examples

Rather than relying the various org-html* variables to modify the <head> element and other pieces of the template, it’s possible to render the whole thing yourself:

(defun cool-html-template (contents info) (concat "<!DOCTYPE html>\n" "<html lang=\"en\">\n" " <head>\n" " <title>" (org-export-data (plist-get info :title) info) "</title>\n" " </head>\n" " <body>\n" (indent-string contents 4) "\n" " </body>\n" "</html>")) (org-export-define-derived-backend 'cool-html 'html :translate-alist '((template . cool-html-template)))

resulting in this majestic display:

 <!DOCTYPE html> <html lang="en"> <head> <title>cool crab's cool page</title> </head> <body> 🦀 </body> </html>

Rather than using org-html-{pre,post}amble, it’s possible to render the inner-template as well, e.g. to wrap the document in <main>:

(defun cool-html-inner-template (contents info) (concat "<main>\n" (indent-string contents 2) "\n</main>")) (org-export-define-derived-backend 'cool-html 'html :translate-alist '((template . cool-html-template) (inner-template . cool-html-inner-template)))

 <!DOCTYPE html> <html lang="en"> <head> <title>cool crab's cool page</title> </head> <body> <main> <img src="/cool-crab.png" alt="a cool-looking crab on the beach, wearing dark sunglasses and enjoying a refreshing beverage" /> </main> </body> </html>

The info alist

I haven’t yet mentioned the info argument passed into the transcoder functions. The Org export reference documentation describes this as a “communication channel”, it’s an alist used to pass around metadata.

There’s a lot of data in there, including keywords defined in the Org document. This makes it useful to implement custom options that can be set in the document to influence the generated HTML.

There are convenience functions for accessing data within the info alist, like org-export-get-date.

But wait, there’s more

I’ve only scratched the surface of what org-export backends can do.

You can also go nuts with the Org parse tree before it’s exported using filters, and modify the plain markup before parsing using hooks. See the Advanced Export Configuration manual page for details.

To sum up, export backends provide extremely flexible and elegant rendering of Org documents, and you can do a lot more with a backend compared to the available export options.

Acknowledgements

Thanks to Jason Stewart for feedback on an earlier draft of this post.

Footnotes:

https://thibaultmarin.github.io/blog/posts/2016-11-13-Personal_website_in_org.html

https://taingram.org/blog/org-mode-blog.html

https://www.brautaset.org/posts/blogging-with-org-mode.html

⁴

These examples are generated by Org Babel!

⁵

See the org-element-all-elements and org-element-all-objects variables for the list of types, also documented in the Org element API. In addition to those, org-export adds the template, inner-template, and plain-text types.

Disabling CLAT46 on macOS Ventura

2023-08-12T00:00:00Z

Long story short: I run an IPv6-only network at home. Explaining why is out of scope here.

Like Ondřej Caletka and Jason Healy have already discussed in their respective posts, the past couple of macOS releases have been shipping with IPv4-to-IPv6 client-side translation (CLAT46).

Essentially this means you can transparently interact with IPv4 addresses without an IPv4 connection, provided that your IPv6 connection supports NAT64:

 $ sudo networksetup -setv4off Wi-Fi $ ping -c1 1.1.1.1 PING 1.1.1.1 (1.1.1.1): 56 data bytes 64 bytes from 1.1.1.1: icmp_seq=0 ttl=59 time=6.691 ms --- 1.1.1.1 ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 6.691/6.691/6.691/0.000 ms

See the previous links for how CLAT works in detail.

Auto-configuration on macOS

This is what little I’ve found out about the implementation.

During network setup, the com.apple.IPConfiguration subsystem decides whether CLAT should be enabled, and logs this in Console:

 default 08:46:40.447314+0300 configd RTADV en0: CLAT46 enabled using address 192.0.0.2 default 08:46:42.059657+0300 configd RTADV en0: removed CLAT46 address 192.0.0.2

The IPConfiguration agent can be tinkered with using ipconfig, which returns the CLAT status through ipconfig getsummary <interface> when active:

 $ ipconfig getsummary en0 <dictionary> { IPv6 : <array> { 0 : <dictionary> { RTADV : <dictionary> { CLAT46Active : TRUE ... } ... } } ... }

There’s also a sysctl flag called net.inet6.ip6.clat_debug that I assume outputs additional details in the logs. This might require running sudo ipconfig setverbose 1.

Problems in macOS Ventura

More recently, macOS Ventura seems to have relaxed the requirements to automatically enable CLAT.

My home network uses OpenWrt, and it’s configured IPv6-only with SLAAC (without DHCPv6) and has both NAT66 and NAT64 (using Jool). I haven’t set the RFC 7050 Pref64 RA flag, nor does the network have a DHCPv4 server that might advertise the RFC 8925 IPv6-Only Preferred option.

After upgrading to macOS Ventura, I was very confused to see IPv4 traffic supposedly working, until I learned about the CLAT. I think auto-configuring CLAT is great for IPv6 adoption, but it seems there are still some bugs:

Firefox and other software now stubbornly prefer IPv4, although explicit IPv6 connections are handled correctly. Even attempting to force curl’s happy eyeballs implementation to give IPv6 a generous head start doesn’t work, and an IPv4 connection is made immediately:
```
 $ curl -vvv --happy-eyeballs-timeout-ms=1000 https://icanhazip.com * Trying 104.18.114.97:443... * Connected to icanhazip.com (104.18.114.97) port 443 (#0) 
```
A more annoying bug is one that Jason Healy wrote about, where CLAT routes are sometimes misconfigured after switching networks. This has also happened to me, often after waking my laptop from sleep in the morning.

Disconnecting and reconnecting fixes this right away, but the issue is usually back the next morning. Restarting macOS gets rid of this issue for a while, but it’ll eventually return.

Workaround

I wanted to validate that CLAT was really the source of these issues. The first thing I tried was disabling IPv4 altogether in System Settings, which can also be done using networksetup:

 $ sudo networksetup -setv4off Wi-Fi

Unfortunately CLAT doesn’t care about whether IPv4 is enabled, and it’s auto-configured nevertheless. Next, I simply deleted the configured IPv4 address using ifconfig:

 $ sudo ifconfig en0 delete 192.0.0.2

This did the trick, and the issues I’d seen went away immediately.

Unfortunately just removing the address doesn’t persist anywhere, and after reconnecting to the network, CLAT was auto-configured again with the same settings.

After perusing manpages and the morsels of macOS IPv6 documentation I could find, I was unable to find a nice CLAT-specific flag that could disable it.

Eventually I was able to figure out a workaround: configuring IPv4 with a link-local address from 169.254.0.0/16. Here’s an example, you might want to pick a random address from that block instead of this one:

 $ sudo networksetup -setmanual Wi-Fi 169.254.1.1 255.255.255.0

ipconfig getsummary en0 now reports ConfigMethod : LinkLocal for IPv4, and more crucially, CLAT46Active is nowhere to be seen.