Using Lua on Nginx to fix Finder's WebDAV quirks

Lua support for Nginx is provided by the ngx_http_lua_module courtesy of the OpenResty project, who have their own Nginx distribution with additional Lua-based features. However, it’s entirely possible to use the module with the mainline version of Nginx as well, which is what I’m doing.

Loading the module is as simple as calling load_module:

load_module /usr/lib/nginx/modules/ngx_http_lua_module.so;

Setting the appropriate Lua load paths with lua_package_path and lua_package_cpath (for Lua files and compiled modules respectively) allows loading Lua libraries:

lua_package_path  '/usr/share/lua/5.1/?.lua;;';
lua_package_cpath '/usr/lib/x86_64-linux-gnu/lua/5.1/?.so;;';

Once configured, you can call Lua code with the various configuration directives, e.g.:

location / {
  content_by_lua_block {
    ngx.say("Hello from Lua!")
  }
}

Here’s a quick example: my name (Leo Nikkilä) has multiple ways of being represented in Unicode. The most common is to use the precomposed code point U+00E4 LATIN SMALL LETTER A WITH DIAERESIS (ä). In UTF-8, this is:

$ printf ä | od -An -tx1 -v
 c3 a4

However, you can also write the same letter by composing U+0061 LATIN SMALL LETTER A (a) and U+0308 COMBINING DIAERESIS (◌̈) to form ä which looks identical, but is different on the wire:

$ printf ä | od -An -tx1 -v
 61 cc 88

This makes it difficult to compare strings, especially arbitrary inputs that should match pre-existing values like path names, which is why Unicode defines various normalisation forms to deal with the issue. The most common forms are the ones above: NFC (“Normalization Form Canonical Composition”, preferring (pre)composed code points) and NFD (“Normalization Form Canonical Decomposition”, preferring decomposed code points)².

Most file systems these days use NFC to encode Unicode path names³.

However, Apple’s (now legacy) HFS+ file system for Mac OS X used NFD to store path names. This caused problems with interoperability, giving rise to utilities (like Björn Jacke’s convmv) to rename files transferred over from Mac OS X.

This is still evident in HFS+’s successor APFS, which supports both NFC and NFD. Similar to how file systems provide case-insensitivity, APFS is able to recall a file using its original name, while preventing you from creating another file with the same name in another form.

Unfortunately Apple hasn’t completely moved away from NFD: as of macOS Sonoma 14.2.1, Finder’s WebDAV implementation still expects the server to encode paths in NFD! I’m guessing this is something Apple did to support WebDAV on macOS Server and HFS+, which was supported since Mac OS X Server 10.0.

Ultimately this means that if your WebDAV server uses NFC for path names (extremely likely since macOS Server is now discontinued) and a path’s NFC and NFD representations differ, Finder is unable to access that path through WebDAV.

The PROPFIND response body includes URL-encoded path names:

<?xml version="1.0" encoding="utf-8" ?>
<D:multistatus xmlns:D="DAV:">
<D:response>
<D:href>/Leo%20Nikkil%C3%A4</D:href>
<D:propstat>
<D:prop>
<D:displayname>Leo Nikkilä</D:displayname>
</D:prop>
<D:status>HTTP/1.1 200 OK</D:status>
</D:propstat>
</D:response>
</D:multistatus>

To work around this issue, we should mangle <D:href> elements from NFC to NFD. (The above should be /Leo%20Nikkila%CC%88.) We should also mangle request paths back from NFD to NFC when Finder tries to access them⁴.

Nginx can serve content from Lua, but we can also rewrite requests and responses from other Nginx modules, so that we can still serve responses using ngx_http_dav_module and ngx_http_dav_ext_module but tweak things where needed.

rewrite_by_lua_block allows modifying the request, e.g. rewriting the request URI and HTTP headers:

location / {
  rewrite_by_lua_block {
    ngx.req.set_uri("/new/path/")
    ngx.req.set_header("Destination", "/new/destination/")
  }
}

header_filter_by_lua_block and body_filter_by_lua_block allow modifying the response, e.g. rewriting response headers and the response body:

location / {
  header_filter_by_lua_block {
    -- Reset Content-Length since we're changing the body length.
    ngx.header.content_length = nil
  }
  body_filter_by_lua_block {
    -- Append a comment to the response body. The response is chunked,
    -- so we're processing every chunk separately here.
    let buf, eof = ngx.arg[1], ngx.arg[2]
    if eof then
       ngx.arg[1] = { buf, "<!-- Hello from Lua! -->" }
    end
  }
}

These are enough to implement Rob’s WebDAV configuration and rewrite NFC to NFD and back where necessary.

Rewriting request URIs and Destination headers was the easy part. Rewriting NFC to NFD was more difficult. Since the PROPFIND response is XML, I needed to parse the XML in Lua using LuaExpat. I could then decide what to do with different parts of the response and rewrite the <D:href> elements only⁵.

To make things more difficult, I couldn’t find a widely available Lua library to transform NFC into NFD and back, although luautf8 recently gained support for isnfc and normalize_nfc thanks to Alex Dowad. Since the normalisation data is now included, the NFD functions would be a nice addition as well. I’ve been going through the C source code but unfortunately haven’t found the time to attempt this myself.

For now, I ended up stealing the little-known ustring module from Wikimedia’s Scribunto codebase, which implements functions for both transformations in pure Lua. The downside is that it isn’t available in any repositories, so I had to vendor in the whole thing.

This took quite a bit of work, but I’m glad I didn’t have to change my name⁶ to access my files with Finder.

You’ll find the resulting Lua scripts and how to install them at https://git.sr.ht/~lnikkila/nginx-webdav-quirks. For discoverability, I’ve also made the repository available at a GitHub mirror.

While working on this, I discovered that Finder requests some magic files from the server after connecting, e.g. /.ql_disablethumbnails and /.ql_disablecache.

I’ve been looking for a way to globally disable thumbnail generation while browsing remote files since this generates a lot of traffic. Creating empty files at these paths seems to work, but browsing will inexplicably get much slower.

Based on the access logs, Finder is downloading entire files from the server to determine which file types they are, whereas normally it only looks at the header/trailer of each file. This doesn’t make much sense, but I thought I’d document this here since the problem is being discussed but the magic files are rarely mentioned.

I ended up disabling this hack, but maybe someone is able to figure out something that works. Clearly Apple is aware of similar problems since they’ve also implemented the DSDontWriteNetworkStores flag.