Ruby Xocolatl

HFS+ and Unicode madness

This is an old post that I wrote a while ago, but never published. It is here just for historical|nostalgic|unburdening reasons. Original post begins here:

Annoying scenario: A file with unicode characters in its filename, like Pétalo.java:

$ javac Pétalo.java
Pe?talo.java:1: illegal character: \8730
public class Pétalo {
              ^
Pe?talo.java:1: illegal character: \169
public class Pétalo {
              ^
2 errors

Gist ↗Restored in 2026, these were gists that stopped rendering.

Why? That’s easy, because Mac OS X Java wants everything with MacRoman encoding (WTF??2026That was Java's platform default encoding, which was MacRoman on old Macs. Since JDK 18 (JEP 400) the default is UTF-8 everywhere.). Then I tried with the -encoding flag and tab-autocompletion for the filename:

javac -encoding UTF8 Pétalo.java
Pe?talo.java:1: class P?talo is public, should be declared in a file named P?talo.java
public class P?talo {
       ^
1 error

Gist ↗

That is not good. When using autocompletion the filename is represented in NFD or Normalization Form Canonical Decomposition (é => 65 cc81) but the class name in the file uses NFC or Normalization Form Canonical Composition (é => c3a9). What a mess! Do you see the difference?:

$ echo Pétalo | xxd
0000000: 5065 cc81 7461 6c6f 0a                   Pe..talo.
$ echo Pétalo | xxd
0000000: 50c3 a974 616c 6f0a                      P..talo.

Gist ↗

You can’t see the difference until you see the actual bytes of the string, and that’s why Apple tries to protect us by storing everything in the Filesystem with the same representation: NFD.2026HFS+ did this (a variant of NFD, to be exact). APFS doesn't rewrite names. It stores the bytes you give it and compares names ignoring normalization, so the two Pétalo files are the same file but keep their own bytes. The first echo was done with autocompletion while the latter was typed directly with the keyboard. So if I compile the Java source code by fully typing the filename, it succeeds.

¿What about version control?

Then, this leads to a major problem. When someone in the world (including you and me) creates a file, he types its name with the keyboard, which as I have shown, gives NFC in Mac OS X. But with HFS, the file is stored using NFD and if you clone a git repo created in another OS you’ll see that you have untracked files without modifying anything!!2026Fixed on the git side by core.precomposeunicode (git 1.7.12, 2012), which is on by default for new repos on macOS. Git turns the NFD names it reads from disk back into NFC, so this rarely happens anymore.

UFS to the rescue (kind of…)

Maybe you could say “well, just use another filesystem”, and I’ve done that, but it seems that Apple uses its decomposition algorithms everywhere.

First I tried ZFS as I thought “That FS should work correctly in OSX, they can’t spoil it”. But I installed the read-write implementation from MacOS Forge and it did the same stupid conversion (UPDATE: ZFS is discontinued in MacOS Forge :( ).

FAT is a no-no since it only accepts ASCII characters for the filename and I don’t want to start a remote server nor a Virtual Machine every time I have to work with those files. This should work with my beloved OS!! And in fact, I have successfully made everything I’ve needed to work in Mac OS, until now :(.

So I needed to try UFS, I looked at Disk Utility, but it doesn’t support UFS and newfs fails to format my external USB disk.

But since I used hdiutil to create Pallet’s dmg, I thought of having a UFS disk image and guess what, you can!! This is the magic command:

$ hdiutil create -size 100m -fs UFS -volname UFS UFS.dmg 

Gist ↗

Now I can mount my dmg, compile and use git!!

But still there are two little problems:

  • Snow Leopard doesn’t support UFS.2026Lion (10.7) dropped reading and mounting UFS entirely, so the disk image trick is history too. I can’t even mount my UFS disk image :(
  • Every app that uses Cocoa to open files can’t read files with NFC filenames.2026Fixed on APFS, NFC names open fine now. This means “forget about TextMate”. Noooooooo!!

Update

Now I’ve updated to Snow Leopard and it is very disappointing and sad that I have to use a virtual machine to do my job.

The good thing is that I can work with TextMate via ssh and MacFusion.

Bad things: MacOS X can’t solve my problem, I can’t open files with NFC names in TextMate (but emacs.app still loves me) and although it seems there are some people working with File Systems at Apple I think this won’t be solved anytime soon.2026heldHeld, "not anytime soon" was right. It took seven years and a new file system (APFS, 2017), plus git's fix in 2012.