---
layout: post
title: "HFS+ and Unicode madness"
date: 2010-04-25
comments: false
status: fixed
reviewed: 2026-10
status_note: >
  True in 2010, not an issue anymore. Macs moved from HFS+ to APFS in 2017,
  which keeps file names the way you typed them, git learned to fix this in
  2012, and Java uses UTF-8 by default since JDK 18. The NFC vs NFD explanation
  is still right, and it still explains why the same é can be two different
  strings.
---

<p><strong>This is an old post that I wrote a while ago, but never published. It is here just for historical|nostalgic|unburdening reasons. Original post begins here:</strong></p>

Annoying scenario: A file with unicode characters in its filename, like `Pétalo.java`:

```console
$ javac Pétalo.java
Pe?talo.java:1: illegal character: \8730
public class Pétalo {
              ^
Pe?talo.java:1: illegal character: \169
public class Pétalo {
              ^
2 errors
```

<small>[Gist ↗](https://gist.github.com/juanger/489706)</small>[^2026r-hfs]

[^2026r-hfs]: Restored in 2026, these were gists that stopped rendering.

<p>Why? That’s easy, because Mac OS X Java wants everything with MacRoman encoding (<span class='caps'>WTF</span>??<label for="e26-macroman" class="margin-toggle errata-number"></label><input type="checkbox" id="e26-macroman" class="margin-toggle"/><span class="sidenote errata"><span class="errata-year">2026</span>That was Java's platform default encoding, which was MacRoman on old Macs. Since JDK 18 (JEP 400) the default is UTF-8 everywhere.</span>). Then I tried with the -encoding flag and tab-autocompletion for the filename:</p>

```console
javac -encoding UTF8 Pétalo.java
Pe?talo.java:1: class P?talo is public, should be declared in a file named P?talo.java
public class P?talo {
       ^
1 error
```

<small>[Gist ↗](https://gist.github.com/juanger/489709)</small>

<p>That is not good. When using autocompletion the filename is represented in <span class='caps'>NFD</span> or Normalization Form Canonical Decomposition (é =&gt; 65 cc81) but the class name in the file uses <span class='caps'>NFC</span> or Normalization Form Canonical Composition (é =&gt; c3a9). What a mess! Do you see the difference?:</p>

```console
$ echo Pétalo | xxd
0000000: 5065 cc81 7461 6c6f 0a                   Pe..talo.
$ echo Pétalo | xxd
0000000: 50c3 a974 616c 6f0a                      P..talo.
```

<small>[Gist ↗](https://gist.github.com/juanger/489712)</small>

<p>You can’t see the difference until you see the actual bytes of the string, and that’s why Apple tries to protect us by storing everything in the Filesystem with the same representation: <span class='caps'>NFD</span>.<label for="e26-nfd" class="margin-toggle errata-number"></label><input type="checkbox" id="e26-nfd" class="margin-toggle"/><span class="sidenote errata"><span class="errata-year">2026</span>HFS+ did this (a variant of NFD, to be exact). APFS doesn't rewrite names. It stores the bytes you give it and compares names ignoring normalization, so the two Pétalo files are the same file but keep their own bytes.</span> The first echo was done with autocompletion while the latter was typed directly with the keyboard. So if I compile the Java source code by fully typing the filename, it succeeds.</p><h3>¿What about version control?</h3><p>Then, this leads to a major problem. When someone in the world (including you and me) creates a file, he types its name with the keyboard, which as I have shown, gives <span class='caps'>NFC</span> in Mac OS X. But with <span class='caps'>HFS</span>, the file is stored using <span class='caps'>NFD</span> and if you clone a git repo created in another OS you’ll see that you have untracked files without modifying anything!!<label for="e26-precompose" class="margin-toggle errata-number"></label><input type="checkbox" id="e26-precompose" class="margin-toggle"/><span class="sidenote errata"><span class="errata-year">2026</span>Fixed on the git side by <code>core.precomposeunicode</code> (git 1.7.12, 2012), which is on by default for new repos on macOS. Git turns the NFD names it reads from disk back into NFC, so this rarely happens anymore.</span></p><h3><span class='caps'>UFS</span> to the rescue (kind of…)</h3><p>Maybe you could say “well, just use another filesystem”, and I’ve done that, but it seems that Apple uses its decomposition algorithms everywhere.</p><p>First I tried <span class='caps'>ZFS</span> as I thought “That FS should work correctly in <span class='caps'>OSX</span>, they can’t spoil it”. But I installed the read-write implementation from MacOS Forge and it did the same stupid conversion (<span class='caps'>UPDATE</span>: <span class='caps'>ZFS</span> is discontinued in MacOS Forge :( ).</p><p><span class='caps'>FAT</span> is a no-no since it only accepts <span class='caps'>ASCII</span> characters for the filename and I don’t want to start a remote server nor a Virtual Machine every time I have to work with those files. This should work with my beloved OS!! And in fact, I have successfully made everything I’ve needed to work in Mac OS, until now :(.</p><p>So I needed to try <span class='caps'>UFS</span>, I looked at Disk Utility, but it doesn’t support <span class='caps'>UFS</span> and newfs fails to format my external <span class='caps'>USB</span> disk.</p><p>But since I used hdiutil to create Pallet’s dmg, I thought of having a <span class='caps'>UFS</span> disk image and guess what, you can!! This is the magic command:</p>

```console
$ hdiutil create -size 100m -fs UFS -volname UFS UFS.dmg 
```

<small>[Gist ↗](https://gist.github.com/juanger/489715)</small>

<p>Now I can mount my dmg, compile and use git!!</p><p>But still there are two little problems:</p><ul><li>Snow Leopard doesn’t support <span class='caps'>UFS</span>.<label for="e26-ufs" class="margin-toggle errata-number"></label><input type="checkbox" id="e26-ufs" class="margin-toggle"/><span class="sidenote errata"><span class="errata-year">2026</span>Lion (10.7) dropped reading and mounting UFS entirely, so the disk image trick is history too.</span> I can’t even mount my <span class='caps'>UFS</span> disk image :(</li>
<li>Every app that uses Cocoa to open files can’t read files with <span class='caps'>NFC</span> filenames.<label for="e26-cocoanfc" class="margin-toggle errata-number"></label><input type="checkbox" id="e26-cocoanfc" class="margin-toggle"/><span class="sidenote errata"><span class="errata-year">2026</span>Fixed on APFS, NFC names open fine now.</span> This means “forget about TextMate”. Noooooooo!!</li>
</ul><h3>Update</h3><p>Now I’ve updated to Snow Leopard and it is very disappointing and sad that I have to use a virtual machine to do my job.</p><p>The good thing is that I can work with TextMate via ssh and MacFusion.</p><p>Bad things: MacOS X can’t solve my problem, I can’t open files with <span class='caps'>NFC</span> names in TextMate (but emacs.app still loves me) and although it seems there are some people working with File Systems at Apple I think this won’t be solved anytime soon.<label for="e26-solved" class="margin-toggle verdict-toggle"><span class="verdict-word">held</span>⊕</label><input type="checkbox" id="e26-solved" class="margin-toggle"/><span class="marginnote errata verdict" data-verdict="held"><span class="errata-year">2026</span><span class="verdict-word">held</span>Held, "not anytime soon" was right. It took seven years and a new file system (APFS, 2017), plus git's fix in 2012.</span></p>
