TL;DR. Here’s how to format a drive that is readable and writable on all operating systems.

The Problem and Current Solution

Portable USB drives are great, but they can be a source of frustration when used across multiple operating systems. Each operating system flavor has its own preferred filesystem format: Linux uses ext4, OSX uses HFS+, Windows uses NTFS, and Solaris and Unix use ZFS 1. Enabling read/write access to any of these formats across different operating systems is cumbersome and usually involves third-party software.

For this reason, most drives come preformatted with a FAT32 filesystem that can be understood by all operating systems, but this is not without its own set of problems.

Problems with the Current Solution

Note. To convey exact size information, this post will use IEC binary prefix notation (Ki-, Mi-, Gi-, Ti-, etc.) where 1 KiB (kibibyte) = 1024 bytes, and SI binary prefix notation (K-, M-, G-, T-, etc) where 1 KB (kilobyte) = 1000 bytes.

Hard disk drives and solid state drives use SI notation and really mean it. A 2 TB drive has 2,000,000,000,000 bytes, not 2,199,023,255,552. Until recently, operating systems calculated size using IEC units, but displayed SI units; so a 2 TB drive would appear as “1.82 TB” (meaning 1.82 TiB). Newer operating systems are starting to display sizes correctly by using SI units for storage media and IEC units for RAM size. Note, however, that usable drive space is always slightly less than its capacity because the filesystem itself requires space for its records.

FAT32 is old. Very old. It was designed for use on 32-bit systems. Although it can be read natively by Linux, OSX, and Windows, its age shows when used on large devices. Since the filesystem can have a maximum of only 232 sectors, the maximum amount of usable space ranges from 2 TiB (512-byte sectors) to 16 TiB (4096-byte sectors) 2. Drives with more than 2 TB of storage volume are commonplace now. Maximum file size is 232 − 1 bytes (one byte shy of 4 GiB). High-definition movies easily approach this limit. The number of files in the system and lengths of file paths are quite limited as well.

A Better Solution

Enter exFAT, the 64-bit upgrade to FAT32. It is supported by all relatively modern operating systems and does not have the same limitations of FAT32. exFAT has been available on Windows since Windows XP SP2, on OS X since Snow Leopard (10.6.5 to be precise), and on Linux via FUSE (available in your favorite distro’s package manager or on Github). The maximum filesystem size ranges from 128 PiB (512-byte sector size) to 1 EiB (4096-byte sector size) 3 with similar (un-)restrictions on maximum file size. Filenames are limited to 255 characters, but this should not pose any problems to average users.

Problems with the Better Solution

Implementing this solution was harder than anticipated. Ironically, only exFAT drives formatted on Windows were readable and writable across all other operating systems, but getting this to format correctly was a challenge. Even then, Windows did not get it completely right.

Traditionally, a drive is organized into one or more partitions, each of which contains a filesystem. Partition metadata (name, type, flags, start sector, end sector, etc.) is stored in a partition table at the start of the drive. There are two partition table formats: Master Boot Record (MBR), which is on its way out, and GUID Partition Table (GPT). So the goal for our universally read-/writable drive is GPT with an exFAT partition.

Windows has limited support for partitioning through the Disk Management utility, but no way to create a partition table. Windows allows MBR drives to be formatted as exFAT, but GPT drives can only be formatted as NTFS. I used dd to destroy the partition table to see what Windows would do. Windows allows a drive without a partition table to be formatted as exFAT, but doing so formats the root device, leaving it without a partition table. Yuck.

Once I understood how Windows expects an exFAT-formatted drive to be laid out, I was able to format it correctly in Linux, but I have not yet been able to do so in OSX. Hence, the only way (that I have found) to make a properly formatted universally read-/writable exFAT drive is to use Linux. There are two secrets that I had to learn, and both are closely related to formatting an NTFS partition:

  1. When left to their own methods, Linux and Mac set an incorrect partition type identifier in the partition table. The correct type (as set by Windows) is 0x07 (MBR) or EBD0A0A2-B9E5-4433-87C0-68B6B72699C7 (GPT), the same type that is used for NTFS partitions.

  2. On GPT partitions, Windows needs the msftdata flag set. Depending on the utility used, selecting the appropriate type will also set this flag. This flag is also used for NTFS partitions.

One final potential issue is sector alignment: Partitions work best when their sectors match the physical sectors in the underlying hardware. This is why the specifying the right start sector is important. Most drive formatting utilities will detect and choose the correct starting sector automatically.

Update 2017-10-22: Thanks to Mamading Ceesay for pointing out the following additional caveat: exFAT’s number of physical sectors per cluster, also called “clustor size” or “allocation units”, can cause problems on Mac OS X (now called macOS) if larger than 1024 KB. Good cluster sizes for interoperability are 128 KB and 256 KB.

A Solution to the Problems with the Better Solution

Our goal is a drive with the following characteristics. We will use gdisk to format the drive.

CAUTION. Be sure to back up any important data on your drive. The following commands will destroy all data on the drive.
% sudo gdisk /dev/sdX
GPT fdisk (gdisk) version 0.8.8

Partition table scan:
  MBR: not present
  BSD: not present
  APM: not present
  GPT: not present

Creating new GPT entries.

Command (? for help):

First, create a new GPT partition table

Command (? for help): o
This option deletes all partitions and creates a new protective MBR.
Proceed? (Y/N): Y

Now create a partition. The defaults will create a new partition that spans the whole drive with the first sector already aligned. Be sure to choose the correct type 0700!

Command (? for help): n
Partition number (1-128, default 1):
First sector (34-16326462, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-16326462, default = 16326462) or {+-}size{KMGTP}:
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300): 0700
Changed type of partition to 'Microsoft basic data'

Write the changes to the drive and exit

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): Y
OK; writing new GUID partition table (GPT) to /dev/sdX.
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.

Finally, format our new partition with the exFAT filesystem.

% sudo mkfs.exfat /dev/sdX1
mkexfatfs 1.0.1
Creating... done.
Flushing... done.
File system created successfully.

Update 2017-10-22: mkfs.exfat will use the following default cluster sizes depending on the volume size, but the -s flag can be used to provide a value manually. Please refer to the manual (man mkfs.exfat) for more details.

The drive should now be readable and writable on Linux, OSX, and Windows.

  1. I’d venture to say that ZFS is the best filesystem available today due to its data integrity, file compression and deduplication, snapshotting, and RAID management. ↩︎

  2. Older hard drives used 512-byte sectors, but most modern drives have made the jump to 4096-byte sectors. Even though the drive has larger apparent size, a 4096-byte sector size means that the space quantum is 4 KiB, which can lead to a lot of “wasted” space. For example, a 5 KiB file overflows a 4 KiB sector by a single kilobyte. Therefore it needs to be stored across two sectors, an effective 8 KiB with 3 KiB of slack space. This is why the “file size” and “size on disk” differ. ↩︎

  3. exFAT can theoretically support a much larger filesystem, but the 48-bit IDE/ATA addressing scheme used by the motherboard is the limiting factor in this case. ↩︎