blog home about get remote Apple support

stupid APFS tricks

APFS does a few things differently than HFS does. Some of it is under-the-hood stuff that has to do with encryption and security. But in terms of your day-to-day use, there are a couple of odd tricks it can perform.

Multiple volumes per disk partition

Most file systems create one volume per disk partition. But APFS allows multiple volumes per partition, within what it calls a container. So, if you open Disk Utillity, and choose Show All Devices from the View menu, what you’ll find is that under each drive partition (usually only one), if it’s formatted as APFS, you’ll see a container, and then within the container, you’ll see one or more volumes. I haven’t yet figured out a good use of secondary APFS volumes within a container that wouldn’t be as well solved by just making a folder on the primary volume.

If you do add a second volume to an APFS container, even though it will appear as though you added a second (or third or whatever) drive, only the first can be used for booting macOS. If you want to have more than one bootable macOS version on the same drive, that you can switch to using the option key during startup or with the Startup Disk control panel, you’ll instead need to partition the drive to get a  second APFS container.

However, Apple has. In Catalina, your macOS drive, though shown to you in the Finder as though it is a single volume like it was in all versions of macOS before it, is actually split across two volumes. One of these is the system drive, and contains parts of the operating system that don’t ever change, such as core applications like Mail, Safari, etc, as well as the unseen parts of the OS that keeps your computer running. So, from Apple’s perspective, the only reason they would change is if something were trying to modify the way the system behaves. Because that could potentially be the action of malicious software, Apple has designated this unchanging system volume of macOS to be readable, but not writeable. That would mean that bad software (or good software, for that matter) could try to change those parts of the system, but they’d be simply unable to.

Meanwhile, everything else in the OS, plus all your personal files, are designated in a writeable volume, as usual, within the same container. Then Apple glues these two volumes together in an APFS volume group, and makes their common areas (like the Applications folder) appear as a single folder via gluing the complementary files togehter with firmlinks. It’s all crazy complicated and insane. If you go looking in the basement, you’ll see two Applications folders on two volumes, each containing different things. But the Finder shows them to you as one.

Copies that don’t take up twice as much space

In HFS+, or really most file systems, if you duplicate a file (or folder), it writes a copy of all that file’s data, which, depending on its size, can take a while, and then, logically enough, you get a second file that takes up exactly as much space as the first one.

In APFS, something cool and crazy happens. What I just described is what looks like what happens. And your second file does in fact appear to take up as much space as the first one. But it doesn’t. It takes up almost nothing. That’s because what gets copied is not the file data itself, but just a reference to it. You can’t actually see that this happened, other than that it only took a few seconds to copy even a very large file or folder. And in fact it can be quite confusing, because the math on your drive no longer makes sense. If you had a 50 GB file and copy it under APFS, it looks like you’re using up 100 GB of whatever you have available. But you’re not. You’re still using 50 GB.

If you then open the second file, and change it, APFS only saves the parts of the data that changed, and changes its internal referencing so that the copied file now reflects the same data in the original that has not changed, except for the new parts that you have changed. So, in the example above, if you changed, say, 5 GB of data in the file, the two files now together occupy, in reality, 55 GB. This is known as copy-on-write, and is completely bananas.

I have used this feature of APFS to great effect when merging large photo libraries. Rule one of how I work is always set things up so I can go back a step if I need to. No one-way moves, if they can be avoided. Merging photos into a Photos library might not result in what I want, if something goes wrong. Under HFS+, I would have had to duplicate the whole Photos library first, so that it could be reverted to if need be. If that was a large Photos library — say, 200 GB — then I’d need 200 extra GB lying around. Plus the copying takes time. And, if I’m doing multiple iterations, then I need a LOT of extra space. And time.

APFS solves this problem in perfect fashion. If I duplicate that 200 GB photo library, it takes a short amount time, and takes, for all intents and purposes, no extra disk space. When I merge in new photos, those get added to the total disk space used. If I don’t get the results I expect, after the merge, no problem. I throw out the merged library, return to the pre-merged copy, and start over.

What’s challenging about copy-on-write is its lack of obviousness, and lack of consistency. As far as I know, it only happens during Finder copies. If you use the Terminal to copy, or you use “Save As…” in an Application, then you’ll get a new file that does not share data with the original, so it takes up as much space as the original, just as you would have under HFS+. And there’s no way to see, in APFS, that a file is a “references copy” rather than a full copy, nor see how much disk space is really used by it.

Snapshots

APFS has a very cool thing called snapshots. It makes your entire disk able to be reverted to a previous state. I’ve used this feature when troubleshooting a misbehaving Mac, because it enabled me to try something, say “nope, wasn’t that”, and then go back to the disk as it was, so I could try something else. When you’re not connected to your Time Machine drive, your Mac automatically makes snapshots, disk space permitting, and these are used to fill in the blanks when you reconnect. Snapshots are automatically made during macOS updates, but you can also make them on demand using “tmutil” in Terminal, or Carbon Copy Cloner (which can also restore from a snapshot without reverting the whole disk).

Copying complications

Copy-on-write and multiple volumes can cause complications when copying a whole disk with a file-based tool such as Carbon Copy Cloner. If you copy a file in the APFS Finder, and that copy gets copied to another disk, it’s going to necessarily copy all of its data. That means that, in the copy-on-write above example, your target disk will not have two related 50 GB files that in reality total 55 GB on your disk. It will end up with what you’d expect: two unrelated files that take up 50 GB each. I can imagine this would cause a complication if you are copying a nearly full disk to another disk of the same size, because that receiving disk might, counterintuitively, not be large enough.

The solution to this problem is to “block copy” the disk. Blocks are just groupings of bytes, which are themselves groupings of bits, which are to computers what atoms are to physical objects. Everything on your disk is just an ordering of bits, which are clustered into blocks. In other words, blocks are “beneath” the file system; the file system stores its data iin the blocks. So, if you ignore the concept of a file and just copy every raw block on a disk to another disk, you’ll still preserve all the relationships of copied files.

To block copy an APFS volume to another disk, or disk image, you can use Disk Utility. You need to first delete all snapshots in the volume, using “tmutil” in the Terminal, or Carbon Copy Cloner. You then need to unmount all volumes within the container. You then use the “Restore” button in Disk Utility, and specify the container as a source, and an APFS volume on the target, which will be replaced by what you’re copying. (You can’t specify the target container itself). However, Disk Utility apparently doesn’t do everything the right way in Catalina, when it comes to writing to the new disk, so you might need to go into Terminal to actually accomplish what you want. The links at the bottom of this page may be helpful for this.

One quirk of Apple’s container-copying tools — whether Disk Utility, or Terminal commands — is that, despite block-copying the entire original container, they only allow you to restore one volume from a multi-volume container. Why this is, I cannot say. What happens is this: the container (or image file representing the container) is copied as a single file to an APFS volume on the target. (If needed, any existing container on the target will be deleted, and a new container and volume will be created to receive the container-as-file.) Then, via a process Apple calls “inverting,” your chosen volume is extracted from the container-as-file, and, depending on what options you selected, it either gets added it as an additional volume of the target container, or it outright replaces the target volume, leaving you with, in effect, a block-copied APFS volume. Inverting will fail if you have any snapshots present. You need to delete them first.

If you have multiple volumes in your original container, you’ll need to repeat the process, making a point to add a volume to the target container, rather than replacing it; and you’ll have to go through the ineffecient process of recopying the entirety of the container for every volume you want from it.

There is a workaround, though — you can in fact block copy the whole container without Apple’s tools, using the “dd” commnd in the Terminal, and it will bring over all volumes contained within. It does require care to execute correctly, because if there is any chance that the original and the copy are present at the same time, the Mac can’t distinguish between them. So the volume ID’s need to be changed on the copy. And that gets complicated if the container is encrypted, because you can’t just change the ID. And the container is always encrypted on a T2-equipped Mac, if I understand things accurately, whether or not you have FileVault enabled. And I’d imagine that the partition (or whole disk) on the target disk would need to be the identical size as the source, or bad things might happen. (You could always resize partitions later once the copy is complete.)

This can also cause complications when copying a whole disk with a file-based tool such as Carbon Copy Cloner. If you copy a “references copy” made in the APFS Finder to another disk, it’s going to necessarily copy all of its data. That means that, in the above example, your target disk will not have two related files that 55 GB on your disk, but two separate files that take up 50 GB each. I can imagine this would cause a complication if you are copying a nearly full disk that has copies of files to another disk of the same size, because that receiving disk might, counterintuitively, not be large enough.

The solution to this problem is to “block copy” the disk. Blocks are just groupings of bytes, which are themselves groupings of bits, which are to computers what atoms are to physical objects. Everything on your disk is just an ordering of bits, which are clustered into blocks. In other words, blocks are “beneath” the file system; the file system stores its data iin the blocks. So, if you ignore the concept of a file and just copy every raw block on a disk to another disk, you’ll still preserve all the relationships of copied files.

To block copy an APFS container to another disk, or disk image, you can use Disk Utility. You use the “Restore” button, and specify the container as a source, and an APFS volume on the target which will be replaced with the one you’re copying (you can’t specify the container itself). However, Disk Utility apparently doesn’t do everything the right way in Catalina, when it comes to writing to the new disk, so you might need to go into Terminal to actually accomplish what you want. The links at the bottom of this page may be helpful for this.

References

I had to do some deep diving to research APFS behaviors, and the following resources were amazingly helpful:

https://gist.github.com/darwin/3c92ac089cf99beb54f1108b2e8b4b9f

https://github.com/toolbits/imagingAPFS

https://eclecticlight.co/2019/08/14/apfs-tools-suck/

https://eclecticlight.co/2017/04/08/how-to-make-your-own-apfs-volume/

https://blog.macsales.com/44912-new-capabilities-limitations-how-disk-utilitys-restore-feature-changed-in-high-sierra/

http://blog.tempel.org/2019/05/cloning-apfs-volumes-containers-apfs.html

https://apple.stackexchange.com/questions/361269/create-disk-image-disabled

“man asr” in Terminal

 

Sign up to get our blog posts by email

Sign up for our monthly email newsletter

Meta