bcachefs

Compression

Added 2016-07-26 11:34:27 +0000 UTC

Finally figured out how to make compressed disk usage accounting work. It's a surprisingly thorny issue - I'll have to write more about it later.

The TL;DR is - disk usage is only allowed to increase when you're getting a disk reservation (which is also where you'd get -ENOSPC). We have to be able to arbitrarily move data around (or rewrite fragmented data) without the amount of disk space used - however we define that number - increasing. But extents getting partially overwritten - in particular, compressed extents that are split by being overwritten in the middle - really throws a wrench into things.

Anyways, up till now the compression code has been in pretty good shape (gzip and lz4 currently supported) - the only thing that's been missing is teaching the disk usage accounting code to go off of compressed size, not uncompressed size - without that, bcachefs won't actually let you store more data in a filesystem than if it wasn't compressed.

But that should finally be fixed in the next couple days, and compression will finally be useful. I'm really looking forward to having this done - I'm using a Chromebook Pixel as my main laptop, and the 64 GB SSD is rather cramped.

It looks like getting some throttling for writeback is going to be more urgent now - when doing a write requires allocating a bounce buffer (for the compressed data) for the duration of the write, you don't want writeback queuing up tons of writes and allocating all your memory for bounce buffers.