AWS compression algorithm switched from gzip to zstd: Saving 30% storage space

Adrian cockcroft, the former vice president of Amazon, broke the news on Twitter , saying that after the AWS compression algorithm was switched from gzip to zstd, a huge amount of memory was saved, the compressed S3 storage was reduced by about 30%, and the space saved can reach the scale of EB (1 EB). = 1024 PB = 1024 * 1024 TB).

An Amazon employee added to the statement, saying that what Amazon changed was not the way of compressing data stored by customers, but the way S3 itself stores service data (mainly logs) – S3 itself switched from gzip logs to ztsd logs, making Storage costs are reduced by 30%.

However, Amazon did not issue an announcement related to changing the compression technology. Careful Holmes netizens found that Amazon S3 had a price reduction of 31% at the end of November 2021, which just corresponds to the 30% reduction in storage costs.

About Zstd

Zstd (Zstandard) is a fast lossless compression algorithm open sourced by Facebook. It is mainly used in real-time compression scenarios at the zlib level and has a better compression ratio.

Zstd can also provide stronger compression ratios at the expense of compression speed, with the ratio of speed to compression ratio configurable incrementally.

Zstd is a compression technology with excellent performance. Different from compression algorithms such as zlib , lz4, and xz, Zstd seeks a solution that combines compression performance and compression rate. 

Zstd also provides a special compression mode “dictionary compression” for small data, which supports generating dictionary files in training mode to improve the compression ratio for small data packets.

The following is a performance comparison of common compression algorithms: