2 perf: Improve level 1 compression speed in most scenarios by 6% by @gbtucker and @terrelln
3 api: Move the advanced API, including all functions in the staging section, to the stable section
4 api: Make ZSTD_e_flush and ZSTD_e_end block for maximum forward progress
5 api: Rename ZSTD_CCtxParam_getParameter to ZSTD_CCtxParams_getParameter
6 api: Rename ZSTD_CCtxParam_setParameter to ZSTD_CCtxParams_setParameter
7 api: Don't export ZSTDMT functions from the shared library by default
8 api: Require ZSTD_MULTITHREAD to be defined to use ZSTDMT
9 api: Add ZSTD_decompressBound() to provide an upper bound on decompressed size by @shakeelrao
10 api: Fix ZSTD_decompressDCtx() corner cases with a dictionary
11 api: Move ZSTD_getDictID_*() functions to the stable section
12 api: Add ZSTD_c_literalCompressionMode flag to enable or disable literal compression by @terrelln
13 api: Allow compression parameters to be set when a dictionary is used
14 api: Allow setting parameters before or after ZSTD_CCtx_loadDictionary() is called
15 api: Fix ZSTD_estimateCStreamSize_usingCCtxParams()
16 api: Setting ZSTD_d_maxWindowLog to 0 means use the default
17 cli: Ensure that a dictionary is not used to compress itself by @shakeelrao
18 cli: Add --[no-]compress-literals flag to enable or disable literal compression
19 doc: Update the examples to use the advanced API
20 doc: Explain how to transition from old streaming functions to the advanced API in the header
21 build: Improve the Windows release packages
22 build: Improve CMake build by @hjmjohnson
23 build: Build fixes for FreeBSD by @lwhsu
24 build: Remove redundant warnings by @thatsafunnyname
25 build: Fix tests on OpenBSD by @bket
26 build: Extend fuzzer build system to work with the new clang engine
27 build: CMake now creates the libzstd.so.1 symlink
28 build: Improve Menson build by @lzutao
29 misc: Fix symbolic link detection on FreeBSD
30 misc: Use physical core count for -T0 on FreeBSD by @cemeyer
31 misc: Fix zstd --list on truncated files by @kostmo
32 misc: Improve logging in debug mode by @felixhandte
33 misc: Add CirrusCI tests by @lwhsu
34 misc: Optimize dictionary memory usage in corner cases
35 misc: Improve the dictionary builder on small or homogeneous data
36 misc: Fix spelling across the repo by @jsoref
39 perf: better decompression speed on large files (+7%) and cold dictionaries (+15%)
40 perf: slightly better compression ratio at high compression modes
41 api : finalized advanced API, last stage before "stable" status
42 api : new --rsyncable mode, by @terrelln
43 api : support decompression of empty frames into NULL (used to be an error) (#1385)
44 build: new set of macros to build a minimal size decoder, by @felixhandte
45 build: fix compilation on MIPS32, reported by @clbr (#1441)
46 build: fix compilation with multiple -arch flags, by @ryandesign
47 build: highly upgraded meson build, by @lzutao
48 build: improved buck support, by @obelisk
49 build: fix cmake script : can create debug build, by @pitrou
50 build: Makefile : grep works on both colored consoles and systems without color support
51 build: fixed zstd-pgo, by @bmwiedemann
52 cli : support ZSTD_CLEVEL environment variable, by @yijinfb (#1423)
53 cli : --no-progress flag, preserving final summary (#1371), by @terrelln
54 cli : ensure destination file is not source file (#1422)
55 cli : clearer error messages, especially when input file not present
56 doc : clarified zstd_compression_format.md, by @ulikunitz
57 misc: fixed zstdgrep, returns 1 on failure, by @lzutao
58 misc: NEWS renamed as CHANGELOG, in accordance with fboss
61 perf: slightly better decompression speed on clang (depending on hardware target)
62 fix : performance of dictionary compression for small input < 4 KB at levels 9 and 10
63 build: no longer build backtrace by default in release mode; restrict further automatic mode
64 build: control backtrace support through build macro BACKTRACE
65 misc: added man pages for zstdless and zstdgrep, by @samrussell
68 perf: much faster dictionary builder, by @jenniferliu
69 perf: faster dictionary compression on small data when using multiple contexts, by @felixhandte
70 perf: faster dictionary decompression when using a very large number of dictionaries simultaneously
71 cli : fix : does no longer overwrite destination when source does not exist (#1082)
72 cli : new command --adapt, for automatic compression level adaptation
73 api : fix : block api can be streamed with > 4 GB, reported by @catid
74 api : reduced ZSTD_DDict size by 2 KB
75 api : minimum negative compression level is defined, and can be queried using ZSTD_minCLevel().
76 build: support Haiku target, by @korli
77 build: Read Legacy format is limited to v0.5+ by default. Can be changed at compile time with macro ZSTD_LEGACY_SUPPORT.
78 doc : zstd_compression_format.md updated to match wording in IETF RFC 8478
79 misc: tests/paramgrill, a parameter optimizer, by @GeorgeLu97
82 perf: much faster dictionary compression, by @felixhandte
83 perf: small quality improvement for dictionary generation, by @terrelln
84 perf: slightly improved high compression levels (notably level 19)
85 mem : automatic memory release for long duration contexts
86 cli : fix : overlapLog can be manually set
87 cli : fix : decoding invalid lz4 frames
88 api : fix : performance degradation for dictionary compression when using advanced API, by @terrelln
89 api : change : clarify ZSTD_CCtx_reset() vs ZSTD_CCtx_resetParameters(), by @terrelln
90 build: select custom libzstd scope through control macros, by @GeorgeLu97
91 build: OpenBSD patch, by @bket
92 build: make and make all are compatible with -j
93 doc : clarify zstd_compression_format.md, updated for IETF RFC process
94 misc: pzstd compatible with reproducible compilation, by @lamby
97 perf: faster speed (especially decoding speed) on recent cpus (haswell+)
98 perf: much better performance associating --long with multi-threading, by @terrelln
99 perf: better compression at levels 13-15
100 cli : asynchronous compression by default, for faster experience (use --single-thread for former behavior)
101 cli : smoother status report in multi-threading mode
102 cli : added command --fast=#, for faster compression modes
103 cli : fix crash when not overwriting existing files, by Pádraig Brady (@pixelb)
104 api : `nbThreads` becomes `nbWorkers` : 1 triggers asynchronous mode
105 api : compression levels can be negative, for even more speed
106 api : ZSTD_getFrameProgression() : get precise progress status of ZSTDMT anytime
107 api : ZSTDMT can accept new compression parameters during compression
108 api : implemented all advanced dictionary decompression prototypes
109 build: improved meson recipe, by Shawn Landden (@shawnl)
110 build: VS2017 scripts, by @HaydnTrigg
111 misc: all /contrib projects fixed
112 misc: added /contrib/docker script by @gyscos
115 perf: faster zstd_opt strategy (levels 16-19)
116 fix : bug #944 : multithreading with shared ditionary and large data, reported by @gsliepen
117 cli : fix : content size written in header by default
118 cli : fix : improved LZ4 format support, by @felixhandte
119 cli : new : hidden command `-S`, to benchmark multiple files while generating one result per file
120 api : fix : support large skippable frames, by @terrelln
121 api : fix : streaming interface was adding a useless 3-bytes null block to small frames
122 api : change : when setting `pledgedSrcSize`, use `ZSTD_CONTENTSIZE_UNKNOWN` macro value to mean "unknown"
123 build: fix : compilation under rhel6 and centos6, reported by @pixelb
124 build: added `check` target
127 new : long range mode, using --long command, by Stella Lau (@stellamplau)
128 new : ability to generate and decode magicless frames (#591)
129 changed : maximum nb of threads reduced to 200, to avoid address space exhaustion in 32-bits mode
130 fix : multi-threading compression works with custom allocators
131 fix : ZSTD_sizeof_CStream() was over-evaluating memory usage
132 fix : a rare compression bug when compression generates very large distances and bunch of other conditions (only possible at --ultra -22)
133 fix : 32-bits build can now decode large offsets (levels 21+)
134 cli : added LZ4 frame support by default, by Felix Handte (@felixhandte)
135 cli : improved --list output
136 cli : new : can split input file for dictionary training, using command -B#
137 cli : new : clean operation artefact on Ctrl-C interruption
138 cli : fix : do not change /dev/null permissions when using command -t with root access, reported by @mike155 (#851)
139 cli : fix : write file size in header in multiple-files mode
140 api : added macro ZSTD_COMPRESSBOUND() for static allocation
141 api : experimental : new advanced decompression API
142 api : fix : sizeof_CCtx() used to over-estimate
143 build: fix : no-multithread variant compiles without pool.c dependency, reported by Mitchell Blank Jr (@mitchblank) (#819)
144 build: better compatibility with reproducible builds, by Bernhard M. Wiedemann (@bmwiedemann) (#818)
145 example : added streaming_memory_usage
146 license : changed /examples license to BSD + GPLv2
147 license : fix a few header files to reflect new license (#825)
150 New license : BSD + GPLv2
151 perf: substantially decreased memory usage in Multi-threading mode, thanks to reports by Tino Reichardt (@mcmilk)
152 perf: Multi-threading supports up to 256 threads. Cap at 256 when more are requested (#760)
153 cli : improved and fixed --list command, by @ib (#772)
154 cli : command -vV to list supported formats, by @ib (#771)
155 build : fixed binary variants, reported by @svenha (#788)
156 build : fix Visual compilation for non x86/x64 targets, reported by Greg Slazinski (@GregSlazinski) (#718)
157 API exp : breaking change : ZSTD_getframeHeader() provides more information
158 API exp : breaking change : pinned down values of error codes
159 doc : fixed huffman example, by Ulrich Kunitz (@ulikunitz)
160 new : contrib/adaptive-compression, I/O driven compression strength, by Paul Cruz (@paulcruz74)
161 new : contrib/long_distance_matching, statistics by Stella Lau (@stellamplau)
162 updated : contrib/linux-kernel, by Nick Terrell (@terrelln)
165 cli : new : `--list` command, by Paul Cruz
166 cli : changed : xz/lzma support enabled by default
167 cli : changed : `-t *` continue processing list after a decompression error
168 API : added : ZSTD_versionString()
169 API : promoted to stable status : ZSTD_getFrameContentSize(), by Sean Purcell
170 API exp : new advanced API : ZSTD_compress_generic(), ZSTD_CCtx_setParameter()
171 API exp : new : API for static or external allocation : ZSTD_initStatic?Ctx()
172 API exp : added : ZSTD_decompressBegin_usingDDict(), requested by Guy Riddle (#700)
173 API exp : clarified memory estimation / measurement functions.
174 API exp : changed : strongest strategy renamed ZSTD_btultra, fastest strategy ZSTD_fast set to 1
175 tools : decodecorpus can generate random dictionary-compressed samples, by Paul Cruz
176 new : contrib/seekable_format, demo and API, by Sean Purcell
177 changed : contrib/linux-kernel, updated version and license, by Nick Terrell
180 cli : changed : Multithreading enabled by default (use target zstd-nomt or HAVE_THREAD=0 to disable)
181 cli : new : command -T0 means "detect and use nb of cores", by Sean Purcell
182 cli : new : zstdmt symlink hardwired to `zstd -T0`
183 cli : new : command --threads=# (#671)
184 cli : changed : cover dictionary builder by default, for improved quality, by Nick Terrell
185 cli : new : commands --train-cover and --train-legacy, to select dictionary algorithm and parameters
186 cli : experimental targets `zstd4` and `xzstd4`, with support for lz4 format, by Sean Purcell
187 cli : fix : does not output compressed data on console
188 cli : fix : ignore symbolic links unless --force specified,
189 API : breaking change : ZSTD_createCDict_advanced(), only use compressionParameters as argument
190 API : added : prototypes ZSTD_*_usingCDict_advanced(), for direct control over frameParameters.
191 API : improved: ZSTDMT_compressCCtx() reduced memory usage
192 API : fix : ZSTDMT_compressCCtx() now provides srcSize in header (#634)
193 API : fix : src size stored in frame header is controlled at end of frame
194 API : fix : enforced consistent rules for pledgedSrcSize==0 (#641)
195 API : fix : error code "GENERIC" replaced by "dstSizeTooSmall" when appropriate
196 build: improved cmake script, by @Majlen
197 build: enabled Multi-threading support for *BSD, by Baptiste Daroussin
198 tools: updated Paramgrill. Command -O# provides best parameters for sample and speed target.
199 new : contrib/linux-kernel version, by Nick Terrell
202 cli : new : can compress in *.gz format, using --format=gzip command, by Przemyslaw Skibinski
203 cli : new : advanced benchmark command --priority=rt
204 cli : fix : write on sparse-enabled file systems in 32-bits mode, by @ds77
205 cli : fix : --rm remains silent when input is stdin
206 cli : experimental : xzstd, with support for xz/lzma decoding, by Przemyslaw Skibinski
207 speed : improved decompression speed in streaming mode for single shot scenarios (+5%)
208 memory: DDict (decompression dictionary) memory usage down from 150 KB to 20 KB
209 arch: 32-bits variant able to generate and decode very long matches (>32 MB), by Sean Purcell
210 API : new : ZSTD_findFrameCompressedSize(), ZSTD_getFrameContentSize(), ZSTD_findDecompressedSize()
211 API : changed : dropped support of legacy versions <= v0.3 (can be changed by modifying ZSTD_LEGACY_SUPPORT value)
212 build : new: meson build system in contrib/meson, by Dima Krasner
213 build : improved cmake script, by @Majlen
214 build : added -Wformat-security flag, as recommended by Padraig Brady
215 doc : new : educational decoder, by Sean Purcell
218 cli : zstd can decompress .gz files (can be disabled with `make zstd-nogz` or `make HAVE_ZLIB=0`)
219 cli : new : experimental target `make zstdmt`, with multi-threading support
220 cli : new : improved dictionary builder "cover" (experimental), by Nick Terrell, based on prior work by Giuseppe Ottaviano.
221 cli : new : advanced commands for detailed parameters, by Przemyslaw Skibinski
222 cli : fix zstdless on Mac OS-X, by Andrew Janke
223 cli : fix #232 "compress non-files"
224 dictBuilder : improved dictionary generation quality, thanks to Nick Terrell
225 API : new : lib/compress/ZSTDMT_compress.h multithreading API (experimental)
226 API : new : ZSTD_create?Dict_byReference(), requested by Bartosz Taudul
227 API : new : ZDICT_finalizeDictionary()
228 API : fix : ZSTD_initCStream_usingCDict() properly writes dictID into frame header, by Gregory Szorc (#511)
229 API : fix : all symbols properly exposed in libzstd, by Nick Terrell
230 build : support for Solaris target, by Przemyslaw Skibinski
231 doc : clarified specification, by Sean Purcell
234 API : streaming : decompression : changed : automatic implicit reset when chain-decoding new frames without init
235 API : experimental : added : dictID retrieval functions, and ZSTD_initCStream_srcSize()
236 API : zbuff : changed : prototypes now generate deprecation warnings
237 lib : improved : faster decompression speed at ultra compression settings and 32-bits mode
238 lib : changed : only public ZSTD_ symbols are now exposed
239 lib : changed : reduced usage of stack memory
240 lib : fixed : several corner case bugs, by Nick Terrell
241 cli : new : gzstd, experimental version able to decode .gz files, by Przemyslaw Skibinski
242 cli : new : preserve file attributes
243 cli : new : added zstdless and zstdgrep tools
244 cli : fixed : status displays total amount decoded, even for file consisting of multiple frames (like pzstd)
245 cli : fixed : zstdcat
246 zlib_wrapper : added support for gz* functions, by Przemyslaw Skibinski
247 install : better compatibility with FreeBSD, by Dimitry Andric
248 source tree : changed : zbuff source files moved to lib/deprecated
251 New : command -M#, --memory=, --memlimit=, --memlimit-decompress= to limit allowed memory consumption
252 New : doc/zstd_manual.html, by Przemyslaw Skibinski
253 Improved : slightly better compression ratio at --ultra levels (>= 20)
254 Improved : better memory usage when using streaming compression API, thanks to @Rogier-5 report
255 Added : API : ZSTD_initCStream_usingCDict(), ZSTD_initDStream_usingDDict() (experimental section)
256 Added : example/multiple_streaming_compression.c
257 Changed : zstd_errors.h is now installed within /include (and replaces errors_public.h)
259 Fixed : zstd-small, zstd-compress and zstd-decompress compilation targets
262 New : contrib/pzstd, parallel version of zstd, by Nick Terrell
263 added : NetBSD install target (#338)
264 Improved : speed for batches of small files
265 Improved : speed of zlib wrapper, by Przemyslaw Skibinski
266 Changed : libzstd on Windows supports legacy formats, by Christophe Chevalier
267 Fixed : CLI -d output to stdout by default when input is stdin (#322)
268 Fixed : CLI correctly detects console on Mac OS-X
269 Fixed : CLI supports recursive mode `-r` on Mac OS-X
270 Fixed : Legacy decoders use unified error codes, reported by benrg (#341), fixed by Przemyslaw Skibinski
271 Fixed : compatibility with OpenBSD, reported by Juan Francisco Cantero Hurtado (#319)
272 Fixed : compatibility with Hurd, by Przemyslaw Skibinski (#365)
273 Fixed : zstd-pgo, reported by octoploid (#329)
276 Change Licensing, all project is now BSD, Copyright Facebook
277 Small decompression speed improvement
278 API : Streaming API supports legacy format
279 API : ZDICT_getDictID(), ZSTD_sizeof_{CCtx, DCtx, CStream, DStream}(), ZSTD_setDStreamParameter()
280 CLI supports legacy formats v0.4+
281 Fixed : compression fails on certain huge files, reported by Jesse McGrew
282 Enhanced documentation, by Przemyslaw Skibinski
286 Changed : --ultra now enables levels beyond 19
287 Changed : -i# now selects benchmark time in second
288 Fixed : ZSTD_compress* can now compress > 4 GB in a single pass, reported by Nick Terrell
289 Fixed : speed regression on specific patterns (#272)
290 Fixed : support for Z_SYNC_FLUSH, by Dmitry Krot (#291)
291 Fixed : ICC compilation, by Przemyslaw Skibinski
294 Improved : better speed on clang and gcc -O2, thanks to Eric Biggers
295 New : Build on FreeBSD and DragonFly, thanks to JrMarino
296 Changed : modified API : ZSTD_compressEnd()
297 Fixed : legacy mode with ZSTD_HEAPMODE=0, by Christopher Bergqvist
298 Fixed : premature end of frame when zero-sized raw block, reported by Eric Biggers
299 Fixed : large dictionaries (> 384 KB), reported by Ilona Papava
300 Fixed : checksum correctly checked in single-pass mode
301 Fixed : combined --test amd --rm, reported by Andreas M. Nilsson
302 Modified : minor compression level adaptations
303 Updated : compression format specification to v0.2.0
304 changed : zstd.h moved to /lib directory
307 Transition version, supporting decoding of v0.8.x
310 Added : homebrew for Mac, by Daniel Cade
311 Added : more examples
312 Fixed : segfault when using small dictionaries, reported by Felix Handte
313 Modified : default compression level for CLI is now 3
314 Updated : specification, to v0.1.1
317 New : compression format specification
318 New : `--` separator, stating that all following arguments are file names. Suggested by Chip Turner.
319 New : `ZSTD_getDecompressedSize()`
320 New : OpenBSD target, by Juan Francisco Cantero Hurtado
321 New : `examples` directory
322 fixed : dictBuilder using HC levels, reported by Bartosz Taudul
323 fixed : legacy support from ZSTD_decompress_usingDDict(), reported by Felix Handte
324 fixed : multi-blocks decoding with intermediate uncompressed blocks, reported by Greg Slazinski
325 modified : removed "mem.h" and "error_public.h" dependencies from "zstd.h" (experimental section)
326 modified : legacy functions no longer need magic number
329 fixed : ZSTD_decompressBlock() using multiple consecutive blocks. Reported by Greg Slazinski.
330 fixed : potential segfault on very large files (many gigabytes). Reported by Chip Turner.
331 fixed : CLI displays system error message when destination file cannot be created (#231). Reported by Chip Turner.
334 fixed : ZBUFF_compressEnd() called multiple times with too small `dst` buffer, reported by Christophe Chevalier
335 fixed : dictBuilder fails if first sample is too small, reported by Руслан Ковалёв
336 fixed : corruption issue, reported by cj
337 modified : checksum enabled by default in command line mode
340 New : Support for directory compression, using `-r`, thanks to Przemyslaw Skibinski
341 New : Command `--rm`, to remove source file after successful de/compression
342 New : Visual build scripts, by Christophe Chevalier
343 New : Support for Sparse File-systems (do not use space for zero-filled sectors)
344 New : Frame checksum support
345 New : Support pass-through mode (when using `-df`)
346 API : more efficient Dictionary API : `ZSTD_compress_usingCDict()`, `ZSTD_decompress_usingDDict()`
347 API : create dictionary files from custom content, by Giuseppe Ottaviano
348 API : support for custom malloc/free functions
349 New : controllable Dictionary ID
350 New : Support for skippable frames
353 New : zlib wrapper API, thanks to Przemyslaw Skibinski
354 New : Ability to compile compressor / decompressor separately
355 Changed : new lib directory structure
356 Fixed : Legacy codec v0.5 compatible with dictionary decompression
357 Fixed : Decoder corruption error (#173)
358 Fixed : null-string roundtrip (#176)
359 New : benchmark mode can select directory as input
360 Experimental : midipix support, VMS support
363 Stronger high compression modes, thanks to Przemyslaw Skibinski
364 API : ZSTD_getFrameParams() provides size of decompressed content
365 New : highest compression modes require `--ultra` command to fully unleash their capacity
366 Fixed : zstd cli return error code > 0 and removes dst file artifact when decompression fails, thanks to Chip Turner
369 New : Optimal parsing => Very high compression modes, thanks to Przemyslaw Skibinski
370 Changed : Dictionary builder integrated into libzstd and zstd cli
371 Changed (!) : zstd cli now uses "multiple input files" as default mode. See `zstd -h`.
372 Fix : high compression modes for big-endian platforms
373 New : zstd cli : `-t` | `--test` command
376 New : dictionary builder utility
377 Changed : streaming & dictionary API
378 Improved : better compression of small data
381 Improved : small compression speed improvement in HC mode
382 Changed : `zstd_decompress.c` has ZSTD_LEGACY_SUPPORT to 0 by default
386 fix : fast compression mode on Windows
387 New : cmake configuration file, thanks to Artyom Dymchenko
388 Improved : high compression mode on repetitive data
389 New : block-level API
390 New : ZSTD_duplicateCCtx()
393 new : -m/--multiple : compress/decompress multiple files
396 Fixed : high compression modes for Windows 32 bits
397 new : external dictionary API extended to buffered mode and accessible through command line
398 new : windows DLL project, thanks to Christophe Chevalier
401 new : external dictionary API
405 Generic minor improvements for small blocks
406 Fixed : big-endian compatibility, by Peter Harris (#85)
409 Fixed : ZSTD_LEGACY_SUPPORT=0 build mode (reported by Luben)
413 Command line utility compatible with high compression levels
414 Removed zstdhc => merged into zstd
415 Added : ZBUFF API (see zstd_buffered.h)
416 Rolling buffer support
422 minor generic compression improvements
428 Small compression ratio improvement
434 Small compression ratio improvement
437 HC mode : compression levels 2-26
440 Fix : Visual Studio 2013 & 2015 release compilation, by Christophe Chevalier
443 Fix : Read errors, advanced fuzzer tests, by Hanno Böck
446 **Breaking format change**
447 Faster decompression speed
448 Can still decode v0.1 format
451 fix uninitialization warning, reported by Evan Nemerson
454 frame concatenation support
458 detects write-flush errors