diff options
| author | Conrad Meyer <cem@FreeBSD.org> | 2019-11-06 06:42:00 +0000 |
|---|---|---|
| committer | Conrad Meyer <cem@FreeBSD.org> | 2019-11-06 06:42:00 +0000 |
| commit | ea68403922c3b53b00fc999fcb3eaef1feb50177 (patch) | |
| tree | 9870b0c695852e26fb9f8e19df8d7f5cb6616141 /doc/zstd_manual.html | |
| parent | 90f4bdbe917eaf678feca2b0ff9647b5ae8bbb9d (diff) | |
Notes
Diffstat (limited to 'doc/zstd_manual.html')
| -rw-r--r-- | doc/zstd_manual.html | 328 |
1 files changed, 205 insertions, 123 deletions
diff --git a/doc/zstd_manual.html b/doc/zstd_manual.html index 6d8e74fe52746..43c5555b8ca85 100644 --- a/doc/zstd_manual.html +++ b/doc/zstd_manual.html @@ -1,10 +1,10 @@ <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"> -<title>zstd 1.4.2 Manual</title> +<title>zstd 1.4.4 Manual</title> </head> <body> -<h1>zstd 1.4.2 Manual</h1> +<h1>zstd 1.4.4 Manual</h1> <hr> <a name="Contents"></a><h2>Contents</h2> <ol> @@ -27,10 +27,16 @@ <li><a href="#Chapter17">Advanced compression functions</a></li> <li><a href="#Chapter18">Advanced decompression functions</a></li> <li><a href="#Chapter19">Advanced streaming functions</a></li> -<li><a href="#Chapter20">Buffer-less and synchronous inner streaming functions</a></li> -<li><a href="#Chapter21">Buffer-less streaming compression (synchronous mode)</a></li> -<li><a href="#Chapter22">Buffer-less streaming decompression (synchronous mode)</a></li> -<li><a href="#Chapter23">Block level API</a></li> +<li><a href="#Chapter20">! ZSTD_initCStream_usingDict() :</a></li> +<li><a href="#Chapter21">! ZSTD_initCStream_advanced() :</a></li> +<li><a href="#Chapter22">! ZSTD_initCStream_usingCDict() :</a></li> +<li><a href="#Chapter23">! ZSTD_initCStream_usingCDict_advanced() :</a></li> +<li><a href="#Chapter24">This function is deprecated, and is equivalent to:</a></li> +<li><a href="#Chapter25">This function is deprecated, and is equivalent to:</a></li> +<li><a href="#Chapter26">Buffer-less and synchronous inner streaming functions</a></li> +<li><a href="#Chapter27">Buffer-less streaming compression (synchronous mode)</a></li> +<li><a href="#Chapter28">Buffer-less streaming decompression (synchronous mode)</a></li> +<li><a href="#Chapter29">Block level API</a></li> </ol> <hr> <a name="Chapter1"></a><h2>Introduction</h2><pre> @@ -157,9 +163,13 @@ size_t ZSTD_freeCCtx(ZSTD_CCtx* cctx); void* dst, size_t dstCapacity, const void* src, size_t srcSize, int compressionLevel); -</b><p> Same as ZSTD_compress(), using an explicit ZSTD_CCtx - The function will compress at requested compression level, - ignoring any other parameter +</b><p> Same as ZSTD_compress(), using an explicit ZSTD_CCtx. + Important : in order to behave similarly to `ZSTD_compress()`, + this function compresses at requested compression level, + __ignoring any other parameter__ . + If any advanced parameter was set using the advanced API, + they will all be reset. Only `compressionLevel` remains. + </p></pre><BR> <h3>Decompression context</h3><pre> When decompressing many times, @@ -199,18 +209,26 @@ size_t ZSTD_freeDCtx(ZSTD_DCtx* dctx); </b>/* compression parameters<b> * Note: When compressing with a ZSTD_CDict these parameters are superseded - * by the parameters used to construct the ZSTD_CDict. See ZSTD_CCtx_refCDict() - * for more info (superseded-by-cdict). */ - ZSTD_c_compressionLevel=100, </b>/* Update all compression parameters according to pre-defined cLevel table<b> + * by the parameters used to construct the ZSTD_CDict. + * See ZSTD_CCtx_refCDict() for more info (superseded-by-cdict). */ + ZSTD_c_compressionLevel=100, </b>/* Set compression parameters according to pre-defined cLevel table.<b> + * Note that exact compression parameters are dynamically determined, + * depending on both compression level and srcSize (when known). * Default level is ZSTD_CLEVEL_DEFAULT==3. * Special: value 0 means default, which is controlled by ZSTD_CLEVEL_DEFAULT. * Note 1 : it's possible to pass a negative compression level. - * Note 2 : setting a level sets all default values of other compression parameters */ + * Note 2 : setting a level resets all other compression parameters to default */ + </b>/* Advanced compression parameters :<b> + * It's possible to pin down compression parameters to some specific values. + * In which case, these values are no longer dynamically selected by the compressor */ ZSTD_c_windowLog=101, </b>/* Maximum allowed back-reference distance, expressed as power of 2.<b> + * This will set a memory budget for streaming decompression, + * with larger values requiring more memory + * and typically compressing more. * Must be clamped between ZSTD_WINDOWLOG_MIN and ZSTD_WINDOWLOG_MAX. * Special: value 0 means "use default windowLog". * Note: Using a windowLog greater than ZSTD_WINDOWLOG_LIMIT_DEFAULT - * requires explicitly allowing such window size at decompression stage if using streaming. */ + * requires explicitly allowing such size at streaming decompression stage. */ ZSTD_c_hashLog=102, </b>/* Size of the initial probe table, as a power of 2.<b> * Resulting memory usage is (1 << (hashLog+2)). * Must be clamped between ZSTD_HASHLOG_MIN and ZSTD_HASHLOG_MAX. @@ -221,13 +239,13 @@ size_t ZSTD_freeDCtx(ZSTD_DCtx* dctx); * Resulting memory usage is (1 << (chainLog+2)). * Must be clamped between ZSTD_CHAINLOG_MIN and ZSTD_CHAINLOG_MAX. * Larger tables result in better and slower compression. - * This parameter is useless when using "fast" strategy. + * This parameter is useless for "fast" strategy. * It's still useful when using "dfast" strategy, * in which case it defines a secondary probe table. * Special: value 0 means "use default chainLog". */ ZSTD_c_searchLog=104, </b>/* Number of search attempts, as a power of 2.<b> * More attempts result in better and slower compression. - * This parameter is useless when using "fast" and "dFast" strategies. + * This parameter is useless for "fast" and "dFast" strategies. * Special: value 0 means "use default searchLog". */ ZSTD_c_minMatch=105, </b>/* Minimum size of searched matches.<b> * Note that Zstandard can still find matches of smaller size, @@ -282,7 +300,7 @@ size_t ZSTD_freeDCtx(ZSTD_DCtx* dctx); ZSTD_c_contentSizeFlag=200, </b>/* Content size will be written into frame header _whenever known_ (default:1)<b> * Content size must be known at the beginning of compression. * This is automatically the case when using ZSTD_compress2(), - * For streaming variants, content size must be provided with ZSTD_CCtx_setPledgedSrcSize() */ + * For streaming scenarios, content size must be provided with ZSTD_CCtx_setPledgedSrcSize() */ ZSTD_c_checksumFlag=201, </b>/* A 32-bits checksum of content is written at end of frame (default:0) */<b> ZSTD_c_dictIDFlag=202, </b>/* When applicable, dictionary's ID is written into frame header (default:1) */<b> @@ -301,7 +319,7 @@ size_t ZSTD_freeDCtx(ZSTD_DCtx* dctx); * Each compression job is completed in parallel, so this value can indirectly impact the nb of active threads. * 0 means default, which is dynamically determined based on compression parameters. * Job size must be a minimum of overlap size, or 1 MB, whichever is largest. - * The minimum size is automatically and transparently enforced */ + * The minimum size is automatically and transparently enforced. */ ZSTD_c_overlapLog=402, </b>/* Control the overlap size, as a fraction of window size.<b> * The overlap size is an amount of data reloaded from previous job at the beginning of a new job. * It helps preserve compression ratio, while each job is compressed in parallel. @@ -324,6 +342,7 @@ size_t ZSTD_freeDCtx(ZSTD_DCtx* dctx); * ZSTD_c_forceAttachDict * ZSTD_c_literalCompressionMode * ZSTD_c_targetCBlockSize + * ZSTD_c_srcSizeHint * Because they are not stable, it's necessary to define ZSTD_STATIC_LINKING_ONLY to access them. * note : never ever use experimentalParam? names directly; * also, the enums values themselves are unstable and can still change. @@ -334,6 +353,7 @@ size_t ZSTD_freeDCtx(ZSTD_DCtx* dctx); ZSTD_c_experimentalParam4=1001, ZSTD_c_experimentalParam5=1002, ZSTD_c_experimentalParam6=1003, + ZSTD_c_experimentalParam7=1004 } ZSTD_cParameter; </b></pre><BR> <pre><b>typedef struct { @@ -672,12 +692,17 @@ size_t ZSTD_freeDStream(ZSTD_DStream* zds); <pre><b>ZSTD_CDict* ZSTD_createCDict(const void* dictBuffer, size_t dictSize, int compressionLevel); -</b><p> When compressing multiple messages / blocks using the same dictionary, it's recommended to load it only once. - ZSTD_createCDict() will create a digested dictionary, ready to start future compression operations without startup cost. +</b><p> When compressing multiple messages or blocks using the same dictionary, + it's recommended to digest the dictionary only once, since it's a costly operation. + ZSTD_createCDict() will create a state from digesting a dictionary. + The resulting state can be used for future compression operations with very limited startup cost. ZSTD_CDict can be created once and shared by multiple threads concurrently, since its usage is read-only. - `dictBuffer` can be released after ZSTD_CDict creation, because its content is copied within CDict. - Consider experimental function `ZSTD_createCDict_byReference()` if you prefer to not duplicate `dictBuffer` content. - Note : A ZSTD_CDict can be created from an empty dictBuffer, but it is inefficient when used to compress small data. + @dictBuffer can be released after ZSTD_CDict creation, because its content is copied within CDict. + Note 1 : Consider experimental function `ZSTD_createCDict_byReference()` if you prefer to not duplicate @dictBuffer content. + Note 2 : A ZSTD_CDict can be created from an empty @dictBuffer, + in which case the only thing that it transports is the @compressionLevel. + This can be useful in a pipeline featuring ZSTD_compress_usingCDict() exclusively, + expecting a ZSTD_CDict parameter with any data, including those without a known dictionary. </p></pre><BR> <pre><b>size_t ZSTD_freeCDict(ZSTD_CDict* CDict); @@ -794,7 +819,7 @@ size_t ZSTD_freeDStream(ZSTD_DStream* zds); Note 3 : Referencing a prefix involves building tables, which are dependent on compression parameters. It's a CPU consuming operation, with non-negligible impact on latency. If there is a need to use the same prefix multiple times, consider loadDictionary instead. - Note 4 : By default, the prefix is interpreted as raw content (ZSTD_dm_rawContent). + Note 4 : By default, the prefix is interpreted as raw content (ZSTD_dct_rawContent). Use experimental ZSTD_CCtx_refPrefix_advanced() to alter dictionary interpretation. </p></pre><BR> @@ -838,7 +863,7 @@ size_t ZSTD_freeDStream(ZSTD_DStream* zds); Note 2 : Prefix buffer is referenced. It **must** outlive decompression. Prefix buffer must remain unmodified up to the end of frame, reached when ZSTD_decompressStream() returns 0. - Note 3 : By default, the prefix is treated as raw content (ZSTD_dm_rawContent). + Note 3 : By default, the prefix is treated as raw content (ZSTD_dct_rawContent). Use ZSTD_CCtx_refPrefix_advanced() to alter dictMode (Experimental section) Note 4 : Referencing a raw content prefix has almost no cpu nor memory cost. A full dictionary is more costly, as it requires building tables. @@ -865,6 +890,24 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict); <BR></pre> <pre><b>typedef struct { + unsigned int matchPos; </b>/* Match pos in dst */<b> + </b>/* If seqDef.offset > 3, then this is seqDef.offset - 3<b> + * If seqDef.offset < 3, then this is the corresponding repeat offset + * But if seqDef.offset < 3 and litLength == 0, this is the + * repeat offset before the corresponding repeat offset + * And if seqDef.offset == 3 and litLength == 0, this is the + * most recent repeat offset - 1 + */ + unsigned int offset; + unsigned int litLength; </b>/* Literal length */<b> + unsigned int matchLength; </b>/* Match length */<b> + </b>/* 0 when seq not rep and seqDef.offset otherwise<b> + * when litLength == 0 this will be <= 4, otherwise <= 3 like normal + */ + unsigned int rep; +} ZSTD_Sequence; +</b></pre><BR> +<pre><b>typedef struct { unsigned windowLog; </b>/**< largest match distance : larger == more compression, more memory needed during decompression */<b> unsigned chainLog; </b>/**< fully searched segment : larger == more compression, slower, more memory (useless for fast) */<b> unsigned hashLog; </b>/**< dispatch table : larger == faster, more memory */<b> @@ -893,21 +936,12 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict); </b></pre><BR> <pre><b>typedef enum { ZSTD_dlm_byCopy = 0, </b>/**< Copy dictionary content internally */<b> - ZSTD_dlm_byRef = 1, </b>/**< Reference dictionary content -- the dictionary buffer must outlive its users. */<b> + ZSTD_dlm_byRef = 1 </b>/**< Reference dictionary content -- the dictionary buffer must outlive its users. */<b> } ZSTD_dictLoadMethod_e; </b></pre><BR> <pre><b>typedef enum { - </b>/* Opened question : should we have a format ZSTD_f_auto ?<b> - * Today, it would mean exactly the same as ZSTD_f_zstd1. - * But, in the future, should several formats become supported, - * on the compression side, it would mean "default format". - * On the decompression side, it would mean "automatic format detection", - * so that ZSTD_f_zstd1 would mean "accept *only* zstd frames". - * Since meaning is a little different, another option could be to define different enums for compression and decompression. - * This question could be kept for later, when there are actually multiple formats to support, - * but there is also the question of pinning enum values, and pinning value `0` is especially important */ ZSTD_f_zstd1 = 0, </b>/* zstd frame format, specified in zstd_compression_format.md (default) */<b> - ZSTD_f_zstd1_magicless = 1, </b>/* Variant of zstd frame format, without initial 4-bytes magic number.<b> + ZSTD_f_zstd1_magicless = 1 </b>/* Variant of zstd frame format, without initial 4-bytes magic number.<b> * Useful to save 4 bytes per generated frame. * Decoder cannot recognise automatically this format, requiring this instruction. */ } ZSTD_format_e; @@ -918,7 +952,7 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict); * to evolve and should be considered only in the context of extremely * advanced performance tuning. * - * Zstd currently supports the use of a CDict in two ways: + * Zstd currently supports the use of a CDict in three ways: * * - The contents of the CDict can be copied into the working context. This * means that the compression can search both the dictionary and input @@ -934,6 +968,12 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict); * working context's tables can be reused). For small inputs, this can be * faster than copying the CDict's tables. * + * - The CDict's tables are not used at all, and instead we use the working + * context alone to reload the dictionary and use params based on the source + * size. See ZSTD_compress_insertDictionary() and ZSTD_compress_usingDict(). + * This method is effective when the dictionary sizes are very small relative + * to the input size, and the input size is fairly large to begin with. + * * Zstd has a simple internal heuristic that selects which strategy to use * at the beginning of a compression. However, if experimentation shows that * Zstd is making poor choices, it is possible to override that choice with @@ -942,6 +982,7 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict); ZSTD_dictDefaultAttach = 0, </b>/* Use the default heuristic. */<b> ZSTD_dictForceAttach = 1, </b>/* Never copy the dictionary. */<b> ZSTD_dictForceCopy = 2, </b>/* Always copy the dictionary. */<b> + ZSTD_dictForceLoad = 3 </b>/* Always reload the dictionary */<b> } ZSTD_dictAttachPref_e; </b></pre><BR> <pre><b>typedef enum { @@ -950,7 +991,7 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict); * levels will be compressed. */ ZSTD_lcm_huffman = 1, </b>/**< Always attempt Huffman compression. Uncompressed literals will still be<b> * emitted if Huffman compression is not profitable. */ - ZSTD_lcm_uncompressed = 2, </b>/**< Always emit uncompressed literals. */<b> + ZSTD_lcm_uncompressed = 2 </b>/**< Always emit uncompressed literals. */<b> } ZSTD_literalCompressionMode_e; </b></pre><BR> <a name="Chapter15"></a><h2>Frame size functions</h2><pre></pre> @@ -999,20 +1040,38 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict); or an error code (if srcSize is too small) </p></pre><BR> +<pre><b>size_t ZSTD_getSequences(ZSTD_CCtx* zc, ZSTD_Sequence* outSeqs, + size_t outSeqsSize, const void* src, size_t srcSize); +</b><p> Extract sequences from the sequence store + zc can be used to insert custom compression params. + This function invokes ZSTD_compress2 + @return : number of sequences extracted + +</p></pre><BR> + <a name="Chapter16"></a><h2>Memory management</h2><pre></pre> <pre><b>size_t ZSTD_estimateCCtxSize(int compressionLevel); size_t ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams); size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params); size_t ZSTD_estimateDCtxSize(void); -</b><p> These functions make it possible to estimate memory usage - of a future {D,C}Ctx, before its creation. - ZSTD_estimateCCtxSize() will provide a budget large enough for any compression level up to selected one. - It will also consider src size to be arbitrarily "large", which is worst case. - If srcSize is known to always be small, ZSTD_estimateCCtxSize_usingCParams() can provide a tighter estimation. - ZSTD_estimateCCtxSize_usingCParams() can be used in tandem with ZSTD_getCParams() to create cParams from compressionLevel. - ZSTD_estimateCCtxSize_usingCCtxParams() can be used in tandem with ZSTD_CCtxParams_setParameter(). Only single-threaded compression is supported. This function will return an error code if ZSTD_c_nbWorkers is >= 1. - Note : CCtx size estimation is only correct for single-threaded compression. +</b><p> These functions make it possible to estimate memory usage of a future + {D,C}Ctx, before its creation. + + ZSTD_estimateCCtxSize() will provide a budget large enough for any + compression level up to selected one. Unlike ZSTD_estimateCStreamSize*(), + this estimate does not include space for a window buffer, so this estimate + is guaranteed to be enough for single-shot compressions, but not streaming + compressions. It will however assume the input may be arbitrarily large, + which is the worst case. If srcSize is known to always be small, + ZSTD_estimateCCtxSize_usingCParams() can provide a tighter estimation. + ZSTD_estimateCCtxSize_usingCParams() can be used in tandem with + ZSTD_getCParams() to create cParams from compressionLevel. + ZSTD_estimateCCtxSize_usingCCtxParams() can be used in tandem with + ZSTD_CCtxParams_setParameter(). + + Note: only single-threaded compression is supported. This function will + return an error code if ZSTD_c_nbWorkers is >= 1. </p></pre><BR> <pre><b>size_t ZSTD_estimateCStreamSize(int compressionLevel); @@ -1085,7 +1144,8 @@ static ZSTD_customMem const ZSTD_defaultCMem = { NULL, NULL, NULL }; </b>/**< t </b><p> Create a digested dictionary for compression Dictionary content is just referenced, not duplicated. As a consequence, `dictBuffer` **must** outlive CDict, - and its content must remain unmodified throughout the lifetime of CDict. + and its content must remain unmodified throughout the lifetime of CDict. + note: equivalent to ZSTD_createCDict_advanced(), with dictLoadMethod==ZSTD_dlm_byRef </p></pre><BR> <pre><b>ZSTD_compressionParameters ZSTD_getCParams(int compressionLevel, unsigned long long estimatedSrcSize, size_t dictSize); @@ -1116,7 +1176,9 @@ static ZSTD_customMem const ZSTD_defaultCMem = { NULL, NULL, NULL }; </b>/**< t const void* src, size_t srcSize, const void* dict,size_t dictSize, ZSTD_parameters params); -</b><p> Same as ZSTD_compress_usingDict(), with fine-tune control over compression parameters (by structure) +</b><p> Note : this function is now DEPRECATED. + It can be replaced by ZSTD_compress2(), in combination with ZSTD_CCtx_setParameter() and other parameter setters. + This prototype will be marked as deprecated and generate compilation warning on reaching v1.5.x </p></pre><BR> <pre><b>size_t ZSTD_compress_usingCDict_advanced(ZSTD_CCtx* cctx, @@ -1124,7 +1186,9 @@ static ZSTD_customMem const ZSTD_defaultCMem = { NULL, NULL, NULL }; </b>/**< t const void* src, size_t srcSize, const ZSTD_CDict* cdict, ZSTD_frameParameters fParams); -</b><p> Same as ZSTD_compress_usingCDict(), with fine-tune control over frame parameters +</b><p> Note : this function is now REDUNDANT. + It can be replaced by ZSTD_compress2(), in combination with ZSTD_CCtx_loadDictionary() and other parameter setters. + This prototype will be marked as deprecated and generate compilation warning in some future version </p></pre><BR> <pre><b>size_t ZSTD_CCtx_loadDictionary_byReference(ZSTD_CCtx* cctx, const void* dict, size_t dictSize); @@ -1301,53 +1365,67 @@ size_t ZSTD_freeCCtxParams(ZSTD_CCtx_params* params); * pledgedSrcSize must be correct. If it is not known at init time, use * ZSTD_CONTENTSIZE_UNKNOWN. Note that, for compatibility with older programs, * "0" also disables frame content size field. It may be enabled in the future. + * Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x */ -size_t ZSTD_initCStream_srcSize(ZSTD_CStream* zcs, int compressionLevel, unsigned long long pledgedSrcSize); -</b>/**! ZSTD_initCStream_usingDict() :<b> - * This function is deprecated, and is equivalent to: - * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); - * ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel); - * ZSTD_CCtx_loadDictionary(zcs, dict, dictSize); - * - * Creates of an internal CDict (incompatible with static CCtx), except if - * dict == NULL or dictSize < 8, in which case no dict is used. - * Note: dict is loaded with ZSTD_dm_auto (treated as a full zstd dictionary if - * it begins with ZSTD_MAGIC_DICTIONARY, else as raw content) and ZSTD_dlm_byCopy. - */ -size_t ZSTD_initCStream_usingDict(ZSTD_CStream* zcs, const void* dict, size_t dictSize, int compressionLevel); -</b>/**! ZSTD_initCStream_advanced() :<b> - * This function is deprecated, and is approximately equivalent to: - * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); - * ZSTD_CCtx_setZstdParams(zcs, params); // Set the zstd params and leave the rest as-is - * ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize); - * ZSTD_CCtx_loadDictionary(zcs, dict, dictSize); - * - * pledgedSrcSize must be correct. If srcSize is not known at init time, use - * value ZSTD_CONTENTSIZE_UNKNOWN. dict is loaded with ZSTD_dm_auto and ZSTD_dlm_byCopy. - */ -size_t ZSTD_initCStream_advanced(ZSTD_CStream* zcs, const void* dict, size_t dictSize, - ZSTD_parameters params, unsigned long long pledgedSrcSize); -</b>/**! ZSTD_initCStream_usingCDict() :<b> - * This function is deprecated, and equivalent to: - * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); - * ZSTD_CCtx_refCDict(zcs, cdict); - * - * note : cdict will just be referenced, and must outlive compression session - */ -size_t ZSTD_initCStream_usingCDict(ZSTD_CStream* zcs, const ZSTD_CDict* cdict); -</b>/**! ZSTD_initCStream_usingCDict_advanced() :<b> - * This function is deprecated, and is approximately equivalent to: - * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); - * ZSTD_CCtx_setZstdFrameParams(zcs, fParams); // Set the zstd frame params and leave the rest as-is - * ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize); - * ZSTD_CCtx_refCDict(zcs, cdict); - * - * same as ZSTD_initCStream_usingCDict(), with control over frame parameters. - * pledgedSrcSize must be correct. If srcSize is not known at init time, use - * value ZSTD_CONTENTSIZE_UNKNOWN. - */ -size_t ZSTD_initCStream_usingCDict_advanced(ZSTD_CStream* zcs, const ZSTD_CDict* cdict, ZSTD_frameParameters fParams, unsigned long long pledgedSrcSize); +size_t +ZSTD_initCStream_srcSize(ZSTD_CStream* zcs, + int compressionLevel, + unsigned long long pledgedSrcSize); </pre></b><BR> +<a name="Chapter20"></a><h2>! ZSTD_initCStream_usingDict() :</h2><pre> This function is deprecated, and is equivalent to: + ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); + ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel); + ZSTD_CCtx_loadDictionary(zcs, dict, dictSize); + + Creates of an internal CDict (incompatible with static CCtx), except if + dict == NULL or dictSize < 8, in which case no dict is used. + Note: dict is loaded with ZSTD_dct_auto (treated as a full zstd dictionary if + it begins with ZSTD_MAGIC_DICTIONARY, else as raw content) and ZSTD_dlm_byCopy. + Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x + +<BR></pre> + +<a name="Chapter21"></a><h2>! ZSTD_initCStream_advanced() :</h2><pre> This function is deprecated, and is approximately equivalent to: + ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); + // Pseudocode: Set each zstd parameter and leave the rest as-is. + for ((param, value) : params) { + ZSTD_CCtx_setParameter(zcs, param, value); + } + ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize); + ZSTD_CCtx_loadDictionary(zcs, dict, dictSize); + + dict is loaded with ZSTD_dct_auto and ZSTD_dlm_byCopy. + pledgedSrcSize must be correct. + If srcSize is not known at init time, use value ZSTD_CONTENTSIZE_UNKNOWN. + Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x + +<BR></pre> + +<a name="Chapter22"></a><h2>! ZSTD_initCStream_usingCDict() :</h2><pre> This function is deprecated, and equivalent to: + ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); + ZSTD_CCtx_refCDict(zcs, cdict); + + note : cdict will just be referenced, and must outlive compression session + Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x + +<BR></pre> + +<a name="Chapter23"></a><h2>! ZSTD_initCStream_usingCDict_advanced() :</h2><pre> This function is DEPRECATED, and is approximately equivalent to: + ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); + // Pseudocode: Set each zstd frame parameter and leave the rest as-is. + for ((fParam, value) : fParams) { + ZSTD_CCtx_setParameter(zcs, fParam, value); + } + ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize); + ZSTD_CCtx_refCDict(zcs, cdict); + + same as ZSTD_initCStream_usingCDict(), with control over frame parameters. + pledgedSrcSize must be correct. If srcSize is not known at init time, use + value ZSTD_CONTENTSIZE_UNKNOWN. + Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x + +<BR></pre> + <pre><b>size_t ZSTD_resetCStream(ZSTD_CStream* zcs, unsigned long long pledgedSrcSize); </b><p> This function is deprecated, and is equivalent to: ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); @@ -1361,6 +1439,7 @@ size_t ZSTD_initCStream_usingCDict_advanced(ZSTD_CStream* zcs, const ZSTD_CDict* For the time being, pledgedSrcSize==0 is interpreted as "srcSize unknown" for compatibility with older programs, but it will change to mean "empty" in future version, so use macro ZSTD_CONTENTSIZE_UNKNOWN instead. @return : 0, or an error code (which can be tested using ZSTD_isError()) + Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x </p></pre><BR> @@ -1395,34 +1474,35 @@ size_t ZSTD_initCStream_usingCDict_advanced(ZSTD_CStream* zcs, const ZSTD_CDict* * ZSTD_DCtx_loadDictionary(zds, dict, dictSize); * * note: no dictionary will be used if dict == NULL or dictSize < 8 + * Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x */ size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dict, size_t dictSize); -</b>/**<b> - * This function is deprecated, and is equivalent to: - * - * ZSTD_DCtx_reset(zds, ZSTD_reset_session_only); - * ZSTD_DCtx_refDDict(zds, ddict); - * - * note : ddict is referenced, it must outlive decompression session - */ -size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* zds, const ZSTD_DDict* ddict); -</b>/**<b> - * This function is deprecated, and is equivalent to: - * - * ZSTD_DCtx_reset(zds, ZSTD_reset_session_only); - * - * re-use decompression parameters from previous init; saves dictionary loading - */ -size_t ZSTD_resetDStream(ZSTD_DStream* zds); </pre></b><BR> -<a name="Chapter20"></a><h2>Buffer-less and synchronous inner streaming functions</h2><pre> +<a name="Chapter24"></a><h2>This function is deprecated, and is equivalent to:</h2><pre> + ZSTD_DCtx_reset(zds, ZSTD_reset_session_only); + ZSTD_DCtx_refDDict(zds, ddict); + + note : ddict is referenced, it must outlive decompression session + Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x + +<BR></pre> + +<a name="Chapter25"></a><h2>This function is deprecated, and is equivalent to:</h2><pre> + ZSTD_DCtx_reset(zds, ZSTD_reset_session_only); + + re-use decompression parameters from previous init; saves dictionary loading + Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x + +<BR></pre> + +<a name="Chapter26"></a><h2>Buffer-less and synchronous inner streaming functions</h2><pre> This is an advanced API, giving full control over buffer management, for users which need direct control over memory. But it's also a complex one, with several restrictions, documented below. Prefer normal streaming API for an easier experience. <BR></pre> -<a name="Chapter21"></a><h2>Buffer-less streaming compression (synchronous mode)</h2><pre> +<a name="Chapter27"></a><h2>Buffer-less streaming compression (synchronous mode)</h2><pre> A ZSTD_CCtx object is required to track streaming operations. Use ZSTD_createCCtx() / ZSTD_freeCCtx() to manage resource. ZSTD_CCtx object can be re-used multiple times within successive compression operations. @@ -1458,7 +1538,7 @@ size_t ZSTD_compressBegin_usingCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict); size_t ZSTD_compressBegin_usingCDict_advanced(ZSTD_CCtx* const cctx, const ZSTD_CDict* const cdict, ZSTD_frameParameters const fParams, unsigned long long const pledgedSrcSize); </b>/* compression parameters are already set within cdict. pledgedSrcSize must be correct. If srcSize is not known, use macro ZSTD_CONTENTSIZE_UNKNOWN */<b> size_t ZSTD_copyCCtx(ZSTD_CCtx* cctx, const ZSTD_CCtx* preparedCCtx, unsigned long long pledgedSrcSize); </b>/**< note: if pledgedSrcSize is not known, use ZSTD_CONTENTSIZE_UNKNOWN */<b> </pre></b><BR> -<a name="Chapter22"></a><h2>Buffer-less streaming decompression (synchronous mode)</h2><pre> +<a name="Chapter28"></a><h2>Buffer-less streaming decompression (synchronous mode)</h2><pre> A ZSTD_DCtx object is required to track streaming operations. Use ZSTD_createDCtx() / ZSTD_freeDCtx() to manage it. A ZSTD_DCtx object can be re-used multiple times. @@ -1554,10 +1634,10 @@ size_t ZSTD_decodingBufferSize_min(unsigned long long windowSize, unsigned long <pre><b>typedef enum { ZSTDnit_frameHeader, ZSTDnit_blockHeader, ZSTDnit_block, ZSTDnit_lastBlock, ZSTDnit_checksum, ZSTDnit_skippableFrame } ZSTD_nextInputType_e; </b></pre><BR> -<a name="Chapter23"></a><h2>Block level API</h2><pre></pre> +<a name="Chapter29"></a><h2>Block level API</h2><pre></pre> -<pre><b></b><p> Frame metadata cost is typically ~18 bytes, which can be non-negligible for very small blocks (< 100 bytes). - User will have to take in charge required information to regenerate data, such as compressed and content sizes. +<pre><b></b><p> Frame metadata cost is typically ~12 bytes, which can be non-negligible for very small blocks (< 100 bytes). + But users will have to take in charge needed metadata to regenerate data, such as compressed and content sizes. A few rules to respect : - Compressing and decompressing require a context structure @@ -1568,12 +1648,14 @@ size_t ZSTD_decodingBufferSize_min(unsigned long long windowSize, unsigned long + copyCCtx() and copyDCtx() can be used too - Block size is limited, it must be <= ZSTD_getBlockSize() <= ZSTD_BLOCKSIZE_MAX == 128 KB + If input is larger than a block size, it's necessary to split input data into multiple blocks - + For inputs larger than a single block, really consider using regular ZSTD_compress() instead. - Frame metadata is not that costly, and quickly becomes negligible as source size grows larger. - - When a block is considered not compressible enough, ZSTD_compressBlock() result will be zero. - In which case, nothing is produced into `dst` ! - + User must test for such outcome and deal directly with uncompressed data - + ZSTD_decompressBlock() doesn't accept uncompressed data as input !!! + + For inputs larger than a single block, consider using regular ZSTD_compress() instead. + Frame metadata is not that costly, and quickly becomes negligible as source size grows larger than a block. + - When a block is considered not compressible enough, ZSTD_compressBlock() result will be 0 (zero) ! + ===> In which case, nothing is produced into `dst` ! + + User __must__ test for such outcome and deal directly with uncompressed data + + A block cannot be declared incompressible if ZSTD_compressBlock() return value was != 0. + Doing so would mess up with statistics history, leading to potential data corruption. + + ZSTD_decompressBlock() _doesn't accept uncompressed data as input_ !! + In case of multiple successive blocks, should some of them be uncompressed, decoder must be informed of their existence in order to follow proper history. Use ZSTD_insertBlock() for such a case. |