Import Zstd 1.4.5vendor/zstd/1.4.5vendor/zstd

292 files changed, 5668 insertions, 37340 deletions

diff --git a/CHANGELOG b/CHANGELOG
index 3b882d4cda51..0ed939a5bbb1 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -1,3 +1,29 @@
+v1.4.5
+fix : Compression ratio regression on huge files (> 3 GB) using high levels (--ultra) and multithreading, by @terrelln
+perf: Improved decompression speed: x64 : +10% (clang) / +5% (gcc); ARM : from +15% to +50%, depending on SoC, by @terrelln
+perf: Automatically downsizes ZSTD_DCtx when too large for too long (#2069, by @bimbashreshta)
+perf: Improved fast compression speed on aarch64 (#2040, ~+3%, by @caoyzh)
+perf: Small level 1 compression speed gains (depending on compiler)
+cli : New --patch-from command, create and apply patches from files, by @bimbashreshta
+cli : New --filelist= : Provide a list of files to operate upon from a file
+cli : -b -d command can now benchmark decompression on multiple files
+cli : New --no-content-size command
+cli : New --show-default-cparams information command
+api : ZDICT_finalizeDictionary() is promoted to stable (#2111)
+api : new experimental parameter ZSTD_d_stableOutBuffer (#2094)
+build: Generate a single-file libzstd library (#2065, by @cwoffenden)
+build: Relative includes no longer require -I compiler flags for zstd lib subdirs (#2103, by @felixhandte)
+build: zstd now compiles cleanly under -pedantic (#2099)
+build: zstd now compiles with make-4.3
+build: Support mingw cross-compilation from Linux, by @Ericson2314
+build: Meson multi-thread build fix on windows
+build: Some misc icc fixes backed by new ci test on travis
+misc: bitflip analyzer tool, by @felixhandte
+misc: Extend largeNbDicts benchmark to compression
+misc: Edit-distance match finder in contrib/
+doc : Improved beginner CONTRIBUTING.md docs
+doc : New issue templates for zstd
+
 v1.4.4
 perf: Improved decompression speed, by > 10%, by @terrelln
 perf: Better compression speed when re-using a context, by @felixhandte
@@ -14,7 +40,8 @@ cli: commands --stream-size=# and --size-hint=#, by @nmagerko
 cli: command --exclude-compressed, by @shashank0791
 cli: faster `-t` test mode
 cli: improved some error messages, by @vangyzen
-cli: rare deadlock condition within dictionary builder, by @terrelln
+cli: fix command `-D dictionary` on Windows, reported by @artyompetrov
+cli: fix rare deadlock condition within dictionary builder, by @terrelln
 build: single-file decoder with emscripten compilation script, by @cwoffenden
 build: fixed zlibWrapper compilation on Visual Studio, reported by @bluenlive
 build: fixed deprecation warning for certain gcc version, reported by @jasonma163
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index dd013f8084fa..637e37188550 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -26,6 +26,356 @@ to do this once to work on any of Facebook's open source projects.
 
 Complete your CLA here: <https://code.facebook.com/cla>
 
+## Workflow
+Zstd uses a branch-based workflow for making changes to the codebase. Typically, zstd
+will use a new branch per sizable topic. For smaller changes, it is okay to lump multiple
+related changes into a branch.
+
+Our contribution process works in three main stages:
+1. Local development
+    * Update:
+        * Checkout your fork of zstd if you have not already
+        ```
+        git checkout https://github.com/<username>/zstd
+        cd zstd
+        ```
+        * Update your local dev branch
+        ```
+        git pull https://github.com/facebook/zstd dev
+        git push origin dev
+        ```
+    * Topic and development:
+        * Make a new branch on your fork about the topic you're developing for
+        ```
+        # branch names should be consise but sufficiently informative
+        git checkout -b <branch-name>
+        git push origin <branch-name>
+        ```
+        * Make commits and push
+        ```
+        # make some changes =
+        git add -u && git commit -m <message>
+        git push origin <branch-name>
+        ```
+        * Note: run local tests to ensure that your changes didn't break existing functionality
+            * Quick check
+            ```
+            make shortest
+            ```
+            * Longer check
+            ```
+            make test
+            ```
+2. Code Review and CI tests
+    * Ensure CI tests pass:
+        * Before sharing anything to the community, make sure that all CI tests pass on your local fork.
+        See our section on setting up your CI environment for more information on how to do this.
+        * Ensure that static analysis passes on your development machine. See the Static Analysis section
+        below to see how to do this.
+    * Create a pull request:
+        * When you are ready to share you changes to the community, create a pull request from your branch
+        to facebook:dev. You can do this very easily by clicking 'Create Pull Request' on your fork's home
+        page.
+        * From there, select the branch where you made changes as your source branch and facebook:dev
+        as the destination.
+        * Examine the diff presented between the two branches to make sure there is nothing unexpected.
+    * Write a good pull request description:
+        * While there is no strict template that our contributors follow, we would like them to
+        sufficiently summarize and motivate the changes they are proposing. We recommend all pull requests,
+        at least indirectly, address the following points.
+            * Is this pull request important and why?
+            * Is it addressing an issue? If so, what issue? (provide links for convenience please)
+            * Is this a new feature? If so, why is it useful and/or necessary?
+            * Are there background references and documents that reviewers should be aware of to properly assess this change?
+        * Note: make sure to point out any design and architectural decisions that you made and the rationale behind them.
+        * Note: if you have been working with a specific user and would like them to review your work, make sure you mention them using (@<username>)
+    * Submit the pull request and iterate with feedback.
+3. Merge and Release
+    * Getting approval:
+        * You will have to iterate on your changes with feedback from other collaborators to reach a point
+        where your pull request can be safely merged.
+        * To avoid too many comments on style and convention, make sure that you have a
+        look at our style section below before creating a pull request.
+        * Eventually, someone from the zstd team will approve your pull request and not long after merge it into
+        the dev branch.
+    * Housekeeping:
+        * Most PRs are linked with one or more Github issues. If this is the case for your PR, make sure
+        the corresponding issue is mentioned. If your change 'fixes' or completely addresses the
+        issue at hand, then please indicate this by requesting that an issue be closed by commenting.
+        * Just because your changes have been merged does not mean the topic or larger issue is complete. Remember
+        that the change must make it to an official zstd release for it to be meaningful. We recommend
+        that contributers track the activity on their pull request and corresponding issue(s) page(s) until
+        their change makes it to the next release of zstd. Users will often discover bugs in your code or
+        suggest ways to refine and improve your initial changes even after the pull request is merged.
+
+## Static Analysis
+Static analysis is a process for examining the correctness or validity of a program without actually
+executing it. It usually helps us find many simple bugs. Zstd uses clang's `scan-build` tool for
+static analysis. You can install it by following the instructions for your OS on https://clang-analyzer.llvm.org/scan-build.
+
+Once installed, you can ensure that our static analysis tests pass on your local development machine
+by running:
+```
+make staticAnalyze
+```
+
+In general, you can use `scan-build` to static analyze any build script. For example, to static analyze
+just `contrib/largeNbDicts` and nothing else, you can run:
+
+```
+scan-build make -C contrib/largeNbDicts largeNbDicts
+```
+
+## Performance
+Performance is extremely important for zstd and we only merge pull requests whose performance
+landscape and corresponding trade-offs have been adequately analyzed, reproduced, and presented.
+This high bar for performance means that every PR which has the potential to
+impact performance takes a very long time for us to properly review. That being said, we
+always welcome contributions to improve performance (or worsen performance for the trade-off of
+something else). Please keep the following in mind before submitting a performance related PR:
+
+1. Zstd isn't as old as gzip but it has been around for time now and its evolution is
+very well documented via past Github issues and pull requests. It may be the case that your
+particular performance optimization has already been considered in the past. Please take some
+time to search through old issues and pull requests using keywords specific to your
+would-be PR. Of course, just because a topic has already been discussed (and perhaps rejected
+on some grounds) in the past, doesn't mean it isn't worth bringing up again. But even in that case,
+it will be helpful for you to have context from that topic's history before contributing.
+2. The distinction between noise and actual performance gains can unfortunately be very subtle
+especially when microbenchmarking extremely small wins or losses. The only remedy to getting
+something subtle merged is extensive benchmarking. You will be doing us a great favor if you
+take the time to run extensive, long-duration, and potentially cross-(os, platform, process, etc)
+benchmarks on your end before submitting a PR. Of course, you will not be able to benchmark
+your changes on every single processor and os out there (and neither will we) but do that best
+you can:) We've adding some things to think about when benchmarking below in the Benchmarking
+Performance section which might be helpful for you.
+3. Optimizing performance for a certain OS, processor vendor, compiler, or network system is a perfectly
+legitimate thing to do as long as it does not harm the overall performance health of Zstd.
+This is a hard balance to strike but please keep in mind other aspects of Zstd when
+submitting changes that are clang-specific, windows-specific, etc.
+
+## Benchmarking Performance
+Performance microbenchmarking is a tricky subject but also essential for Zstd. We value empirical
+testing over theoretical speculation. This guide it not perfect but for most scenarios, it
+is a good place to start.
+
+### Stability
+Unfortunately, the most important aspect in being able to benchmark reliably is to have a stable
+benchmarking machine. A virtual machine, a machine with shared resources, or your laptop
+will typically not be stable enough to obtain reliable benchmark results. If you can get your
+hands on a desktop, this is usually a better scenario.
+
+Of course, benchmarking can be done on non-hyper-stable machines as well. You will just have to
+do a little more work to ensure that you are in fact measuring the changes you've made not and
+noise. Here are some things you can do to make your benchmarks more stable:
+
+1. The most simple thing you can do to drastically improve the stability of your benchmark is
+to run it multiple times and then aggregate the results of those runs. As a general rule of
+thumb, the smaller the change you are trying to measure, the more samples of benchmark runs
+you will have to aggregate over to get reliable results. Here are some additional things to keep in
+mind when running multiple trials:
+    * How you aggregate your samples are important. You might be tempted to use the mean of your
+    results. While this is certainly going to be a more stable number than a raw single sample
+    benchmark number, you might have more luck by taking the median. The mean is not robust to
+    outliers whereas the median is. Better still, you could simply take the fastest speed your
+    benchmark achieved on each run since that is likely the fastest your process will be
+    capable of running your code. In our experience, this (aggregating by just taking the sample
+    with the fastest running time) has been the most stable approach.
+    * The more samples you have, the more stable your benchmarks should be. You can verify
+    your improved stability by looking at the size of your confidence intervals as you
+    increase your sample count. These should get smaller and smaller. Eventually hopefully
+    smaller than the performance win you are expecting.
+    * Most processors will take some time to get `hot` when running anything. The observations
+    you collect during that time period will very different from the true performance number. Having
+    a very large number of sample will help alleviate this problem slightly but you can also
+    address is directly by simply not including the first `n` iterations of your benchmark in
+    your aggregations. You can determine `n` by simply looking at the results from each iteration
+    and then hand picking a good threshold after which the variance in results seems to stabilize.
+2. You cannot really get reliable benchmarks if your host machine is simultaneously running
+another cpu/memory-intensive application in the background. If you are running benchmarks on your
+personal laptop for instance, you should close all applications (including your code editor and
+browser) before running your benchmarks. You might also have invisible background applications
+running. You can see what these are by looking at either Activity Monitor on Mac or Task Manager
+on Windows. You will get more stable benchmark results of you end those processes as well.
+    * If you have multiple cores, you can even run your benchmark on a reserved core to prevent
+    pollution from other OS and user processes. There are a number of ways to do this depending
+    on your OS:
+        * On linux boxes, you have use https://github.com/lpechacek/cpuset.
+        * On Windows, you can "Set Processor Affinity" using https://www.thewindowsclub.com/processor-affinity-windows
+        * On Mac, you can try to use their dedicated affinity API https://developer.apple.com/library/archive/releasenotes/Performance/RN-AffinityAPI/#//apple_ref/doc/uid/TP40006635-CH1-DontLinkElementID_2
+3. To benchmark, you will likely end up writing a separate c/c++ program that will link libzstd.
+Dynamically linking your library will introduce some added variation (not a large amount but
+definitely some). Statically linking libzstd will be more stable. Static libraries should
+be enabled by default when building zstd.
+4. Use a profiler with a good high resolution timer. See the section below on profiling for
+details on this.
+5. Disable frequency scaling, turbo boost and address space randomization (this will vary by OS)
+6. Try to avoid storage. On some systems you can use tmpfs. Putting the program, inputs and outputs on
+tmpfs avoids touching a real storage system, which can have a pretty big variability.
+
+Also check our LLVM's guide on benchmarking here: https://llvm.org/docs/Benchmarking.html
+
+### Zstd benchmark
+The fastest signal you can get regarding your performance changes is via the in-build zstd cli
+bench option. You can run Zstd as you typically would for your scenario using some set of options
+and then additionally also specify the `-b#` option. Doing this will run our benchmarking pipeline
+for that options you have just provided. If you want to look at the internals of how this
+benchmarking script works, you can check out programs/benchzstd.c
+
+For example: say you have made a change that you believe improves the speed of zstd level 1. The
+very first thing you should use to asses whether you actually achieved any sort of improvement
+is `zstd -b`. You might try to do something like this. Note: you can use the `-i` option to
+specify a running time for your benchmark in seconds (default is 3 seconds).
+Usually, the longer the running time, the more stable your results will be.
+
+```
+$ git checkout <commit-before-your-change>
+$ make && cp zstd zstd-old
+$ git checkout <commit-after-your-change>
+$ make && cp zstd zstd-new
+$ zstd-old -i5 -b1 <your-test-data>
+ 1<your-test-data>         :      8990 ->      3992 (2.252), 302.6 MB/s , 626.4 MB/s
+$ zstd-new -i5 -b1 <your-test-data>
+ 1<your-test-data>         :      8990 ->      3992 (2.252), 302.8 MB/s , 628.4 MB/s
+```
+
+Unless your performance win is large enough to be visible despite the intrinsic noise
+on your computer, benchzstd alone will likely not be enough to validate the impact of your
+changes. For example, the results of the example above indicate that effectively nothing
+changed but there could be a small <3% improvement that the noise on the host machine
+obscured. So unless you see a large performance win (10-15% consistently) using just
+this method of evaluation will not be sufficient.
+
+### Profiling
+There are a number of great profilers out there. We're going to briefly mention how you can
+profile your code using `instruments` on mac, `perf` on linux and `visual studio profiler`
+on windows.
+
+Say you have an idea for a change that you think will provide some good performance gains
+for level 1 compression on Zstd. Typically this means, you have identified a section of
+code that you think can be made to run faster.
+
+The first thing you will want to do is make sure that the piece of code is actually taking up
+a notable amount of time to run. It is usually not worth optimzing something which accounts for less than
+0.0001% of the total running time. Luckily, there are tools to help with this.
+Profilers will let you see how much time your code spends inside a particular function.
+If your target code snippit is only part of a function, it might be worth trying to
+isolate that snippit by moving it to its own function (this is usually not necessary but
+might be).
+
+Most profilers (including the profilers dicusssed below) will generate a call graph of
+functions for you. Your goal will be to find your function of interest in this call grapch
+and then inspect the time spent inside of it. You might also want to to look at the
+annotated assembly which most profilers will provide you with.
+
+#### Instruments
+We will once again consider the scenario where you think you've identified a piece of code
+whose performance can be improved upon. Follow these steps to profile your code using
+Instruments.
+
+1. Open Instruments
+2. Select `Time Profiler` from the list of standard templates
+3. Close all other applications except for your instruments window and your terminal
+4. Run your benchmarking script from your terminal window
+    * You will want a benchmark that runs for at least a few seconds (5 seconds will
+    usually be long enough). This way the profiler will have something to work with
+    and you will have ample time to attach your profiler to this process:)
+    * I will just use benchzstd as my bencharmking script for this example:
+```
+$ zstd -b1 -i5 <my-data> # this will run for 5 seconds
+```
+5. Once you run your benchmarking script, switch back over to instruments and attach your
+process to the time profiler. You can do this by:
+    * Clicking on the `All Processes` drop down in the top left of the toolbar.
+    * Selecting your process from the dropdown. In my case, it is just going to be labled
+    `zstd`
+    * Hitting the bright red record circle button on the top left of the toolbar
+6. You profiler will now start collecting metrics from your bencharking script. Once
+you think you have collected enough samples (usually this is the case after 3 seconds of
+recording), stop your profiler.
+7. Make sure that in toolbar of the bottom window, `profile` is selected.
+8. You should be able to see your call graph.
+    * If you don't see the call graph or an incomplete call graph, make sure you have compiled
+    zstd and your benchmarking scripg using debug flags. On mac and linux, this just means
+    you will have to supply the `-g` flag alone with your build script. You might also
+    have to provide the `-fno-omit-frame-pointer` flag
+9. Dig down the graph to find your function call and then inspect it by double clicking
+the list item. You will be able to see the annotated source code and the assembly side by
+side.
+
+#### Perf
+
+This wiki has a pretty detailed tutorial on getting started working with perf so we'll
+leave you to check that out of you're getting started:
+
+https://perf.wiki.kernel.org/index.php/Tutorial
+
+Some general notes on perf:
+* Use `perf stat -r # <bench-program>` to quickly get some relevant timing and
+counter statistics. Perf uses a high resolution timer and this is likely one
+of the first things your team will run when assessing your PR.
+* Perf has a long list of hardware counters that can be viewed with `perf --list`.
+When measuring optimizations, something worth trying is to make sure the handware
+counters you expect to be impacted by your change are in fact being so. For example,
+if you expect the L1 cache misses to decrease with your change, you can look at the
+counter `L1-dcache-load-misses`
+* Perf hardware counters will not work on a virtual machine.
+
+#### Visual Studio
+
+TODO
+
+
+## Setting up continuous integration (CI) on your fork
+Zstd uses a number of different continuous integration (CI) tools to ensure that new changes
+are well tested before they make it to an official release. Specifically, we use the platforms
+travis-ci, circle-ci, and appveyor.
+
+Changes cannot be merged into the main dev branch unless they pass all of our CI tests.
+The easiest way to run these CI tests on your own before submitting a PR to our dev branch
+is to configure your personal fork of zstd with each of the CI platforms. Below, you'll find
+instructions for doing this.
+
+### travis-ci
+Follow these steps to link travis-ci with your github fork of zstd
+
+1. Make sure you are logged into your github account
+2. Go to https://travis-ci.org/
+3. Click 'Sign in with Github' on the top right
+4. Click 'Authorize travis-ci'
+5. Click 'Activate all repositories using Github Apps'
+6. Select 'Only select repositories' and select your fork of zstd from the drop down
+7. Click 'Approve and Install'
+8. Click 'Sign in with Github' again. This time, it will be for travis-pro (which will let you view your tests on the web dashboard)
+9. Click 'Authorize travis-pro'
+10. You should have travis set up on your fork now.
+
+### circle-ci
+TODO
+
+### appveyor
+Follow these steps to link circle-ci with your girhub fork of zstd
+
+1. Make sure you are logged into your github account
+2. Go to https://www.appveyor.com/
+3. Click 'Sign in' on the top right
+4. Select 'Github' on the left panel
+5. Click 'Authorize appveyor'
+6. You might be asked to select which repositories you want to give appveyor permission to. Select your fork of zstd if you're prompted
+7. You should have appveyor set up on your fork now.
+
+### General notes on CI
+CI tests run every time a pull request (PR) is created or updated. The exact tests
+that get run will depend on the destination branch you specify. Some tests take
+longer to run than others. Currently, our CI is set up to run a short
+series of tests when creating a PR to the dev branch and a longer series of tests
+when creating a PR to the master branch. You can look in the configuration files
+of the respective CI platform for more information on what gets run when.
+
+Most people will just want to create a PR with the destination set to their local dev
+branch of zstd. You can then find the status of the tests on the PR's page. You can also
+re-run tests and cancel running tests from the PR page or from the respective CI's dashboard.
+
 ## Issues
 We use GitHub issues to track public bugs. Please ensure your description is
 clear and has sufficient instructions to be able to reproduce the issue.
@@ -34,7 +384,7 @@ Facebook has a [bounty program](https://www.facebook.com/whitehat/) for the safe
 disclosure of security bugs. In those cases, please go through the process
 outlined on that page and do not file a public issue.
 
-## Coding Style  
+## Coding Style
 * 4 spaces for indentation rather than tabs
 
 ## License
diff --git a/Makefile b/Makefile
index efb555c35b39..2c1d34604fe9 100644
--- a/Makefile
+++ b/Makefile
@@ -1,10 +1,11 @@
 # ################################################################
-# Copyright (c) 2015-present, Yann Collet, Facebook, Inc.
+# Copyright (c) 2015-2020, Yann Collet, Facebook, Inc.
 # All rights reserved.
 #
 # This source code is licensed under both the BSD-style license (found in the
 # LICENSE file in the root directory of this source tree) and the GPLv2 (found
 # in the COPYING file in the root directory of this source tree).
+# You may select, at your option, one of the above-listed licenses.
 # ################################################################
 
 PRGDIR   = programs
@@ -17,7 +18,16 @@ FUZZDIR  = $(TESTDIR)/fuzz
 # Define nul output
 VOID = /dev/null
 
-ifneq (,$(filter Windows%,$(OS)))
+# When cross-compiling from linux to windows, you might
+# need to specify this as "Windows." Fedora build fails
+# without it.
+#
+# Note: mingw-w64 build from linux to windows does not
+# fail on other tested distros (ubuntu, debian) even
+# without manually specifying the TARGET_SYSTEM.
+TARGET_SYSTEM ?= $(OS)
+
+ifneq (,$(filter Windows%,$(TARGET_SYSTEM)))
 EXT =.exe
 else
 EXT =
@@ -35,7 +45,7 @@ allmost: allzstd zlibwrapper
 
 # skip zwrapper, can't build that on alternate architectures without the proper zlib installed
 .PHONY: allzstd
-allzstd: lib
+allzstd: lib-all
 	$(MAKE) -C $(PRGDIR) all
 	$(MAKE) -C $(TESTDIR) all
 
@@ -45,7 +55,7 @@ all32:
 	$(MAKE) -C $(TESTDIR) all32
 
 .PHONY: lib lib-release libzstd.a
-lib lib-release :
+lib lib-release lib-all :
 	@$(MAKE) -C $(ZSTDDIR) $@
 
 .PHONY: zstd zstd-release
@@ -80,6 +90,13 @@ shortest:
 .PHONY: check
 check: shortest
 
+.PHONY: automated_benchmarking
+automated_benchmarking:
+	$(MAKE) -C $(TESTDIR) $@
+
+.PHONY: benchmarking
+benchmarking: automated_benchmarking
+
 ## examples: build all examples in `/examples` directory
 .PHONY: examples
 examples: lib
@@ -101,7 +118,8 @@ contrib: lib
 	$(MAKE) -C contrib/pzstd all
 	$(MAKE) -C contrib/seekable_format/examples all
 	$(MAKE) -C contrib/largeNbDicts all
-	cd contrib/single_file_decoder/ ; ./build_test.sh
+	cd contrib/single_file_libs/ ; ./build_decoder_test.sh
+	cd contrib/single_file_libs/ ; ./build_library_test.sh
 
 .PHONY: cleanTabs
 cleanTabs:
@@ -337,7 +355,7 @@ endif
 
 ifneq (,$(filter MSYS%,$(shell uname)))
 HOST_OS = MSYS
-CMAKE_PARAMS = -G"MSYS Makefiles" -DZSTD_MULTITHREAD_SUPPORT:BOOL=OFF -DZSTD_BUILD_STATIC:BOOL=ON -DZSTD_BUILD_TESTS:BOOL=ON
+CMAKE_PARAMS = -G"MSYS Makefiles" -DCMAKE_BUILD_TYPE=Debug -DZSTD_MULTITHREAD_SUPPORT:BOOL=OFF -DZSTD_BUILD_STATIC:BOOL=ON -DZSTD_BUILD_TESTS:BOOL=ON
 endif
 
 
@@ -349,11 +367,15 @@ cmakebuild:
 	cmake --version
 	$(RM) -r $(BUILDIR)/cmake/build
 	mkdir $(BUILDIR)/cmake/build
-	cd $(BUILDIR)/cmake/build ; cmake -DCMAKE_INSTALL_PREFIX:PATH=~/install_test_dir $(CMAKE_PARAMS) .. ; $(MAKE) install ; $(MAKE) uninstall
+	cd $(BUILDIR)/cmake/build; cmake -DCMAKE_INSTALL_PREFIX:PATH=~/install_test_dir $(CMAKE_PARAMS) ..
+	$(MAKE) -C $(BUILDIR)/cmake/build -j4;
+	$(MAKE) -C $(BUILDIR)/cmake/build install;
+	$(MAKE) -C $(BUILDIR)/cmake/build uninstall;
+	cd $(BUILDIR)/cmake/build; ctest -V -L Medium
 
-c90build: clean
+c89build: clean
 	$(CC) -v
-	CFLAGS="-std=c90 -Werror" $(MAKE) allmost  # will fail, due to missing support for `long long`
+	CFLAGS="-std=c89 -Werror" $(MAKE) allmost  # will fail, due to missing support for `long long`
 
 gnu90build: clean
 	$(CC) -v
diff --git a/README.md b/README.md
index 9c5f9201307c..5c300fdc49aa 100644
--- a/README.md
+++ b/README.md
@@ -31,10 +31,10 @@ a list of known ports and bindings is provided on [Zstandard homepage](http://ww
 ## Benchmarks
 
 For reference, several fast compression algorithms were tested and compared
-on a server running Arch Linux (`Linux version 5.0.5-arch1-1`),
+on a server running Arch Linux (`Linux version 5.5.11-arch1-1`),
 with a Core i9-9900K CPU @ 5.0GHz,
 using [lzbench], an open-source in-memory benchmark by @inikep
-compiled with [gcc] 8.2.1,
+compiled with [gcc] 9.3.0,
 on the [Silesia compression corpus].
 
 [lzbench]: https://github.com/inikep/lzbench
@@ -43,18 +43,26 @@ on the [Silesia compression corpus].
 
 | Compressor name         | Ratio | Compression| Decompress.|
 | ---------------         | ------| -----------| ---------- |
-| **zstd 1.4.0 -1**       | 2.884 |   530 MB/s |  1360 MB/s |
-| zlib 1.2.11 -1          | 2.743 |   110 MB/s |   440 MB/s |
-| brotli 1.0.7 -0         | 2.701 |   430 MB/s |   470 MB/s |
-| quicklz 1.5.0 -1        | 2.238 |   600 MB/s |   800 MB/s |
-| lzo1x 2.09 -1           | 2.106 |   680 MB/s |   950 MB/s |
-| lz4 1.8.3               | 2.101 |   800 MB/s |  4220 MB/s |
-| snappy 1.1.4            | 2.073 |   580 MB/s |  2020 MB/s |
-| lzf 3.6 -1              | 2.077 |   440 MB/s |   930 MB/s |
+| **zstd 1.4.5 -1**       | 2.884 |   500 MB/s |  1660 MB/s |
+| zlib 1.2.11 -1          | 2.743 |    90 MB/s |   400 MB/s |
+| brotli 1.0.7 -0         | 2.703 |   400 MB/s |   450 MB/s |
+| **zstd 1.4.5 --fast=1** | 2.434 |   570 MB/s |  2200 MB/s |
+| **zstd 1.4.5 --fast=3** | 2.312 |   640 MB/s |  2300 MB/s |
+| quicklz 1.5.0 -1        | 2.238 |   560 MB/s |   710 MB/s |
+| **zstd 1.4.5 --fast=5** | 2.178 |   700 MB/s |  2420 MB/s |
+| lzo1x 2.10 -1           | 2.106 |   690 MB/s |   820 MB/s |
+| lz4 1.9.2               | 2.101 |   740 MB/s |  4530 MB/s |
+| **zstd 1.4.5 --fast=7** | 2.096 |   750 MB/s |  2480 MB/s |
+| lzf 3.6 -1              | 2.077 |   410 MB/s |   860 MB/s |
+| snappy 1.1.8            | 2.073 |   560 MB/s |  1790 MB/s |
 
 [zlib]: http://www.zlib.net/
 [LZ4]: http://www.lz4.org/
 
+The negative compression levels, specified with `--fast=#`,
+offer faster compression and decompression speed in exchange for some loss in
+compression ratio compared to level 1, as seen in the table above.
+
 Zstd can also offer stronger compression ratios at the cost of compression speed.
 Speed vs Compression trade-off is configurable by small increments.
 Decompression speed is preserved and remains roughly the same at all settings,
@@ -143,6 +151,18 @@ example about how Meson is used to build this project.
 
 Note that default build type is **release**.
 
+### VCPKG
+You can build and install zstd [vcpkg](https://github.com/Microsoft/vcpkg/) dependency manager:
+
+    git clone https://github.com/Microsoft/vcpkg.git
+    cd vcpkg
+    ./bootstrap-vcpkg.sh
+    ./vcpkg integrate install
+    ./vcpkg install zstd
+
+The zstd port in vcpkg is kept up to date by Microsoft team members and community contributors.
+If the version is out of date, please [create an issue or pull request](https://github.com/Microsoft/vcpkg) on the vcpkg repository.
+
 ### Visual Studio (Windows)
 
 Going into `build` directory, you will find additional possibilities:
diff --git a/TESTING.md b/TESTING.md
index 551981b14053..7e5305178b97 100644
--- a/TESTING.md
+++ b/TESTING.md
@@ -11,7 +11,7 @@ They consist of the following tests:
 - Compilation on all supported targets (x86, x86_64, ARM, AArch64, PowerPC, and PowerPC64)
 - Compilation on various versions of gcc, clang, and g++
 - `tests/playTests.sh` on x86_64, without the tests on long data (CLI tests)
-- Small tests (`tests/legacy.c`, `tests/longmatch.c`, `tests/symbols.c`) on x64_64
+- Small tests (`tests/legacy.c`, `tests/longmatch.c`) on x64_64
 
 Medium Tests
 ------------
@@ -19,7 +19,7 @@ Medium tests run on every commit and pull request to `dev` branch, on TravisCI.
 They consist of the following tests:
 - The following tests run with UBsan and Asan on x86_64 and x86, as well as with
   Msan on x86_64
-  - `tests/playTests.sh --test-long-data`
+  - `tests/playTests.sh --test-large-data`
   - Fuzzer tests: `tests/fuzzer.c`, `tests/zstreamtest.c`, and `tests/decodecorpus.c`
 - `tests/zstreamtest.c` under Tsan (streaming mode, including multithreaded mode)
 - Valgrind Test (`make -C tests valgrindTest`) (testing CLI and fuzzer under valgrind)
diff --git a/appveyor.yml b/appveyor.yml
index dd2c02ac4826..5d77b3103481 100644
--- a/appveyor.yml
+++ b/appveyor.yml
@@ -14,7 +14,7 @@
     - COMPILER: "gcc"
       HOST:     "mingw"
       PLATFORM: "x64"
-      SCRIPT:   "make allzstd MOREFLAGS=-static && make -C tests test-symbols fullbench-lib"
+      SCRIPT:   "make allzstd MOREFLAGS=-static && make -C tests fullbench-lib"
       ARTIFACT: "true"
       BUILD:    "true"
     - COMPILER: "gcc"
@@ -169,7 +169,8 @@
   - SET "FUZZERTEST=-T30s"
   - if [%HOST%]==[visual] if [%CONFIGURATION%]==[Release] (
       CD tests &&
-      SET ZSTD=./zstd.exe &&
+      SET ZSTD_BIN=./zstd.exe&&
+      SET DATAGEN_BIN=./datagen.exe&&
       sh -e playTests.sh --test-large-data &&
       fullbench.exe -i1 &&
       fullbench.exe -i1 -P0 &&
@@ -188,6 +189,9 @@
   environment:
     matrix:
     - COMPILER: "gcc"
+      HOST:     "cygwin"
+      PLATFORM: "x64"
+    - COMPILER: "gcc"
       HOST:     "mingw"
       PLATFORM: "x64"
       SCRIPT:   "CPPFLAGS=-DDEBUGLEVEL=2 CFLAGS=-Werror make -j allzstd DEBUGLEVEL=2"
@@ -220,6 +224,14 @@
   install:
   - ECHO Installing %COMPILER% %PLATFORM% %CONFIGURATION%
   - SET PATH_ORIGINAL=%PATH%
+  - if [%HOST%]==[cygwin] (
+      ECHO Installing Cygwin Packages &&
+      C:\cygwin64\setup-x86_64.exe -qnNdO -R "C:\cygwin64" -g -P ^
+        gcc-g++,^
+        gcc,^
+        cmake,^
+        make
+    )
   - if [%HOST%]==[mingw] (
       SET "PATH_MINGW32=C:\mingw-w64\i686-6.3.0-posix-dwarf-rt_v5-rev1\mingw32\bin" &&
       SET "PATH_MINGW64=C:\mingw-w64\x86_64-6.3.0-posix-seh-rt_v5-rev1\mingw64\bin" &&
@@ -232,6 +244,17 @@
 
   build_script:
   - ECHO Building %COMPILER% %PLATFORM% %CONFIGURATION%
+  - if [%HOST%]==[cygwin] (
+      set CHERE_INVOKING=yes &&
+      set CC=%COMPILER% &&
+      C:\cygwin64\bin\bash --login -c "
+        set -e;
+        cd build/cmake;
+        CFLAGS='-Werror' cmake -G 'Unix Makefiles' -DCMAKE_BUILD_TYPE=Debug -DZSTD_BUILD_TESTS:BOOL=ON -DZSTD_FUZZER_FLAGS=-T30s -DZSTD_ZSTREAM_FLAGS=-T30s .;
+        make -j4;
+        ctest -V -L Medium;
+      "
+    )
   - if [%HOST%]==[mingw] (
       ( if [%PLATFORM%]==[x64] (
         SET "PATH=%PATH_MINGW64%;%PATH_ORIGINAL%"
diff --git a/contrib/cleanTabs b/contrib/cleanTabs
deleted file mode 100755
index 215913a90ace..000000000000
--- a/contrib/cleanTabs
+++ /dev/null
@@ -1,2 +0,0 @@
-#!/bin/sh
-sed -i '' $'s/\t/    /g' ../lib/**/*.{h,c} ../programs/*.{h,c} ../tests/*.c ./**/*.{h,cpp} ../examples/*.c ../zlibWrapper/*.{h,c}
diff --git a/contrib/docker/Dockerfile b/contrib/docker/Dockerfile
deleted file mode 100644
index e06a32c0dac7..000000000000
--- a/contrib/docker/Dockerfile
+++ /dev/null
@@ -1,20 +0,0 @@
-# Dockerfile
-# First image to build the binary
-FROM alpine as builder
-
-RUN apk --no-cache add make gcc libc-dev
-COPY . /src
-RUN mkdir /pkg && cd /src && make && make DESTDIR=/pkg install
-
-# Second minimal image to only keep the built binary
-FROM alpine
-
-# Copy the built files
-COPY --from=builder /pkg /
-
-# Copy the license as well
-RUN mkdir -p /usr/local/share/licenses/zstd
-COPY --from=builder /src/LICENSE /usr/local/share/licences/zstd/
-
-# Just run `zstd` if no other command is given
-CMD ["/usr/local/bin/zstd"]
diff --git a/contrib/docker/README.md b/contrib/docker/README.md
deleted file mode 100644
index 43f6d7a1ae1a..000000000000
--- a/contrib/docker/README.md
+++ /dev/null
@@ -1,20 +0,0 @@
-
-## Requirement
-
-The `Dockerfile` script requires a version of `docker` >= 17.05
-
-## Installing docker
-
-The official docker install docs use a ppa with a modern version available:
-https://docs.docker.com/install/linux/docker-ce/ubuntu/
-
-## How to run
-
-`docker build -t zstd .`
-
-## test
-
-```
-echo foo | docker run -i --rm zstd | docker run -i --rm zstd zstdcat
-foo
-```
diff --git a/contrib/experimental_dict_builders/benchmarkDictBuilder/Makefile b/contrib/experimental_dict_builders/benchmarkDictBuilder/Makefile
deleted file mode 100644
index 72ce04f2a56b..000000000000
--- a/contrib/experimental_dict_builders/benchmarkDictBuilder/Makefile
+++ /dev/null
@@ -1,44 +0,0 @@
-ARG :=
-
-CC ?= gcc
-CFLAGS ?= -O3
-INCLUDES := -I ../randomDictBuilder -I ../../../programs -I ../../../lib/common -I ../../../lib -I ../../../lib/dictBuilder
-
-RANDOM_FILE := ../randomDictBuilder/random.c
-IO_FILE := ../randomDictBuilder/io.c
-
-all: run clean
-
-.PHONY: run
-run: benchmark
-	echo "Benchmarking with $(ARG)"
-	./benchmark $(ARG)
-
-.PHONY: test
-test: benchmarkTest clean
-
-.PHONY: benchmarkTest
-benchmarkTest: benchmark test.sh
-	sh test.sh
-
-benchmark: benchmark.o io.o random.o libzstd.a
-	$(CC) $(CFLAGS) benchmark.o io.o random.o libzstd.a -o benchmark
-
-benchmark.o: benchmark.c
-	$(CC) $(CFLAGS) $(INCLUDES) -c benchmark.c
-
-random.o: $(RANDOM_FILE)
-	$(CC) $(CFLAGS) $(INCLUDES) -c $(RANDOM_FILE)
-
-io.o: $(IO_FILE)
-	$(CC) $(CFLAGS) $(INCLUDES) -c $(IO_FILE)
-
-libzstd.a:
-	$(MAKE) -C ../../../lib libzstd.a
-	mv ../../../lib/libzstd.a .
-
-.PHONY: clean
-clean:
-	rm -f *.o benchmark libzstd.a
-	$(MAKE) -C ../../../lib clean
-	echo "Cleaning is completed"
diff --git a/contrib/experimental_dict_builders/benchmarkDictBuilder/README.md b/contrib/experimental_dict_builders/benchmarkDictBuilder/README.md
deleted file mode 100644
index 6a6c7f1d2169..000000000000
--- a/contrib/experimental_dict_builders/benchmarkDictBuilder/README.md
+++ /dev/null
@@ -1,849 +0,0 @@
-Benchmarking Dictionary Builder
-
-### Permitted Argument:
-Input File/Directory (in=fileName): required; file/directory used to build dictionary; if directory, will operate recursively for files inside directory; can include multiple files/directories, each following "in="
-
-###Running Test:
-make test
-
-###Usage:
-Benchmark given input files: make ARG= followed by permitted arguments
-
-### Examples:
-make ARG="in=../../../lib/dictBuilder in=../../../lib/compress"
-
-###Benchmarking Result:
-- First Cover is optimize cover, second Cover uses optimized d and k from first one.
-- For every f value of fastCover, the first one is optimize fastCover and the second one uses optimized d and k from first one. This is run for accel values from 1 to 10.
-- Fourth column is chosen d and fifth column is chosen k
-
-github:
-NODICT       0.000004       2.999642        
-RANDOM       0.024560       8.791189        
-LEGACY       0.727109       8.173529        
-COVER       40.565676       10.652243        8          1298
-COVER       3.608284       10.652243        8          1298
-FAST f=15 a=1       4.181024       10.570882        8          1154
-FAST f=15 a=1       0.040788       10.570882        8          1154
-FAST f=15 a=2       3.548352       10.574287        6          1970
-FAST f=15 a=2       0.035535       10.574287        6          1970
-FAST f=15 a=3       3.287364       10.613950        6          1010
-FAST f=15 a=3       0.032182       10.613950        6          1010
-FAST f=15 a=4       3.184976       10.573883        6          1058
-FAST f=15 a=4       0.029878       10.573883        6          1058
-FAST f=15 a=5       3.045513       10.580640        8          1154
-FAST f=15 a=5       0.022162       10.580640        8          1154
-FAST f=15 a=6       3.003296       10.583677        6          1010
-FAST f=15 a=6       0.028091       10.583677        6          1010
-FAST f=15 a=7       2.952655       10.622551        6          1106
-FAST f=15 a=7       0.02724       10.622551        6          1106
-FAST f=15 a=8       2.945674       10.614657        6          1010
-FAST f=15 a=8       0.027264       10.614657        6          1010
-FAST f=15 a=9       3.153439       10.564018        8          1154
-FAST f=15 a=9       0.020635       10.564018        8          1154
-FAST f=15 a=10       2.950416       10.511454        6          1010
-FAST f=15 a=10       0.026606       10.511454        6          1010
-FAST f=16 a=1       3.970029       10.681035        8          1154
-FAST f=16 a=1       0.038188       10.681035        8          1154
-FAST f=16 a=2       3.422892       10.484978        6          1874
-FAST f=16 a=2       0.034702       10.484978        6          1874
-FAST f=16 a=3       3.215836       10.632631        8          1154
-FAST f=16 a=3       0.026084       10.632631        8          1154
-FAST f=16 a=4       3.081353       10.626533        6          1106
-FAST f=16 a=4       0.030032       10.626533        6          1106
-FAST f=16 a=5       3.041241       10.545027        8          1922
-FAST f=16 a=5       0.022882       10.545027        8          1922
-FAST f=16 a=6       2.989390       10.638284        6          1874
-FAST f=16 a=6       0.028308       10.638284        6          1874
-FAST f=16 a=7       3.001581       10.797136        6          1106
-FAST f=16 a=7       0.027479       10.797136        6          1106
-FAST f=16 a=8       2.984107       10.658356        8          1058
-FAST f=16 a=8       0.021099       10.658356        8          1058
-FAST f=16 a=9       2.925788       10.523869        6          1010
-FAST f=16 a=9       0.026905       10.523869        6          1010
-FAST f=16 a=10       2.889605       10.745841        6          1874
-FAST f=16 a=10       0.026846       10.745841        6          1874
-FAST f=17 a=1       4.031953       10.672080        8          1202
-FAST f=17 a=1       0.040658       10.672080        8          1202
-FAST f=17 a=2       3.458107       10.589352        8          1106
-FAST f=17 a=2       0.02926       10.589352        8          1106
-FAST f=17 a=3       3.291189       10.662714        8          1154
-FAST f=17 a=3       0.026531       10.662714        8          1154
-FAST f=17 a=4       3.154950       10.549456        8          1346
-FAST f=17 a=4       0.024991       10.549456        8          1346
-FAST f=17 a=5       3.092271       10.541670        6          1202
-FAST f=17 a=5       0.038285       10.541670        6          1202
-FAST f=17 a=6       3.166146       10.729112        6          1874
-FAST f=17 a=6       0.038217       10.729112        6          1874
-FAST f=17 a=7       3.035467       10.810485        6          1106
-FAST f=17 a=7       0.036655       10.810485        6          1106
-FAST f=17 a=8       3.035668       10.530532        6          1058
-FAST f=17 a=8       0.037715       10.530532        6          1058
-FAST f=17 a=9       2.987917       10.589802        8          1922
-FAST f=17 a=9       0.02217       10.589802        8          1922
-FAST f=17 a=10       2.981647       10.722579        8          1106
-FAST f=17 a=10       0.021948       10.722579        8          1106
-FAST f=18 a=1       4.067144       10.634943        8          1154
-FAST f=18 a=1       0.041386       10.634943        8          1154
-FAST f=18 a=2       3.507377       10.546230        6          1970
-FAST f=18 a=2       0.037572       10.546230        6          1970
-FAST f=18 a=3       3.323015       10.648061        8          1154
-FAST f=18 a=3       0.028306       10.648061        8          1154
-FAST f=18 a=4       3.216735       10.705402        6          1010
-FAST f=18 a=4       0.030755       10.705402        6          1010
-FAST f=18 a=5       3.175794       10.588154        8          1874
-FAST f=18 a=5       0.025315       10.588154        8          1874
-FAST f=18 a=6       3.127459       10.751104        8          1106
-FAST f=18 a=6       0.023897       10.751104        8          1106
-FAST f=18 a=7       3.083017       10.780402        6          1106
-FAST f=18 a=7       0.029158       10.780402        6          1106
-FAST f=18 a=8       3.069700       10.547226        8          1346
-FAST f=18 a=8       0.024046       10.547226        8          1346
-FAST f=18 a=9       3.056591       10.674759        6          1010
-FAST f=18 a=9       0.028496       10.674759        6          1010
-FAST f=18 a=10       3.063588       10.737578        8          1106
-FAST f=18 a=10       0.023033       10.737578        8          1106
-FAST f=19 a=1       4.164041       10.650333        8          1154
-FAST f=19 a=1       0.042906       10.650333        8          1154
-FAST f=19 a=2       3.585409       10.577066        6          1058
-FAST f=19 a=2       0.038994       10.577066        6          1058
-FAST f=19 a=3       3.439643       10.639403        8          1154
-FAST f=19 a=3       0.028427       10.639403        8          1154
-FAST f=19 a=4       3.268869       10.554410        8          1298
-FAST f=19 a=4       0.026866       10.554410        8          1298
-FAST f=19 a=5       3.238225       10.615109        6          1010
-FAST f=19 a=5       0.03078       10.615109        6          1010
-FAST f=19 a=6       3.199558       10.609782        6          1874
-FAST f=19 a=6       0.030099       10.609782        6          1874
-FAST f=19 a=7       3.132395       10.794753        6          1106
-FAST f=19 a=7       0.028964       10.794753        6          1106
-FAST f=19 a=8       3.148446       10.554842        8          1298
-FAST f=19 a=8       0.024277       10.554842        8          1298
-FAST f=19 a=9       3.108324       10.668763        6          1010
-FAST f=19 a=9       0.02896       10.668763        6          1010
-FAST f=19 a=10       3.159863       10.757347        8          1106
-FAST f=19 a=10       0.023351       10.757347        8          1106
-FAST f=20 a=1       4.462698       10.661788        8          1154
-FAST f=20 a=1       0.047174       10.661788        8          1154
-FAST f=20 a=2       3.820269       10.678612        6          1106
-FAST f=20 a=2       0.040807       10.678612        6          1106
-FAST f=20 a=3       3.644955       10.648424        8          1154
-FAST f=20 a=3       0.031398       10.648424        8          1154
-FAST f=20 a=4       3.546257       10.559756        8          1298
-FAST f=20 a=4       0.029856       10.559756        8          1298
-FAST f=20 a=5       3.485248       10.646637        6          1010
-FAST f=20 a=5       0.033756       10.646637        6          1010
-FAST f=20 a=6       3.490438       10.775824        8          1106
-FAST f=20 a=6       0.028338       10.775824        8          1106
-FAST f=20 a=7       3.631289       10.801795        6          1106
-FAST f=20 a=7       0.035228       10.801795        6          1106
-FAST f=20 a=8       3.758936       10.545116        8          1346
-FAST f=20 a=8       0.027495       10.545116        8          1346
-FAST f=20 a=9       3.707024       10.677454        6          1010
-FAST f=20 a=9       0.031326       10.677454        6          1010
-FAST f=20 a=10       3.586593       10.756017        8          1106
-FAST f=20 a=10       0.027122       10.756017        8          1106
-FAST f=21 a=1       5.701396       10.655398        8          1154
-FAST f=21 a=1       0.067744       10.655398        8          1154
-FAST f=21 a=2       5.270542       10.650743        6          1106
-FAST f=21 a=2       0.052999       10.650743        6          1106
-FAST f=21 a=3       4.945294       10.652380        8          1154
-FAST f=21 a=3       0.052678       10.652380        8          1154
-FAST f=21 a=4       4.894079       10.543185        8          1298
-FAST f=21 a=4       0.04997       10.543185        8          1298
-FAST f=21 a=5       4.785417       10.630321        6          1010
-FAST f=21 a=5       0.045294       10.630321        6          1010
-FAST f=21 a=6       4.789381       10.664477        6          1874
-FAST f=21 a=6       0.046578       10.664477        6          1874
-FAST f=21 a=7       4.302955       10.805179        6          1106
-FAST f=21 a=7       0.041205       10.805179        6          1106
-FAST f=21 a=8       4.034630       10.551211        8          1298
-FAST f=21 a=8       0.040121       10.551211        8          1298
-FAST f=21 a=9       4.523868       10.799114        6          1010
-FAST f=21 a=9       0.043592       10.799114        6          1010
-FAST f=21 a=10       4.760736       10.750255        8          1106
-FAST f=21 a=10       0.043483       10.750255        8          1106
-FAST f=22 a=1       6.743064       10.640537        8          1154
-FAST f=22 a=1       0.086967       10.640537        8          1154
-FAST f=22 a=2       6.121739       10.626638        6          1970
-FAST f=22 a=2       0.066337       10.626638        6          1970
-FAST f=22 a=3       5.248851       10.640688        8          1154
-FAST f=22 a=3       0.054935       10.640688        8          1154
-FAST f=22 a=4       5.436579       10.588333        8          1298
-FAST f=22 a=4       0.064113       10.588333        8          1298
-FAST f=22 a=5       5.812815       10.652653        6          1010
-FAST f=22 a=5       0.058189       10.652653        6          1010
-FAST f=22 a=6       5.745472       10.666437        6          1874
-FAST f=22 a=6       0.057188       10.666437        6          1874
-FAST f=22 a=7       5.716393       10.806911        6          1106
-FAST f=22 a=7       0.056       10.806911        6          1106
-FAST f=22 a=8       5.698799       10.530784        8          1298
-FAST f=22 a=8       0.0583       10.530784        8          1298
-FAST f=22 a=9       5.710533       10.777391        6          1010
-FAST f=22 a=9       0.054945       10.777391        6          1010
-FAST f=22 a=10       5.685395       10.745023        8          1106
-FAST f=22 a=10       0.056526       10.745023        8          1106
-FAST f=23 a=1       7.836923       10.638828        8          1154
-FAST f=23 a=1       0.099522       10.638828        8          1154
-FAST f=23 a=2       6.627834       10.631061        6          1970
-FAST f=23 a=2       0.066769       10.631061        6          1970
-FAST f=23 a=3       5.602533       10.647288        8          1154
-FAST f=23 a=3       0.064513       10.647288        8          1154
-FAST f=23 a=4       6.005580       10.568747        8          1298
-FAST f=23 a=4       0.062022       10.568747        8          1298
-FAST f=23 a=5       5.481816       10.676921        6          1010
-FAST f=23 a=5       0.058959       10.676921        6          1010
-FAST f=23 a=6       5.460444       10.666194        6          1874
-FAST f=23 a=6       0.057687       10.666194        6          1874
-FAST f=23 a=7       5.659822       10.800377        6          1106
-FAST f=23 a=7       0.06783       10.800377        6          1106
-FAST f=23 a=8       6.826940       10.522167        8          1298
-FAST f=23 a=8       0.070533       10.522167        8          1298
-FAST f=23 a=9       6.804757       10.577799        8          1682
-FAST f=23 a=9       0.069949       10.577799        8          1682
-FAST f=23 a=10       6.774933       10.742093        8          1106
-FAST f=23 a=10       0.068395       10.742093        8          1106
-FAST f=24 a=1       8.444110       10.632783        8          1154
-FAST f=24 a=1       0.094357       10.632783        8          1154
-FAST f=24 a=2       7.289578       10.631061        6          1970
-FAST f=24 a=2       0.098515       10.631061        6          1970
-FAST f=24 a=3       8.619780       10.646289        8          1154
-FAST f=24 a=3       0.098041       10.646289        8          1154
-FAST f=24 a=4       8.508455       10.555199        8          1298
-FAST f=24 a=4       0.093885       10.555199        8          1298
-FAST f=24 a=5       8.471145       10.674363        6          1010
-FAST f=24 a=5       0.088676       10.674363        6          1010
-FAST f=24 a=6       8.426727       10.667228        6          1874
-FAST f=24 a=6       0.087247       10.667228        6          1874
-FAST f=24 a=7       8.356826       10.803027        6          1106
-FAST f=24 a=7       0.085835       10.803027        6          1106
-FAST f=24 a=8       6.756811       10.522049        8          1298
-FAST f=24 a=8       0.07107       10.522049        8          1298
-FAST f=24 a=9       6.548169       10.571882        8          1682
-FAST f=24 a=9       0.0713       10.571882        8          1682
-FAST f=24 a=10       8.238079       10.736453        8          1106
-FAST f=24 a=10       0.07004       10.736453        8          1106
-
-
-hg-commands:
-NODICT       0.000005       2.425276        
-RANDOM       0.046332       3.490331        
-LEGACY       0.720351       3.911682        
-COVER       45.507731       4.132653        8          386
-COVER       1.868810       4.132653        8          386
-FAST f=15 a=1       4.561427       3.866894        8          1202
-FAST f=15 a=1       0.048946       3.866894        8          1202
-FAST f=15 a=2       3.574462       3.892119        8          1538
-FAST f=15 a=2       0.033677       3.892119        8          1538
-FAST f=15 a=3       3.230227       3.888791        6          1346
-FAST f=15 a=3       0.034312       3.888791        6          1346
-FAST f=15 a=4       3.042388       3.899739        8          1010
-FAST f=15 a=4       0.024307       3.899739        8          1010
-FAST f=15 a=5       2.800148       3.896220        8          818
-FAST f=15 a=5       0.022331       3.896220        8          818
-FAST f=15 a=6       2.706518       3.882039        8          578
-FAST f=15 a=6       0.020955       3.882039        8          578
-FAST f=15 a=7       2.701820       3.885430        6          866
-FAST f=15 a=7       0.026074       3.885430        6          866
-FAST f=15 a=8       2.604445       3.906932        8          1826
-FAST f=15 a=8       0.021789       3.906932        8          1826
-FAST f=15 a=9       2.598568       3.870324        6          1682
-FAST f=15 a=9       0.026004       3.870324        6          1682
-FAST f=15 a=10       2.575920       3.920783        8          1442
-FAST f=15 a=10       0.020228       3.920783        8          1442
-FAST f=16 a=1       4.630623       4.001430        8          770
-FAST f=16 a=1       0.047497       4.001430        8          770
-FAST f=16 a=2       3.674721       3.974431        8          1874
-FAST f=16 a=2       0.035761       3.974431        8          1874
-FAST f=16 a=3       3.338384       3.978703        8          1010
-FAST f=16 a=3       0.029436       3.978703        8          1010
-FAST f=16 a=4       3.004412       3.983035        8          1010
-FAST f=16 a=4       0.025744       3.983035        8          1010
-FAST f=16 a=5       2.881892       3.987710        8          770
-FAST f=16 a=5       0.023211       3.987710        8          770
-FAST f=16 a=6       2.807410       3.952717        8          1298
-FAST f=16 a=6       0.023199       3.952717        8          1298
-FAST f=16 a=7       2.819623       3.994627        8          770
-FAST f=16 a=7       0.021806       3.994627        8          770
-FAST f=16 a=8       2.740092       3.954032        8          1826
-FAST f=16 a=8       0.0226       3.954032        8          1826
-FAST f=16 a=9       2.682564       3.969879        6          1442
-FAST f=16 a=9       0.026324       3.969879        6          1442
-FAST f=16 a=10       2.657959       3.969755        8          674
-FAST f=16 a=10       0.020413       3.969755        8          674
-FAST f=17 a=1       4.729228       4.046000        8          530
-FAST f=17 a=1       0.049703       4.046000        8          530
-FAST f=17 a=2       3.764510       3.991519        8          1970
-FAST f=17 a=2       0.038195       3.991519        8          1970
-FAST f=17 a=3       3.416992       4.006296        6          914
-FAST f=17 a=3       0.036244       4.006296        6          914
-FAST f=17 a=4       3.145626       3.979182        8          1970
-FAST f=17 a=4       0.028676       3.979182        8          1970
-FAST f=17 a=5       2.995070       4.050070        8          770
-FAST f=17 a=5       0.025707       4.050070        8          770
-FAST f=17 a=6       2.911833       4.040024        8          770
-FAST f=17 a=6       0.02453       4.040024        8          770
-FAST f=17 a=7       2.894796       4.015884        8          818
-FAST f=17 a=7       0.023956       4.015884        8          818
-FAST f=17 a=8       2.789962       4.039303        8          530
-FAST f=17 a=8       0.023219       4.039303        8          530
-FAST f=17 a=9       2.787625       3.996762        8          1634
-FAST f=17 a=9       0.023651       3.996762        8          1634
-FAST f=17 a=10       2.754796       4.005059        8          1058
-FAST f=17 a=10       0.022537       4.005059        8          1058
-FAST f=18 a=1       4.779117       4.038214        8          242
-FAST f=18 a=1       0.048814       4.038214        8          242
-FAST f=18 a=2       3.829753       4.045768        8          722
-FAST f=18 a=2       0.036541       4.045768        8          722
-FAST f=18 a=3       3.495053       4.021497        8          770
-FAST f=18 a=3       0.032648       4.021497        8          770
-FAST f=18 a=4       3.221395       4.039623        8          770
-FAST f=18 a=4       0.027818       4.039623        8          770
-FAST f=18 a=5       3.059369       4.050414        8          530
-FAST f=18 a=5       0.026296       4.050414        8          530
-FAST f=18 a=6       3.019292       4.010714        6          962
-FAST f=18 a=6       0.031104       4.010714        6          962
-FAST f=18 a=7       2.949322       4.031439        6          770
-FAST f=18 a=7       0.030745       4.031439        6          770
-FAST f=18 a=8       2.876425       4.032088        6          386
-FAST f=18 a=8       0.027407       4.032088        6          386
-FAST f=18 a=9       2.850958       4.053372        8          674
-FAST f=18 a=9       0.023799       4.053372        8          674
-FAST f=18 a=10       2.884352       4.020148        8          1730
-FAST f=18 a=10       0.024401       4.020148        8          1730
-FAST f=19 a=1       4.815669       4.061203        8          674
-FAST f=19 a=1       0.051425       4.061203        8          674
-FAST f=19 a=2       3.951356       4.013822        8          1442
-FAST f=19 a=2       0.039968       4.013822        8          1442
-FAST f=19 a=3       3.554682       4.050425        8          722
-FAST f=19 a=3       0.032725       4.050425        8          722
-FAST f=19 a=4       3.242585       4.054677        8          722
-FAST f=19 a=4       0.028194       4.054677        8          722
-FAST f=19 a=5       3.105909       4.064524        8          818
-FAST f=19 a=5       0.02675       4.064524        8          818
-FAST f=19 a=6       3.059901       4.036857        8          1250
-FAST f=19 a=6       0.026396       4.036857        8          1250
-FAST f=19 a=7       3.016151       4.068234        6          770
-FAST f=19 a=7       0.031501       4.068234        6          770
-FAST f=19 a=8       2.962902       4.077509        8          530
-FAST f=19 a=8       0.023333       4.077509        8          530
-FAST f=19 a=9       2.899607       4.067328        8          530
-FAST f=19 a=9       0.024553       4.067328        8          530
-FAST f=19 a=10       2.950978       4.059901        8          434
-FAST f=19 a=10       0.023852       4.059901        8          434
-FAST f=20 a=1       5.259834       4.027579        8          1634
-FAST f=20 a=1       0.061123       4.027579        8          1634
-FAST f=20 a=2       4.382150       4.025093        8          1634
-FAST f=20 a=2       0.048009       4.025093        8          1634
-FAST f=20 a=3       4.104323       4.060842        8          530
-FAST f=20 a=3       0.040965       4.060842        8          530
-FAST f=20 a=4       3.853340       4.023504        6          914
-FAST f=20 a=4       0.041072       4.023504        6          914
-FAST f=20 a=5       3.728841       4.018089        6          1634
-FAST f=20 a=5       0.037469       4.018089        6          1634
-FAST f=20 a=6       3.683045       4.069138        8          578
-FAST f=20 a=6       0.028011       4.069138        8          578
-FAST f=20 a=7       3.726973       4.063160        8          722
-FAST f=20 a=7       0.028437       4.063160        8          722
-FAST f=20 a=8       3.555073       4.057690        8          386
-FAST f=20 a=8       0.027588       4.057690        8          386
-FAST f=20 a=9       3.551095       4.067253        8          482
-FAST f=20 a=9       0.025976       4.067253        8          482
-FAST f=20 a=10       3.490127       4.068518        8          530
-FAST f=20 a=10       0.025971       4.068518        8          530
-FAST f=21 a=1       7.343816       4.064945        8          770
-FAST f=21 a=1       0.085035       4.064945        8          770
-FAST f=21 a=2       5.930894       4.048206        8          386
-FAST f=21 a=2       0.067349       4.048206        8          386
-FAST f=21 a=3       6.770775       4.063417        8          578
-FAST f=21 a=3       0.077104       4.063417        8          578
-FAST f=21 a=4       6.889409       4.066761        8          626
-FAST f=21 a=4       0.0717       4.066761        8          626
-FAST f=21 a=5       6.714896       4.051813        8          914
-FAST f=21 a=5       0.071026       4.051813        8          914
-FAST f=21 a=6       6.539890       4.047263        8          1922
-FAST f=21 a=6       0.07127       4.047263        8          1922
-FAST f=21 a=7       6.511052       4.068373        8          482
-FAST f=21 a=7       0.065467       4.068373        8          482
-FAST f=21 a=8       6.458788       4.071597        8          482
-FAST f=21 a=8       0.063817       4.071597        8          482
-FAST f=21 a=9       6.377591       4.052905        8          434
-FAST f=21 a=9       0.063112       4.052905        8          434
-FAST f=21 a=10       6.360752       4.047773        8          530
-FAST f=21 a=10       0.063606       4.047773        8          530
-FAST f=22 a=1       10.523471       4.040812        8          962
-FAST f=22 a=1       0.14214       4.040812        8          962
-FAST f=22 a=2       9.454758       4.059396        8          914
-FAST f=22 a=2       0.118343       4.059396        8          914
-FAST f=22 a=3       9.043197       4.043019        8          1922
-FAST f=22 a=3       0.109798       4.043019        8          1922
-FAST f=22 a=4       8.716261       4.044819        8          770
-FAST f=22 a=4       0.099687       4.044819        8          770
-FAST f=22 a=5       8.529472       4.070576        8          530
-FAST f=22 a=5       0.093127       4.070576        8          530
-FAST f=22 a=6       8.424241       4.070565        8          722
-FAST f=22 a=6       0.093703       4.070565        8          722
-FAST f=22 a=7       8.403391       4.070591        8          578
-FAST f=22 a=7       0.089763       4.070591        8          578
-FAST f=22 a=8       8.285221       4.089171        8          530
-FAST f=22 a=8       0.087716       4.089171        8          530
-FAST f=22 a=9       8.282506       4.047470        8          722
-FAST f=22 a=9       0.089773       4.047470        8          722
-FAST f=22 a=10       8.241809       4.064151        8          818
-FAST f=22 a=10       0.090413       4.064151        8          818
-FAST f=23 a=1       12.389208       4.051635        6          530
-FAST f=23 a=1       0.147796       4.051635        6          530
-FAST f=23 a=2       11.300910       4.042835        6          914
-FAST f=23 a=2       0.133178       4.042835        6          914
-FAST f=23 a=3       10.879455       4.047415        8          626
-FAST f=23 a=3       0.129571       4.047415        8          626
-FAST f=23 a=4       10.522718       4.038269        6          914
-FAST f=23 a=4       0.118121       4.038269        6          914
-FAST f=23 a=5       10.348043       4.066884        8          434
-FAST f=23 a=5       0.112098       4.066884        8          434
-FAST f=23 a=6       10.238630       4.048635        8          1010
-FAST f=23 a=6       0.120281       4.048635        8          1010
-FAST f=23 a=7       10.213255       4.061809        8          530
-FAST f=23 a=7       0.1121       4.061809        8          530
-FAST f=23 a=8       10.107879       4.074104        8          818
-FAST f=23 a=8       0.116544       4.074104        8          818
-FAST f=23 a=9       10.063424       4.064811        8          674
-FAST f=23 a=9       0.109045       4.064811        8          674
-FAST f=23 a=10       10.035801       4.054918        8          530
-FAST f=23 a=10       0.108735       4.054918        8          530
-FAST f=24 a=1       14.963878       4.073490        8          722
-FAST f=24 a=1       0.206344       4.073490        8          722
-FAST f=24 a=2       13.833472       4.036100        8          962
-FAST f=24 a=2       0.17486       4.036100        8          962
-FAST f=24 a=3       13.404631       4.026281        6          1106
-FAST f=24 a=3       0.153961       4.026281        6          1106
-FAST f=24 a=4       13.041164       4.065448        8          674
-FAST f=24 a=4       0.155509       4.065448        8          674
-FAST f=24 a=5       12.879412       4.054636        8          674
-FAST f=24 a=5       0.148282       4.054636        8          674
-FAST f=24 a=6       12.773736       4.081376        8          530
-FAST f=24 a=6       0.142563       4.081376        8          530
-FAST f=24 a=7       12.711310       4.059834        8          770
-FAST f=24 a=7       0.149321       4.059834        8          770
-FAST f=24 a=8       12.635459       4.052050        8          1298
-FAST f=24 a=8       0.15095       4.052050        8          1298
-FAST f=24 a=9       12.558104       4.076516        8          722
-FAST f=24 a=9       0.144361       4.076516        8          722
-FAST f=24 a=10       10.661348       4.062137        8          818
-FAST f=24 a=10       0.108232       4.062137        8          818
-
-
-hg-changelog:
-NODICT       0.000017       1.377590        
-RANDOM       0.186171       2.097487        
-LEGACY       1.670867       2.058907        
-COVER       173.561948       2.189685        8          98
-COVER       4.811180       2.189685        8          98
-FAST f=15 a=1       18.685906       2.129682        8          434
-FAST f=15 a=1       0.173376       2.129682        8          434
-FAST f=15 a=2       12.928259       2.131890        8          482
-FAST f=15 a=2       0.102582       2.131890        8          482
-FAST f=15 a=3       11.132343       2.128027        8          386
-FAST f=15 a=3       0.077122       2.128027        8          386
-FAST f=15 a=4       10.120683       2.125797        8          434
-FAST f=15 a=4       0.065175       2.125797        8          434
-FAST f=15 a=5       9.479092       2.127697        8          386
-FAST f=15 a=5       0.057905       2.127697        8          386
-FAST f=15 a=6       9.159523       2.127132        8          1682
-FAST f=15 a=6       0.058604       2.127132        8          1682
-FAST f=15 a=7       8.724003       2.129914        8          434
-FAST f=15 a=7       0.0493       2.129914        8          434
-FAST f=15 a=8       8.595001       2.127137        8          338
-FAST f=15 a=8       0.0474       2.127137        8          338
-FAST f=15 a=9       8.356405       2.125512        8          482
-FAST f=15 a=9       0.046126       2.125512        8          482
-FAST f=15 a=10       8.207111       2.126066        8          338
-FAST f=15 a=10       0.043292       2.126066        8          338
-FAST f=16 a=1       18.464436       2.144040        8          242
-FAST f=16 a=1       0.172156       2.144040        8          242
-FAST f=16 a=2       12.844825       2.148171        8          194
-FAST f=16 a=2       0.099619       2.148171        8          194
-FAST f=16 a=3       11.082568       2.140837        8          290
-FAST f=16 a=3       0.079165       2.140837        8          290
-FAST f=16 a=4       10.066749       2.144405        8          386
-FAST f=16 a=4       0.068411       2.144405        8          386
-FAST f=16 a=5       9.501121       2.140720        8          386
-FAST f=16 a=5       0.061316       2.140720        8          386
-FAST f=16 a=6       9.179332       2.139478        8          386
-FAST f=16 a=6       0.056322       2.139478        8          386
-FAST f=16 a=7       8.849438       2.142412        8          194
-FAST f=16 a=7       0.050493       2.142412        8          194
-FAST f=16 a=8       8.810919       2.143454        8          434
-FAST f=16 a=8       0.051304       2.143454        8          434
-FAST f=16 a=9       8.553900       2.140339        8          194
-FAST f=16 a=9       0.047285       2.140339        8          194
-FAST f=16 a=10       8.398027       2.143130        8          386
-FAST f=16 a=10       0.046386       2.143130        8          386
-FAST f=17 a=1       18.644657       2.157192        8          98
-FAST f=17 a=1       0.173884       2.157192        8          98
-FAST f=17 a=2       13.071242       2.159830        8          146
-FAST f=17 a=2       0.10388       2.159830        8          146
-FAST f=17 a=3       11.332366       2.153654        6          194
-FAST f=17 a=3       0.08983       2.153654        6          194
-FAST f=17 a=4       10.362413       2.156813        8          242
-FAST f=17 a=4       0.070389       2.156813        8          242
-FAST f=17 a=5       9.808159       2.155098        6          338
-FAST f=17 a=5       0.072661       2.155098        6          338
-FAST f=17 a=6       9.451165       2.153845        6          146
-FAST f=17 a=6       0.064959       2.153845        6          146
-FAST f=17 a=7       9.163097       2.155424        6          242
-FAST f=17 a=7       0.064323       2.155424        6          242
-FAST f=17 a=8       9.047276       2.156640        8          242
-FAST f=17 a=8       0.053382       2.156640        8          242
-FAST f=17 a=9       8.807671       2.152396        8          146
-FAST f=17 a=9       0.049617       2.152396        8          146
-FAST f=17 a=10       8.649827       2.152370        8          146
-FAST f=17 a=10       0.047849       2.152370        8          146
-FAST f=18 a=1       18.809502       2.168116        8          98
-FAST f=18 a=1       0.175226       2.168116        8          98
-FAST f=18 a=2       13.756502       2.170870        6          242
-FAST f=18 a=2       0.119507       2.170870        6          242
-FAST f=18 a=3       12.059748       2.163094        6          98
-FAST f=18 a=3       0.093912       2.163094        6          98
-FAST f=18 a=4       11.410294       2.172372        8          98
-FAST f=18 a=4       0.073048       2.172372        8          98
-FAST f=18 a=5       10.560297       2.166388        8          98
-FAST f=18 a=5       0.065136       2.166388        8          98
-FAST f=18 a=6       10.071390       2.162672        8          98
-FAST f=18 a=6       0.059402       2.162672        8          98
-FAST f=18 a=7       10.084214       2.166624        6          194
-FAST f=18 a=7       0.073276       2.166624        6          194
-FAST f=18 a=8       9.953226       2.167454        8          98
-FAST f=18 a=8       0.053659       2.167454        8          98
-FAST f=18 a=9       8.982461       2.161593        6          146
-FAST f=18 a=9       0.05955       2.161593        6          146
-FAST f=18 a=10       8.986092       2.164373        6          242
-FAST f=18 a=10       0.059135       2.164373        6          242
-FAST f=19 a=1       18.908277       2.176021        8          98
-FAST f=19 a=1       0.177316       2.176021        8          98
-FAST f=19 a=2       13.471313       2.176103        8          98
-FAST f=19 a=2       0.106344       2.176103        8          98
-FAST f=19 a=3       11.571406       2.172812        8          98
-FAST f=19 a=3       0.083293       2.172812        8          98
-FAST f=19 a=4       10.632775       2.177770        6          146
-FAST f=19 a=4       0.079864       2.177770        6          146
-FAST f=19 a=5       10.030190       2.175574        6          146
-FAST f=19 a=5       0.07223       2.175574        6          146
-FAST f=19 a=6       9.717818       2.169997        8          98
-FAST f=19 a=6       0.060049       2.169997        8          98
-FAST f=19 a=7       9.397531       2.172770        8          146
-FAST f=19 a=7       0.057188       2.172770        8          146
-FAST f=19 a=8       9.281061       2.175822        8          98
-FAST f=19 a=8       0.053711       2.175822        8          98
-FAST f=19 a=9       9.165242       2.169849        6          146
-FAST f=19 a=9       0.059898       2.169849        6          146
-FAST f=19 a=10       9.048763       2.173394        8          98
-FAST f=19 a=10       0.049757       2.173394        8          98
-FAST f=20 a=1       21.166917       2.183923        6          98
-FAST f=20 a=1       0.205425       2.183923        6          98
-FAST f=20 a=2       15.642753       2.182349        6          98
-FAST f=20 a=2       0.135957       2.182349        6          98
-FAST f=20 a=3       14.053730       2.173544        6          98
-FAST f=20 a=3       0.11266       2.173544        6          98
-FAST f=20 a=4       15.270019       2.183656        8          98
-FAST f=20 a=4       0.107892       2.183656        8          98
-FAST f=20 a=5       15.497927       2.174661        6          98
-FAST f=20 a=5       0.100305       2.174661        6          98
-FAST f=20 a=6       13.973505       2.172391        8          98
-FAST f=20 a=6       0.087565       2.172391        8          98
-FAST f=20 a=7       14.083296       2.172443        8          98
-FAST f=20 a=7       0.078062       2.172443        8          98
-FAST f=20 a=8       12.560048       2.175581        8          98
-FAST f=20 a=8       0.070282       2.175581        8          98
-FAST f=20 a=9       13.078645       2.173975        6          146
-FAST f=20 a=9       0.081041       2.173975        6          146
-FAST f=20 a=10       12.823328       2.177778        8          98
-FAST f=20 a=10       0.074522       2.177778        8          98
-FAST f=21 a=1       29.825370       2.183057        6          98
-FAST f=21 a=1       0.334453       2.183057        6          98
-FAST f=21 a=2       29.476474       2.182752        8          98
-FAST f=21 a=2       0.286602       2.182752        8          98
-FAST f=21 a=3       25.937186       2.175867        8          98
-FAST f=21 a=3       0.17626       2.175867        8          98
-FAST f=21 a=4       20.413865       2.179780        8          98
-FAST f=21 a=4       0.206085       2.179780        8          98
-FAST f=21 a=5       20.541889       2.178328        6          146
-FAST f=21 a=5       0.199157       2.178328        6          146
-FAST f=21 a=6       21.090670       2.174443        6          146
-FAST f=21 a=6       0.190645       2.174443        6          146
-FAST f=21 a=7       20.221569       2.177384        6          146
-FAST f=21 a=7       0.184278       2.177384        6          146
-FAST f=21 a=8       20.322357       2.179456        6          98
-FAST f=21 a=8       0.178458       2.179456        6          98
-FAST f=21 a=9       20.683912       2.174396        6          146
-FAST f=21 a=9       0.190829       2.174396        6          146
-FAST f=21 a=10       20.840865       2.174905        8          98
-FAST f=21 a=10       0.172515       2.174905        8          98
-FAST f=22 a=1       36.822827       2.181612        6          98
-FAST f=22 a=1       0.437389       2.181612        6          98
-FAST f=22 a=2       30.616902       2.183142        8          98
-FAST f=22 a=2       0.324284       2.183142        8          98
-FAST f=22 a=3       28.472482       2.178130        8          98
-FAST f=22 a=3       0.236538       2.178130        8          98
-FAST f=22 a=4       25.847028       2.181878        8          98
-FAST f=22 a=4       0.263744       2.181878        8          98
-FAST f=22 a=5       27.095881       2.180775        8          98
-FAST f=22 a=5       0.24988       2.180775        8          98
-FAST f=22 a=6       25.939172       2.170916        8          98
-FAST f=22 a=6       0.240033       2.170916        8          98
-FAST f=22 a=7       27.064194       2.177849        8          98
-FAST f=22 a=7       0.242383       2.177849        8          98
-FAST f=22 a=8       25.140221       2.178216        8          98
-FAST f=22 a=8       0.237601       2.178216        8          98
-FAST f=22 a=9       25.505283       2.177455        6          146
-FAST f=22 a=9       0.223217       2.177455        6          146
-FAST f=22 a=10       24.529362       2.176705        6          98
-FAST f=22 a=10       0.222876       2.176705        6          98
-FAST f=23 a=1       39.127310       2.183006        6          98
-FAST f=23 a=1       0.417338       2.183006        6          98
-FAST f=23 a=2       32.468161       2.183524        6          98
-FAST f=23 a=2       0.351645       2.183524        6          98
-FAST f=23 a=3       31.577620       2.172604        6          98
-FAST f=23 a=3       0.319659       2.172604        6          98
-FAST f=23 a=4       30.129247       2.183932        6          98
-FAST f=23 a=4       0.307239       2.183932        6          98
-FAST f=23 a=5       29.103376       2.183529        6          146
-FAST f=23 a=5       0.285533       2.183529        6          146
-FAST f=23 a=6       29.776045       2.174367        8          98
-FAST f=23 a=6       0.276846       2.174367        8          98
-FAST f=23 a=7       28.940407       2.178022        6          146
-FAST f=23 a=7       0.274082       2.178022        6          146
-FAST f=23 a=8       29.256009       2.179462        6          98
-FAST f=23 a=8       0.26949       2.179462        6          98
-FAST f=23 a=9       29.347312       2.170407        8          98
-FAST f=23 a=9       0.265034       2.170407        8          98
-FAST f=23 a=10       29.140081       2.171762        8          98
-FAST f=23 a=10       0.259183       2.171762        8          98
-FAST f=24 a=1       44.871179       2.182115        6          98
-FAST f=24 a=1       0.509433       2.182115        6          98
-FAST f=24 a=2       38.694867       2.180549        8          98
-FAST f=24 a=2       0.406695       2.180549        8          98
-FAST f=24 a=3       38.363769       2.172821        8          98
-FAST f=24 a=3       0.359581       2.172821        8          98
-FAST f=24 a=4       36.580797       2.184142        8          98
-FAST f=24 a=4       0.340614       2.184142        8          98
-FAST f=24 a=5       33.125701       2.183301        8          98
-FAST f=24 a=5       0.324874       2.183301        8          98
-FAST f=24 a=6       34.776068       2.173019        6          146
-FAST f=24 a=6       0.340397       2.173019        6          146
-FAST f=24 a=7       34.417625       2.176561        6          146
-FAST f=24 a=7       0.308223       2.176561        6          146
-FAST f=24 a=8       35.470291       2.182161        6          98
-FAST f=24 a=8       0.307724       2.182161        6          98
-FAST f=24 a=9       34.927252       2.172682        6          146
-FAST f=24 a=9       0.300598       2.172682        6          146
-FAST f=24 a=10       33.238355       2.173395        6          98
-FAST f=24 a=10       0.249916       2.173395        6          98
-
-
-hg-manifest:
-NODICT       0.000004       1.866377        
-RANDOM       0.696346       2.309436        
-LEGACY       7.064527       2.506977        
-COVER       876.312865       2.582528        8          434
-COVER       35.684533       2.582528        8          434
-FAST f=15 a=1       76.618201       2.404013        8          1202
-FAST f=15 a=1       0.700722       2.404013        8          1202
-FAST f=15 a=2       49.213058       2.409248        6          1826
-FAST f=15 a=2       0.473393       2.409248        6          1826
-FAST f=15 a=3       41.753197       2.409677        8          1490
-FAST f=15 a=3       0.336848       2.409677        8          1490
-FAST f=15 a=4       38.648295       2.407996        8          1538
-FAST f=15 a=4       0.283952       2.407996        8          1538
-FAST f=15 a=5       36.144936       2.402895        8          1874
-FAST f=15 a=5       0.270128       2.402895        8          1874
-FAST f=15 a=6       35.484675       2.394873        8          1586
-FAST f=15 a=6       0.251637       2.394873        8          1586
-FAST f=15 a=7       34.280599       2.397311        8          1778
-FAST f=15 a=7       0.23984       2.397311        8          1778
-FAST f=15 a=8       32.122572       2.396089        6          1490
-FAST f=15 a=8       0.251508       2.396089        6          1490
-FAST f=15 a=9       29.909842       2.390092        6          1970
-FAST f=15 a=9       0.251233       2.390092        6          1970
-FAST f=15 a=10       30.102938       2.400086        6          1682
-FAST f=15 a=10       0.23688       2.400086        6          1682
-FAST f=16 a=1       67.750401       2.475460        6          1346
-FAST f=16 a=1       0.796035       2.475460        6          1346
-FAST f=16 a=2       52.812027       2.480860        6          1730
-FAST f=16 a=2       0.480384       2.480860        6          1730
-FAST f=16 a=3       44.179259       2.469304        8          1970
-FAST f=16 a=3       0.332657       2.469304        8          1970
-FAST f=16 a=4       37.612728       2.478208        6          1970
-FAST f=16 a=4       0.32498       2.478208        6          1970
-FAST f=16 a=5       35.056222       2.475568        6          1298
-FAST f=16 a=5       0.302824       2.475568        6          1298
-FAST f=16 a=6       34.713012       2.486079        8          1730
-FAST f=16 a=6       0.24755       2.486079        8          1730
-FAST f=16 a=7       33.713687       2.477180        6          1682
-FAST f=16 a=7       0.280358       2.477180        6          1682
-FAST f=16 a=8       31.571412       2.475418        8          1538
-FAST f=16 a=8       0.241241       2.475418        8          1538
-FAST f=16 a=9       31.608069       2.478263        8          1922
-FAST f=16 a=9       0.241764       2.478263        8          1922
-FAST f=16 a=10       31.358002       2.472263        8          1442
-FAST f=16 a=10       0.221661       2.472263        8          1442
-FAST f=17 a=1       66.185775       2.536085        6          1346
-FAST f=17 a=1       0.713549       2.536085        6          1346
-FAST f=17 a=2       50.365000       2.546105        8          1298
-FAST f=17 a=2       0.467846       2.546105        8          1298
-FAST f=17 a=3       42.712843       2.536250        8          1298
-FAST f=17 a=3       0.34047       2.536250        8          1298
-FAST f=17 a=4       39.514227       2.535555        8          1442
-FAST f=17 a=4       0.302989       2.535555        8          1442
-FAST f=17 a=5       35.189292       2.524925        8          1202
-FAST f=17 a=5       0.273451       2.524925        8          1202
-FAST f=17 a=6       35.791683       2.523466        8          1202
-FAST f=17 a=6       0.268261       2.523466        8          1202
-FAST f=17 a=7       37.416136       2.526625        6          1010
-FAST f=17 a=7       0.277558       2.526625        6          1010
-FAST f=17 a=8       37.084707       2.533274        6          1250
-FAST f=17 a=8       0.285104       2.533274        6          1250
-FAST f=17 a=9       34.183814       2.532765        8          1298
-FAST f=17 a=9       0.235133       2.532765        8          1298
-FAST f=17 a=10       31.149235       2.528722        8          1346
-FAST f=17 a=10       0.232679       2.528722        8          1346
-FAST f=18 a=1       72.942176       2.559857        6          386
-FAST f=18 a=1       0.718618       2.559857        6          386
-FAST f=18 a=2       51.690440       2.559572        8          290
-FAST f=18 a=2       0.403978       2.559572        8          290
-FAST f=18 a=3       45.344908       2.561040        8          962
-FAST f=18 a=3       0.357205       2.561040        8          962
-FAST f=18 a=4       39.804522       2.558446        8          1010
-FAST f=18 a=4       0.310526       2.558446        8          1010
-FAST f=18 a=5       38.134888       2.561811        8          626
-FAST f=18 a=5       0.273743       2.561811        8          626
-FAST f=18 a=6       35.091890       2.555518        8          722
-FAST f=18 a=6       0.260135       2.555518        8          722
-FAST f=18 a=7       34.639523       2.562938        8          290
-FAST f=18 a=7       0.234294       2.562938        8          290
-FAST f=18 a=8       36.076431       2.563567        8          1586
-FAST f=18 a=8       0.274075       2.563567        8          1586
-FAST f=18 a=9       36.376433       2.560950        8          722
-FAST f=18 a=9       0.240106       2.560950        8          722
-FAST f=18 a=10       32.624790       2.559340        8          578
-FAST f=18 a=10       0.234704       2.559340        8          578
-FAST f=19 a=1       70.513761       2.572441        8          194
-FAST f=19 a=1       0.726112       2.572441        8          194
-FAST f=19 a=2       59.263032       2.574560        8          482
-FAST f=19 a=2       0.451554       2.574560        8          482
-FAST f=19 a=3       51.509594       2.571546        6          194
-FAST f=19 a=3       0.393014       2.571546        6          194
-FAST f=19 a=4       55.393906       2.573386        8          482
-FAST f=19 a=4       0.38819       2.573386        8          482
-FAST f=19 a=5       43.201736       2.567589        8          674
-FAST f=19 a=5       0.292155       2.567589        8          674
-FAST f=19 a=6       42.911687       2.572666        6          434
-FAST f=19 a=6       0.303988       2.572666        6          434
-FAST f=19 a=7       44.687591       2.573613        6          290
-FAST f=19 a=7       0.308721       2.573613        6          290
-FAST f=19 a=8       37.372868       2.571039        6          194
-FAST f=19 a=8       0.287137       2.571039        6          194
-FAST f=19 a=9       36.074230       2.566473        6          482
-FAST f=19 a=9       0.280721       2.566473        6          482
-FAST f=19 a=10       33.731720       2.570306        8          194
-FAST f=19 a=10       0.224073       2.570306        8          194
-FAST f=20 a=1       79.670634       2.581146        6          290
-FAST f=20 a=1       0.899986       2.581146        6          290
-FAST f=20 a=2       58.827141       2.579782        8          386
-FAST f=20 a=2       0.602288       2.579782        8          386
-FAST f=20 a=3       51.289004       2.579627        8          722
-FAST f=20 a=3       0.446091       2.579627        8          722
-FAST f=20 a=4       47.711068       2.581508        8          722
-FAST f=20 a=4       0.473007       2.581508        8          722
-FAST f=20 a=5       47.402929       2.578062        6          434
-FAST f=20 a=5       0.497131       2.578062        6          434
-FAST f=20 a=6       54.797102       2.577365        8          482
-FAST f=20 a=6       0.515061       2.577365        8          482
-FAST f=20 a=7       51.370877       2.583050        8          386
-FAST f=20 a=7       0.402878       2.583050        8          386
-FAST f=20 a=8       51.437931       2.574875        6          242
-FAST f=20 a=8       0.453094       2.574875        6          242
-FAST f=20 a=9       44.105456       2.576700        6          242
-FAST f=20 a=9       0.456633       2.576700        6          242
-FAST f=20 a=10       44.447580       2.578305        8          338
-FAST f=20 a=10       0.409121       2.578305        8          338
-FAST f=21 a=1       113.031686       2.582449        6          242
-FAST f=21 a=1       1.456971       2.582449        6          242
-FAST f=21 a=2       97.700932       2.582124        8          194
-FAST f=21 a=2       1.072078       2.582124        8          194
-FAST f=21 a=3       96.563648       2.585479        8          434
-FAST f=21 a=3       0.949528       2.585479        8          434
-FAST f=21 a=4       90.597813       2.582366        6          386
-FAST f=21 a=4       0.76944       2.582366        6          386
-FAST f=21 a=5       86.815980       2.579043        8          434
-FAST f=21 a=5       0.858167       2.579043        8          434
-FAST f=21 a=6       91.235820       2.578378        8          530
-FAST f=21 a=6       0.684274       2.578378        8          530
-FAST f=21 a=7       84.392788       2.581243        8          386
-FAST f=21 a=7       0.814386       2.581243        8          386
-FAST f=21 a=8       82.052310       2.582547        8          338
-FAST f=21 a=8       0.822633       2.582547        8          338
-FAST f=21 a=9       74.696074       2.579319        8          194
-FAST f=21 a=9       0.811028       2.579319        8          194
-FAST f=21 a=10       76.211170       2.578766        8          290
-FAST f=21 a=10       0.809715       2.578766        8          290
-FAST f=22 a=1       138.976871       2.580478        8          194
-FAST f=22 a=1       1.748932       2.580478        8          194
-FAST f=22 a=2       120.164097       2.583633        8          386
-FAST f=22 a=2       1.333239       2.583633        8          386
-FAST f=22 a=3       111.986474       2.582566        6          194
-FAST f=22 a=3       1.305734       2.582566        6          194
-FAST f=22 a=4       108.548148       2.583068        6          194
-FAST f=22 a=4       1.314026       2.583068        6          194
-FAST f=22 a=5       103.173017       2.583495        6          290
-FAST f=22 a=5       1.228664       2.583495        6          290
-FAST f=22 a=6       108.421262       2.582349        8          530
-FAST f=22 a=6       1.076773       2.582349        8          530
-FAST f=22 a=7       103.284127       2.581022        8          386
-FAST f=22 a=7       1.112117       2.581022        8          386
-FAST f=22 a=8       96.330279       2.581073        8          290
-FAST f=22 a=8       1.109303       2.581073        8          290
-FAST f=22 a=9       97.651348       2.580075        6          194
-FAST f=22 a=9       0.933032       2.580075        6          194
-FAST f=22 a=10       101.660621       2.584886        8          194
-FAST f=22 a=10       0.796823       2.584886        8          194
-FAST f=23 a=1       159.322978       2.581474        6          242
-FAST f=23 a=1       2.015878       2.581474        6          242
-FAST f=23 a=2       134.331775       2.581619        8          194
-FAST f=23 a=2       1.545845       2.581619        8          194
-FAST f=23 a=3       127.724552       2.579888        6          338
-FAST f=23 a=3       1.444496       2.579888        6          338
-FAST f=23 a=4       126.077675       2.578137        6          242
-FAST f=23 a=4       1.364394       2.578137        6          242
-FAST f=23 a=5       124.914027       2.580843        8          338
-FAST f=23 a=5       1.116059       2.580843        8          338
-FAST f=23 a=6       122.874153       2.577637        6          338
-FAST f=23 a=6       1.164584       2.577637        6          338
-FAST f=23 a=7       123.099257       2.582715        6          386
-FAST f=23 a=7       1.354042       2.582715        6          386
-FAST f=23 a=8       122.026753       2.577681        8          194
-FAST f=23 a=8       1.210966       2.577681        8          194
-FAST f=23 a=9       121.164312       2.584599        6          290
-FAST f=23 a=9       1.174859       2.584599        6          290
-FAST f=23 a=10       117.462222       2.580358        8          194
-FAST f=23 a=10       1.075258       2.580358        8          194
-FAST f=24 a=1       169.539659       2.581642        6          194
-FAST f=24 a=1       1.916804       2.581642        6          194
-FAST f=24 a=2       160.539270       2.580421        6          290
-FAST f=24 a=2       1.71087       2.580421        6          290
-FAST f=24 a=3       155.455874       2.580449        6          242
-FAST f=24 a=3       1.60307       2.580449        6          242
-FAST f=24 a=4       147.630320       2.582953        6          338
-FAST f=24 a=4       1.396364       2.582953        6          338
-FAST f=24 a=5       133.767428       2.580589        6          290
-FAST f=24 a=5       1.19933       2.580589        6          290
-FAST f=24 a=6       146.437535       2.579453        8          194
-FAST f=24 a=6       1.385405       2.579453        8          194
-FAST f=24 a=7       147.227507       2.584155        8          386
-FAST f=24 a=7       1.48942       2.584155        8          386
-FAST f=24 a=8       138.005773       2.584115        8          194
-FAST f=24 a=8       1.352       2.584115        8          194
-FAST f=24 a=9       141.442625       2.582902        8          290
-FAST f=24 a=9       1.39647       2.582902        8          290
-FAST f=24 a=10       142.157446       2.582701        8          434
-FAST f=24 a=10       1.498889       2.582701        8          434
diff --git a/contrib/experimental_dict_builders/benchmarkDictBuilder/benchmark.c b/contrib/experimental_dict_builders/benchmarkDictBuilder/benchmark.c
deleted file mode 100644
index cd943797bdea..000000000000
--- a/contrib/experimental_dict_builders/benchmarkDictBuilder/benchmark.c
+++ /dev/null
@@ -1,442 +0,0 @@
-#include <stdio.h>  /* fprintf */
-#include <stdlib.h> /* malloc, free, qsort */
-#include <string.h>   /* strcmp, strlen */
-#include <errno.h>    /* errno */
-#include <ctype.h>
-#include <time.h>
-#include "random.h"
-#include "dictBuilder.h"
-#include "zstd_internal.h" /* includes zstd.h */
-#include "io.h"
-#include "util.h"
-#include "zdict.h"
-
-
-
-/*-*************************************
-*  Console display
-***************************************/
-#define DISPLAY(...)         fprintf(stderr, __VA_ARGS__)
-#define DISPLAYLEVEL(l, ...) if (displayLevel>=l) { DISPLAY(__VA_ARGS__); }
-
-static const U64 g_refreshRate = SEC_TO_MICRO / 6;
-static UTIL_time_t g_displayClock = UTIL_TIME_INITIALIZER;
-
-#define DISPLAYUPDATE(l, ...) { if (displayLevel>=l) { \
-            if ((UTIL_clockSpanMicro(g_displayClock) > g_refreshRate) || (displayLevel>=4)) \
-            { g_displayClock = UTIL_getTime(); DISPLAY(__VA_ARGS__); \
-            if (displayLevel>=4) fflush(stderr); } } }
-
-
-/*-*************************************
-*  Exceptions
-***************************************/
-#ifndef DEBUG
-#  define DEBUG 0
-#endif
-#define DEBUGOUTPUT(...) if (DEBUG) DISPLAY(__VA_ARGS__);
-#define EXM_THROW(error, ...)                                             \
-{                                                                         \
-    DEBUGOUTPUT("Error defined at %s, line %i : \n", __FILE__, __LINE__); \
-    DISPLAY("Error %i : ", error);                                        \
-    DISPLAY(__VA_ARGS__);                                                 \
-    DISPLAY("\n");                                                        \
-    exit(error);                                                          \
-}
-
-
-/*-*************************************
-*  Constants
-***************************************/
-static const unsigned g_defaultMaxDictSize = 110 KB;
-#define DEFAULT_CLEVEL 3
-#define DEFAULT_DISPLAYLEVEL 2
-
-
-/*-*************************************
-*  Struct
-***************************************/
-typedef struct {
-  const void* dictBuffer;
-  size_t dictSize;
-} dictInfo;
-
-
-/*-*************************************
-* Dictionary related operations
-***************************************/
-/** createDictFromFiles() :
- *  Based on type of param given, train dictionary using the corresponding algorithm
- *  @return dictInfo containing dictionary buffer and dictionary size
- */
-dictInfo* createDictFromFiles(sampleInfo *info, unsigned maxDictSize,
-                  ZDICT_random_params_t *randomParams, ZDICT_cover_params_t *coverParams,
-                  ZDICT_legacy_params_t *legacyParams, ZDICT_fastCover_params_t *fastParams) {
-    unsigned const displayLevel = randomParams ? randomParams->zParams.notificationLevel :
-                                  coverParams ? coverParams->zParams.notificationLevel :
-                                  legacyParams ? legacyParams->zParams.notificationLevel :
-                                  fastParams ? fastParams->zParams.notificationLevel :
-                                  DEFAULT_DISPLAYLEVEL;   /* no dict */
-    void* const dictBuffer = malloc(maxDictSize);
-
-    dictInfo* dInfo = NULL;
-
-    /* Checks */
-    if (!dictBuffer)
-        EXM_THROW(12, "not enough memory for trainFromFiles");   /* should not happen */
-
-    {   size_t dictSize;
-        if(randomParams) {
-          dictSize = ZDICT_trainFromBuffer_random(dictBuffer, maxDictSize, info->srcBuffer,
-                                               info->samplesSizes, info->nbSamples, *randomParams);
-        }else if(coverParams) {
-          /* Run the optimize version if either k or d is not provided */
-          if (!coverParams->d || !coverParams->k){
-            dictSize = ZDICT_optimizeTrainFromBuffer_cover(dictBuffer, maxDictSize, info->srcBuffer,
-                                                  info->samplesSizes, info->nbSamples, coverParams);
-          } else {
-            dictSize = ZDICT_trainFromBuffer_cover(dictBuffer, maxDictSize, info->srcBuffer,
-                                                  info->samplesSizes, info->nbSamples, *coverParams);
-          }
-        } else if(legacyParams) {
-          dictSize = ZDICT_trainFromBuffer_legacy(dictBuffer, maxDictSize, info->srcBuffer,
-                                               info->samplesSizes, info->nbSamples, *legacyParams);
-        } else if(fastParams) {
-          /* Run the optimize version if either k or d is not provided */
-          if (!fastParams->d || !fastParams->k) {
-            dictSize = ZDICT_optimizeTrainFromBuffer_fastCover(dictBuffer, maxDictSize, info->srcBuffer,
-                                                  info->samplesSizes, info->nbSamples, fastParams);
-          } else {
-            dictSize = ZDICT_trainFromBuffer_fastCover(dictBuffer, maxDictSize, info->srcBuffer,
-                                                  info->samplesSizes, info->nbSamples, *fastParams);
-          }
-        } else {
-          dictSize = 0;
-        }
-        if (ZDICT_isError(dictSize)) {
-            DISPLAYLEVEL(1, "dictionary training failed : %s \n", ZDICT_getErrorName(dictSize));   /* should not happen */
-            free(dictBuffer);
-            return dInfo;
-        }
-        dInfo = (dictInfo *)malloc(sizeof(dictInfo));
-        dInfo->dictBuffer = dictBuffer;
-        dInfo->dictSize = dictSize;
-    }
-    return dInfo;
-}
-
-
-/** compressWithDict() :
- *  Compress samples from sample buffer given dictionary stored on dictionary buffer and compression level
- *  @return compression ratio
- */
-double compressWithDict(sampleInfo *srcInfo, dictInfo* dInfo, int compressionLevel, int displayLevel) {
-  /* Local variables */
-  size_t totalCompressedSize = 0;
-  size_t totalOriginalSize = 0;
-  const unsigned hasDict = dInfo->dictSize > 0 ? 1 : 0;
-  double cRatio;
-  size_t dstCapacity;
-  int i;
-
-  /* Pointers */
-  ZSTD_CDict *cdict = NULL;
-  ZSTD_CCtx* cctx = NULL;
-  size_t *offsets = NULL;
-  void* dst = NULL;
-
-  /* Allocate dst with enough space to compress the maximum sized sample */
-  {
-    size_t maxSampleSize = 0;
-    for (i = 0; i < srcInfo->nbSamples; i++) {
-      maxSampleSize = MAX(srcInfo->samplesSizes[i], maxSampleSize);
-    }
-    dstCapacity = ZSTD_compressBound(maxSampleSize);
-    dst = malloc(dstCapacity);
-  }
-
-  /* Calculate offset for each sample */
-  offsets = (size_t *)malloc((srcInfo->nbSamples + 1) * sizeof(size_t));
-  offsets[0] = 0;
-  for (i = 1; i <= srcInfo->nbSamples; i++) {
-    offsets[i] = offsets[i - 1] + srcInfo->samplesSizes[i - 1];
-  }
-
-  /* Create the cctx */
-  cctx = ZSTD_createCCtx();
-  if(!cctx || !dst) {
-    cRatio = -1;
-    goto _cleanup;
-  }
-
-  /* Create CDict if there's a dictionary stored on buffer */
-  if (hasDict) {
-    cdict = ZSTD_createCDict(dInfo->dictBuffer, dInfo->dictSize, compressionLevel);
-    if(!cdict) {
-      cRatio = -1;
-      goto _cleanup;
-    }
-  }
-
-  /* Compress each sample and sum their sizes*/
-  const BYTE *const samples = (const BYTE *)srcInfo->srcBuffer;
-  for (i = 0; i < srcInfo->nbSamples; i++) {
-    size_t compressedSize;
-    if(hasDict) {
-      compressedSize = ZSTD_compress_usingCDict(cctx, dst, dstCapacity, samples + offsets[i], srcInfo->samplesSizes[i], cdict);
-    } else {
-      compressedSize = ZSTD_compressCCtx(cctx, dst, dstCapacity,samples + offsets[i], srcInfo->samplesSizes[i], compressionLevel);
-    }
-    if (ZSTD_isError(compressedSize)) {
-      cRatio = -1;
-      goto _cleanup;
-    }
-    totalCompressedSize += compressedSize;
-  }
-
-  /* Sum original sizes */
-  for (i = 0; i<srcInfo->nbSamples; i++) {
-    totalOriginalSize += srcInfo->samplesSizes[i];
-  }
-
-  /* Calculate compression ratio */
-  DISPLAYLEVEL(2, "original size is %lu\n", totalOriginalSize);
-  DISPLAYLEVEL(2, "compressed size is %lu\n", totalCompressedSize);
-  cRatio = (double)totalOriginalSize/(double)totalCompressedSize;
-
-_cleanup:
-  free(dst);
-  free(offsets);
-  ZSTD_freeCCtx(cctx);
-  ZSTD_freeCDict(cdict);
-  return cRatio;
-}
-
-
-/** FreeDictInfo() :
- *  Free memory allocated for dictInfo
- */
-void freeDictInfo(dictInfo* info) {
-  if (!info) return;
-  if (info->dictBuffer) free((void*)(info->dictBuffer));
-  free(info);
-}
-
-
-
-/*-********************************************************
-  *  Benchmarking functions
-**********************************************************/
-/** benchmarkDictBuilder() :
- *  Measure how long a dictionary builder takes and compression ratio with the dictionary built
- *  @return 0 if benchmark successfully, 1 otherwise
- */
-int benchmarkDictBuilder(sampleInfo *srcInfo, unsigned maxDictSize, ZDICT_random_params_t *randomParam,
-                        ZDICT_cover_params_t *coverParam, ZDICT_legacy_params_t *legacyParam,
-                        ZDICT_fastCover_params_t *fastParam) {
-  /* Local variables */
-  const unsigned displayLevel = randomParam ? randomParam->zParams.notificationLevel :
-                                coverParam ? coverParam->zParams.notificationLevel :
-                                legacyParam ? legacyParam->zParams.notificationLevel :
-                                fastParam ? fastParam->zParams.notificationLevel:
-                                DEFAULT_DISPLAYLEVEL;   /* no dict */
-  const char* name = randomParam ? "RANDOM" :
-                    coverParam ? "COVER" :
-                    legacyParam ? "LEGACY" :
-                    fastParam ? "FAST":
-                    "NODICT";    /* no dict */
-  const unsigned cLevel = randomParam ? randomParam->zParams.compressionLevel :
-                          coverParam ? coverParam->zParams.compressionLevel :
-                          legacyParam ? legacyParam->zParams.compressionLevel :
-                          fastParam ? fastParam->zParams.compressionLevel:
-                          DEFAULT_CLEVEL;   /* no dict */
-  int result = 0;
-
-  /* Calculate speed */
-  const UTIL_time_t begin = UTIL_getTime();
-  dictInfo* dInfo = createDictFromFiles(srcInfo, maxDictSize, randomParam, coverParam, legacyParam, fastParam);
-  const U64 timeMicro = UTIL_clockSpanMicro(begin);
-  const double timeSec = timeMicro / (double)SEC_TO_MICRO;
-  if (!dInfo) {
-    DISPLAYLEVEL(1, "%s does not train successfully\n", name);
-    result = 1;
-    goto _cleanup;
-  }
-  DISPLAYLEVEL(1, "%s took %f seconds to execute \n", name, timeSec);
-
-  /* Calculate compression ratio */
-  const double cRatio = compressWithDict(srcInfo, dInfo, cLevel, displayLevel);
-  if (cRatio < 0) {
-    DISPLAYLEVEL(1, "Compressing with %s dictionary does not work\n", name);
-    result = 1;
-    goto _cleanup;
-
-  }
-  DISPLAYLEVEL(1, "Compression ratio with %s dictionary is %f\n", name, cRatio);
-
-_cleanup:
-  freeDictInfo(dInfo);
-  return result;
-}
-
-
-
-int main(int argCount, const char* argv[])
-{
-  const int displayLevel = DEFAULT_DISPLAYLEVEL;
-  const char* programName = argv[0];
-  int result = 0;
-
-  /* Initialize arguments to default values */
-  unsigned k = 200;
-  unsigned d = 8;
-  unsigned f;
-  unsigned accel;
-  unsigned i;
-  const unsigned cLevel = DEFAULT_CLEVEL;
-  const unsigned dictID = 0;
-  const unsigned maxDictSize = g_defaultMaxDictSize;
-
-  /* Initialize table to store input files */
-  const char** filenameTable = (const char**)malloc(argCount * sizeof(const char*));
-  unsigned filenameIdx = 0;
-
-  char* fileNamesBuf = NULL;
-  unsigned fileNamesNb = filenameIdx;
-  const int followLinks = 0;
-  const char** extendedFileList = NULL;
-
-  /* Parse arguments */
-  for (i = 1; i < argCount; i++) {
-    const char* argument = argv[i];
-    if (longCommandWArg(&argument, "in=")) {
-      filenameTable[filenameIdx] = argument;
-      filenameIdx++;
-      continue;
-    }
-    DISPLAYLEVEL(1, "benchmark: Incorrect parameters\n");
-    return 1;
-  }
-
-  /* Get the list of all files recursively (because followLinks==0)*/
-  extendedFileList = UTIL_createFileList(filenameTable, filenameIdx, &fileNamesBuf,
-                                        &fileNamesNb, followLinks);
-  if (extendedFileList) {
-    unsigned u;
-    for (u=0; u<fileNamesNb; u++) DISPLAYLEVEL(4, "%u %s\n", u, extendedFileList[u]);
-    free((void*)filenameTable);
-    filenameTable = extendedFileList;
-    filenameIdx = fileNamesNb;
-  }
-
-  /* get sampleInfo */
-  size_t blockSize = 0;
-  sampleInfo* srcInfo= getSampleInfo(filenameTable,
-                    filenameIdx, blockSize, maxDictSize, displayLevel);
-
-  /* set up zParams */
-  ZDICT_params_t zParams;
-  zParams.compressionLevel = cLevel;
-  zParams.notificationLevel = displayLevel;
-  zParams.dictID = dictID;
-
-  /* with no dict */
-  {
-    const int noDictResult = benchmarkDictBuilder(srcInfo, maxDictSize, NULL, NULL, NULL, NULL);
-    if(noDictResult) {
-      result = 1;
-      goto _cleanup;
-    }
-  }
-
-  /* for random */
-  {
-    ZDICT_random_params_t randomParam;
-    randomParam.zParams = zParams;
-    randomParam.k = k;
-    const int randomResult = benchmarkDictBuilder(srcInfo, maxDictSize, &randomParam, NULL, NULL, NULL);
-    DISPLAYLEVEL(2, "k=%u\n", randomParam.k);
-    if(randomResult) {
-      result = 1;
-      goto _cleanup;
-    }
-  }
-
-  /* for legacy */
-  {
-    ZDICT_legacy_params_t legacyParam;
-    legacyParam.zParams = zParams;
-    legacyParam.selectivityLevel = 9;
-    const int legacyResult = benchmarkDictBuilder(srcInfo, maxDictSize, NULL, NULL, &legacyParam, NULL);
-    DISPLAYLEVEL(2, "selectivityLevel=%u\n", legacyParam.selectivityLevel);
-    if(legacyResult) {
-      result = 1;
-      goto _cleanup;
-    }
-  }
-
-  /* for cover */
-  {
-    /* for cover (optimizing k and d) */
-    ZDICT_cover_params_t coverParam;
-    memset(&coverParam, 0, sizeof(coverParam));
-    coverParam.zParams = zParams;
-    coverParam.splitPoint = 1.0;
-    coverParam.steps = 40;
-    coverParam.nbThreads = 1;
-    const int coverOptResult = benchmarkDictBuilder(srcInfo, maxDictSize, NULL, &coverParam, NULL, NULL);
-    DISPLAYLEVEL(2, "k=%u\nd=%u\nsteps=%u\nsplit=%u\n", coverParam.k, coverParam.d, coverParam.steps, (unsigned)(coverParam.splitPoint * 100));
-    if(coverOptResult) {
-      result = 1;
-      goto _cleanup;
-    }
-
-    /* for cover (with k and d provided) */
-    const int coverResult = benchmarkDictBuilder(srcInfo, maxDictSize, NULL, &coverParam, NULL, NULL);
-    DISPLAYLEVEL(2, "k=%u\nd=%u\nsteps=%u\nsplit=%u\n", coverParam.k, coverParam.d, coverParam.steps, (unsigned)(coverParam.splitPoint * 100));
-    if(coverResult) {
-      result = 1;
-      goto _cleanup;
-    }
-
-  }
-
-  /* for fastCover */
-  for (f = 15; f < 25; f++){
-    DISPLAYLEVEL(2, "current f is %u\n", f);
-    for (accel = 1; accel < 11; accel++) {
-      DISPLAYLEVEL(2, "current accel is %u\n", accel);
-      /* for fastCover (optimizing k and d) */
-      ZDICT_fastCover_params_t fastParam;
-      memset(&fastParam, 0, sizeof(fastParam));
-      fastParam.zParams = zParams;
-      fastParam.f = f;
-      fastParam.steps = 40;
-      fastParam.nbThreads = 1;
-      fastParam.accel = accel;
-      const int fastOptResult = benchmarkDictBuilder(srcInfo, maxDictSize, NULL, NULL, NULL, &fastParam);
-      DISPLAYLEVEL(2, "k=%u\nd=%u\nf=%u\nsteps=%u\nsplit=%u\naccel=%u\n", fastParam.k, fastParam.d, fastParam.f, fastParam.steps, (unsigned)(fastParam.splitPoint * 100), fastParam.accel);
-      if(fastOptResult) {
-        result = 1;
-        goto _cleanup;
-      }
-
-      /* for fastCover (with k and d provided) */
-      for (i = 0; i < 5; i++) {
-        const int fastResult = benchmarkDictBuilder(srcInfo, maxDictSize, NULL, NULL, NULL, &fastParam);
-        DISPLAYLEVEL(2, "k=%u\nd=%u\nf=%u\nsteps=%u\nsplit=%u\naccel=%u\n", fastParam.k, fastParam.d, fastParam.f, fastParam.steps, (unsigned)(fastParam.splitPoint * 100), fastParam.accel);
-        if(fastResult) {
-          result = 1;
-          goto _cleanup;
-        }
-      }
-    }
-  }
-
-
-  /* Free allocated memory */
-_cleanup:
-  UTIL_freeFileList(extendedFileList, fileNamesBuf);
-  freeSampleInfo(srcInfo);
-  return result;
-}
diff --git a/contrib/experimental_dict_builders/benchmarkDictBuilder/dictBuilder.h b/contrib/experimental_dict_builders/benchmarkDictBuilder/dictBuilder.h
deleted file mode 100644
index 781ec8c2f39e..000000000000
--- a/contrib/experimental_dict_builders/benchmarkDictBuilder/dictBuilder.h
+++ /dev/null
@@ -1,6 +0,0 @@
-/* ZDICT_trainFromBuffer_legacy() :
- * issue : samplesBuffer need to be followed by a noisy guard band.
- * work around : duplicate the buffer, and add the noise */
-size_t ZDICT_trainFromBuffer_legacy(void* dictBuffer, size_t dictBufferCapacity,
-                                    const void* samplesBuffer, const size_t* samplesSizes, unsigned nbSamples,
-                                    ZDICT_legacy_params_t params);
diff --git a/contrib/experimental_dict_builders/benchmarkDictBuilder/test.sh b/contrib/experimental_dict_builders/benchmarkDictBuilder/test.sh
deleted file mode 100755
index 5eaf5930a3c6..000000000000
--- a/contrib/experimental_dict_builders/benchmarkDictBuilder/test.sh
+++ /dev/null
@@ -1,2 +0,0 @@
-echo "Benchmark with in=../../lib/common"
-./benchmark in=../../../lib/common
diff --git a/contrib/experimental_dict_builders/fastCover/Makefile b/contrib/experimental_dict_builders/fastCover/Makefile
deleted file mode 100644
index 3ba24790ce01..000000000000
--- a/contrib/experimental_dict_builders/fastCover/Makefile
+++ /dev/null
@@ -1,54 +0,0 @@
-ARG :=
-
-CC ?= gcc
-CFLAGS ?= -O3 -g
-INCLUDES := -I ../../../programs -I ../randomDictBuilder -I ../../../lib/common -I ../../../lib -I ../../../lib/dictBuilder
-
-IO_FILE := ../randomDictBuilder/io.c
-
-TEST_INPUT := ../../../lib
-TEST_OUTPUT := fastCoverDict
-
-all: main run clean
-
-.PHONY: test
-test: main testrun testshell clean
-
-.PHONY: run
-run:
-	echo "Building a fastCover dictionary with given arguments"
-	./main $(ARG)
-
-main: main.o io.o fastCover.o libzstd.a
-	$(CC) $(CFLAGS) main.o io.o fastCover.o libzstd.a -o main
-
-main.o: main.c
-	$(CC) $(CFLAGS) $(INCLUDES) -c main.c
-
-fastCover.o: fastCover.c
-	$(CC) $(CFLAGS) $(INCLUDES) -c fastCover.c
-
-io.o: $(IO_FILE)
-	$(CC) $(CFLAGS) $(INCLUDES) -c $(IO_FILE)
-
-libzstd.a:
-	$(MAKE) MOREFLAGS=-g -C ../../../lib libzstd.a
-	mv ../../../lib/libzstd.a .
-
-.PHONY: testrun
-testrun: main
-	echo "Run with $(TEST_INPUT) and $(TEST_OUTPUT) "
-	./main in=$(TEST_INPUT) out=$(TEST_OUTPUT)
-	zstd -be3 -D $(TEST_OUTPUT) -r $(TEST_INPUT) -q
-	rm -f $(TEST_OUTPUT)
-
-.PHONY: testshell
-testshell: test.sh
-	sh test.sh
-	echo "Finish running test.sh"
-
-.PHONY: clean
-clean:
-	rm -f *.o main libzstd.a
-	$(MAKE) -C ../../../lib clean
-	echo "Cleaning is completed"
diff --git a/contrib/experimental_dict_builders/fastCover/README.md b/contrib/experimental_dict_builders/fastCover/README.md
deleted file mode 100644
index ad377743f2a7..000000000000
--- a/contrib/experimental_dict_builders/fastCover/README.md
+++ /dev/null
@@ -1,24 +0,0 @@
-FastCover Dictionary Builder
-
-### Permitted Arguments:
-Input File/Directory (in=fileName): required; file/directory used to build dictionary; if directory, will operate recursively for files inside directory; can include multiple files/directories, each following "in="
-Output Dictionary (out=dictName): if not provided, default to fastCoverDict
-Dictionary ID (dictID=#): nonnegative number; if not provided, default to 0
-Maximum Dictionary Size (maxdict=#): positive number; in bytes, if not provided, default to 110KB
-Size of Selected Segment (k=#): positive number; in bytes; if not provided, default to 200
-Size of Dmer (d=#): either 6 or 8; if not provided, default to 8
-Number of steps (steps=#): positive number, if not provided, default to 32
-Percentage of samples used for training(split=#): positive number; if not provided, default to 100
-
-
-###Running Test:
-make test
-
-
-###Usage:
-To build a FASTCOVER dictionary with the provided arguments: make ARG= followed by arguments
-If k or d is not provided, the optimize version of FASTCOVER is run.
-
-### Examples:
-make ARG="in=../../../lib/dictBuilder out=dict100 dictID=520"
-make ARG="in=../../../lib/dictBuilder in=../../../lib/compress"
diff --git a/contrib/experimental_dict_builders/fastCover/fastCover.c b/contrib/experimental_dict_builders/fastCover/fastCover.c
deleted file mode 100644
index 0a338bde2b20..000000000000
--- a/contrib/experimental_dict_builders/fastCover/fastCover.c
+++ /dev/null
@@ -1,809 +0,0 @@
-/*-*************************************
-*  Dependencies
-***************************************/
-#include <stdio.h>  /* fprintf */
-#include <stdlib.h> /* malloc, free, qsort */
-#include <string.h> /* memset */
-#include <time.h>   /* clock */
-#include "mem.h" /* read */
-#include "pool.h"
-#include "threading.h"
-#include "fastCover.h"
-#include "zstd_internal.h" /* includes zstd.h */
-#include "zdict.h"
-
-
-/*-*************************************
-*  Constants
-***************************************/
-#define FASTCOVER_MAX_SAMPLES_SIZE (sizeof(size_t) == 8 ? ((U32)-1) : ((U32)1 GB))
-#define FASTCOVER_MAX_F 32
-#define DEFAULT_SPLITPOINT 1.0
-
-/*-*************************************
-*  Console display
-***************************************/
-static int g_displayLevel = 2;
-#define DISPLAY(...)                                                           \
-  {                                                                            \
-    fprintf(stderr, __VA_ARGS__);                                              \
-    fflush(stderr);                                                            \
-  }
-#define LOCALDISPLAYLEVEL(displayLevel, l, ...)                                \
-  if (displayLevel >= l) {                                                     \
-    DISPLAY(__VA_ARGS__);                                                      \
-  } /* 0 : no display;   1: errors;   2: default;  3: details;  4: debug */
-#define DISPLAYLEVEL(l, ...) LOCALDISPLAYLEVEL(g_displayLevel, l, __VA_ARGS__)
-
-#define LOCALDISPLAYUPDATE(displayLevel, l, ...)                               \
-  if (displayLevel >= l) {                                                     \
-    if ((clock() - g_time > refreshRate) || (displayLevel >= 4)) {             \
-      g_time = clock();                                                        \
-      DISPLAY(__VA_ARGS__);                                                    \
-    }                                                                          \
-  }
-#define DISPLAYUPDATE(l, ...) LOCALDISPLAYUPDATE(g_displayLevel, l, __VA_ARGS__)
-static const clock_t refreshRate = CLOCKS_PER_SEC * 15 / 100;
-static clock_t g_time = 0;
-
-
-/*-*************************************
-* Hash Functions
-***************************************/
-static const U64 prime6bytes = 227718039650203ULL;
-static size_t ZSTD_hash6(U64 u, U32 h) { return (size_t)(((u  << (64-48)) * prime6bytes) >> (64-h)) ; }
-static size_t ZSTD_hash6Ptr(const void* p, U32 h) { return ZSTD_hash6(MEM_readLE64(p), h); }
-
-static const U64 prime8bytes = 0xCF1BBCDCB7A56463ULL;
-static size_t ZSTD_hash8(U64 u, U32 h) { return (size_t)(((u) * prime8bytes) >> (64-h)) ; }
-static size_t ZSTD_hash8Ptr(const void* p, U32 h) { return ZSTD_hash8(MEM_readLE64(p), h); }
-
-
-/**
- * Hash the d-byte value pointed to by p and mod 2^f
- */
-static size_t FASTCOVER_hashPtrToIndex(const void* p, U32 h, unsigned d) {
-  if (d == 6) {
-    return ZSTD_hash6Ptr(p, h) & ((1 << h) - 1);
-  }
-  return ZSTD_hash8Ptr(p, h) & ((1 << h) - 1);
-}
-
-
-/*-*************************************
-* Context
-***************************************/
-typedef struct {
-  const BYTE *samples;
-  size_t *offsets;
-  const size_t *samplesSizes;
-  size_t nbSamples;
-  size_t nbTrainSamples;
-  size_t nbTestSamples;
-  size_t nbDmers;
-  U32 *freqs;
-  U16 *segmentFreqs;
-  unsigned d;
-} FASTCOVER_ctx_t;
-
-
-/*-*************************************
-*  Helper functions
-***************************************/
-/**
- * Returns the sum of the sample sizes.
- */
-static size_t FASTCOVER_sum(const size_t *samplesSizes, unsigned nbSamples) {
-  size_t sum = 0;
-  unsigned i;
-  for (i = 0; i < nbSamples; ++i) {
-    sum += samplesSizes[i];
-  }
-  return sum;
-}
-
-
-/*-*************************************
-*  fast functions
-***************************************/
-/**
- * A segment is a range in the source as well as the score of the segment.
- */
-typedef struct {
-  U32 begin;
-  U32 end;
-  U32 score;
-} FASTCOVER_segment_t;
-
-
-/**
- * Selects the best segment in an epoch.
- * Segments of are scored according to the function:
- *
- * Let F(d) be the frequency of all dmers with hash value d.
- * Let S_i be hash value of the dmer at position i of segment S which has length k.
- *
- *     Score(S) = F(S_1) + F(S_2) + ... + F(S_{k-d+1})
- *
- * Once the dmer with hash value d is in the dictionary we set F(d) = F(d)/2.
- */
-static FASTCOVER_segment_t FASTCOVER_selectSegment(const FASTCOVER_ctx_t *ctx,
-                                                  U32 *freqs, U32 begin,U32 end,
-                                                  ZDICT_fastCover_params_t parameters) {
-  /* Constants */
-  const U32 k = parameters.k;
-  const U32 d = parameters.d;
-  const U32 dmersInK = k - d + 1;
-  /* Try each segment (activeSegment) and save the best (bestSegment) */
-  FASTCOVER_segment_t bestSegment = {0, 0, 0};
-  FASTCOVER_segment_t activeSegment;
-  /* Reset the activeDmers in the segment */
-  /* The activeSegment starts at the beginning of the epoch. */
-  activeSegment.begin = begin;
-  activeSegment.end = begin;
-  activeSegment.score = 0;
-  {
-    /* Slide the activeSegment through the whole epoch.
-     * Save the best segment in bestSegment.
-     */
-    while (activeSegment.end < end) {
-      /* Get hash value of current dmer */
-      const size_t index = FASTCOVER_hashPtrToIndex(ctx->samples + activeSegment.end, parameters.f, ctx->d);
-      /* Add frequency of this index to score if this is the first occurrence of index in active segment */
-      if (ctx->segmentFreqs[index] == 0) {
-        activeSegment.score += freqs[index];
-      }
-      ctx->segmentFreqs[index] += 1;
-      /* Increment end of segment */
-      activeSegment.end += 1;
-      /* If the window is now too large, drop the first position */
-      if (activeSegment.end - activeSegment.begin == dmersInK + 1) {
-        /* Get hash value of the dmer to be eliminated from active segment */
-        const size_t delIndex = FASTCOVER_hashPtrToIndex(ctx->samples + activeSegment.begin, parameters.f, ctx->d);
-        ctx->segmentFreqs[delIndex] -= 1;
-        /* Subtract frequency of this index from score if this is the last occurrence of this index in active segment */
-        if (ctx->segmentFreqs[delIndex] == 0) {
-          activeSegment.score -= freqs[delIndex];
-        }
-        /* Increment start of segment */
-        activeSegment.begin += 1;
-      }
-      /* If this segment is the best so far save it */
-      if (activeSegment.score > bestSegment.score) {
-        bestSegment = activeSegment;
-      }
-    }
-    /* Zero out rest of segmentFreqs array */
-    while (activeSegment.begin < end) {
-      const size_t delIndex = FASTCOVER_hashPtrToIndex(ctx->samples + activeSegment.begin, parameters.f, ctx->d);
-      ctx->segmentFreqs[delIndex] -= 1;
-      activeSegment.begin += 1;
-    }
-  }
-  {
-    /* Trim off the zero frequency head and tail from the segment. */
-    U32 newBegin = bestSegment.end;
-    U32 newEnd = bestSegment.begin;
-    U32 pos;
-    for (pos = bestSegment.begin; pos != bestSegment.end; ++pos) {
-      const size_t index = FASTCOVER_hashPtrToIndex(ctx->samples + pos, parameters.f, ctx->d);
-      U32 freq = freqs[index];
-      if (freq != 0) {
-        newBegin = MIN(newBegin, pos);
-        newEnd = pos + 1;
-      }
-    }
-    bestSegment.begin = newBegin;
-    bestSegment.end = newEnd;
-  }
-  {
-    /*  Zero the frequency of hash value of each dmer covered by the chosen segment. */
-    U32 pos;
-    for (pos = bestSegment.begin; pos != bestSegment.end; ++pos) {
-      const size_t i = FASTCOVER_hashPtrToIndex(ctx->samples + pos, parameters.f, ctx->d);
-      freqs[i] = 0;
-    }
-  }
-  return bestSegment;
-}
-
-/**
- * Check the validity of the parameters.
- * Returns non-zero if the parameters are valid and 0 otherwise.
- */
-static int FASTCOVER_checkParameters(ZDICT_fastCover_params_t parameters,
-                                 size_t maxDictSize) {
-  /* k, d, and f are required parameters */
-  if (parameters.d == 0 || parameters.k == 0 || parameters.f == 0) {
-    return 0;
-  }
-  /* d has to be 6 or 8 */
-  if (parameters.d != 6 && parameters.d != 8) {
-    return 0;
-  }
-  /* 0 < f <= FASTCOVER_MAX_F */
-  if (parameters.f > FASTCOVER_MAX_F) {
-    return 0;
-  }
-  /* k <= maxDictSize */
-  if (parameters.k > maxDictSize) {
-    return 0;
-  }
-  /* d <= k */
-  if (parameters.d > parameters.k) {
-    return 0;
-  }
-  /* 0 < splitPoint <= 1 */
-  if (parameters.splitPoint <= 0 || parameters.splitPoint > 1) {
-    return 0;
-  }
-  return 1;
-}
-
-
-/**
- * Clean up a context initialized with `FASTCOVER_ctx_init()`.
- */
-static void FASTCOVER_ctx_destroy(FASTCOVER_ctx_t *ctx) {
-  if (!ctx) {
-    return;
-  }
-  if (ctx->segmentFreqs) {
-    free(ctx->segmentFreqs);
-    ctx->segmentFreqs = NULL;
-  }
-  if (ctx->freqs) {
-    free(ctx->freqs);
-    ctx->freqs = NULL;
-  }
-  if (ctx->offsets) {
-    free(ctx->offsets);
-    ctx->offsets = NULL;
-  }
-}
-
-/**
- * Calculate for frequency of hash value of each dmer in ctx->samples
- */
-static void FASTCOVER_computeFrequency(U32 *freqs, unsigned f, FASTCOVER_ctx_t *ctx){
-  size_t start; /* start of current dmer */
-  for (unsigned i = 0; i < ctx->nbTrainSamples; i++) {
-    size_t currSampleStart = ctx->offsets[i];
-    size_t currSampleEnd = ctx->offsets[i+1];
-    start = currSampleStart;
-    while (start + ctx->d <= currSampleEnd) {
-      const size_t dmerIndex = FASTCOVER_hashPtrToIndex(ctx->samples + start, f, ctx->d);
-      freqs[dmerIndex]++;
-      start++;
-    }
-  }
-}
-
-/**
- * Prepare a context for dictionary building.
- * The context is only dependent on the parameter `d` and can used multiple
- * times.
- * Returns 1 on success or zero on error.
- * The context must be destroyed with `FASTCOVER_ctx_destroy()`.
- */
-static int FASTCOVER_ctx_init(FASTCOVER_ctx_t *ctx, const void *samplesBuffer,
-                          const size_t *samplesSizes, unsigned nbSamples,
-                          unsigned d, double splitPoint, unsigned f) {
-  const BYTE *const samples = (const BYTE *)samplesBuffer;
-  const size_t totalSamplesSize = FASTCOVER_sum(samplesSizes, nbSamples);
-  /* Split samples into testing and training sets */
-  const unsigned nbTrainSamples = splitPoint < 1.0 ? (unsigned)((double)nbSamples * splitPoint) : nbSamples;
-  const unsigned nbTestSamples = splitPoint < 1.0 ? nbSamples - nbTrainSamples : nbSamples;
-  const size_t trainingSamplesSize = splitPoint < 1.0 ? FASTCOVER_sum(samplesSizes, nbTrainSamples) : totalSamplesSize;
-  const size_t testSamplesSize = splitPoint < 1.0 ? FASTCOVER_sum(samplesSizes + nbTrainSamples, nbTestSamples) : totalSamplesSize;
-  /* Checks */
-  if (totalSamplesSize < MAX(d, sizeof(U64)) ||
-      totalSamplesSize >= (size_t)FASTCOVER_MAX_SAMPLES_SIZE) {
-    DISPLAYLEVEL(1, "Total samples size is too large (%u MB), maximum size is %u MB\n",
-                 (U32)(totalSamplesSize >> 20), (FASTCOVER_MAX_SAMPLES_SIZE >> 20));
-    return 0;
-  }
-  /* Check if there are at least 5 training samples */
-  if (nbTrainSamples < 5) {
-    DISPLAYLEVEL(1, "Total number of training samples is %u and is invalid.", nbTrainSamples);
-    return 0;
-  }
-  /* Check if there's testing sample */
-  if (nbTestSamples < 1) {
-    DISPLAYLEVEL(1, "Total number of testing samples is %u and is invalid.", nbTestSamples);
-    return 0;
-  }
-  /* Zero the context */
-  memset(ctx, 0, sizeof(*ctx));
-  DISPLAYLEVEL(2, "Training on %u samples of total size %u\n", nbTrainSamples,
-               (U32)trainingSamplesSize);
-  DISPLAYLEVEL(2, "Testing on %u samples of total size %u\n", nbTestSamples,
-               (U32)testSamplesSize);
-
-  ctx->samples = samples;
-  ctx->samplesSizes = samplesSizes;
-  ctx->nbSamples = nbSamples;
-  ctx->nbTrainSamples = nbTrainSamples;
-  ctx->nbTestSamples = nbTestSamples;
-  ctx->nbDmers = trainingSamplesSize - d + 1;
-  ctx->d = d;
-
-  /* The offsets of each file */
-  ctx->offsets = (size_t *)malloc((nbSamples + 1) * sizeof(size_t));
-  if (!ctx->offsets) {
-    DISPLAYLEVEL(1, "Failed to allocate scratch buffers\n");
-    FASTCOVER_ctx_destroy(ctx);
-    return 0;
-  }
-
-  /* Fill offsets from the samplesSizes */
-  {
-    U32 i;
-    ctx->offsets[0] = 0;
-    for (i = 1; i <= nbSamples; ++i) {
-      ctx->offsets[i] = ctx->offsets[i - 1] + samplesSizes[i - 1];
-    }
-  }
-
-  /* Initialize frequency array of size 2^f */
-  ctx->freqs = (U32 *)calloc((1 << f), sizeof(U32));
-  ctx->segmentFreqs = (U16 *)calloc((1 << f), sizeof(U16));
-  DISPLAYLEVEL(2, "Computing frequencies\n");
-  FASTCOVER_computeFrequency(ctx->freqs, f, ctx);
-
-  return 1;
-}
-
-
-/**
- * Given the prepared context build the dictionary.
- */
-static size_t FASTCOVER_buildDictionary(const FASTCOVER_ctx_t *ctx, U32 *freqs,
-                                    void *dictBuffer,
-                                    size_t dictBufferCapacity,
-                                    ZDICT_fastCover_params_t parameters){
-  BYTE *const dict = (BYTE *)dictBuffer;
-  size_t tail = dictBufferCapacity;
-  /* Divide the data up into epochs of equal size.
-   * We will select at least one segment from each epoch.
-   */
-  const U32 epochs = MAX(1, (U32)(dictBufferCapacity / parameters.k));
-  const U32 epochSize = (U32)(ctx->nbDmers / epochs);
-  size_t epoch;
-  DISPLAYLEVEL(2, "Breaking content into %u epochs of size %u\n", epochs,
-               epochSize);
-  /* Loop through the epochs until there are no more segments or the dictionary
-   * is full.
-   */
-  for (epoch = 0; tail > 0; epoch = (epoch + 1) % epochs) {
-    const U32 epochBegin = (U32)(epoch * epochSize);
-    const U32 epochEnd = epochBegin + epochSize;
-    size_t segmentSize;
-    /* Select a segment */
-    FASTCOVER_segment_t segment = FASTCOVER_selectSegment(
-        ctx, freqs, epochBegin, epochEnd, parameters);
-
-    /* If the segment covers no dmers, then we are out of content */
-    if (segment.score == 0) {
-      break;
-    }
-
-    /* Trim the segment if necessary and if it is too small then we are done */
-    segmentSize = MIN(segment.end - segment.begin + parameters.d - 1, tail);
-    if (segmentSize < parameters.d) {
-      break;
-    }
-
-    /* We fill the dictionary from the back to allow the best segments to be
-     * referenced with the smallest offsets.
-     */
-    tail -= segmentSize;
-    memcpy(dict + tail, ctx->samples + segment.begin, segmentSize);
-    DISPLAYUPDATE(
-        2, "\r%u%%       ",
-        (U32)(((dictBufferCapacity - tail) * 100) / dictBufferCapacity));
-  }
-  DISPLAYLEVEL(2, "\r%79s\r", "");
-  return tail;
-}
-
-
-/**
- * FASTCOVER_best_t is used for two purposes:
- * 1. Synchronizing threads.
- * 2. Saving the best parameters and dictionary.
- *
- * All of the methods except FASTCOVER_best_init() are thread safe if zstd is
- * compiled with multithreaded support.
- */
-typedef struct fast_best_s {
-  ZSTD_pthread_mutex_t mutex;
-  ZSTD_pthread_cond_t cond;
-  size_t liveJobs;
-  void *dict;
-  size_t dictSize;
-  ZDICT_fastCover_params_t parameters;
-  size_t compressedSize;
-} FASTCOVER_best_t;
-
-/**
- * Initialize the `FASTCOVER_best_t`.
- */
-static void FASTCOVER_best_init(FASTCOVER_best_t *best) {
-  if (best==NULL) return; /* compatible with init on NULL */
-  (void)ZSTD_pthread_mutex_init(&best->mutex, NULL);
-  (void)ZSTD_pthread_cond_init(&best->cond, NULL);
-  best->liveJobs = 0;
-  best->dict = NULL;
-  best->dictSize = 0;
-  best->compressedSize = (size_t)-1;
-  memset(&best->parameters, 0, sizeof(best->parameters));
-}
-
-/**
- * Wait until liveJobs == 0.
- */
-static void FASTCOVER_best_wait(FASTCOVER_best_t *best) {
-  if (!best) {
-    return;
-  }
-  ZSTD_pthread_mutex_lock(&best->mutex);
-  while (best->liveJobs != 0) {
-    ZSTD_pthread_cond_wait(&best->cond, &best->mutex);
-  }
-  ZSTD_pthread_mutex_unlock(&best->mutex);
-}
-
-/**
- * Call FASTCOVER_best_wait() and then destroy the FASTCOVER_best_t.
- */
-static void FASTCOVER_best_destroy(FASTCOVER_best_t *best) {
-  if (!best) {
-    return;
-  }
-  FASTCOVER_best_wait(best);
-  if (best->dict) {
-    free(best->dict);
-  }
-  ZSTD_pthread_mutex_destroy(&best->mutex);
-  ZSTD_pthread_cond_destroy(&best->cond);
-}
-
-/**
- * Called when a thread is about to be launched.
- * Increments liveJobs.
- */
-static void FASTCOVER_best_start(FASTCOVER_best_t *best) {
-  if (!best) {
-    return;
-  }
-  ZSTD_pthread_mutex_lock(&best->mutex);
-  ++best->liveJobs;
-  ZSTD_pthread_mutex_unlock(&best->mutex);
-}
-
-/**
- * Called when a thread finishes executing, both on error or success.
- * Decrements liveJobs and signals any waiting threads if liveJobs == 0.
- * If this dictionary is the best so far save it and its parameters.
- */
-static void FASTCOVER_best_finish(FASTCOVER_best_t *best, size_t compressedSize,
-                              ZDICT_fastCover_params_t parameters, void *dict,
-                              size_t dictSize) {
-  if (!best) {
-    return;
-  }
-  {
-    size_t liveJobs;
-    ZSTD_pthread_mutex_lock(&best->mutex);
-    --best->liveJobs;
-    liveJobs = best->liveJobs;
-    /* If the new dictionary is better */
-    if (compressedSize < best->compressedSize) {
-      /* Allocate space if necessary */
-      if (!best->dict || best->dictSize < dictSize) {
-        if (best->dict) {
-          free(best->dict);
-        }
-        best->dict = malloc(dictSize);
-        if (!best->dict) {
-          best->compressedSize = ERROR(GENERIC);
-          best->dictSize = 0;
-          return;
-        }
-      }
-      /* Save the dictionary, parameters, and size */
-      memcpy(best->dict, dict, dictSize);
-      best->dictSize = dictSize;
-      best->parameters = parameters;
-      best->compressedSize = compressedSize;
-    }
-    ZSTD_pthread_mutex_unlock(&best->mutex);
-    if (liveJobs == 0) {
-      ZSTD_pthread_cond_broadcast(&best->cond);
-    }
-  }
-}
-
-/**
- * Parameters for FASTCOVER_tryParameters().
- */
-typedef struct FASTCOVER_tryParameters_data_s {
-  const FASTCOVER_ctx_t *ctx;
-  FASTCOVER_best_t *best;
-  size_t dictBufferCapacity;
-  ZDICT_fastCover_params_t parameters;
-} FASTCOVER_tryParameters_data_t;
-
-/**
- * Tries a set of parameters and updates the FASTCOVER_best_t with the results.
- * This function is thread safe if zstd is compiled with multithreaded support.
- * It takes its parameters as an *OWNING* opaque pointer to support threading.
- */
-static void FASTCOVER_tryParameters(void *opaque) {
-  /* Save parameters as local variables */
-  FASTCOVER_tryParameters_data_t *const data = (FASTCOVER_tryParameters_data_t *)opaque;
-  const FASTCOVER_ctx_t *const ctx = data->ctx;
-  const ZDICT_fastCover_params_t parameters = data->parameters;
-  size_t dictBufferCapacity = data->dictBufferCapacity;
-  size_t totalCompressedSize = ERROR(GENERIC);
-  /* Allocate space for hash table, dict, and freqs */
-  BYTE *const dict = (BYTE * const)malloc(dictBufferCapacity);
-  U32 *freqs = (U32*) malloc((1 << parameters.f) * sizeof(U32));
-  if (!dict || !freqs) {
-    DISPLAYLEVEL(1, "Failed to allocate buffers: out of memory\n");
-    goto _cleanup;
-  }
-  /* Copy the frequencies because we need to modify them */
-  memcpy(freqs, ctx->freqs, (1 << parameters.f) * sizeof(U32));
-  /* Build the dictionary */
-  {
-    const size_t tail = FASTCOVER_buildDictionary(ctx, freqs, dict,
-                                              dictBufferCapacity, parameters);
-
-    dictBufferCapacity = ZDICT_finalizeDictionary(
-        dict, dictBufferCapacity, dict + tail, dictBufferCapacity - tail,
-        ctx->samples, ctx->samplesSizes, (unsigned)ctx->nbTrainSamples,
-        parameters.zParams);
-    if (ZDICT_isError(dictBufferCapacity)) {
-      DISPLAYLEVEL(1, "Failed to finalize dictionary\n");
-      goto _cleanup;
-    }
-  }
-  /* Check total compressed size */
-  {
-    /* Pointers */
-    ZSTD_CCtx *cctx;
-    ZSTD_CDict *cdict;
-    void *dst;
-    /* Local variables */
-    size_t dstCapacity;
-    size_t i;
-    /* Allocate dst with enough space to compress the maximum sized sample */
-    {
-      size_t maxSampleSize = 0;
-      i = parameters.splitPoint < 1.0 ? ctx->nbTrainSamples : 0;
-      for (; i < ctx->nbSamples; ++i) {
-        maxSampleSize = MAX(ctx->samplesSizes[i], maxSampleSize);
-      }
-      dstCapacity = ZSTD_compressBound(maxSampleSize);
-      dst = malloc(dstCapacity);
-    }
-    /* Create the cctx and cdict */
-    cctx = ZSTD_createCCtx();
-    cdict = ZSTD_createCDict(dict, dictBufferCapacity,
-                             parameters.zParams.compressionLevel);
-    if (!dst || !cctx || !cdict) {
-      goto _compressCleanup;
-    }
-    /* Compress each sample and sum their sizes (or error) */
-    totalCompressedSize = dictBufferCapacity;
-    i = parameters.splitPoint < 1.0 ? ctx->nbTrainSamples : 0;
-    for (; i < ctx->nbSamples; ++i) {
-      const size_t size = ZSTD_compress_usingCDict(
-          cctx, dst, dstCapacity, ctx->samples + ctx->offsets[i],
-          ctx->samplesSizes[i], cdict);
-      if (ZSTD_isError(size)) {
-        totalCompressedSize = ERROR(GENERIC);
-        goto _compressCleanup;
-      }
-      totalCompressedSize += size;
-    }
-  _compressCleanup:
-    ZSTD_freeCCtx(cctx);
-    ZSTD_freeCDict(cdict);
-    if (dst) {
-      free(dst);
-    }
-  }
-
-_cleanup:
-  FASTCOVER_best_finish(data->best, totalCompressedSize, parameters, dict,
-                    dictBufferCapacity);
-  free(data);
-  if (dict) {
-    free(dict);
-  }
-  if (freqs) {
-    free(freqs);
-  }
-}
-
-ZDICTLIB_API size_t ZDICT_trainFromBuffer_fastCover(
-    void *dictBuffer, size_t dictBufferCapacity, const void *samplesBuffer,
-    const size_t *samplesSizes, unsigned nbSamples, ZDICT_fastCover_params_t parameters) {
-    BYTE* const dict = (BYTE*)dictBuffer;
-    FASTCOVER_ctx_t ctx;
-    parameters.splitPoint = 1.0;
-    /* Initialize global data */
-    g_displayLevel = parameters.zParams.notificationLevel;
-    /* Checks */
-    if (!FASTCOVER_checkParameters(parameters, dictBufferCapacity)) {
-      DISPLAYLEVEL(1, "FASTCOVER parameters incorrect\n");
-      return ERROR(GENERIC);
-    }
-    if (nbSamples == 0) {
-      DISPLAYLEVEL(1, "FASTCOVER must have at least one input file\n");
-      return ERROR(GENERIC);
-    }
-    if (dictBufferCapacity < ZDICT_DICTSIZE_MIN) {
-      DISPLAYLEVEL(1, "dictBufferCapacity must be at least %u\n",
-                   ZDICT_DICTSIZE_MIN);
-      return ERROR(dstSize_tooSmall);
-    }
-    /* Initialize context */
-    if (!FASTCOVER_ctx_init(&ctx, samplesBuffer, samplesSizes, nbSamples,
-                            parameters.d, parameters.splitPoint, parameters.f)) {
-      DISPLAYLEVEL(1, "Failed to initialize context\n");
-      return ERROR(GENERIC);
-    }
-    /* Build the dictionary */
-    DISPLAYLEVEL(2, "Building dictionary\n");
-    {
-      const size_t tail = FASTCOVER_buildDictionary(&ctx, ctx.freqs, dictBuffer,
-                                                dictBufferCapacity, parameters);
-
-      const size_t dictionarySize = ZDICT_finalizeDictionary(
-          dict, dictBufferCapacity, dict + tail, dictBufferCapacity - tail,
-          samplesBuffer, samplesSizes, (unsigned)ctx.nbTrainSamples,
-          parameters.zParams);
-      if (!ZSTD_isError(dictionarySize)) {
-          DISPLAYLEVEL(2, "Constructed dictionary of size %u\n",
-                      (U32)dictionarySize);
-      }
-      FASTCOVER_ctx_destroy(&ctx);
-      return dictionarySize;
-    }
-}
-
-
-
-ZDICTLIB_API size_t ZDICT_optimizeTrainFromBuffer_fastCover(
-    void *dictBuffer, size_t dictBufferCapacity, const void *samplesBuffer,
-    const size_t *samplesSizes, unsigned nbSamples,
-    ZDICT_fastCover_params_t *parameters) {
-    /* constants */
-    const unsigned nbThreads = parameters->nbThreads;
-    const double splitPoint =
-        parameters->splitPoint <= 0.0 ? DEFAULT_SPLITPOINT : parameters->splitPoint;
-    const unsigned kMinD = parameters->d == 0 ? 6 : parameters->d;
-    const unsigned kMaxD = parameters->d == 0 ? 8 : parameters->d;
-    const unsigned kMinK = parameters->k == 0 ? 50 : parameters->k;
-    const unsigned kMaxK = parameters->k == 0 ? 2000 : parameters->k;
-    const unsigned kSteps = parameters->steps == 0 ? 40 : parameters->steps;
-    const unsigned kStepSize = MAX((kMaxK - kMinK) / kSteps, 1);
-    const unsigned kIterations =
-        (1 + (kMaxD - kMinD) / 2) * (1 + (kMaxK - kMinK) / kStepSize);
-    const unsigned f = parameters->f == 0 ? 23 : parameters->f;
-
-    /* Local variables */
-    const int displayLevel = parameters->zParams.notificationLevel;
-    unsigned iteration = 1;
-    unsigned d;
-    unsigned k;
-    FASTCOVER_best_t best;
-    POOL_ctx *pool = NULL;
-
-    /* Checks */
-    if (splitPoint <= 0 || splitPoint > 1) {
-      LOCALDISPLAYLEVEL(displayLevel, 1, "Incorrect splitPoint\n");
-      return ERROR(GENERIC);
-    }
-    if (kMinK < kMaxD || kMaxK < kMinK) {
-      LOCALDISPLAYLEVEL(displayLevel, 1, "Incorrect k\n");
-      return ERROR(GENERIC);
-    }
-    if (nbSamples == 0) {
-      DISPLAYLEVEL(1, "FASTCOVER must have at least one input file\n");
-      return ERROR(GENERIC);
-    }
-    if (dictBufferCapacity < ZDICT_DICTSIZE_MIN) {
-      DISPLAYLEVEL(1, "dictBufferCapacity must be at least %u\n",
-                   ZDICT_DICTSIZE_MIN);
-      return ERROR(dstSize_tooSmall);
-    }
-    if (nbThreads > 1) {
-      pool = POOL_create(nbThreads, 1);
-      if (!pool) {
-        return ERROR(memory_allocation);
-      }
-    }
-    /* Initialization */
-    FASTCOVER_best_init(&best);
-    /* Turn down global display level to clean up display at level 2 and below */
-    g_displayLevel = displayLevel == 0 ? 0 : displayLevel - 1;
-    /* Loop through d first because each new value needs a new context */
-    LOCALDISPLAYLEVEL(displayLevel, 2, "Trying %u different sets of parameters\n",
-                      kIterations);
-    for (d = kMinD; d <= kMaxD; d += 2) {
-      /* Initialize the context for this value of d */
-      FASTCOVER_ctx_t ctx;
-      LOCALDISPLAYLEVEL(displayLevel, 3, "d=%u\n", d);
-      if (!FASTCOVER_ctx_init(&ctx, samplesBuffer, samplesSizes, nbSamples, d, splitPoint, f)) {
-        LOCALDISPLAYLEVEL(displayLevel, 1, "Failed to initialize context\n");
-        FASTCOVER_best_destroy(&best);
-        POOL_free(pool);
-        return ERROR(GENERIC);
-      }
-      /* Loop through k reusing the same context */
-      for (k = kMinK; k <= kMaxK; k += kStepSize) {
-        /* Prepare the arguments */
-        FASTCOVER_tryParameters_data_t *data = (FASTCOVER_tryParameters_data_t *)malloc(
-            sizeof(FASTCOVER_tryParameters_data_t));
-        LOCALDISPLAYLEVEL(displayLevel, 3, "k=%u\n", k);
-        if (!data) {
-          LOCALDISPLAYLEVEL(displayLevel, 1, "Failed to allocate parameters\n");
-          FASTCOVER_best_destroy(&best);
-          FASTCOVER_ctx_destroy(&ctx);
-          POOL_free(pool);
-          return ERROR(GENERIC);
-        }
-        data->ctx = &ctx;
-        data->best = &best;
-        data->dictBufferCapacity = dictBufferCapacity;
-        data->parameters = *parameters;
-        data->parameters.k = k;
-        data->parameters.d = d;
-        data->parameters.f = f;
-        data->parameters.splitPoint = splitPoint;
-        data->parameters.steps = kSteps;
-        data->parameters.zParams.notificationLevel = g_displayLevel;
-        /* Check the parameters */
-        if (!FASTCOVER_checkParameters(data->parameters, dictBufferCapacity)) {
-          DISPLAYLEVEL(1, "fastCover parameters incorrect\n");
-          free(data);
-          continue;
-        }
-        /* Call the function and pass ownership of data to it */
-        FASTCOVER_best_start(&best);
-        if (pool) {
-          POOL_add(pool, &FASTCOVER_tryParameters, data);
-        } else {
-          FASTCOVER_tryParameters(data);
-        }
-        /* Print status */
-        LOCALDISPLAYUPDATE(displayLevel, 2, "\r%u%%       ",
-                           (U32)((iteration * 100) / kIterations));
-        ++iteration;
-      }
-      FASTCOVER_best_wait(&best);
-      FASTCOVER_ctx_destroy(&ctx);
-    }
-    LOCALDISPLAYLEVEL(displayLevel, 2, "\r%79s\r", "");
-    /* Fill the output buffer and parameters with output of the best parameters */
-    {
-      const size_t dictSize = best.dictSize;
-      if (ZSTD_isError(best.compressedSize)) {
-        const size_t compressedSize = best.compressedSize;
-        FASTCOVER_best_destroy(&best);
-        POOL_free(pool);
-        return compressedSize;
-      }
-      *parameters = best.parameters;
-      memcpy(dictBuffer, best.dict, dictSize);
-      FASTCOVER_best_destroy(&best);
-      POOL_free(pool);
-      return dictSize;
-    }
-
-}
diff --git a/contrib/experimental_dict_builders/fastCover/fastCover.h b/contrib/experimental_dict_builders/fastCover/fastCover.h
deleted file mode 100644
index 958e9f423930..000000000000
--- a/contrib/experimental_dict_builders/fastCover/fastCover.h
+++ /dev/null
@@ -1,57 +0,0 @@
-#include <stdio.h>  /* fprintf */
-#include <stdlib.h> /* malloc, free, qsort */
-#include <string.h> /* memset */
-#include <time.h>   /* clock */
-#include "mem.h" /* read */
-#include "pool.h"
-#include "threading.h"
-#include "zstd_internal.h" /* includes zstd.h */
-#ifndef ZDICT_STATIC_LINKING_ONLY
-#define ZDICT_STATIC_LINKING_ONLY
-#endif
-#include "zdict.h"
-
-
-typedef struct {
-    unsigned k;                  /* Segment size : constraint: 0 < k : Reasonable range [16, 2048+] */
-    unsigned d;                  /* dmer size : constraint: 0 < d <= k : Reasonable range [6, 16] */
-    unsigned f;                  /* log of size of frequency array */
-    unsigned steps;              /* Number of steps : Only used for optimization : 0 means default (32) : Higher means more parameters checked */
-    unsigned nbThreads;          /* Number of threads : constraint: 0 < nbThreads : 1 means single-threaded : Only used for optimization : Ignored if ZSTD_MULTITHREAD is not defined */
-    double splitPoint;           /* Percentage of samples used for training: the first nbSamples * splitPoint samples will be used to training, the last nbSamples * (1 - splitPoint) samples will be used for testing, 0 means default (1.0), 1.0 when all samples are used for both training and testing */
-    ZDICT_params_t zParams;
-} ZDICT_fastCover_params_t;
-
-
-/*! ZDICT_optimizeTrainFromBuffer_fastCover():
- *  Train a dictionary from an array of samples using a modified version of the COVER algorithm.
- *  Samples must be stored concatenated in a single flat buffer `samplesBuffer`,
- *  supplied with an array of sizes `samplesSizes`, providing the size of each sample, in order.
- *  The resulting dictionary will be saved into `dictBuffer`.
- *  All of the parameters except for f are optional.
- *  If d is non-zero then we don't check multiple values of d, otherwise we check d = {6, 8, 10, 12, 14, 16}.
- *  if steps is zero it defaults to its default value.
- *  If k is non-zero then we don't check multiple values of k, otherwise we check steps values in [16, 2048].
- *
- *  @return: size of dictionary stored into `dictBuffer` (<= `dictBufferCapacity`)
- *           or an error code, which can be tested with ZDICT_isError().
- *           On success `*parameters` contains the parameters selected.
- */
- ZDICTLIB_API size_t ZDICT_optimizeTrainFromBuffer_fastCover(
-     void *dictBuffer, size_t dictBufferCapacity, const void *samplesBuffer,
-     const size_t *samplesSizes, unsigned nbSamples,
-     ZDICT_fastCover_params_t *parameters);
-
-
-/*! ZDICT_trainFromBuffer_fastCover():
- *  Train a dictionary from an array of samples using a modified version of the COVER algorithm.
- *  Samples must be stored concatenated in a single flat buffer `samplesBuffer`,
- *  supplied with an array of sizes `samplesSizes`, providing the size of each sample, in order.
- *  The resulting dictionary will be saved into `dictBuffer`.
- *  d, k, and f are required.
- *  @return: size of dictionary stored into `dictBuffer` (<= `dictBufferCapacity`)
- *           or an error code, which can be tested with ZDICT_isError().
- */
-ZDICTLIB_API size_t ZDICT_trainFromBuffer_fastCover(
-    void *dictBuffer, size_t dictBufferCapacity, const void *samplesBuffer,
-    const size_t *samplesSizes, unsigned nbSamples, ZDICT_fastCover_params_t parameters);
diff --git a/contrib/experimental_dict_builders/fastCover/main.c b/contrib/experimental_dict_builders/fastCover/main.c
deleted file mode 100644
index df7d91812e29..000000000000
--- a/contrib/experimental_dict_builders/fastCover/main.c
+++ /dev/null
@@ -1,183 +0,0 @@
-#include <stdio.h>  /* fprintf */
-#include <stdlib.h> /* malloc, free, qsort */
-#include <string.h>   /* strcmp, strlen */
-#include <errno.h>    /* errno */
-#include <ctype.h>
-#include "fastCover.h"
-#include "io.h"
-#include "util.h"
-#include "zdict.h"
-
-
-/*-*************************************
-*  Console display
-***************************************/
-#define DISPLAY(...)         fprintf(stderr, __VA_ARGS__)
-#define DISPLAYLEVEL(l, ...) if (displayLevel>=l) { DISPLAY(__VA_ARGS__); }
-
-static const U64 g_refreshRate = SEC_TO_MICRO / 6;
-static UTIL_time_t g_displayClock = UTIL_TIME_INITIALIZER;
-
-#define DISPLAYUPDATE(l, ...) { if (displayLevel>=l) { \
-            if ((UTIL_clockSpanMicro(g_displayClock) > g_refreshRate) || (displayLevel>=4)) \
-            { g_displayClock = UTIL_getTime(); DISPLAY(__VA_ARGS__); \
-            if (displayLevel>=4) fflush(stderr); } } }
-
-
-/*-*************************************
-*  Exceptions
-***************************************/
-#ifndef DEBUG
-#  define DEBUG 0
-#endif
-#define DEBUGOUTPUT(...) if (DEBUG) DISPLAY(__VA_ARGS__);
-#define EXM_THROW(error, ...)                                             \
-{                                                                         \
-    DEBUGOUTPUT("Error defined at %s, line %i : \n", __FILE__, __LINE__); \
-    DISPLAY("Error %i : ", error);                                        \
-    DISPLAY(__VA_ARGS__);                                                 \
-    DISPLAY("\n");                                                        \
-    exit(error);                                                          \
-}
-
-
-/*-*************************************
-*  Constants
-***************************************/
-static const unsigned g_defaultMaxDictSize = 110 KB;
-#define DEFAULT_CLEVEL 3
-
-
-/*-*************************************
-*  FASTCOVER
-***************************************/
-int FASTCOVER_trainFromFiles(const char* dictFileName, sampleInfo *info,
-                          unsigned maxDictSize,
-                          ZDICT_fastCover_params_t *params) {
-    unsigned const displayLevel = params->zParams.notificationLevel;
-    void* const dictBuffer = malloc(maxDictSize);
-
-    int result = 0;
-
-    /* Checks */
-    if (!dictBuffer)
-        EXM_THROW(12, "not enough memory for trainFromFiles");   /* should not happen */
-
-    {   size_t dictSize;
-        /* Run the optimize version if either k or d is not provided */
-        if (!params->d || !params->k) {
-          dictSize = ZDICT_optimizeTrainFromBuffer_fastCover(dictBuffer, maxDictSize, info->srcBuffer,
-                                               info->samplesSizes, info->nbSamples, params);
-        } else {
-          dictSize = ZDICT_trainFromBuffer_fastCover(dictBuffer, maxDictSize, info->srcBuffer,
-                                               info->samplesSizes, info->nbSamples, *params);
-        }
-        DISPLAYLEVEL(2, "k=%u\nd=%u\nf=%u\nsteps=%u\nsplit=%u\n", params->k, params->d, params->f, params->steps, (unsigned)(params->splitPoint*100));
-        if (ZDICT_isError(dictSize)) {
-            DISPLAYLEVEL(1, "dictionary training failed : %s \n", ZDICT_getErrorName(dictSize));   /* should not happen */
-            result = 1;
-            goto _done;
-        }
-        /* save dict */
-        DISPLAYLEVEL(2, "Save dictionary of size %u into file %s \n", (U32)dictSize, dictFileName);
-        saveDict(dictFileName, dictBuffer, dictSize);
-    }
-
-    /* clean up */
-_done:
-    free(dictBuffer);
-    return result;
-}
-
-
-
-int main(int argCount, const char* argv[])
-{
-  int displayLevel = 2;
-  const char* programName = argv[0];
-  int operationResult = 0;
-
-  /* Initialize arguments to default values */
-  unsigned k = 0;
-  unsigned d = 0;
-  unsigned f = 23;
-  unsigned steps = 32;
-  unsigned nbThreads = 1;
-  unsigned split = 100;
-  const char* outputFile = "fastCoverDict";
-  unsigned dictID = 0;
-  unsigned maxDictSize = g_defaultMaxDictSize;
-
-  /* Initialize table to store input files */
-  const char** filenameTable = (const char**)malloc(argCount * sizeof(const char*));
-  unsigned filenameIdx = 0;
-
-  char* fileNamesBuf = NULL;
-  unsigned fileNamesNb = filenameIdx;
-  int followLinks = 0; /* follow directory recursively */
-  const char** extendedFileList = NULL;
-
-  /* Parse arguments */
-  for (int i = 1; i < argCount; i++) {
-    const char* argument = argv[i];
-    if (longCommandWArg(&argument, "k=")) { k = readU32FromChar(&argument); continue; }
-    if (longCommandWArg(&argument, "d=")) { d = readU32FromChar(&argument); continue; }
-    if (longCommandWArg(&argument, "f=")) { f = readU32FromChar(&argument); continue; }
-    if (longCommandWArg(&argument, "steps=")) { steps = readU32FromChar(&argument); continue; }
-    if (longCommandWArg(&argument, "split=")) { split = readU32FromChar(&argument); continue; }
-    if (longCommandWArg(&argument, "dictID=")) { dictID = readU32FromChar(&argument); continue; }
-    if (longCommandWArg(&argument, "maxdict=")) { maxDictSize = readU32FromChar(&argument); continue; }
-    if (longCommandWArg(&argument, "in=")) {
-      filenameTable[filenameIdx] = argument;
-      filenameIdx++;
-      continue;
-    }
-    if (longCommandWArg(&argument, "out=")) {
-      outputFile = argument;
-      continue;
-    }
-    DISPLAYLEVEL(1, "Incorrect parameters\n");
-    operationResult = 1;
-    return operationResult;
-  }
-
-  /* Get the list of all files recursively (because followLinks==0)*/
-  extendedFileList = UTIL_createFileList(filenameTable, filenameIdx, &fileNamesBuf,
-                                        &fileNamesNb, followLinks);
-  if (extendedFileList) {
-      unsigned u;
-      for (u=0; u<fileNamesNb; u++) DISPLAYLEVEL(4, "%u %s\n", u, extendedFileList[u]);
-      free((void*)filenameTable);
-      filenameTable = extendedFileList;
-      filenameIdx = fileNamesNb;
-  }
-
-  size_t blockSize = 0;
-
-  /* Set up zParams */
-  ZDICT_params_t zParams;
-  zParams.compressionLevel = DEFAULT_CLEVEL;
-  zParams.notificationLevel = displayLevel;
-  zParams.dictID = dictID;
-
-  /* Set up fastCover params */
-  ZDICT_fastCover_params_t params;
-  params.zParams = zParams;
-  params.k = k;
-  params.d = d;
-  params.f = f;
-  params.steps = steps;
-  params.nbThreads = nbThreads;
-  params.splitPoint = (double)split/100;
-
-  /* Build dictionary */
-  sampleInfo* info = getSampleInfo(filenameTable,
-                    filenameIdx, blockSize, maxDictSize, zParams.notificationLevel);
-  operationResult = FASTCOVER_trainFromFiles(outputFile, info, maxDictSize, &params);
-
-  /* Free allocated memory */
-  UTIL_freeFileList(extendedFileList, fileNamesBuf);
-  freeSampleInfo(info);
-
-  return operationResult;
-}
diff --git a/contrib/experimental_dict_builders/fastCover/test.sh b/contrib/experimental_dict_builders/fastCover/test.sh
deleted file mode 100755
index f86915b59fc5..000000000000
--- a/contrib/experimental_dict_builders/fastCover/test.sh
+++ /dev/null
@@ -1,15 +0,0 @@
-echo "Building fastCover dictionary with in=../../lib/common f=20 out=dict1"
-./main in=../../../lib/common f=20 out=dict1
-zstd -be3 -D dict1 -r ../../../lib/common -q
-echo "Building fastCover dictionary with in=../../lib/common k=500 d=6 f=24 out=dict2 dictID=100 maxdict=140000"
-./main in=../../../lib/common k=500 d=6 f=24 out=dict2 dictID=100 maxdict=140000
-zstd -be3 -D dict2 -r ../../../lib/common -q
-echo "Building fastCover dictionary with 2 sample sources"
-./main in=../../../lib/common in=../../../lib/compress out=dict3
-zstd -be3 -D dict3 -r ../../../lib/common -q
-echo "Removing dict1 dict2 dict3"
-rm -f dict1 dict2 dict3
-
-echo "Testing with invalid parameters, should fail"
-! ./main in=../../../lib/common r=10
-! ./main in=../../../lib/common d=10
diff --git a/contrib/experimental_dict_builders/randomDictBuilder/Makefile b/contrib/experimental_dict_builders/randomDictBuilder/Makefile
deleted file mode 100644
index bbd40e47c312..000000000000
--- a/contrib/experimental_dict_builders/randomDictBuilder/Makefile
+++ /dev/null
@@ -1,52 +0,0 @@
-ARG :=
-
-CC ?= gcc
-CFLAGS ?= -O3
-INCLUDES := -I ../../../programs -I ../../../lib/common -I ../../../lib -I ../../../lib/dictBuilder
-
-TEST_INPUT := ../../../lib
-TEST_OUTPUT := randomDict
-
-all: main run clean
-
-.PHONY: test
-test: main testrun testshell clean
-
-.PHONY: run
-run:
-	echo "Building a random dictionary with given arguments"
-	./main $(ARG)
-
-main: main.o io.o random.o libzstd.a
-	$(CC) $(CFLAGS) main.o io.o random.o libzstd.a -o main
-
-main.o: main.c
-	$(CC) $(CFLAGS) $(INCLUDES) -c main.c
-
-random.o: random.c
-	$(CC) $(CFLAGS) $(INCLUDES) -c random.c
-
-io.o: io.c
-	$(CC) $(CFLAGS) $(INCLUDES) -c io.c
-
-libzstd.a:
-	$(MAKE) -C ../../../lib libzstd.a
-	mv ../../../lib/libzstd.a .
-
-.PHONY: testrun
-testrun: main
-	echo "Run with $(TEST_INPUT) and $(TEST_OUTPUT) "
-	./main in=$(TEST_INPUT) out=$(TEST_OUTPUT)
-	zstd -be3 -D $(TEST_OUTPUT) -r $(TEST_INPUT) -q
-	rm -f $(TEST_OUTPUT)
-
-.PHONY: testshell
-testshell: test.sh
-	sh test.sh
-	echo "Finish running test.sh"
-
-.PHONY: clean
-clean:
-	rm -f *.o main libzstd.a
-	$(MAKE) -C ../../../lib clean
-	echo "Cleaning is completed"
diff --git a/contrib/experimental_dict_builders/randomDictBuilder/README.md b/contrib/experimental_dict_builders/randomDictBuilder/README.md
deleted file mode 100644
index da12a4280541..000000000000
--- a/contrib/experimental_dict_builders/randomDictBuilder/README.md
+++ /dev/null
@@ -1,20 +0,0 @@
-Random Dictionary Builder
-
-### Permitted Arguments:
-Input File/Directory (in=fileName): required; file/directory used to build dictionary; if directory, will operate recursively for files inside directory; can include multiple files/directories, each following "in="
-Output Dictionary (out=dictName): if not provided, default to defaultDict
-Dictionary ID (dictID=#): nonnegative number; if not provided, default to 0
-Maximum Dictionary Size (maxdict=#): positive number; in bytes, if not provided, default to 110KB
-Size of Randomly Selected Segment (k=#): positive number; in bytes; if not provided, default to 200
-
-###Running Test:
-make test
-
-
-###Usage:
-To build a random dictionary with the provided arguments: make ARG= followed by arguments
-
-
-### Examples:
-make ARG="in=../../../lib/dictBuilder out=dict100 dictID=520"
-make ARG="in=../../../lib/dictBuilder in=../../../lib/compress"
diff --git a/contrib/experimental_dict_builders/randomDictBuilder/io.c b/contrib/experimental_dict_builders/randomDictBuilder/io.c
deleted file mode 100644
index bfe39eaed6b1..000000000000
--- a/contrib/experimental_dict_builders/randomDictBuilder/io.c
+++ /dev/null
@@ -1,284 +0,0 @@
-#include <stdio.h>  /* fprintf */
-#include <stdlib.h> /* malloc, free, qsort */
-#include <string.h>   /* strcmp, strlen */
-#include <errno.h>    /* errno */
-#include <ctype.h>
-#include "io.h"
-#include "fileio.h"   /* stdinmark, stdoutmark, ZSTD_EXTENSION */
-#include "platform.h"         /* Large Files support */
-#include "util.h"
-#include "zdict.h"
-
-/*-*************************************
-*  Console display
-***************************************/
-#define DISPLAY(...)         fprintf(stderr, __VA_ARGS__)
-#define DISPLAYLEVEL(l, ...) if (displayLevel>=l) { DISPLAY(__VA_ARGS__); }
-
-static const U64 g_refreshRate = SEC_TO_MICRO / 6;
-static UTIL_time_t g_displayClock = UTIL_TIME_INITIALIZER;
-
-#define DISPLAYUPDATE(l, ...) { if (displayLevel>=l) { \
-            if ((UTIL_clockSpanMicro(g_displayClock) > g_refreshRate) || (displayLevel>=4)) \
-            { g_displayClock = UTIL_getTime(); DISPLAY(__VA_ARGS__); \
-            if (displayLevel>=4) fflush(stderr); } } }
-
-/*-*************************************
-*  Exceptions
-***************************************/
-#ifndef DEBUG
-#  define DEBUG 0
-#endif
-#define DEBUGOUTPUT(...) if (DEBUG) DISPLAY(__VA_ARGS__);
-#define EXM_THROW(error, ...)                                             \
-{                                                                         \
-    DEBUGOUTPUT("Error defined at %s, line %i : \n", __FILE__, __LINE__); \
-    DISPLAY("Error %i : ", error);                                        \
-    DISPLAY(__VA_ARGS__);                                                 \
-    DISPLAY("\n");                                                        \
-    exit(error);                                                          \
-}
-
-
-/*-*************************************
-*  Constants
-***************************************/
-
-#define SAMPLESIZE_MAX (128 KB)
-#define RANDOM_MAX_SAMPLES_SIZE (sizeof(size_t) == 8 ? ((U32)-1) : ((U32)1 GB))
-#define RANDOM_MEMMULT 9
-static const size_t g_maxMemory = (sizeof(size_t) == 4) ?
-                          (2 GB - 64 MB) : ((size_t)(512 MB) << sizeof(size_t));
-
-#define NOISELENGTH 32
-
-
-/*-*************************************
-*  Commandline related functions
-***************************************/
-unsigned readU32FromChar(const char** stringPtr){
-    const char errorMsg[] = "error: numeric value too large";
-    unsigned result = 0;
-    while ((**stringPtr >='0') && (**stringPtr <='9')) {
-        unsigned const max = (((unsigned)(-1)) / 10) - 1;
-        if (result > max) exit(1);
-        result *= 10, result += **stringPtr - '0', (*stringPtr)++ ;
-    }
-    if ((**stringPtr=='K') || (**stringPtr=='M')) {
-        unsigned const maxK = ((unsigned)(-1)) >> 10;
-        if (result > maxK) exit(1);
-        result <<= 10;
-        if (**stringPtr=='M') {
-            if (result > maxK) exit(1);
-            result <<= 10;
-        }
-        (*stringPtr)++;  /* skip `K` or `M` */
-        if (**stringPtr=='i') (*stringPtr)++;
-        if (**stringPtr=='B') (*stringPtr)++;
-    }
-    return result;
-}
-
-unsigned longCommandWArg(const char** stringPtr, const char* longCommand){
-    size_t const comSize = strlen(longCommand);
-    int const result = !strncmp(*stringPtr, longCommand, comSize);
-    if (result) *stringPtr += comSize;
-    return result;
-}
-
-
-/* ********************************************************
-*  File related operations
-**********************************************************/
-/** loadFiles() :
- *  load samples from files listed in fileNamesTable into buffer.
- *  works even if buffer is too small to load all samples.
- *  Also provides the size of each sample into sampleSizes table
- *  which must be sized correctly, using DiB_fileStats().
- * @return : nb of samples effectively loaded into `buffer`
- * *bufferSizePtr is modified, it provides the amount data loaded within buffer.
- *  sampleSizes is filled with the size of each sample.
- */
-static unsigned loadFiles(void* buffer, size_t* bufferSizePtr, size_t* sampleSizes,
-                          unsigned sstSize, const char** fileNamesTable, unsigned nbFiles,
-                          size_t targetChunkSize, unsigned displayLevel) {
-    char* const buff = (char*)buffer;
-    size_t pos = 0;
-    unsigned nbLoadedChunks = 0, fileIndex;
-
-    for (fileIndex=0; fileIndex<nbFiles; fileIndex++) {
-        const char* const fileName = fileNamesTable[fileIndex];
-        unsigned long long const fs64 = UTIL_getFileSize(fileName);
-        unsigned long long remainingToLoad = (fs64 == UTIL_FILESIZE_UNKNOWN) ? 0 : fs64;
-        U32 const nbChunks = targetChunkSize ? (U32)((fs64 + (targetChunkSize-1)) / targetChunkSize) : 1;
-        U64 const chunkSize = targetChunkSize ? MIN(targetChunkSize, fs64) : fs64;
-        size_t const maxChunkSize = (size_t)MIN(chunkSize, SAMPLESIZE_MAX);
-        U32 cnb;
-        FILE* const f = fopen(fileName, "rb");
-        if (f==NULL) EXM_THROW(10, "zstd: dictBuilder: %s %s ", fileName, strerror(errno));
-        DISPLAYUPDATE(2, "Loading %s...       \r", fileName);
-        for (cnb=0; cnb<nbChunks; cnb++) {
-            size_t const toLoad = (size_t)MIN(maxChunkSize, remainingToLoad);
-            if (toLoad > *bufferSizePtr-pos) break;
-            {   size_t const readSize = fread(buff+pos, 1, toLoad, f);
-                if (readSize != toLoad) EXM_THROW(11, "Pb reading %s", fileName);
-                pos += readSize;
-                sampleSizes[nbLoadedChunks++] = toLoad;
-                remainingToLoad -= targetChunkSize;
-                if (nbLoadedChunks == sstSize) { /* no more space left in sampleSizes table */
-                    fileIndex = nbFiles;  /* stop there */
-                    break;
-                }
-                if (toLoad < targetChunkSize) {
-                    fseek(f, (long)(targetChunkSize - toLoad), SEEK_CUR);
-        }   }   }
-        fclose(f);
-    }
-    DISPLAYLEVEL(2, "\r%79s\r", "");
-    *bufferSizePtr = pos;
-    DISPLAYLEVEL(4, "loaded : %u KB \n", (U32)(pos >> 10))
-    return nbLoadedChunks;
-}
-
-#define rotl32(x,r) ((x << r) | (x >> (32 - r)))
-static U32 getRand(U32* src)
-{
-    static const U32 prime1 = 2654435761U;
-    static const U32 prime2 = 2246822519U;
-    U32 rand32 = *src;
-    rand32 *= prime1;
-    rand32 ^= prime2;
-    rand32  = rotl32(rand32, 13);
-    *src = rand32;
-    return rand32 >> 5;
-}
-
-/* shuffle() :
- * shuffle a table of file names in a semi-random way
- * It improves dictionary quality by reducing "locality" impact, so if sample set is very large,
- * it will load random elements from it, instead of just the first ones. */
-static void shuffle(const char** fileNamesTable, unsigned nbFiles) {
-    U32 seed = 0xFD2FB528;
-    unsigned i;
-    for (i = nbFiles - 1; i > 0; --i) {
-        unsigned const j = getRand(&seed) % (i + 1);
-        const char* const tmp = fileNamesTable[j];
-        fileNamesTable[j] = fileNamesTable[i];
-        fileNamesTable[i] = tmp;
-    }
-}
-
-
-/*-********************************************************
-*  Dictionary training functions
-**********************************************************/
-size_t findMaxMem(unsigned long long requiredMem) {
-    size_t const step = 8 MB;
-    void* testmem = NULL;
-
-    requiredMem = (((requiredMem >> 23) + 1) << 23);
-    requiredMem += step;
-    if (requiredMem > g_maxMemory) requiredMem = g_maxMemory;
-
-    while (!testmem) {
-        testmem = malloc((size_t)requiredMem);
-        requiredMem -= step;
-    }
-
-    free(testmem);
-    return (size_t)requiredMem;
-}
-
-void saveDict(const char* dictFileName,
-                         const void* buff, size_t buffSize) {
-    FILE* const f = fopen(dictFileName, "wb");
-    if (f==NULL) EXM_THROW(3, "cannot open %s ", dictFileName);
-
-    { size_t const n = fwrite(buff, 1, buffSize, f);
-      if (n!=buffSize) EXM_THROW(4, "%s : write error", dictFileName) }
-
-    { size_t const n = (size_t)fclose(f);
-      if (n!=0) EXM_THROW(5, "%s : flush error", dictFileName) }
-}
-
-/*! getFileStats() :
- *  Given a list of files, and a chunkSize (0 == no chunk, whole files)
- *  provides the amount of data to be loaded and the resulting nb of samples.
- *  This is useful primarily for allocation purpose => sample buffer, and sample sizes table.
- */
-static fileStats getFileStats(const char** fileNamesTable, unsigned nbFiles,
-                              size_t chunkSize, unsigned displayLevel) {
-    fileStats fs;
-    unsigned n;
-    memset(&fs, 0, sizeof(fs));
-    for (n=0; n<nbFiles; n++) {
-        U64 const fileSize = UTIL_getFileSize(fileNamesTable[n]);
-        U64 const srcSize = (fileSize == UTIL_FILESIZE_UNKNOWN) ? 0 : fileSize;
-        U32 const nbSamples = (U32)(chunkSize ? (srcSize + (chunkSize-1)) / chunkSize : 1);
-        U64 const chunkToLoad = chunkSize ? MIN(chunkSize, srcSize) : srcSize;
-        size_t const cappedChunkSize = (size_t)MIN(chunkToLoad, SAMPLESIZE_MAX);
-        fs.totalSizeToLoad += cappedChunkSize * nbSamples;
-        fs.oneSampleTooLarge |= (chunkSize > 2*SAMPLESIZE_MAX);
-        fs.nbSamples += nbSamples;
-    }
-    DISPLAYLEVEL(4, "Preparing to load : %u KB \n", (U32)(fs.totalSizeToLoad >> 10));
-    return fs;
-}
-
-
-
-
-sampleInfo* getSampleInfo(const char** fileNamesTable, unsigned nbFiles, size_t chunkSize,
-                          unsigned maxDictSize, const unsigned displayLevel) {
-    fileStats const fs = getFileStats(fileNamesTable, nbFiles, chunkSize, displayLevel);
-    size_t* const sampleSizes = (size_t*)malloc(fs.nbSamples * sizeof(size_t));
-    size_t const memMult = RANDOM_MEMMULT;
-    size_t const maxMem =  findMaxMem(fs.totalSizeToLoad * memMult) / memMult;
-    size_t loadedSize = (size_t) MIN ((unsigned long long)maxMem, fs.totalSizeToLoad);
-    void* const srcBuffer = malloc(loadedSize+NOISELENGTH);
-
-    /* Checks */
-    if ((!sampleSizes) || (!srcBuffer))
-        EXM_THROW(12, "not enough memory for trainFromFiles");   /* should not happen */
-    if (fs.oneSampleTooLarge) {
-        DISPLAYLEVEL(2, "!  Warning : some sample(s) are very large \n");
-        DISPLAYLEVEL(2, "!  Note that dictionary is only useful for small samples. \n");
-        DISPLAYLEVEL(2, "!  As a consequence, only the first %u bytes of each sample are loaded \n", SAMPLESIZE_MAX);
-    }
-    if (fs.nbSamples < 5) {
-        DISPLAYLEVEL(2, "!  Warning : nb of samples too low for proper processing ! \n");
-        DISPLAYLEVEL(2, "!  Please provide _one file per sample_. \n");
-        DISPLAYLEVEL(2, "!  Alternatively, split files into fixed-size blocks representative of samples, with -B# \n");
-        EXM_THROW(14, "nb of samples too low");   /* we now clearly forbid this case */
-    }
-    if (fs.totalSizeToLoad < (unsigned long long)(8 * maxDictSize)) {
-        DISPLAYLEVEL(2, "!  Warning : data size of samples too small for target dictionary size \n");
-        DISPLAYLEVEL(2, "!  Samples should be about 100x larger than target dictionary size \n");
-    }
-
-    /* init */
-    if (loadedSize < fs.totalSizeToLoad)
-        DISPLAYLEVEL(1, "Not enough memory; training on %u MB only...\n", (unsigned)(loadedSize >> 20));
-
-    /* Load input buffer */
-    DISPLAYLEVEL(3, "Shuffling input files\n");
-    shuffle(fileNamesTable, nbFiles);
-    nbFiles = loadFiles(srcBuffer, &loadedSize, sampleSizes, fs.nbSamples,
-                        fileNamesTable, nbFiles, chunkSize, displayLevel);
-
-    sampleInfo *info = (sampleInfo *)malloc(sizeof(sampleInfo));
-
-    info->nbSamples = fs.nbSamples;
-    info->samplesSizes = sampleSizes;
-    info->srcBuffer = srcBuffer;
-
-    return info;
-}
-
-
-void freeSampleInfo(sampleInfo *info) {
-    if (!info) return;
-    if (info->samplesSizes) free((void*)(info->samplesSizes));
-    if (info->srcBuffer) free((void*)(info->srcBuffer));
-    free(info);
-}
diff --git a/contrib/experimental_dict_builders/randomDictBuilder/io.h b/contrib/experimental_dict_builders/randomDictBuilder/io.h
deleted file mode 100644
index 0ee24604eed2..000000000000
--- a/contrib/experimental_dict_builders/randomDictBuilder/io.h
+++ /dev/null
@@ -1,60 +0,0 @@
-#include <stdio.h>  /* fprintf */
-#include <stdlib.h> /* malloc, free, qsort */
-#include <string.h>   /* strcmp, strlen */
-#include <errno.h>    /* errno */
-#include <ctype.h>
-#include "zstd_internal.h" /* includes zstd.h */
-#include "fileio.h"   /* stdinmark, stdoutmark, ZSTD_EXTENSION */
-#include "platform.h"         /* Large Files support */
-#include "util.h"
-#include "zdict.h"
-
-
-/*-*************************************
-*  Structs
-***************************************/
-typedef struct {
-    U64 totalSizeToLoad;
-    unsigned oneSampleTooLarge;
-    unsigned nbSamples;
-} fileStats;
-
-typedef struct {
-  const void* srcBuffer;
-  const size_t *samplesSizes;
-  size_t nbSamples;
-}sampleInfo;
-
-
-
-/*! getSampleInfo():
- *  Load from input files and add samples to buffer
- * @return: a sampleInfo struct containing infomation about buffer where samples are stored,
- *          size of each sample, and total number of samples
- */
-sampleInfo* getSampleInfo(const char** fileNamesTable, unsigned nbFiles, size_t chunkSize,
-                          unsigned maxDictSize, const unsigned displayLevel);
-
-
-
-/*! freeSampleInfo():
- *  Free memory allocated for info
- */
-void freeSampleInfo(sampleInfo *info);
-
-
-
-/*! saveDict():
- *  Save data stored on buff to dictFileName
- */
-void saveDict(const char* dictFileName, const void* buff, size_t buffSize);
-
-
-unsigned readU32FromChar(const char** stringPtr);
-
-/** longCommandWArg() :
- *  check if *stringPtr is the same as longCommand.
- *  If yes, @return 1 and advances *stringPtr to the position which immediately follows longCommand.
- * @return 0 and doesn't modify *stringPtr otherwise.
- */
-unsigned longCommandWArg(const char** stringPtr, const char* longCommand);
diff --git a/contrib/experimental_dict_builders/randomDictBuilder/main.c b/contrib/experimental_dict_builders/randomDictBuilder/main.c
deleted file mode 100644
index 3ad885746090..000000000000
--- a/contrib/experimental_dict_builders/randomDictBuilder/main.c
+++ /dev/null
@@ -1,161 +0,0 @@
-#include <stdio.h>  /* fprintf */
-#include <stdlib.h> /* malloc, free, qsort */
-#include <string.h>   /* strcmp, strlen */
-#include <errno.h>    /* errno */
-#include <ctype.h>
-#include "random.h"
-#include "io.h"
-#include "util.h"
-#include "zdict.h"
-
-
-/*-*************************************
-*  Console display
-***************************************/
-#define DISPLAY(...)         fprintf(stderr, __VA_ARGS__)
-#define DISPLAYLEVEL(l, ...) if (displayLevel>=l) { DISPLAY(__VA_ARGS__); }
-
-static const U64 g_refreshRate = SEC_TO_MICRO / 6;
-static UTIL_time_t g_displayClock = UTIL_TIME_INITIALIZER;
-
-#define DISPLAYUPDATE(l, ...) { if (displayLevel>=l) { \
-            if ((UTIL_clockSpanMicro(g_displayClock) > g_refreshRate) || (displayLevel>=4)) \
-            { g_displayClock = UTIL_getTime(); DISPLAY(__VA_ARGS__); \
-            if (displayLevel>=4) fflush(stderr); } } }
-
-
-/*-*************************************
-*  Exceptions
-***************************************/
-#ifndef DEBUG
-#  define DEBUG 0
-#endif
-#define DEBUGOUTPUT(...) if (DEBUG) DISPLAY(__VA_ARGS__);
-#define EXM_THROW(error, ...)                                             \
-{                                                                         \
-    DEBUGOUTPUT("Error defined at %s, line %i : \n", __FILE__, __LINE__); \
-    DISPLAY("Error %i : ", error);                                        \
-    DISPLAY(__VA_ARGS__);                                                 \
-    DISPLAY("\n");                                                        \
-    exit(error);                                                          \
-}
-
-
-/*-*************************************
-*  Constants
-***************************************/
-static const unsigned g_defaultMaxDictSize = 110 KB;
-#define DEFAULT_CLEVEL 3
-#define DEFAULT_k 200
-#define DEFAULT_OUTPUTFILE "defaultDict"
-#define DEFAULT_DICTID 0
-
-
-
-/*-*************************************
-*  RANDOM
-***************************************/
-int RANDOM_trainFromFiles(const char* dictFileName, sampleInfo *info,
-                          unsigned maxDictSize,
-                          ZDICT_random_params_t *params) {
-    unsigned const displayLevel = params->zParams.notificationLevel;
-    void* const dictBuffer = malloc(maxDictSize);
-
-    int result = 0;
-
-    /* Checks */
-    if (!dictBuffer)
-        EXM_THROW(12, "not enough memory for trainFromFiles");   /* should not happen */
-
-    {   size_t dictSize;
-        dictSize = ZDICT_trainFromBuffer_random(dictBuffer, maxDictSize, info->srcBuffer,
-                                             info->samplesSizes, info->nbSamples, *params);
-        DISPLAYLEVEL(2, "k=%u\n", params->k);
-        if (ZDICT_isError(dictSize)) {
-            DISPLAYLEVEL(1, "dictionary training failed : %s \n", ZDICT_getErrorName(dictSize));   /* should not happen */
-            result = 1;
-            goto _done;
-        }
-        /* save dict */
-        DISPLAYLEVEL(2, "Save dictionary of size %u into file %s \n", (U32)dictSize, dictFileName);
-        saveDict(dictFileName, dictBuffer, dictSize);
-    }
-
-    /* clean up */
-_done:
-    free(dictBuffer);
-    return result;
-}
-
-
-
-int main(int argCount, const char* argv[])
-{
-  int displayLevel = 2;
-  const char* programName = argv[0];
-  int operationResult = 0;
-
-  /* Initialize arguments to default values */
-  unsigned k = DEFAULT_k;
-  const char* outputFile = DEFAULT_OUTPUTFILE;
-  unsigned dictID = DEFAULT_DICTID;
-  unsigned maxDictSize = g_defaultMaxDictSize;
-
-  /* Initialize table to store input files */
-  const char** filenameTable = (const char**)malloc(argCount * sizeof(const char*));
-  unsigned filenameIdx = 0;
-
-  /* Parse arguments */
-  for (int i = 1; i < argCount; i++) {
-    const char* argument = argv[i];
-    if (longCommandWArg(&argument, "k=")) { k = readU32FromChar(&argument); continue; }
-    if (longCommandWArg(&argument, "dictID=")) { dictID = readU32FromChar(&argument); continue; }
-    if (longCommandWArg(&argument, "maxdict=")) { maxDictSize = readU32FromChar(&argument); continue; }
-    if (longCommandWArg(&argument, "in=")) {
-      filenameTable[filenameIdx] = argument;
-      filenameIdx++;
-      continue;
-    }
-    if (longCommandWArg(&argument, "out=")) {
-      outputFile = argument;
-      continue;
-    }
-    DISPLAYLEVEL(1, "Incorrect parameters\n");
-    operationResult = 1;
-    return operationResult;
-  }
-
-  char* fileNamesBuf = NULL;
-  unsigned fileNamesNb = filenameIdx;
-  int followLinks = 0; /* follow directory recursively */
-  const char** extendedFileList = NULL;
-  extendedFileList = UTIL_createFileList(filenameTable, filenameIdx, &fileNamesBuf,
-                                        &fileNamesNb, followLinks);
-  if (extendedFileList) {
-      unsigned u;
-      for (u=0; u<fileNamesNb; u++) DISPLAYLEVEL(4, "%u %s\n", u, extendedFileList[u]);
-      free((void*)filenameTable);
-      filenameTable = extendedFileList;
-      filenameIdx = fileNamesNb;
-  }
-
-  size_t blockSize = 0;
-
-  ZDICT_random_params_t params;
-  ZDICT_params_t zParams;
-  zParams.compressionLevel = DEFAULT_CLEVEL;
-  zParams.notificationLevel = displayLevel;
-  zParams.dictID = dictID;
-  params.zParams = zParams;
-  params.k = k;
-
-  sampleInfo* info = getSampleInfo(filenameTable,
-                    filenameIdx, blockSize, maxDictSize, zParams.notificationLevel);
-  operationResult = RANDOM_trainFromFiles(outputFile, info, maxDictSize, &params);
-
-  /* Free allocated memory */
-  UTIL_freeFileList(extendedFileList, fileNamesBuf);
-  freeSampleInfo(info);
-
-  return operationResult;
-}
diff --git a/contrib/experimental_dict_builders/randomDictBuilder/random.c b/contrib/experimental_dict_builders/randomDictBuilder/random.c
deleted file mode 100644
index 5276bea96a56..000000000000
--- a/contrib/experimental_dict_builders/randomDictBuilder/random.c
+++ /dev/null
@@ -1,163 +0,0 @@
-/*-*************************************
-*  Dependencies
-***************************************/
-#include <stdio.h>            /* fprintf */
-#include <stdlib.h>           /* malloc, free, qsort */
-#include <string.h>           /* memset */
-#include <time.h>             /* clock */
-#include "random.h"
-#include "util.h"             /* UTIL_getFileSize, UTIL_getTotalFileSize */
-#ifndef ZDICT_STATIC_LINKING_ONLY
-#define ZDICT_STATIC_LINKING_ONLY
-#endif
-#include "zdict.h"
-
-/*-*************************************
-*  Console display
-***************************************/
-#define DISPLAY(...)         fprintf(stderr, __VA_ARGS__)
-#define DISPLAYLEVEL(l, ...) if (displayLevel>=l) { DISPLAY(__VA_ARGS__); }
-
-#define LOCALDISPLAYUPDATE(displayLevel, l, ...)                               \
-  if (displayLevel >= l) {                                                     \
-    if ((clock() - g_time > refreshRate) || (displayLevel >= 4)) {             \
-      g_time = clock();                                                        \
-      DISPLAY(__VA_ARGS__);                                                    \
-    }                                                                          \
-  }
-#define DISPLAYUPDATE(l, ...) LOCALDISPLAYUPDATE(displayLevel, l, __VA_ARGS__)
-static const clock_t refreshRate = CLOCKS_PER_SEC * 15 / 100;
-static clock_t g_time = 0;
-
-
-
-/* ********************************************************
-*  Random Dictionary Builder
-**********************************************************/
-/**
- * Returns the sum of the sample sizes.
- */
-static size_t RANDOM_sum(const size_t *samplesSizes, unsigned nbSamples) {
-  size_t sum = 0;
-  unsigned i;
-  for (i = 0; i < nbSamples; ++i) {
-    sum += samplesSizes[i];
-  }
-  return sum;
-}
-
-
-/**
- * A segment is an inclusive range in the source.
- */
-typedef struct {
-  U32 begin;
-  U32 end;
-} RANDOM_segment_t;
-
-
-/**
- * Selects a random segment from totalSamplesSize - k + 1 possible segments
- */
-static RANDOM_segment_t RANDOM_selectSegment(const size_t totalSamplesSize,
-                                            ZDICT_random_params_t parameters) {
-    const U32 k = parameters.k;
-    RANDOM_segment_t segment;
-    unsigned index;
-
-    /* Randomly generate a number from 0 to sampleSizes - k */
-    index = rand()%(totalSamplesSize - k + 1);
-
-    /* inclusive */
-    segment.begin = index;
-    segment.end = index + k - 1;
-
-    return segment;
-}
-
-
-/**
- * Check the validity of the parameters.
- * Returns non-zero if the parameters are valid and 0 otherwise.
- */
-static int RANDOM_checkParameters(ZDICT_random_params_t parameters,
-                                  size_t maxDictSize) {
-    /* k is a required parameter */
-    if (parameters.k == 0) {
-      return 0;
-    }
-    /* k <= maxDictSize */
-    if (parameters.k > maxDictSize) {
-      return 0;
-    }
-    return 1;
-}
-
-
-/**
- * Given the prepared context build the dictionary.
- */
-static size_t RANDOM_buildDictionary(const size_t totalSamplesSize, const BYTE *samples,
-                                    void *dictBuffer, size_t dictBufferCapacity,
-                                    ZDICT_random_params_t parameters) {
-    BYTE *const dict = (BYTE *)dictBuffer;
-    size_t tail = dictBufferCapacity;
-    const int displayLevel = parameters.zParams.notificationLevel;
-    while (tail > 0) {
-
-      /* Select a segment */
-      RANDOM_segment_t segment = RANDOM_selectSegment(totalSamplesSize, parameters);
-
-      size_t segmentSize;
-      segmentSize = MIN(segment.end - segment.begin + 1, tail);
-
-      tail -= segmentSize;
-      memcpy(dict + tail, samples + segment.begin, segmentSize);
-      DISPLAYUPDATE(
-          2, "\r%u%%       ",
-          (U32)(((dictBufferCapacity - tail) * 100) / dictBufferCapacity));
-    }
-
-    return tail;
-}
-
-
-
-
-ZDICTLIB_API size_t ZDICT_trainFromBuffer_random(
-    void *dictBuffer, size_t dictBufferCapacity,
-    const void *samplesBuffer, const size_t *samplesSizes, unsigned nbSamples,
-    ZDICT_random_params_t parameters) {
-      const int displayLevel = parameters.zParams.notificationLevel;
-      BYTE* const dict = (BYTE*)dictBuffer;
-      /* Checks */
-      if (!RANDOM_checkParameters(parameters, dictBufferCapacity)) {
-          DISPLAYLEVEL(1, "k is incorrect\n");
-          return ERROR(GENERIC);
-      }
-      if (nbSamples == 0) {
-        DISPLAYLEVEL(1, "Random must have at least one input file\n");
-        return ERROR(GENERIC);
-      }
-      if (dictBufferCapacity < ZDICT_DICTSIZE_MIN) {
-        DISPLAYLEVEL(1, "dictBufferCapacity must be at least %u\n",
-                     ZDICT_DICTSIZE_MIN);
-        return ERROR(dstSize_tooSmall);
-      }
-      const size_t totalSamplesSize = RANDOM_sum(samplesSizes, nbSamples);
-      const BYTE *const samples = (const BYTE *)samplesBuffer;
-
-      DISPLAYLEVEL(2, "Building dictionary\n");
-      {
-        const size_t tail = RANDOM_buildDictionary(totalSamplesSize, samples,
-                                  dictBuffer, dictBufferCapacity, parameters);
-        const size_t dictSize = ZDICT_finalizeDictionary(
-            dict, dictBufferCapacity, dict + tail, dictBufferCapacity - tail,
-            samplesBuffer, samplesSizes, nbSamples, parameters.zParams);
-        if (!ZSTD_isError(dictSize)) {
-            DISPLAYLEVEL(2, "Constructed dictionary of size %u\n",
-                          (U32)dictSize);
-        }
-        return dictSize;
-      }
-}
diff --git a/contrib/experimental_dict_builders/randomDictBuilder/random.h b/contrib/experimental_dict_builders/randomDictBuilder/random.h
deleted file mode 100644
index 352775f950c4..000000000000
--- a/contrib/experimental_dict_builders/randomDictBuilder/random.h
+++ /dev/null
@@ -1,29 +0,0 @@
-#include <stdio.h>  /* fprintf */
-#include <stdlib.h> /* malloc, free, qsort */
-#include <string.h> /* memset */
-#include <time.h>   /* clock */
-#include "zstd_internal.h" /* includes zstd.h */
-#ifndef ZDICT_STATIC_LINKING_ONLY
-#define ZDICT_STATIC_LINKING_ONLY
-#endif
-#include "zdict.h"
-
-
-
-typedef struct {
-    unsigned k; /* Segment size : constraint: 0 < k : Reasonable range [16, 2048+]; Default to 200 */
-    ZDICT_params_t zParams;
-} ZDICT_random_params_t;
-
-
-/*! ZDICT_trainFromBuffer_random():
- *  Train a dictionary from an array of samples.
- *  Samples must be stored concatenated in a single flat buffer `samplesBuffer`,
- *  supplied with an array of sizes `samplesSizes`, providing the size of each sample, in order.
- *  The resulting dictionary will be saved into `dictBuffer`.
- * @return: size of dictionary stored into `dictBuffer` (<= `dictBufferCapacity`)
- *          or an error code, which can be tested with ZDICT_isError().
- */
-ZDICTLIB_API size_t ZDICT_trainFromBuffer_random( void *dictBuffer, size_t dictBufferCapacity,
-    const void *samplesBuffer, const size_t *samplesSizes, unsigned nbSamples,
-    ZDICT_random_params_t parameters);
diff --git a/contrib/experimental_dict_builders/randomDictBuilder/test.sh b/contrib/experimental_dict_builders/randomDictBuilder/test.sh
deleted file mode 100755
index 1eb732e52a09..000000000000
--- a/contrib/experimental_dict_builders/randomDictBuilder/test.sh
+++ /dev/null
@@ -1,14 +0,0 @@
-echo "Building random dictionary with in=../../lib/common k=200 out=dict1"
-./main in=../../../lib/common k=200 out=dict1
-zstd -be3 -D dict1 -r ../../../lib/common -q
-echo "Building random dictionary with in=../../lib/common k=500 out=dict2 dictID=100 maxdict=140000"
-./main in=../../../lib/common k=500 out=dict2 dictID=100 maxdict=140000
-zstd -be3 -D dict2 -r ../../../lib/common -q
-echo "Building random dictionary with 2 sample sources"
-./main in=../../../lib/common in=../../../lib/compress out=dict3
-zstd -be3 -D dict3 -r ../../../lib/common -q
-echo "Removing dict1 dict2 dict3"
-rm -f dict1 dict2 dict3
-
-echo "Testing with invalid parameters, should fail"
-! ./main r=10
diff --git a/contrib/gen_html/Makefile b/contrib/gen_html/Makefile
deleted file mode 100644
index 425f266c4e46..000000000000
--- a/contrib/gen_html/Makefile
+++ /dev/null
@@ -1,51 +0,0 @@
-# ################################################################
-# Copyright (c) 2016-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under both the BSD-style license (found in the
-# LICENSE file in the root directory of this source tree) and the GPLv2 (found
-# in the COPYING file in the root directory of this source tree).
-# ################################################################
-
-CXXFLAGS ?= -O3
-CXXFLAGS += -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wno-comment
-CXXFLAGS += $(MOREFLAGS)
-FLAGS   = $(CPPFLAGS) $(CXXFLAGS) $(LDFLAGS)
-
-ZSTDAPI = ../../lib/zstd.h
-ZSTDMANUAL = ../../doc/zstd_manual.html
-LIBVER_MAJOR_SCRIPT:=`sed -n '/define ZSTD_VERSION_MAJOR/s/.*[[:blank:]]\([0-9][0-9]*\).*/\1/p' < $(ZSTDAPI)`
-LIBVER_MINOR_SCRIPT:=`sed -n '/define ZSTD_VERSION_MINOR/s/.*[[:blank:]]\([0-9][0-9]*\).*/\1/p' < $(ZSTDAPI)`
-LIBVER_PATCH_SCRIPT:=`sed -n '/define ZSTD_VERSION_RELEASE/s/.*[[:blank:]]\([0-9][0-9]*\).*/\1/p' < $(ZSTDAPI)`
-LIBVER_SCRIPT:= $(LIBVER_MAJOR_SCRIPT).$(LIBVER_MINOR_SCRIPT).$(LIBVER_PATCH_SCRIPT)
-LIBVER := $(shell echo $(LIBVER_SCRIPT))
-
-
-# Define *.exe as extension for Windows systems
-ifneq (,$(filter Windows%,$(OS)))
-EXT =.exe
-else
-EXT =
-endif
-
-
-.PHONY: default
-default: gen_html
-
-.PHONY: all
-all: manual
-
-gen_html: gen_html.cpp
-	$(CXX) $(FLAGS) $^ -o $@$(EXT)
-
-$(ZSTDMANUAL): gen_html $(ZSTDAPI)
-	echo "Update zstd manual in /doc"
-	./gen_html $(LIBVER) $(ZSTDAPI) $(ZSTDMANUAL)
-
-.PHONY: manual
-manual: gen_html $(ZSTDMANUAL)
-
-.PHONY: clean
-clean:
-	@$(RM) gen_html$(EXT)
-	@echo Cleaning completed
diff --git a/contrib/gen_html/README.md b/contrib/gen_html/README.md
deleted file mode 100644
index 63a4caa25061..000000000000
--- a/contrib/gen_html/README.md
+++ /dev/null
@@ -1,31 +0,0 @@
-gen_html - a program for automatic generation of zstd manual 
-============================================================
-
-#### Introduction
-
-This simple C++ program generates a single-page HTML manual from `zstd.h`.
-
-The format of recognized comment blocks is following:
-- comments of type `/*!` mean: this is a function declaration; switch comments with declarations
-- comments of type `/**` and `/*-` mean: this is a comment; use a `<H2>` header for the first line
-- comments of type `/*=` and `/**=` mean: use a `<H3>` header and show also all functions until first empty line
-- comments of type `/*X` where `X` is different from above-mentioned are ignored
-
-Moreover:
-- `ZSTDLIB_API` is removed to improve readability
-- `typedef` are detected and included even if uncommented
-- comments of type `/**<` and `/*!<` are detected and only function declaration is highlighted (bold)
-
-
-#### Usage
-
-The program requires 3 parameters:
-```
-gen_html [zstd_version] [input_file] [output_html]
-```
-
-To compile program and generate zstd manual we have used: 
-```
-make
-./gen_html.exe 1.1.1 ../../lib/zstd.h zstd_manual.html
-```
diff --git a/contrib/gen_html/gen-zstd-manual.sh b/contrib/gen_html/gen-zstd-manual.sh
deleted file mode 100755
index 57a8b6ea512a..000000000000
--- a/contrib/gen_html/gen-zstd-manual.sh
+++ /dev/null
@@ -1,9 +0,0 @@
-#!/bin/sh
-
-LIBVER_MAJOR_SCRIPT=`sed -n '/define ZSTD_VERSION_MAJOR/s/.*[[:blank:]]\([0-9][0-9]*\).*/\1/p' < ../../lib/zstd.h`
-LIBVER_MINOR_SCRIPT=`sed -n '/define ZSTD_VERSION_MINOR/s/.*[[:blank:]]\([0-9][0-9]*\).*/\1/p' < ../../lib/zstd.h`
-LIBVER_PATCH_SCRIPT=`sed -n '/define ZSTD_VERSION_RELEASE/s/.*[[:blank:]]\([0-9][0-9]*\).*/\1/p' < ../../lib/zstd.h`
-LIBVER_SCRIPT=$LIBVER_MAJOR_SCRIPT.$LIBVER_MINOR_SCRIPT.$LIBVER_PATCH_SCRIPT
-
-echo ZSTD_VERSION=$LIBVER_SCRIPT
-./gen_html $LIBVER_SCRIPT ../../lib/zstd.h ./zstd_manual.html
diff --git a/contrib/gen_html/gen_html.cpp b/contrib/gen_html/gen_html.cpp
deleted file mode 100644
index 90d5b21a3aa6..000000000000
--- a/contrib/gen_html/gen_html.cpp
+++ /dev/null
@@ -1,224 +0,0 @@
-/*
- * Copyright (c) 2016-present, Przemyslaw Skibinski, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-
-#include <iostream>
-#include <fstream>
-#include <sstream>
-#include <vector>
-using namespace std;
-
-
-/* trim string at the beginning and at the end */
-void trim(string& s, string characters)
-{
-    size_t p = s.find_first_not_of(characters);
-    s.erase(0, p);
-
-    p = s.find_last_not_of(characters);
-    if (string::npos != p)
-       s.erase(p+1);
-}
-
-
-/* trim C++ style comments */
-void trim_comments(string &s)
-{
-    size_t spos, epos;
-
-    spos = s.find("/*");
-    epos = s.find("*/");
-    s = s.substr(spos+3, epos-(spos+3));
-}
-
-
-/* get lines until a given terminator */
-vector<string> get_lines(vector<string>& input, int& linenum, string terminator)
-{
-    vector<string> out;
-    string line;
-    size_t epos;
-
-    while ((size_t)linenum < input.size()) {
-        line = input[linenum];
-
-        if (terminator.empty() && line.empty()) { linenum--; break; }
-
-        epos = line.find(terminator);
-        if (!terminator.empty() && epos!=string::npos) {
-            out.push_back(line);
-            break;
-        }
-        out.push_back(line);
-        linenum++;
-    }
-    return out;
-}
-
-
-/* print line with ZSTDLIB_API removed and C++ comments not bold */
-void print_line(stringstream &sout, string line)
-{
-    size_t spos;
-
-    if (line.substr(0,12) == "ZSTDLIB_API ") line = line.substr(12);
-    spos = line.find("/*");
-    if (spos!=string::npos) {
-        sout << line.substr(0, spos);
-        sout << "</b>" << line.substr(spos) << "<b>" << endl;
-    } else {
-      //  fprintf(stderr, "lines=%s\n", line.c_str());
-        sout << line << endl;
-    }
-}
-
-
-int main(int argc, char *argv[]) {
-    char exclam;
-    int linenum, chapter = 1;
-    vector<string> input, lines, comments, chapters;
-    string line, version;
-    size_t spos, l;
-    stringstream sout;
-    ifstream istream;
-    ofstream ostream;
-
-    if (argc < 4) {
-        cout << "usage: " << argv[0] << " [zstd_version] [input_file] [output_html]" << endl;
-        return 1;
-    }
-
-    version = "zstd " + string(argv[1]) + " Manual";
-
-    istream.open(argv[2], ifstream::in);
-    if (!istream.is_open()) {
-        cout << "Error opening file " << argv[2] << endl;
-        return 1;
-    }
-
-    ostream.open(argv[3], ifstream::out);
-    if (!ostream.is_open()) {
-        cout << "Error opening file " << argv[3] << endl;
-        return 1;
-   }
-
-    while (getline(istream, line)) {
-        input.push_back(line);
-    }
-
-    for (linenum=0; (size_t)linenum < input.size(); linenum++) {
-        line = input[linenum];
-
-        /* typedefs are detected and included even if uncommented */
-        if (line.substr(0,7) == "typedef" && line.find("{")!=string::npos) {
-            lines = get_lines(input, linenum, "}");
-            sout << "<pre><b>";
-            for (l=0; l<lines.size(); l++) {
-                print_line(sout, lines[l]);
-            }
-            sout << "</b></pre><BR>" << endl;
-            continue;
-        }
-
-        /* comments of type /**< and /*!< are detected and only function declaration is highlighted (bold) */
-        if ((line.find("/**<")!=string::npos || line.find("/*!<")!=string::npos) && line.find("*/")!=string::npos) {
-            sout << "<pre><b>";
-            print_line(sout, line);
-            sout << "</b></pre><BR>" << endl;
-            continue;
-        }
-
-        spos = line.find("/**=");
-        if (spos==string::npos) {
-            spos = line.find("/*!");
-            if (spos==string::npos)
-                spos = line.find("/**");
-            if (spos==string::npos)
-                spos = line.find("/*-");
-            if (spos==string::npos)
-                spos = line.find("/*=");
-            if (spos==string::npos)
-                continue;
-            exclam = line[spos+2];
-        }
-        else exclam = '=';
-
-        comments = get_lines(input, linenum, "*/");
-        if (!comments.empty()) comments[0] = line.substr(spos+3);
-        if (!comments.empty()) comments[comments.size()-1] = comments[comments.size()-1].substr(0, comments[comments.size()-1].find("*/"));
-        for (l=0; l<comments.size(); l++) {
-            if (comments[l].find(" *")==0) comments[l] = comments[l].substr(2);
-            else if (comments[l].find("  *")==0) comments[l] = comments[l].substr(3);
-            trim(comments[l], "*-=");
-        }
-        while (!comments.empty() && comments[comments.size()-1].empty()) comments.pop_back(); // remove empty line at the end
-        while (!comments.empty() && comments[0].empty()) comments.erase(comments.begin()); // remove empty line at the start
-
-        /* comments of type /*! mean: this is a function declaration; switch comments with declarations */
-        if (exclam == '!') {
-            if (!comments.empty()) comments.erase(comments.begin()); /* remove first line like "ZSTD_XXX() :" */
-            linenum++;
-            lines = get_lines(input, linenum, "");
-
-            sout << "<pre><b>";
-            for (l=0; l<lines.size(); l++) {
-              //  fprintf(stderr, "line[%d]=%s\n", l, lines[l].c_str());
-                string fline = lines[l];
-                if (fline.substr(0, 12) == "ZSTDLIB_API " ||
-                    fline.substr(0, 12) == string(12, ' '))
-                  fline = fline.substr(12);
-                print_line(sout, fline);
-            }
-            sout << "</b><p>";
-            for (l=0; l<comments.size(); l++) {
-                print_line(sout, comments[l]);
-            }
-            sout << "</p></pre><BR>" << endl << endl;
-        } else if (exclam == '=') { /* comments of type /*= and /**= mean: use a <H3> header and show also all functions until first empty line */
-            trim(comments[0], " ");
-            sout << "<h3>" << comments[0] << "</h3><pre>";
-            for (l=1; l<comments.size(); l++) {
-                print_line(sout, comments[l]);
-            }
-            sout << "</pre><b><pre>";
-            lines = get_lines(input, ++linenum, "");
-            for (l=0; l<lines.size(); l++) {
-                print_line(sout, lines[l]);
-            }
-            sout << "</pre></b><BR>" << endl;
-        } else { /* comments of type /** and /*- mean: this is a comment; use a <H2> header for the first line */
-            if (comments.empty()) continue;
-
-            trim(comments[0], " ");
-            sout << "<a name=\"Chapter" << chapter << "\"></a><h2>" << comments[0] << "</h2><pre>";
-            chapters.push_back(comments[0]);
-            chapter++;
-
-            for (l=1; l<comments.size(); l++) {
-                print_line(sout, comments[l]);
-            }
-            if (comments.size() > 1)
-                sout << "<BR></pre>" << endl << endl;
-            else
-                sout << "</pre>" << endl << endl;
-        }
-    }
-
-    ostream << "<html>\n<head>\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=ISO-8859-1\">\n<title>" << version << "</title>\n</head>\n<body>" << endl;
-    ostream << "<h1>" << version << "</h1>\n";
-
-    ostream << "<hr>\n<a name=\"Contents\"></a><h2>Contents</h2>\n<ol>\n";
-    for (size_t i=0; i<chapters.size(); i++)
-        ostream << "<li><a href=\"#Chapter" << i+1 << "\">" << chapters[i].c_str() << "</a></li>\n";
-    ostream << "</ol>\n<hr>\n";
-
-    ostream << sout.str();
-    ostream << "</html>" << endl << "</body>" << endl;
-
-    return 0;
-}
diff --git a/contrib/largeNbDicts/Makefile b/contrib/largeNbDicts/Makefile
deleted file mode 100644
index 4c055b0ed3fe..000000000000
--- a/contrib/largeNbDicts/Makefile
+++ /dev/null
@@ -1,58 +0,0 @@
-# ################################################################
-# Copyright (c) 2018-present, Yann Collet, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under both the BSD-style license (found in the
-# LICENSE file in the root directory of this source tree) and the GPLv2 (found
-# in the COPYING file in the root directory of this source tree).
-# ################################################################
-
-PROGDIR = ../../programs
-LIBDIR  = ../../lib
-
-LIBZSTD = $(LIBDIR)/libzstd.a
-
-CPPFLAGS+= -I$(LIBDIR) -I$(LIBDIR)/common -I$(LIBDIR)/dictBuilder -I$(PROGDIR)
-
-CFLAGS  ?= -O3
-CFLAGS  += -std=gnu99
-DEBUGFLAGS= -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow \
-            -Wstrict-aliasing=1 -Wswitch-enum \
-            -Wstrict-prototypes -Wundef -Wpointer-arith \
-            -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings \
-            -Wredundant-decls
-CFLAGS  += $(DEBUGFLAGS) $(MOREFLAGS)
-
-
-default: largeNbDicts
-
-all : largeNbDicts
-
-largeNbDicts: util.o timefn.o benchfn.o datagen.o xxhash.o largeNbDicts.c $(LIBZSTD)
-	$(CC) $(CPPFLAGS) $(CFLAGS) $^ $(LDFLAGS) -o $@
-
-.PHONY: $(LIBZSTD)
-$(LIBZSTD):
-	$(MAKE) -C $(LIBDIR) libzstd.a CFLAGS="$(CFLAGS)"
-
-benchfn.o: $(PROGDIR)/benchfn.c
-	$(CC) $(CPPFLAGS) $(CFLAGS) $^ -c
-
-timefn.o: $(PROGDIR)/timefn.c
-	$(CC) $(CPPFLAGS) $(CFLAGS) $^ -c
-
-datagen.o: $(PROGDIR)/datagen.c
-	$(CC) $(CPPFLAGS) $(CFLAGS) $^ -c
-
-util.o: $(PROGDIR)/util.c
-	$(CC) $(CPPFLAGS) $(CFLAGS) $^ -c
-
-
-xxhash.o : $(LIBDIR)/common/xxhash.c
-	$(CC) $(CPPFLAGS) $(CFLAGS) $^ -c
-
-
-clean:
-	$(RM) *.o
-	$(MAKE) -C $(LIBDIR) clean > /dev/null
-	$(RM) largeNbDicts
diff --git a/contrib/largeNbDicts/README.md b/contrib/largeNbDicts/README.md
deleted file mode 100644
index f29bcdfe8e37..000000000000
--- a/contrib/largeNbDicts/README.md
+++ /dev/null
@@ -1,25 +0,0 @@
-largeNbDicts
-=====================
-
-`largeNbDicts` is a benchmark test tool
-dedicated to the specific scenario of
-dictionary decompression using a very large number of dictionaries.
-When dictionaries are constantly changing, they are always "cold",
-suffering from increased latency due to cache misses.
-
-The tool is created in a bid to investigate performance for this scenario,
-and experiment mitigation techniques.
-
-Command line :
-```
-largeNbDicts [Options] filename(s)
-
-Options :
--r           : recursively load all files in subdirectories (default: off)
--B#          : split input into blocks of size # (default: no split)
--#           : use compression level # (default: 3)
--D #         : use # as a dictionary (default: create one)
--i#          : nb benchmark rounds (default: 6)
---nbDicts=#  : set nb of dictionaries to # (default: one per block)
--h           : help (this text)
-```
diff --git a/contrib/largeNbDicts/largeNbDicts.c b/contrib/largeNbDicts/largeNbDicts.c
deleted file mode 100644
index 627a6910576f..000000000000
--- a/contrib/largeNbDicts/largeNbDicts.c
+++ /dev/null
@@ -1,817 +0,0 @@
-/*
- * Copyright (c) 2018-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-/* largeNbDicts
- * This is a benchmark test tool
- * dedicated to the specific case of dictionary decompression
- * using a very large nb of dictionaries
- * thus suffering latency from lots of cache misses.
- * It's created in a bid to investigate performance and find optimizations. */
-
-
-/*---  Dependencies  ---*/
-
-#include <stddef.h>   /* size_t */
-#include <stdlib.h>   /* malloc, free, abort */
-#include <stdio.h>    /* fprintf */
-#include <limits.h>   /* UINT_MAX */
-#include <assert.h>   /* assert */
-
-#include "util.h"
-#include "benchfn.h"
-#define ZSTD_STATIC_LINKING_ONLY
-#include "zstd.h"
-#include "zdict.h"
-
-
-/*---  Constants  --- */
-
-#define KB  *(1<<10)
-#define MB  *(1<<20)
-
-#define BLOCKSIZE_DEFAULT 0  /* no slicing into blocks */
-#define DICTSIZE  (4 KB)
-#define CLEVEL_DEFAULT 3
-
-#define BENCH_TIME_DEFAULT_S   6
-#define RUN_TIME_DEFAULT_MS    1000
-#define BENCH_TIME_DEFAULT_MS (BENCH_TIME_DEFAULT_S * RUN_TIME_DEFAULT_MS)
-
-#define DISPLAY_LEVEL_DEFAULT 3
-
-#define BENCH_SIZE_MAX (1200 MB)
-
-
-/*---  Macros  ---*/
-
-#define CONTROL(c)   { if (!(c)) abort(); }
-#undef MIN
-#define MIN(a,b)     ((a) < (b) ? (a) : (b))
-
-
-/*---  Display Macros  ---*/
-
-#define DISPLAY(...)         fprintf(stdout, __VA_ARGS__)
-#define DISPLAYLEVEL(l, ...) { if (g_displayLevel>=l) { DISPLAY(__VA_ARGS__); } }
-static int g_displayLevel = DISPLAY_LEVEL_DEFAULT;   /* 0 : no display,  1: errors,  2 : + result + interaction + warnings,  3 : + progression,  4 : + information */
-
-
-/*---  buffer_t  ---*/
-
-typedef struct {
-    void* ptr;
-    size_t size;
-    size_t capacity;
-} buffer_t;
-
-static const buffer_t kBuffNull = { NULL, 0, 0 };
-
-/* @return : kBuffNull if any error */
-static buffer_t createBuffer(size_t capacity)
-{
-    assert(capacity > 0);
-    void* const ptr = malloc(capacity);
-    if (ptr==NULL) return kBuffNull;
-
-    buffer_t buffer;
-    buffer.ptr = ptr;
-    buffer.capacity = capacity;
-    buffer.size = 0;
-    return buffer;
-}
-
-static void freeBuffer(buffer_t buff)
-{
-    free(buff.ptr);
-}
-
-
-static void fillBuffer_fromHandle(buffer_t* buff, FILE* f)
-{
-    size_t const readSize = fread(buff->ptr, 1, buff->capacity, f);
-    buff->size = readSize;
-}
-
-
-/* @return : kBuffNull if any error */
-static buffer_t createBuffer_fromFile(const char* fileName)
-{
-    U64 const fileSize = UTIL_getFileSize(fileName);
-    size_t const bufferSize = (size_t) fileSize;
-
-    if (fileSize == UTIL_FILESIZE_UNKNOWN) return kBuffNull;
-    assert((U64)bufferSize == fileSize);   /* check overflow */
-
-    {   FILE* const f = fopen(fileName, "rb");
-        if (f == NULL) return kBuffNull;
-
-        buffer_t buff = createBuffer(bufferSize);
-        CONTROL(buff.ptr != NULL);
-
-        fillBuffer_fromHandle(&buff, f);
-        CONTROL(buff.size == buff.capacity);
-
-        fclose(f);   /* do nothing specific if fclose() fails */
-        return buff;
-    }
-}
-
-
-/* @return : kBuffNull if any error */
-static buffer_t
-createDictionaryBuffer(const char* dictionaryName,
-                       const void* srcBuffer,
-                       const size_t* srcBlockSizes, size_t nbBlocks,
-                       size_t requestedDictSize)
-{
-    if (dictionaryName) {
-        DISPLAYLEVEL(3, "loading dictionary %s \n", dictionaryName);
-        return createBuffer_fromFile(dictionaryName);  /* note : result might be kBuffNull */
-
-    } else {
-
-        DISPLAYLEVEL(3, "creating dictionary, of target size %u bytes \n",
-                        (unsigned)requestedDictSize);
-        void* const dictBuffer = malloc(requestedDictSize);
-        CONTROL(dictBuffer != NULL);
-
-        assert(nbBlocks <= UINT_MAX);
-        size_t const dictSize = ZDICT_trainFromBuffer(dictBuffer, requestedDictSize,
-                                                      srcBuffer,
-                                                      srcBlockSizes, (unsigned)nbBlocks);
-        CONTROL(!ZSTD_isError(dictSize));
-
-        buffer_t result;
-        result.ptr = dictBuffer;
-        result.capacity = requestedDictSize;
-        result.size = dictSize;
-        return result;
-    }
-}
-
-
-/*! BMK_loadFiles() :
- *  Loads `buffer`, with content from files listed within `fileNamesTable`.
- *  Fills `buffer` entirely.
- * @return : 0 on success, !=0 on error */
-static int loadFiles(void* buffer, size_t bufferSize,
-                     size_t* fileSizes,
-                     const char* const * fileNamesTable, unsigned nbFiles)
-{
-    size_t pos = 0, totalSize = 0;
-
-    for (unsigned n=0; n<nbFiles; n++) {
-        U64 fileSize = UTIL_getFileSize(fileNamesTable[n]);
-        if (UTIL_isDirectory(fileNamesTable[n])) {
-            fileSizes[n] = 0;
-            continue;
-        }
-        if (fileSize == UTIL_FILESIZE_UNKNOWN) {
-            fileSizes[n] = 0;
-            continue;
-        }
-
-        FILE* const f = fopen(fileNamesTable[n], "rb");
-        assert(f!=NULL);
-
-        assert(pos <= bufferSize);
-        assert(fileSize <= bufferSize - pos);
-
-        {   size_t const readSize = fread(((char*)buffer)+pos, 1, (size_t)fileSize, f);
-            assert(readSize == fileSize);
-            pos += readSize;
-        }
-        fileSizes[n] = (size_t)fileSize;
-        totalSize += (size_t)fileSize;
-        fclose(f);
-    }
-
-    assert(totalSize == bufferSize);
-    return 0;
-}
-
-
-
-/*---  slice_collection_t  ---*/
-
-typedef struct {
-    void** slicePtrs;
-    size_t* capacities;
-    size_t nbSlices;
-} slice_collection_t;
-
-static const slice_collection_t kNullCollection = { NULL, NULL, 0 };
-
-static void freeSliceCollection(slice_collection_t collection)
-{
-    free(collection.slicePtrs);
-    free(collection.capacities);
-}
-
-/* shrinkSizes() :
- * downsizes sizes of slices within collection, according to `newSizes`.
- * every `newSizes` entry must be <= than its corresponding collection size */
-void shrinkSizes(slice_collection_t collection,
-                 const size_t* newSizes)  /* presumed same size as collection */
-{
-    size_t const nbSlices = collection.nbSlices;
-    for (size_t blockNb = 0; blockNb < nbSlices; blockNb++) {
-        assert(newSizes[blockNb] <= collection.capacities[blockNb]);
-        collection.capacities[blockNb] = newSizes[blockNb];
-    }
-}
-
-
-/* splitSlices() :
- * nbSlices : if == 0, nbSlices is automatically determined from srcSlices and blockSize.
- *            otherwise, creates exactly nbSlices slices,
- *            by either truncating input (when smaller)
- *            or repeating input from beginning */
-static slice_collection_t
-splitSlices(slice_collection_t srcSlices, size_t blockSize, size_t nbSlices)
-{
-    if (blockSize==0) blockSize = (size_t)(-1);   /* means "do not cut" */
-    size_t nbSrcBlocks = 0;
-    for (size_t ssnb=0; ssnb < srcSlices.nbSlices; ssnb++) {
-        size_t pos = 0;
-        while (pos <= srcSlices.capacities[ssnb]) {
-            nbSrcBlocks++;
-            pos += blockSize;
-        }
-    }
-
-    if (nbSlices == 0) nbSlices = nbSrcBlocks;
-
-    void** const sliceTable = (void**)malloc(nbSlices * sizeof(*sliceTable));
-    size_t* const capacities = (size_t*)malloc(nbSlices * sizeof(*capacities));
-    if (sliceTable == NULL || capacities == NULL) {
-        free(sliceTable);
-        free(capacities);
-        return kNullCollection;
-    }
-
-    size_t ssnb = 0;
-    for (size_t sliceNb=0; sliceNb < nbSlices; ) {
-        ssnb = (ssnb + 1) % srcSlices.nbSlices;
-        size_t pos = 0;
-        char* const ptr = (char*)srcSlices.slicePtrs[ssnb];
-        while (pos < srcSlices.capacities[ssnb] && sliceNb < nbSlices) {
-            size_t const size = MIN(blockSize, srcSlices.capacities[ssnb] - pos);
-            sliceTable[sliceNb] = ptr + pos;
-            capacities[sliceNb] = size;
-            sliceNb++;
-            pos += blockSize;
-        }
-    }
-
-    slice_collection_t result;
-    result.nbSlices = nbSlices;
-    result.slicePtrs = sliceTable;
-    result.capacities = capacities;
-    return result;
-}
-
-
-static size_t sliceCollection_totalCapacity(slice_collection_t sc)
-{
-    size_t totalSize = 0;
-    for (size_t n=0; n<sc.nbSlices; n++)
-        totalSize += sc.capacities[n];
-    return totalSize;
-}
-
-
-/* ---  buffer collection  --- */
-
-typedef struct {
-    buffer_t buffer;
-    slice_collection_t slices;
-} buffer_collection_t;
-
-
-static void freeBufferCollection(buffer_collection_t bc)
-{
-    freeBuffer(bc.buffer);
-    freeSliceCollection(bc.slices);
-}
-
-
-static buffer_collection_t
-createBufferCollection_fromSliceCollectionSizes(slice_collection_t sc)
-{
-    size_t const bufferSize = sliceCollection_totalCapacity(sc);
-
-    buffer_t buffer = createBuffer(bufferSize);
-    CONTROL(buffer.ptr != NULL);
-
-    size_t const nbSlices = sc.nbSlices;
-    void** const slices = (void**)malloc(nbSlices * sizeof(*slices));
-    CONTROL(slices != NULL);
-
-    size_t* const capacities = (size_t*)malloc(nbSlices * sizeof(*capacities));
-    CONTROL(capacities != NULL);
-
-    char* const ptr = (char*)buffer.ptr;
-    size_t pos = 0;
-    for (size_t n=0; n < nbSlices; n++) {
-        capacities[n] = sc.capacities[n];
-        slices[n] = ptr + pos;
-        pos += capacities[n];
-    }
-
-    buffer_collection_t result;
-    result.buffer = buffer;
-    result.slices.nbSlices = nbSlices;
-    result.slices.capacities = capacities;
-    result.slices.slicePtrs = slices;
-    return result;
-}
-
-
-/* @return : kBuffNull if any error */
-static buffer_collection_t
-createBufferCollection_fromFiles(const char* const * fileNamesTable, unsigned nbFiles)
-{
-    U64 const totalSizeToLoad = UTIL_getTotalFileSize(fileNamesTable, nbFiles);
-    assert(totalSizeToLoad != UTIL_FILESIZE_UNKNOWN);
-    assert(totalSizeToLoad <= BENCH_SIZE_MAX);
-    size_t const loadedSize = (size_t)totalSizeToLoad;
-    assert(loadedSize > 0);
-    void* const srcBuffer = malloc(loadedSize);
-    assert(srcBuffer != NULL);
-
-    assert(nbFiles > 0);
-    size_t* const fileSizes = (size_t*)calloc(nbFiles, sizeof(*fileSizes));
-    assert(fileSizes != NULL);
-
-    /* Load input buffer */
-    int const errorCode = loadFiles(srcBuffer, loadedSize,
-                                    fileSizes,
-                                    fileNamesTable, nbFiles);
-    assert(errorCode == 0);
-
-    void** sliceTable = (void**)malloc(nbFiles * sizeof(*sliceTable));
-    assert(sliceTable != NULL);
-
-    char* const ptr = (char*)srcBuffer;
-    size_t pos = 0;
-    unsigned fileNb = 0;
-    for ( ; (pos < loadedSize) && (fileNb < nbFiles); fileNb++) {
-        sliceTable[fileNb] = ptr + pos;
-        pos += fileSizes[fileNb];
-    }
-    assert(pos == loadedSize);
-    assert(fileNb == nbFiles);
-
-
-    buffer_t buffer;
-    buffer.ptr = srcBuffer;
-    buffer.capacity = loadedSize;
-    buffer.size = loadedSize;
-
-    slice_collection_t slices;
-    slices.slicePtrs = sliceTable;
-    slices.capacities = fileSizes;
-    slices.nbSlices = nbFiles;
-
-    buffer_collection_t bc;
-    bc.buffer = buffer;
-    bc.slices = slices;
-    return bc;
-}
-
-
-
-
-/*---  ddict_collection_t  ---*/
-
-typedef struct {
-    ZSTD_DDict** ddicts;
-    size_t nbDDict;
-} ddict_collection_t;
-
-static const ddict_collection_t kNullDDictCollection = { NULL, 0 };
-
-static void freeDDictCollection(ddict_collection_t ddictc)
-{
-    for (size_t dictNb=0; dictNb < ddictc.nbDDict; dictNb++) {
-        ZSTD_freeDDict(ddictc.ddicts[dictNb]);
-    }
-    free(ddictc.ddicts);
-}
-
-/* returns .buffers=NULL if operation fails */
-static ddict_collection_t createDDictCollection(const void* dictBuffer, size_t dictSize, size_t nbDDict)
-{
-    ZSTD_DDict** const ddicts = malloc(nbDDict * sizeof(ZSTD_DDict*));
-    assert(ddicts != NULL);
-    if (ddicts==NULL) return kNullDDictCollection;
-    for (size_t dictNb=0; dictNb < nbDDict; dictNb++) {
-        ddicts[dictNb] = ZSTD_createDDict(dictBuffer, dictSize);
-        assert(ddicts[dictNb] != NULL);
-    }
-    ddict_collection_t ddictc;
-    ddictc.ddicts = ddicts;
-    ddictc.nbDDict = nbDDict;
-    return ddictc;
-}
-
-
-/* mess with addresses, so that linear scanning dictionaries != linear address scanning */
-void shuffleDictionaries(ddict_collection_t dicts)
-{
-    size_t const nbDicts = dicts.nbDDict;
-    for (size_t r=0; r<nbDicts; r++) {
-        size_t const d = rand() % nbDicts;
-        ZSTD_DDict* tmpd = dicts.ddicts[d];
-        dicts.ddicts[d] = dicts.ddicts[r];
-        dicts.ddicts[r] = tmpd;
-    }
-    for (size_t r=0; r<nbDicts; r++) {
-        size_t const d1 = rand() % nbDicts;
-        size_t const d2 = rand() % nbDicts;
-        ZSTD_DDict* tmpd = dicts.ddicts[d1];
-        dicts.ddicts[d1] = dicts.ddicts[d2];
-        dicts.ddicts[d2] = tmpd;
-    }
-}
-
-
-/* ---   Compression  --- */
-
-/* compressBlocks() :
- * @return : total compressed size of all blocks,
- *        or 0 if error.
- */
-static size_t compressBlocks(size_t* cSizes,   /* optional (can be NULL). If present, must contain at least nbBlocks fields */
-                             slice_collection_t dstBlockBuffers,
-                             slice_collection_t srcBlockBuffers,
-                             ZSTD_CDict* cdict, int cLevel)
-{
-    size_t const nbBlocks = srcBlockBuffers.nbSlices;
-    assert(dstBlockBuffers.nbSlices == srcBlockBuffers.nbSlices);
-
-    ZSTD_CCtx* const cctx = ZSTD_createCCtx();
-    assert(cctx != NULL);
-
-    size_t totalCSize = 0;
-    for (size_t blockNb=0; blockNb < nbBlocks; blockNb++) {
-        size_t cBlockSize;
-        if (cdict == NULL) {
-            cBlockSize = ZSTD_compressCCtx(cctx,
-                            dstBlockBuffers.slicePtrs[blockNb], dstBlockBuffers.capacities[blockNb],
-                            srcBlockBuffers.slicePtrs[blockNb], srcBlockBuffers.capacities[blockNb],
-                            cLevel);
-        } else {
-            cBlockSize = ZSTD_compress_usingCDict(cctx,
-                            dstBlockBuffers.slicePtrs[blockNb], dstBlockBuffers.capacities[blockNb],
-                            srcBlockBuffers.slicePtrs[blockNb], srcBlockBuffers.capacities[blockNb],
-                            cdict);
-        }
-        CONTROL(!ZSTD_isError(cBlockSize));
-        if (cSizes) cSizes[blockNb] = cBlockSize;
-        totalCSize += cBlockSize;
-    }
-    return totalCSize;
-}
-
-
-/* ---  Benchmark  --- */
-
-typedef struct {
-    ZSTD_DCtx* dctx;
-    size_t nbDicts;
-    size_t dictNb;
-    ddict_collection_t dictionaries;
-} decompressInstructions;
-
-decompressInstructions createDecompressInstructions(ddict_collection_t dictionaries)
-{
-    decompressInstructions di;
-    di.dctx = ZSTD_createDCtx();
-    assert(di.dctx != NULL);
-    di.nbDicts = dictionaries.nbDDict;
-    di.dictNb = 0;
-    di.dictionaries = dictionaries;
-    return di;
-}
-
-void freeDecompressInstructions(decompressInstructions di)
-{
-    ZSTD_freeDCtx(di.dctx);
-}
-
-/* benched function */
-size_t decompress(const void* src, size_t srcSize, void* dst, size_t dstCapacity, void* payload)
-{
-    decompressInstructions* const di = (decompressInstructions*) payload;
-
-    size_t const result = ZSTD_decompress_usingDDict(di->dctx,
-                                        dst, dstCapacity,
-                                        src, srcSize,
-                                        di->dictionaries.ddicts[di->dictNb]);
-
-    di->dictNb = di->dictNb + 1;
-    if (di->dictNb >= di->nbDicts) di->dictNb = 0;
-
-    return result;
-}
-
-
-static int benchMem(slice_collection_t dstBlocks,
-                    slice_collection_t srcBlocks,
-                    ddict_collection_t dictionaries,
-                    int nbRounds)
-{
-    assert(dstBlocks.nbSlices == srcBlocks.nbSlices);
-
-    unsigned const ms_per_round = RUN_TIME_DEFAULT_MS;
-    unsigned const total_time_ms = nbRounds * ms_per_round;
-
-    double bestSpeed = 0.;
-
-    BMK_timedFnState_t* const benchState =
-            BMK_createTimedFnState(total_time_ms, ms_per_round);
-    decompressInstructions di = createDecompressInstructions(dictionaries);
-    BMK_benchParams_t const bp = {
-        .benchFn = decompress,
-        .benchPayload = &di,
-        .initFn = NULL,
-        .initPayload = NULL,
-        .errorFn = ZSTD_isError,
-        .blockCount = dstBlocks.nbSlices,
-        .srcBuffers = (const void* const*) srcBlocks.slicePtrs,
-        .srcSizes = srcBlocks.capacities,
-        .dstBuffers = dstBlocks.slicePtrs,
-        .dstCapacities = dstBlocks.capacities,
-        .blockResults = NULL
-    };
-
-    for (;;) {
-        BMK_runOutcome_t const outcome = BMK_benchTimedFn(benchState, bp);
-        CONTROL(BMK_isSuccessful_runOutcome(outcome));
-
-        BMK_runTime_t const result = BMK_extract_runTime(outcome);
-        double const dTime_ns = result.nanoSecPerRun;
-        double const dTime_sec = (double)dTime_ns / 1000000000;
-        size_t const srcSize = result.sumOfReturn;
-        double const dSpeed_MBps = (double)srcSize / dTime_sec / (1 MB);
-        if (dSpeed_MBps > bestSpeed) bestSpeed = dSpeed_MBps;
-        DISPLAY("Decompression Speed : %.1f MB/s \r", bestSpeed);
-        fflush(stdout);
-        if (BMK_isCompleted_TimedFn(benchState)) break;
-    }
-    DISPLAY("\n");
-
-    freeDecompressInstructions(di);
-    BMK_freeTimedFnState(benchState);
-
-    return 0;   /* success */
-}
-
-
-/*! bench() :
- *  fileName : file to load for benchmarking purpose
- *  dictionary : optional (can be NULL), file to load as dictionary,
- *              if none provided : will be calculated on the fly by the program.
- * @return : 0 is success, 1+ otherwise */
-int bench(const char** fileNameTable, unsigned nbFiles,
-          const char* dictionary,
-          size_t blockSize, int clevel,
-          unsigned nbDictMax, unsigned nbBlocks,
-          int nbRounds)
-{
-    int result = 0;
-
-    DISPLAYLEVEL(3, "loading %u files... \n", nbFiles);
-    buffer_collection_t const srcs = createBufferCollection_fromFiles(fileNameTable, nbFiles);
-    CONTROL(srcs.buffer.ptr != NULL);
-    buffer_t srcBuffer = srcs.buffer;
-    size_t const srcSize = srcBuffer.size;
-    DISPLAYLEVEL(3, "created src buffer of size %.1f MB \n",
-                    (double)srcSize / (1 MB));
-
-    slice_collection_t const srcSlices = splitSlices(srcs.slices, blockSize, nbBlocks);
-    nbBlocks = (unsigned)(srcSlices.nbSlices);
-    DISPLAYLEVEL(3, "split input into %u blocks ", nbBlocks);
-    if (blockSize)
-        DISPLAYLEVEL(3, "of max size %u bytes ", (unsigned)blockSize);
-    DISPLAYLEVEL(3, "\n");
-    size_t const totalSrcSlicesSize = sliceCollection_totalCapacity(srcSlices);
-
-
-    size_t* const dstCapacities = malloc(nbBlocks * sizeof(*dstCapacities));
-    CONTROL(dstCapacities != NULL);
-    size_t dstBufferCapacity = 0;
-    for (size_t bnb=0; bnb<nbBlocks; bnb++) {
-        dstCapacities[bnb] = ZSTD_compressBound(srcSlices.capacities[bnb]);
-        dstBufferCapacity += dstCapacities[bnb];
-    }
-
-    buffer_t dstBuffer = createBuffer(dstBufferCapacity);
-    CONTROL(dstBuffer.ptr != NULL);
-
-    void** const sliceTable = malloc(nbBlocks * sizeof(*sliceTable));
-    CONTROL(sliceTable != NULL);
-
-    {   char* const ptr = dstBuffer.ptr;
-        size_t pos = 0;
-        for (size_t snb=0; snb < nbBlocks; snb++) {
-            sliceTable[snb] = ptr + pos;
-            pos += dstCapacities[snb];
-    }   }
-
-    slice_collection_t dstSlices;
-    dstSlices.capacities = dstCapacities;
-    dstSlices.slicePtrs = sliceTable;
-    dstSlices.nbSlices = nbBlocks;
-
-
-    /* dictionary determination */
-    buffer_t const dictBuffer = createDictionaryBuffer(dictionary,
-                                srcs.buffer.ptr,
-                                srcs.slices.capacities, srcs.slices.nbSlices,
-                                DICTSIZE);
-    CONTROL(dictBuffer.ptr != NULL);
-
-    ZSTD_CDict* const cdict = ZSTD_createCDict(dictBuffer.ptr, dictBuffer.size, clevel);
-    CONTROL(cdict != NULL);
-
-    size_t const cTotalSizeNoDict = compressBlocks(NULL, dstSlices, srcSlices, NULL, clevel);
-    CONTROL(cTotalSizeNoDict != 0);
-    DISPLAYLEVEL(3, "compressing at level %u without dictionary : Ratio=%.2f  (%u bytes) \n",
-                    clevel,
-                    (double)totalSrcSlicesSize / cTotalSizeNoDict, (unsigned)cTotalSizeNoDict);
-
-    size_t* const cSizes = malloc(nbBlocks * sizeof(size_t));
-    CONTROL(cSizes != NULL);
-
-    size_t const cTotalSize = compressBlocks(cSizes, dstSlices, srcSlices, cdict, clevel);
-    CONTROL(cTotalSize != 0);
-    DISPLAYLEVEL(3, "compressed using a %u bytes dictionary : Ratio=%.2f  (%u bytes) \n",
-                    (unsigned)dictBuffer.size,
-                    (double)totalSrcSlicesSize / cTotalSize, (unsigned)cTotalSize);
-
-    /* now dstSlices contain the real compressed size of each block, instead of the maximum capacity */
-    shrinkSizes(dstSlices, cSizes);
-
-    size_t const dictMem = ZSTD_estimateDDictSize(dictBuffer.size, ZSTD_dlm_byCopy);
-    unsigned const nbDicts = nbDictMax ? nbDictMax : nbBlocks;
-    size_t const allDictMem = dictMem * nbDicts;
-    DISPLAYLEVEL(3, "generating %u dictionaries, using %.1f MB of memory \n",
-                    nbDicts, (double)allDictMem / (1 MB));
-
-    ddict_collection_t const dictionaries = createDDictCollection(dictBuffer.ptr, dictBuffer.size, nbDicts);
-    CONTROL(dictionaries.ddicts != NULL);
-
-    shuffleDictionaries(dictionaries);
-
-    buffer_collection_t resultCollection = createBufferCollection_fromSliceCollectionSizes(srcSlices);
-    CONTROL(resultCollection.buffer.ptr != NULL);
-
-    result = benchMem(resultCollection.slices, dstSlices, dictionaries, nbRounds);
-
-    /* free all heap objects in reverse order */
-    freeBufferCollection(resultCollection);
-    freeDDictCollection(dictionaries);
-    free(cSizes);
-    ZSTD_freeCDict(cdict);
-    freeBuffer(dictBuffer);
-    freeSliceCollection(dstSlices);
-    freeBuffer(dstBuffer);
-    freeSliceCollection(srcSlices);
-    freeBufferCollection(srcs);
-
-    return result;
-}
-
-
-
-/* ---  Command Line  --- */
-
-/*! readU32FromChar() :
- * @return : unsigned integer value read from input in `char` format.
- *  allows and interprets K, KB, KiB, M, MB and MiB suffix.
- *  Will also modify `*stringPtr`, advancing it to position where it stopped reading.
- *  Note : function will exit() program if digit sequence overflows */
-static unsigned readU32FromChar(const char** stringPtr)
-{
-    unsigned result = 0;
-    while ((**stringPtr >='0') && (**stringPtr <='9')) {
-        unsigned const max = (((unsigned)(-1)) / 10) - 1;
-        assert(result <= max);   /* check overflow */
-        result *= 10, result += **stringPtr - '0', (*stringPtr)++ ;
-    }
-    if ((**stringPtr=='K') || (**stringPtr=='M')) {
-        unsigned const maxK = ((unsigned)(-1)) >> 10;
-        assert(result <= maxK);   /* check overflow */
-        result <<= 10;
-        if (**stringPtr=='M') {
-            assert(result <= maxK);   /* check overflow */
-            result <<= 10;
-        }
-        (*stringPtr)++;  /* skip `K` or `M` */
-        if (**stringPtr=='i') (*stringPtr)++;
-        if (**stringPtr=='B') (*stringPtr)++;
-    }
-    return result;
-}
-
-/** longCommandWArg() :
- *  check if *stringPtr is the same as longCommand.
- *  If yes, @return 1 and advances *stringPtr to the position which immediately follows longCommand.
- * @return 0 and doesn't modify *stringPtr otherwise.
- */
-static unsigned longCommandWArg(const char** stringPtr, const char* longCommand)
-{
-    size_t const comSize = strlen(longCommand);
-    int const result = !strncmp(*stringPtr, longCommand, comSize);
-    if (result) *stringPtr += comSize;
-    return result;
-}
-
-
-int usage(const char* exeName)
-{
-    DISPLAY (" \n");
-    DISPLAY (" %s [Options] filename(s) \n", exeName);
-    DISPLAY (" \n");
-    DISPLAY ("Options : \n");
-    DISPLAY ("-r          : recursively load all files in subdirectories (default: off) \n");
-    DISPLAY ("-B#         : split input into blocks of size # (default: no split) \n");
-    DISPLAY ("-#          : use compression level # (default: %u) \n", CLEVEL_DEFAULT);
-    DISPLAY ("-D #        : use # as a dictionary (default: create one) \n");
-    DISPLAY ("-i#         : nb benchmark rounds (default: %u) \n", BENCH_TIME_DEFAULT_S);
-    DISPLAY ("--nbBlocks=#: use # blocks for bench (default: one per file) \n");
-    DISPLAY ("--nbDicts=# : create # dictionaries for bench (default: one per block) \n");
-    DISPLAY ("-h          : help (this text) \n");
-    return 0;
-}
-
-int bad_usage(const char* exeName)
-{
-    DISPLAY (" bad usage : \n");
-    usage(exeName);
-    return 1;
-}
-
-int main (int argc, const char** argv)
-{
-    int recursiveMode = 0;
-    int nbRounds = BENCH_TIME_DEFAULT_S;
-    const char* const exeName = argv[0];
-
-    if (argc < 2) return bad_usage(exeName);
-
-    const char** nameTable = (const char**)malloc(argc * sizeof(const char*));
-    assert(nameTable != NULL);
-    unsigned nameIdx = 0;
-
-    const char* dictionary = NULL;
-    int cLevel = CLEVEL_DEFAULT;
-    size_t blockSize = BLOCKSIZE_DEFAULT;
-    unsigned nbDicts = 0;  /* determine nbDicts automatically: 1 dictionary per block */
-    unsigned nbBlocks = 0; /* determine nbBlocks automatically, from source and blockSize */
-
-    for (int argNb = 1; argNb < argc ; argNb++) {
-        const char* argument = argv[argNb];
-        if (!strcmp(argument, "-h")) { free(nameTable); return usage(exeName); }
-        if (!strcmp(argument, "-r")) { recursiveMode = 1; continue; }
-        if (!strcmp(argument, "-D")) { argNb++; assert(argNb < argc); dictionary = argv[argNb]; continue; }
-        if (longCommandWArg(&argument, "-i")) { nbRounds = readU32FromChar(&argument); continue; }
-        if (longCommandWArg(&argument, "--dictionary=")) { dictionary = argument; continue; }
-        if (longCommandWArg(&argument, "-B")) { blockSize = readU32FromChar(&argument); continue; }
-        if (longCommandWArg(&argument, "--blockSize=")) { blockSize = readU32FromChar(&argument); continue; }
-        if (longCommandWArg(&argument, "--nbDicts=")) { nbDicts = readU32FromChar(&argument); continue; }
-        if (longCommandWArg(&argument, "--nbBlocks=")) { nbBlocks = readU32FromChar(&argument); continue; }
-        if (longCommandWArg(&argument, "--clevel=")) { cLevel = readU32FromChar(&argument); continue; }
-        if (longCommandWArg(&argument, "-")) { cLevel = readU32FromChar(&argument); continue; }
-        /* anything that's not a command is a filename */
-        nameTable[nameIdx++] = argument;
-    }
-
-    const char** filenameTable = nameTable;
-    unsigned nbFiles = nameIdx;
-    char* buffer_containing_filenames = NULL;
-
-    if (recursiveMode) {
-#ifndef UTIL_HAS_CREATEFILELIST
-        assert(0);   /* missing capability, do not run */
-#endif
-        filenameTable = UTIL_createFileList(nameTable, nameIdx, &buffer_containing_filenames, &nbFiles, 1 /* follow_links */);
-    }
-
-    int result = bench(filenameTable, nbFiles, dictionary, blockSize, cLevel, nbDicts, nbBlocks, nbRounds);
-
-    free(buffer_containing_filenames);
-    free(nameTable);
-
-    return result;
-}
diff --git a/contrib/premake/premake4.lua b/contrib/premake/premake4.lua
deleted file mode 100644
index 6675e2e481c1..000000000000
--- a/contrib/premake/premake4.lua
+++ /dev/null
@@ -1,6 +0,0 @@
--- Include zstd.lua in your GENie or premake4 file, which exposes a project_zstd function
-dofile('zstd.lua')
-
-solution 'example'
-	configurations { 'Debug', 'Release' }
-	project_zstd('../../lib/')
diff --git a/contrib/premake/zstd.lua b/contrib/premake/zstd.lua
deleted file mode 100644
index df1ace3ee8ea..000000000000
--- a/contrib/premake/zstd.lua
+++ /dev/null
@@ -1,80 +0,0 @@
--- This GENie/premake file copies the behavior of the Makefile in the lib folder.
--- Basic usage: project_zstd(ZSTD_DIR)
-
-function project_zstd(dir, compression, decompression, deprecated, dictbuilder, legacy)
-	if compression == nil then compression = true end
-	if decompression == nil then decompression = true end
-	if deprecated == nil then deprecated = false end
-	if dictbuilder == nil then dictbuilder = false end
-
-	if legacy == nil then legacy = 0 end
-
-	if not compression then
-		dictbuilder = false
-		deprecated = false
-	end
-
-	if not decompression then
-		legacy = 0
-		deprecated = false
-	end
-
-	project 'zstd'
-		kind 'StaticLib'
-		language 'C'
-
-		files {
-			dir .. 'zstd.h',
-			dir .. 'common/**.c',
-			dir .. 'common/**.h'
-		}
-
-		if compression then
-			files {
-				dir .. 'compress/**.c',
-				dir .. 'compress/**.h'
-			}
-		end
-
-		if decompression then
-			files {
-				dir .. 'decompress/**.c',
-				dir .. 'decompress/**.h'
-			}
-		end
-
-		if dictbuilder then
-			files {
-				dir .. 'dictBuilder/**.c',
-				dir .. 'dictBuilder/**.h'
-			}
-		end
-
-		if deprecated then
-			files {
-				dir .. 'deprecated/**.c',
-				dir .. 'deprecated/**.h'
-			}
-		end
-
-		if legacy ~= 0 then
-			if legacy >= 8 then
-				files {
-					dir .. 'legacy/zstd_v0' .. (legacy - 7) .. '.*'
-				}
-			end
-			includedirs {
-				dir .. 'legacy'
-			}
-		end
-
-		includedirs {
-			dir,
-			dir .. 'common'
-		}
-
-		defines {
-			'XXH_NAMESPACE=ZSTD_',
-			'ZSTD_LEGACY_SUPPORT=' .. legacy
-		}
-end
diff --git a/contrib/pzstd/BUCK b/contrib/pzstd/BUCK
deleted file mode 100644
index d04eeedd8a1a..000000000000
--- a/contrib/pzstd/BUCK
+++ /dev/null
@@ -1,72 +0,0 @@
-cxx_library(
-    name='libpzstd',
-    visibility=['PUBLIC'],
-    header_namespace='',
-    exported_headers=[
-        'ErrorHolder.h',
-        'Logging.h',
-        'Pzstd.h',
-    ],
-    headers=[
-        'SkippableFrame.h',
-    ],
-    srcs=[
-        'Pzstd.cpp',
-        'SkippableFrame.cpp',
-    ],
-    deps=[
-        ':options',
-        '//contrib/pzstd/utils:utils',
-        '//lib:mem',
-        '//lib:zstd',
-    ],
-)
-
-cxx_library(
-    name='options',
-    visibility=['PUBLIC'],
-    header_namespace='',
-    exported_headers=['Options.h'],
-    srcs=['Options.cpp'],
-    deps=[
-        '//contrib/pzstd/utils:scope_guard',
-        '//lib:zstd',
-        '//programs:util',
-    ],
-)
-
-cxx_binary(
-    name='pzstd',
-    visibility=['PUBLIC'],
-    srcs=['main.cpp'],
-    deps=[
-        ':libpzstd',
-        ':options',
-    ],
-)
-
-# Must run "make googletest" first
-cxx_library(
-    name='gtest',
-    srcs=glob([
-        'googletest/googletest/src/gtest-all.cc',
-        'googletest/googlemock/src/gmock-all.cc',
-        'googletest/googlemock/src/gmock_main.cc',
-    ]),
-    header_namespace='',
-    exported_headers=subdir_glob([
-        ('googletest/googletest/include', '**/*.h'),
-        ('googletest/googlemock/include', '**/*.h'),
-    ]),
-    headers=subdir_glob([
-        ('googletest/googletest', 'src/*.cc'),
-        ('googletest/googletest', 'src/*.h'),
-        ('googletest/googlemock', 'src/*.cc'),
-        ('googletest/googlemock', 'src/*.h'),
-    ]),
-    platform_linker_flags=[
-        ('android', []),
-        ('', ['-lpthread']),
-    ],
-    visibility=['PUBLIC'],
-)
diff --git a/contrib/pzstd/ErrorHolder.h b/contrib/pzstd/ErrorHolder.h
deleted file mode 100644
index 829651c5961e..000000000000
--- a/contrib/pzstd/ErrorHolder.h
+++ /dev/null
@@ -1,54 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-#pragma once
-
-#include <atomic>
-#include <cassert>
-#include <stdexcept>
-#include <string>
-
-namespace pzstd {
-
-// Coordinates graceful shutdown of the pzstd pipeline
-class ErrorHolder {
-  std::atomic<bool> error_;
-  std::string message_;
-
- public:
-  ErrorHolder() : error_(false) {}
-
-  bool hasError() noexcept {
-    return error_.load();
-  }
-
-  void setError(std::string message) noexcept {
-    // Given multiple possibly concurrent calls, exactly one will ever succeed.
-    bool expected = false;
-    if (error_.compare_exchange_strong(expected, true)) {
-      message_ = std::move(message);
-    }
-  }
-
-  bool check(bool predicate, std::string message) noexcept {
-    if (!predicate) {
-      setError(std::move(message));
-    }
-    return !hasError();
-  }
-
-  std::string getError() noexcept {
-    error_.store(false);
-    return std::move(message_);
-  }
-
-  ~ErrorHolder() {
-    assert(!hasError());
-  }
-};
-}
diff --git a/contrib/pzstd/Logging.h b/contrib/pzstd/Logging.h
deleted file mode 100644
index 16a63932c0a3..000000000000
--- a/contrib/pzstd/Logging.h
+++ /dev/null
@@ -1,72 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-#pragma once
-
-#include <cstdio>
-#include <mutex>
-
-namespace pzstd {
-
-constexpr int ERROR = 1;
-constexpr int INFO = 2;
-constexpr int DEBUG = 3;
-constexpr int VERBOSE = 4;
-
-class Logger {
-  std::mutex mutex_;
-  FILE* out_;
-  const int level_;
-
-  using Clock = std::chrono::system_clock;
-  Clock::time_point lastUpdate_;
-  std::chrono::milliseconds refreshRate_;
-
- public:
-  explicit Logger(int level, FILE* out = stderr)
-      : out_(out), level_(level), lastUpdate_(Clock::now()),
-        refreshRate_(150) {}
-
-
-  bool logsAt(int level) {
-    return level <= level_;
-  }
-
-  template <typename... Args>
-  void operator()(int level, const char *fmt, Args... args) {
-    if (level > level_) {
-      return;
-    }
-    std::lock_guard<std::mutex> lock(mutex_);
-    std::fprintf(out_, fmt, args...);
-  }
-
-  template <typename... Args>
-  void update(int level, const char *fmt, Args... args) {
-    if (level > level_) {
-      return;
-    }
-    std::lock_guard<std::mutex> lock(mutex_);
-    auto now = Clock::now();
-    if (now - lastUpdate_ > refreshRate_) {
-      lastUpdate_ = now;
-      std::fprintf(out_, "\r");
-      std::fprintf(out_, fmt, args...);
-    }
-  }
-
-  void clear(int level) {
-    if (level > level_) {
-      return;
-    }
-    std::lock_guard<std::mutex> lock(mutex_);
-    std::fprintf(out_, "\r%79s\r", "");
-  }
-};
-
-}
diff --git a/contrib/pzstd/Makefile b/contrib/pzstd/Makefile
deleted file mode 100644
index 8d2b1932e91c..000000000000
--- a/contrib/pzstd/Makefile
+++ /dev/null
@@ -1,271 +0,0 @@
-# ################################################################
-# Copyright (c) 2016-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under both the BSD-style license (found in the
-# LICENSE file in the root directory of this source tree) and the GPLv2 (found
-# in the COPYING file in the root directory of this source tree).
-# ################################################################
-
-# Standard variables for installation
-DESTDIR ?=
-PREFIX  ?= /usr/local
-BINDIR  := $(DESTDIR)$(PREFIX)/bin
-
-ZSTDDIR = ../../lib
-PROGDIR = ../../programs
-
-# External program to use to run tests, e.g. qemu or valgrind
-TESTPROG  ?=
-# Flags to pass to the tests
-TESTFLAGS ?=
-
-# We use gcc/clang to generate the header dependencies of files
-DEPFLAGS = -MMD -MP -MF $*.Td
-POSTCOMPILE = mv -f $*.Td $*.d
-
-# CFLAGS, CXXFLAGS, CPPFLAGS, and LDFLAGS are for the users to override
-CFLAGS   ?= -O3 -Wall -Wextra
-CXXFLAGS ?= -O3 -Wall -Wextra -pedantic
-CPPFLAGS ?=
-LDFLAGS  ?=
-
-# Include flags
-PZSTD_INC  = -I$(ZSTDDIR) -I$(ZSTDDIR)/common -I$(PROGDIR) -I.
-GTEST_INC  = -isystem googletest/googletest/include
-
-PZSTD_CPPFLAGS  = $(PZSTD_INC)
-PZSTD_CCXXFLAGS =
-PZSTD_CFLAGS    = $(PZSTD_CCXXFLAGS)
-PZSTD_CXXFLAGS  = $(PZSTD_CCXXFLAGS) -std=c++11
-PZSTD_LDFLAGS   =
-EXTRA_FLAGS     =
-ALL_CFLAGS      = $(EXTRA_FLAGS) $(CPPFLAGS) $(PZSTD_CPPFLAGS) $(CFLAGS)   $(PZSTD_CFLAGS)
-ALL_CXXFLAGS    = $(EXTRA_FLAGS) $(CPPFLAGS) $(PZSTD_CPPFLAGS) $(CXXFLAGS) $(PZSTD_CXXFLAGS)
-ALL_LDFLAGS     = $(EXTRA_FLAGS) $(CXXFLAGS) $(LDFLAGS) $(PZSTD_LDFLAGS)
-
-
-# gtest libraries need to go before "-lpthread" because they depend on it.
-GTEST_LIB  = -L googletest/build/googlemock/gtest
-LIBS       =
-
-# Compilation commands
-LD_COMMAND  = $(CXX) $^          $(ALL_LDFLAGS) $(LIBS) -pthread -o $@
-CC_COMMAND  = $(CC)  $(DEPFLAGS) $(ALL_CFLAGS)   -c $<  -o $@
-CXX_COMMAND = $(CXX) $(DEPFLAGS) $(ALL_CXXFLAGS) -c $<  -o $@
-
-# Get a list of all zstd files so we rebuild the static library when we need to
-ZSTDCOMMON_FILES := $(wildcard $(ZSTDDIR)/common/*.c) \
-                    $(wildcard $(ZSTDDIR)/common/*.h)
-ZSTDCOMP_FILES   := $(wildcard $(ZSTDDIR)/compress/*.c) \
-                    $(wildcard $(ZSTDDIR)/compress/*.h)
-ZSTDDECOMP_FILES := $(wildcard $(ZSTDDIR)/decompress/*.c) \
-                    $(wildcard $(ZSTDDIR)/decompress/*.h)
-ZSTDPROG_FILES   := $(wildcard $(PROGDIR)/*.c) \
-                    $(wildcard $(PROGDIR)/*.h)
-ZSTD_FILES       := $(wildcard $(ZSTDDIR)/*.h) \
-                    $(ZSTDDECOMP_FILES) $(ZSTDCOMMON_FILES) $(ZSTDCOMP_FILES) \
-                    $(ZSTDPROG_FILES)
-
-# List all the pzstd source files so we can determine their dependencies
-PZSTD_SRCS  := $(wildcard *.cpp)
-PZSTD_TESTS := $(wildcard test/*.cpp)
-UTILS_TESTS := $(wildcard utils/test/*.cpp)
-ALL_SRCS    := $(PZSTD_SRCS) $(PZSTD_TESTS) $(UTILS_TESTS)
-
-
-# Define *.exe as extension for Windows systems
-ifneq (,$(filter Windows%,$(OS)))
-EXT =.exe
-else
-EXT =
-endif
-
-# Standard targets
-.PHONY: default
-default: all
-
-.PHONY: test-pzstd
-test-pzstd: TESTFLAGS=--gtest_filter=-*ExtremelyLarge*
-test-pzstd: clean googletest pzstd tests check
-
-.PHONY: test-pzstd32
-test-pzstd32: clean googletest32 all32 check
-
-.PHONY: test-pzstd-tsan
-test-pzstd-tsan: LDFLAGS=-fuse-ld=gold
-test-pzstd-tsan: TESTFLAGS=--gtest_filter=-*ExtremelyLarge*
-test-pzstd-tsan: clean googletest tsan check
-
-.PHONY: test-pzstd-asan
-test-pzstd-asan: LDFLAGS=-fuse-ld=gold
-test-pzstd-asan: TESTFLAGS=--gtest_filter=-*ExtremelyLarge*
-test-pzstd-asan: clean asan check
-
-.PHONY: check
-check:
-	$(TESTPROG) ./utils/test/BufferTest$(EXT) $(TESTFLAGS)
-	$(TESTPROG) ./utils/test/RangeTest$(EXT) $(TESTFLAGS)
-	$(TESTPROG) ./utils/test/ResourcePoolTest$(EXT) $(TESTFLAGS)
-	$(TESTPROG) ./utils/test/ScopeGuardTest$(EXT) $(TESTFLAGS)
-	$(TESTPROG) ./utils/test/ThreadPoolTest$(EXT) $(TESTFLAGS)
-	$(TESTPROG) ./utils/test/WorkQueueTest$(EXT) $(TESTFLAGS)
-	$(TESTPROG) ./test/OptionsTest$(EXT) $(TESTFLAGS)
-	$(TESTPROG) ./test/PzstdTest$(EXT) $(TESTFLAGS)
-
-.PHONY: install
-install: PZSTD_CPPFLAGS += -DNDEBUG
-install: pzstd$(EXT)
-	install -d -m 755 $(BINDIR)/
-	install -m 755 pzstd$(EXT) $(BINDIR)/pzstd$(EXT)
-
-.PHONY: uninstall
-uninstall:
-	$(RM) $(BINDIR)/pzstd$(EXT)
-
-# Targets for many different builds
-.PHONY: all
-all: PZSTD_CPPFLAGS += -DNDEBUG
-all: pzstd$(EXT)
-
-.PHONY: debug
-debug: EXTRA_FLAGS += -g
-debug: pzstd$(EXT) tests roundtrip
-
-.PHONY: tsan
-tsan: PZSTD_CCXXFLAGS += -fsanitize=thread -fPIC
-tsan: PZSTD_LDFLAGS   += -fsanitize=thread
-tsan: debug
-
-.PHONY: asan
-asan: EXTRA_FLAGS += -fsanitize=address
-asan: debug
-
-.PHONY: ubsan
-ubsan: EXTRA_FLAGS += -fsanitize=undefined
-ubsan: debug
-
-.PHONY: all32
-all32: EXTRA_FLAGS += -m32
-all32: all tests roundtrip
-
-.PHONY: debug32
-debug32: EXTRA_FLAGS += -m32
-debug32: debug
-
-.PHONY: asan32
-asan32: EXTRA_FLAGS += -m32
-asan32: asan
-
-.PHONY: tsan32
-tsan32: EXTRA_FLAGS += -m32
-tsan32: tsan
-
-.PHONY: ubsan32
-ubsan32: EXTRA_FLAGS += -m32
-ubsan32: ubsan
-
-# Run long round trip tests
-.PHONY: roundtripcheck
-roundtripcheck: roundtrip check
-	$(TESTPROG) ./test/RoundTripTest$(EXT) $(TESTFLAGS)
-
-# Build the main binary
-pzstd$(EXT): main.o $(PROGDIR)/util.o Options.o Pzstd.o SkippableFrame.o $(ZSTDDIR)/libzstd.a
-	$(LD_COMMAND)
-
-# Target that depends on all the tests
-.PHONY: tests
-tests: EXTRA_FLAGS += -Wno-deprecated-declarations
-tests: $(patsubst %,%$(EXT),$(basename $(PZSTD_TESTS) $(UTILS_TESTS)))
-
-# Build the round trip tests
-.PHONY: roundtrip
-roundtrip: EXTRA_FLAGS += -Wno-deprecated-declarations
-roundtrip: test/RoundTripTest$(EXT)
-
-# Use the static library that zstd builds for simplicity and
-# so we get the compiler options correct
-$(ZSTDDIR)/libzstd.a: $(ZSTD_FILES)
-	CFLAGS="$(ALL_CFLAGS)" LDFLAGS="$(ALL_LDFLAGS)" $(MAKE) -C $(ZSTDDIR) libzstd.a
-
-# Rules to build the tests
-test/RoundTripTest$(EXT): test/RoundTripTest.o $(PROGDIR)/datagen.o \
-                          $(PROGDIR)/util.o Options.o \
-                          Pzstd.o SkippableFrame.o $(ZSTDDIR)/libzstd.a
-	$(LD_COMMAND)
-
-test/%Test$(EXT): PZSTD_LDFLAGS += $(GTEST_LIB)
-test/%Test$(EXT): LIBS += -lgtest -lgtest_main
-test/%Test$(EXT): test/%Test.o $(PROGDIR)/datagen.o \
-                  $(PROGDIR)/util.o Options.o Pzstd.o \
-                  SkippableFrame.o $(ZSTDDIR)/libzstd.a
-	$(LD_COMMAND)
-
-utils/test/%Test$(EXT): PZSTD_LDFLAGS += $(GTEST_LIB)
-utils/test/%Test$(EXT): LIBS += -lgtest -lgtest_main
-utils/test/%Test$(EXT): utils/test/%Test.o
-	$(LD_COMMAND)
-
-
-GTEST_CMAKEFLAGS =
-
-# Install googletest
-.PHONY: googletest
-googletest: PZSTD_CCXXFLAGS += -fPIC
-googletest:
-	@$(RM) -rf googletest
-	@git clone https://github.com/google/googletest
-	@mkdir -p googletest/build
-	@cd googletest/build && cmake $(GTEST_CMAKEFLAGS) -DCMAKE_CXX_FLAGS="$(ALL_CXXFLAGS)" .. && $(MAKE)
-
-.PHONY: googletest32
-googletest32: PZSTD_CCXXFLAGS  += -m32
-googletest32: googletest
-
-.PHONY: googletest-mingw64
-googletest-mingw64: GTEST_CMAKEFLAGS += -G "MSYS Makefiles"
-googletest-mingw64: googletest
-
-.PHONY: clean
-clean:
-	$(RM) -f *.o pzstd$(EXT) *.Td *.d
-	$(RM) -f test/*.o test/*Test$(EXT) test/*.Td test/*.d
-	$(RM) -f utils/test/*.o utils/test/*Test$(EXT) utils/test/*.Td utils/test/*.d
-	$(RM) -f $(PROGDIR)/*.o $(PROGDIR)/*.Td $(PROGDIR)/*.d
-	$(MAKE) -C $(ZSTDDIR) clean
-	@echo Cleaning completed
-
-
-# Cancel implicit rules
-%.o: %.c
-%.o: %.cpp
-
-# Object file rules
-%.o: %.c
-	$(CC_COMMAND)
-	$(POSTCOMPILE)
-
-$(PROGDIR)/%.o: $(PROGDIR)/%.c
-	$(CC_COMMAND)
-	$(POSTCOMPILE)
-
-%.o: %.cpp
-	$(CXX_COMMAND)
-	$(POSTCOMPILE)
-
-test/%.o: PZSTD_CPPFLAGS += $(GTEST_INC)
-test/%.o: test/%.cpp
-	$(CXX_COMMAND)
-	$(POSTCOMPILE)
-
-utils/test/%.o: PZSTD_CPPFLAGS += $(GTEST_INC)
-utils/test/%.o: utils/test/%.cpp
-	$(CXX_COMMAND)
-	$(POSTCOMPILE)
-
-# Dependency file stuff
-.PRECIOUS: %.d test/%.d utils/test/%.d
-
-# Include rules that specify header file dependencies
--include $(patsubst %,%.d,$(basename $(ALL_SRCS)))
diff --git a/contrib/pzstd/Options.cpp b/contrib/pzstd/Options.cpp
deleted file mode 100644
index 2123f8894c3e..000000000000
--- a/contrib/pzstd/Options.cpp
+++ /dev/null
@@ -1,428 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-#include "Options.h"
-#include "util.h"
-#include "utils/ScopeGuard.h"
-
-#include <algorithm>
-#include <cassert>
-#include <cstdio>
-#include <cstring>
-#include <iterator>
-#include <thread>
-#include <vector>
-
-
-namespace pzstd {
-
-namespace {
-unsigned defaultNumThreads() {
-#ifdef PZSTD_NUM_THREADS
-  return PZSTD_NUM_THREADS;
-#else
-  return std::thread::hardware_concurrency();
-#endif
-}
-
-unsigned parseUnsigned(const char **arg) {
-  unsigned result = 0;
-  while (**arg >= '0' && **arg <= '9') {
-    result *= 10;
-    result += **arg - '0';
-    ++(*arg);
-  }
-  return result;
-}
-
-const char *getArgument(const char *options, const char **argv, int &i,
-                        int argc) {
-  if (options[1] != 0) {
-    return options + 1;
-  }
-  ++i;
-  if (i == argc) {
-    std::fprintf(stderr, "Option -%c requires an argument, but none provided\n",
-                 *options);
-    return nullptr;
-  }
-  return argv[i];
-}
-
-const std::string kZstdExtension = ".zst";
-constexpr char kStdIn[] = "-";
-constexpr char kStdOut[] = "-";
-constexpr unsigned kDefaultCompressionLevel = 3;
-constexpr unsigned kMaxNonUltraCompressionLevel = 19;
-
-#ifdef _WIN32
-const char nullOutput[] = "nul";
-#else
-const char nullOutput[] = "/dev/null";
-#endif
-
-void notSupported(const char *option) {
-  std::fprintf(stderr, "Operation not supported: %s\n", option);
-}
-
-void usage() {
-  std::fprintf(stderr, "Usage:\n");
-  std::fprintf(stderr, "  pzstd [args] [FILE(s)]\n");
-  std::fprintf(stderr, "Parallel ZSTD options:\n");
-  std::fprintf(stderr, "  -p, --processes   #    : number of threads to use for (de)compression (default:<numcpus>)\n");
-
-  std::fprintf(stderr, "ZSTD options:\n");
-  std::fprintf(stderr, "  -#                     : # compression level (1-%d, default:%d)\n", kMaxNonUltraCompressionLevel, kDefaultCompressionLevel);
-  std::fprintf(stderr, "  -d, --decompress       : decompression\n");
-  std::fprintf(stderr, "  -o                file : result stored into `file` (only if 1 input file)\n");
-  std::fprintf(stderr, "  -f, --force            : overwrite output without prompting, (de)compress links\n");
-  std::fprintf(stderr, "      --rm               : remove source file(s) after successful (de)compression\n");
-  std::fprintf(stderr, "  -k, --keep             : preserve source file(s) (default)\n");
-  std::fprintf(stderr, "  -h, --help             : display help and exit\n");
-  std::fprintf(stderr, "  -V, --version          : display version number and exit\n");
-  std::fprintf(stderr, "  -v, --verbose          : verbose mode; specify multiple times to increase log level (default:2)\n");
-  std::fprintf(stderr, "  -q, --quiet            : suppress warnings; specify twice to suppress errors too\n");
-  std::fprintf(stderr, "  -c, --stdout           : force write to standard output, even if it is the console\n");
-#ifdef UTIL_HAS_CREATEFILELIST
-  std::fprintf(stderr, "  -r                     : operate recursively on directories\n");
-#endif
-  std::fprintf(stderr, "      --ultra            : enable levels beyond %i, up to %i (requires more memory)\n", kMaxNonUltraCompressionLevel, ZSTD_maxCLevel());
-  std::fprintf(stderr, "  -C, --check            : integrity check (default)\n");
-  std::fprintf(stderr, "      --no-check         : no integrity check\n");
-  std::fprintf(stderr, "  -t, --test             : test compressed file integrity\n");
-  std::fprintf(stderr, "  --                     : all arguments after \"--\" are treated as files\n");
-}
-} // anonymous namespace
-
-Options::Options()
-    : numThreads(defaultNumThreads()), maxWindowLog(23),
-      compressionLevel(kDefaultCompressionLevel), decompress(false),
-      overwrite(false), keepSource(true), writeMode(WriteMode::Auto),
-      checksum(true), verbosity(2) {}
-
-Options::Status Options::parse(int argc, const char **argv) {
-  bool test = false;
-  bool recursive = false;
-  bool ultra = false;
-  bool forceStdout = false;
-  bool followLinks = false;
-  // Local copy of input files, which are pointers into argv.
-  std::vector<const char *> localInputFiles;
-  for (int i = 1; i < argc; ++i) {
-    const char *arg = argv[i];
-    // Protect against empty arguments
-    if (arg[0] == 0) {
-      continue;
-    }
-    // Everything after "--" is an input file
-    if (!std::strcmp(arg, "--")) {
-      ++i;
-      std::copy(argv + i, argv + argc, std::back_inserter(localInputFiles));
-      break;
-    }
-    // Long arguments that don't have a short option
-    {
-      bool isLongOption = true;
-      if (!std::strcmp(arg, "--rm")) {
-        keepSource = false;
-      } else if (!std::strcmp(arg, "--ultra")) {
-        ultra = true;
-        maxWindowLog = 0;
-      } else if (!std::strcmp(arg, "--no-check")) {
-        checksum = false;
-      } else if (!std::strcmp(arg, "--sparse")) {
-        writeMode = WriteMode::Sparse;
-        notSupported("Sparse mode");
-        return Status::Failure;
-      } else if (!std::strcmp(arg, "--no-sparse")) {
-        writeMode = WriteMode::Regular;
-        notSupported("Sparse mode");
-        return Status::Failure;
-      } else if (!std::strcmp(arg, "--dictID")) {
-        notSupported(arg);
-        return Status::Failure;
-      } else if (!std::strcmp(arg, "--no-dictID")) {
-        notSupported(arg);
-        return Status::Failure;
-      } else {
-        isLongOption = false;
-      }
-      if (isLongOption) {
-        continue;
-      }
-    }
-    // Arguments with a short option simply set their short option.
-    const char *options = nullptr;
-    if (!std::strcmp(arg, "--processes")) {
-      options = "p";
-    } else if (!std::strcmp(arg, "--version")) {
-      options = "V";
-    } else if (!std::strcmp(arg, "--help")) {
-      options = "h";
-    } else if (!std::strcmp(arg, "--decompress")) {
-      options = "d";
-    } else if (!std::strcmp(arg, "--force")) {
-      options = "f";
-    } else if (!std::strcmp(arg, "--stdout")) {
-      options = "c";
-    } else if (!std::strcmp(arg, "--keep")) {
-      options = "k";
-    } else if (!std::strcmp(arg, "--verbose")) {
-      options = "v";
-    } else if (!std::strcmp(arg, "--quiet")) {
-      options = "q";
-    } else if (!std::strcmp(arg, "--check")) {
-      options = "C";
-    } else if (!std::strcmp(arg, "--test")) {
-      options = "t";
-    } else if (arg[0] == '-' && arg[1] != 0) {
-      options = arg + 1;
-    } else {
-      localInputFiles.emplace_back(arg);
-      continue;
-    }
-    assert(options != nullptr);
-
-    bool finished = false;
-    while (!finished && *options != 0) {
-      // Parse the compression level
-      if (*options >= '0' && *options <= '9') {
-        compressionLevel = parseUnsigned(&options);
-        continue;
-      }
-
-      switch (*options) {
-      case 'h':
-      case 'H':
-        usage();
-        return Status::Message;
-      case 'V':
-        std::fprintf(stderr, "PZSTD version: %s.\n", ZSTD_VERSION_STRING);
-        return Status::Message;
-      case 'p': {
-        finished = true;
-        const char *optionArgument = getArgument(options, argv, i, argc);
-        if (optionArgument == nullptr) {
-          return Status::Failure;
-        }
-        if (*optionArgument < '0' || *optionArgument > '9') {
-          std::fprintf(stderr, "Option -p expects a number, but %s provided\n",
-                       optionArgument);
-          return Status::Failure;
-        }
-        numThreads = parseUnsigned(&optionArgument);
-        if (*optionArgument != 0) {
-          std::fprintf(stderr,
-                       "Option -p expects a number, but %u%s provided\n",
-                       numThreads, optionArgument);
-          return Status::Failure;
-        }
-        break;
-      }
-      case 'o': {
-        finished = true;
-        const char *optionArgument = getArgument(options, argv, i, argc);
-        if (optionArgument == nullptr) {
-          return Status::Failure;
-        }
-        outputFile = optionArgument;
-        break;
-      }
-      case 'C':
-        checksum = true;
-        break;
-      case 'k':
-        keepSource = true;
-        break;
-      case 'd':
-        decompress = true;
-        break;
-      case 'f':
-        overwrite = true;
-        forceStdout = true;
-        followLinks = true;
-        break;
-      case 't':
-        test = true;
-        decompress = true;
-        break;
-#ifdef UTIL_HAS_CREATEFILELIST
-      case 'r':
-        recursive = true;
-        break;
-#endif
-      case 'c':
-        outputFile = kStdOut;
-        forceStdout = true;
-        break;
-      case 'v':
-        ++verbosity;
-        break;
-      case 'q':
-        --verbosity;
-        // Ignore them for now
-        break;
-      // Unsupported options from Zstd
-      case 'D':
-      case 's':
-        notSupported("Zstd dictionaries.");
-        return Status::Failure;
-      case 'b':
-      case 'e':
-      case 'i':
-      case 'B':
-        notSupported("Zstd benchmarking options.");
-        return Status::Failure;
-      default:
-        std::fprintf(stderr, "Invalid argument: %s\n", arg);
-        return Status::Failure;
-      }
-      if (!finished) {
-        ++options;
-      }
-    } // while (*options != 0);
-  }   // for (int i = 1; i < argc; ++i);
-
-  // Set options for test mode
-  if (test) {
-    outputFile = nullOutput;
-    keepSource = true;
-  }
-
-  // Input file defaults to standard input if not provided.
-  if (localInputFiles.empty()) {
-    localInputFiles.emplace_back(kStdIn);
-  }
-
-  // Check validity of input files
-  if (localInputFiles.size() > 1) {
-    const auto it = std::find(localInputFiles.begin(), localInputFiles.end(),
-                              std::string{kStdIn});
-    if (it != localInputFiles.end()) {
-      std::fprintf(
-          stderr,
-          "Cannot specify standard input when handling multiple files\n");
-      return Status::Failure;
-    }
-  }
-  if (localInputFiles.size() > 1 || recursive) {
-    if (!outputFile.empty() && outputFile != nullOutput) {
-      std::fprintf(
-          stderr,
-          "Cannot specify an output file when handling multiple inputs\n");
-      return Status::Failure;
-    }
-  }
-
-  g_utilDisplayLevel = verbosity;
-  // Remove local input files that are symbolic links
-  if (!followLinks) {
-      std::remove_if(localInputFiles.begin(), localInputFiles.end(),
-                     [&](const char *path) {
-                        bool isLink = UTIL_isLink(path);
-                        if (isLink && verbosity >= 2) {
-                            std::fprintf(
-                                    stderr,
-                                    "Warning : %s is symbolic link, ignoring\n",
-                                    path);
-                        }
-                        return isLink;
-                    });
-  }
-
-  // Translate input files/directories into files to (de)compress
-  if (recursive) {
-    char *scratchBuffer = nullptr;
-    unsigned numFiles = 0;
-    const char **files =
-        UTIL_createFileList(localInputFiles.data(), localInputFiles.size(),
-                            &scratchBuffer, &numFiles, followLinks);
-    if (files == nullptr) {
-      std::fprintf(stderr, "Error traversing directories\n");
-      return Status::Failure;
-    }
-    auto guard =
-        makeScopeGuard([&] { UTIL_freeFileList(files, scratchBuffer); });
-    if (numFiles == 0) {
-      std::fprintf(stderr, "No files found\n");
-      return Status::Failure;
-    }
-    inputFiles.resize(numFiles);
-    std::copy(files, files + numFiles, inputFiles.begin());
-  } else {
-    inputFiles.resize(localInputFiles.size());
-    std::copy(localInputFiles.begin(), localInputFiles.end(),
-              inputFiles.begin());
-  }
-  localInputFiles.clear();
-  assert(!inputFiles.empty());
-
-  // If reading from standard input, default to standard output
-  if (inputFiles[0] == kStdIn && outputFile.empty()) {
-    assert(inputFiles.size() == 1);
-    outputFile = "-";
-  }
-
-  if (inputFiles[0] == kStdIn && IS_CONSOLE(stdin)) {
-    assert(inputFiles.size() == 1);
-    std::fprintf(stderr, "Cannot read input from interactive console\n");
-    return Status::Failure;
-  }
-  if (outputFile == "-" && IS_CONSOLE(stdout) && !(forceStdout && decompress)) {
-    std::fprintf(stderr, "Will not write to console stdout unless -c or -f is "
-                         "specified and decompressing\n");
-    return Status::Failure;
-  }
-
-  // Check compression level
-  {
-    unsigned maxCLevel =
-        ultra ? ZSTD_maxCLevel() : kMaxNonUltraCompressionLevel;
-    if (compressionLevel > maxCLevel || compressionLevel == 0) {
-      std::fprintf(stderr, "Invalid compression level %u.\n", compressionLevel);
-      return Status::Failure;
-    }
-  }
-
-  // Check that numThreads is set
-  if (numThreads == 0) {
-    std::fprintf(stderr, "Invalid arguments: # of threads not specified "
-                         "and unable to determine hardware concurrency.\n");
-    return Status::Failure;
-  }
-
-  // Modify verbosity
-  // If we are piping input and output, turn off interaction
-  if (inputFiles[0] == kStdIn && outputFile == kStdOut && verbosity == 2) {
-    verbosity = 1;
-  }
-  // If we are in multi-file mode, turn off interaction
-  if (inputFiles.size() > 1 && verbosity == 2) {
-    verbosity = 1;
-  }
-
-  return Status::Success;
-}
-
-std::string Options::getOutputFile(const std::string &inputFile) const {
-  if (!outputFile.empty()) {
-    return outputFile;
-  }
-  // Attempt to add/remove zstd extension from the input file
-  if (decompress) {
-    int stemSize = inputFile.size() - kZstdExtension.size();
-    if (stemSize > 0 && inputFile.substr(stemSize) == kZstdExtension) {
-      return inputFile.substr(0, stemSize);
-    } else {
-      return "";
-    }
-  } else {
-    return inputFile + kZstdExtension;
-  }
-}
-}
diff --git a/contrib/pzstd/Options.h b/contrib/pzstd/Options.h
deleted file mode 100644
index f4f2aaa499cb..000000000000
--- a/contrib/pzstd/Options.h
+++ /dev/null
@@ -1,68 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-#pragma once
-
-#define ZSTD_STATIC_LINKING_ONLY
-#include "zstd.h"
-#undef ZSTD_STATIC_LINKING_ONLY
-
-#include <cstdint>
-#include <string>
-#include <vector>
-
-namespace pzstd {
-
-struct Options {
-  enum class WriteMode { Regular, Auto, Sparse };
-
-  unsigned numThreads;
-  unsigned maxWindowLog;
-  unsigned compressionLevel;
-  bool decompress;
-  std::vector<std::string> inputFiles;
-  std::string outputFile;
-  bool overwrite;
-  bool keepSource;
-  WriteMode writeMode;
-  bool checksum;
-  int verbosity;
-
-  enum class Status {
-    Success, // Successfully parsed options
-    Failure, // Failure to parse options
-    Message  // Options specified to print a message (e.g. "-h")
-  };
-
-  Options();
-  Options(unsigned numThreads, unsigned maxWindowLog, unsigned compressionLevel,
-          bool decompress, std::vector<std::string> inputFiles,
-          std::string outputFile, bool overwrite, bool keepSource,
-          WriteMode writeMode, bool checksum, int verbosity)
-      : numThreads(numThreads), maxWindowLog(maxWindowLog),
-        compressionLevel(compressionLevel), decompress(decompress),
-        inputFiles(std::move(inputFiles)), outputFile(std::move(outputFile)),
-        overwrite(overwrite), keepSource(keepSource), writeMode(writeMode),
-        checksum(checksum), verbosity(verbosity) {}
-
-  Status parse(int argc, const char **argv);
-
-  ZSTD_parameters determineParameters() const {
-    ZSTD_parameters params = ZSTD_getParams(compressionLevel, 0, 0);
-    params.fParams.contentSizeFlag = 0;
-    params.fParams.checksumFlag = checksum;
-    if (maxWindowLog != 0 && params.cParams.windowLog > maxWindowLog) {
-      params.cParams.windowLog = maxWindowLog;
-      params.cParams = ZSTD_adjustCParams(params.cParams, 0, 0);
-    }
-    return params;
-  }
-
-  std::string getOutputFile(const std::string &inputFile) const;
-};
-}
diff --git a/contrib/pzstd/Pzstd.cpp b/contrib/pzstd/Pzstd.cpp
deleted file mode 100644
index 652187c3bd0e..000000000000
--- a/contrib/pzstd/Pzstd.cpp
+++ /dev/null
@@ -1,611 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-#include "platform.h"   /* Large Files support, SET_BINARY_MODE */
-#include "Pzstd.h"
-#include "SkippableFrame.h"
-#include "utils/FileSystem.h"
-#include "utils/Range.h"
-#include "utils/ScopeGuard.h"
-#include "utils/ThreadPool.h"
-#include "utils/WorkQueue.h"
-
-#include <chrono>
-#include <cinttypes>
-#include <cstddef>
-#include <cstdio>
-#include <memory>
-#include <string>
-
-
-namespace pzstd {
-
-namespace {
-#ifdef _WIN32
-const std::string nullOutput = "nul";
-#else
-const std::string nullOutput = "/dev/null";
-#endif
-}
-
-using std::size_t;
-
-static std::uintmax_t fileSizeOrZero(const std::string &file) {
-  if (file == "-") {
-    return 0;
-  }
-  std::error_code ec;
-  auto size = file_size(file, ec);
-  if (ec) {
-    size = 0;
-  }
-  return size;
-}
-
-static std::uint64_t handleOneInput(const Options &options,
-                             const std::string &inputFile,
-                             FILE* inputFd,
-                             const std::string &outputFile,
-                             FILE* outputFd,
-                             SharedState& state) {
-  auto inputSize = fileSizeOrZero(inputFile);
-  // WorkQueue outlives ThreadPool so in the case of error we are certain
-  // we don't accidentally try to call push() on it after it is destroyed
-  WorkQueue<std::shared_ptr<BufferWorkQueue>> outs{options.numThreads + 1};
-  std::uint64_t bytesRead;
-  std::uint64_t bytesWritten;
-  {
-    // Initialize the (de)compression thread pool with numThreads
-    ThreadPool executor(options.numThreads);
-    // Run the reader thread on an extra thread
-    ThreadPool readExecutor(1);
-    if (!options.decompress) {
-      // Add a job that reads the input and starts all the compression jobs
-      readExecutor.add(
-          [&state, &outs, &executor, inputFd, inputSize, &options, &bytesRead] {
-            bytesRead = asyncCompressChunks(
-                state,
-                outs,
-                executor,
-                inputFd,
-                inputSize,
-                options.numThreads,
-                options.determineParameters());
-          });
-      // Start writing
-      bytesWritten = writeFile(state, outs, outputFd, options.decompress);
-    } else {
-      // Add a job that reads the input and starts all the decompression jobs
-      readExecutor.add([&state, &outs, &executor, inputFd, &bytesRead] {
-        bytesRead = asyncDecompressFrames(state, outs, executor, inputFd);
-      });
-      // Start writing
-      bytesWritten = writeFile(state, outs, outputFd, options.decompress);
-    }
-  }
-  if (!state.errorHolder.hasError()) {
-    std::string inputFileName = inputFile == "-" ? "stdin" : inputFile;
-    std::string outputFileName = outputFile == "-" ? "stdout" : outputFile;
-    if (!options.decompress) {
-      double ratio = static_cast<double>(bytesWritten) /
-                     static_cast<double>(bytesRead + !bytesRead);
-      state.log(INFO, "%-20s :%6.2f%%   (%6" PRIu64 " => %6" PRIu64
-                   " bytes, %s)\n",
-                   inputFileName.c_str(), ratio * 100, bytesRead, bytesWritten,
-                   outputFileName.c_str());
-    } else {
-      state.log(INFO, "%-20s: %" PRIu64 " bytes \n",
-                   inputFileName.c_str(),bytesWritten);
-    }
-  }
-  return bytesWritten;
-}
-
-static FILE *openInputFile(const std::string &inputFile,
-                           ErrorHolder &errorHolder) {
-  if (inputFile == "-") {
-    SET_BINARY_MODE(stdin);
-    return stdin;
-  }
-  // Check if input file is a directory
-  {
-    std::error_code ec;
-    if (is_directory(inputFile, ec)) {
-      errorHolder.setError("Output file is a directory -- ignored");
-      return nullptr;
-    }
-  }
-  auto inputFd = std::fopen(inputFile.c_str(), "rb");
-  if (!errorHolder.check(inputFd != nullptr, "Failed to open input file")) {
-    return nullptr;
-  }
-  return inputFd;
-}
-
-static FILE *openOutputFile(const Options &options,
-                            const std::string &outputFile,
-                            SharedState& state) {
-  if (outputFile == "-") {
-    SET_BINARY_MODE(stdout);
-    return stdout;
-  }
-  // Check if the output file exists and then open it
-  if (!options.overwrite && outputFile != nullOutput) {
-    auto outputFd = std::fopen(outputFile.c_str(), "rb");
-    if (outputFd != nullptr) {
-      std::fclose(outputFd);
-      if (!state.log.logsAt(INFO)) {
-        state.errorHolder.setError("Output file exists");
-        return nullptr;
-      }
-      state.log(
-          INFO,
-          "pzstd: %s already exists; do you wish to overwrite (y/n) ? ",
-          outputFile.c_str());
-      int c = getchar();
-      if (c != 'y' && c != 'Y') {
-        state.errorHolder.setError("Not overwritten");
-        return nullptr;
-      }
-    }
-  }
-  auto outputFd = std::fopen(outputFile.c_str(), "wb");
-  if (!state.errorHolder.check(
-          outputFd != nullptr, "Failed to open output file")) {
-    return nullptr;
-  }
-  return outputFd;
-}
-
-int pzstdMain(const Options &options) {
-  int returnCode = 0;
-  SharedState state(options);
-  for (const auto& input : options.inputFiles) {
-    // Setup the shared state
-    auto printErrorGuard = makeScopeGuard([&] {
-      if (state.errorHolder.hasError()) {
-        returnCode = 1;
-        state.log(ERROR, "pzstd: %s: %s.\n", input.c_str(),
-                  state.errorHolder.getError().c_str());
-      }
-    });
-    // Open the input file
-    auto inputFd = openInputFile(input, state.errorHolder);
-    if (inputFd == nullptr) {
-      continue;
-    }
-    auto closeInputGuard = makeScopeGuard([&] { std::fclose(inputFd); });
-    // Open the output file
-    auto outputFile = options.getOutputFile(input);
-    if (!state.errorHolder.check(outputFile != "",
-                           "Input file does not have extension .zst")) {
-      continue;
-    }
-    auto outputFd = openOutputFile(options, outputFile, state);
-    if (outputFd == nullptr) {
-      continue;
-    }
-    auto closeOutputGuard = makeScopeGuard([&] { std::fclose(outputFd); });
-    // (de)compress the file
-    handleOneInput(options, input, inputFd, outputFile, outputFd, state);
-    if (state.errorHolder.hasError()) {
-      continue;
-    }
-    // Delete the input file if necessary
-    if (!options.keepSource) {
-      // Be sure that we are done and have written everything before we delete
-      if (!state.errorHolder.check(std::fclose(inputFd) == 0,
-                             "Failed to close input file")) {
-        continue;
-      }
-      closeInputGuard.dismiss();
-      if (!state.errorHolder.check(std::fclose(outputFd) == 0,
-                             "Failed to close output file")) {
-        continue;
-      }
-      closeOutputGuard.dismiss();
-      if (std::remove(input.c_str()) != 0) {
-        state.errorHolder.setError("Failed to remove input file");
-        continue;
-      }
-    }
-  }
-  // Returns 1 if any of the files failed to (de)compress.
-  return returnCode;
-}
-
-/// Construct a `ZSTD_inBuffer` that points to the data in `buffer`.
-static ZSTD_inBuffer makeZstdInBuffer(const Buffer& buffer) {
-  return ZSTD_inBuffer{buffer.data(), buffer.size(), 0};
-}
-
-/**
- * Advance `buffer` and `inBuffer` by the amount of data read, as indicated by
- * `inBuffer.pos`.
- */
-void advance(Buffer& buffer, ZSTD_inBuffer& inBuffer) {
-  auto pos = inBuffer.pos;
-  inBuffer.src = static_cast<const unsigned char*>(inBuffer.src) + pos;
-  inBuffer.size -= pos;
-  inBuffer.pos = 0;
-  return buffer.advance(pos);
-}
-
-/// Construct a `ZSTD_outBuffer` that points to the data in `buffer`.
-static ZSTD_outBuffer makeZstdOutBuffer(Buffer& buffer) {
-  return ZSTD_outBuffer{buffer.data(), buffer.size(), 0};
-}
-
-/**
- * Split `buffer` and advance `outBuffer` by the amount of data written, as
- * indicated by `outBuffer.pos`.
- */
-Buffer split(Buffer& buffer, ZSTD_outBuffer& outBuffer) {
-  auto pos = outBuffer.pos;
-  outBuffer.dst = static_cast<unsigned char*>(outBuffer.dst) + pos;
-  outBuffer.size -= pos;
-  outBuffer.pos = 0;
-  return buffer.splitAt(pos);
-}
-
-/**
- * Stream chunks of input from `in`, compress it, and stream it out to `out`.
- *
- * @param state        The shared state
- * @param in           Queue that we `pop()` input buffers from
- * @param out          Queue that we `push()` compressed output buffers to
- * @param maxInputSize An upper bound on the size of the input
- */
-static void compress(
-    SharedState& state,
-    std::shared_ptr<BufferWorkQueue> in,
-    std::shared_ptr<BufferWorkQueue> out,
-    size_t maxInputSize) {
-  auto& errorHolder = state.errorHolder;
-  auto guard = makeScopeGuard([&] { out->finish(); });
-  // Initialize the CCtx
-  auto ctx = state.cStreamPool->get();
-  if (!errorHolder.check(ctx != nullptr, "Failed to allocate ZSTD_CStream")) {
-    return;
-  }
-  {
-    auto err = ZSTD_resetCStream(ctx.get(), 0);
-    if (!errorHolder.check(!ZSTD_isError(err), ZSTD_getErrorName(err))) {
-      return;
-    }
-  }
-
-  // Allocate space for the result
-  auto outBuffer = Buffer(ZSTD_compressBound(maxInputSize));
-  auto zstdOutBuffer = makeZstdOutBuffer(outBuffer);
-  {
-    Buffer inBuffer;
-    // Read a buffer in from the input queue
-    while (in->pop(inBuffer) && !errorHolder.hasError()) {
-      auto zstdInBuffer = makeZstdInBuffer(inBuffer);
-      // Compress the whole buffer and send it to the output queue
-      while (!inBuffer.empty() && !errorHolder.hasError()) {
-        if (!errorHolder.check(
-                !outBuffer.empty(), "ZSTD_compressBound() was too small")) {
-          return;
-        }
-        // Compress
-        auto err =
-            ZSTD_compressStream(ctx.get(), &zstdOutBuffer, &zstdInBuffer);
-        if (!errorHolder.check(!ZSTD_isError(err), ZSTD_getErrorName(err))) {
-          return;
-        }
-        // Split the compressed data off outBuffer and pass to the output queue
-        out->push(split(outBuffer, zstdOutBuffer));
-        // Forget about the data we already compressed
-        advance(inBuffer, zstdInBuffer);
-      }
-    }
-  }
-  // Write the epilog
-  size_t bytesLeft;
-  do {
-    if (!errorHolder.check(
-            !outBuffer.empty(), "ZSTD_compressBound() was too small")) {
-      return;
-    }
-    bytesLeft = ZSTD_endStream(ctx.get(), &zstdOutBuffer);
-    if (!errorHolder.check(
-            !ZSTD_isError(bytesLeft), ZSTD_getErrorName(bytesLeft))) {
-      return;
-    }
-    out->push(split(outBuffer, zstdOutBuffer));
-  } while (bytesLeft != 0 && !errorHolder.hasError());
-}
-
-/**
- * Calculates how large each independently compressed frame should be.
- *
- * @param size       The size of the source if known, 0 otherwise
- * @param numThreads The number of threads available to run compression jobs on
- * @param params     The zstd parameters to be used for compression
- */
-static size_t calculateStep(
-    std::uintmax_t size,
-    size_t numThreads,
-    const ZSTD_parameters &params) {
-  (void)size;
-  (void)numThreads;
-  return size_t{1} << (params.cParams.windowLog + 2);
-}
-
-namespace {
-enum class FileStatus { Continue, Done, Error };
-/// Determines the status of the file descriptor `fd`.
-FileStatus fileStatus(FILE* fd) {
-  if (std::feof(fd)) {
-    return FileStatus::Done;
-  } else if (std::ferror(fd)) {
-    return FileStatus::Error;
-  }
-  return FileStatus::Continue;
-}
-} // anonymous namespace
-
-/**
- * Reads `size` data in chunks of `chunkSize` and puts it into `queue`.
- * Will read less if an error or EOF occurs.
- * Returns the status of the file after all of the reads have occurred.
- */
-static FileStatus
-readData(BufferWorkQueue& queue, size_t chunkSize, size_t size, FILE* fd,
-         std::uint64_t *totalBytesRead) {
-  Buffer buffer(size);
-  while (!buffer.empty()) {
-    auto bytesRead =
-        std::fread(buffer.data(), 1, std::min(chunkSize, buffer.size()), fd);
-    *totalBytesRead += bytesRead;
-    queue.push(buffer.splitAt(bytesRead));
-    auto status = fileStatus(fd);
-    if (status != FileStatus::Continue) {
-      return status;
-    }
-  }
-  return FileStatus::Continue;
-}
-
-std::uint64_t asyncCompressChunks(
-    SharedState& state,
-    WorkQueue<std::shared_ptr<BufferWorkQueue>>& chunks,
-    ThreadPool& executor,
-    FILE* fd,
-    std::uintmax_t size,
-    size_t numThreads,
-    ZSTD_parameters params) {
-  auto chunksGuard = makeScopeGuard([&] { chunks.finish(); });
-  std::uint64_t bytesRead = 0;
-
-  // Break the input up into chunks of size `step` and compress each chunk
-  // independently.
-  size_t step = calculateStep(size, numThreads, params);
-  state.log(DEBUG, "Chosen frame size: %zu\n", step);
-  auto status = FileStatus::Continue;
-  while (status == FileStatus::Continue && !state.errorHolder.hasError()) {
-    // Make a new input queue that we will put the chunk's input data into.
-    auto in = std::make_shared<BufferWorkQueue>();
-    auto inGuard = makeScopeGuard([&] { in->finish(); });
-    // Make a new output queue that compress will put the compressed data into.
-    auto out = std::make_shared<BufferWorkQueue>();
-    // Start compression in the thread pool
-    executor.add([&state, in, out, step] {
-      return compress(
-          state, std::move(in), std::move(out), step);
-    });
-    // Pass the output queue to the writer thread.
-    chunks.push(std::move(out));
-    state.log(VERBOSE, "%s\n", "Starting a new frame");
-    // Fill the input queue for the compression job we just started
-    status = readData(*in, ZSTD_CStreamInSize(), step, fd, &bytesRead);
-  }
-  state.errorHolder.check(status != FileStatus::Error, "Error reading input");
-  return bytesRead;
-}
-
-/**
- * Decompress a frame, whose data is streamed into `in`, and stream the output
- * to `out`.
- *
- * @param state        The shared state
- * @param in           Queue that we `pop()` input buffers from. It contains
- *                      exactly one compressed frame.
- * @param out          Queue that we `push()` decompressed output buffers to
- */
-static void decompress(
-    SharedState& state,
-    std::shared_ptr<BufferWorkQueue> in,
-    std::shared_ptr<BufferWorkQueue> out) {
-  auto& errorHolder = state.errorHolder;
-  auto guard = makeScopeGuard([&] { out->finish(); });
-  // Initialize the DCtx
-  auto ctx = state.dStreamPool->get();
-  if (!errorHolder.check(ctx != nullptr, "Failed to allocate ZSTD_DStream")) {
-    return;
-  }
-  {
-    auto err = ZSTD_resetDStream(ctx.get());
-    if (!errorHolder.check(!ZSTD_isError(err), ZSTD_getErrorName(err))) {
-      return;
-    }
-  }
-
-  const size_t outSize = ZSTD_DStreamOutSize();
-  Buffer inBuffer;
-  size_t returnCode = 0;
-  // Read a buffer in from the input queue
-  while (in->pop(inBuffer) && !errorHolder.hasError()) {
-    auto zstdInBuffer = makeZstdInBuffer(inBuffer);
-    // Decompress the whole buffer and send it to the output queue
-    while (!inBuffer.empty() && !errorHolder.hasError()) {
-      // Allocate a buffer with at least outSize bytes.
-      Buffer outBuffer(outSize);
-      auto zstdOutBuffer = makeZstdOutBuffer(outBuffer);
-      // Decompress
-      returnCode =
-          ZSTD_decompressStream(ctx.get(), &zstdOutBuffer, &zstdInBuffer);
-      if (!errorHolder.check(
-              !ZSTD_isError(returnCode), ZSTD_getErrorName(returnCode))) {
-        return;
-      }
-      // Pass the buffer with the decompressed data to the output queue
-      out->push(split(outBuffer, zstdOutBuffer));
-      // Advance past the input we already read
-      advance(inBuffer, zstdInBuffer);
-      if (returnCode == 0) {
-        // The frame is over, prepare to (maybe) start a new frame
-        ZSTD_initDStream(ctx.get());
-      }
-    }
-  }
-  if (!errorHolder.check(returnCode <= 1, "Incomplete block")) {
-    return;
-  }
-  // We've given ZSTD_decompressStream all of our data, but there may still
-  // be data to read.
-  while (returnCode == 1) {
-    // Allocate a buffer with at least outSize bytes.
-    Buffer outBuffer(outSize);
-    auto zstdOutBuffer = makeZstdOutBuffer(outBuffer);
-    // Pass in no input.
-    ZSTD_inBuffer zstdInBuffer{nullptr, 0, 0};
-    // Decompress
-    returnCode =
-        ZSTD_decompressStream(ctx.get(), &zstdOutBuffer, &zstdInBuffer);
-    if (!errorHolder.check(
-            !ZSTD_isError(returnCode), ZSTD_getErrorName(returnCode))) {
-      return;
-    }
-    // Pass the buffer with the decompressed data to the output queue
-    out->push(split(outBuffer, zstdOutBuffer));
-  }
-}
-
-std::uint64_t asyncDecompressFrames(
-    SharedState& state,
-    WorkQueue<std::shared_ptr<BufferWorkQueue>>& frames,
-    ThreadPool& executor,
-    FILE* fd) {
-  auto framesGuard = makeScopeGuard([&] { frames.finish(); });
-  std::uint64_t totalBytesRead = 0;
-
-  // Split the source up into its component frames.
-  // If we find our recognized skippable frame we know the next frames size
-  // which means that we can decompress each standard frame in independently.
-  // Otherwise, we will decompress using only one decompression task.
-  const size_t chunkSize = ZSTD_DStreamInSize();
-  auto status = FileStatus::Continue;
-  while (status == FileStatus::Continue && !state.errorHolder.hasError()) {
-    // Make a new input queue that we will put the frames's bytes into.
-    auto in = std::make_shared<BufferWorkQueue>();
-    auto inGuard = makeScopeGuard([&] { in->finish(); });
-    // Make a output queue that decompress will put the decompressed data into
-    auto out = std::make_shared<BufferWorkQueue>();
-
-    size_t frameSize;
-    {
-      // Calculate the size of the next frame.
-      // frameSize is 0 if the frame info can't be decoded.
-      Buffer buffer(SkippableFrame::kSize);
-      auto bytesRead = std::fread(buffer.data(), 1, buffer.size(), fd);
-      totalBytesRead += bytesRead;
-      status = fileStatus(fd);
-      if (bytesRead == 0 && status != FileStatus::Continue) {
-        break;
-      }
-      buffer.subtract(buffer.size() - bytesRead);
-      frameSize = SkippableFrame::tryRead(buffer.range());
-      in->push(std::move(buffer));
-    }
-    if (frameSize == 0) {
-      // We hit a non SkippableFrame, so this will be the last job.
-      // Make sure that we don't use too much memory
-      in->setMaxSize(64);
-      out->setMaxSize(64);
-    }
-    // Start decompression in the thread pool
-    executor.add([&state, in, out] {
-      return decompress(state, std::move(in), std::move(out));
-    });
-    // Pass the output queue to the writer thread
-    frames.push(std::move(out));
-    if (frameSize == 0) {
-      // We hit a non SkippableFrame ==> not compressed by pzstd or corrupted
-      // Pass the rest of the source to this decompression task
-      state.log(VERBOSE, "%s\n",
-          "Input not in pzstd format, falling back to serial decompression");
-      while (status == FileStatus::Continue && !state.errorHolder.hasError()) {
-        status = readData(*in, chunkSize, chunkSize, fd, &totalBytesRead);
-      }
-      break;
-    }
-    state.log(VERBOSE, "Decompressing a frame of size %zu", frameSize);
-    // Fill the input queue for the decompression job we just started
-    status = readData(*in, chunkSize, frameSize, fd, &totalBytesRead);
-  }
-  state.errorHolder.check(status != FileStatus::Error, "Error reading input");
-  return totalBytesRead;
-}
-
-/// Write `data` to `fd`, returns true iff success.
-static bool writeData(ByteRange data, FILE* fd) {
-  while (!data.empty()) {
-    data.advance(std::fwrite(data.begin(), 1, data.size(), fd));
-    if (std::ferror(fd)) {
-      return false;
-    }
-  }
-  return true;
-}
-
-std::uint64_t writeFile(
-    SharedState& state,
-    WorkQueue<std::shared_ptr<BufferWorkQueue>>& outs,
-    FILE* outputFd,
-    bool decompress) {
-  auto& errorHolder = state.errorHolder;
-  auto lineClearGuard = makeScopeGuard([&state] {
-    state.log.clear(INFO);
-  });
-  std::uint64_t bytesWritten = 0;
-  std::shared_ptr<BufferWorkQueue> out;
-  // Grab the output queue for each decompression job (in order).
-  while (outs.pop(out)) {
-    if (errorHolder.hasError()) {
-      continue;
-    }
-    if (!decompress) {
-      // If we are compressing and want to write skippable frames we can't
-      // start writing before compression is done because we need to know the
-      // compressed size.
-      // Wait for the compressed size to be available and write skippable frame
-      SkippableFrame frame(out->size());
-      if (!writeData(frame.data(), outputFd)) {
-        errorHolder.setError("Failed to write output");
-        return bytesWritten;
-      }
-      bytesWritten += frame.kSize;
-    }
-    // For each chunk of the frame: Pop it from the queue and write it
-    Buffer buffer;
-    while (out->pop(buffer) && !errorHolder.hasError()) {
-      if (!writeData(buffer.range(), outputFd)) {
-        errorHolder.setError("Failed to write output");
-        return bytesWritten;
-      }
-      bytesWritten += buffer.size();
-      state.log.update(INFO, "Written: %u MB   ",
-                static_cast<std::uint32_t>(bytesWritten >> 20));
-    }
-  }
-  return bytesWritten;
-}
-}
diff --git a/contrib/pzstd/Pzstd.h b/contrib/pzstd/Pzstd.h
deleted file mode 100644
index 79d1fcca2653..000000000000
--- a/contrib/pzstd/Pzstd.h
+++ /dev/null
@@ -1,150 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-#pragma once
-
-#include "ErrorHolder.h"
-#include "Logging.h"
-#include "Options.h"
-#include "utils/Buffer.h"
-#include "utils/Range.h"
-#include "utils/ResourcePool.h"
-#include "utils/ThreadPool.h"
-#include "utils/WorkQueue.h"
-#define ZSTD_STATIC_LINKING_ONLY
-#include "zstd.h"
-#undef ZSTD_STATIC_LINKING_ONLY
-
-#include <cstddef>
-#include <cstdint>
-#include <memory>
-
-namespace pzstd {
-/**
- * Runs pzstd with `options` and returns the number of bytes written.
- * An error occurred if `errorHandler.hasError()`.
- *
- * @param options      The pzstd options to use for (de)compression
- * @returns            0 upon success and non-zero on failure.
- */
-int pzstdMain(const Options& options);
-
-class SharedState {
- public:
-  SharedState(const Options& options) : log(options.verbosity) {
-    if (!options.decompress) {
-      auto parameters = options.determineParameters();
-      cStreamPool.reset(new ResourcePool<ZSTD_CStream>{
-          [this, parameters]() -> ZSTD_CStream* {
-            this->log(VERBOSE, "%s\n", "Creating new ZSTD_CStream");
-            auto zcs = ZSTD_createCStream();
-            if (zcs) {
-              auto err = ZSTD_initCStream_advanced(
-                  zcs, nullptr, 0, parameters, 0);
-              if (ZSTD_isError(err)) {
-                ZSTD_freeCStream(zcs);
-                return nullptr;
-              }
-            }
-            return zcs;
-          },
-          [](ZSTD_CStream *zcs) {
-            ZSTD_freeCStream(zcs);
-          }});
-    } else {
-      dStreamPool.reset(new ResourcePool<ZSTD_DStream>{
-          [this]() -> ZSTD_DStream* {
-            this->log(VERBOSE, "%s\n", "Creating new ZSTD_DStream");
-            auto zds = ZSTD_createDStream();
-            if (zds) {
-              auto err = ZSTD_initDStream(zds);
-              if (ZSTD_isError(err)) {
-                ZSTD_freeDStream(zds);
-                return nullptr;
-              }
-            }
-            return zds;
-          },
-          [](ZSTD_DStream *zds) {
-            ZSTD_freeDStream(zds);
-          }});
-    }
-  }
-
-  ~SharedState() {
-    // The resource pools have references to this, so destroy them first.
-    cStreamPool.reset();
-    dStreamPool.reset();
-  }
-
-  Logger log;
-  ErrorHolder errorHolder;
-  std::unique_ptr<ResourcePool<ZSTD_CStream>> cStreamPool;
-  std::unique_ptr<ResourcePool<ZSTD_DStream>> dStreamPool;
-};
-
-/**
- * Streams input from `fd`, breaks input up into chunks, and compresses each
- * chunk independently.  Output of each chunk gets streamed to a queue, and
- * the output queues get put into `chunks` in order.
- *
- * @param state        The shared state
- * @param chunks       Each compression jobs output queue gets `pushed()` here
- *                      as soon as it is available
- * @param executor     The thread pool to run compression jobs in
- * @param fd           The input file descriptor
- * @param size         The size of the input file if known, 0 otherwise
- * @param numThreads   The number of threads in the thread pool
- * @param parameters   The zstd parameters to use for compression
- * @returns            The number of bytes read from the file
- */
-std::uint64_t asyncCompressChunks(
-    SharedState& state,
-    WorkQueue<std::shared_ptr<BufferWorkQueue>>& chunks,
-    ThreadPool& executor,
-    FILE* fd,
-    std::uintmax_t size,
-    std::size_t numThreads,
-    ZSTD_parameters parameters);
-
-/**
- * Streams input from `fd`.  If pzstd headers are available it breaks the input
- * up into independent frames.  It sends each frame to an independent
- * decompression job.  Output of each frame gets streamed to a queue, and
- * the output queues get put into `frames` in order.
- *
- * @param state        The shared state
- * @param frames       Each decompression jobs output queue gets `pushed()` here
- *                      as soon as it is available
- * @param executor     The thread pool to run compression jobs in
- * @param fd           The input file descriptor
- * @returns            The number of bytes read from the file
- */
-std::uint64_t asyncDecompressFrames(
-    SharedState& state,
-    WorkQueue<std::shared_ptr<BufferWorkQueue>>& frames,
-    ThreadPool& executor,
-    FILE* fd);
-
-/**
- * Streams input in from each queue in `outs` in order, and writes the data to
- * `outputFd`.
- *
- * @param state        The shared state
- * @param outs         A queue of output queues, one for each
- *                      (de)compression job.
- * @param outputFd     The file descriptor to write to
- * @param decompress   Are we decompressing?
- * @returns            The number of bytes written
- */
-std::uint64_t writeFile(
-    SharedState& state,
-    WorkQueue<std::shared_ptr<BufferWorkQueue>>& outs,
-    FILE* outputFd,
-    bool decompress);
-}
diff --git a/contrib/pzstd/README.md b/contrib/pzstd/README.md
deleted file mode 100644
index 84d945815838..000000000000
--- a/contrib/pzstd/README.md
+++ /dev/null
@@ -1,56 +0,0 @@
-# Parallel Zstandard (PZstandard)
-
-Parallel Zstandard is a Pigz-like tool for Zstandard.
-It provides Zstandard format compatible compression and decompression that is able to utilize multiple cores.
-It breaks the input up into equal sized chunks and compresses each chunk independently into a Zstandard frame.
-It then concatenates the frames together to produce the final compressed output.
-Pzstandard will write a 12 byte header for each frame that is a skippable frame in the Zstandard format, which tells PZstandard the size of the next compressed frame.
-PZstandard supports parallel decompression of files compressed with PZstandard.
-When decompressing files compressed with Zstandard, PZstandard does IO in one thread, and decompression in another.
-
-## Usage
-
-PZstandard supports the same command line interface as Zstandard, but also provides the `-p` option to specify the number of threads.
-Dictionary mode is not currently supported.
-
-Basic usage
-
-    pzstd input-file -o output-file -p num-threads -#          # Compression
-    pzstd -d input-file -o output-file -p num-threads          # Decompression
-
-PZstandard also supports piping and fifo pipes
-
-    cat input-file | pzstd -p num-threads -# -c > /dev/null
-
-For more options
-
-    pzstd --help
-
-PZstandard tries to pick a smart default number of threads if not specified (displayed in `pzstd --help`).
-If this number is not suitable, during compilation you can define `PZSTD_NUM_THREADS` to the number of threads you prefer.
-
-## Benchmarks
-
-As a reference, PZstandard and Pigz were compared on an Intel Core i7 @ 3.1 GHz, each using 4 threads, with the [Silesia compression corpus](http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia).
-
-Compression Speed vs Ratio with 4 Threads | Decompression Speed with 4 Threads
-------------------------------------------|-----------------------------------
-![Compression Speed vs Ratio](images/Cspeed.png "Compression Speed vs Ratio") | ![Decompression Speed](images/Dspeed.png "Decompression Speed")
-
-The test procedure was to run each of the following commands 2 times for each compression level, and take the minimum time.
-
-    time pzstd -# -p 4    -c silesia.tar     > silesia.tar.zst
-    time pzstd -d -p 4    -c silesia.tar.zst > /dev/null
-
-    time pigz  -# -p 4 -k -c silesia.tar     > silesia.tar.gz
-    time pigz  -d -p 4 -k -c silesia.tar.gz  > /dev/null
-
-PZstandard was tested using compression levels 1-19, and Pigz was tested using compression levels 1-9.
-Pigz cannot do parallel decompression, it simply does each of reading, decompression, and writing on separate threads.
-
-## Tests
-
-Tests require that you have [gtest](https://github.com/google/googletest) installed.
-Set `GTEST_INC` and `GTEST_LIB` in `Makefile` to specify the location of the gtest headers and libraries.
-Alternatively, run `make googletest`, which will clone googletest and build it.
-Run `make tests && make check` to run tests.
diff --git a/contrib/pzstd/SkippableFrame.cpp b/contrib/pzstd/SkippableFrame.cpp
deleted file mode 100644
index 769866dfc815..000000000000
--- a/contrib/pzstd/SkippableFrame.cpp
+++ /dev/null
@@ -1,30 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-#include "SkippableFrame.h"
-#include "mem.h"
-#include "utils/Range.h"
-
-#include <cstdio>
-
-using namespace pzstd;
-
-SkippableFrame::SkippableFrame(std::uint32_t size) : frameSize_(size) {
-  MEM_writeLE32(data_.data(), kSkippableFrameMagicNumber);
-  MEM_writeLE32(data_.data() + 4, kFrameContentsSize);
-  MEM_writeLE32(data_.data() + 8, frameSize_);
-}
-
-/* static */ std::size_t SkippableFrame::tryRead(ByteRange bytes) {
-  if (bytes.size() < SkippableFrame::kSize ||
-      MEM_readLE32(bytes.begin()) != kSkippableFrameMagicNumber ||
-      MEM_readLE32(bytes.begin() + 4) != kFrameContentsSize) {
-    return 0;
-  }
-  return MEM_readLE32(bytes.begin() + 8);
-}
diff --git a/contrib/pzstd/SkippableFrame.h b/contrib/pzstd/SkippableFrame.h
deleted file mode 100644
index 60deed0405be..000000000000
--- a/contrib/pzstd/SkippableFrame.h
+++ /dev/null
@@ -1,64 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-#pragma once
-
-#include "utils/Range.h"
-
-#include <array>
-#include <cstddef>
-#include <cstdint>
-#include <cstdio>
-
-namespace pzstd {
-/**
- * We put a skippable frame before each frame.
- * It contains a skippable frame magic number, the size of the skippable frame,
- * and the size of the next frame.
- * Each skippable frame is exactly 12 bytes in little endian format.
- * The first 8 bytes are for compatibility with the ZSTD format.
- * If we have N threads, the output will look like
- *
- * [0x184D2A50|4|size1] [frame1 of size size1]
- * [0x184D2A50|4|size2] [frame2 of size size2]
- * ...
- * [0x184D2A50|4|sizeN] [frameN of size sizeN]
- *
- * Each sizeX is 4 bytes.
- *
- * These skippable frames should allow us to skip through the compressed file
- * and only load at most N pages.
- */
-class SkippableFrame {
- public:
-  static constexpr std::size_t kSize = 12;
-
- private:
-  std::uint32_t frameSize_;
-  std::array<std::uint8_t, kSize> data_;
-  static constexpr std::uint32_t kSkippableFrameMagicNumber = 0x184D2A50;
-  // Could be improved if the size fits in less bytes
-  static constexpr std::uint32_t kFrameContentsSize = kSize - 8;
-
- public:
-   // Write the skippable frame to data_ in LE format.
-  explicit SkippableFrame(std::uint32_t size);
-
-  // Read the skippable frame from bytes in LE format.
-  static std::size_t tryRead(ByteRange bytes);
-
-  ByteRange data() const {
-    return {data_.data(), data_.size()};
-  }
-
-  // Size of the next frame.
-  std::size_t frameSize() const {
-    return frameSize_;
-  }
-};
-}
diff --git a/contrib/pzstd/images/Cspeed.png b/contrib/pzstd/images/Cspeed.png
deleted file mode 100644
index aca4f663ea2e..000000000000
--- a/contrib/pzstd/images/Cspeed.png
+++ /dev/null
diff --git a/contrib/pzstd/images/Dspeed.png b/contrib/pzstd/images/Dspeed.png
deleted file mode 100644
index e48881bcd05b..000000000000
--- a/contrib/pzstd/images/Dspeed.png
+++ /dev/null
diff --git a/contrib/pzstd/main.cpp b/contrib/pzstd/main.cpp
deleted file mode 100644
index b93f043b16b1..000000000000
--- a/contrib/pzstd/main.cpp
+++ /dev/null
@@ -1,27 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-#include "ErrorHolder.h"
-#include "Options.h"
-#include "Pzstd.h"
-
-using namespace pzstd;
-
-int main(int argc, const char** argv) {
-  Options options;
-  switch (options.parse(argc, argv)) {
-  case Options::Status::Failure:
-    return 1;
-  case Options::Status::Message:
-    return 0;
-  default:
-    break;
-  }
-
-  return pzstdMain(options);
-}
diff --git a/contrib/pzstd/test/BUCK b/contrib/pzstd/test/BUCK
deleted file mode 100644
index 6d3fdd3c269b..000000000000
--- a/contrib/pzstd/test/BUCK
+++ /dev/null
@@ -1,37 +0,0 @@
-cxx_test(
-    name='options_test',
-    srcs=['OptionsTest.cpp'],
-    deps=['//contrib/pzstd:options'],
-)
-
-cxx_test(
-    name='pzstd_test',
-    srcs=['PzstdTest.cpp'],
-    deps=[
-        ':round_trip',
-        '//contrib/pzstd:libpzstd',
-        '//contrib/pzstd/utils:scope_guard',
-        '//programs:datagen',
-    ],
-)
-
-cxx_binary(
-    name='round_trip_test',
-    srcs=['RoundTripTest.cpp'],
-    deps=[
-        ':round_trip',
-        '//contrib/pzstd/utils:scope_guard',
-        '//programs:datagen',
-    ]
-)
-
-cxx_library(
-    name='round_trip',
-    header_namespace='test',
-    exported_headers=['RoundTrip.h'],
-    deps=[
-        '//contrib/pzstd:libpzstd',
-        '//contrib/pzstd:options',
-        '//contrib/pzstd/utils:scope_guard',
-    ]
-)
diff --git a/contrib/pzstd/test/OptionsTest.cpp b/contrib/pzstd/test/OptionsTest.cpp
deleted file mode 100644
index e601148255d4..000000000000
--- a/contrib/pzstd/test/OptionsTest.cpp
+++ /dev/null
@@ -1,536 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-#include "Options.h"
-
-#include <array>
-#include <gtest/gtest.h>
-
-using namespace pzstd;
-
-namespace pzstd {
-bool operator==(const Options &lhs, const Options &rhs) {
-  return lhs.numThreads == rhs.numThreads &&
-         lhs.maxWindowLog == rhs.maxWindowLog &&
-         lhs.compressionLevel == rhs.compressionLevel &&
-         lhs.decompress == rhs.decompress && lhs.inputFiles == rhs.inputFiles &&
-         lhs.outputFile == rhs.outputFile && lhs.overwrite == rhs.overwrite &&
-         lhs.keepSource == rhs.keepSource && lhs.writeMode == rhs.writeMode &&
-         lhs.checksum == rhs.checksum && lhs.verbosity == rhs.verbosity;
-}
-
-std::ostream &operator<<(std::ostream &out, const Options &opt) {
-  out << "{";
-  {
-    out << "\n\t"
-        << "numThreads: " << opt.numThreads;
-    out << ",\n\t"
-        << "maxWindowLog: " << opt.maxWindowLog;
-    out << ",\n\t"
-        << "compressionLevel: " << opt.compressionLevel;
-    out << ",\n\t"
-        << "decompress: " << opt.decompress;
-    out << ",\n\t"
-        << "inputFiles: {";
-    {
-      bool first = true;
-      for (const auto &file : opt.inputFiles) {
-        if (!first) {
-          out << ",";
-        }
-        first = false;
-        out << "\n\t\t" << file;
-      }
-    }
-    out << "\n\t}";
-    out << ",\n\t"
-        << "outputFile: " << opt.outputFile;
-    out << ",\n\t"
-        << "overwrite: " << opt.overwrite;
-    out << ",\n\t"
-        << "keepSource: " << opt.keepSource;
-    out << ",\n\t"
-        << "writeMode: " << static_cast<int>(opt.writeMode);
-    out << ",\n\t"
-        << "checksum: " << opt.checksum;
-    out << ",\n\t"
-        << "verbosity: " << opt.verbosity;
-  }
-  out << "\n}";
-  return out;
-}
-}
-
-namespace {
-#ifdef _WIN32
-const char nullOutput[] = "nul";
-#else
-const char nullOutput[] = "/dev/null";
-#endif
-
-constexpr auto autoMode = Options::WriteMode::Auto;
-} // anonymous namespace
-
-#define EXPECT_SUCCESS(...) EXPECT_EQ(Options::Status::Success, __VA_ARGS__)
-#define EXPECT_FAILURE(...) EXPECT_EQ(Options::Status::Failure, __VA_ARGS__)
-#define EXPECT_MESSAGE(...) EXPECT_EQ(Options::Status::Message, __VA_ARGS__)
-
-template <typename... Args>
-std::array<const char *, sizeof...(Args) + 1> makeArray(Args... args) {
-  return {{nullptr, args...}};
-}
-
-TEST(Options, ValidInputs) {
-  {
-    Options options;
-    auto args = makeArray("--processes", "5", "-o", "x", "y", "-f");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    Options expected = {5,    23,   3,        false, {"y"}, "x",
-                        true, true, autoMode, true,  2};
-    EXPECT_EQ(expected, options);
-  }
-  {
-    Options options;
-    auto args = makeArray("-p", "1", "input", "-19");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    Options expected = {1,     23,   19,       false, {"input"}, "",
-                        false, true, autoMode, true,  2};
-    EXPECT_EQ(expected, options);
-  }
-  {
-    Options options;
-    auto args =
-        makeArray("--ultra", "-22", "-p", "1", "-o", "x", "-d", "x.zst", "-f");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    Options expected = {1,    0,    22,       true, {"x.zst"}, "x",
-                        true, true, autoMode, true, 2};
-    EXPECT_EQ(expected, options);
-  }
-  {
-    Options options;
-    auto args = makeArray("--processes", "100", "hello.zst", "--decompress",
-                          "--force");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    Options expected = {100,  23,       3,    true, {"hello.zst"}, "", true,
-                        true, autoMode, true, 2};
-    EXPECT_EQ(expected, options);
-  }
-  {
-    Options options;
-    auto args = makeArray("x", "-dp", "1", "-c");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    Options expected = {1,     23,   3,        true, {"x"}, "-",
-                        false, true, autoMode, true, 2};
-    EXPECT_EQ(expected, options);
-  }
-  {
-    Options options;
-    auto args = makeArray("x", "-dp", "1", "--stdout");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    Options expected = {1,     23,   3,        true, {"x"}, "-",
-                        false, true, autoMode, true, 2};
-    EXPECT_EQ(expected, options);
-  }
-  {
-    Options options;
-    auto args = makeArray("-p", "1", "x", "-5", "-fo", "-", "--ultra", "-d");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    Options expected = {1,    0,    5,        true, {"x"}, "-",
-                        true, true, autoMode, true, 2};
-    EXPECT_EQ(expected, options);
-  }
-  {
-    Options options;
-    auto args = makeArray("silesia.tar", "-o", "silesia.tar.pzstd", "-p", "2");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    Options expected = {2,
-                        23,
-                        3,
-                        false,
-                        {"silesia.tar"},
-                        "silesia.tar.pzstd",
-                        false,
-                        true,
-                        autoMode,
-                        true,
-                        2};
-    EXPECT_EQ(expected, options);
-  }
-  {
-    Options options;
-    auto args = makeArray("x", "-p", "1");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-  }
-  {
-    Options options;
-    auto args = makeArray("x", "-p", "1");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-  }
-}
-
-TEST(Options, GetOutputFile) {
-  {
-    Options options;
-    auto args = makeArray("x");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    EXPECT_EQ("x.zst", options.getOutputFile(options.inputFiles[0]));
-  }
-  {
-    Options options;
-    auto args = makeArray("x", "y", "-o", nullOutput);
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    EXPECT_EQ(nullOutput, options.getOutputFile(options.inputFiles[0]));
-  }
-  {
-    Options options;
-    auto args = makeArray("x.zst", "-do", nullOutput);
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    EXPECT_EQ(nullOutput, options.getOutputFile(options.inputFiles[0]));
-  }
-  {
-    Options options;
-    auto args = makeArray("x.zst", "-d");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    EXPECT_EQ("x", options.getOutputFile(options.inputFiles[0]));
-  }
-  {
-    Options options;
-    auto args = makeArray("xzst", "-d");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    EXPECT_EQ("", options.getOutputFile(options.inputFiles[0]));
-  }
-  {
-    Options options;
-    auto args = makeArray("xzst", "-doxx");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    EXPECT_EQ("xx", options.getOutputFile(options.inputFiles[0]));
-  }
-}
-
-TEST(Options, MultipleFiles) {
-  {
-    Options options;
-    auto args = makeArray("x", "y", "z");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    Options expected;
-    expected.inputFiles = {"x", "y", "z"};
-    expected.verbosity = 1;
-    EXPECT_EQ(expected, options);
-  }
-  {
-    Options options;
-    auto args = makeArray("x", "y", "z", "-o", nullOutput);
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    Options expected;
-    expected.inputFiles = {"x", "y", "z"};
-    expected.outputFile = nullOutput;
-    expected.verbosity = 1;
-    EXPECT_EQ(expected, options);
-  }
-  {
-    Options options;
-    auto args = makeArray("x", "y", "-o-");
-    EXPECT_FAILURE(options.parse(args.size(), args.data()));
-  }
-  {
-    Options options;
-    auto args = makeArray("x", "y", "-o", "file");
-    EXPECT_FAILURE(options.parse(args.size(), args.data()));
-  }
-  {
-    Options options;
-    auto args = makeArray("-qqvd12qp4", "-f", "x", "--", "--rm", "-c");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    Options expected = {4,  23,   12,   true,     {"x", "--rm", "-c"},
-                        "", true, true, autoMode, true,
-                        0};
-    EXPECT_EQ(expected, options);
-  }
-}
-
-TEST(Options, NumThreads) {
-  {
-    Options options;
-    auto args = makeArray("x", "-dfo", "-");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-  }
-  {
-    Options options;
-    auto args = makeArray("x", "-p", "0", "-fo", "-");
-    EXPECT_FAILURE(options.parse(args.size(), args.data()));
-  }
-  {
-    Options options;
-    auto args = makeArray("-f", "-p", "-o", "-");
-    EXPECT_FAILURE(options.parse(args.size(), args.data()));
-  }
-}
-
-TEST(Options, BadCompressionLevel) {
-  {
-    Options options;
-    auto args = makeArray("x", "-20");
-    EXPECT_FAILURE(options.parse(args.size(), args.data()));
-  }
-  {
-    Options options;
-    auto args = makeArray("x", "--ultra", "-23");
-    EXPECT_FAILURE(options.parse(args.size(), args.data()));
-  }
-  {
-    Options options;
-    auto args = makeArray("x", "--1"); // negative 1?
-    EXPECT_FAILURE(options.parse(args.size(), args.data()));
-  }
-}
-
-TEST(Options, InvalidOption) {
-  {
-    Options options;
-    auto args = makeArray("x", "-x");
-    EXPECT_FAILURE(options.parse(args.size(), args.data()));
-  }
-}
-
-TEST(Options, BadOutputFile) {
-  {
-    Options options;
-    auto args = makeArray("notzst", "-d", "-p", "1");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    EXPECT_EQ("", options.getOutputFile(options.inputFiles.front()));
-  }
-}
-
-TEST(Options, BadOptionsWithArguments) {
-  {
-    Options options;
-    auto args = makeArray("x", "-pf");
-    EXPECT_FAILURE(options.parse(args.size(), args.data()));
-  }
-  {
-    Options options;
-    auto args = makeArray("x", "-p", "10f");
-    EXPECT_FAILURE(options.parse(args.size(), args.data()));
-  }
-  {
-    Options options;
-    auto args = makeArray("x", "-p");
-    EXPECT_FAILURE(options.parse(args.size(), args.data()));
-  }
-  {
-    Options options;
-    auto args = makeArray("x", "-o");
-    EXPECT_FAILURE(options.parse(args.size(), args.data()));
-  }
-  {
-    Options options;
-    auto args = makeArray("x", "-o");
-    EXPECT_FAILURE(options.parse(args.size(), args.data()));
-  }
-}
-
-TEST(Options, KeepSource) {
-  {
-    Options options;
-    auto args = makeArray("x", "--rm", "-k");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    EXPECT_EQ(true, options.keepSource);
-  }
-  {
-    Options options;
-    auto args = makeArray("x", "--rm", "--keep");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    EXPECT_EQ(true, options.keepSource);
-  }
-  {
-    Options options;
-    auto args = makeArray("x");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    EXPECT_EQ(true, options.keepSource);
-  }
-  {
-    Options options;
-    auto args = makeArray("x", "--rm");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    EXPECT_EQ(false, options.keepSource);
-  }
-}
-
-TEST(Options, Verbosity) {
-  {
-    Options options;
-    auto args = makeArray("x");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    EXPECT_EQ(2, options.verbosity);
-  }
-  {
-    Options options;
-    auto args = makeArray("--quiet", "-qq", "x");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    EXPECT_EQ(-1, options.verbosity);
-  }
-  {
-    Options options;
-    auto args = makeArray("x", "y");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    EXPECT_EQ(1, options.verbosity);
-  }
-  {
-    Options options;
-    auto args = makeArray("--", "x", "y");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    EXPECT_EQ(1, options.verbosity);
-  }
-  {
-    Options options;
-    auto args = makeArray("-qv", "x", "y");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    EXPECT_EQ(1, options.verbosity);
-  }
-  {
-    Options options;
-    auto args = makeArray("-v", "x", "y");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    EXPECT_EQ(3, options.verbosity);
-  }
-  {
-    Options options;
-    auto args = makeArray("-v", "x");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    EXPECT_EQ(3, options.verbosity);
-  }
-}
-
-TEST(Options, TestMode) {
-  {
-    Options options;
-    auto args = makeArray("x", "-t");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    EXPECT_EQ(true, options.keepSource);
-    EXPECT_EQ(true, options.decompress);
-    EXPECT_EQ(nullOutput, options.outputFile);
-  }
-  {
-    Options options;
-    auto args = makeArray("x", "--test", "--rm", "-ohello");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    EXPECT_EQ(true, options.keepSource);
-    EXPECT_EQ(true, options.decompress);
-    EXPECT_EQ(nullOutput, options.outputFile);
-  }
-}
-
-TEST(Options, Checksum) {
-  {
-    Options options;
-    auto args = makeArray("x.zst", "--no-check", "-Cd");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    EXPECT_EQ(true, options.checksum);
-  }
-  {
-    Options options;
-    auto args = makeArray("x");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    EXPECT_EQ(true, options.checksum);
-  }
-  {
-    Options options;
-    auto args = makeArray("x", "--no-check", "--check");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    EXPECT_EQ(true, options.checksum);
-  }
-  {
-    Options options;
-    auto args = makeArray("x", "--no-check");
-    EXPECT_SUCCESS(options.parse(args.size(), args.data()));
-    EXPECT_EQ(false, options.checksum);
-  }
-}
-
-TEST(Options, InputFiles) {
-  {
-    Options options;
-    auto args = makeArray("-cd");
-    options.parse(args.size(), args.data());
-    EXPECT_EQ(1, options.inputFiles.size());
-    EXPECT_EQ("-", options.inputFiles[0]);
-    EXPECT_EQ("-", options.outputFile);
-  }
-  {
-    Options options;
-    auto args = makeArray();
-    options.parse(args.size(), args.data());
-    EXPECT_EQ(1, options.inputFiles.size());
-    EXPECT_EQ("-", options.inputFiles[0]);
-    EXPECT_EQ("-", options.outputFile);
-  }
-  {
-    Options options;
-    auto args = makeArray("-d");
-    options.parse(args.size(), args.data());
-    EXPECT_EQ(1, options.inputFiles.size());
-    EXPECT_EQ("-", options.inputFiles[0]);
-    EXPECT_EQ("-", options.outputFile);
-  }
-  {
-    Options options;
-    auto args = makeArray("x", "-");
-    EXPECT_FAILURE(options.parse(args.size(), args.data()));
-  }
-}
-
-TEST(Options, InvalidOptions) {
-  {
-    Options options;
-    auto args = makeArray("-ibasdf");
-    EXPECT_FAILURE(options.parse(args.size(), args.data()));
-  }
-  {
-    Options options;
-    auto args = makeArray("- ");
-    EXPECT_FAILURE(options.parse(args.size(), args.data()));
-  }
-  {
-    Options options;
-    auto args = makeArray("-n15");
-    EXPECT_FAILURE(options.parse(args.size(), args.data()));
-  }
-  {
-    Options options;
-    auto args = makeArray("-0", "x");
-    EXPECT_FAILURE(options.parse(args.size(), args.data()));
-  }
-}
-
-TEST(Options, Extras) {
-  {
-    Options options;
-    auto args = makeArray("-h");
-    EXPECT_MESSAGE(options.parse(args.size(), args.data()));
-  }
-  {
-    Options options;
-    auto args = makeArray("-H");
-    EXPECT_MESSAGE(options.parse(args.size(), args.data()));
-  }
-  {
-    Options options;
-    auto args = makeArray("-V");
-    EXPECT_MESSAGE(options.parse(args.size(), args.data()));
-  }
-  {
-    Options options;
-    auto args = makeArray("--help");
-    EXPECT_MESSAGE(options.parse(args.size(), args.data()));
-  }
-  {
-    Options options;
-    auto args = makeArray("--version");
-    EXPECT_MESSAGE(options.parse(args.size(), args.data()));
-  }
-}
diff --git a/contrib/pzstd/test/PzstdTest.cpp b/contrib/pzstd/test/PzstdTest.cpp
deleted file mode 100644
index 5c7d66310805..000000000000
--- a/contrib/pzstd/test/PzstdTest.cpp
+++ /dev/null
@@ -1,149 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-#include "Pzstd.h"
-extern "C" {
-#include "datagen.h"
-}
-#include "test/RoundTrip.h"
-#include "utils/ScopeGuard.h"
-
-#include <cstddef>
-#include <cstdio>
-#include <gtest/gtest.h>
-#include <memory>
-#include <random>
-
-using namespace std;
-using namespace pzstd;
-
-TEST(Pzstd, SmallSizes) {
-  unsigned seed = std::random_device{}();
-  std::fprintf(stderr, "Pzstd.SmallSizes seed: %u\n", seed);
-  std::mt19937 gen(seed);
-
-  for (unsigned len = 1; len < 256; ++len) {
-    if (len % 16 == 0) {
-      std::fprintf(stderr, "%u / 16\n", len / 16);
-    }
-    std::string inputFile = std::tmpnam(nullptr);
-    auto guard = makeScopeGuard([&] { std::remove(inputFile.c_str()); });
-    {
-      static uint8_t buf[256];
-      RDG_genBuffer(buf, len, 0.5, 0.0, gen());
-      auto fd = std::fopen(inputFile.c_str(), "wb");
-      auto written = std::fwrite(buf, 1, len, fd);
-      std::fclose(fd);
-      ASSERT_EQ(written, len);
-    }
-    for (unsigned numThreads = 1; numThreads <= 2; ++numThreads) {
-      for (unsigned level = 1; level <= 4; level *= 4) {
-        auto errorGuard = makeScopeGuard([&] {
-          std::fprintf(stderr, "# threads: %u\n", numThreads);
-          std::fprintf(stderr, "compression level: %u\n", level);
-        });
-        Options options;
-        options.overwrite = true;
-        options.inputFiles = {inputFile};
-        options.numThreads = numThreads;
-        options.compressionLevel = level;
-        options.verbosity = 1;
-        ASSERT_TRUE(roundTrip(options));
-        errorGuard.dismiss();
-      }
-    }
-  }
-}
-
-TEST(Pzstd, LargeSizes) {
-  unsigned seed = std::random_device{}();
-  std::fprintf(stderr, "Pzstd.LargeSizes seed: %u\n", seed);
-  std::mt19937 gen(seed);
-
-  for (unsigned len = 1 << 20; len <= (1 << 24); len *= 2) {
-    std::string inputFile = std::tmpnam(nullptr);
-    auto guard = makeScopeGuard([&] { std::remove(inputFile.c_str()); });
-    {
-      std::unique_ptr<uint8_t[]> buf(new uint8_t[len]);
-      RDG_genBuffer(buf.get(), len, 0.5, 0.0, gen());
-      auto fd = std::fopen(inputFile.c_str(), "wb");
-      auto written = std::fwrite(buf.get(), 1, len, fd);
-      std::fclose(fd);
-      ASSERT_EQ(written, len);
-    }
-    for (unsigned numThreads = 1; numThreads <= 16; numThreads *= 4) {
-      for (unsigned level = 1; level <= 4; level *= 4) {
-        auto errorGuard = makeScopeGuard([&] {
-          std::fprintf(stderr, "# threads: %u\n", numThreads);
-          std::fprintf(stderr, "compression level: %u\n", level);
-        });
-        Options options;
-        options.overwrite = true;
-        options.inputFiles = {inputFile};
-        options.numThreads = std::min(numThreads, options.numThreads);
-        options.compressionLevel = level;
-        options.verbosity = 1;
-        ASSERT_TRUE(roundTrip(options));
-        errorGuard.dismiss();
-      }
-    }
-  }
-}
-
-TEST(Pzstd, DISABLED_ExtremelyLargeSize) {
-  unsigned seed = std::random_device{}();
-  std::fprintf(stderr, "Pzstd.ExtremelyLargeSize seed: %u\n", seed);
-  std::mt19937 gen(seed);
-
-  std::string inputFile = std::tmpnam(nullptr);
-  auto guard = makeScopeGuard([&] { std::remove(inputFile.c_str()); });
-
-  {
-    // Write 4GB + 64 MB
-    constexpr size_t kLength = 1 << 26;
-    std::unique_ptr<uint8_t[]> buf(new uint8_t[kLength]);
-    auto fd = std::fopen(inputFile.c_str(), "wb");
-    auto closeGuard = makeScopeGuard([&] { std::fclose(fd); });
-    for (size_t i = 0; i < (1 << 6) + 1; ++i) {
-      RDG_genBuffer(buf.get(), kLength, 0.5, 0.0, gen());
-      auto written = std::fwrite(buf.get(), 1, kLength, fd);
-      if (written != kLength) {
-        std::fprintf(stderr, "Failed to write file, skipping test\n");
-        return;
-      }
-    }
-  }
-
-  Options options;
-  options.overwrite = true;
-  options.inputFiles = {inputFile};
-  options.compressionLevel = 1;
-  if (options.numThreads == 0) {
-    options.numThreads = 1;
-  }
-  ASSERT_TRUE(roundTrip(options));
-}
-
-TEST(Pzstd, ExtremelyCompressible) {
-  std::string inputFile = std::tmpnam(nullptr);
-  auto guard = makeScopeGuard([&] { std::remove(inputFile.c_str()); });
-  {
-    std::unique_ptr<uint8_t[]> buf(new uint8_t[10000]);
-    std::memset(buf.get(), 'a', 10000);
-    auto fd = std::fopen(inputFile.c_str(), "wb");
-    auto written = std::fwrite(buf.get(), 1, 10000, fd);
-    std::fclose(fd);
-    ASSERT_EQ(written, 10000);
-  }
-  Options options;
-  options.overwrite = true;
-  options.inputFiles = {inputFile};
-  options.numThreads = 1;
-  options.compressionLevel = 1;
-  ASSERT_TRUE(roundTrip(options));
-}
diff --git a/contrib/pzstd/test/RoundTrip.h b/contrib/pzstd/test/RoundTrip.h
deleted file mode 100644
index c6364ecb4227..000000000000
--- a/contrib/pzstd/test/RoundTrip.h
+++ /dev/null
@@ -1,86 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-#pragma once
-
-#include "Options.h"
-#include "Pzstd.h"
-#include "utils/ScopeGuard.h"
-
-#include <cstdio>
-#include <string>
-#include <cstdint>
-#include <memory>
-
-namespace pzstd {
-
-inline bool check(std::string source, std::string decompressed) {
-  std::unique_ptr<std::uint8_t[]> sBuf(new std::uint8_t[1024]);
-  std::unique_ptr<std::uint8_t[]> dBuf(new std::uint8_t[1024]);
-
-  auto sFd = std::fopen(source.c_str(), "rb");
-  auto dFd = std::fopen(decompressed.c_str(), "rb");
-  auto guard = makeScopeGuard([&] {
-    std::fclose(sFd);
-    std::fclose(dFd);
-  });
-
-  size_t sRead, dRead;
-
-  do {
-    sRead = std::fread(sBuf.get(), 1, 1024, sFd);
-    dRead = std::fread(dBuf.get(), 1, 1024, dFd);
-    if (std::ferror(sFd) || std::ferror(dFd)) {
-      return false;
-    }
-    if (sRead != dRead) {
-      return false;
-    }
-
-    for (size_t i = 0; i < sRead; ++i) {
-      if (sBuf.get()[i] != dBuf.get()[i]) {
-        return false;
-      }
-    }
-  } while (sRead == 1024);
-  if (!std::feof(sFd) || !std::feof(dFd)) {
-    return false;
-  }
-  return true;
-}
-
-inline bool roundTrip(Options& options) {
-  if (options.inputFiles.size() != 1) {
-    return false;
-  }
-  std::string source = options.inputFiles.front();
-  std::string compressedFile = std::tmpnam(nullptr);
-  std::string decompressedFile = std::tmpnam(nullptr);
-  auto guard = makeScopeGuard([&] {
-    std::remove(compressedFile.c_str());
-    std::remove(decompressedFile.c_str());
-  });
-
-  {
-    options.outputFile = compressedFile;
-    options.decompress = false;
-    if (pzstdMain(options) != 0) {
-      return false;
-    }
-  }
-  {
-    options.decompress = true;
-    options.inputFiles.front() = compressedFile;
-    options.outputFile = decompressedFile;
-    if (pzstdMain(options) != 0) {
-      return false;
-    }
-  }
-  return check(source, decompressedFile);
-}
-}
diff --git a/contrib/pzstd/test/RoundTripTest.cpp b/contrib/pzstd/test/RoundTripTest.cpp
deleted file mode 100644
index 36af0673ae6a..000000000000
--- a/contrib/pzstd/test/RoundTripTest.cpp
+++ /dev/null
@@ -1,86 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-extern "C" {
-#include "datagen.h"
-}
-#include "Options.h"
-#include "test/RoundTrip.h"
-#include "utils/ScopeGuard.h"
-
-#include <cstddef>
-#include <cstdio>
-#include <cstdlib>
-#include <memory>
-#include <random>
-
-using namespace std;
-using namespace pzstd;
-
-namespace {
-string
-writeData(size_t size, double matchProba, double litProba, unsigned seed) {
-  std::unique_ptr<uint8_t[]> buf(new uint8_t[size]);
-  RDG_genBuffer(buf.get(), size, matchProba, litProba, seed);
-  string file = tmpnam(nullptr);
-  auto fd = std::fopen(file.c_str(), "wb");
-  auto guard = makeScopeGuard([&] { std::fclose(fd); });
-  auto bytesWritten = std::fwrite(buf.get(), 1, size, fd);
-  if (bytesWritten != size) {
-    std::abort();
-  }
-  return file;
-}
-
-template <typename Generator>
-string generateInputFile(Generator& gen) {
-  // Use inputs ranging from 1 Byte to 2^16 Bytes
-  std::uniform_int_distribution<size_t> size{1, 1 << 16};
-  std::uniform_real_distribution<> prob{0, 1};
-  return writeData(size(gen), prob(gen), prob(gen), gen());
-}
-
-template <typename Generator>
-Options generateOptions(Generator& gen, const string& inputFile) {
-  Options options;
-  options.inputFiles = {inputFile};
-  options.overwrite = true;
-
-  std::uniform_int_distribution<unsigned> numThreads{1, 32};
-  std::uniform_int_distribution<unsigned> compressionLevel{1, 10};
-
-  options.numThreads = numThreads(gen);
-  options.compressionLevel = compressionLevel(gen);
-
-  return options;
-}
-}
-
-int main() {
-  std::mt19937 gen(std::random_device{}());
-
-  auto newlineGuard = makeScopeGuard([] { std::fprintf(stderr, "\n"); });
-  for (unsigned i = 0; i < 10000; ++i) {
-    if (i % 100 == 0) {
-      std::fprintf(stderr, "Progress: %u%%\r", i / 100);
-    }
-    auto inputFile = generateInputFile(gen);
-    auto inputGuard = makeScopeGuard([&] { std::remove(inputFile.c_str()); });
-    for (unsigned i = 0; i < 10; ++i) {
-      auto options = generateOptions(gen, inputFile);
-      if (!roundTrip(options)) {
-        std::fprintf(stderr, "numThreads: %u\n", options.numThreads);
-        std::fprintf(stderr, "level: %u\n", options.compressionLevel);
-        std::fprintf(stderr, "decompress? %u\n", (unsigned)options.decompress);
-        std::fprintf(stderr, "file: %s\n", inputFile.c_str());
-        return 1;
-      }
-    }
-  }
-  return 0;
-}
diff --git a/contrib/pzstd/utils/BUCK b/contrib/pzstd/utils/BUCK
deleted file mode 100644
index e757f412070b..000000000000
--- a/contrib/pzstd/utils/BUCK
+++ /dev/null
@@ -1,75 +0,0 @@
-cxx_library(
-    name='buffer',
-    visibility=['PUBLIC'],
-    header_namespace='utils',
-    exported_headers=['Buffer.h'],
-    deps=[':range'],
-)
-
-cxx_library(
-    name='file_system',
-    visibility=['PUBLIC'],
-    header_namespace='utils',
-    exported_headers=['FileSystem.h'],
-    deps=[':range'],
-)
-
-cxx_library(
-    name='likely',
-    visibility=['PUBLIC'],
-    header_namespace='utils',
-    exported_headers=['Likely.h'],
-)
-
-cxx_library(
-    name='range',
-    visibility=['PUBLIC'],
-    header_namespace='utils',
-    exported_headers=['Range.h'],
-    deps=[':likely'],
-)
-
-cxx_library(
-    name='resource_pool',
-    visibility=['PUBLIC'],
-    header_namespace='utils',
-    exported_headers=['ResourcePool.h'],
-)
-
-cxx_library(
-    name='scope_guard',
-    visibility=['PUBLIC'],
-    header_namespace='utils',
-    exported_headers=['ScopeGuard.h'],
-)
-
-cxx_library(
-    name='thread_pool',
-    visibility=['PUBLIC'],
-    header_namespace='utils',
-    exported_headers=['ThreadPool.h'],
-    deps=[':work_queue'],
-)
-
-cxx_library(
-    name='work_queue',
-    visibility=['PUBLIC'],
-    header_namespace='utils',
-    exported_headers=['WorkQueue.h'],
-    deps=[':buffer'],
-)
-
-cxx_library(
-    name='utils',
-    visibility=['PUBLIC'],
-    deps=[
-        ':buffer',
-        ':file_system',
-        ':likely',
-        ':range',
-        ':resource_pool',
-        ':scope_guard',
-        ':thread_pool',
-        ':work_queue',
-    ],
-)
diff --git a/contrib/pzstd/utils/Buffer.h b/contrib/pzstd/utils/Buffer.h
deleted file mode 100644
index f69c3b4d9f7a..000000000000
--- a/contrib/pzstd/utils/Buffer.h
+++ /dev/null
@@ -1,99 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-#pragma once
-
-#include "utils/Range.h"
-
-#include <array>
-#include <cstddef>
-#include <memory>
-
-namespace pzstd {
-
-/**
- * A `Buffer` has a pointer to a shared buffer, and a range of the buffer that
- * it owns.
- * The idea is that you can allocate one buffer, and write chunks into it
- * and break off those chunks.
- * The underlying buffer is reference counted, and will be destroyed when all
- * `Buffer`s that reference it are destroyed.
- */
-class Buffer {
-  std::shared_ptr<unsigned char> buffer_;
-  MutableByteRange range_;
-
-  static void delete_buffer(unsigned char* buffer) {
-    delete[] buffer;
-  }
-
- public:
-  /// Construct an empty buffer that owns no data.
-  explicit Buffer() {}
-
-  /// Construct a `Buffer` that owns a new underlying buffer of size `size`.
-  explicit Buffer(std::size_t size)
-      : buffer_(new unsigned char[size], delete_buffer),
-        range_(buffer_.get(), buffer_.get() + size) {}
-
-  explicit Buffer(std::shared_ptr<unsigned char> buffer, MutableByteRange data)
-      : buffer_(buffer), range_(data) {}
-
-  Buffer(Buffer&&) = default;
-  Buffer& operator=(Buffer&&) & = default;
-
-  /**
-   * Splits the data into two pieces: [begin, begin + n), [begin + n, end).
-   * Their data both points into the same underlying buffer.
-   * Modifies the original `Buffer` to point to only [begin + n, end).
-   *
-   * @param n  The offset to split at.
-   * @returns  A buffer that owns the data [begin, begin + n).
-   */
-  Buffer splitAt(std::size_t n) {
-    auto firstPiece = range_.subpiece(0, n);
-    range_.advance(n);
-    return Buffer(buffer_, firstPiece);
-  }
-
-  /// Modifies the buffer to point to the range [begin + n, end).
-  void advance(std::size_t n) {
-    range_.advance(n);
-  }
-
-  /// Modifies the buffer to point to the range [begin, end - n).
-  void subtract(std::size_t n) {
-    range_.subtract(n);
-  }
-
-  /// Returns a read only `Range` pointing to the `Buffer`s data.
-  ByteRange range() const {
-    return range_;
-  }
-  /// Returns a mutable `Range` pointing to the `Buffer`s data.
-  MutableByteRange range() {
-    return range_;
-  }
-
-  const unsigned char* data() const {
-    return range_.data();
-  }
-
-  unsigned char* data() {
-    return range_.data();
-  }
-
-  std::size_t size() const {
-    return range_.size();
-  }
-
-  bool empty() const {
-    return range_.empty();
-  }
-};
-}
diff --git a/contrib/pzstd/utils/FileSystem.h b/contrib/pzstd/utils/FileSystem.h
deleted file mode 100644
index 3cfbe86e507e..000000000000
--- a/contrib/pzstd/utils/FileSystem.h
+++ /dev/null
@@ -1,94 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-#pragma once
-
-#include "utils/Range.h"
-
-#include <sys/stat.h>
-#include <cerrno>
-#include <cstdint>
-#include <system_error>
-
-// A small subset of `std::filesystem`.
-// `std::filesystem` should be a drop in replacement.
-// See http://en.cppreference.com/w/cpp/filesystem for documentation.
-
-namespace pzstd {
-
-// using file_status = ... causes gcc to emit a false positive warning
-#if defined(_MSC_VER)
-typedef struct ::_stat64 file_status;
-#else
-typedef struct ::stat file_status;
-#endif
-
-/// http://en.cppreference.com/w/cpp/filesystem/status
-inline file_status status(StringPiece path, std::error_code& ec) noexcept {
-  file_status status;
-#if defined(_MSC_VER)
-  const auto error = ::_stat64(path.data(), &status);
-#else
-  const auto error = ::stat(path.data(), &status);
-#endif
-  if (error) {
-    ec.assign(errno, std::generic_category());
-  } else {
-    ec.clear();
-  }
-  return status;
-}
-
-/// http://en.cppreference.com/w/cpp/filesystem/is_regular_file
-inline bool is_regular_file(file_status status) noexcept {
-#if defined(S_ISREG)
-  return S_ISREG(status.st_mode);
-#elif !defined(S_ISREG) && defined(S_IFMT) && defined(S_IFREG)
-  return (status.st_mode & S_IFMT) == S_IFREG;
-#else
-  static_assert(false, "No POSIX stat() support.");
-#endif
-}
-
-/// http://en.cppreference.com/w/cpp/filesystem/is_regular_file
-inline bool is_regular_file(StringPiece path, std::error_code& ec) noexcept {
-  return is_regular_file(status(path, ec));
-}
-
-/// http://en.cppreference.com/w/cpp/filesystem/is_directory
-inline bool is_directory(file_status status) noexcept {
-#if defined(S_ISDIR)
-  return S_ISDIR(status.st_mode);
-#elif !defined(S_ISDIR) && defined(S_IFMT) && defined(S_IFDIR)
-  return (status.st_mode & S_IFMT) == S_IFDIR;
-#else
-  static_assert(false, "NO POSIX stat() support.");
-#endif
-}
-
-/// http://en.cppreference.com/w/cpp/filesystem/is_directory
-inline bool is_directory(StringPiece path, std::error_code& ec) noexcept {
-  return is_directory(status(path, ec));
-}
-
-/// http://en.cppreference.com/w/cpp/filesystem/file_size
-inline std::uintmax_t file_size(
-    StringPiece path,
-    std::error_code& ec) noexcept {
-  auto stat = status(path, ec);
-  if (ec) {
-    return -1;
-  }
-  if (!is_regular_file(stat)) {
-    ec.assign(ENOTSUP, std::generic_category());
-    return -1;
-  }
-  ec.clear();
-  return stat.st_size;
-}
-}
diff --git a/contrib/pzstd/utils/Likely.h b/contrib/pzstd/utils/Likely.h
deleted file mode 100644
index 7cea8da2771f..000000000000
--- a/contrib/pzstd/utils/Likely.h
+++ /dev/null
@@ -1,28 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
- 
-/**
- * Compiler hints to indicate the fast path of an "if" branch: whether
- * the if condition is likely to be true or false.
- *
- * @author Tudor Bosman (tudorb@fb.com)
- */
-
-#pragma once
-
-#undef LIKELY
-#undef UNLIKELY
-
-#if defined(__GNUC__) && __GNUC__ >= 4
-#define LIKELY(x) (__builtin_expect((x), 1))
-#define UNLIKELY(x) (__builtin_expect((x), 0))
-#else
-#define LIKELY(x) (x)
-#define UNLIKELY(x) (x)
-#endif
diff --git a/contrib/pzstd/utils/Range.h b/contrib/pzstd/utils/Range.h
deleted file mode 100644
index fedb5d786c68..000000000000
--- a/contrib/pzstd/utils/Range.h
+++ /dev/null
@@ -1,131 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
- 
-/**
- * A subset of `folly/Range.h`.
- * All code copied verbatim modulo formatting
- */
-#pragma once
-
-#include "utils/Likely.h"
-
-#include <cstddef>
-#include <cstring>
-#include <stdexcept>
-#include <string>
-#include <type_traits>
-
-namespace pzstd {
-
-namespace detail {
-/*
- *Use IsCharPointer<T>::type to enable const char* or char*.
- *Use IsCharPointer<T>::const_type to enable only const char*.
-*/
-template <class T>
-struct IsCharPointer {};
-
-template <>
-struct IsCharPointer<char*> {
-  typedef int type;
-};
-
-template <>
-struct IsCharPointer<const char*> {
-  typedef int const_type;
-  typedef int type;
-};
-
-} // namespace detail
-
-template <typename Iter>
-class Range {
-  Iter b_;
-  Iter e_;
-
- public:
-  using size_type = std::size_t;
-  using iterator = Iter;
-  using const_iterator = Iter;
-  using value_type = typename std::remove_reference<
-      typename std::iterator_traits<Iter>::reference>::type;
-  using reference = typename std::iterator_traits<Iter>::reference;
-
-  constexpr Range() : b_(), e_() {}
-  constexpr Range(Iter begin, Iter end) : b_(begin), e_(end) {}
-
-  constexpr Range(Iter begin, size_type size) : b_(begin), e_(begin + size) {}
-
-  template <class T = Iter, typename detail::IsCharPointer<T>::type = 0>
-  /* implicit */ Range(Iter str) : b_(str), e_(str + std::strlen(str)) {}
-
-  template <class T = Iter, typename detail::IsCharPointer<T>::const_type = 0>
-  /* implicit */ Range(const std::string& str)
-      : b_(str.data()), e_(b_ + str.size()) {}
-
-  // Allow implicit conversion from Range<From> to Range<To> if From is
-  // implicitly convertible to To.
-  template <
-      class OtherIter,
-      typename std::enable_if<
-          (!std::is_same<Iter, OtherIter>::value &&
-           std::is_convertible<OtherIter, Iter>::value),
-          int>::type = 0>
-  constexpr /* implicit */ Range(const Range<OtherIter>& other)
-      : b_(other.begin()), e_(other.end()) {}
-
-  Range(const Range&) = default;
-  Range(Range&&) = default;
-
-  Range& operator=(const Range&) & = default;
-  Range& operator=(Range&&) & = default;
-
-  constexpr size_type size() const {
-    return e_ - b_;
-  }
-  bool empty() const {
-    return b_ == e_;
-  }
-  Iter data() const {
-    return b_;
-  }
-  Iter begin() const {
-    return b_;
-  }
-  Iter end() const {
-    return e_;
-  }
-
-  void advance(size_type n) {
-    if (UNLIKELY(n > size())) {
-      throw std::out_of_range("index out of range");
-    }
-    b_ += n;
-  }
-
-  void subtract(size_type n) {
-    if (UNLIKELY(n > size())) {
-      throw std::out_of_range("index out of range");
-    }
-    e_ -= n;
-  }
-
-  Range subpiece(size_type first, size_type length = std::string::npos) const {
-    if (UNLIKELY(first > size())) {
-      throw std::out_of_range("index out of range");
-    }
-
-    return Range(b_ + first, std::min(length, size() - first));
-  }
-};
-
-using ByteRange = Range<const unsigned char*>;
-using MutableByteRange = Range<unsigned char*>;
-using StringPiece = Range<const char*>;
-}
diff --git a/contrib/pzstd/utils/ResourcePool.h b/contrib/pzstd/utils/ResourcePool.h
deleted file mode 100644
index 8dfcdd765909..000000000000
--- a/contrib/pzstd/utils/ResourcePool.h
+++ /dev/null
@@ -1,96 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-#pragma once
-
-#include <cassert>
-#include <functional>
-#include <memory>
-#include <mutex>
-#include <vector>
-
-namespace pzstd {
-
-/**
- * An unbounded pool of resources.
- * A `ResourcePool<T>` requires a factory function that takes allocates `T*` and
- * a free function that frees a `T*`.
- * Calling `ResourcePool::get()` will give you a new `ResourcePool::UniquePtr`
- * to a `T`, and when it goes out of scope the resource will be returned to the
- * pool.
- * The `ResourcePool<T>` *must* survive longer than any resources it hands out.
- * Remember that `ResourcePool<T>` hands out mutable `T`s, so make sure to clean
- * up the resource before or after every use.
- */
-template <typename T>
-class ResourcePool {
- public:
-  class Deleter;
-  using Factory = std::function<T*()>;
-  using Free = std::function<void(T*)>;
-  using UniquePtr = std::unique_ptr<T, Deleter>;
-
- private:
-  std::mutex mutex_;
-  Factory factory_;
-  Free free_;
-  std::vector<T*> resources_;
-  unsigned inUse_;
-
- public:
-  /**
-   * Creates a `ResourcePool`.
-   *
-   * @param factory  The function to use to create new resources.
-   * @param free     The function to use to free resources created by `factory`.
-   */
-  ResourcePool(Factory factory, Free free)
-      : factory_(std::move(factory)), free_(std::move(free)), inUse_(0) {}
-
-  /**
-   * @returns  A unique pointer to a resource.  The resource is null iff
-   *           there are no available resources and `factory()` returns null.
-   */
-  UniquePtr get() {
-    std::lock_guard<std::mutex> lock(mutex_);
-    if (!resources_.empty()) {
-      UniquePtr resource{resources_.back(), Deleter{*this}};
-      resources_.pop_back();
-      ++inUse_;
-      return resource;
-    }
-    UniquePtr resource{factory_(), Deleter{*this}};
-    ++inUse_;
-    return resource;
-  }
-
-  ~ResourcePool() noexcept {
-    assert(inUse_ == 0);
-    for (const auto resource : resources_) {
-      free_(resource);
-    }
-  }
-
-  class Deleter {
-    ResourcePool *pool_;
-  public:
-    explicit Deleter(ResourcePool &pool) : pool_(&pool) {}
-
-    void operator() (T *resource) {
-      std::lock_guard<std::mutex> lock(pool_->mutex_);
-      // Make sure we don't put null resources into the pool
-      if (resource) {
-        pool_->resources_.push_back(resource);
-      }
-      assert(pool_->inUse_ > 0);
-      --pool_->inUse_;
-    }
-  };
-};
-
-}
diff --git a/contrib/pzstd/utils/ScopeGuard.h b/contrib/pzstd/utils/ScopeGuard.h
deleted file mode 100644
index 31768f43d22c..000000000000
--- a/contrib/pzstd/utils/ScopeGuard.h
+++ /dev/null
@@ -1,50 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-#pragma once
-
-#include <utility>
-
-namespace pzstd {
-
-/**
- * Dismissable scope guard.
- * `Function` must be callable and take no parameters.
- * Unless `dissmiss()` is called, the callable is executed upon destruction of
- * `ScopeGuard`.
- *
- * Example:
- *
- *   auto guard = makeScopeGuard([&] { cleanup(); });
- */
-template <typename Function>
-class ScopeGuard {
-  Function function;
-  bool dismissed;
-
- public:
-  explicit ScopeGuard(Function&& function)
-      : function(std::move(function)), dismissed(false) {}
-
-  void dismiss() {
-    dismissed = true;
-  }
-
-  ~ScopeGuard() noexcept {
-    if (!dismissed) {
-      function();
-    }
-  }
-};
-
-/// Creates a scope guard from `function`.
-template <typename Function>
-ScopeGuard<Function> makeScopeGuard(Function&& function) {
-  return ScopeGuard<Function>(std::forward<Function>(function));
-}
-}
diff --git a/contrib/pzstd/utils/ThreadPool.h b/contrib/pzstd/utils/ThreadPool.h
deleted file mode 100644
index 8ece8e0da4eb..000000000000
--- a/contrib/pzstd/utils/ThreadPool.h
+++ /dev/null
@@ -1,58 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-#pragma once
-
-#include "utils/WorkQueue.h"
-
-#include <cstddef>
-#include <functional>
-#include <thread>
-#include <vector>
-
-namespace pzstd {
-/// A simple thread pool that pulls tasks off its queue in FIFO order.
-class ThreadPool {
-  std::vector<std::thread> threads_;
-
-  WorkQueue<std::function<void()>> tasks_;
-
- public:
-  /// Constructs a thread pool with `numThreads` threads.
-  explicit ThreadPool(std::size_t numThreads) {
-    threads_.reserve(numThreads);
-    for (std::size_t i = 0; i < numThreads; ++i) {
-      threads_.emplace_back([this] {
-        std::function<void()> task;
-        while (tasks_.pop(task)) {
-          task();
-        }
-      });
-    }
-  }
-
-  /// Finishes all tasks currently in the queue.
-  ~ThreadPool() {
-    tasks_.finish();
-    for (auto& thread : threads_) {
-      thread.join();
-    }
-  }
-
-  /**
-   * Adds `task` to the queue of tasks to execute. Since `task` is a
-   * `std::function<>`, it cannot be a move only type. So any lambda passed must
-   * not capture move only types (like `std::unique_ptr`).
-   *
-   * @param task  The task to execute.
-   */
-  void add(std::function<void()> task) {
-    tasks_.push(std::move(task));
-  }
-};
-}
diff --git a/contrib/pzstd/utils/WorkQueue.h b/contrib/pzstd/utils/WorkQueue.h
deleted file mode 100644
index 1d14d922c648..000000000000
--- a/contrib/pzstd/utils/WorkQueue.h
+++ /dev/null
@@ -1,181 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-#pragma once
-
-#include "utils/Buffer.h"
-
-#include <atomic>
-#include <cassert>
-#include <cstddef>
-#include <condition_variable>
-#include <cstddef>
-#include <functional>
-#include <mutex>
-#include <queue>
-
-namespace pzstd {
-
-/// Unbounded thread-safe work queue.
-template <typename T>
-class WorkQueue {
-  // Protects all member variable access
-  std::mutex mutex_;
-  std::condition_variable readerCv_;
-  std::condition_variable writerCv_;
-  std::condition_variable finishCv_;
-
-  std::queue<T> queue_;
-  bool done_;
-  std::size_t maxSize_;
-
-  // Must have lock to call this function
-  bool full() const {
-    if (maxSize_ == 0) {
-      return false;
-    }
-    return queue_.size() >= maxSize_;
-  }
-
- public:
-  /**
-   * Constructs an empty work queue with an optional max size.
-   * If `maxSize == 0` the queue size is unbounded.
-   *
-   * @param maxSize The maximum allowed size of the work queue.
-   */
-  WorkQueue(std::size_t maxSize = 0) : done_(false), maxSize_(maxSize) {}
-
-  /**
-   * Push an item onto the work queue.  Notify a single thread that work is
-   * available.  If `finish()` has been called, do nothing and return false.
-   * If `push()` returns false, then `item` has not been moved from.
-   *
-   * @param item  Item to push onto the queue.
-   * @returns     True upon success, false if `finish()` has been called.  An
-   *               item was pushed iff `push()` returns true.
-   */
-  bool push(T&& item) {
-    {
-      std::unique_lock<std::mutex> lock(mutex_);
-      while (full() && !done_) {
-        writerCv_.wait(lock);
-      }
-      if (done_) {
-        return false;
-      }
-      queue_.push(std::move(item));
-    }
-    readerCv_.notify_one();
-    return true;
-  }
-
-  /**
-   * Attempts to pop an item off the work queue.  It will block until data is
-   * available or `finish()` has been called.
-   *
-   * @param[out] item  If `pop` returns `true`, it contains the popped item.
-   *                    If `pop` returns `false`, it is unmodified.
-   * @returns          True upon success.  False if the queue is empty and
-   *                    `finish()` has been called.
-   */
-  bool pop(T& item) {
-    {
-      std::unique_lock<std::mutex> lock(mutex_);
-      while (queue_.empty() && !done_) {
-        readerCv_.wait(lock);
-      }
-      if (queue_.empty()) {
-        assert(done_);
-        return false;
-      }
-      item = std::move(queue_.front());
-      queue_.pop();
-    }
-    writerCv_.notify_one();
-    return true;
-  }
-
-  /**
-   * Sets the maximum queue size.  If `maxSize == 0` then it is unbounded.
-   *
-   * @param maxSize The new maximum queue size.
-   */
-  void setMaxSize(std::size_t maxSize) {
-    {
-      std::lock_guard<std::mutex> lock(mutex_);
-      maxSize_ = maxSize;
-    }
-    writerCv_.notify_all();
-  }
-
-  /**
-   * Promise that `push()` won't be called again, so once the queue is empty
-   * there will never any more work.
-   */
-  void finish() {
-    {
-      std::lock_guard<std::mutex> lock(mutex_);
-      assert(!done_);
-      done_ = true;
-    }
-    readerCv_.notify_all();
-    writerCv_.notify_all();
-    finishCv_.notify_all();
-  }
-
-  /// Blocks until `finish()` has been called (but the queue may not be empty).
-  void waitUntilFinished() {
-    std::unique_lock<std::mutex> lock(mutex_);
-    while (!done_) {
-      finishCv_.wait(lock);
-    }
-  }
-};
-
-/// Work queue for `Buffer`s that knows the total number of bytes in the queue.
-class BufferWorkQueue {
-  WorkQueue<Buffer> queue_;
-  std::atomic<std::size_t> size_;
-
- public:
-  BufferWorkQueue(std::size_t maxSize = 0) : queue_(maxSize), size_(0) {}
-
-  void push(Buffer buffer) {
-    size_.fetch_add(buffer.size());
-    queue_.push(std::move(buffer));
-  }
-
-  bool pop(Buffer& buffer) {
-    bool result = queue_.pop(buffer);
-    if (result) {
-      size_.fetch_sub(buffer.size());
-    }
-    return result;
-  }
-
-  void setMaxSize(std::size_t maxSize) {
-    queue_.setMaxSize(maxSize);
-  }
-
-  void finish() {
-    queue_.finish();
-  }
-
-  /**
-   * Blocks until `finish()` has been called.
-   *
-   * @returns The total number of bytes of all the `Buffer`s currently in the
-   *           queue.
-   */
-  std::size_t size() {
-    queue_.waitUntilFinished();
-    return size_.load();
-  }
-};
-}
diff --git a/contrib/pzstd/utils/test/BUCK b/contrib/pzstd/utils/test/BUCK
deleted file mode 100644
index a5113cab6b0e..000000000000
--- a/contrib/pzstd/utils/test/BUCK
+++ /dev/null
@@ -1,35 +0,0 @@
-cxx_test(
-    name='buffer_test',
-    srcs=['BufferTest.cpp'],
-    deps=['//contrib/pzstd/utils:buffer'],
-)
-
-cxx_test(
-    name='range_test',
-    srcs=['RangeTest.cpp'],
-    deps=['//contrib/pzstd/utils:range'],
-)
-
-cxx_test(
-    name='resource_pool_test',
-    srcs=['ResourcePoolTest.cpp'],
-    deps=['//contrib/pzstd/utils:resource_pool'],
-)
-
-cxx_test(
-    name='scope_guard_test',
-    srcs=['ScopeGuardTest.cpp'],
-    deps=['//contrib/pzstd/utils:scope_guard'],
-)
-
-cxx_test(
-    name='thread_pool_test',
-    srcs=['ThreadPoolTest.cpp'],
-    deps=['//contrib/pzstd/utils:thread_pool'],
-)
-
-cxx_test(
-    name='work_queue_test',
-    srcs=['RangeTest.cpp'],
-    deps=['//contrib/pzstd/utils:work_queue'],
-)
diff --git a/contrib/pzstd/utils/test/BufferTest.cpp b/contrib/pzstd/utils/test/BufferTest.cpp
deleted file mode 100644
index fbba74e82628..000000000000
--- a/contrib/pzstd/utils/test/BufferTest.cpp
+++ /dev/null
@@ -1,89 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-#include "utils/Buffer.h"
-#include "utils/Range.h"
-
-#include <gtest/gtest.h>
-#include <memory>
-
-using namespace pzstd;
-
-namespace {
-void deleter(const unsigned char* buf) {
-  delete[] buf;
-}
-}
-
-TEST(Buffer, Constructors) {
-  Buffer empty;
-  EXPECT_TRUE(empty.empty());
-  EXPECT_EQ(0, empty.size());
-
-  Buffer sized(5);
-  EXPECT_FALSE(sized.empty());
-  EXPECT_EQ(5, sized.size());
-
-  Buffer moved(std::move(sized));
-  EXPECT_FALSE(sized.empty());
-  EXPECT_EQ(5, sized.size());
-
-  Buffer assigned;
-  assigned = std::move(moved);
-  EXPECT_FALSE(sized.empty());
-  EXPECT_EQ(5, sized.size());
-}
-
-TEST(Buffer, BufferManagement) {
-  std::shared_ptr<unsigned char> buf(new unsigned char[10], deleter);
-  {
-    Buffer acquired(buf, MutableByteRange(buf.get(), buf.get() + 10));
-    EXPECT_EQ(2, buf.use_count());
-    Buffer moved(std::move(acquired));
-    EXPECT_EQ(2, buf.use_count());
-    Buffer assigned;
-    assigned = std::move(moved);
-    EXPECT_EQ(2, buf.use_count());
-
-    Buffer split = assigned.splitAt(5);
-    EXPECT_EQ(3, buf.use_count());
-
-    split.advance(1);
-    assigned.subtract(1);
-    EXPECT_EQ(3, buf.use_count());
-  }
-  EXPECT_EQ(1, buf.use_count());
-}
-
-TEST(Buffer, Modifiers) {
-  Buffer buf(10);
-  {
-    unsigned char i = 0;
-    for (auto& byte : buf.range()) {
-      byte = i++;
-    }
-  }
-
-  auto prefix = buf.splitAt(2);
-
-  ASSERT_EQ(2, prefix.size());
-  EXPECT_EQ(0, *prefix.data());
-
-  ASSERT_EQ(8, buf.size());
-  EXPECT_EQ(2, *buf.data());
-
-  buf.advance(2);
-  EXPECT_EQ(4, *buf.data());
-
-  EXPECT_EQ(9, *(buf.range().end() - 1));
-
-  buf.subtract(2);
-  EXPECT_EQ(7, *(buf.range().end() - 1));
-
-  EXPECT_EQ(4, buf.size());
-}
diff --git a/contrib/pzstd/utils/test/RangeTest.cpp b/contrib/pzstd/utils/test/RangeTest.cpp
deleted file mode 100644
index 755b50fa6e80..000000000000
--- a/contrib/pzstd/utils/test/RangeTest.cpp
+++ /dev/null
@@ -1,82 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-#include "utils/Range.h"
-
-#include <gtest/gtest.h>
-#include <string>
-
-using namespace pzstd;
-
-// Range is directly copied from folly.
-// Just some sanity tests to make sure everything seems to work.
-
-TEST(Range, Constructors) {
-  StringPiece empty;
-  EXPECT_TRUE(empty.empty());
-  EXPECT_EQ(0, empty.size());
-
-  std::string str = "hello";
-  {
-    Range<std::string::const_iterator> piece(str.begin(), str.end());
-    EXPECT_EQ(5, piece.size());
-    EXPECT_EQ('h', *piece.data());
-    EXPECT_EQ('o', *(piece.end() - 1));
-  }
-
-  {
-    StringPiece piece(str.data(), str.size());
-    EXPECT_EQ(5, piece.size());
-    EXPECT_EQ('h', *piece.data());
-    EXPECT_EQ('o', *(piece.end() - 1));
-  }
-
-  {
-    StringPiece piece(str);
-    EXPECT_EQ(5, piece.size());
-    EXPECT_EQ('h', *piece.data());
-    EXPECT_EQ('o', *(piece.end() - 1));
-  }
-
-  {
-    StringPiece piece(str.c_str());
-    EXPECT_EQ(5, piece.size());
-    EXPECT_EQ('h', *piece.data());
-    EXPECT_EQ('o', *(piece.end() - 1));
-  }
-}
-
-TEST(Range, Modifiers) {
-  StringPiece range("hello world");
-  ASSERT_EQ(11, range.size());
-
-  {
-    auto hello = range.subpiece(0, 5);
-    EXPECT_EQ(5, hello.size());
-    EXPECT_EQ('h', *hello.data());
-    EXPECT_EQ('o', *(hello.end() - 1));
-  }
-  {
-    auto hello = range;
-    hello.subtract(6);
-    EXPECT_EQ(5, hello.size());
-    EXPECT_EQ('h', *hello.data());
-    EXPECT_EQ('o', *(hello.end() - 1));
-  }
-  {
-    auto world = range;
-    world.advance(6);
-    EXPECT_EQ(5, world.size());
-    EXPECT_EQ('w', *world.data());
-    EXPECT_EQ('d', *(world.end() - 1));
-  }
-
-  std::string expected = "hello world";
-  EXPECT_EQ(expected, std::string(range.begin(), range.end()));
-  EXPECT_EQ(expected, std::string(range.data(), range.size()));
-}
diff --git a/contrib/pzstd/utils/test/ResourcePoolTest.cpp b/contrib/pzstd/utils/test/ResourcePoolTest.cpp
deleted file mode 100644
index 6fe145180be9..000000000000
--- a/contrib/pzstd/utils/test/ResourcePoolTest.cpp
+++ /dev/null
@@ -1,72 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-#include "utils/ResourcePool.h"
-
-#include <gtest/gtest.h>
-#include <atomic>
-#include <thread>
-
-using namespace pzstd;
-
-TEST(ResourcePool, FullTest) {
-  unsigned numCreated = 0;
-  unsigned numDeleted = 0;
-  {
-    ResourcePool<int> pool(
-      [&numCreated] { ++numCreated; return new int{5}; },
-      [&numDeleted](int *x) { ++numDeleted; delete x; });
-
-    {
-      auto i = pool.get();
-      EXPECT_EQ(5, *i);
-      *i = 6;
-    }
-    {
-      auto i = pool.get();
-      EXPECT_EQ(6, *i);
-      auto j = pool.get();
-      EXPECT_EQ(5, *j);
-      *j = 7;
-    }
-    {
-      auto i = pool.get();
-      EXPECT_EQ(6, *i);
-      auto j = pool.get();
-      EXPECT_EQ(7, *j);
-    }
-  }
-  EXPECT_EQ(2, numCreated);
-  EXPECT_EQ(numCreated, numDeleted);
-}
-
-TEST(ResourcePool, ThreadSafe) {
-  std::atomic<unsigned> numCreated{0};
-  std::atomic<unsigned> numDeleted{0};
-  {
-    ResourcePool<int> pool(
-      [&numCreated] { ++numCreated; return new int{0}; },
-      [&numDeleted](int *x) { ++numDeleted; delete x; });
-    auto push = [&pool] {
-      for (int i = 0; i < 100; ++i) {
-        auto x = pool.get();
-        ++*x;
-      }
-    };
-    std::thread t1{push};
-    std::thread t2{push};
-    t1.join();
-    t2.join();
-
-    auto x = pool.get();
-    auto y = pool.get();
-    EXPECT_EQ(200, *x + *y);
-  }
-  EXPECT_GE(2, numCreated);
-  EXPECT_EQ(numCreated, numDeleted);
-}
diff --git a/contrib/pzstd/utils/test/ScopeGuardTest.cpp b/contrib/pzstd/utils/test/ScopeGuardTest.cpp
deleted file mode 100644
index 7bc624da79b2..000000000000
--- a/contrib/pzstd/utils/test/ScopeGuardTest.cpp
+++ /dev/null
@@ -1,28 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-#include "utils/ScopeGuard.h"
-
-#include <gtest/gtest.h>
-
-using namespace pzstd;
-
-TEST(ScopeGuard, Dismiss) {
-  {
-    auto guard = makeScopeGuard([&] { EXPECT_TRUE(false); });
-    guard.dismiss();
-  }
-}
-
-TEST(ScopeGuard, Executes) {
-  bool executed = false;
-  {
-    auto guard = makeScopeGuard([&] { executed = true; });
-  }
-  EXPECT_TRUE(executed);
-}
diff --git a/contrib/pzstd/utils/test/ThreadPoolTest.cpp b/contrib/pzstd/utils/test/ThreadPoolTest.cpp
deleted file mode 100644
index 703fd4c9ca17..000000000000
--- a/contrib/pzstd/utils/test/ThreadPoolTest.cpp
+++ /dev/null
@@ -1,71 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-#include "utils/ThreadPool.h"
-
-#include <gtest/gtest.h>
-#include <atomic>
-#include <iostream>
-#include <thread>
-#include <vector>
-
-using namespace pzstd;
-
-TEST(ThreadPool, Ordering) {
-  std::vector<int> results;
-
-  {
-    ThreadPool executor(1);
-    for (int i = 0; i < 10; ++i) {
-      executor.add([ &results, i ] { results.push_back(i); });
-    }
-  }
-
-  for (int i = 0; i < 10; ++i) {
-    EXPECT_EQ(i, results[i]);
-  }
-}
-
-TEST(ThreadPool, AllJobsFinished) {
-  std::atomic<unsigned> numFinished{0};
-  std::atomic<bool> start{false};
-  {
-    std::cerr << "Creating executor" << std::endl;
-    ThreadPool executor(5);
-    for (int i = 0; i < 10; ++i) {
-      executor.add([ &numFinished, &start ] {
-        while (!start.load()) {
-          std::this_thread::yield();
-        }
-        ++numFinished;
-      });
-    }
-    std::cerr << "Starting" << std::endl;
-    start.store(true);
-    std::cerr << "Finishing" << std::endl;
-  }
-  EXPECT_EQ(10, numFinished.load());
-}
-
-TEST(ThreadPool, AddJobWhileJoining) {
-  std::atomic<bool> done{false};
-  {
-    ThreadPool executor(1);
-    executor.add([&executor, &done] {
-      while (!done.load()) {
-        std::this_thread::yield();
-      }
-      // Sleep for a second to be sure that we are joining
-      std::this_thread::sleep_for(std::chrono::seconds(1));
-      executor.add([] {
-        EXPECT_TRUE(false);
-      });
-    });
-    done.store(true);
-  }
-}
diff --git a/contrib/pzstd/utils/test/WorkQueueTest.cpp b/contrib/pzstd/utils/test/WorkQueueTest.cpp
deleted file mode 100644
index 14cf77304f21..000000000000
--- a/contrib/pzstd/utils/test/WorkQueueTest.cpp
+++ /dev/null
@@ -1,282 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-#include "utils/Buffer.h"
-#include "utils/WorkQueue.h"
-
-#include <gtest/gtest.h>
-#include <iostream>
-#include <memory>
-#include <mutex>
-#include <thread>
-#include <vector>
-
-using namespace pzstd;
-
-namespace {
-struct Popper {
-  WorkQueue<int>* queue;
-  int* results;
-  std::mutex* mutex;
-
-  void operator()() {
-    int result;
-    while (queue->pop(result)) {
-      std::lock_guard<std::mutex> lock(*mutex);
-      results[result] = result;
-    }
-  }
-};
-}
-
-TEST(WorkQueue, SingleThreaded) {
-  WorkQueue<int> queue;
-  int result;
-
-  queue.push(5);
-  EXPECT_TRUE(queue.pop(result));
-  EXPECT_EQ(5, result);
-
-  queue.push(1);
-  queue.push(2);
-  EXPECT_TRUE(queue.pop(result));
-  EXPECT_EQ(1, result);
-  EXPECT_TRUE(queue.pop(result));
-  EXPECT_EQ(2, result);
-
-  queue.push(1);
-  queue.push(2);
-  queue.finish();
-  EXPECT_TRUE(queue.pop(result));
-  EXPECT_EQ(1, result);
-  EXPECT_TRUE(queue.pop(result));
-  EXPECT_EQ(2, result);
-  EXPECT_FALSE(queue.pop(result));
-
-  queue.waitUntilFinished();
-}
-
-TEST(WorkQueue, SPSC) {
-  WorkQueue<int> queue;
-  const int max = 100;
-
-  for (int i = 0; i < 10; ++i) {
-    queue.push(int{i});
-  }
-
-  std::thread thread([ &queue, max ] {
-    int result;
-    for (int i = 0;; ++i) {
-      if (!queue.pop(result)) {
-        EXPECT_EQ(i, max);
-        break;
-      }
-      EXPECT_EQ(i, result);
-    }
-  });
-
-  std::this_thread::yield();
-  for (int i = 10; i < max; ++i) {
-    queue.push(int{i});
-  }
-  queue.finish();
-
-  thread.join();
-}
-
-TEST(WorkQueue, SPMC) {
-  WorkQueue<int> queue;
-  std::vector<int> results(50, -1);
-  std::mutex mutex;
-  std::vector<std::thread> threads;
-  for (int i = 0; i < 5; ++i) {
-    threads.emplace_back(Popper{&queue, results.data(), &mutex});
-  }
-
-  for (int i = 0; i < 50; ++i) {
-    queue.push(int{i});
-  }
-  queue.finish();
-
-  for (auto& thread : threads) {
-    thread.join();
-  }
-
-  for (int i = 0; i < 50; ++i) {
-    EXPECT_EQ(i, results[i]);
-  }
-}
-
-TEST(WorkQueue, MPMC) {
-  WorkQueue<int> queue;
-  std::vector<int> results(100, -1);
-  std::mutex mutex;
-  std::vector<std::thread> popperThreads;
-  for (int i = 0; i < 4; ++i) {
-    popperThreads.emplace_back(Popper{&queue, results.data(), &mutex});
-  }
-
-  std::vector<std::thread> pusherThreads;
-  for (int i = 0; i < 2; ++i) {
-    auto min = i * 50;
-    auto max = (i + 1) * 50;
-    pusherThreads.emplace_back(
-        [ &queue, min, max ] {
-          for (int i = min; i < max; ++i) {
-            queue.push(int{i});
-          }
-        });
-  }
-
-  for (auto& thread : pusherThreads) {
-    thread.join();
-  }
-  queue.finish();
-
-  for (auto& thread : popperThreads) {
-    thread.join();
-  }
-
-  for (int i = 0; i < 100; ++i) {
-    EXPECT_EQ(i, results[i]);
-  }
-}
-
-TEST(WorkQueue, BoundedSizeWorks) {
-  WorkQueue<int> queue(1);
-  int result;
-  queue.push(5);
-  queue.pop(result);
-  queue.push(5);
-  queue.pop(result);
-  queue.push(5);
-  queue.finish();
-  queue.pop(result);
-  EXPECT_EQ(5, result);
-}
-
-TEST(WorkQueue, BoundedSizePushAfterFinish) {
-  WorkQueue<int> queue(1);
-  int result;
-  queue.push(5);
-  std::thread pusher([&queue] {
-    queue.push(6);
-  });
-  // Dirtily try and make sure that pusher has run.
-  std::this_thread::sleep_for(std::chrono::seconds(1));
-  queue.finish();
-  EXPECT_TRUE(queue.pop(result));
-  EXPECT_EQ(5, result);
-  EXPECT_FALSE(queue.pop(result));
-
-  pusher.join();
-}
-
-TEST(WorkQueue, SetMaxSize) {
-  WorkQueue<int> queue(2);
-  int result;
-  queue.push(5);
-  queue.push(6);
-  queue.setMaxSize(1);
-  std::thread pusher([&queue] {
-    queue.push(7);
-  });
-  // Dirtily try and make sure that pusher has run.
-  std::this_thread::sleep_for(std::chrono::seconds(1));
-  queue.finish();
-  EXPECT_TRUE(queue.pop(result));
-  EXPECT_EQ(5, result);
-  EXPECT_TRUE(queue.pop(result));
-  EXPECT_EQ(6, result);
-  EXPECT_FALSE(queue.pop(result));
-
-  pusher.join();
-}
-
-TEST(WorkQueue, BoundedSizeMPMC) {
-  WorkQueue<int> queue(10);
-  std::vector<int> results(200, -1);
-  std::mutex mutex;
-  std::cerr << "Creating popperThreads" << std::endl;
-  std::vector<std::thread> popperThreads;
-  for (int i = 0; i < 4; ++i) {
-    popperThreads.emplace_back(Popper{&queue, results.data(), &mutex});
-  }
-
-  std::cerr << "Creating pusherThreads" << std::endl;
-  std::vector<std::thread> pusherThreads;
-  for (int i = 0; i < 2; ++i) {
-    auto min = i * 100;
-    auto max = (i + 1) * 100;
-    pusherThreads.emplace_back(
-        [ &queue, min, max ] {
-          for (int i = min; i < max; ++i) {
-            queue.push(int{i});
-          }
-        });
-  }
-
-  std::cerr << "Joining pusherThreads" << std::endl;
-  for (auto& thread : pusherThreads) {
-    thread.join();
-  }
-  std::cerr << "Finishing queue" << std::endl;
-  queue.finish();
-
-  std::cerr << "Joining popperThreads" << std::endl;
-  for (auto& thread : popperThreads) {
-    thread.join();
-  }
-
-  std::cerr << "Inspecting results" << std::endl;
-  for (int i = 0; i < 200; ++i) {
-    EXPECT_EQ(i, results[i]);
-  }
-}
-
-TEST(WorkQueue, FailedPush) {
-  WorkQueue<std::unique_ptr<int>> queue;
-  std::unique_ptr<int> x(new int{5});
-  EXPECT_TRUE(queue.push(std::move(x)));
-  EXPECT_EQ(nullptr, x);
-  queue.finish();
-  x.reset(new int{6});
-  EXPECT_FALSE(queue.push(std::move(x)));
-  EXPECT_NE(nullptr, x);
-  EXPECT_EQ(6, *x);
-}
-
-TEST(BufferWorkQueue, SizeCalculatedCorrectly) {
-  {
-    BufferWorkQueue queue;
-    queue.finish();
-    EXPECT_EQ(0, queue.size());
-  }
-  {
-    BufferWorkQueue queue;
-    queue.push(Buffer(10));
-    queue.finish();
-    EXPECT_EQ(10, queue.size());
-  }
-  {
-    BufferWorkQueue queue;
-    queue.push(Buffer(10));
-    queue.push(Buffer(5));
-    queue.finish();
-    EXPECT_EQ(15, queue.size());
-  }
-  {
-    BufferWorkQueue queue;
-    queue.push(Buffer(10));
-    queue.push(Buffer(5));
-    queue.finish();
-    Buffer buffer;
-    queue.pop(buffer);
-    EXPECT_EQ(5, queue.size());
-  }
-}
diff --git a/contrib/seekable_format/examples/Makefile b/contrib/seekable_format/examples/Makefile
deleted file mode 100644
index 543780f75d34..000000000000
--- a/contrib/seekable_format/examples/Makefile
+++ /dev/null
@@ -1,53 +0,0 @@
-# ################################################################
-# Copyright (c) 2017-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under both the BSD-style license (found in the
-# LICENSE file in the root directory of this source tree) and the GPLv2 (found
-# in the COPYING file in the root directory of this source tree).
-# ################################################################
-
-# This Makefile presumes libzstd is built, using `make` in / or /lib/
-
-ZSTDLIB_PATH = ../../../lib
-ZSTDLIB_NAME = libzstd.a
-ZSTDLIB = $(ZSTDLIB_PATH)/$(ZSTDLIB_NAME)
-
-CPPFLAGS += -I../ -I../../../lib -I../../../lib/common
-
-CFLAGS ?= -O3
-CFLAGS += -g
-
-SEEKABLE_OBJS = ../zstdseek_compress.c ../zstdseek_decompress.c $(ZSTDLIB)
-
-.PHONY: default all clean test
-
-default: all
-
-all: seekable_compression seekable_decompression seekable_decompression_mem \
-	parallel_processing
-
-$(ZSTDLIB):
-	make -C $(ZSTDLIB_PATH) $(ZSTDLIB_NAME)
-
-seekable_compression : seekable_compression.c $(SEEKABLE_OBJS)
-	$(CC) $(CPPFLAGS) $(CFLAGS) $^ $(LDFLAGS) -o $@
-
-seekable_decompression : seekable_decompression.c $(SEEKABLE_OBJS)
-	$(CC) $(CPPFLAGS) $(CFLAGS) $^ $(LDFLAGS) -o $@
-
-seekable_decompression_mem : seekable_decompression_mem.c $(SEEKABLE_OBJS)
-	$(CC) $(CPPFLAGS) $(CFLAGS) $^ $(LDFLAGS) -o $@
-
-parallel_processing : parallel_processing.c $(SEEKABLE_OBJS)
-	$(CC) $(CPPFLAGS) $(CFLAGS) $^ $(LDFLAGS) -o $@ -pthread
-
-parallel_compression : parallel_compression.c $(SEEKABLE_OBJS)
-	$(CC) $(CPPFLAGS) $(CFLAGS) $^ $(LDFLAGS) -o $@ -pthread
-
-clean:
-	@rm -f core *.o tmp* result* *.zst \
-		seekable_compression seekable_decompression \
-		seekable_decompression_mem \
-		parallel_processing parallel_compression
-	@echo Cleaning completed
diff --git a/contrib/seekable_format/examples/parallel_compression.c b/contrib/seekable_format/examples/parallel_compression.c
deleted file mode 100644
index 69644d2b3c80..000000000000
--- a/contrib/seekable_format/examples/parallel_compression.c
+++ /dev/null
@@ -1,215 +0,0 @@
-/*
- * Copyright (c) 2017-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-
-#include <stdlib.h>    // malloc, free, exit, atoi
-#include <stdio.h>     // fprintf, perror, feof, fopen, etc.
-#include <string.h>    // strlen, memset, strcat
-#define ZSTD_STATIC_LINKING_ONLY
-#include <zstd.h>      // presumes zstd library is installed
-#include <zstd_errors.h>
-#if defined(WIN32) || defined(_WIN32)
-#  include <windows.h>
-#  define SLEEP(x) Sleep(x)
-#else
-#  include <unistd.h>
-#  define SLEEP(x) usleep(x * 1000)
-#endif
-
-#define XXH_NAMESPACE ZSTD_
-#include "xxhash.h"
-
-#include "pool.h"      // use zstd thread pool for demo
-
-#include "zstd_seekable.h"
-
-static void* malloc_orDie(size_t size)
-{
-    void* const buff = malloc(size);
-    if (buff) return buff;
-    /* error */
-    perror("malloc:");
-    exit(1);
-}
-
-static FILE* fopen_orDie(const char *filename, const char *instruction)
-{
-    FILE* const inFile = fopen(filename, instruction);
-    if (inFile) return inFile;
-    /* error */
-    perror(filename);
-    exit(3);
-}
-
-static size_t fread_orDie(void* buffer, size_t sizeToRead, FILE* file)
-{
-    size_t const readSize = fread(buffer, 1, sizeToRead, file);
-    if (readSize == sizeToRead) return readSize;   /* good */
-    if (feof(file)) return readSize;   /* good, reached end of file */
-    /* error */
-    perror("fread");
-    exit(4);
-}
-
-static size_t fwrite_orDie(const void* buffer, size_t sizeToWrite, FILE* file)
-{
-    size_t const writtenSize = fwrite(buffer, 1, sizeToWrite, file);
-    if (writtenSize == sizeToWrite) return sizeToWrite;   /* good */
-    /* error */
-    perror("fwrite");
-    exit(5);
-}
-
-static size_t fclose_orDie(FILE* file)
-{
-    if (!fclose(file)) return 0;
-    /* error */
-    perror("fclose");
-    exit(6);
-}
-
-static void fseek_orDie(FILE* file, long int offset, int origin)
-{
-    if (!fseek(file, offset, origin)) {
-        if (!fflush(file)) return;
-    }
-    /* error */
-    perror("fseek");
-    exit(7);
-}
-
-static long int ftell_orDie(FILE* file)
-{
-    long int off = ftell(file);
-    if (off != -1) return off;
-    /* error */
-    perror("ftell");
-    exit(8);
-}
-
-struct job {
-    const void* src;
-    size_t srcSize;
-    void* dst;
-    size_t dstSize;
-
-    unsigned checksum;
-
-    int compressionLevel;
-    int done;
-};
-
-static void compressFrame(void* opaque)
-{
-    struct job* job = opaque;
-
-    job->checksum = XXH64(job->src, job->srcSize, 0);
-
-    size_t ret = ZSTD_compress(job->dst, job->dstSize, job->src, job->srcSize, job->compressionLevel);
-    if (ZSTD_isError(ret)) {
-        fprintf(stderr, "ZSTD_compress() error : %s \n", ZSTD_getErrorName(ret));
-        exit(20);
-    }
-
-    job->dstSize = ret;
-    job->done = 1;
-}
-
-static void compressFile_orDie(const char* fname, const char* outName, int cLevel, unsigned frameSize, int nbThreads)
-{
-    POOL_ctx* pool = POOL_create(nbThreads, nbThreads);
-    if (pool == NULL) { fprintf(stderr, "POOL_create() error \n"); exit(9); }
-
-    FILE* const fin  = fopen_orDie(fname, "rb");
-    FILE* const fout = fopen_orDie(outName, "wb");
-
-    if (ZSTD_compressBound(frameSize) > 0xFFFFFFFFU) { fprintf(stderr, "Frame size too large \n"); exit(10); }
-    unsigned dstSize = ZSTD_compressBound(frameSize);
-
-
-    fseek_orDie(fin, 0, SEEK_END);
-    long int length = ftell_orDie(fin);
-    fseek_orDie(fin, 0, SEEK_SET);
-
-    size_t numFrames = (length + frameSize - 1) / frameSize;
-
-    struct job* jobs = malloc_orDie(sizeof(struct job) * numFrames);
-
-    size_t i;
-    for(i = 0; i < numFrames; i++) {
-        void* in = malloc_orDie(frameSize);
-        void* out = malloc_orDie(dstSize);
-
-        size_t inSize = fread_orDie(in, frameSize, fin);
-
-        jobs[i].src = in;
-        jobs[i].srcSize = inSize;
-        jobs[i].dst = out;
-        jobs[i].dstSize = dstSize;
-        jobs[i].compressionLevel = cLevel;
-        jobs[i].done = 0;
-        POOL_add(pool, compressFrame, &jobs[i]);
-    }
-
-    ZSTD_frameLog* fl = ZSTD_seekable_createFrameLog(1);
-    if (fl == NULL) { fprintf(stderr, "ZSTD_seekable_createFrameLog() failed \n"); exit(11); }
-    for (i = 0; i < numFrames; i++) {
-        while (!jobs[i].done) SLEEP(5); /* wake up every 5 milliseconds to check */
-        fwrite_orDie(jobs[i].dst, jobs[i].dstSize, fout);
-        free((void*)jobs[i].src);
-        free(jobs[i].dst);
-
-        size_t ret = ZSTD_seekable_logFrame(fl, jobs[i].dstSize, jobs[i].srcSize, jobs[i].checksum);
-        if (ZSTD_isError(ret)) { fprintf(stderr, "ZSTD_seekable_logFrame() error : %s \n", ZSTD_getErrorName(ret)); }
-    }
-
-    {   unsigned char seekTableBuff[1024];
-        ZSTD_outBuffer out = {seekTableBuff, 1024, 0};
-        while (ZSTD_seekable_writeSeekTable(fl, &out) != 0) {
-            fwrite_orDie(seekTableBuff, out.pos, fout);
-            out.pos = 0;
-        }
-        fwrite_orDie(seekTableBuff, out.pos, fout);
-    }
-
-    ZSTD_seekable_freeFrameLog(fl);
-    free(jobs);
-    fclose_orDie(fout);
-    fclose_orDie(fin);
-}
-
-static const char* createOutFilename_orDie(const char* filename)
-{
-    size_t const inL = strlen(filename);
-    size_t const outL = inL + 5;
-    void* outSpace = malloc_orDie(outL);
-    memset(outSpace, 0, outL);
-    strcat(outSpace, filename);
-    strcat(outSpace, ".zst");
-    return (const char*)outSpace;
-}
-
-int main(int argc, const char** argv) {
-    const char* const exeName = argv[0];
-    if (argc!=4) {
-        printf("wrong arguments\n");
-        printf("usage:\n");
-        printf("%s FILE FRAME_SIZE NB_THREADS\n", exeName);
-        return 1;
-    }
-
-    {   const char* const inFileName = argv[1];
-        unsigned const frameSize = (unsigned)atoi(argv[2]);
-        int const nbThreads = atoi(argv[3]);
-
-        const char* const outFileName = createOutFilename_orDie(inFileName);
-        compressFile_orDie(inFileName, outFileName, 5, frameSize, nbThreads);
-    }
-
-    return 0;
-}
diff --git a/contrib/seekable_format/examples/parallel_processing.c b/contrib/seekable_format/examples/parallel_processing.c
deleted file mode 100644
index 36226b49fd3c..000000000000
--- a/contrib/seekable_format/examples/parallel_processing.c
+++ /dev/null
@@ -1,194 +0,0 @@
-/*
- * Copyright (c) 2017-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-
-/*
- * A simple demo that sums up all the bytes in the file in parallel using
- * seekable decompression and the zstd thread pool
- */
-
-#include <stdlib.h>    // malloc, exit
-#include <stdio.h>     // fprintf, perror, feof
-#include <string.h>    // strerror
-#include <errno.h>     // errno
-#define ZSTD_STATIC_LINKING_ONLY
-#include <zstd.h>      // presumes zstd library is installed
-#include <zstd_errors.h>
-#if defined(WIN32) || defined(_WIN32)
-#  include <windows.h>
-#  define SLEEP(x) Sleep(x)
-#else
-#  include <unistd.h>
-#  define SLEEP(x) usleep(x * 1000)
-#endif
-
-#include "pool.h"      // use zstd thread pool for demo
-
-#include "zstd_seekable.h"
-
-#define MIN(a, b) ((a) < (b) ? (a) : (b))
-
-static void* malloc_orDie(size_t size)
-{
-    void* const buff = malloc(size);
-    if (buff) return buff;
-    /* error */
-    perror("malloc");
-    exit(1);
-}
-
-static void* realloc_orDie(void* ptr, size_t size)
-{
-    ptr = realloc(ptr, size);
-    if (ptr) return ptr;
-    /* error */
-    perror("realloc");
-    exit(1);
-}
-
-static FILE* fopen_orDie(const char *filename, const char *instruction)
-{
-    FILE* const inFile = fopen(filename, instruction);
-    if (inFile) return inFile;
-    /* error */
-    perror(filename);
-    exit(3);
-}
-
-static size_t fread_orDie(void* buffer, size_t sizeToRead, FILE* file)
-{
-    size_t const readSize = fread(buffer, 1, sizeToRead, file);
-    if (readSize == sizeToRead) return readSize;   /* good */
-    if (feof(file)) return readSize;   /* good, reached end of file */
-    /* error */
-    perror("fread");
-    exit(4);
-}
-
-static size_t fwrite_orDie(const void* buffer, size_t sizeToWrite, FILE* file)
-{
-    size_t const writtenSize = fwrite(buffer, 1, sizeToWrite, file);
-    if (writtenSize == sizeToWrite) return sizeToWrite;   /* good */
-    /* error */
-    perror("fwrite");
-    exit(5);
-}
-
-static size_t fclose_orDie(FILE* file)
-{
-    if (!fclose(file)) return 0;
-    /* error */
-    perror("fclose");
-    exit(6);
-}
-
-static void fseek_orDie(FILE* file, long int offset, int origin) {
-    if (!fseek(file, offset, origin)) {
-        if (!fflush(file)) return;
-    }
-    /* error */
-    perror("fseek");
-    exit(7);
-}
-
-struct sum_job {
-    const char* fname;
-    unsigned long long sum;
-    unsigned frameNb;
-    int done;
-};
-
-static void sumFrame(void* opaque)
-{
-    struct sum_job* job = (struct sum_job*)opaque;
-    job->done = 0;
-
-    FILE* const fin = fopen_orDie(job->fname, "rb");
-
-    ZSTD_seekable* const seekable = ZSTD_seekable_create();
-    if (seekable==NULL) { fprintf(stderr, "ZSTD_seekable_create() error \n"); exit(10); }
-
-    size_t const initResult = ZSTD_seekable_initFile(seekable, fin);
-    if (ZSTD_isError(initResult)) { fprintf(stderr, "ZSTD_seekable_init() error : %s \n", ZSTD_getErrorName(initResult)); exit(11); }
-
-    size_t const frameSize = ZSTD_seekable_getFrameDecompressedSize(seekable, job->frameNb);
-    unsigned char* data = malloc_orDie(frameSize);
-
-    size_t result = ZSTD_seekable_decompressFrame(seekable, data, frameSize, job->frameNb);
-    if (ZSTD_isError(result)) { fprintf(stderr, "ZSTD_seekable_decompressFrame() error : %s \n", ZSTD_getErrorName(result)); exit(12); }
-
-    unsigned long long sum = 0;
-    size_t i;
-    for (i = 0; i < frameSize; i++) {
-        sum += data[i];
-    }
-    job->sum = sum;
-    job->done = 1;
-
-    fclose(fin);
-    ZSTD_seekable_free(seekable);
-    free(data);
-}
-
-static void sumFile_orDie(const char* fname, int nbThreads)
-{
-    POOL_ctx* pool = POOL_create(nbThreads, nbThreads);
-    if (pool == NULL) { fprintf(stderr, "POOL_create() error \n"); exit(9); }
-
-    FILE* const fin = fopen_orDie(fname, "rb");
-
-    ZSTD_seekable* const seekable = ZSTD_seekable_create();
-    if (seekable==NULL) { fprintf(stderr, "ZSTD_seekable_create() error \n"); exit(10); }
-
-    size_t const initResult = ZSTD_seekable_initFile(seekable, fin);
-    if (ZSTD_isError(initResult)) { fprintf(stderr, "ZSTD_seekable_init() error : %s \n", ZSTD_getErrorName(initResult)); exit(11); }
-
-    unsigned const numFrames = ZSTD_seekable_getNumFrames(seekable);
-    struct sum_job* jobs = (struct sum_job*)malloc(numFrames * sizeof(struct sum_job));
-
-    unsigned fnb;
-    for (fnb = 0; fnb < numFrames; fnb++) {
-        jobs[fnb] = (struct sum_job){ fname, 0, fnb, 0 };
-        POOL_add(pool, sumFrame, &jobs[fnb]);
-    }
-
-    unsigned long long total = 0;
-
-    for (fnb = 0; fnb < numFrames; fnb++) {
-        while (!jobs[fnb].done) SLEEP(5); /* wake up every 5 milliseconds to check */
-        total += jobs[fnb].sum;
-    }
-
-    printf("Sum: %llu\n", total);
-
-    POOL_free(pool);
-    ZSTD_seekable_free(seekable);
-    fclose(fin);
-    free(jobs);
-}
-
-
-int main(int argc, const char** argv)
-{
-    const char* const exeName = argv[0];
-
-    if (argc!=3) {
-        fprintf(stderr, "wrong arguments\n");
-        fprintf(stderr, "usage:\n");
-        fprintf(stderr, "%s FILE NB_THREADS\n", exeName);
-        return 1;
-    }
-
-    {
-        const char* const inFilename = argv[1];
-        int const nbThreads = atoi(argv[2]);
-        sumFile_orDie(inFilename, nbThreads);
-    }
-
-    return 0;
-}
diff --git a/contrib/seekable_format/examples/seekable_compression.c b/contrib/seekable_format/examples/seekable_compression.c
deleted file mode 100644
index 9a331a89531e..000000000000
--- a/contrib/seekable_format/examples/seekable_compression.c
+++ /dev/null
@@ -1,133 +0,0 @@
-/*
- * Copyright (c) 2017-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-
-#include <stdlib.h>    // malloc, free, exit, atoi
-#include <stdio.h>     // fprintf, perror, feof, fopen, etc.
-#include <string.h>    // strlen, memset, strcat
-#define ZSTD_STATIC_LINKING_ONLY
-#include <zstd.h>      // presumes zstd library is installed
-
-#include "zstd_seekable.h"
-
-static void* malloc_orDie(size_t size)
-{
-    void* const buff = malloc(size);
-    if (buff) return buff;
-    /* error */
-    perror("malloc:");
-    exit(1);
-}
-
-static FILE* fopen_orDie(const char *filename, const char *instruction)
-{
-    FILE* const inFile = fopen(filename, instruction);
-    if (inFile) return inFile;
-    /* error */
-    perror(filename);
-    exit(3);
-}
-
-static size_t fread_orDie(void* buffer, size_t sizeToRead, FILE* file)
-{
-    size_t const readSize = fread(buffer, 1, sizeToRead, file);
-    if (readSize == sizeToRead) return readSize;   /* good */
-    if (feof(file)) return readSize;   /* good, reached end of file */
-    /* error */
-    perror("fread");
-    exit(4);
-}
-
-static size_t fwrite_orDie(const void* buffer, size_t sizeToWrite, FILE* file)
-{
-    size_t const writtenSize = fwrite(buffer, 1, sizeToWrite, file);
-    if (writtenSize == sizeToWrite) return sizeToWrite;   /* good */
-    /* error */
-    perror("fwrite");
-    exit(5);
-}
-
-static size_t fclose_orDie(FILE* file)
-{
-    if (!fclose(file)) return 0;
-    /* error */
-    perror("fclose");
-    exit(6);
-}
-
-static void compressFile_orDie(const char* fname, const char* outName, int cLevel, unsigned frameSize)
-{
-    FILE* const fin  = fopen_orDie(fname, "rb");
-    FILE* const fout = fopen_orDie(outName, "wb");
-    size_t const buffInSize = ZSTD_CStreamInSize();    /* can always read one full block */
-    void*  const buffIn  = malloc_orDie(buffInSize);
-    size_t const buffOutSize = ZSTD_CStreamOutSize();  /* can always flush a full block */
-    void*  const buffOut = malloc_orDie(buffOutSize);
-
-    ZSTD_seekable_CStream* const cstream = ZSTD_seekable_createCStream();
-    if (cstream==NULL) { fprintf(stderr, "ZSTD_seekable_createCStream() error \n"); exit(10); }
-    size_t const initResult = ZSTD_seekable_initCStream(cstream, cLevel, 1, frameSize);
-    if (ZSTD_isError(initResult)) { fprintf(stderr, "ZSTD_seekable_initCStream() error : %s \n", ZSTD_getErrorName(initResult)); exit(11); }
-
-    size_t read, toRead = buffInSize;
-    while( (read = fread_orDie(buffIn, toRead, fin)) ) {
-        ZSTD_inBuffer input = { buffIn, read, 0 };
-        while (input.pos < input.size) {
-            ZSTD_outBuffer output = { buffOut, buffOutSize, 0 };
-            toRead = ZSTD_seekable_compressStream(cstream, &output , &input);   /* toRead is guaranteed to be <= ZSTD_CStreamInSize() */
-            if (ZSTD_isError(toRead)) { fprintf(stderr, "ZSTD_seekable_compressStream() error : %s \n", ZSTD_getErrorName(toRead)); exit(12); }
-            if (toRead > buffInSize) toRead = buffInSize;   /* Safely handle case when `buffInSize` is manually changed to a value < ZSTD_CStreamInSize()*/
-            fwrite_orDie(buffOut, output.pos, fout);
-        }
-    }
-
-    while (1) {
-        ZSTD_outBuffer output = { buffOut, buffOutSize, 0 };
-        size_t const remainingToFlush = ZSTD_seekable_endStream(cstream, &output);   /* close stream */
-        if (ZSTD_isError(remainingToFlush)) { fprintf(stderr, "ZSTD_seekable_endStream() error : %s \n", ZSTD_getErrorName(remainingToFlush)); exit(13); }
-        fwrite_orDie(buffOut, output.pos, fout);
-        if (!remainingToFlush) break;
-    }
-
-    ZSTD_seekable_freeCStream(cstream);
-    fclose_orDie(fout);
-    fclose_orDie(fin);
-    free(buffIn);
-    free(buffOut);
-}
-
-static char* createOutFilename_orDie(const char* filename)
-{
-    size_t const inL = strlen(filename);
-    size_t const outL = inL + 5;
-    void* outSpace = malloc_orDie(outL);
-    memset(outSpace, 0, outL);
-    strcat(outSpace, filename);
-    strcat(outSpace, ".zst");
-    return (char*)outSpace;
-}
-
-int main(int argc, const char** argv) {
-    const char* const exeName = argv[0];
-    if (argc!=3) {
-        printf("wrong arguments\n");
-        printf("usage:\n");
-        printf("%s FILE FRAME_SIZE\n", exeName);
-        return 1;
-    }
-
-    {   const char* const inFileName = argv[1];
-        unsigned const frameSize = (unsigned)atoi(argv[2]);
-
-        char* const outFileName = createOutFilename_orDie(inFileName);
-        compressFile_orDie(inFileName, outFileName, 5, frameSize);
-        free(outFileName);
-    }
-
-    return 0;
-}
diff --git a/contrib/seekable_format/examples/seekable_decompression.c b/contrib/seekable_format/examples/seekable_decompression.c
deleted file mode 100644
index 7050e0fa5c64..000000000000
--- a/contrib/seekable_format/examples/seekable_decompression.c
+++ /dev/null
@@ -1,138 +0,0 @@
-/*
- * Copyright (c) 2017-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-
-
-#include <stdlib.h>    // malloc, exit
-#include <stdio.h>     // fprintf, perror, feof
-#include <string.h>    // strerror
-#include <errno.h>     // errno
-#define ZSTD_STATIC_LINKING_ONLY
-#include <zstd.h>      // presumes zstd library is installed
-#include <zstd_errors.h>
-
-#include "zstd_seekable.h"
-
-#define MIN(a, b) ((a) < (b) ? (a) : (b))
-
-static void* malloc_orDie(size_t size)
-{
-    void* const buff = malloc(size);
-    if (buff) return buff;
-    /* error */
-    perror("malloc");
-    exit(1);
-}
-
-static void* realloc_orDie(void* ptr, size_t size)
-{
-    ptr = realloc(ptr, size);
-    if (ptr) return ptr;
-    /* error */
-    perror("realloc");
-    exit(1);
-}
-
-static FILE* fopen_orDie(const char *filename, const char *instruction)
-{
-    FILE* const inFile = fopen(filename, instruction);
-    if (inFile) return inFile;
-    /* error */
-    perror(filename);
-    exit(3);
-}
-
-static size_t fread_orDie(void* buffer, size_t sizeToRead, FILE* file)
-{
-    size_t const readSize = fread(buffer, 1, sizeToRead, file);
-    if (readSize == sizeToRead) return readSize;   /* good */
-    if (feof(file)) return readSize;   /* good, reached end of file */
-    /* error */
-    perror("fread");
-    exit(4);
-}
-
-static size_t fwrite_orDie(const void* buffer, size_t sizeToWrite, FILE* file)
-{
-    size_t const writtenSize = fwrite(buffer, 1, sizeToWrite, file);
-    if (writtenSize == sizeToWrite) return sizeToWrite;   /* good */
-    /* error */
-    perror("fwrite");
-    exit(5);
-}
-
-static size_t fclose_orDie(FILE* file)
-{
-    if (!fclose(file)) return 0;
-    /* error */
-    perror("fclose");
-    exit(6);
-}
-
-static void fseek_orDie(FILE* file, long int offset, int origin) {
-    if (!fseek(file, offset, origin)) {
-        if (!fflush(file)) return;
-    }
-    /* error */
-    perror("fseek");
-    exit(7);
-}
-
-
-static void decompressFile_orDie(const char* fname, off_t startOffset, off_t endOffset)
-{
-    FILE* const fin  = fopen_orDie(fname, "rb");
-    FILE* const fout = stdout;
-    size_t const buffOutSize = ZSTD_DStreamOutSize();  /* Guarantee to successfully flush at least one complete compressed block in all circumstances. */
-    void*  const buffOut = malloc_orDie(buffOutSize);
-
-    ZSTD_seekable* const seekable = ZSTD_seekable_create();
-    if (seekable==NULL) { fprintf(stderr, "ZSTD_seekable_create() error \n"); exit(10); }
-
-    size_t const initResult = ZSTD_seekable_initFile(seekable, fin);
-    if (ZSTD_isError(initResult)) { fprintf(stderr, "ZSTD_seekable_init() error : %s \n", ZSTD_getErrorName(initResult)); exit(11); }
-
-    while (startOffset < endOffset) {
-        size_t const result = ZSTD_seekable_decompress(seekable, buffOut, MIN(endOffset - startOffset, buffOutSize), startOffset);
-
-        if (ZSTD_isError(result)) {
-            fprintf(stderr, "ZSTD_seekable_decompress() error : %s \n",
-                    ZSTD_getErrorName(result));
-            exit(12);
-        }
-        fwrite_orDie(buffOut, result, fout);
-        startOffset += result;
-    }
-
-    ZSTD_seekable_free(seekable);
-    fclose_orDie(fin);
-    fclose_orDie(fout);
-    free(buffOut);
-}
-
-
-int main(int argc, const char** argv)
-{
-    const char* const exeName = argv[0];
-
-    if (argc!=4) {
-        fprintf(stderr, "wrong arguments\n");
-        fprintf(stderr, "usage:\n");
-        fprintf(stderr, "%s FILE START END\n", exeName);
-        return 1;
-    }
-
-    {
-        const char* const inFilename = argv[1];
-        off_t const startOffset = atoll(argv[2]);
-        off_t const endOffset = atoll(argv[3]);
-        decompressFile_orDie(inFilename, startOffset, endOffset);
-    }
-
-    return 0;
-}
diff --git a/contrib/seekable_format/examples/seekable_decompression_mem.c b/contrib/seekable_format/examples/seekable_decompression_mem.c
deleted file mode 100644
index c36d2221f97e..000000000000
--- a/contrib/seekable_format/examples/seekable_decompression_mem.c
+++ /dev/null
@@ -1,144 +0,0 @@
-/*
- * Copyright (c) 2017-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-
-
-#include <stdlib.h>    // malloc, exit
-#include <stdio.h>     // fprintf, perror, feof
-#include <string.h>    // strerror
-#include <errno.h>     // errno
-#define ZSTD_STATIC_LINKING_ONLY
-#include <zstd.h>      // presumes zstd library is installed
-#include <zstd_errors.h>
-
-#include "zstd_seekable.h"
-
-#define MIN(a, b) ((a) < (b) ? (a) : (b))
-
-#define MAX_FILE_SIZE (8 * 1024 * 1024)
-
-static void* malloc_orDie(size_t size)
-{
-    void* const buff = malloc(size);
-    if (buff) return buff;
-    /* error */
-    perror("malloc");
-    exit(1);
-}
-
-static void* realloc_orDie(void* ptr, size_t size)
-{
-    ptr = realloc(ptr, size);
-    if (ptr) return ptr;
-    /* error */
-    perror("realloc");
-    exit(1);
-}
-
-static FILE* fopen_orDie(const char *filename, const char *instruction)
-{
-    FILE* const inFile = fopen(filename, instruction);
-    if (inFile) return inFile;
-    /* error */
-    perror(filename);
-    exit(3);
-}
-
-static size_t fread_orDie(void* buffer, size_t sizeToRead, FILE* file)
-{
-    size_t const readSize = fread(buffer, 1, sizeToRead, file);
-    if (readSize == sizeToRead) return readSize;   /* good */
-    if (feof(file)) return readSize;   /* good, reached end of file */
-    /* error */
-    perror("fread");
-    exit(4);
-}
-
-static size_t fwrite_orDie(const void* buffer, size_t sizeToWrite, FILE* file)
-{
-    size_t const writtenSize = fwrite(buffer, 1, sizeToWrite, file);
-    if (writtenSize == sizeToWrite) return sizeToWrite;   /* good */
-    /* error */
-    perror("fwrite");
-    exit(5);
-}
-
-static size_t fclose_orDie(FILE* file)
-{
-    if (!fclose(file)) return 0;
-    /* error */
-    perror("fclose");
-    exit(6);
-}
-
-static void fseek_orDie(FILE* file, long int offset, int origin) {
-    if (!fseek(file, offset, origin)) {
-        if (!fflush(file)) return;
-    }
-    /* error */
-    perror("fseek");
-    exit(7);
-}
-
-
-static void decompressFile_orDie(const char* fname, off_t startOffset, off_t endOffset)
-{
-    FILE* const fin  = fopen_orDie(fname, "rb");
-    FILE* const fout = stdout;
-    // Just for demo purposes, assume file is <= MAX_FILE_SIZE
-    void*  const buffIn = malloc_orDie(MAX_FILE_SIZE);
-    size_t const inSize = fread_orDie(buffIn, MAX_FILE_SIZE, fin);
-    size_t const buffOutSize = ZSTD_DStreamOutSize();  /* Guarantee to successfully flush at least one complete compressed block in all circumstances. */
-    void*  const buffOut = malloc_orDie(buffOutSize);
-
-    ZSTD_seekable* const seekable = ZSTD_seekable_create();
-    if (seekable==NULL) { fprintf(stderr, "ZSTD_seekable_create() error \n"); exit(10); }
-
-    size_t const initResult = ZSTD_seekable_initBuff(seekable, buffIn, inSize);
-    if (ZSTD_isError(initResult)) { fprintf(stderr, "ZSTD_seekable_init() error : %s \n", ZSTD_getErrorName(initResult)); exit(11); }
-
-    while (startOffset < endOffset) {
-        size_t const result = ZSTD_seekable_decompress(seekable, buffOut, MIN(endOffset - startOffset, buffOutSize), startOffset);
-
-        if (ZSTD_isError(result)) {
-            fprintf(stderr, "ZSTD_seekable_decompress() error : %s \n",
-                    ZSTD_getErrorName(result));
-            exit(12);
-        }
-        fwrite_orDie(buffOut, result, fout);
-        startOffset += result;
-    }
-
-    ZSTD_seekable_free(seekable);
-    fclose_orDie(fin);
-    fclose_orDie(fout);
-    free(buffIn);
-    free(buffOut);
-}
-
-
-int main(int argc, const char** argv)
-{
-    const char* const exeName = argv[0];
-
-    if (argc!=4) {
-        fprintf(stderr, "wrong arguments\n");
-        fprintf(stderr, "usage:\n");
-        fprintf(stderr, "%s FILE START END\n", exeName);
-        return 1;
-    }
-
-    {
-        const char* const inFilename = argv[1];
-        off_t const startOffset = atoll(argv[2]);
-        off_t const endOffset = atoll(argv[3]);
-        decompressFile_orDie(inFilename, startOffset, endOffset);
-    }
-
-    return 0;
-}
diff --git a/contrib/seekable_format/zstd_seekable.h b/contrib/seekable_format/zstd_seekable.h
deleted file mode 100644
index 7ffd1ba0a72b..000000000000
--- a/contrib/seekable_format/zstd_seekable.h
+++ /dev/null
@@ -1,186 +0,0 @@
-#ifndef SEEKABLE_H
-#define SEEKABLE_H
-
-#if defined (__cplusplus)
-extern "C" {
-#endif
-
-#include <stdio.h>
-#include "zstd.h"   /* ZSTDLIB_API */
-
-
-#define ZSTD_seekTableFooterSize 9
-
-#define ZSTD_SEEKABLE_MAGICNUMBER 0x8F92EAB1
-
-#define ZSTD_SEEKABLE_MAXFRAMES 0x8000000U
-
-/* Limit the maximum size to avoid any potential issues storing the compressed size */
-#define ZSTD_SEEKABLE_MAX_FRAME_DECOMPRESSED_SIZE 0x80000000U
-
-/*-****************************************************************************
-*  Seekable Format
-*
-*  The seekable format splits the compressed data into a series of "frames",
-*  each compressed individually so that decompression of a section in the
-*  middle of an archive only requires zstd to decompress at most a frame's
-*  worth of extra data, instead of the entire archive.
-******************************************************************************/
-
-typedef struct ZSTD_seekable_CStream_s ZSTD_seekable_CStream;
-typedef struct ZSTD_seekable_s ZSTD_seekable;
-
-/*-****************************************************************************
-*  Seekable compression - HowTo
-*  A ZSTD_seekable_CStream object is required to tracking streaming operation.
-*  Use ZSTD_seekable_createCStream() and ZSTD_seekable_freeCStream() to create/
-*  release resources.
-*
-*  Streaming objects are reusable to avoid allocation and deallocation,
-*  to start a new compression operation call ZSTD_seekable_initCStream() on the
-*  compressor.
-*
-*  Data streamed to the seekable compressor will automatically be split into
-*  frames of size `maxFrameSize` (provided in ZSTD_seekable_initCStream()),
-*  or if none is provided, will be cut off whenever ZSTD_seekable_endFrame() is
-*  called or when the default maximum frame size (2GB) is reached.
-*
-*  Use ZSTD_seekable_initCStream() to initialize a ZSTD_seekable_CStream object
-*  for a new compression operation.
-*  `maxFrameSize` indicates the size at which to automatically start a new
-*  seekable frame.  `maxFrameSize == 0` implies the default maximum size.
-*  `checksumFlag` indicates whether or not the seek table should include frame
-*  checksums on the uncompressed data for verification.
-*  @return : a size hint for input to provide for compression, or an error code
-*            checkable with ZSTD_isError()
-*
-*  Use ZSTD_seekable_compressStream() repetitively to consume input stream.
-*  The function will automatically update both `pos` fields.
-*  Note that it may not consume the entire input, in which case `pos < size`,
-*  and it's up to the caller to present again remaining data.
-*  @return : a size hint, preferred nb of bytes to use as input for next
-*            function call or an error code, which can be tested using
-*            ZSTD_isError().
-*            Note 1 : it's just a hint, to help latency a little, any other
-*                     value will work fine.
-*
-*  At any time, call ZSTD_seekable_endFrame() to end the current frame and
-*  start a new one.
-*
-*  ZSTD_seekable_endStream() will end the current frame, and then write the seek
-*  table so that decompressors can efficiently find compressed frames.
-*  ZSTD_seekable_endStream() may return a number > 0 if it was unable to flush
-*  all the necessary data to `output`.  In this case, it should be called again
-*  until all remaining data is flushed out and 0 is returned.
-******************************************************************************/
-
-/*===== Seekable compressor management =====*/
-ZSTDLIB_API ZSTD_seekable_CStream* ZSTD_seekable_createCStream(void);
-ZSTDLIB_API size_t ZSTD_seekable_freeCStream(ZSTD_seekable_CStream* zcs);
-
-/*===== Seekable compression functions =====*/
-ZSTDLIB_API size_t ZSTD_seekable_initCStream(ZSTD_seekable_CStream* zcs, int compressionLevel, int checksumFlag, unsigned maxFrameSize);
-ZSTDLIB_API size_t ZSTD_seekable_compressStream(ZSTD_seekable_CStream* zcs, ZSTD_outBuffer* output, ZSTD_inBuffer* input);
-ZSTDLIB_API size_t ZSTD_seekable_endFrame(ZSTD_seekable_CStream* zcs, ZSTD_outBuffer* output);
-ZSTDLIB_API size_t ZSTD_seekable_endStream(ZSTD_seekable_CStream* zcs, ZSTD_outBuffer* output);
-
-/*= Raw seek table API
- *  These functions allow for the seek table to be constructed directly.
- *  This table can then be appended to a file of concatenated frames.
- *  This allows the frames to be compressed independently, even in parallel,
- *  and compiled together afterward into a seekable archive.
- *
- *  Use ZSTD_seekable_createFrameLog() to allocate and initialize a tracking
- *  structure.
- *
- *  Call ZSTD_seekable_logFrame() once for each frame in the archive.
- *  checksum is optional, and will not be used if checksumFlag was 0 when the
- *  frame log was created.  If present, it should be the least significant 32
- *  bits of the XXH64 hash of the uncompressed data.
- *
- *  Call ZSTD_seekable_writeSeekTable to serialize the data into a seek table.
- *  If the entire table was written, the return value will be 0.  Otherwise,
- *  it will be equal to the number of bytes left to write. */
-typedef struct ZSTD_frameLog_s ZSTD_frameLog;
-ZSTDLIB_API ZSTD_frameLog* ZSTD_seekable_createFrameLog(int checksumFlag);
-ZSTDLIB_API size_t ZSTD_seekable_freeFrameLog(ZSTD_frameLog* fl);
-ZSTDLIB_API size_t ZSTD_seekable_logFrame(ZSTD_frameLog* fl, unsigned compressedSize, unsigned decompressedSize, unsigned checksum);
-ZSTDLIB_API size_t ZSTD_seekable_writeSeekTable(ZSTD_frameLog* fl, ZSTD_outBuffer* output);
-
-/*-****************************************************************************
-*  Seekable decompression - HowTo
-*  A ZSTD_seekable object is required to tracking the seekTable.
-*
-*  Call ZSTD_seekable_init* to initialize a ZSTD_seekable object with the
-*  the seek table provided in the input.
-*  There are three modes for ZSTD_seekable_init:
-*    - ZSTD_seekable_initBuff() : An in-memory API.  The data contained in
-*      `src` should be the entire seekable file, including the seek table.
-*      `src` should be kept alive and unmodified until the ZSTD_seekable object
-*      is freed or reset.
-*    - ZSTD_seekable_initFile() : A simplified file API using stdio.  fread and
-*      fseek will be used to access the required data for building the seek
-*      table and doing decompression operations.  `src` should not be closed
-*      or modified until the ZSTD_seekable object is freed or reset.
-*    - ZSTD_seekable_initAdvanced() : A general API allowing the client to
-*      provide its own read and seek callbacks.
-*        + ZSTD_seekable_read() : read exactly `n` bytes into `buffer`.
-*                                 Premature EOF should be treated as an error.
-*        + ZSTD_seekable_seek() : seek the read head to `offset` from `origin`,
-*                                 where origin is either SEEK_SET (beginning of
-*                                 file), or SEEK_END (end of file).
-*  Both functions should return a non-negative value in case of success, and a
-*  negative value in case of failure.  If implementing using this API and
-*  stdio, be careful with files larger than 4GB and fseek.  All of these
-*  functions return an error code checkable with ZSTD_isError().
-*
-*  Call ZSTD_seekable_decompress to decompress `dstSize` bytes at decompressed
-*  offset `offset`.  ZSTD_seekable_decompress may have to decompress the entire
-*  prefix of the frame before the desired data if it has not already processed
-*  this section. If ZSTD_seekable_decompress is called multiple times for a
-*  consecutive range of data, it will efficiently retain the decompressor object
-*  and avoid redecompressing frame prefixes.  The return value is the number of
-*  bytes decompressed, or an error code checkable with ZSTD_isError().
-*
-*  The seek table access functions can be used to obtain the data contained
-*  in the seek table.  If frameIndex is larger than the value returned by
-*  ZSTD_seekable_getNumFrames(), they will return error codes checkable with
-*  ZSTD_isError().  Note that since the offset access functions return
-*  unsigned long long instead of size_t, in this case they will instead return
-*  the value ZSTD_SEEKABLE_FRAMEINDEX_TOOLARGE.
-******************************************************************************/
-
-/*===== Seekable decompressor management =====*/
-ZSTDLIB_API ZSTD_seekable* ZSTD_seekable_create(void);
-ZSTDLIB_API size_t ZSTD_seekable_free(ZSTD_seekable* zs);
-
-/*===== Seekable decompression functions =====*/
-ZSTDLIB_API size_t ZSTD_seekable_initBuff(ZSTD_seekable* zs, const void* src, size_t srcSize);
-ZSTDLIB_API size_t ZSTD_seekable_initFile(ZSTD_seekable* zs, FILE* src);
-ZSTDLIB_API size_t ZSTD_seekable_decompress(ZSTD_seekable* zs, void* dst, size_t dstSize, unsigned long long offset);
-ZSTDLIB_API size_t ZSTD_seekable_decompressFrame(ZSTD_seekable* zs, void* dst, size_t dstSize, unsigned frameIndex);
-
-#define ZSTD_SEEKABLE_FRAMEINDEX_TOOLARGE (0ULL-2)
-/*===== Seek Table access functions =====*/
-ZSTDLIB_API unsigned ZSTD_seekable_getNumFrames(ZSTD_seekable* const zs);
-ZSTDLIB_API unsigned long long ZSTD_seekable_getFrameCompressedOffset(ZSTD_seekable* const zs, unsigned frameIndex);
-ZSTDLIB_API unsigned long long ZSTD_seekable_getFrameDecompressedOffset(ZSTD_seekable* const zs, unsigned frameIndex);
-ZSTDLIB_API size_t ZSTD_seekable_getFrameCompressedSize(ZSTD_seekable* const zs, unsigned frameIndex);
-ZSTDLIB_API size_t ZSTD_seekable_getFrameDecompressedSize(ZSTD_seekable* const zs, unsigned frameIndex);
-ZSTDLIB_API unsigned ZSTD_seekable_offsetToFrameIndex(ZSTD_seekable* const zs, unsigned long long offset);
-
-/*===== Seekable advanced I/O API =====*/
-typedef int(ZSTD_seekable_read)(void* opaque, void* buffer, size_t n);
-typedef int(ZSTD_seekable_seek)(void* opaque, long long offset, int origin);
-typedef struct {
-    void* opaque;
-    ZSTD_seekable_read* read;
-    ZSTD_seekable_seek* seek;
-} ZSTD_seekable_customFile;
-ZSTDLIB_API size_t ZSTD_seekable_initAdvanced(ZSTD_seekable* zs, ZSTD_seekable_customFile src);
-
-#if defined (__cplusplus)
-}
-#endif
-
-#endif
diff --git a/contrib/seekable_format/zstd_seekable_compression_format.md b/contrib/seekable_format/zstd_seekable_compression_format.md
deleted file mode 100644
index bf3080f7bbed..000000000000
--- a/contrib/seekable_format/zstd_seekable_compression_format.md
+++ /dev/null
@@ -1,116 +0,0 @@
-# Zstandard Seekable Format
-
-### Notices
-
-Copyright (c) 2017-present Facebook, Inc.
-
-Permission is granted to copy and distribute this document
-for any purpose and without charge,
-including translations into other languages
-and incorporation into compilations,
-provided that the copyright notice and this notice are preserved,
-and that any substantive changes or deletions from the original
-are clearly marked.
-Distribution of this document is unlimited.
-
-### Version
-0.1.0 (11/04/17)
-
-## Introduction
-This document defines a format for compressed data to be stored so that subranges of the data can be efficiently decompressed without requiring the entire document to be decompressed.
-This is done by splitting up the input data into frames,
-each of which are compressed independently,
-and so can be decompressed independently.
-Decompression then takes advantage of a provided 'seek table', which allows the decompressor to immediately jump to the desired data.  This is done in a way that is compatible with the original Zstandard format by placing the seek table in a Zstandard skippable frame.
-
-### Overall conventions
-In this document:
-- square brackets i.e. `[` and `]` are used to indicate optional fields or parameters.
-- the naming convention for identifiers is `Mixed_Case_With_Underscores`
-- All numeric fields are little-endian unless specified otherwise
-
-## Format
-
-The format consists of a number of frames (Zstandard compressed frames and skippable frames), followed by a final skippable frame at the end containing the seek table.
-
-### Seek Table Format
-The structure of the seek table frame is as follows:
-
-|`Skippable_Magic_Number`|`Frame_Size`|`[Seek_Table_Entries]`|`Seek_Table_Footer`|
-|------------------------|------------|----------------------|-------------------|
-| 4 bytes                | 4 bytes    | 8-12 bytes each      | 9 bytes           |
-
-__`Skippable_Magic_Number`__
-
-Value : 0x184D2A5E.
-This is for compatibility with [Zstandard skippable frames].
-Since it is legal for other Zstandard skippable frames to use the same
-magic number, it is not recommended for a decoder to recognize frames
-solely on this.
-
-__`Frame_Size`__
-
-The total size of the skippable frame, not including the `Skippable_Magic_Number` or `Frame_Size`.
-This is for compatibility with [Zstandard skippable frames].
-
-[Zstandard skippable frames]: https://github.com/facebook/zstd/blob/master/doc/zstd_compression_format.md#skippable-frames
-
-#### `Seek_Table_Footer`
-The seek table footer format is as follows:
-
-|`Number_Of_Frames`|`Seek_Table_Descriptor`|`Seekable_Magic_Number`|
-|------------------|-----------------------|-----------------------|
-| 4 bytes          | 1 byte                | 4 bytes               |
-
-__`Seekable_Magic_Number`__
-
-Value : 0x8F92EAB1.
-This value must be the last bytes present in the compressed file so that decoders
-can efficiently find it and determine if there is an actual seek table present.
-
-__`Number_Of_Frames`__
-
-The number of stored frames in the data.
-
-__`Seek_Table_Descriptor`__
-
-A bitfield describing the format of the seek table.
-
-| Bit number | Field name                |
-| ---------- | ----------                |
-| 7          | `Checksum_Flag`           |
-| 6-2        | `Reserved_Bits`           |
-| 1-0        | `Unused_Bits`             |
-
-While only `Checksum_Flag` currently exists, there are 7 other bits in this field that can be used for future changes to the format,
-for example the addition of inline dictionaries.
-
-__`Checksum_Flag`__
-
-If the checksum flag is set, each of the seek table entries contains a 4 byte checksum of the uncompressed data contained in its frame.
-
-`Reserved_Bits` are not currently used but may be used in the future for breaking changes, so a compliant decoder should ensure they are set to 0.  `Unused_Bits` may be used in the future for non-breaking changes, so a compliant decoder should not interpret these bits.
-
-#### __`Seek_Table_Entries`__
-
-`Seek_Table_Entries` consists of `Number_Of_Frames` (one for each frame in the data, not including the seek table frame) entries of the following form, in sequence:
-
-|`Compressed_Size`|`Decompressed_Size`|`[Checksum]`|
-|-----------------|-------------------|------------|
-| 4 bytes         | 4 bytes           | 4 bytes    |
-
-__`Compressed_Size`__
-
-The compressed size of the frame.
-The cumulative sum of the `Compressed_Size` fields of frames `0` to `i` gives the offset in the compressed file of frame `i+1`.
-
-__`Decompressed_Size`__
-
-The size of the decompressed data contained in the frame.  For skippable or otherwise empty frames, this value is 0.
-
-__`Checksum`__
-
-Only present if `Checksum_Flag` is set in the `Seek_Table_Descriptor`.  Value : the least significant 32 bits of the XXH64 digest of the uncompressed data, stored in little-endian format.
-
-## Version Changes
-- 0.1.0: initial version
diff --git a/contrib/seekable_format/zstdseek_compress.c b/contrib/seekable_format/zstdseek_compress.c
deleted file mode 100644
index 5a75714fac5b..000000000000
--- a/contrib/seekable_format/zstdseek_compress.c
+++ /dev/null
@@ -1,369 +0,0 @@
-/*
- * Copyright (c) 2017-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-
-#include <stdlib.h>     /* malloc, free */
-#include <limits.h>     /* UINT_MAX */
-#include <assert.h>
-
-#define XXH_STATIC_LINKING_ONLY
-#define XXH_NAMESPACE ZSTD_
-#include "xxhash.h"
-
-#define ZSTD_STATIC_LINKING_ONLY
-#include "zstd.h"
-#include "zstd_errors.h"
-#include "mem.h"
-#include "zstd_seekable.h"
-
-#define CHECK_Z(f) { size_t const ret = (f); if (ret != 0) return ret; }
-
-#undef ERROR
-#define ERROR(name) ((size_t)-ZSTD_error_##name)
-
-#undef MIN
-#undef MAX
-#define MIN(a, b) ((a) < (b) ? (a) : (b))
-#define MAX(a, b) ((a) > (b) ? (a) : (b))
-
-typedef struct {
-    U32 cSize;
-    U32 dSize;
-    U32 checksum;
-} framelogEntry_t;
-
-struct ZSTD_frameLog_s {
-    framelogEntry_t* entries;
-    U32 size;
-    U32 capacity;
-
-    int checksumFlag;
-
-    /* for use when streaming out the seek table */
-    U32 seekTablePos;
-    U32 seekTableIndex;
-} framelog_t;
-
-struct ZSTD_seekable_CStream_s {
-    ZSTD_CStream* cstream;
-    ZSTD_frameLog framelog;
-
-    U32 frameCSize;
-    U32 frameDSize;
-
-    XXH64_state_t xxhState;
-
-    U32 maxFrameSize;
-
-    int writingSeekTable;
-};
-
-size_t ZSTD_seekable_frameLog_allocVec(ZSTD_frameLog* fl)
-{
-    /* allocate some initial space */
-    size_t const FRAMELOG_STARTING_CAPACITY = 16;
-    fl->entries = (framelogEntry_t*)malloc(
-            sizeof(framelogEntry_t) * FRAMELOG_STARTING_CAPACITY);
-    if (fl->entries == NULL) return ERROR(memory_allocation);
-    fl->capacity = FRAMELOG_STARTING_CAPACITY;
-
-    return 0;
-}
-
-size_t ZSTD_seekable_frameLog_freeVec(ZSTD_frameLog* fl)
-{
-    if (fl != NULL) free(fl->entries);
-    return 0;
-}
-
-ZSTD_frameLog* ZSTD_seekable_createFrameLog(int checksumFlag)
-{
-    ZSTD_frameLog* fl = malloc(sizeof(ZSTD_frameLog));
-    if (fl == NULL) return NULL;
-
-    if (ZSTD_isError(ZSTD_seekable_frameLog_allocVec(fl))) {
-        free(fl);
-        return NULL;
-    }
-
-    fl->checksumFlag = checksumFlag;
-    fl->seekTablePos = 0;
-    fl->seekTableIndex = 0;
-    fl->size = 0;
-
-    return fl;
-}
-
-size_t ZSTD_seekable_freeFrameLog(ZSTD_frameLog* fl)
-{
-    ZSTD_seekable_frameLog_freeVec(fl);
-    free(fl);
-    return 0;
-}
-
-ZSTD_seekable_CStream* ZSTD_seekable_createCStream()
-{
-    ZSTD_seekable_CStream* zcs = malloc(sizeof(ZSTD_seekable_CStream));
-
-    if (zcs == NULL) return NULL;
-
-    memset(zcs, 0, sizeof(*zcs));
-
-    zcs->cstream = ZSTD_createCStream();
-    if (zcs->cstream == NULL) goto failed1;
-
-    if (ZSTD_isError(ZSTD_seekable_frameLog_allocVec(&zcs->framelog))) goto failed2;
-
-    return zcs;
-
-failed2:
-    ZSTD_freeCStream(zcs->cstream);
-failed1:
-    free(zcs);
-    return NULL;
-}
-
-size_t ZSTD_seekable_freeCStream(ZSTD_seekable_CStream* zcs)
-{
-    if (zcs == NULL) return 0; /* support free on null */
-    ZSTD_freeCStream(zcs->cstream);
-    ZSTD_seekable_frameLog_freeVec(&zcs->framelog);
-    free(zcs);
-
-    return 0;
-}
-
-size_t ZSTD_seekable_initCStream(ZSTD_seekable_CStream* zcs,
-                                 int compressionLevel,
-                                 int checksumFlag,
-                                 unsigned maxFrameSize)
-{
-    zcs->framelog.size = 0;
-    zcs->frameCSize = 0;
-    zcs->frameDSize = 0;
-
-    /* make sure maxFrameSize has a reasonable value */
-    if (maxFrameSize > ZSTD_SEEKABLE_MAX_FRAME_DECOMPRESSED_SIZE) {
-        return ERROR(frameParameter_unsupported);
-    }
-
-    zcs->maxFrameSize = maxFrameSize
-                                ? maxFrameSize
-                                : ZSTD_SEEKABLE_MAX_FRAME_DECOMPRESSED_SIZE;
-
-    zcs->framelog.checksumFlag = checksumFlag;
-    if (zcs->framelog.checksumFlag) {
-        XXH64_reset(&zcs->xxhState, 0);
-    }
-
-    zcs->framelog.seekTablePos = 0;
-    zcs->framelog.seekTableIndex = 0;
-    zcs->writingSeekTable = 0;
-
-    return ZSTD_initCStream(zcs->cstream, compressionLevel);
-}
-
-size_t ZSTD_seekable_logFrame(ZSTD_frameLog* fl,
-                              unsigned compressedSize,
-                              unsigned decompressedSize,
-                              unsigned checksum)
-{
-    if (fl->size == ZSTD_SEEKABLE_MAXFRAMES)
-        return ERROR(frameIndex_tooLarge);
-
-    /* grow the buffer if required */
-    if (fl->size == fl->capacity) {
-        /* exponential size increase for constant amortized runtime */
-        size_t const newCapacity = fl->capacity * 2;
-        framelogEntry_t* const newEntries = realloc(fl->entries,
-                sizeof(framelogEntry_t) * newCapacity);
-
-        if (newEntries == NULL) return ERROR(memory_allocation);
-
-        fl->entries = newEntries;
-        assert(newCapacity <= UINT_MAX);
-        fl->capacity = (U32)newCapacity;
-    }
-
-    fl->entries[fl->size] = (framelogEntry_t){
-            compressedSize, decompressedSize, checksum
-    };
-    fl->size++;
-
-    return 0;
-}
-
-size_t ZSTD_seekable_endFrame(ZSTD_seekable_CStream* zcs, ZSTD_outBuffer* output)
-{
-    size_t const prevOutPos = output->pos;
-    /* end the frame */
-    size_t ret = ZSTD_endStream(zcs->cstream, output);
-
-    zcs->frameCSize += output->pos - prevOutPos;
-
-    /* need to flush before doing the rest */
-    if (ret) return ret;
-
-    /* frame done */
-
-    /* store the frame data for later */
-    ret = ZSTD_seekable_logFrame(
-            &zcs->framelog, zcs->frameCSize, zcs->frameDSize,
-            zcs->framelog.checksumFlag
-                    ? XXH64_digest(&zcs->xxhState) & 0xFFFFFFFFU
-                    : 0);
-    if (ret) return ret;
-
-    /* reset for the next frame */
-    zcs->frameCSize = 0;
-    zcs->frameDSize = 0;
-
-    ZSTD_resetCStream(zcs->cstream, 0);
-    if (zcs->framelog.checksumFlag)
-        XXH64_reset(&zcs->xxhState, 0);
-
-    return 0;
-}
-
-size_t ZSTD_seekable_compressStream(ZSTD_seekable_CStream* zcs, ZSTD_outBuffer* output, ZSTD_inBuffer* input)
-{
-    const BYTE* const inBase = (const BYTE*) input->src + input->pos;
-    size_t inLen = input->size - input->pos;
-
-    inLen = MIN(inLen, (size_t)(zcs->maxFrameSize - zcs->frameDSize));
-
-    /* if we haven't finished flushing the last frame, don't start writing a new one */
-    if (inLen > 0) {
-        ZSTD_inBuffer inTmp = { inBase, inLen, 0 };
-        size_t const prevOutPos = output->pos;
-
-        size_t const ret = ZSTD_compressStream(zcs->cstream, output, &inTmp);
-
-        if (zcs->framelog.checksumFlag) {
-            XXH64_update(&zcs->xxhState, inBase, inTmp.pos);
-        }
-
-        zcs->frameCSize += output->pos - prevOutPos;
-        zcs->frameDSize += inTmp.pos;
-
-        input->pos += inTmp.pos;
-
-        if (ZSTD_isError(ret)) return ret;
-    }
-
-    if (zcs->maxFrameSize == zcs->frameDSize) {
-        /* log the frame and start over */
-        size_t const ret = ZSTD_seekable_endFrame(zcs, output);
-        if (ZSTD_isError(ret)) return ret;
-
-        /* get the client ready for the next frame */
-        return (size_t)zcs->maxFrameSize;
-    }
-
-    return (size_t)(zcs->maxFrameSize - zcs->frameDSize);
-}
-
-static inline size_t ZSTD_seekable_seekTableSize(const ZSTD_frameLog* fl)
-{
-    size_t const sizePerFrame = 8 + (fl->checksumFlag?4:0);
-    size_t const seekTableLen = ZSTD_SKIPPABLEHEADERSIZE +
-                                sizePerFrame * fl->size +
-                                ZSTD_seekTableFooterSize;
-
-    return seekTableLen;
-}
-
-static inline size_t ZSTD_stwrite32(ZSTD_frameLog* fl,
-                                    ZSTD_outBuffer* output, U32 const value,
-                                    U32 const offset)
-{
-    if (fl->seekTablePos < offset + 4) {
-        BYTE tmp[4]; /* so that we can work with buffers too small to write a whole word to */
-        size_t const lenWrite =
-                MIN(output->size - output->pos, offset + 4 - fl->seekTablePos);
-        MEM_writeLE32(tmp, value);
-        memcpy((BYTE*)output->dst + output->pos,
-               tmp + (fl->seekTablePos - offset), lenWrite);
-        output->pos += lenWrite;
-        fl->seekTablePos += lenWrite;
-
-        if (lenWrite < 4) return ZSTD_seekable_seekTableSize(fl) - fl->seekTablePos;
-    }
-    return 0;
-}
-
-size_t ZSTD_seekable_writeSeekTable(ZSTD_frameLog* fl, ZSTD_outBuffer* output)
-{
-    /* seekTableIndex: the current index in the table and
-     * seekTableSize: the amount of the table written so far
-     *
-     * This function is written this way so that if it has to return early
-     * because of a small buffer, it can keep going where it left off.
-     */
-
-    size_t const sizePerFrame = 8 + (fl->checksumFlag?4:0);
-    size_t const seekTableLen = ZSTD_seekable_seekTableSize(fl);
-
-    CHECK_Z(ZSTD_stwrite32(fl, output, ZSTD_MAGIC_SKIPPABLE_START | 0xE, 0));
-    assert(seekTableLen <= (size_t)UINT_MAX);
-    CHECK_Z(ZSTD_stwrite32(fl, output, (U32)seekTableLen - ZSTD_SKIPPABLEHEADERSIZE, 4));
-
-    while (fl->seekTableIndex < fl->size) {
-        unsigned long long const start = ZSTD_SKIPPABLEHEADERSIZE + sizePerFrame * fl->seekTableIndex;
-        assert(start + 8 <= UINT_MAX);
-        CHECK_Z(ZSTD_stwrite32(fl, output,
-                               fl->entries[fl->seekTableIndex].cSize,
-                               (U32)start + 0));
-
-        CHECK_Z(ZSTD_stwrite32(fl, output,
-                               fl->entries[fl->seekTableIndex].dSize,
-                               (U32)start + 4));
-
-        if (fl->checksumFlag) {
-            CHECK_Z(ZSTD_stwrite32(
-                    fl, output, fl->entries[fl->seekTableIndex].checksum,
-                    (U32)start + 8));
-        }
-
-        fl->seekTableIndex++;
-    }
-
-    assert(seekTableLen <= UINT_MAX);
-    CHECK_Z(ZSTD_stwrite32(fl, output, fl->size,
-                           (U32)seekTableLen - ZSTD_seekTableFooterSize));
-
-    if (output->size - output->pos < 1) return seekTableLen - fl->seekTablePos;
-    if (fl->seekTablePos < seekTableLen - 4) {
-        BYTE sfd = 0;
-        sfd |= (fl->checksumFlag) << 7;
-
-        ((BYTE*)output->dst)[output->pos] = sfd;
-        output->pos++;
-        fl->seekTablePos++;
-    }
-
-    CHECK_Z(ZSTD_stwrite32(fl, output, ZSTD_SEEKABLE_MAGICNUMBER,
-                           (U32)seekTableLen - 4));
-
-    if (fl->seekTablePos != seekTableLen) return ERROR(GENERIC);
-    return 0;
-}
-
-size_t ZSTD_seekable_endStream(ZSTD_seekable_CStream* zcs, ZSTD_outBuffer* output)
-{
-    if (!zcs->writingSeekTable && zcs->frameDSize) {
-        const size_t endFrame = ZSTD_seekable_endFrame(zcs, output);
-        if (ZSTD_isError(endFrame)) return endFrame;
-        /* return an accurate size hint */
-        if (endFrame) return endFrame + ZSTD_seekable_seekTableSize(&zcs->framelog);
-    }
-
-    zcs->writingSeekTable = 1;
-
-    return ZSTD_seekable_writeSeekTable(&zcs->framelog, output);
-}
diff --git a/contrib/seekable_format/zstdseek_decompress.c b/contrib/seekable_format/zstdseek_decompress.c
deleted file mode 100644
index abfd1e902717..000000000000
--- a/contrib/seekable_format/zstdseek_decompress.c
+++ /dev/null
@@ -1,467 +0,0 @@
-/*
- * Copyright (c) 2017-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-/* *********************************************************
-*  Turn on Large Files support (>4GB) for 32-bit Linux/Unix
-***********************************************************/
-#if !defined(__64BIT__) || defined(__MINGW32__)       /* No point defining Large file for 64 bit but MinGW-w64 requires it */
-#  if !defined(_FILE_OFFSET_BITS)
-#    define _FILE_OFFSET_BITS 64                      /* turn off_t into a 64-bit type for ftello, fseeko */
-#  endif
-#  if !defined(_LARGEFILE_SOURCE)                     /* obsolete macro, replaced with _FILE_OFFSET_BITS */
-#    define _LARGEFILE_SOURCE 1                       /* Large File Support extension (LFS) - fseeko, ftello */
-#  endif
-#  if defined(_AIX) || defined(__hpux)
-#    define _LARGE_FILES                              /* Large file support on 32-bits AIX and HP-UX */
-#  endif
-#endif
-
-/* ************************************************************
-* Avoid fseek()'s 2GiB barrier with MSVC, macOS, *BSD, MinGW
-***************************************************************/
-#if defined(_MSC_VER) && _MSC_VER >= 1400
-#   define LONG_SEEK _fseeki64
-#elif !defined(__64BIT__) && (PLATFORM_POSIX_VERSION >= 200112L) /* No point defining Large file for 64 bit */
-#  define LONG_SEEK fseeko
-#elif defined(__MINGW32__) && !defined(__STRICT_ANSI__) && !defined(__NO_MINGW_LFS) && defined(__MSVCRT__)
-#   define LONG_SEEK fseeko64
-#elif defined(_WIN32) && !defined(__DJGPP__)
-#   include <windows.h>
-    static int LONG_SEEK(FILE* file, __int64 offset, int origin) {
-        LARGE_INTEGER off;
-        DWORD method;
-        off.QuadPart = offset;
-        if (origin == SEEK_END)
-            method = FILE_END;
-        else if (origin == SEEK_CUR)
-            method = FILE_CURRENT;
-        else
-            method = FILE_BEGIN;
-
-        if (SetFilePointerEx((HANDLE) _get_osfhandle(_fileno(file)), off, NULL, method))
-            return 0;
-        else
-            return -1;
-    }
-#else
-#   define LONG_SEEK fseek
-#endif
-
-#include <stdlib.h>  /* malloc, free */
-#include <stdio.h>   /* FILE* */
-#include <limits.h>  /* UNIT_MAX */
-#include <assert.h>
-
-#define XXH_STATIC_LINKING_ONLY
-#define XXH_NAMESPACE ZSTD_
-#include "xxhash.h"
-
-#define ZSTD_STATIC_LINKING_ONLY
-#include "zstd.h"
-#include "zstd_errors.h"
-#include "mem.h"
-#include "zstd_seekable.h"
-
-#undef ERROR
-#define ERROR(name) ((size_t)-ZSTD_error_##name)
-
-#define CHECK_IO(f) { int const errcod = (f); if (errcod < 0) return ERROR(seekableIO); }
-
-#undef MIN
-#undef MAX
-#define MIN(a, b) ((a) < (b) ? (a) : (b))
-#define MAX(a, b) ((a) > (b) ? (a) : (b))
-
-/* Special-case callbacks for FILE* and in-memory modes, so that we can treat
- * them the same way as the advanced API */
-static int ZSTD_seekable_read_FILE(void* opaque, void* buffer, size_t n)
-{
-    size_t const result = fread(buffer, 1, n, (FILE*)opaque);
-    if (result != n) {
-        return -1;
-    }
-    return 0;
-}
-
-static int ZSTD_seekable_seek_FILE(void* opaque, long long offset, int origin)
-{
-    int const ret = LONG_SEEK((FILE*)opaque, offset, origin);
-    if (ret) return ret;
-    return fflush((FILE*)opaque);
-}
-
-typedef struct {
-    const void *ptr;
-    size_t size;
-    size_t pos;
-} buffWrapper_t;
-
-static int ZSTD_seekable_read_buff(void* opaque, void* buffer, size_t n)
-{
-    buffWrapper_t* buff = (buffWrapper_t*) opaque;
-    if (buff->pos + n > buff->size) return -1;
-    memcpy(buffer, (const BYTE*)buff->ptr + buff->pos, n);
-    buff->pos += n;
-    return 0;
-}
-
-static int ZSTD_seekable_seek_buff(void* opaque, long long offset, int origin)
-{
-    buffWrapper_t* const buff = (buffWrapper_t*) opaque;
-    unsigned long long newOffset;
-    switch (origin) {
-    case SEEK_SET:
-        newOffset = offset;
-        break;
-    case SEEK_CUR:
-        newOffset = (unsigned long long)buff->pos + offset;
-        break;
-    case SEEK_END:
-        newOffset = (unsigned long long)buff->size + offset;
-        break;
-    default:
-        assert(0);  /* not possible */
-    }
-    if (newOffset > buff->size) {
-        return -1;
-    }
-    buff->pos = newOffset;
-    return 0;
-}
-
-typedef struct {
-    U64 cOffset;
-    U64 dOffset;
-    U32 checksum;
-} seekEntry_t;
-
-typedef struct {
-    seekEntry_t* entries;
-    size_t tableLen;
-
-    int checksumFlag;
-} seekTable_t;
-
-#define SEEKABLE_BUFF_SIZE ZSTD_BLOCKSIZE_MAX
-
-struct ZSTD_seekable_s {
-    ZSTD_DStream* dstream;
-    seekTable_t seekTable;
-    ZSTD_seekable_customFile src;
-
-    U64 decompressedOffset;
-    U32 curFrame;
-
-    BYTE inBuff[SEEKABLE_BUFF_SIZE]; /* need to do our own input buffering */
-    BYTE outBuff[SEEKABLE_BUFF_SIZE]; /* so we can efficiently decompress the
-                                         starts of chunks before we get to the
-                                         desired section */
-    ZSTD_inBuffer in; /* maintain continuity across ZSTD_seekable_decompress operations */
-    buffWrapper_t buffWrapper; /* for `src.opaque` in in-memory mode */
-
-    XXH64_state_t xxhState;
-};
-
-ZSTD_seekable* ZSTD_seekable_create(void)
-{
-    ZSTD_seekable* zs = malloc(sizeof(ZSTD_seekable));
-
-    if (zs == NULL) return NULL;
-
-    /* also initializes stage to zsds_init */
-    memset(zs, 0, sizeof(*zs));
-
-    zs->dstream = ZSTD_createDStream();
-    if (zs->dstream == NULL) {
-        free(zs);
-        return NULL;
-    }
-
-    return zs;
-}
-
-size_t ZSTD_seekable_free(ZSTD_seekable* zs)
-{
-    if (zs == NULL) return 0; /* support free on null */
-    ZSTD_freeDStream(zs->dstream);
-    free(zs->seekTable.entries);
-    free(zs);
-
-    return 0;
-}
-
-/** ZSTD_seekable_offsetToFrameIndex() :
- *  Performs a binary search to find the last frame with a decompressed offset
- *  <= pos
- *  @return : the frame's index */
-unsigned ZSTD_seekable_offsetToFrameIndex(ZSTD_seekable* const zs, unsigned long long pos)
-{
-    U32 lo = 0;
-    U32 hi = (U32)zs->seekTable.tableLen;
-    assert(zs->seekTable.tableLen <= UINT_MAX);
-
-    if (pos >= zs->seekTable.entries[zs->seekTable.tableLen].dOffset) {
-        return (U32)zs->seekTable.tableLen;
-    }
-
-    while (lo + 1 < hi) {
-        U32 const mid = lo + ((hi - lo) >> 1);
-        if (zs->seekTable.entries[mid].dOffset <= pos) {
-            lo = mid;
-        } else {
-            hi = mid;
-        }
-    }
-    return lo;
-}
-
-unsigned ZSTD_seekable_getNumFrames(ZSTD_seekable* const zs)
-{
-    assert(zs->seekTable.tableLen <= UINT_MAX);
-    return (unsigned)zs->seekTable.tableLen;
-}
-
-unsigned long long ZSTD_seekable_getFrameCompressedOffset(ZSTD_seekable* const zs, unsigned frameIndex)
-{
-    if (frameIndex >= zs->seekTable.tableLen) return ZSTD_SEEKABLE_FRAMEINDEX_TOOLARGE;
-    return zs->seekTable.entries[frameIndex].cOffset;
-}
-
-unsigned long long ZSTD_seekable_getFrameDecompressedOffset(ZSTD_seekable* const zs, unsigned frameIndex)
-{
-    if (frameIndex >= zs->seekTable.tableLen) return ZSTD_SEEKABLE_FRAMEINDEX_TOOLARGE;
-    return zs->seekTable.entries[frameIndex].dOffset;
-}
-
-size_t ZSTD_seekable_getFrameCompressedSize(ZSTD_seekable* const zs, unsigned frameIndex)
-{
-    if (frameIndex >= zs->seekTable.tableLen) return ERROR(frameIndex_tooLarge);
-    return zs->seekTable.entries[frameIndex + 1].cOffset -
-           zs->seekTable.entries[frameIndex].cOffset;
-}
-
-size_t ZSTD_seekable_getFrameDecompressedSize(ZSTD_seekable* const zs, unsigned frameIndex)
-{
-    if (frameIndex > zs->seekTable.tableLen) return ERROR(frameIndex_tooLarge);
-    return zs->seekTable.entries[frameIndex + 1].dOffset -
-           zs->seekTable.entries[frameIndex].dOffset;
-}
-
-static size_t ZSTD_seekable_loadSeekTable(ZSTD_seekable* zs)
-{
-    int checksumFlag;
-    ZSTD_seekable_customFile src = zs->src;
-    /* read the footer, fixed size */
-    CHECK_IO(src.seek(src.opaque, -(int)ZSTD_seekTableFooterSize, SEEK_END));
-    CHECK_IO(src.read(src.opaque, zs->inBuff, ZSTD_seekTableFooterSize));
-
-    if (MEM_readLE32(zs->inBuff + 5) != ZSTD_SEEKABLE_MAGICNUMBER) {
-        return ERROR(prefix_unknown);
-    }
-
-    {   BYTE const sfd = zs->inBuff[4];
-        checksumFlag = sfd >> 7;
-
-        /* check reserved bits */
-        if ((checksumFlag >> 2) & 0x1f) {
-            return ERROR(corruption_detected);
-        }
-    }
-
-    {   U32 const numFrames = MEM_readLE32(zs->inBuff);
-        U32 const sizePerEntry = 8 + (checksumFlag?4:0);
-        U32 const tableSize = sizePerEntry * numFrames;
-        U32 const frameSize = tableSize + ZSTD_seekTableFooterSize + ZSTD_SKIPPABLEHEADERSIZE;
-
-        U32 remaining = frameSize - ZSTD_seekTableFooterSize; /* don't need to re-read footer */
-        {
-            U32 const toRead = MIN(remaining, SEEKABLE_BUFF_SIZE);
-
-            CHECK_IO(src.seek(src.opaque, -(S64)frameSize, SEEK_END));
-            CHECK_IO(src.read(src.opaque, zs->inBuff, toRead));
-
-            remaining -= toRead;
-        }
-
-        if (MEM_readLE32(zs->inBuff) != (ZSTD_MAGIC_SKIPPABLE_START | 0xE)) {
-            return ERROR(prefix_unknown);
-        }
-        if (MEM_readLE32(zs->inBuff+4) + ZSTD_SKIPPABLEHEADERSIZE != frameSize) {
-            return ERROR(prefix_unknown);
-        }
-
-        {   /* Allocate an extra entry at the end so that we can do size
-             * computations on the last element without special case */
-            seekEntry_t* entries = (seekEntry_t*)malloc(sizeof(seekEntry_t) * (numFrames + 1));
-
-            U32 idx = 0;
-            U32 pos = 8;
-
-
-            U64 cOffset = 0;
-            U64 dOffset = 0;
-
-            if (!entries) {
-                free(entries);
-                return ERROR(memory_allocation);
-            }
-
-            /* compute cumulative positions */
-            for (; idx < numFrames; idx++) {
-                if (pos + sizePerEntry > SEEKABLE_BUFF_SIZE) {
-                    U32 const offset = SEEKABLE_BUFF_SIZE - pos;
-                    U32 const toRead = MIN(remaining, SEEKABLE_BUFF_SIZE - offset);
-                    memmove(zs->inBuff, zs->inBuff + pos, offset); /* move any data we haven't read yet */
-                    CHECK_IO(src.read(src.opaque, zs->inBuff+offset, toRead));
-                    remaining -= toRead;
-                    pos = 0;
-                }
-                entries[idx].cOffset = cOffset;
-                entries[idx].dOffset = dOffset;
-
-                cOffset += MEM_readLE32(zs->inBuff + pos);
-                pos += 4;
-                dOffset += MEM_readLE32(zs->inBuff + pos);
-                pos += 4;
-                if (checksumFlag) {
-                    entries[idx].checksum = MEM_readLE32(zs->inBuff + pos);
-                    pos += 4;
-                }
-            }
-            entries[numFrames].cOffset = cOffset;
-            entries[numFrames].dOffset = dOffset;
-
-            zs->seekTable.entries = entries;
-            zs->seekTable.tableLen = numFrames;
-            zs->seekTable.checksumFlag = checksumFlag;
-            return 0;
-        }
-    }
-}
-
-size_t ZSTD_seekable_initBuff(ZSTD_seekable* zs, const void* src, size_t srcSize)
-{
-    zs->buffWrapper = (buffWrapper_t){src, srcSize, 0};
-    {   ZSTD_seekable_customFile srcFile = {&zs->buffWrapper,
-                                            &ZSTD_seekable_read_buff,
-                                            &ZSTD_seekable_seek_buff};
-        return ZSTD_seekable_initAdvanced(zs, srcFile); }
-}
-
-size_t ZSTD_seekable_initFile(ZSTD_seekable* zs, FILE* src)
-{
-    ZSTD_seekable_customFile srcFile = {src, &ZSTD_seekable_read_FILE,
-                                        &ZSTD_seekable_seek_FILE};
-    return ZSTD_seekable_initAdvanced(zs, srcFile);
-}
-
-size_t ZSTD_seekable_initAdvanced(ZSTD_seekable* zs, ZSTD_seekable_customFile src)
-{
-    zs->src = src;
-
-    {   const size_t seekTableInit = ZSTD_seekable_loadSeekTable(zs);
-        if (ZSTD_isError(seekTableInit)) return seekTableInit; }
-
-    zs->decompressedOffset = (U64)-1;
-    zs->curFrame = (U32)-1;
-
-    {   const size_t dstreamInit = ZSTD_initDStream(zs->dstream);
-        if (ZSTD_isError(dstreamInit)) return dstreamInit; }
-    return 0;
-}
-
-size_t ZSTD_seekable_decompress(ZSTD_seekable* zs, void* dst, size_t len, unsigned long long offset)
-{
-    U32 targetFrame = ZSTD_seekable_offsetToFrameIndex(zs, offset);
-    do {
-        /* check if we can continue from a previous decompress job */
-        if (targetFrame != zs->curFrame || offset != zs->decompressedOffset) {
-            zs->decompressedOffset = zs->seekTable.entries[targetFrame].dOffset;
-            zs->curFrame = targetFrame;
-
-            CHECK_IO(zs->src.seek(zs->src.opaque,
-                                  zs->seekTable.entries[targetFrame].cOffset,
-                                  SEEK_SET));
-            zs->in = (ZSTD_inBuffer){zs->inBuff, 0, 0};
-            XXH64_reset(&zs->xxhState, 0);
-            ZSTD_resetDStream(zs->dstream);
-        }
-
-        while (zs->decompressedOffset < offset + len) {
-            size_t toRead;
-            ZSTD_outBuffer outTmp;
-            size_t prevOutPos;
-            if (zs->decompressedOffset < offset) {
-                /* dummy decompressions until we get to the target offset */
-                outTmp = (ZSTD_outBuffer){zs->outBuff, MIN(SEEKABLE_BUFF_SIZE, offset - zs->decompressedOffset), 0};
-            } else {
-                outTmp = (ZSTD_outBuffer){dst, len, zs->decompressedOffset - offset};
-            }
-
-            prevOutPos = outTmp.pos;
-            toRead = ZSTD_decompressStream(zs->dstream, &outTmp, &zs->in);
-            if (ZSTD_isError(toRead)) {
-                return toRead;
-            }
-
-            if (zs->seekTable.checksumFlag) {
-                XXH64_update(&zs->xxhState, (BYTE*)outTmp.dst + prevOutPos,
-                             outTmp.pos - prevOutPos);
-            }
-            zs->decompressedOffset += outTmp.pos - prevOutPos;
-
-            if (toRead == 0) {
-                /* frame complete */
-
-                /* verify checksum */
-                if (zs->seekTable.checksumFlag &&
-                    (XXH64_digest(&zs->xxhState) & 0xFFFFFFFFU) !=
-                            zs->seekTable.entries[targetFrame].checksum) {
-                    return ERROR(corruption_detected);
-                }
-
-                if (zs->decompressedOffset < offset + len) {
-                    /* go back to the start and force a reset of the stream */
-                    targetFrame = ZSTD_seekable_offsetToFrameIndex(zs, zs->decompressedOffset);
-                }
-                break;
-            }
-
-            /* read in more data if we're done with this buffer */
-            if (zs->in.pos == zs->in.size) {
-                toRead = MIN(toRead, SEEKABLE_BUFF_SIZE);
-                CHECK_IO(zs->src.read(zs->src.opaque, zs->inBuff, toRead));
-                zs->in.size = toRead;
-                zs->in.pos = 0;
-            }
-        }
-    } while (zs->decompressedOffset != offset + len);
-
-    return len;
-}
-
-size_t ZSTD_seekable_decompressFrame(ZSTD_seekable* zs, void* dst, size_t dstSize, unsigned frameIndex)
-{
-    if (frameIndex >= zs->seekTable.tableLen) {
-        return ERROR(frameIndex_tooLarge);
-    }
-
-    {
-        size_t const decompressedSize =
-                zs->seekTable.entries[frameIndex + 1].dOffset -
-                zs->seekTable.entries[frameIndex].dOffset;
-        if (dstSize < decompressedSize) {
-            return ERROR(dstSize_tooSmall);
-        }
-        return ZSTD_seekable_decompress(
-                zs, dst, decompressedSize,
-                zs->seekTable.entries[frameIndex].dOffset);
-    }
-}
diff --git a/contrib/snap/snapcraft.yaml b/contrib/snap/snapcraft.yaml
deleted file mode 100644
index 0a77946ae0aa..000000000000
--- a/contrib/snap/snapcraft.yaml
+++ /dev/null
@@ -1,28 +0,0 @@
-name: zstd
-version: git
-summary: Zstandard - Fast real-time compression algorithm
-description: |
-  Zstandard, or zstd as short version, is a fast lossless compression
-  algorithm, targeting real-time compression scenarios at zlib-level and better
-  compression ratios. It's backed by a very fast entropy stage, provided by
-  Huff0 and FSE library
-
-grade: devel # must be 'stable' to release into candidate/stable channels
-confinement: devmode # use 'strict' once you have the right plugs and slots
-
-apps:
-  zstd:
-    command: usr/local/bin/zstd
-    plugs: [home, removable-media]
-  zstdgrep:
-    command: usr/local/bin/zstdgrep
-    plugs: [home, removable-media]
-  zstdless:
-    command: usr/local/bin/zstdless
-    plugs: [home, removable-media]
-
-parts:
-  zstd:
-    source: .
-    plugin: make
-    build-packages: [g++]
diff --git a/doc/educational_decoder/Makefile b/doc/educational_decoder/Makefile
index 704f867661a7..316c6eadc4ac 100644
--- a/doc/educational_decoder/Makefile
+++ b/doc/educational_decoder/Makefile
@@ -1,10 +1,11 @@
 # ################################################################
-# Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+# Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
 # All rights reserved.
 #
 # This source code is licensed under both the BSD-style license (found in the
 # LICENSE file in the root directory of this source tree) and the GPLv2 (found
 # in the COPYING file in the root directory of this source tree).
+# You may select, at your option, one of the above-listed licenses.
 # ################################################################
 
 ZSTD ?= zstd   # note: requires zstd installation on local system
@@ -36,7 +37,7 @@ harness: $(HARNESS_FILES)
 	$(CC) $(FLAGS) $^ -o $@
 
 clean:
-	@$(RM) harness
+	@$(RM) harness *.o
 	@$(RM) -rf harness.dSYM  # MacOS specific
 
 test: harness
@@ -59,4 +60,3 @@ test: harness
 	@./harness tmp.zst tmp dictionary
 	@$(DIFF) -s tmp README.md
 	@$(RM) tmp* dictionary
-	@$(MAKE) clean
diff --git a/doc/educational_decoder/README.md b/doc/educational_decoder/README.md
index e3b9bf58e5ae..c89451ca0784 100644
--- a/doc/educational_decoder/README.md
+++ b/doc/educational_decoder/README.md
@@ -13,6 +13,13 @@ It also contains implementations of Huffman and FSE table decoding.
 [Zstandard format specification]: https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md
 [format specification]: https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md
 
+While the library's primary objective is code clarity,
+it also happens to compile into a small object file.
+The object file can be made even smaller by removing error messages,
+using the macro directive `ZDEC_NO_MESSAGE` at compilation time.
+This can be reduced even further by foregoing dictionary support,
+by defining `ZDEC_NO_DICTIONARY`.
+
 `harness.c` provides a simple test harness around the decoder:
 
     harness <input-file> <output-file> [dictionary]
diff --git a/doc/educational_decoder/harness.c b/doc/educational_decoder/harness.c
index a704f6bdb29f..1403a6ed655b 100644
--- a/doc/educational_decoder/harness.c
+++ b/doc/educational_decoder/harness.c
@@ -1,10 +1,11 @@
 /*
- * Copyright (c) 2017-present, Facebook, Inc.
+ * Copyright (c) 2017-2020, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
  * LICENSE file in the root directory of this source tree) and the GPLv2 (found
  * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
  */
 
 #include <stdio.h>
@@ -21,108 +22,98 @@ typedef unsigned char u8;
 // Protect against allocating too much memory for output
 #define MAX_OUTPUT_SIZE ((size_t)1024 * 1024 * 1024)
 
-static size_t read_file(const char *path, u8 **ptr)
+// Error message then exit
+#define ERR_OUT(...) { fprintf(stderr, __VA_ARGS__); exit(1); }
+
+
+typedef struct {
+    u8* address;
+    size_t size;
+} buffer_s;
+
+static void freeBuffer(buffer_s b) { free(b.address); }
+
+static buffer_s read_file(const char *path)
 {
     FILE* const f = fopen(path, "rb");
-    if (!f) {
-        fprintf(stderr, "failed to open file %s \n", path);
-        exit(1);
-    }
+    if (!f) ERR_OUT("failed to open file %s \n", path);
 
     fseek(f, 0L, SEEK_END);
     size_t const size = (size_t)ftell(f);
     rewind(f);
 
-    *ptr = malloc(size);
-    if (!ptr) {
-        fprintf(stderr, "failed to allocate memory to hold %s \n", path);
-        exit(1);
-    }
+    void* const ptr = malloc(size);
+    if (!ptr) ERR_OUT("failed to allocate memory to hold %s \n", path);
 
-    size_t const read = fread(*ptr, 1, size, f);
-    if (read != size) {  /* must read everything in one pass */
-        fprintf(stderr, "error while reading file %s \n", path);
-        exit(1);
-    }
+    size_t const read = fread(ptr, 1, size, f);
+    if (read != size) ERR_OUT("error while reading file %s \n", path);
 
     fclose(f);
-
-    return read;
+    buffer_s const b = { ptr, size };
+    return b;
 }
 
-static void write_file(const char *path, const u8 *ptr, size_t size)
+static void write_file(const char* path, const u8* ptr, size_t size)
 {
     FILE* const f = fopen(path, "wb");
-    if (!f) {
-        fprintf(stderr, "failed to open file %s \n", path);
-        exit(1);
-    }
+    if (!f) ERR_OUT("failed to open file %s \n", path);
 
     size_t written = 0;
     while (written < size) {
         written += fwrite(ptr+written, 1, size, f);
-        if (ferror(f)) {
-            fprintf(stderr, "error while writing file %s\n", path);
-            exit(1);
-    }   }
+        if (ferror(f)) ERR_OUT("error while writing file %s\n", path);
+    }
 
     fclose(f);
 }
 
 int main(int argc, char **argv)
 {
-    if (argc < 3) {
-        fprintf(stderr, "usage: %s <file.zst> <out_path> [dictionary] \n",
-                argv[0]);
+    if (argc < 3)
+        ERR_OUT("usage: %s <file.zst> <out_path> [dictionary] \n", argv[0]);
 
-        return 1;
-    }
+    buffer_s const input = read_file(argv[1]);
 
-    u8* input;
-    size_t const input_size = read_file(argv[1], &input);
-
-    u8* dict = NULL;
-    size_t dict_size = 0;
+    buffer_s dict = { NULL, 0 };
     if (argc >= 4) {
-        dict_size = read_file(argv[3], &dict);
+        dict = read_file(argv[3]);
     }
 
-    size_t out_capacity = ZSTD_get_decompressed_size(input, input_size);
+    size_t out_capacity = ZSTD_get_decompressed_size(input.address, input.size);
     if (out_capacity == (size_t)-1) {
-        out_capacity = MAX_COMPRESSION_RATIO * input_size;
+        out_capacity = MAX_COMPRESSION_RATIO * input.size;
         fprintf(stderr, "WARNING: Compressed data does not contain "
                         "decompressed size, going to assume the compression "
                         "ratio is at most %d (decompressed size of at most "
                         "%u) \n",
                 MAX_COMPRESSION_RATIO, (unsigned)out_capacity);
     }
-    if (out_capacity > MAX_OUTPUT_SIZE) {
-        fprintf(stderr,
-                "Required output size too large for this implementation \n");
-        return 1;
-    }
+    if (out_capacity > MAX_OUTPUT_SIZE)
+        ERR_OUT("Required output size too large for this implementation \n");
 
     u8* const output = malloc(out_capacity);
-    if (!output) {
-        fprintf(stderr, "failed to allocate memory \n");
-        return 1;
-    }
+    if (!output) ERR_OUT("failed to allocate memory \n");
 
     dictionary_t* const parsed_dict = create_dictionary();
-    if (dict) {
-        parse_dictionary(parsed_dict, dict, dict_size);
+    if (dict.size) {
+#if defined (ZDEC_NO_DICTIONARY)
+        printf("dict.size = %zu \n", dict.size);
+        ERR_OUT("no dictionary support \n");
+#else
+        parse_dictionary(parsed_dict, dict.address, dict.size);
+#endif
     }
     size_t const decompressed_size =
         ZSTD_decompress_with_dict(output, out_capacity,
-                                  input, input_size,
+                                  input.address, input.size,
                                   parsed_dict);
 
     free_dictionary(parsed_dict);
 
     write_file(argv[2], output, decompressed_size);
 
-    free(input);
+    freeBuffer(input);
+    freeBuffer(dict);
     free(output);
-    free(dict);
     return 0;
 }
diff --git a/doc/educational_decoder/zstd_decompress.c b/doc/educational_decoder/zstd_decompress.c
index 64e1b8738d06..605918b39f85 100644
--- a/doc/educational_decoder/zstd_decompress.c
+++ b/doc/educational_decoder/zstd_decompress.c
@@ -1,34 +1,52 @@
 /*
- * Copyright (c) 2017-present, Facebook, Inc.
+ * Copyright (c) 2017-2020, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
  * LICENSE file in the root directory of this source tree) and the GPLv2 (found
  * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
  */
 
 /// Zstandard educational decoder implementation
 /// See https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md
 
-#include <stdint.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
+#include <stdint.h>   // uint8_t, etc.
+#include <stdlib.h>   // malloc, free, exit
+#include <stdio.h>    // fprintf
+#include <string.h>   // memset, memcpy
 #include "zstd_decompress.h"
 
-/******* UTILITY MACROS AND TYPES *********************************************/
-// Max block size decompressed size is 128 KB and literal blocks can't be
-// larger than their block
-#define MAX_LITERALS_SIZE ((size_t)128 * 1024)
 
+/******* IMPORTANT CONSTANTS *********************************************/
+
+// Zstandard frame
+// "Magic_Number
+// 4 Bytes, little-endian format. Value : 0xFD2FB528"
+#define ZSTD_MAGIC_NUMBER 0xFD2FB528U
+
+// The size of `Block_Content` is limited by `Block_Maximum_Size`,
+#define ZSTD_BLOCK_SIZE_MAX ((size_t)128 * 1024)
+
+// literal blocks can't be larger than their block
+#define MAX_LITERALS_SIZE ZSTD_BLOCK_SIZE_MAX
+
+
+/******* UTILITY MACROS AND TYPES *********************************************/
 #define MAX(a, b) ((a) > (b) ? (a) : (b))
 #define MIN(a, b) ((a) < (b) ? (a) : (b))
 
+#if defined(ZDEC_NO_MESSAGE)
+#define MESSAGE(...)
+#else
+#define MESSAGE(...)  fprintf(stderr, "" __VA_ARGS__)
+#endif
+
 /// This decoder calls exit(1) when it encounters an error, however a production
 /// library should propagate error codes
 #define ERROR(s)                                                               \
     do {                                                                       \
-        fprintf(stderr, "Error: %s\n", s);                                     \
+        MESSAGE("Error: %s\n", s);                                     \
         exit(1);                                                               \
     } while (0)
 #define INP_SIZE()                                                             \
@@ -39,12 +57,12 @@
 #define BAD_ALLOC() ERROR("Memory allocation error")
 #define IMPOSSIBLE() ERROR("An impossibility has occurred")
 
-typedef uint8_t u8;
+typedef uint8_t  u8;
 typedef uint16_t u16;
 typedef uint32_t u32;
 typedef uint64_t u64;
 
-typedef int8_t i8;
+typedef int8_t  i8;
 typedef int16_t i16;
 typedef int32_t i32;
 typedef int64_t i64;
@@ -176,10 +194,6 @@ static void HUF_init_dtable_usingweights(HUF_dtable *const table,
 
 /// Free the malloc'ed parts of a decoding table
 static void HUF_free_dtable(HUF_dtable *const dtable);
-
-/// Deep copy a decoding table, so that it can be used and free'd without
-/// impacting the source table.
-static void HUF_copy_dtable(HUF_dtable *const dst, const HUF_dtable *const src);
 /*** END HUFFMAN PRIMITIVES ***********/
 
 /*** FSE PRIMITIVES *******************/
@@ -241,10 +255,6 @@ static void FSE_init_dtable_rle(FSE_dtable *const dtable, const u8 symb);
 
 /// Free the malloc'ed parts of a decoding table
 static void FSE_free_dtable(FSE_dtable *const dtable);
-
-/// Deep copy a decoding table, so that it can be used and free'd without
-/// impacting the source table.
-static void FSE_copy_dtable(FSE_dtable *const dst, const FSE_dtable *const src);
 /*** END FSE PRIMITIVES ***************/
 
 /******* END IMPLEMENTATION PRIMITIVE PROTOTYPES ******************************/
@@ -373,7 +383,7 @@ static void execute_match_copy(frame_context_t *const ctx, size_t offset,
 
 size_t ZSTD_decompress(void *const dst, const size_t dst_len,
                        const void *const src, const size_t src_len) {
-    dictionary_t* uninit_dict = create_dictionary();
+    dictionary_t* const uninit_dict = create_dictionary();
     size_t const decomp_size = ZSTD_decompress_with_dict(dst, dst_len, src,
                                                          src_len, uninit_dict);
     free_dictionary(uninit_dict);
@@ -417,12 +427,7 @@ static void decompress_data(frame_context_t *const ctx, ostream_t *const out,
 static void decode_frame(ostream_t *const out, istream_t *const in,
                          const dictionary_t *const dict) {
     const u32 magic_number = (u32)IO_read_bits(in, 32);
-    // Zstandard frame
-    //
-    // "Magic_Number
-    //
-    // 4 Bytes, little-endian format. Value : 0xFD2FB528"
-    if (magic_number == 0xFD2FB528U) {
+    if (magic_number == ZSTD_MAGIC_NUMBER) {
         // ZSTD frame
         decode_data_frame(out, in, dict);
 
@@ -576,43 +581,6 @@ static void parse_frame_header(frame_header_t *const header,
     }
 }
 
-/// A dictionary acts as initializing values for the frame context before
-/// decompression, so we implement it by applying it's predetermined
-/// tables and content to the context before beginning decompression
-static void frame_context_apply_dict(frame_context_t *const ctx,
-                                     const dictionary_t *const dict) {
-    // If the content pointer is NULL then it must be an empty dict
-    if (!dict || !dict->content)
-        return;
-
-    // If the requested dictionary_id is non-zero, the correct dictionary must
-    // be present
-    if (ctx->header.dictionary_id != 0 &&
-        ctx->header.dictionary_id != dict->dictionary_id) {
-        ERROR("Wrong dictionary provided");
-    }
-
-    // Copy the dict content to the context for references during sequence
-    // execution
-    ctx->dict_content = dict->content;
-    ctx->dict_content_len = dict->content_size;
-
-    // If it's a formatted dict copy the precomputed tables in so they can
-    // be used in the table repeat modes
-    if (dict->dictionary_id != 0) {
-        // Deep copy the entropy tables so they can be freed independently of
-        // the dictionary struct
-        HUF_copy_dtable(&ctx->literals_dtable, &dict->literals_dtable);
-        FSE_copy_dtable(&ctx->ll_dtable, &dict->ll_dtable);
-        FSE_copy_dtable(&ctx->of_dtable, &dict->of_dtable);
-        FSE_copy_dtable(&ctx->ml_dtable, &dict->ml_dtable);
-
-        // Copy the repeated offsets
-        memcpy(ctx->previous_offsets, dict->previous_offsets,
-               sizeof(ctx->previous_offsets));
-    }
-}
-
 /// Decompress the data from a frame block by block
 static void decompress_data(frame_context_t *const ctx, ostream_t *const out,
                             istream_t *const in) {
@@ -1411,7 +1379,7 @@ size_t ZSTD_get_decompressed_size(const void *src, const size_t src_len) {
     {
         const u32 magic_number = (u32)IO_read_bits(&in, 32);
 
-        if (magic_number == 0xFD2FB528U) {
+        if (magic_number == ZSTD_MAGIC_NUMBER) {
             // ZSTD frame
             frame_header_t header;
             parse_frame_header(&header, &in);
@@ -1431,17 +1399,33 @@ size_t ZSTD_get_decompressed_size(const void *src, const size_t src_len) {
 /******* END OUTPUT SIZE COUNTING *********************************************/
 
 /******* DICTIONARY PARSING ***************************************************/
-#define DICT_SIZE_ERROR() ERROR("Dictionary size cannot be less than 8 bytes")
-#define NULL_SRC() ERROR("Tried to create dictionary with pointer to null src");
-
 dictionary_t* create_dictionary() {
-    dictionary_t* dict = calloc(1, sizeof(dictionary_t));
+    dictionary_t* const dict = calloc(1, sizeof(dictionary_t));
     if (!dict) {
         BAD_ALLOC();
     }
     return dict;
 }
 
+/// Free an allocated dictionary
+void free_dictionary(dictionary_t *const dict) {
+    HUF_free_dtable(&dict->literals_dtable);
+    FSE_free_dtable(&dict->ll_dtable);
+    FSE_free_dtable(&dict->of_dtable);
+    FSE_free_dtable(&dict->ml_dtable);
+
+    free(dict->content);
+
+    memset(dict, 0, sizeof(dictionary_t));
+
+    free(dict);
+}
+
+
+#if !defined(ZDEC_NO_DICTIONARY)
+#define DICT_SIZE_ERROR() ERROR("Dictionary size cannot be less than 8 bytes")
+#define NULL_SRC() ERROR("Tried to create dictionary with pointer to null src");
+
 static void init_dictionary_content(dictionary_t *const dict,
                                     istream_t *const in);
 
@@ -1513,19 +1497,93 @@ static void init_dictionary_content(dictionary_t *const dict,
     memcpy(dict->content, content, dict->content_size);
 }
 
-/// Free an allocated dictionary
-void free_dictionary(dictionary_t *const dict) {
-    HUF_free_dtable(&dict->literals_dtable);
-    FSE_free_dtable(&dict->ll_dtable);
-    FSE_free_dtable(&dict->of_dtable);
-    FSE_free_dtable(&dict->ml_dtable);
+static void HUF_copy_dtable(HUF_dtable *const dst,
+                            const HUF_dtable *const src) {
+    if (src->max_bits == 0) {
+        memset(dst, 0, sizeof(HUF_dtable));
+        return;
+    }
 
-    free(dict->content);
+    const size_t size = (size_t)1 << src->max_bits;
+    dst->max_bits = src->max_bits;
 
-    memset(dict, 0, sizeof(dictionary_t));
+    dst->symbols = malloc(size);
+    dst->num_bits = malloc(size);
+    if (!dst->symbols || !dst->num_bits) {
+        BAD_ALLOC();
+    }
 
-    free(dict);
+    memcpy(dst->symbols, src->symbols, size);
+    memcpy(dst->num_bits, src->num_bits, size);
 }
+
+static void FSE_copy_dtable(FSE_dtable *const dst, const FSE_dtable *const src) {
+    if (src->accuracy_log == 0) {
+        memset(dst, 0, sizeof(FSE_dtable));
+        return;
+    }
+
+    size_t size = (size_t)1 << src->accuracy_log;
+    dst->accuracy_log = src->accuracy_log;
+
+    dst->symbols = malloc(size);
+    dst->num_bits = malloc(size);
+    dst->new_state_base = malloc(size * sizeof(u16));
+    if (!dst->symbols || !dst->num_bits || !dst->new_state_base) {
+        BAD_ALLOC();
+    }
+
+    memcpy(dst->symbols, src->symbols, size);
+    memcpy(dst->num_bits, src->num_bits, size);
+    memcpy(dst->new_state_base, src->new_state_base, size * sizeof(u16));
+}
+
+/// A dictionary acts as initializing values for the frame context before
+/// decompression, so we implement it by applying it's predetermined
+/// tables and content to the context before beginning decompression
+static void frame_context_apply_dict(frame_context_t *const ctx,
+                                     const dictionary_t *const dict) {
+    // If the content pointer is NULL then it must be an empty dict
+    if (!dict || !dict->content)
+        return;
+
+    // If the requested dictionary_id is non-zero, the correct dictionary must
+    // be present
+    if (ctx->header.dictionary_id != 0 &&
+        ctx->header.dictionary_id != dict->dictionary_id) {
+        ERROR("Wrong dictionary provided");
+    }
+
+    // Copy the dict content to the context for references during sequence
+    // execution
+    ctx->dict_content = dict->content;
+    ctx->dict_content_len = dict->content_size;
+
+    // If it's a formatted dict copy the precomputed tables in so they can
+    // be used in the table repeat modes
+    if (dict->dictionary_id != 0) {
+        // Deep copy the entropy tables so they can be freed independently of
+        // the dictionary struct
+        HUF_copy_dtable(&ctx->literals_dtable, &dict->literals_dtable);
+        FSE_copy_dtable(&ctx->ll_dtable, &dict->ll_dtable);
+        FSE_copy_dtable(&ctx->of_dtable, &dict->of_dtable);
+        FSE_copy_dtable(&ctx->ml_dtable, &dict->ml_dtable);
+
+        // Copy the repeated offsets
+        memcpy(ctx->previous_offsets, dict->previous_offsets,
+               sizeof(ctx->previous_offsets));
+    }
+}
+
+#else  // ZDEC_NO_DICTIONARY is defined
+
+static void frame_context_apply_dict(frame_context_t *const ctx,
+                                     const dictionary_t *const dict) {
+    (void)ctx;
+    if (dict && dict->content) ERROR("dictionary not supported");
+}
+
+#endif
 /******* END DICTIONARY PARSING ***********************************************/
 
 /******* IO STREAM OPERATIONS *************************************************/
@@ -1945,26 +2003,6 @@ static void HUF_free_dtable(HUF_dtable *const dtable) {
     free(dtable->num_bits);
     memset(dtable, 0, sizeof(HUF_dtable));
 }
-
-static void HUF_copy_dtable(HUF_dtable *const dst,
-                            const HUF_dtable *const src) {
-    if (src->max_bits == 0) {
-        memset(dst, 0, sizeof(HUF_dtable));
-        return;
-    }
-
-    const size_t size = (size_t)1 << src->max_bits;
-    dst->max_bits = src->max_bits;
-
-    dst->symbols = malloc(size);
-    dst->num_bits = malloc(size);
-    if (!dst->symbols || !dst->num_bits) {
-        BAD_ALLOC();
-    }
-
-    memcpy(dst->symbols, src->symbols, size);
-    memcpy(dst->num_bits, src->num_bits, size);
-}
 /******* END HUFFMAN PRIMITIVES ***********************************************/
 
 /******* FSE PRIMITIVES *******************************************************/
@@ -2279,25 +2317,4 @@ static void FSE_free_dtable(FSE_dtable *const dtable) {
     free(dtable->new_state_base);
     memset(dtable, 0, sizeof(FSE_dtable));
 }
-
-static void FSE_copy_dtable(FSE_dtable *const dst, const FSE_dtable *const src) {
-    if (src->accuracy_log == 0) {
-        memset(dst, 0, sizeof(FSE_dtable));
-        return;
-    }
-
-    size_t size = (size_t)1 << src->accuracy_log;
-    dst->accuracy_log = src->accuracy_log;
-
-    dst->symbols = malloc(size);
-    dst->num_bits = malloc(size);
-    dst->new_state_base = malloc(size * sizeof(u16));
-    if (!dst->symbols || !dst->num_bits || !dst->new_state_base) {
-        BAD_ALLOC();
-    }
-
-    memcpy(dst->symbols, src->symbols, size);
-    memcpy(dst->num_bits, src->num_bits, size);
-    memcpy(dst->new_state_base, src->new_state_base, size * sizeof(u16));
-}
 /******* END FSE PRIMITIVES ***************************************************/
diff --git a/doc/educational_decoder/zstd_decompress.h b/doc/educational_decoder/zstd_decompress.h
index 74b18533850a..2b44eee95cec 100644
--- a/doc/educational_decoder/zstd_decompress.h
+++ b/doc/educational_decoder/zstd_decompress.h
@@ -1,10 +1,11 @@
 /*
- * Copyright (c) 2016-present, Facebook, Inc.
+ * Copyright (c) 2016-2020, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
  * LICENSE file in the root directory of this source tree) and the GPLv2 (found
  * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
  */
 
 #include <stddef.h>   /* size_t */
diff --git a/doc/zstd_compression_format.md b/doc/zstd_compression_format.md
index 90ac0fe9bcd5..fc61726fc98c 100644
--- a/doc/zstd_compression_format.md
+++ b/doc/zstd_compression_format.md
@@ -16,7 +16,7 @@ Distribution of this document is unlimited.
 
 ### Version
 
-0.3.4 (16/08/19)
+0.3.5 (13/11/19)
 
 
 Introduction
@@ -341,6 +341,8 @@ The structure of a block is as follows:
 |:--------------:|:---------------:|
 |    3 bytes     |     n bytes     |
 
+__`Block_Header`__
+
 `Block_Header` uses 3 bytes, written using __little-endian__ convention.
 It contains 3 fields :
 
@@ -385,17 +387,30 @@ There are 4 block types :
 __`Block_Size`__
 
 The upper 21 bits of `Block_Header` represent the `Block_Size`.
+
 When `Block_Type` is `Compressed_Block` or `Raw_Block`,
-`Block_Size` is the size of `Block_Content`, hence excluding `Block_Header`.  
-When `Block_Type` is `RLE_Block`, `Block_Content`’s size is always 1,
-and `Block_Size` represents the number of times this byte must be repeated.
-A block can contain and decompress into any number of bytes (even zero),
-up to `Block_Maximum_Decompressed_Size`, which is the smallest of:
--  Window_Size
+`Block_Size` is the size of `Block_Content` (hence excluding `Block_Header`).  
+
+When `Block_Type` is `RLE_Block`, since `Block_Content`’s size is always 1,
+`Block_Size` represents the number of times this byte must be repeated.
+
+`Block_Size` is limited by `Block_Maximum_Size` (see below).
+
+__`Block_Content`__ and __`Block_Maximum_Size`__
+
+The size of `Block_Content` is limited by `Block_Maximum_Size`,
+which is the smallest of:
+-  `Window_Size`
 -  128 KB
 
-If this condition cannot be respected when generating a `Compressed_Block`,
-the block must be sent uncompressed instead (`Raw_Block`).
+`Block_Maximum_Size` is constant for a given frame.
+This maximum is applicable to both the decompressed size
+and the compressed size of any block in the frame.
+
+The reasoning for this limit is that a decoder can read this information
+at the beginning of a frame and use it to allocate buffers.
+The guarantees on the size of blocks ensure that
+the buffers will be large enough for any following block of the valid frame.
 
 
 Compressed Blocks
@@ -1658,6 +1673,7 @@ or at least provide a meaningful error code explaining for which reason it canno
 
 Version changes
 ---------------
+- 0.3.5 : clarifications for Block_Maximum_Size
 - 0.3.4 : clarifications for FSE decoding table
 - 0.3.3 : clarifications for field Block_Size
 - 0.3.2 : remove additional block size restriction on compressed blocks
diff --git a/doc/zstd_manual.html b/doc/zstd_manual.html
index 43c5555b8ca8..fe58f78cb153 100644
--- a/doc/zstd_manual.html
+++ b/doc/zstd_manual.html
@@ -1,10 +1,10 @@
 <html>
 <head>
 <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
-<title>zstd 1.4.4 Manual</title>
+<title>zstd 1.4.5 Manual</title>
 </head>
 <body>
-<h1>zstd 1.4.4 Manual</h1>
+<h1>zstd 1.4.5 Manual</h1>
 <hr>
 <a name="Contents"></a><h2>Contents</h2>
 <ol>
@@ -217,7 +217,10 @@ size_t     ZSTD_freeDCtx(ZSTD_DCtx* dctx);
                               * Default level is ZSTD_CLEVEL_DEFAULT==3.
                               * Special: value 0 means default, which is controlled by ZSTD_CLEVEL_DEFAULT.
                               * Note 1 : it's possible to pass a negative compression level.
-                              * Note 2 : setting a level resets all other compression parameters to default */
+                              * Note 2 : setting a level does not automatically set all other compression parameters
+                              *   to default. Setting this will however eventually dynamically impact the compression
+                              *   parameters which have not been manually set. The manually set
+                              *   ones will 'stick'. */
     </b>/* Advanced compression parameters :<b>
      * It's possible to pin down compression parameters to some specific values.
      * In which case, these values are no longer dynamically selected by the compressor */
@@ -451,11 +454,13 @@ size_t     ZSTD_freeDCtx(ZSTD_DCtx* dctx);
     </b>/* note : additional experimental parameters are also available<b>
      * within the experimental section of the API.
      * At the time of this writing, they include :
-     * ZSTD_c_format
+     * ZSTD_d_format
+     * ZSTD_d_stableOutBuffer
      * Because they are not stable, it's necessary to define ZSTD_STATIC_LINKING_ONLY to access them.
      * note : never ever use experimentalParam? names directly
      */
-     ZSTD_d_experimentalParam1=1000
+     ZSTD_d_experimentalParam1=1000,
+     ZSTD_d_experimentalParam2=1001
 
 } ZSTD_dParameter;
 </b></pre><BR>
@@ -1055,23 +1060,28 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
 size_t ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams);
 size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params);
 size_t ZSTD_estimateDCtxSize(void);
-</b><p>  These functions make it possible to estimate memory usage of a future
-  {D,C}Ctx, before its creation.
-
-  ZSTD_estimateCCtxSize() will provide a budget large enough for any
-  compression level up to selected one. Unlike ZSTD_estimateCStreamSize*(),
-  this estimate does not include space for a window buffer, so this estimate
-  is guaranteed to be enough for single-shot compressions, but not streaming
-  compressions. It will however assume the input may be arbitrarily large,
-  which is the worst case. If srcSize is known to always be small,
-  ZSTD_estimateCCtxSize_usingCParams() can provide a tighter estimation.
-  ZSTD_estimateCCtxSize_usingCParams() can be used in tandem with
-  ZSTD_getCParams() to create cParams from compressionLevel.
-  ZSTD_estimateCCtxSize_usingCCtxParams() can be used in tandem with
-  ZSTD_CCtxParams_setParameter().
-
-  Note: only single-threaded compression is supported. This function will
-  return an error code if ZSTD_c_nbWorkers is >= 1. 
+</b><p>  These functions make it possible to estimate memory usage
+  of a future {D,C}Ctx, before its creation.
+
+  ZSTD_estimateCCtxSize() will provide a memory budget large enough
+  for any compression level up to selected one.
+  Note : Unlike ZSTD_estimateCStreamSize*(), this estimate
+         does not include space for a window buffer.
+         Therefore, the estimation is only guaranteed for single-shot compressions, not streaming.
+  The estimate will assume the input may be arbitrarily large,
+  which is the worst case.
+
+  When srcSize can be bound by a known and rather "small" value,
+  this fact can be used to provide a tighter estimation
+  because the CCtx compression context will need less memory.
+  This tighter estimation can be provided by more advanced functions
+  ZSTD_estimateCCtxSize_usingCParams(), which can be used in tandem with ZSTD_getCParams(),
+  and ZSTD_estimateCCtxSize_usingCCtxParams(), which can be used in tandem with ZSTD_CCtxParams_setParameter().
+  Both can be used to estimate memory using custom compression parameters and arbitrary srcSize limits.
+
+  Note 2 : only single-threaded compression is supported.
+  ZSTD_estimateCCtxSize_usingCCtxParams() will return an error code if ZSTD_c_nbWorkers is >= 1.
+ 
 </p></pre><BR>
 
 <pre><b>size_t ZSTD_estimateCStreamSize(int compressionLevel);
diff --git a/examples/Makefile b/examples/Makefile
index 65ea8abad5d6..1ae6bce83d0c 100644
--- a/examples/Makefile
+++ b/examples/Makefile
@@ -1,10 +1,11 @@
 # ################################################################
-# Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+# Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
 # All rights reserved.
 #
 # This source code is licensed under both the BSD-style license (found in the
 # LICENSE file in the root directory of this source tree) and the GPLv2 (found
 # in the COPYING file in the root directory of this source tree).
+# You may select, at your option, one of the above-listed licenses.
 # ################################################################
 
 # This Makefile presumes libzstd is installed, using `sudo make install`
diff --git a/examples/common.h b/examples/common.h
index a714cbb72c36..4492c7e4efa7 100644
--- a/examples/common.h
+++ b/examples/common.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/examples/dictionary_compression.c b/examples/dictionary_compression.c
index 9efdb785c112..d9aad45a7b07 100644
--- a/examples/dictionary_compression.c
+++ b/examples/dictionary_compression.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020 Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/examples/dictionary_decompression.c b/examples/dictionary_decompression.c
index f683bbb43800..7e50986e37aa 100644
--- a/examples/dictionary_decompression.c
+++ b/examples/dictionary_decompression.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/examples/multiple_simple_compression.c b/examples/multiple_simple_compression.c
index a44ac8b442f9..e409467b226b 100644
--- a/examples/multiple_simple_compression.c
+++ b/examples/multiple_simple_compression.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/examples/multiple_streaming_compression.c b/examples/multiple_streaming_compression.c
index ad98b1bd1b09..8a4dc96c1121 100644
--- a/examples/multiple_streaming_compression.c
+++ b/examples/multiple_streaming_compression.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/examples/simple_compression.c b/examples/simple_compression.c
index 019a143d4c85..618080b338f7 100644
--- a/examples/simple_compression.c
+++ b/examples/simple_compression.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/examples/simple_decompression.c b/examples/simple_decompression.c
index 1aa57c7b0934..e108987c625d 100644
--- a/examples/simple_decompression.c
+++ b/examples/simple_decompression.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/examples/streaming_compression.c b/examples/streaming_compression.c
index d0b04895f018..f0f1065b1d27 100644
--- a/examples/streaming_compression.c
+++ b/examples/streaming_compression.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/examples/streaming_decompression.c b/examples/streaming_decompression.c
index d26b45b34c74..26eda3441b7f 100644
--- a/examples/streaming_decompression.c
+++ b/examples/streaming_decompression.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/examples/streaming_memory_usage.c b/examples/streaming_memory_usage.c
index 26835788abed..37dd660e4a64 100644
--- a/examples/streaming_memory_usage.c
+++ b/examples/streaming_memory_usage.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2017-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2017-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/lib/Makefile b/lib/Makefile
index 273ceb90490d..7c6dff02468a 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -1,12 +1,24 @@
 # ################################################################
-# Copyright (c) 2015-present, Yann Collet, Facebook, Inc.
+# Copyright (c) 2015-2020, Yann Collet, Facebook, Inc.
 # All rights reserved.
 #
 # This source code is licensed under both the BSD-style license (found in the
 # LICENSE file in the root directory of this source tree) and the GPLv2 (found
 # in the COPYING file in the root directory of this source tree).
+# You may select, at your option, one of the above-listed licenses.
 # ################################################################
 
+Q = $(if $(filter 1,$(V) $(VERBOSE)),,@)
+
+# When cross-compiling from linux to windows, you might
+# need to specify this as "Windows." Fedora build fails
+# without it.
+#
+# Note: mingw-w64 build from linux to windows does not
+# fail on other tested distros (ubuntu, debian) even
+# without manually specifying the TARGET_SYSTEM.
+TARGET_SYSTEM ?= $(OS)
+
 # Version numbers
 LIBVER_MAJOR_SCRIPT:=`sed -n '/define ZSTD_VERSION_MAJOR/s/.*[[:blank:]]\([0-9][0-9]*\).*/\1/p' < ./zstd.h`
 LIBVER_MINOR_SCRIPT:=`sed -n '/define ZSTD_VERSION_MINOR/s/.*[[:blank:]]\([0-9][0-9]*\).*/\1/p' < ./zstd.h`
@@ -19,11 +31,10 @@ LIBVER := $(shell echo $(LIBVER_SCRIPT))
 VERSION?= $(LIBVER)
 CCVER := $(shell $(CC) --version)
 
-CPPFLAGS+= -I. -I./common -DXXH_NAMESPACE=ZSTD_
-ifeq ($(OS),Windows_NT)   # MinGW assumed
+CPPFLAGS+= -DXXH_NAMESPACE=ZSTD_
+ifeq ($(TARGET_SYSTEM),Windows_NT)   # MinGW assumed
 CPPFLAGS   += -D__USE_MINGW_ANSI_STDIO   # compatibility with %zu formatting
 endif
-CFLAGS  ?= -O3
 DEBUGFLAGS= -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow \
             -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement \
             -Wstrict-prototypes -Wundef -Wpointer-arith \
@@ -50,18 +61,46 @@ ifeq ($(findstring GCC,$(CCVER)),GCC)
 decompress/zstd_decompress_block.o :	CFLAGS+=-fno-tree-vectorize
 endif
 
-ZSTD_LEGACY_SUPPORT ?= 5
+# This is a helper variable that configures a bunch of other variables to new,
+# space-optimized defaults.
+ZSTD_LIB_MINIFY ?= 0
+ifneq ($(ZSTD_LIB_MINIFY), 0)
+	HAVE_CC_OZ ?= $(shell echo "" | $(CC) -Oz -x c -c - -o /dev/null 2> /dev/null && echo 1 || echo 0)
+	ZSTD_LEGACY_SUPPORT ?= 0
+	ZSTD_LIB_DEPRECATED ?= 0
+	HUF_FORCE_DECOMPRESS_X1 ?= 1
+	ZSTD_FORCE_DECOMPRESS_SHORT ?= 1
+	ZSTD_NO_INLINE ?= 1
+	ZSTD_STRIP_ERROR_STRINGS ?= 1
+	ifneq ($(HAVE_CC_OZ), 0)
+		# Some compilers (clang) support an even more space-optimized setting.
+		CFLAGS += -Oz
+	else
+		CFLAGS += -Os
+	endif
+	CFLAGS += -fno-stack-protector -fomit-frame-pointer -fno-ident \
+	          -DDYNAMIC_BMI2=0 -DNDEBUG
+else
+	CFLAGS += -O3
+endif
+
+# Modules
 ZSTD_LIB_COMPRESSION ?= 1
 ZSTD_LIB_DECOMPRESSION ?= 1
 ZSTD_LIB_DICTBUILDER ?= 1
 ZSTD_LIB_DEPRECATED ?= 1
+
+# Legacy support
+ZSTD_LEGACY_SUPPORT ?= 5
+ZSTD_LEGACY_MULTITHREADED_API ?= 0
+
+# Build size optimizations
 HUF_FORCE_DECOMPRESS_X1 ?= 0
 HUF_FORCE_DECOMPRESS_X2 ?= 0
 ZSTD_FORCE_DECOMPRESS_SHORT ?= 0
 ZSTD_FORCE_DECOMPRESS_LONG ?= 0
 ZSTD_NO_INLINE ?= 0
 ZSTD_STRIP_ERROR_STRINGS ?= 0
-ZSTD_LEGACY_MULTITHREADED_API ?= 0
 
 ifeq ($(ZSTD_LIB_COMPRESSION), 0)
 	ZSTD_LIB_DICTBUILDER = 0
@@ -121,7 +160,6 @@ ifneq ($(ZSTD_LEGACY_SUPPORT), 0)
 ifeq ($(shell test $(ZSTD_LEGACY_SUPPORT) -lt 8; echo $$?), 0)
 	ZSTD_FILES += $(shell ls legacy/*.c | $(GREP) 'v0[$(ZSTD_LEGACY_SUPPORT)-7]')
 endif
-	CPPFLAGS += -I./legacy
 endif
 CPPFLAGS  += -DZSTD_LEGACY_SUPPORT=$(ZSTD_LEGACY_SUPPORT)
 
@@ -142,26 +180,26 @@ else
 endif
 
 
-.PHONY: default all clean install uninstall
+.PHONY: default lib-all all clean install uninstall
 
 default: lib-release
 
+# alias
+lib-all: all
+
 all: lib
 
 libzstd.a: ARFLAGS = rcs
 libzstd.a: $(ZSTD_OBJ)
 	@echo compiling static library
-	@$(AR) $(ARFLAGS) $@ $^
-
-libzstd.a-mt: CPPFLAGS += -DZSTD_MULTITHREAD
-libzstd.a-mt: libzstd.a
+	$(Q)$(AR) $(ARFLAGS) $@ $^
 
-ifneq (,$(filter Windows%,$(OS)))
+ifneq (,$(filter Windows%,$(TARGET_SYSTEM)))
 
 LIBZSTD = dll\libzstd.dll
 $(LIBZSTD): $(ZSTD_FILES)
 	@echo compiling dynamic library $(LIBVER)
-	$(CC) $(FLAGS) -DZSTD_DLL_EXPORT=1 -Wl,--out-implib,dll\libzstd.lib -shared $^ -o $@
+	$(CC) $(FLAGS) -DZSTD_DLL_EXPORT=1 -Wl,--out-implib,dll\libzstd.dll.a -shared $^ -o $@
 
 else
 
@@ -169,27 +207,30 @@ LIBZSTD = libzstd.$(SHARED_EXT_VER)
 $(LIBZSTD): LDFLAGS += -shared -fPIC -fvisibility=hidden
 $(LIBZSTD): $(ZSTD_FILES)
 	@echo compiling dynamic library $(LIBVER)
-	@$(CC) $(FLAGS) $^ $(LDFLAGS) $(SONAME_FLAGS) -o $@
+	$(Q)$(CC) $(FLAGS) $^ $(LDFLAGS) $(SONAME_FLAGS) -o $@
 	@echo creating versioned links
-	@ln -sf $@ libzstd.$(SHARED_EXT_MAJOR)
-	@ln -sf $@ libzstd.$(SHARED_EXT)
+	$(Q)ln -sf $@ libzstd.$(SHARED_EXT_MAJOR)
+	$(Q)ln -sf $@ libzstd.$(SHARED_EXT)
 
 endif
 
-
+.PHONY: libzstd
 libzstd : $(LIBZSTD)
 
-libzstd-mt : CPPFLAGS += -DZSTD_MULTITHREAD
-libzstd-mt : libzstd
+.PHONY: lib
+lib : libzstd.a libzstd
 
-lib: libzstd.a libzstd
+.PHONY: lib-mt
+%-mt : CPPFLAGS += -DZSTD_MULTITHREAD
+%-mt : LDFLAGS  += -pthread
+%-mt : %
+	@echo multi-threading build completed
 
-lib-mt: CPPFLAGS += -DZSTD_MULTITHREAD
-lib-mt: lib
+.PHONY: lib-release
+%-release : DEBUGFLAGS :=
+%-release : %
+	@echo release build completed
 
-lib-release lib-release-mt: DEBUGFLAGS :=
-lib-release: lib
-lib-release-mt: lib-mt
 
 # Special case : building library in single-thread mode _and_ without zstdmt_compress.c
 ZSTDMT_FILES = compress/zstdmt_compress.c
@@ -198,20 +239,22 @@ libzstd-nomt: LDFLAGS += -shared -fPIC -fvisibility=hidden
 libzstd-nomt: $(ZSTD_NOMT_FILES)
 	@echo compiling single-thread dynamic library $(LIBVER)
 	@echo files : $(ZSTD_NOMT_FILES)
-	@$(CC) $(FLAGS) $^ $(LDFLAGS) $(SONAME_FLAGS) -o $@
+	$(Q)$(CC) $(FLAGS) $^ $(LDFLAGS) $(SONAME_FLAGS) -o $@
 
 clean:
-	@$(RM) -r *.dSYM   # macOS-specific
-	@$(RM) core *.o *.a *.gcda *.$(SHARED_EXT) *.$(SHARED_EXT).* libzstd.pc
-	@$(RM) dll/libzstd.dll dll/libzstd.lib libzstd-nomt*
-	@$(RM) common/*.o compress/*.o decompress/*.o dictBuilder/*.o legacy/*.o deprecated/*.o
+	$(Q)$(RM) -r *.dSYM   # macOS-specific
+	$(Q)$(RM) core *.o *.a *.gcda *.$(SHARED_EXT) *.$(SHARED_EXT).* libzstd.pc
+	$(Q)$(RM) dll/libzstd.dll dll/libzstd.lib libzstd-nomt*
+	$(Q)$(RM) common/*.o compress/*.o decompress/*.o dictBuilder/*.o legacy/*.o deprecated/*.o
 	@echo Cleaning library completed
 
 #-----------------------------------------------------------------------------
-# make install is validated only for Linux, macOS, BSD, Hurd and Solaris targets
+# make install is validated only for below listed environments
 #-----------------------------------------------------------------------------
 ifneq (,$(filter $(shell uname),Linux Darwin GNU/kFreeBSD GNU OpenBSD FreeBSD NetBSD DragonFly SunOS Haiku))
 
+all: libzstd.pc
+
 DESTDIR     ?=
 # directory variables : GNU conventions prefer lowercase
 # see https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html
@@ -219,11 +262,31 @@ DESTDIR     ?=
 prefix      ?= /usr/local
 PREFIX      ?= $(prefix)
 exec_prefix ?= $(PREFIX)
-libdir      ?= $(exec_prefix)/lib
+EXEC_PREFIX ?= $(exec_prefix)
+libdir      ?= $(EXEC_PREFIX)/lib
 LIBDIR      ?= $(libdir)
 includedir  ?= $(PREFIX)/include
 INCLUDEDIR  ?= $(includedir)
 
+PCLIBDIR ?= $(shell echo "$(LIBDIR)" | sed -n -E -e "s@^$(EXEC_PREFIX)(/|$$)@@p")
+PCINCDIR ?= $(shell echo "$(INCLUDEDIR)" | sed -n -E -e "s@^$(PREFIX)(/|$$)@@p")
+
+ifeq (,$(PCLIBDIR))
+# Additional prefix check is required, since the empty string is technically a
+# valid PCLIBDIR
+ifeq (,$(shell echo "$(LIBDIR)" | sed -n -E -e "\\@^$(EXEC_PREFIX)(/|$$)@ p"))
+$(error configured libdir ($(LIBDIR)) is outside of prefix ($(PREFIX)), can't generate pkg-config file)
+endif
+endif
+
+ifeq (,$(PCINCDIR))
+# Additional prefix check is required, since the empty string is technically a
+# valid PCINCDIR
+ifeq (,$(shell echo "$(INCLUDEDIR)" | sed -n -E -e "\\@^$(PREFIX)(/|$$)@ p"))
+$(error configured includedir ($(INCLUDEDIR)) is outside of exec_prefix ($(EXEC_PREFIX)), can't generate pkg-config file)
+endif
+endif
+
 ifneq (,$(filter $(shell uname),FreeBSD NetBSD DragonFly))
 PKGCONFIGDIR ?= $(PREFIX)/libdata/pkgconfig
 else
@@ -243,47 +306,49 @@ INSTALL_DATA    ?= $(INSTALL) -m 644
 libzstd.pc:
 libzstd.pc: libzstd.pc.in
 	@echo creating pkgconfig
-	@sed -e 's|@PREFIX@|$(PREFIX)|' \
-             -e 's|@VERSION@|$(VERSION)|' \
-             $< >$@
+	$(Q)@sed -E -e 's|@PREFIX@|$(PREFIX)|' \
+          -e 's|@LIBDIR@|$(PCLIBDIR)|' \
+          -e 's|@INCLUDEDIR@|$(PCINCDIR)|' \
+          -e 's|@VERSION@|$(VERSION)|' \
+          $< >$@
 
 install: install-pc install-static install-shared install-includes
 	@echo zstd static and shared library installed
 
 install-pc: libzstd.pc
-	@$(INSTALL) -d -m 755 $(DESTDIR)$(PKGCONFIGDIR)/
-	@$(INSTALL_DATA) libzstd.pc $(DESTDIR)$(PKGCONFIGDIR)/
+	$(Q)$(INSTALL) -d -m 755 $(DESTDIR)$(PKGCONFIGDIR)/
+	$(Q)$(INSTALL_DATA) libzstd.pc $(DESTDIR)$(PKGCONFIGDIR)/
 
 install-static: libzstd.a
 	@echo Installing static library
-	@$(INSTALL) -d -m 755 $(DESTDIR)$(LIBDIR)/
-	@$(INSTALL_DATA) libzstd.a $(DESTDIR)$(LIBDIR)
+	$(Q)$(INSTALL) -d -m 755 $(DESTDIR)$(LIBDIR)/
+	$(Q)$(INSTALL_DATA) libzstd.a $(DESTDIR)$(LIBDIR)
 
 install-shared: libzstd
 	@echo Installing shared library
-	@$(INSTALL) -d -m 755 $(DESTDIR)$(LIBDIR)/
-	@$(INSTALL_PROGRAM) $(LIBZSTD) $(DESTDIR)$(LIBDIR)
-	@ln -sf $(LIBZSTD) $(DESTDIR)$(LIBDIR)/libzstd.$(SHARED_EXT_MAJOR)
-	@ln -sf $(LIBZSTD) $(DESTDIR)$(LIBDIR)/libzstd.$(SHARED_EXT)
+	$(Q)$(INSTALL) -d -m 755 $(DESTDIR)$(LIBDIR)/
+	$(Q)$(INSTALL_PROGRAM) $(LIBZSTD) $(DESTDIR)$(LIBDIR)
+	$(Q)ln -sf $(LIBZSTD) $(DESTDIR)$(LIBDIR)/libzstd.$(SHARED_EXT_MAJOR)
+	$(Q)ln -sf $(LIBZSTD) $(DESTDIR)$(LIBDIR)/libzstd.$(SHARED_EXT)
 
 install-includes:
 	@echo Installing includes
-	@$(INSTALL) -d -m 755 $(DESTDIR)$(INCLUDEDIR)/
-	@$(INSTALL_DATA) zstd.h $(DESTDIR)$(INCLUDEDIR)
-	@$(INSTALL_DATA) common/zstd_errors.h $(DESTDIR)$(INCLUDEDIR)
-	@$(INSTALL_DATA) deprecated/zbuff.h $(DESTDIR)$(INCLUDEDIR)     # prototypes generate deprecation warnings
-	@$(INSTALL_DATA) dictBuilder/zdict.h $(DESTDIR)$(INCLUDEDIR)
+	$(Q)$(INSTALL) -d -m 755 $(DESTDIR)$(INCLUDEDIR)/
+	$(Q)$(INSTALL_DATA) zstd.h $(DESTDIR)$(INCLUDEDIR)
+	$(Q)$(INSTALL_DATA) common/zstd_errors.h $(DESTDIR)$(INCLUDEDIR)
+	$(Q)$(INSTALL_DATA) deprecated/zbuff.h $(DESTDIR)$(INCLUDEDIR)     # prototypes generate deprecation warnings
+	$(Q)$(INSTALL_DATA) dictBuilder/zdict.h $(DESTDIR)$(INCLUDEDIR)
 
 uninstall:
-	@$(RM) $(DESTDIR)$(LIBDIR)/libzstd.a
-	@$(RM) $(DESTDIR)$(LIBDIR)/libzstd.$(SHARED_EXT)
-	@$(RM) $(DESTDIR)$(LIBDIR)/libzstd.$(SHARED_EXT_MAJOR)
-	@$(RM) $(DESTDIR)$(LIBDIR)/$(LIBZSTD)
-	@$(RM) $(DESTDIR)$(PKGCONFIGDIR)/libzstd.pc
-	@$(RM) $(DESTDIR)$(INCLUDEDIR)/zstd.h
-	@$(RM) $(DESTDIR)$(INCLUDEDIR)/zstd_errors.h
-	@$(RM) $(DESTDIR)$(INCLUDEDIR)/zbuff.h   # Deprecated streaming functions
-	@$(RM) $(DESTDIR)$(INCLUDEDIR)/zdict.h
+	$(Q)$(RM) $(DESTDIR)$(LIBDIR)/libzstd.a
+	$(Q)$(RM) $(DESTDIR)$(LIBDIR)/libzstd.$(SHARED_EXT)
+	$(Q)$(RM) $(DESTDIR)$(LIBDIR)/libzstd.$(SHARED_EXT_MAJOR)
+	$(Q)$(RM) $(DESTDIR)$(LIBDIR)/$(LIBZSTD)
+	$(Q)$(RM) $(DESTDIR)$(PKGCONFIGDIR)/libzstd.pc
+	$(Q)$(RM) $(DESTDIR)$(INCLUDEDIR)/zstd.h
+	$(Q)$(RM) $(DESTDIR)$(INCLUDEDIR)/zstd_errors.h
+	$(Q)$(RM) $(DESTDIR)$(INCLUDEDIR)/zbuff.h   # Deprecated streaming functions
+	$(Q)$(RM) $(DESTDIR)$(INCLUDEDIR)/zdict.h
 	@echo zstd libraries successfully uninstalled
 
 endif
diff --git a/lib/README.md b/lib/README.md
index 0062c0d63e04..6ccffb13868c 100644
--- a/lib/README.md
+++ b/lib/README.md
@@ -85,28 +85,48 @@ The file structure is designed to make this selection manually achievable for an
 
 - While invoking `make libzstd`, it's possible to define build macros
         `ZSTD_LIB_COMPRESSION, ZSTD_LIB_DECOMPRESSION`, `ZSTD_LIB_DICTBUILDER`,
-        and `ZSTD_LIB_DEPRECATED` as `0` to forgo compilation of the corresponding features.
-        This will also disable compilation of all dependencies
-        (eg. `ZSTD_LIB_COMPRESSION=0` will also disable dictBuilder).
-
-- There are some additional build macros that can be used to minify the decoder.
-
-  Zstandard often has more than one implementation of a piece of functionality,
-  where each implementation optimizes for different scenarios. For example, the
-  Huffman decoder has complementary implementations that decode the stream one
-  symbol at a time or two symbols at a time. Zstd normally includes both (and
-  dispatches between them at runtime), but by defining `HUF_FORCE_DECOMPRESS_X1`
-  or `HUF_FORCE_DECOMPRESS_X2`, you can force the use of one or the other, avoiding
+        and `ZSTD_LIB_DEPRECATED` as `0` to forgo compilation of the
+        corresponding features. This will also disable compilation of all
+        dependencies (eg. `ZSTD_LIB_COMPRESSION=0` will also disable
+        dictBuilder).
+
+- There are a number of options that can help minimize the binary size of
+  `libzstd`.
+
+  The first step is to select the components needed (using the above-described
+  `ZSTD_LIB_COMPRESSION` etc.).
+
+  The next step is to set `ZSTD_LIB_MINIFY` to `1` when invoking `make`. This
+  disables various optional components and changes the compilation flags to
+  prioritize space-saving.
+
+  Detailed options: Zstandard's code and build environment is set up by default
+  to optimize above all else for performance. In pursuit of this goal, Zstandard
+  makes significant trade-offs in code size. For example, Zstandard often has
+  more than one implementation of a particular component, with each
+  implementation optimized for different scenarios. For example, the Huffman
+  decoder has complementary implementations that decode the stream one symbol at
+  a time or two symbols at a time. Zstd normally includes both (and dispatches
+  between them at runtime), but by defining `HUF_FORCE_DECOMPRESS_X1` or
+  `HUF_FORCE_DECOMPRESS_X2`, you can force the use of one or the other, avoiding
   compilation of the other. Similarly, `ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT`
   and `ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG` force the compilation and use of
   only one or the other of two decompression implementations. The smallest
   binary is achieved by using `HUF_FORCE_DECOMPRESS_X1` and
-  `ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT`.
+  `ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT` (implied by `ZSTD_LIB_MINIFY`).
 
   For squeezing the last ounce of size out, you can also define
   `ZSTD_NO_INLINE`, which disables inlining, and `ZSTD_STRIP_ERROR_STRINGS`,
   which removes the error messages that are otherwise returned by
-  `ZSTD_getErrorName`.
+  `ZSTD_getErrorName` (implied by `ZSTD_LIB_MINIFY`).
+
+  Finally, when integrating into your application, make sure you're doing link-
+  time optimation and unused symbol garbage collection (via some combination of,
+  e.g., `-flto`, `-ffat-lto-objects`, `-fuse-linker-plugin`,
+  `-ffunction-sections`, `-fdata-sections`, `-fmerge-all-constants`,
+  `-Wl,--gc-sections`, `-Wl,-z,norelro`, and an archiver that understands
+  the compiler's intermediate representation, e.g., `AR=gcc-ar`). Consult your
+  compiler's documentation.
 
 - While invoking `make libzstd`, the build macro `ZSTD_LEGACY_MULTITHREADED_API=1`
   will expose the deprecated `ZSTDMT` API exposed by `zstdmt_compress.h` in
diff --git a/lib/common/bitstream.h b/lib/common/bitstream.h
index 1c294b80d13f..37b99c01eed3 100644
--- a/lib/common/bitstream.h
+++ b/lib/common/bitstream.h
@@ -1,35 +1,15 @@
 /* ******************************************************************
-   bitstream
-   Part of FSE library
-   Copyright (C) 2013-present, Yann Collet.
-
-   BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
-
-   Redistribution and use in source and binary forms, with or without
-   modification, are permitted provided that the following conditions are
-   met:
-
-       * Redistributions of source code must retain the above copyright
-   notice, this list of conditions and the following disclaimer.
-       * Redistributions in binary form must reproduce the above
-   copyright notice, this list of conditions and the following disclaimer
-   in the documentation and/or other materials provided with the
-   distribution.
-
-   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-   You can contact the author at :
-   - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ * bitstream
+ * Part of FSE library
+ * Copyright (c) 2013-2020, Yann Collet, Facebook, Inc.
+ *
+ * You can contact the author at :
+ * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
 ****************************************************************** */
 #ifndef BITSTREAM_H_MODULE
 #define BITSTREAM_H_MODULE
@@ -48,6 +28,7 @@ extern "C" {
 *  Dependencies
 ******************************************/
 #include "mem.h"            /* unaligned access routines */
+#include "compiler.h"       /* UNLIKELY() */
 #include "debug.h"          /* assert(), DEBUGLOG(), RAWLOG() */
 #include "error_private.h"  /* error codes and messages */
 
@@ -161,8 +142,7 @@ MEM_STATIC unsigned BIT_highbit32 (U32 val)
     {
 #   if defined(_MSC_VER)   /* Visual */
         unsigned long r=0;
-        _BitScanReverse ( &r, val );
-        return (unsigned) r;
+        return _BitScanReverse ( &r, val ) ? (unsigned)r : 0;
 #   elif defined(__GNUC__) && (__GNUC__ >= 3)   /* Use GCC Intrinsic */
         return __builtin_clz (val) ^ 31;
 #   elif defined(__ICCARM__)    /* IAR Intrinsic */
@@ -411,6 +391,23 @@ MEM_STATIC size_t BIT_readBitsFast(BIT_DStream_t* bitD, unsigned nbBits)
     return value;
 }
 
+/*! BIT_reloadDStreamFast() :
+ *  Similar to BIT_reloadDStream(), but with two differences:
+ *  1. bitsConsumed <= sizeof(bitD->bitContainer)*8 must hold!
+ *  2. Returns BIT_DStream_overflow when bitD->ptr < bitD->limitPtr, at this
+ *     point you must use BIT_reloadDStream() to reload.
+ */
+MEM_STATIC BIT_DStream_status BIT_reloadDStreamFast(BIT_DStream_t* bitD)
+{
+    if (UNLIKELY(bitD->ptr < bitD->limitPtr))
+        return BIT_DStream_overflow;
+    assert(bitD->bitsConsumed <= sizeof(bitD->bitContainer)*8);
+    bitD->ptr -= bitD->bitsConsumed >> 3;
+    bitD->bitsConsumed &= 7;
+    bitD->bitContainer = MEM_readLEST(bitD->ptr);
+    return BIT_DStream_unfinished;
+}
+
 /*! BIT_reloadDStream() :
  *  Refill `bitD` from buffer previously set in BIT_initDStream() .
  *  This function is safe, it guarantees it will not read beyond src buffer.
@@ -422,10 +419,7 @@ MEM_STATIC BIT_DStream_status BIT_reloadDStream(BIT_DStream_t* bitD)
         return BIT_DStream_overflow;
 
     if (bitD->ptr >= bitD->limitPtr) {
-        bitD->ptr -= bitD->bitsConsumed >> 3;
-        bitD->bitsConsumed &= 7;
-        bitD->bitContainer = MEM_readLEST(bitD->ptr);
-        return BIT_DStream_unfinished;
+        return BIT_reloadDStreamFast(bitD);
     }
     if (bitD->ptr == bitD->start) {
         if (bitD->bitsConsumed < sizeof(bitD->bitContainer)*8) return BIT_DStream_endOfBuffer;
diff --git a/lib/common/compiler.h b/lib/common/compiler.h
index 1877a0c1d9be..95e9483521d4 100644
--- a/lib/common/compiler.h
+++ b/lib/common/compiler.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -17,7 +17,7 @@
 /* force inlining */
 
 #if !defined(ZSTD_NO_INLINE)
-#if defined (__GNUC__) || defined(__cplusplus) || defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L   /* C99 */
+#if (defined(__GNUC__) && !defined(__STRICT_ANSI__)) || defined(__cplusplus) || defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L   /* C99 */
 #  define INLINE_KEYWORD inline
 #else
 #  define INLINE_KEYWORD
@@ -114,6 +114,9 @@
 #    include <mmintrin.h>   /* https://msdn.microsoft.com/fr-fr/library/84szxsww(v=vs.90).aspx */
 #    define PREFETCH_L1(ptr)  _mm_prefetch((const char*)(ptr), _MM_HINT_T0)
 #    define PREFETCH_L2(ptr)  _mm_prefetch((const char*)(ptr), _MM_HINT_T1)
+#    elif defined(__aarch64__)
+#     define PREFETCH_L1(ptr)  __asm__ __volatile__("prfm pldl1keep, %0" ::"Q"(*(ptr)))
+#     define PREFETCH_L2(ptr)  __asm__ __volatile__("prfm pldl2keep, %0" ::"Q"(*(ptr)))
 #  elif defined(__GNUC__) && ( (__GNUC__ >= 4) || ( (__GNUC__ == 3) && (__GNUC_MINOR__ >= 1) ) )
 #    define PREFETCH_L1(ptr)  __builtin_prefetch((ptr), 0 /* rw==read */, 3 /* locality */)
 #    define PREFETCH_L2(ptr)  __builtin_prefetch((ptr), 0 /* rw==read */, 2 /* locality */)
@@ -136,7 +139,7 @@
 
 /* vectorization
  * older GCC (pre gcc-4.3 picked as the cutoff) uses a different syntax */
-#if !defined(__clang__) && defined(__GNUC__)
+#if !defined(__INTEL_COMPILER) && !defined(__clang__) && defined(__GNUC__)
 #  if (__GNUC__ == 4 && __GNUC_MINOR__ > 3) || (__GNUC__ >= 5)
 #    define DONT_VECTORIZE __attribute__((optimize("no-tree-vectorize")))
 #  else
@@ -146,6 +149,19 @@
 #  define DONT_VECTORIZE
 #endif
 
+/* Tell the compiler that a branch is likely or unlikely.
+ * Only use these macros if it causes the compiler to generate better code.
+ * If you can remove a LIKELY/UNLIKELY annotation without speed changes in gcc
+ * and clang, please do.
+ */
+#if defined(__GNUC__)
+#define LIKELY(x) (__builtin_expect((x), 1))
+#define UNLIKELY(x) (__builtin_expect((x), 0))
+#else
+#define LIKELY(x) (x)
+#define UNLIKELY(x) (x)
+#endif
+
 /* disable warnings */
 #ifdef _MSC_VER    /* Visual Studio */
 #  include <intrin.h>                    /* For Visual 2005 */
diff --git a/lib/common/cpu.h b/lib/common/cpu.h
index 5f0923fc9289..6e8a974f62d7 100644
--- a/lib/common/cpu.h
+++ b/lib/common/cpu.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2018-present, Facebook, Inc.
+ * Copyright (c) 2018-2020, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/lib/common/debug.c b/lib/common/debug.c
index 3ebdd1cb15a6..f303f4a2e530 100644
--- a/lib/common/debug.c
+++ b/lib/common/debug.c
@@ -1,35 +1,15 @@
 /* ******************************************************************
-   debug
-   Part of FSE library
-   Copyright (C) 2013-present, Yann Collet.
-
-   BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
-
-   Redistribution and use in source and binary forms, with or without
-   modification, are permitted provided that the following conditions are
-   met:
-
-       * Redistributions of source code must retain the above copyright
-   notice, this list of conditions and the following disclaimer.
-       * Redistributions in binary form must reproduce the above
-   copyright notice, this list of conditions and the following disclaimer
-   in the documentation and/or other materials provided with the
-   distribution.
-
-   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-   You can contact the author at :
-   - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ * debug
+ * Part of FSE library
+ * Copyright (c) 2013-2020, Yann Collet, Facebook, Inc.
+ *
+ * You can contact the author at :
+ * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
 ****************************************************************** */
 
 
diff --git a/lib/common/debug.h b/lib/common/debug.h
index b4fc89d49741..ac6224888d8b 100644
--- a/lib/common/debug.h
+++ b/lib/common/debug.h
@@ -1,35 +1,15 @@
 /* ******************************************************************
-   debug
-   Part of FSE library
-   Copyright (C) 2013-present, Yann Collet.
-
-   BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
-
-   Redistribution and use in source and binary forms, with or without
-   modification, are permitted provided that the following conditions are
-   met:
-
-       * Redistributions of source code must retain the above copyright
-   notice, this list of conditions and the following disclaimer.
-       * Redistributions in binary form must reproduce the above
-   copyright notice, this list of conditions and the following disclaimer
-   in the documentation and/or other materials provided with the
-   distribution.
-
-   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-   You can contact the author at :
-   - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ * debug
+ * Part of FSE library
+ * Copyright (c) 2013-2020, Yann Collet, Facebook, Inc.
+ *
+ * You can contact the author at :
+ * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
 ****************************************************************** */
 
 
diff --git a/lib/common/entropy_common.c b/lib/common/entropy_common.c
index b12944e1de93..9d3e4e8e36ab 100644
--- a/lib/common/entropy_common.c
+++ b/lib/common/entropy_common.c
@@ -1,36 +1,16 @@
-/*
-   Common functions of New Generation Entropy library
-   Copyright (C) 2016, Yann Collet.
-
-   BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
-
-   Redistribution and use in source and binary forms, with or without
-   modification, are permitted provided that the following conditions are
-   met:
-
-       * Redistributions of source code must retain the above copyright
-   notice, this list of conditions and the following disclaimer.
-       * Redistributions in binary form must reproduce the above
-   copyright notice, this list of conditions and the following disclaimer
-   in the documentation and/or other materials provided with the
-   distribution.
-
-   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-    You can contact the author at :
-    - FSE+HUF source repository : https://github.com/Cyan4973/FiniteStateEntropy
-    - Public forum : https://groups.google.com/forum/#!forum/lz4c
-*************************************************************************** */
+/* ******************************************************************
+ * Common functions of New Generation Entropy library
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
+ *
+ *  You can contact the author at :
+ *  - FSE+HUF source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ *  - Public forum : https://groups.google.com/forum/#!forum/lz4c
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+****************************************************************** */
 
 /* *************************************
 *  Dependencies
diff --git a/lib/common/error_private.c b/lib/common/error_private.c
index 7c1bb67a23f4..cd437529c12b 100644
--- a/lib/common/error_private.c
+++ b/lib/common/error_private.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -47,6 +47,7 @@ const char* ERR_getErrorString(ERR_enum code)
         /* following error codes are not stable and may be removed or changed in a future version */
     case PREFIX(frameIndex_tooLarge): return "Frame index is too large";
     case PREFIX(seekableIO): return "An I/O error occurred when reading/seeking";
+    case PREFIX(dstBuffer_wrong): return "Destination buffer is wrong";
     case PREFIX(maxCode):
     default: return notErrorCode;
     }
diff --git a/lib/common/error_private.h b/lib/common/error_private.h
index 0d2fa7e34b01..982cf8e9fe6f 100644
--- a/lib/common/error_private.h
+++ b/lib/common/error_private.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -49,7 +49,7 @@ typedef ZSTD_ErrorCode ERR_enum;
 /*-****************************************
 *  Error codes handling
 ******************************************/
-#undef ERROR   /* reported already defined on VS 2015 (Rich Geldreich) */
+#undef ERROR   /* already defined on Visual Studio */
 #define ERROR(name) ZSTD_ERROR(name)
 #define ZSTD_ERROR(name) ((size_t)-PREFIX(name))
 
@@ -57,6 +57,10 @@ ERR_STATIC unsigned ERR_isError(size_t code) { return (code > ERROR(maxCode)); }
 
 ERR_STATIC ERR_enum ERR_getErrorCode(size_t code) { if (!ERR_isError(code)) return (ERR_enum)0; return (ERR_enum) (0-code); }
 
+/* check and forward error code */
+#define CHECK_V_F(e, f) size_t const e = f; if (ERR_isError(e)) return e
+#define CHECK_F(f)   { CHECK_V_F(_var_err__, f); }
+
 
 /*-****************************************
 *  Error Strings
diff --git a/lib/common/fse.h b/lib/common/fse.h
index a7553e3721c4..ff54e70ea75c 100644
--- a/lib/common/fse.h
+++ b/lib/common/fse.h
@@ -1,35 +1,15 @@
 /* ******************************************************************
-   FSE : Finite State Entropy codec
-   Public Prototypes declaration
-   Copyright (C) 2013-2016, Yann Collet.
-
-   BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
-
-   Redistribution and use in source and binary forms, with or without
-   modification, are permitted provided that the following conditions are
-   met:
-
-       * Redistributions of source code must retain the above copyright
-   notice, this list of conditions and the following disclaimer.
-       * Redistributions in binary form must reproduce the above
-   copyright notice, this list of conditions and the following disclaimer
-   in the documentation and/or other materials provided with the
-   distribution.
-
-   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-   You can contact the author at :
-   - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ * FSE : Finite State Entropy codec
+ * Public Prototypes declaration
+ * Copyright (c) 2013-2020, Yann Collet, Facebook, Inc.
+ *
+ * You can contact the author at :
+ * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
 ****************************************************************** */
 
 #if defined (__cplusplus)
diff --git a/lib/common/fse_decompress.c b/lib/common/fse_decompress.c
index 4f0737898209..bcc2223ccc65 100644
--- a/lib/common/fse_decompress.c
+++ b/lib/common/fse_decompress.c
@@ -1,35 +1,15 @@
 /* ******************************************************************
-   FSE : Finite State Entropy decoder
-   Copyright (C) 2013-2015, Yann Collet.
-
-   BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
-
-   Redistribution and use in source and binary forms, with or without
-   modification, are permitted provided that the following conditions are
-   met:
-
-       * Redistributions of source code must retain the above copyright
-   notice, this list of conditions and the following disclaimer.
-       * Redistributions in binary form must reproduce the above
-   copyright notice, this list of conditions and the following disclaimer
-   in the documentation and/or other materials provided with the
-   distribution.
-
-   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-    You can contact the author at :
-    - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
-    - Public forum : https://groups.google.com/forum/#!forum/lz4c
+ * FSE : Finite State Entropy decoder
+ * Copyright (c) 2013-2020, Yann Collet, Facebook, Inc.
+ *
+ *  You can contact the author at :
+ *  - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ *  - Public forum : https://groups.google.com/forum/#!forum/lz4c
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
 ****************************************************************** */
 
 
@@ -51,11 +31,6 @@
 #define FSE_isError ERR_isError
 #define FSE_STATIC_ASSERT(c) DEBUG_STATIC_ASSERT(c)   /* use only *after* variable declarations */
 
-/* check and forward error code */
-#ifndef CHECK_F
-#define CHECK_F(f) { size_t const e = f; if (FSE_isError(e)) return e; }
-#endif
-
 
 /* **************************************************************
 *  Templates
@@ -287,7 +262,7 @@ size_t FSE_decompress_wksp(void* dst, size_t dstCapacity, const void* cSrc, size
     /* normal FSE decoding mode */
     size_t const NCountLength = FSE_readNCount (counting, &maxSymbolValue, &tableLog, istart, cSrcSize);
     if (FSE_isError(NCountLength)) return NCountLength;
-    //if (NCountLength >= cSrcSize) return ERROR(srcSize_wrong);   /* too small input size; supposed to be already checked in NCountLength, only remaining case : NCountLength==cSrcSize */
+    /* if (NCountLength >= cSrcSize) return ERROR(srcSize_wrong); */  /* too small input size; supposed to be already checked in NCountLength, only remaining case : NCountLength==cSrcSize */
     if (tableLog > maxLog) return ERROR(tableLog_tooLarge);
     ip += NCountLength;
     cSrcSize -= NCountLength;
diff --git a/lib/common/huf.h b/lib/common/huf.h
index 6b572c448d9d..ef432685dac8 100644
--- a/lib/common/huf.h
+++ b/lib/common/huf.h
@@ -1,35 +1,15 @@
 /* ******************************************************************
-   huff0 huffman codec,
-   part of Finite State Entropy library
-   Copyright (C) 2013-present, Yann Collet.
-
-   BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
-
-   Redistribution and use in source and binary forms, with or without
-   modification, are permitted provided that the following conditions are
-   met:
-
-       * Redistributions of source code must retain the above copyright
-   notice, this list of conditions and the following disclaimer.
-       * Redistributions in binary form must reproduce the above
-   copyright notice, this list of conditions and the following disclaimer
-   in the documentation and/or other materials provided with the
-   distribution.
-
-   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-   You can contact the author at :
-   - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ * huff0 huffman codec,
+ * part of Finite State Entropy library
+ * Copyright (c) 2013-2020, Yann Collet, Facebook, Inc.
+ *
+ * You can contact the author at :
+ * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
 ****************************************************************** */
 
 #if defined (__cplusplus)
@@ -110,7 +90,7 @@ HUF_PUBLIC_API size_t HUF_compress2 (void* dst, size_t dstCapacity,
 /** HUF_compress4X_wksp() :
  *  Same as HUF_compress2(), but uses externally allocated `workSpace`.
  * `workspace` must have minimum alignment of 4, and be at least as large as HUF_WORKSPACE_SIZE */
-#define HUF_WORKSPACE_SIZE (6 << 10)
+#define HUF_WORKSPACE_SIZE ((6 << 10) + 256)
 #define HUF_WORKSPACE_SIZE_U32 (HUF_WORKSPACE_SIZE / sizeof(U32))
 HUF_PUBLIC_API size_t HUF_compress4X_wksp (void* dst, size_t dstCapacity,
                                      const void* src, size_t srcSize,
@@ -208,6 +188,8 @@ typedef struct HUF_CElt_s HUF_CElt;   /* incomplete type */
 size_t HUF_buildCTable (HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue, unsigned maxNbBits);   /* @return : maxNbBits; CTable and count can overlap. In which case, CTable will overwrite count content */
 size_t HUF_writeCTable (void* dst, size_t maxDstSize, const HUF_CElt* CTable, unsigned maxSymbolValue, unsigned huffLog);
 size_t HUF_compress4X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable);
+size_t HUF_estimateCompressedSize(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue);
+int HUF_validateCTable(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue);
 
 typedef enum {
    HUF_repeat_none,  /**< Cannot use the previous table */
@@ -246,7 +228,7 @@ size_t HUF_readStats(BYTE* huffWeight, size_t hwSize,
 
 /** HUF_readCTable() :
  *  Loading a CTable saved with HUF_writeCTable() */
-size_t HUF_readCTable (HUF_CElt* CTable, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize);
+size_t HUF_readCTable (HUF_CElt* CTable, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize, unsigned *hasZeroWeights);
 
 /** HUF_getNbBits() :
  *  Read nbBits from CTable symbolTable, for symbol `symbolValue` presumed <= HUF_SYMBOLVALUE_MAX
diff --git a/lib/common/mem.h b/lib/common/mem.h
index 530d30c8f758..89c8aea7d221 100644
--- a/lib/common/mem.h
+++ b/lib/common/mem.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/lib/common/pool.c b/lib/common/pool.c
index f575935076cf..aa4b4de0d3f6 100644
--- a/lib/common/pool.c
+++ b/lib/common/pool.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/lib/common/pool.h b/lib/common/pool.h
index 458d37f13c3e..259bafc97570 100644
--- a/lib/common/pool.h
+++ b/lib/common/pool.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -18,7 +18,7 @@ extern "C" {
 
 #include <stddef.h>   /* size_t */
 #define ZSTD_STATIC_LINKING_ONLY   /* ZSTD_customMem */
-#include "zstd.h"
+#include "../zstd.h"
 
 typedef struct POOL_ctx_s POOL_ctx;
 
diff --git a/lib/common/threading.c b/lib/common/threading.c
index 482664bd9ada..e2edb313ebff 100644
--- a/lib/common/threading.c
+++ b/lib/common/threading.c
@@ -2,12 +2,13 @@
  * Copyright (c) 2016 Tino Reichardt
  * All rights reserved.
  *
+ * You can contact the author at:
+ * - zstdmt source repository: https://github.com/mcmilk/zstdmt
+ *
  * This source code is licensed under both the BSD-style license (found in the
  * LICENSE file in the root directory of this source tree) and the GPLv2 (found
  * in the COPYING file in the root directory of this source tree).
- *
- * You can contact the author at:
- * - zstdmt source repository: https://github.com/mcmilk/zstdmt
+ * You may select, at your option, one of the above-listed licenses.
  */
 
 /**
diff --git a/lib/common/threading.h b/lib/common/threading.h
index 3193ca7db86c..fd0060d5aa2a 100644
--- a/lib/common/threading.h
+++ b/lib/common/threading.h
@@ -2,12 +2,13 @@
  * Copyright (c) 2016 Tino Reichardt
  * All rights reserved.
  *
+ * You can contact the author at:
+ * - zstdmt source repository: https://github.com/mcmilk/zstdmt
+ *
  * This source code is licensed under both the BSD-style license (found in the
  * LICENSE file in the root directory of this source tree) and the GPLv2 (found
  * in the COPYING file in the root directory of this source tree).
- *
- * You can contact the author at:
- * - zstdmt source repository: https://github.com/mcmilk/zstdmt
+ * You may select, at your option, one of the above-listed licenses.
  */
 
 #ifndef THREADING_H_938743
diff --git a/lib/common/xxhash.c b/lib/common/xxhash.c
index 99d245962185..597de18fc895 100644
--- a/lib/common/xxhash.c
+++ b/lib/common/xxhash.c
@@ -1,35 +1,15 @@
 /*
-*  xxHash - Fast Hash algorithm
-*  Copyright (C) 2012-2016, Yann Collet
-*
-*  BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
-*
-*  Redistribution and use in source and binary forms, with or without
-*  modification, are permitted provided that the following conditions are
-*  met:
-*
-*  * Redistributions of source code must retain the above copyright
-*  notice, this list of conditions and the following disclaimer.
-*  * Redistributions in binary form must reproduce the above
-*  copyright notice, this list of conditions and the following disclaimer
-*  in the documentation and/or other materials provided with the
-*  distribution.
-*
-*  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-*  "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-*  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-*  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-*  OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-*  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-*  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-*  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-*  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-*  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-*  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-*
-*  You can contact the author at :
-*  - xxHash homepage: http://www.xxhash.com
-*  - xxHash source repository : https://github.com/Cyan4973/xxHash
+ *  xxHash - Fast Hash algorithm
+ *  Copyright (c) 2012-2020, Yann Collet, Facebook, Inc.
+ *
+ *  You can contact the author at :
+ *  - xxHash homepage: http://www.xxhash.com
+ *  - xxHash source repository : https://github.com/Cyan4973/xxHash
+ * 
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
 */
 
 
@@ -115,7 +95,7 @@ static void* XXH_memcpy(void* dest, const void* src, size_t size) { return memcp
 /* *************************************
 *  Compiler Specific Options
 ***************************************/
-#if defined (__GNUC__) || defined(__cplusplus) || defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L   /* C99 */
+#if (defined(__GNUC__) && !defined(__STRICT_ANSI__)) || defined(__cplusplus) || defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L   /* C99 */
 #  define INLINE_KEYWORD inline
 #else
 #  define INLINE_KEYWORD
@@ -729,7 +709,9 @@ FORCE_INLINE_TEMPLATE XXH_errorcode XXH64_update_endian (XXH64_state_t* state, c
     state->total_len += len;
 
     if (state->memsize + len < 32) {  /* fill in tmp buffer */
-        XXH_memcpy(((BYTE*)state->mem64) + state->memsize, input, len);
+        if (input != NULL) {
+            XXH_memcpy(((BYTE*)state->mem64) + state->memsize, input, len);
+        }
         state->memsize += (U32)len;
         return XXH_OK;
     }
diff --git a/lib/common/xxhash.h b/lib/common/xxhash.h
index 9bad1f59f63a..4207eba8328f 100644
--- a/lib/common/xxhash.h
+++ b/lib/common/xxhash.h
@@ -1,35 +1,15 @@
 /*
-   xxHash - Extremely Fast Hash algorithm
-   Header File
-   Copyright (C) 2012-2016, Yann Collet.
-
-   BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
-
-   Redistribution and use in source and binary forms, with or without
-   modification, are permitted provided that the following conditions are
-   met:
-
-       * Redistributions of source code must retain the above copyright
-   notice, this list of conditions and the following disclaimer.
-       * Redistributions in binary form must reproduce the above
-   copyright notice, this list of conditions and the following disclaimer
-   in the documentation and/or other materials provided with the
-   distribution.
-
-   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-   You can contact the author at :
-   - xxHash source repository : https://github.com/Cyan4973/xxHash
+ * xxHash - Extremely Fast Hash algorithm
+ * Header File
+ * Copyright (c) 2012-2020, Yann Collet, Facebook, Inc.
+ *
+ * You can contact the author at :
+ * - xxHash source repository : https://github.com/Cyan4973/xxHash
+ * 
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
 */
 
 /* Notice extracted from xxHash homepage :
diff --git a/lib/common/zstd_common.c b/lib/common/zstd_common.c
index 667f4a27fc69..91fe3323a5ba 100644
--- a/lib/common/zstd_common.c
+++ b/lib/common/zstd_common.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/lib/common/zstd_errors.h b/lib/common/zstd_errors.h
index 92a3433896c5..998398e7e57f 100644
--- a/lib/common/zstd_errors.h
+++ b/lib/common/zstd_errors.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -76,6 +76,7 @@ typedef enum {
   /* following error codes are __NOT STABLE__, they can be removed or changed in future versions */
   ZSTD_error_frameIndex_tooLarge = 100,
   ZSTD_error_seekableIO          = 102,
+  ZSTD_error_dstBuffer_wrong     = 104,
   ZSTD_error_maxCode = 120  /* never EVER use this value directly, it can change in future versions! Use ZSTD_isError() instead */
 } ZSTD_ErrorCode;
 
diff --git a/lib/common/zstd_internal.h b/lib/common/zstd_internal.h
index dcdcbdb81cd7..3bc7e55a0a97 100644
--- a/lib/common/zstd_internal.h
+++ b/lib/common/zstd_internal.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -19,12 +19,15 @@
 /*-*************************************
 *  Dependencies
 ***************************************/
+#ifdef __aarch64__
+#include <arm_neon.h>
+#endif
 #include "compiler.h"
 #include "mem.h"
 #include "debug.h"                 /* assert, DEBUGLOG, RAWLOG, g_debuglevel */
 #include "error_private.h"
 #define ZSTD_STATIC_LINKING_ONLY
-#include "zstd.h"
+#include "../zstd.h"
 #define FSE_STATIC_LINKING_ONLY
 #include "fse.h"
 #define HUF_STATIC_LINKING_ONLY
@@ -54,6 +57,31 @@ extern "C" {
 #define MAX(a,b) ((a)>(b) ? (a) : (b))
 
 /**
+ * Ignore: this is an internal helper.
+ *
+ * This is a helper function to help force C99-correctness during compilation.
+ * Under strict compilation modes, variadic macro arguments can't be empty.
+ * However, variadic function arguments can be. Using a function therefore lets
+ * us statically check that at least one (string) argument was passed,
+ * independent of the compilation flags.
+ */
+static INLINE_KEYWORD UNUSED_ATTR
+void _force_has_format_string(const char *format, ...) {
+  (void)format;
+}
+
+/**
+ * Ignore: this is an internal helper.
+ *
+ * We want to force this function invocation to be syntactically correct, but
+ * we don't want to force runtime evaluation of its arguments.
+ */
+#define _FORCE_HAS_FORMAT_STRING(...) \
+  if (0) { \
+    _force_has_format_string(__VA_ARGS__); \
+  }
+
+/**
  * Return the specified error if the condition evaluates to true.
  *
  * In debug modes, prints additional information.
@@ -62,7 +90,9 @@ extern "C" {
  */
 #define RETURN_ERROR_IF(cond, err, ...) \
   if (cond) { \
-    RAWLOG(3, "%s:%d: ERROR!: check %s failed, returning %s", __FILE__, __LINE__, ZSTD_QUOTE(cond), ZSTD_QUOTE(ERROR(err))); \
+    RAWLOG(3, "%s:%d: ERROR!: check %s failed, returning %s", \
+           __FILE__, __LINE__, ZSTD_QUOTE(cond), ZSTD_QUOTE(ERROR(err))); \
+    _FORCE_HAS_FORMAT_STRING(__VA_ARGS__); \
     RAWLOG(3, ": " __VA_ARGS__); \
     RAWLOG(3, "\n"); \
     return ERROR(err); \
@@ -75,7 +105,9 @@ extern "C" {
  */
 #define RETURN_ERROR(err, ...) \
   do { \
-    RAWLOG(3, "%s:%d: ERROR!: unconditional check failed, returning %s", __FILE__, __LINE__, ZSTD_QUOTE(ERROR(err))); \
+    RAWLOG(3, "%s:%d: ERROR!: unconditional check failed, returning %s", \
+           __FILE__, __LINE__, ZSTD_QUOTE(ERROR(err))); \
+    _FORCE_HAS_FORMAT_STRING(__VA_ARGS__); \
     RAWLOG(3, ": " __VA_ARGS__); \
     RAWLOG(3, "\n"); \
     return ERROR(err); \
@@ -90,7 +122,9 @@ extern "C" {
   do { \
     size_t const err_code = (err); \
     if (ERR_isError(err_code)) { \
-      RAWLOG(3, "%s:%d: ERROR!: forwarding error in %s: %s", __FILE__, __LINE__, ZSTD_QUOTE(err), ERR_getErrorName(err_code)); \
+      RAWLOG(3, "%s:%d: ERROR!: forwarding error in %s: %s", \
+             __FILE__, __LINE__, ZSTD_QUOTE(err), ERR_getErrorName(err_code)); \
+      _FORCE_HAS_FORMAT_STRING(__VA_ARGS__); \
       RAWLOG(3, ": " __VA_ARGS__); \
       RAWLOG(3, "\n"); \
       return err_code; \
@@ -128,6 +162,8 @@ static const size_t ZSTD_did_fieldSize[4] = { 0, 1, 2, 4 };
 static const size_t ZSTD_blockHeaderSize = ZSTD_BLOCKHEADERSIZE;
 typedef enum { bt_raw, bt_rle, bt_compressed, bt_reserved } blockType_e;
 
+#define ZSTD_FRAMECHECKSUMSIZE 4
+
 #define MIN_SEQUENCES_SIZE 1 /* nbSeq==0 */
 #define MIN_CBLOCK_SIZE (1 /*litCSize*/ + 1 /* RLE or RAW */ + MIN_SEQUENCES_SIZE /* nbSeq==0 */)   /* for a non-null block */
 
@@ -191,10 +227,22 @@ static const U32 OF_defaultNormLog = OF_DEFAULTNORMLOG;
 /*-*******************************************
 *  Shared functions to include for inlining
 *********************************************/
-static void ZSTD_copy8(void* dst, const void* src) { memcpy(dst, src, 8); }
+static void ZSTD_copy8(void* dst, const void* src) {
+#ifdef __aarch64__
+    vst1_u8((uint8_t*)dst, vld1_u8((const uint8_t*)src));
+#else
+    memcpy(dst, src, 8);
+#endif
+}
 
 #define COPY8(d,s) { ZSTD_copy8(d,s); d+=8; s+=8; }
-static void ZSTD_copy16(void* dst, const void* src) { memcpy(dst, src, 16); }
+static void ZSTD_copy16(void* dst, const void* src) {
+#ifdef __aarch64__
+    vst1q_u8((uint8_t*)dst, vld1q_u8((const uint8_t*)src));
+#else
+    memcpy(dst, src, 16);
+#endif
+}
 #define COPY16(d,s) { ZSTD_copy16(d,s); d+=16; s+=16; }
 
 #define WILDCOPY_OVERLENGTH 32
@@ -213,7 +261,7 @@ typedef enum {
  *         - ZSTD_overlap_src_before_dst: The src and dst may overlap, but they MUST be at least 8 bytes apart.
  *           The src buffer must be before the dst buffer.
  */
-MEM_STATIC FORCE_INLINE_ATTR DONT_VECTORIZE
+MEM_STATIC FORCE_INLINE_ATTR 
 void ZSTD_wildcopy(void* dst, const void* src, ptrdiff_t length, ZSTD_overlap_e const ovtype)
 {
     ptrdiff_t diff = (BYTE*)dst - (const BYTE*)src;
@@ -230,13 +278,18 @@ void ZSTD_wildcopy(void* dst, const void* src, ptrdiff_t length, ZSTD_overlap_e
         } while (op < oend);
     } else {
         assert(diff >= WILDCOPY_VECLEN || diff <= -WILDCOPY_VECLEN);
-        /* Separate out the first two COPY16() calls because the copy length is
+        /* Separate out the first COPY16() call because the copy length is
          * almost certain to be short, so the branches have different
-         * probabilities.
-         * On gcc-9 unrolling once is +1.6%, twice is +2%, thrice is +1.8%.
-         * On clang-8 unrolling once is +1.4%, twice is +3.3%, thrice is +3%.
+         * probabilities. Since it is almost certain to be short, only do
+         * one COPY16() in the first call. Then, do two calls per loop since
+         * at that point it is more likely to have a high trip count.
          */
-        COPY16(op, ip);
+#ifndef __aarch64__
+        do {
+            COPY16(op, ip);
+        }
+        while (op < oend);
+#else
         COPY16(op, ip);
         if (op >= oend) return;
         do {
@@ -244,9 +297,29 @@ void ZSTD_wildcopy(void* dst, const void* src, ptrdiff_t length, ZSTD_overlap_e
             COPY16(op, ip);
         }
         while (op < oend);
+#endif
+    }
+}
+
+MEM_STATIC size_t ZSTD_limitCopy(void* dst, size_t dstCapacity, const void* src, size_t srcSize)
+{
+    size_t const length = MIN(dstCapacity, srcSize);
+    if (length > 0) {
+        memcpy(dst, src, length);
     }
+    return length;
 }
 
+/* define "workspace is too large" as this number of times larger than needed */
+#define ZSTD_WORKSPACETOOLARGE_FACTOR 3
+
+/* when workspace is continuously too large
+ * during at least this number of times,
+ * context's memory usage is considered wasteful,
+ * because it's sized to handle a worst case scenario which rarely happens.
+ * In which case, resize it down to free some memory */
+#define ZSTD_WORKSPACETOOLARGE_MAXDURATION 128
+
 
 /*-*******************************************
 *  Private declarations
@@ -271,6 +344,31 @@ typedef struct {
     U32   longLengthPos;
 } seqStore_t;
 
+typedef struct {
+    U32 litLength;
+    U32 matchLength;
+} ZSTD_sequenceLength;
+
+/**
+ * Returns the ZSTD_sequenceLength for the given sequences. It handles the decoding of long sequences
+ * indicated by longLengthPos and longLengthID, and adds MINMATCH back to matchLength.
+ */
+MEM_STATIC ZSTD_sequenceLength ZSTD_getSequenceLength(seqStore_t const* seqStore, seqDef const* seq)
+{
+    ZSTD_sequenceLength seqLen;
+    seqLen.litLength = seq->litLength;
+    seqLen.matchLength = seq->matchLength + MINMATCH;
+    if (seqStore->longLengthPos == (U32)(seq - seqStore->sequencesStart)) {
+        if (seqStore->longLengthID == 1) {
+            seqLen.litLength += 0xFFFF;
+        }
+        if (seqStore->longLengthID == 2) {
+            seqLen.matchLength += 0xFFFF;
+        }
+    }
+    return seqLen;
+}
+
 /**
  * Contains the compressed frame size and an upper-bound for the decompressed frame size.
  * Note: before using `compressedSize`, check for errors using ZSTD_isError().
@@ -297,8 +395,7 @@ MEM_STATIC U32 ZSTD_highbit32(U32 val)   /* compress, dictBuilder, decodeCorpus
     {
 #   if defined(_MSC_VER)   /* Visual */
         unsigned long r=0;
-        _BitScanReverse(&r, val);
-        return (unsigned)r;
+        return _BitScanReverse(&r, val) ? (unsigned)r : 0;
 #   elif defined(__GNUC__) && (__GNUC__ >= 3)   /* GCC Intrinsic */
         return __builtin_clz (val) ^ 31;
 #   elif defined(__ICCARM__)    /* IAR Intrinsic */
diff --git a/lib/compress/fse_compress.c b/lib/compress/fse_compress.c
index 68b47e109351..a42759814fdd 100644
--- a/lib/compress/fse_compress.c
+++ b/lib/compress/fse_compress.c
@@ -1,35 +1,15 @@
 /* ******************************************************************
-   FSE : Finite State Entropy encoder
-   Copyright (C) 2013-present, Yann Collet.
-
-   BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
-
-   Redistribution and use in source and binary forms, with or without
-   modification, are permitted provided that the following conditions are
-   met:
-
-       * Redistributions of source code must retain the above copyright
-   notice, this list of conditions and the following disclaimer.
-       * Redistributions in binary form must reproduce the above
-   copyright notice, this list of conditions and the following disclaimer
-   in the documentation and/or other materials provided with the
-   distribution.
-
-   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-    You can contact the author at :
-    - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
-    - Public forum : https://groups.google.com/forum/#!forum/lz4c
+ * FSE : Finite State Entropy encoder
+ * Copyright (c) 2013-2020, Yann Collet, Facebook, Inc.
+ *
+ *  You can contact the author at :
+ *  - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ *  - Public forum : https://groups.google.com/forum/#!forum/lz4c
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
 ****************************************************************** */
 
 /* **************************************************************
@@ -37,14 +17,14 @@
 ****************************************************************/
 #include <stdlib.h>     /* malloc, free, qsort */
 #include <string.h>     /* memcpy, memset */
-#include "compiler.h"
-#include "mem.h"        /* U32, U16, etc. */
-#include "debug.h"      /* assert, DEBUGLOG */
+#include "../common/compiler.h"
+#include "../common/mem.h"        /* U32, U16, etc. */
+#include "../common/debug.h"      /* assert, DEBUGLOG */
 #include "hist.h"       /* HIST_count_wksp */
-#include "bitstream.h"
+#include "../common/bitstream.h"
 #define FSE_STATIC_LINKING_ONLY
-#include "fse.h"
-#include "error_private.h"
+#include "../common/fse.h"
+#include "../common/error_private.h"
 
 
 /* **************************************************************
@@ -645,9 +625,6 @@ size_t FSE_compress_usingCTable (void* dst, size_t dstSize,
 
 size_t FSE_compressBound(size_t size) { return FSE_COMPRESSBOUND(size); }
 
-#define CHECK_V_F(e, f) size_t const e = f; if (ERR_isError(e)) return e
-#define CHECK_F(f)   { CHECK_V_F(_var_err__, f); }
-
 /* FSE_compress_wksp() :
  * Same as FSE_compress2(), but using an externally allocated scratch buffer (`workSpace`).
  * `wkspSize` size must be `(1<<tableLog)`.
diff --git a/lib/compress/hist.c b/lib/compress/hist.c
index 45b7babc1e23..61e08c7968be 100644
--- a/lib/compress/hist.c
+++ b/lib/compress/hist.c
@@ -1,42 +1,22 @@
 /* ******************************************************************
-   hist : Histogram functions
-   part of Finite State Entropy project
-   Copyright (C) 2013-present, Yann Collet.
-
-   BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
-
-   Redistribution and use in source and binary forms, with or without
-   modification, are permitted provided that the following conditions are
-   met:
-
-       * Redistributions of source code must retain the above copyright
-   notice, this list of conditions and the following disclaimer.
-       * Redistributions in binary form must reproduce the above
-   copyright notice, this list of conditions and the following disclaimer
-   in the documentation and/or other materials provided with the
-   distribution.
-
-   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-    You can contact the author at :
-    - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
-    - Public forum : https://groups.google.com/forum/#!forum/lz4c
+ * hist : Histogram functions
+ * part of Finite State Entropy project
+ * Copyright (c) 2013-2020, Yann Collet, Facebook, Inc.
+ *
+ *  You can contact the author at :
+ *  - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ *  - Public forum : https://groups.google.com/forum/#!forum/lz4c
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
 ****************************************************************** */
 
 /* --- dependencies --- */
-#include "mem.h"             /* U32, BYTE, etc. */
-#include "debug.h"           /* assert, DEBUGLOG */
-#include "error_private.h"   /* ERROR */
+#include "../common/mem.h"             /* U32, BYTE, etc. */
+#include "../common/debug.h"           /* assert, DEBUGLOG */
+#include "../common/error_private.h"   /* ERROR */
 #include "hist.h"
 
 
diff --git a/lib/compress/hist.h b/lib/compress/hist.h
index 8b389358dc10..77e3ec4fb192 100644
--- a/lib/compress/hist.h
+++ b/lib/compress/hist.h
@@ -1,36 +1,16 @@
 /* ******************************************************************
-   hist : Histogram functions
-   part of Finite State Entropy project
-   Copyright (C) 2013-present, Yann Collet.
-
-   BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
-
-   Redistribution and use in source and binary forms, with or without
-   modification, are permitted provided that the following conditions are
-   met:
-
-       * Redistributions of source code must retain the above copyright
-   notice, this list of conditions and the following disclaimer.
-       * Redistributions in binary form must reproduce the above
-   copyright notice, this list of conditions and the following disclaimer
-   in the documentation and/or other materials provided with the
-   distribution.
-
-   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-    You can contact the author at :
-    - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
-    - Public forum : https://groups.google.com/forum/#!forum/lz4c
+ * hist : Histogram functions
+ * part of Finite State Entropy project
+ * Copyright (c) 2013-2020, Yann Collet, Facebook, Inc.
+ *
+ *  You can contact the author at :
+ *  - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ *  - Public forum : https://groups.google.com/forum/#!forum/lz4c
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
 ****************************************************************** */
 
 /* --- dependencies --- */
diff --git a/lib/compress/huf_compress.c b/lib/compress/huf_compress.c
index f074f1e0a95d..546879868a53 100644
--- a/lib/compress/huf_compress.c
+++ b/lib/compress/huf_compress.c
@@ -1,35 +1,15 @@
 /* ******************************************************************
-   Huffman encoder, part of New Generation Entropy library
-   Copyright (C) 2013-2016, Yann Collet.
-
-   BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
-
-   Redistribution and use in source and binary forms, with or without
-   modification, are permitted provided that the following conditions are
-   met:
-
-       * Redistributions of source code must retain the above copyright
-   notice, this list of conditions and the following disclaimer.
-       * Redistributions in binary form must reproduce the above
-   copyright notice, this list of conditions and the following disclaimer
-   in the documentation and/or other materials provided with the
-   distribution.
-
-   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-    You can contact the author at :
-    - FSE+HUF source repository : https://github.com/Cyan4973/FiniteStateEntropy
-    - Public forum : https://groups.google.com/forum/#!forum/lz4c
+ * Huffman encoder, part of New Generation Entropy library
+ * Copyright (c) 2013-2020, Yann Collet, Facebook, Inc.
+ *
+ *  You can contact the author at :
+ *  - FSE+HUF source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ *  - Public forum : https://groups.google.com/forum/#!forum/lz4c
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
 ****************************************************************** */
 
 /* **************************************************************
@@ -45,14 +25,14 @@
 ****************************************************************/
 #include <string.h>     /* memcpy, memset */
 #include <stdio.h>      /* printf (debug) */
-#include "compiler.h"
-#include "bitstream.h"
+#include "../common/compiler.h"
+#include "../common/bitstream.h"
 #include "hist.h"
 #define FSE_STATIC_LINKING_ONLY   /* FSE_optimalTableLog_internal */
-#include "fse.h"        /* header compression */
+#include "../common/fse.h"        /* header compression */
 #define HUF_STATIC_LINKING_ONLY
-#include "huf.h"
-#include "error_private.h"
+#include "../common/huf.h"
+#include "../common/error_private.h"
 
 
 /* **************************************************************
@@ -60,8 +40,6 @@
 ****************************************************************/
 #define HUF_isError ERR_isError
 #define HUF_STATIC_ASSERT(c) DEBUG_STATIC_ASSERT(c)   /* use only *after* variable declarations */
-#define CHECK_V_F(e, f) size_t const e = f; if (ERR_isError(e)) return e
-#define CHECK_F(f)   { CHECK_V_F(_var_err__, f); }
 
 
 /* **************************************************************
@@ -110,18 +88,18 @@ static size_t HUF_compressWeights (void* dst, size_t dstSize, const void* weight
     CHECK_F( FSE_normalizeCount(norm, tableLog, count, wtSize, maxSymbolValue) );
 
     /* Write table description header */
-    {   CHECK_V_F(hSize, FSE_writeNCount(op, oend-op, norm, maxSymbolValue, tableLog) );
+    {   CHECK_V_F(hSize, FSE_writeNCount(op, (size_t)(oend-op), norm, maxSymbolValue, tableLog) );
         op += hSize;
     }
 
     /* Compress */
     CHECK_F( FSE_buildCTable_wksp(CTable, norm, maxSymbolValue, tableLog, scratchBuffer, sizeof(scratchBuffer)) );
-    {   CHECK_V_F(cSize, FSE_compress_usingCTable(op, oend - op, weightTable, wtSize, CTable) );
+    {   CHECK_V_F(cSize, FSE_compress_usingCTable(op, (size_t)(oend - op), weightTable, wtSize, CTable) );
         if (cSize == 0) return 0;   /* not enough space for compressed data */
         op += cSize;
     }
 
-    return op-ostart;
+    return (size_t)(op-ostart);
 }
 
 
@@ -169,7 +147,7 @@ size_t HUF_writeCTable (void* dst, size_t maxDstSize,
 }
 
 
-size_t HUF_readCTable (HUF_CElt* CTable, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize)
+size_t HUF_readCTable (HUF_CElt* CTable, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize, unsigned* hasZeroWeights)
 {
     BYTE huffWeight[HUF_SYMBOLVALUE_MAX + 1];   /* init not required, even though some static analyzer may complain */
     U32 rankVal[HUF_TABLELOG_ABSOLUTEMAX + 1];   /* large enough for values from 0 to 16 */
@@ -192,9 +170,11 @@ size_t HUF_readCTable (HUF_CElt* CTable, unsigned* maxSymbolValuePtr, const void
     }   }
 
     /* fill nbBits */
+    *hasZeroWeights = 0;
     {   U32 n; for (n=0; n<nbSymbols; n++) {
             const U32 w = huffWeight[n];
-            CTable[n].nbBits = (BYTE)(tableLog + 1 - w);
+            *hasZeroWeights |= (w == 0);
+            CTable[n].nbBits = (BYTE)(tableLog + 1 - w) & -(w != 0);
     }   }
 
     /* fill val */
@@ -240,7 +220,7 @@ static U32 HUF_setMaxHeight(nodeElt* huffNode, U32 lastNonNull, U32 maxNbBits)
     /* there are several too large elements (at least >= 2) */
     {   int totalCost = 0;
         const U32 baseCost = 1 << (largestBits - maxNbBits);
-        U32 n = lastNonNull;
+        int n = (int)lastNonNull;
 
         while (huffNode[n].nbBits > maxNbBits) {
             totalCost += baseCost - (1 << (largestBits - huffNode[n].nbBits));
@@ -255,22 +235,22 @@ static U32 HUF_setMaxHeight(nodeElt* huffNode, U32 lastNonNull, U32 maxNbBits)
         /* repay normalized cost */
         {   U32 const noSymbol = 0xF0F0F0F0;
             U32 rankLast[HUF_TABLELOG_MAX+2];
-            int pos;
 
             /* Get pos of last (smallest) symbol per rank */
             memset(rankLast, 0xF0, sizeof(rankLast));
             {   U32 currentNbBits = maxNbBits;
+                int pos;
                 for (pos=n ; pos >= 0; pos--) {
                     if (huffNode[pos].nbBits >= currentNbBits) continue;
                     currentNbBits = huffNode[pos].nbBits;   /* < maxNbBits */
-                    rankLast[maxNbBits-currentNbBits] = pos;
+                    rankLast[maxNbBits-currentNbBits] = (U32)pos;
             }   }
 
             while (totalCost > 0) {
-                U32 nBitsToDecrease = BIT_highbit32(totalCost) + 1;
+                U32 nBitsToDecrease = BIT_highbit32((U32)totalCost) + 1;
                 for ( ; nBitsToDecrease > 1; nBitsToDecrease--) {
-                    U32 highPos = rankLast[nBitsToDecrease];
-                    U32 lowPos = rankLast[nBitsToDecrease-1];
+                    U32 const highPos = rankLast[nBitsToDecrease];
+                    U32 const lowPos = rankLast[nBitsToDecrease-1];
                     if (highPos == noSymbol) continue;
                     if (lowPos == noSymbol) break;
                     {   U32 const highTotal = huffNode[highPos].count;
@@ -297,7 +277,8 @@ static U32 HUF_setMaxHeight(nodeElt* huffNode, U32 lastNonNull, U32 maxNbBits)
                 if (rankLast[1] == noSymbol) {  /* special case : no rank 1 symbol (using maxNbBits-1); let's create one from largest rank 0 (using maxNbBits) */
                     while (huffNode[n].nbBits == maxNbBits) n--;
                     huffNode[n+1].nbBits--;
-                    rankLast[1] = n+1;
+                    assert(n >= 0);
+                    rankLast[1] = (U32)(n+1);
                     totalCost++;
                     continue;
                 }
@@ -309,29 +290,36 @@ static U32 HUF_setMaxHeight(nodeElt* huffNode, U32 lastNonNull, U32 maxNbBits)
     return maxNbBits;
 }
 
-
 typedef struct {
     U32 base;
     U32 current;
 } rankPos;
 
-static void HUF_sort(nodeElt* huffNode, const unsigned* count, U32 maxSymbolValue)
+typedef nodeElt huffNodeTable[HUF_CTABLE_WORKSPACE_SIZE_U32];
+
+#define RANK_POSITION_TABLE_SIZE 32
+
+typedef struct {
+  huffNodeTable huffNodeTbl;
+  rankPos rankPosition[RANK_POSITION_TABLE_SIZE];
+} HUF_buildCTable_wksp_tables;
+
+static void HUF_sort(nodeElt* huffNode, const unsigned* count, U32 maxSymbolValue, rankPos* rankPosition)
 {
-    rankPos rank[32];
     U32 n;
 
-    memset(rank, 0, sizeof(rank));
+    memset(rankPosition, 0, sizeof(*rankPosition) * RANK_POSITION_TABLE_SIZE);
     for (n=0; n<=maxSymbolValue; n++) {
         U32 r = BIT_highbit32(count[n] + 1);
-        rank[r].base ++;
+        rankPosition[r].base ++;
     }
-    for (n=30; n>0; n--) rank[n-1].base += rank[n].base;
-    for (n=0; n<32; n++) rank[n].current = rank[n].base;
+    for (n=30; n>0; n--) rankPosition[n-1].base += rankPosition[n].base;
+    for (n=0; n<32; n++) rankPosition[n].current = rankPosition[n].base;
     for (n=0; n<=maxSymbolValue; n++) {
         U32 const c = count[n];
         U32 const r = BIT_highbit32(c+1) + 1;
-        U32 pos = rank[r].current++;
-        while ((pos > rank[r].base) && (c > huffNode[pos-1].count)) {
+        U32 pos = rankPosition[r].current++;
+        while ((pos > rankPosition[r].base) && (c > huffNode[pos-1].count)) {
             huffNode[pos] = huffNode[pos-1];
             pos--;
         }
@@ -343,45 +331,48 @@ static void HUF_sort(nodeElt* huffNode, const unsigned* count, U32 maxSymbolValu
 
 /** HUF_buildCTable_wksp() :
  *  Same as HUF_buildCTable(), but using externally allocated scratch buffer.
- *  `workSpace` must be aligned on 4-bytes boundaries, and be at least as large as a table of HUF_CTABLE_WORKSPACE_SIZE_U32 unsigned.
+ *  `workSpace` must be aligned on 4-bytes boundaries, and be at least as large as sizeof(HUF_buildCTable_wksp_tables).
  */
 #define STARTNODE (HUF_SYMBOLVALUE_MAX+1)
-typedef nodeElt huffNodeTable[HUF_CTABLE_WORKSPACE_SIZE_U32];
+
 size_t HUF_buildCTable_wksp (HUF_CElt* tree, const unsigned* count, U32 maxSymbolValue, U32 maxNbBits, void* workSpace, size_t wkspSize)
 {
-    nodeElt* const huffNode0 = (nodeElt*)workSpace;
+    HUF_buildCTable_wksp_tables* const wksp_tables = (HUF_buildCTable_wksp_tables*)workSpace;
+    nodeElt* const huffNode0 = wksp_tables->huffNodeTbl;
     nodeElt* const huffNode = huffNode0+1;
-    U32 n, nonNullRank;
+    int nonNullRank;
     int lowS, lowN;
-    U16 nodeNb = STARTNODE;
-    U32 nodeRoot;
+    int nodeNb = STARTNODE;
+    int n, nodeRoot;
 
     /* safety checks */
     if (((size_t)workSpace & 3) != 0) return ERROR(GENERIC);  /* must be aligned on 4-bytes boundaries */
-    if (wkspSize < sizeof(huffNodeTable)) return ERROR(workSpace_tooSmall);
+    if (wkspSize < sizeof(HUF_buildCTable_wksp_tables))
+      return ERROR(workSpace_tooSmall);
     if (maxNbBits == 0) maxNbBits = HUF_TABLELOG_DEFAULT;
-    if (maxSymbolValue > HUF_SYMBOLVALUE_MAX) return ERROR(maxSymbolValue_tooLarge);
+    if (maxSymbolValue > HUF_SYMBOLVALUE_MAX)
+      return ERROR(maxSymbolValue_tooLarge);
     memset(huffNode0, 0, sizeof(huffNodeTable));
 
     /* sort, decreasing order */
-    HUF_sort(huffNode, count, maxSymbolValue);
+    HUF_sort(huffNode, count, maxSymbolValue, wksp_tables->rankPosition);
 
     /* init for parents */
-    nonNullRank = maxSymbolValue;
+    nonNullRank = (int)maxSymbolValue;
     while(huffNode[nonNullRank].count == 0) nonNullRank--;
     lowS = nonNullRank; nodeRoot = nodeNb + lowS - 1; lowN = nodeNb;
     huffNode[nodeNb].count = huffNode[lowS].count + huffNode[lowS-1].count;
-    huffNode[lowS].parent = huffNode[lowS-1].parent = nodeNb;
+    huffNode[lowS].parent = huffNode[lowS-1].parent = (U16)nodeNb;
     nodeNb++; lowS-=2;
     for (n=nodeNb; n<=nodeRoot; n++) huffNode[n].count = (U32)(1U<<30);
     huffNode0[0].count = (U32)(1U<<31);  /* fake entry, strong barrier */
 
     /* create parents */
     while (nodeNb <= nodeRoot) {
-        U32 n1 = (huffNode[lowS].count < huffNode[lowN].count) ? lowS-- : lowN++;
-        U32 n2 = (huffNode[lowS].count < huffNode[lowN].count) ? lowS-- : lowN++;
+        int const n1 = (huffNode[lowS].count < huffNode[lowN].count) ? lowS-- : lowN++;
+        int const n2 = (huffNode[lowS].count < huffNode[lowN].count) ? lowS-- : lowN++;
         huffNode[nodeNb].count = huffNode[n1].count + huffNode[n2].count;
-        huffNode[n1].parent = huffNode[n2].parent = nodeNb;
+        huffNode[n1].parent = huffNode[n2].parent = (U16)nodeNb;
         nodeNb++;
     }
 
@@ -393,24 +384,25 @@ size_t HUF_buildCTable_wksp (HUF_CElt* tree, const unsigned* count, U32 maxSymbo
         huffNode[n].nbBits = huffNode[ huffNode[n].parent ].nbBits + 1;
 
     /* enforce maxTableLog */
-    maxNbBits = HUF_setMaxHeight(huffNode, nonNullRank, maxNbBits);
+    maxNbBits = HUF_setMaxHeight(huffNode, (U32)nonNullRank, maxNbBits);
 
     /* fill result into tree (val, nbBits) */
     {   U16 nbPerRank[HUF_TABLELOG_MAX+1] = {0};
         U16 valPerRank[HUF_TABLELOG_MAX+1] = {0};
+        int const alphabetSize = (int)(maxSymbolValue + 1);
         if (maxNbBits > HUF_TABLELOG_MAX) return ERROR(GENERIC);   /* check fit into table */
         for (n=0; n<=nonNullRank; n++)
             nbPerRank[huffNode[n].nbBits]++;
         /* determine stating value per rank */
         {   U16 min = 0;
-            for (n=maxNbBits; n>0; n--) {
+            for (n=(int)maxNbBits; n>0; n--) {
                 valPerRank[n] = min;      /* get starting value within each rank */
                 min += nbPerRank[n];
                 min >>= 1;
         }   }
-        for (n=0; n<=maxSymbolValue; n++)
+        for (n=0; n<alphabetSize; n++)
             tree[huffNode[n].byte].nbBits = huffNode[n].nbBits;   /* push nbBits per symbol, symbol order */
-        for (n=0; n<=maxSymbolValue; n++)
+        for (n=0; n<alphabetSize; n++)
             tree[n].val = valPerRank[tree[n].nbBits]++;   /* assign value within rank, symbol order */
     }
 
@@ -423,11 +415,11 @@ size_t HUF_buildCTable_wksp (HUF_CElt* tree, const unsigned* count, U32 maxSymbo
  */
 size_t HUF_buildCTable (HUF_CElt* tree, const unsigned* count, unsigned maxSymbolValue, unsigned maxNbBits)
 {
-    huffNodeTable nodeTable;
-    return HUF_buildCTable_wksp(tree, count, maxSymbolValue, maxNbBits, nodeTable, sizeof(nodeTable));
+    HUF_buildCTable_wksp_tables workspace;
+    return HUF_buildCTable_wksp(tree, count, maxSymbolValue, maxNbBits, &workspace, sizeof(workspace));
 }
 
-static size_t HUF_estimateCompressedSize(HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue)
+size_t HUF_estimateCompressedSize(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue)
 {
     size_t nbBits = 0;
     int s;
@@ -437,7 +429,7 @@ static size_t HUF_estimateCompressedSize(HUF_CElt* CTable, const unsigned* count
     return nbBits >> 3;
 }
 
-static int HUF_validateCTable(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue) {
+int HUF_validateCTable(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue) {
   int bad = 0;
   int s;
   for (s = 0; s <= (int)maxSymbolValue; ++s) {
@@ -476,7 +468,7 @@ HUF_compress1X_usingCTable_internal_body(void* dst, size_t dstSize,
 
     /* init */
     if (dstSize < 8) return 0;   /* not enough space to compress */
-    { size_t const initErr = BIT_initCStream(&bitC, op, oend-op);
+    { size_t const initErr = BIT_initCStream(&bitC, op, (size_t)(oend-op));
       if (HUF_isError(initErr)) return 0; }
 
     n = srcSize & ~3;  /* join to mod 4 */
@@ -573,7 +565,8 @@ HUF_compress4X_usingCTable_internal(void* dst, size_t dstSize,
     if (srcSize < 12) return 0;   /* no saving possible : too small input */
     op += 6;   /* jumpTable */
 
-    {   CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, oend-op, ip, segmentSize, CTable, bmi2) );
+    assert(op <= oend);
+    {   CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, (size_t)(oend-op), ip, segmentSize, CTable, bmi2) );
         if (cSize==0) return 0;
         assert(cSize <= 65535);
         MEM_writeLE16(ostart, (U16)cSize);
@@ -581,7 +574,8 @@ HUF_compress4X_usingCTable_internal(void* dst, size_t dstSize,
     }
 
     ip += segmentSize;
-    {   CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, oend-op, ip, segmentSize, CTable, bmi2) );
+    assert(op <= oend);
+    {   CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, (size_t)(oend-op), ip, segmentSize, CTable, bmi2) );
         if (cSize==0) return 0;
         assert(cSize <= 65535);
         MEM_writeLE16(ostart+2, (U16)cSize);
@@ -589,7 +583,8 @@ HUF_compress4X_usingCTable_internal(void* dst, size_t dstSize,
     }
 
     ip += segmentSize;
-    {   CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, oend-op, ip, segmentSize, CTable, bmi2) );
+    assert(op <= oend);
+    {   CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, (size_t)(oend-op), ip, segmentSize, CTable, bmi2) );
         if (cSize==0) return 0;
         assert(cSize <= 65535);
         MEM_writeLE16(ostart+4, (U16)cSize);
@@ -597,12 +592,14 @@ HUF_compress4X_usingCTable_internal(void* dst, size_t dstSize,
     }
 
     ip += segmentSize;
-    {   CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, oend-op, ip, iend-ip, CTable, bmi2) );
+    assert(op <= oend);
+    assert(ip <= iend);
+    {   CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, (size_t)(oend-op), ip, (size_t)(iend-ip), CTable, bmi2) );
         if (cSize==0) return 0;
         op += cSize;
     }
 
-    return op-ostart;
+    return (size_t)(op-ostart);
 }
 
 size_t HUF_compress4X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable)
@@ -618,20 +615,21 @@ static size_t HUF_compressCTable_internal(
                 HUF_nbStreams_e nbStreams, const HUF_CElt* CTable, const int bmi2)
 {
     size_t const cSize = (nbStreams==HUF_singleStream) ?
-                         HUF_compress1X_usingCTable_internal(op, oend - op, src, srcSize, CTable, bmi2) :
-                         HUF_compress4X_usingCTable_internal(op, oend - op, src, srcSize, CTable, bmi2);
+                         HUF_compress1X_usingCTable_internal(op, (size_t)(oend - op), src, srcSize, CTable, bmi2) :
+                         HUF_compress4X_usingCTable_internal(op, (size_t)(oend - op), src, srcSize, CTable, bmi2);
     if (HUF_isError(cSize)) { return cSize; }
     if (cSize==0) { return 0; }   /* uncompressible */
     op += cSize;
     /* check compressibility */
+    assert(op >= ostart);
     if ((size_t)(op-ostart) >= srcSize-1) { return 0; }
-    return op-ostart;
+    return (size_t)(op-ostart);
 }
 
 typedef struct {
     unsigned count[HUF_SYMBOLVALUE_MAX + 1];
     HUF_CElt CTable[HUF_SYMBOLVALUE_MAX + 1];
-    huffNodeTable nodeTable;
+    HUF_buildCTable_wksp_tables buildCTable_wksp;
 } HUF_compress_tables_t;
 
 /* HUF_compress_internal() :
@@ -650,6 +648,8 @@ HUF_compress_internal (void* dst, size_t dstSize,
     BYTE* const oend = ostart + dstSize;
     BYTE* op = ostart;
 
+    HUF_STATIC_ASSERT(sizeof(*table) <= HUF_WORKSPACE_SIZE);
+
     /* checks & inits */
     if (((size_t)workSpace & 3) != 0) return ERROR(GENERIC);  /* must be aligned on 4-bytes boundaries */
     if (wkspSize < HUF_WORKSPACE_SIZE) return ERROR(workSpace_tooSmall);
@@ -691,7 +691,7 @@ HUF_compress_internal (void* dst, size_t dstSize,
     huffLog = HUF_optimalTableLog(huffLog, srcSize, maxSymbolValue);
     {   size_t const maxBits = HUF_buildCTable_wksp(table->CTable, table->count,
                                             maxSymbolValue, huffLog,
-                                            table->nodeTable, sizeof(table->nodeTable));
+                                            &table->buildCTable_wksp, sizeof(table->buildCTable_wksp));
         CHECK_F(maxBits);
         huffLog = (U32)maxBits;
         /* Zero unused symbols in CTable, so we can check it for validity */
diff --git a/lib/compress/zstd_compress.c b/lib/compress/zstd_compress.c
index 35346b92cb1a..3f963b1cfff8 100644
--- a/lib/compress/zstd_compress.c
+++ b/lib/compress/zstd_compress.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -13,13 +13,13 @@
 ***************************************/
 #include <limits.h>         /* INT_MAX */
 #include <string.h>         /* memset */
-#include "cpu.h"
-#include "mem.h"
+#include "../common/cpu.h"
+#include "../common/mem.h"
 #include "hist.h"           /* HIST_countFast_wksp */
 #define FSE_STATIC_LINKING_ONLY   /* FSE_encodeSymbol */
-#include "fse.h"
+#include "../common/fse.h"
 #define HUF_STATIC_LINKING_ONLY
-#include "huf.h"
+#include "../common/huf.h"
 #include "zstd_compress_internal.h"
 #include "zstd_compress_sequences.h"
 #include "zstd_compress_literals.h"
@@ -28,11 +28,19 @@
 #include "zstd_lazy.h"
 #include "zstd_opt.h"
 #include "zstd_ldm.h"
+#include "zstd_compress_superblock.h"
 
 
 /*-*************************************
 *  Helper functions
 ***************************************/
+/* ZSTD_compressBound()
+ * Note that the result from this function is only compatible with the "normal"
+ * full-block strategy.
+ * When there are a lot of small blocks due to frequent flush in streaming mode
+ * the overhead of headers can make the compressed data to be larger than the
+ * return value of ZSTD_compressBound().
+ */
 size_t ZSTD_compressBound(size_t srcSize) {
     return ZSTD_COMPRESSBOUND(srcSize);
 }
@@ -82,7 +90,7 @@ ZSTD_CCtx* ZSTD_createCCtx_advanced(ZSTD_customMem customMem)
     }
 }
 
-ZSTD_CCtx* ZSTD_initStaticCCtx(void *workspace, size_t workspaceSize)
+ZSTD_CCtx* ZSTD_initStaticCCtx(void* workspace, size_t workspaceSize)
 {
     ZSTD_cwksp ws;
     ZSTD_CCtx* cctx;
@@ -91,9 +99,8 @@ ZSTD_CCtx* ZSTD_initStaticCCtx(void *workspace, size_t workspaceSize)
     ZSTD_cwksp_init(&ws, workspace, workspaceSize);
 
     cctx = (ZSTD_CCtx*)ZSTD_cwksp_reserve_object(&ws, sizeof(ZSTD_CCtx));
-    if (cctx == NULL) {
-        return NULL;
-    }
+    if (cctx == NULL) return NULL;
+
     memset(cctx, 0, sizeof(ZSTD_CCtx));
     ZSTD_cwksp_move(&cctx->workspace, &ws);
     cctx->staticSize = workspaceSize;
@@ -102,8 +109,7 @@ ZSTD_CCtx* ZSTD_initStaticCCtx(void *workspace, size_t workspaceSize)
     if (!ZSTD_cwksp_check_available(&cctx->workspace, HUF_WORKSPACE_SIZE + 2 * sizeof(ZSTD_compressedBlockState_t))) return NULL;
     cctx->blockState.prevCBlock = (ZSTD_compressedBlockState_t*)ZSTD_cwksp_reserve_object(&cctx->workspace, sizeof(ZSTD_compressedBlockState_t));
     cctx->blockState.nextCBlock = (ZSTD_compressedBlockState_t*)ZSTD_cwksp_reserve_object(&cctx->workspace, sizeof(ZSTD_compressedBlockState_t));
-    cctx->entropyWorkspace = (U32*)ZSTD_cwksp_reserve_object(
-        &cctx->workspace, HUF_WORKSPACE_SIZE);
+    cctx->entropyWorkspace = (U32*)ZSTD_cwksp_reserve_object(&cctx->workspace, HUF_WORKSPACE_SIZE);
     cctx->bmi2 = ZSTD_cpuid_bmi2(ZSTD_cpuid());
     return cctx;
 }
@@ -227,7 +233,7 @@ size_t ZSTD_CCtxParams_reset(ZSTD_CCtx_params* params)
 }
 
 size_t ZSTD_CCtxParams_init(ZSTD_CCtx_params* cctxParams, int compressionLevel) {
-    RETURN_ERROR_IF(!cctxParams, GENERIC);
+    RETURN_ERROR_IF(!cctxParams, GENERIC, "NULL pointer!");
     memset(cctxParams, 0, sizeof(*cctxParams));
     cctxParams->compressionLevel = compressionLevel;
     cctxParams->fParams.contentSizeFlag = 1;
@@ -236,8 +242,8 @@ size_t ZSTD_CCtxParams_init(ZSTD_CCtx_params* cctxParams, int compressionLevel)
 
 size_t ZSTD_CCtxParams_init_advanced(ZSTD_CCtx_params* cctxParams, ZSTD_parameters params)
 {
-    RETURN_ERROR_IF(!cctxParams, GENERIC);
-    FORWARD_IF_ERROR( ZSTD_checkCParams(params.cParams) );
+    RETURN_ERROR_IF(!cctxParams, GENERIC, "NULL pointer!");
+    FORWARD_IF_ERROR( ZSTD_checkCParams(params.cParams) , "");
     memset(cctxParams, 0, sizeof(*cctxParams));
     assert(!ZSTD_checkCParams(params.cParams));
     cctxParams->cParams = params.cParams;
@@ -249,12 +255,12 @@ size_t ZSTD_CCtxParams_init_advanced(ZSTD_CCtx_params* cctxParams, ZSTD_paramete
 /* ZSTD_assignParamsToCCtxParams() :
  * params is presumed valid at this stage */
 static ZSTD_CCtx_params ZSTD_assignParamsToCCtxParams(
-        const ZSTD_CCtx_params* cctxParams, ZSTD_parameters params)
+        const ZSTD_CCtx_params* cctxParams, const ZSTD_parameters* params)
 {
     ZSTD_CCtx_params ret = *cctxParams;
-    assert(!ZSTD_checkCParams(params.cParams));
-    ret.cParams = params.cParams;
-    ret.fParams = params.fParams;
+    assert(!ZSTD_checkCParams(params->cParams));
+    ret.cParams = params->cParams;
+    ret.fParams = params->fParams;
     ret.compressionLevel = ZSTD_CLEVEL_DEFAULT;   /* should not matter, as all cParams are presumed properly defined */
     return ret;
 }
@@ -339,8 +345,13 @@ ZSTD_bounds ZSTD_cParam_getBounds(ZSTD_cParameter param)
         return bounds;
 
     case ZSTD_c_overlapLog:
+#ifdef ZSTD_MULTITHREAD
         bounds.lowerBound = ZSTD_OVERLAPLOG_MIN;
         bounds.upperBound = ZSTD_OVERLAPLOG_MAX;
+#else
+        bounds.lowerBound = 0;
+        bounds.upperBound = 0;
+#endif
         return bounds;
 
     case ZSTD_c_enableLongDistanceMatching:
@@ -408,9 +419,8 @@ ZSTD_bounds ZSTD_cParam_getBounds(ZSTD_cParameter param)
         return bounds;
 
     default:
-        {   ZSTD_bounds const boundError = { ERROR(parameter_unsupported), 0, 0 };
-            return boundError;
-        }
+        bounds.error = ERROR(parameter_unsupported);
+        return bounds;
     }
 }
 
@@ -428,7 +438,7 @@ static size_t ZSTD_cParam_clampBounds(ZSTD_cParameter cParam, int* value)
 
 #define BOUNDCHECK(cParam, val) { \
     RETURN_ERROR_IF(!ZSTD_cParam_withinBounds(cParam,val), \
-                    parameter_outOfBound); \
+                    parameter_outOfBound, "Param out of bounds"); \
 }
 
 
@@ -476,7 +486,7 @@ size_t ZSTD_CCtx_setParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param, int value)
         if (ZSTD_isUpdateAuthorized(param)) {
             cctx->cParamsChanged = 1;
         } else {
-            RETURN_ERROR(stage_wrong);
+            RETURN_ERROR(stage_wrong, "can only set params in ctx init stage");
     }   }
 
     switch(param)
@@ -513,7 +523,7 @@ size_t ZSTD_CCtx_setParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param, int value)
     case ZSTD_c_srcSizeHint:
         break;
 
-    default: RETURN_ERROR(parameter_unsupported);
+    default: RETURN_ERROR(parameter_unsupported, "unknown parameter");
     }
     return ZSTD_CCtxParams_setParameter(&cctx->requestedParams, param, value);
 }
@@ -530,7 +540,7 @@ size_t ZSTD_CCtxParams_setParameter(ZSTD_CCtx_params* CCtxParams,
         return (size_t)CCtxParams->format;
 
     case ZSTD_c_compressionLevel : {
-        FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(param, &value));
+        FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(param, &value), "");
         if (value) {  /* 0 : does not change current level */
             CCtxParams->compressionLevel = value;
         }
@@ -618,7 +628,7 @@ size_t ZSTD_CCtxParams_setParameter(ZSTD_CCtx_params* CCtxParams,
         RETURN_ERROR_IF(value!=0, parameter_unsupported, "not compiled with multithreading");
         return 0;
 #else
-        FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(param, &value));
+        FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(param, &value), "");
         CCtxParams->nbWorkers = value;
         return CCtxParams->nbWorkers;
 #endif
@@ -631,7 +641,7 @@ size_t ZSTD_CCtxParams_setParameter(ZSTD_CCtx_params* CCtxParams,
         /* Adjust to the minimum non-default value. */
         if (value != 0 && value < ZSTDMT_JOBSIZE_MIN)
             value = ZSTDMT_JOBSIZE_MIN;
-        FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(param, &value));
+        FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(param, &value), "");
         assert(value >= 0);
         CCtxParams->jobSize = value;
         return CCtxParams->jobSize;
@@ -642,7 +652,7 @@ size_t ZSTD_CCtxParams_setParameter(ZSTD_CCtx_params* CCtxParams,
         RETURN_ERROR_IF(value!=0, parameter_unsupported, "not compiled with multithreading");
         return 0;
 #else
-        FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(ZSTD_c_overlapLog, &value));
+        FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(ZSTD_c_overlapLog, &value), "");
         CCtxParams->overlapLog = value;
         return CCtxParams->overlapLog;
 #endif
@@ -652,7 +662,7 @@ size_t ZSTD_CCtxParams_setParameter(ZSTD_CCtx_params* CCtxParams,
         RETURN_ERROR_IF(value!=0, parameter_unsupported, "not compiled with multithreading");
         return 0;
 #else
-        FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(ZSTD_c_overlapLog, &value));
+        FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(ZSTD_c_overlapLog, &value), "");
         CCtxParams->rsyncable = value;
         return CCtxParams->rsyncable;
 #endif
@@ -681,7 +691,7 @@ size_t ZSTD_CCtxParams_setParameter(ZSTD_CCtx_params* CCtxParams,
 
     case ZSTD_c_ldmHashRateLog :
         RETURN_ERROR_IF(value > ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN,
-                        parameter_outOfBound);
+                        parameter_outOfBound, "Param out of bounds!");
         CCtxParams->ldmParams.hashRateLog = value;
         return CCtxParams->ldmParams.hashRateLog;
 
@@ -821,8 +831,11 @@ size_t ZSTD_CCtx_setParametersUsingCCtxParams(
         ZSTD_CCtx* cctx, const ZSTD_CCtx_params* params)
 {
     DEBUGLOG(4, "ZSTD_CCtx_setParametersUsingCCtxParams");
-    RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong);
-    RETURN_ERROR_IF(cctx->cdict, stage_wrong);
+    RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong,
+                    "The context is in the wrong stage!");
+    RETURN_ERROR_IF(cctx->cdict, stage_wrong,
+                    "Can't override parameters with cdict attached (some must "
+                    "be inherited from the cdict).");
 
     cctx->requestedParams = *params;
     return 0;
@@ -831,7 +844,8 @@ size_t ZSTD_CCtx_setParametersUsingCCtxParams(
 ZSTDLIB_API size_t ZSTD_CCtx_setPledgedSrcSize(ZSTD_CCtx* cctx, unsigned long long pledgedSrcSize)
 {
     DEBUGLOG(4, "ZSTD_CCtx_setPledgedSrcSize to %u bytes", (U32)pledgedSrcSize);
-    RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong);
+    RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong,
+                    "Can't set pledgedSrcSize when not in init stage.");
     cctx->pledgedSrcSizePlusOne = pledgedSrcSize+1;
     return 0;
 }
@@ -845,7 +859,7 @@ static size_t ZSTD_initLocalDict(ZSTD_CCtx* cctx)
 {
     ZSTD_localDict* const dl = &cctx->localDict;
     ZSTD_compressionParameters const cParams = ZSTD_getCParamsFromCCtxParams(
-            &cctx->requestedParams, 0, dl->dictSize);
+            &cctx->requestedParams, ZSTD_CONTENTSIZE_UNKNOWN, dl->dictSize);
     if (dl->dict == NULL) {
         /* No local dictionary. */
         assert(dl->dictBuffer == NULL);
@@ -869,7 +883,7 @@ static size_t ZSTD_initLocalDict(ZSTD_CCtx* cctx)
             dl->dictContentType,
             cParams,
             cctx->customMem);
-    RETURN_ERROR_IF(!dl->cdict, memory_allocation);
+    RETURN_ERROR_IF(!dl->cdict, memory_allocation, "ZSTD_createCDict_advanced failed");
     cctx->cdict = dl->cdict;
     return 0;
 }
@@ -878,7 +892,8 @@ size_t ZSTD_CCtx_loadDictionary_advanced(
         ZSTD_CCtx* cctx, const void* dict, size_t dictSize,
         ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType)
 {
-    RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong);
+    RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong,
+                    "Can't load a dictionary when ctx is not in init stage.");
     RETURN_ERROR_IF(cctx->staticSize, memory_allocation,
                     "no malloc for static CCtx");
     DEBUGLOG(4, "ZSTD_CCtx_loadDictionary_advanced (size: %u)", (U32)dictSize);
@@ -889,7 +904,7 @@ size_t ZSTD_CCtx_loadDictionary_advanced(
         cctx->localDict.dict = dict;
     } else {
         void* dictBuffer = ZSTD_malloc(dictSize, cctx->customMem);
-        RETURN_ERROR_IF(!dictBuffer, memory_allocation);
+        RETURN_ERROR_IF(!dictBuffer, memory_allocation, "NULL pointer!");
         memcpy(dictBuffer, dict, dictSize);
         cctx->localDict.dictBuffer = dictBuffer;
         cctx->localDict.dict = dictBuffer;
@@ -915,7 +930,8 @@ ZSTDLIB_API size_t ZSTD_CCtx_loadDictionary(ZSTD_CCtx* cctx, const void* dict, s
 
 size_t ZSTD_CCtx_refCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict)
 {
-    RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong);
+    RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong,
+                    "Can't ref a dict when ctx not in init stage.");
     /* Free the existing local cdict (if any) to save memory. */
     ZSTD_clearAllDicts(cctx);
     cctx->cdict = cdict;
@@ -930,11 +946,14 @@ size_t ZSTD_CCtx_refPrefix(ZSTD_CCtx* cctx, const void* prefix, size_t prefixSiz
 size_t ZSTD_CCtx_refPrefix_advanced(
         ZSTD_CCtx* cctx, const void* prefix, size_t prefixSize, ZSTD_dictContentType_e dictContentType)
 {
-    RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong);
+    RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong,
+                    "Can't ref a prefix when ctx not in init stage.");
     ZSTD_clearAllDicts(cctx);
-    cctx->prefixDict.dict = prefix;
-    cctx->prefixDict.dictSize = prefixSize;
-    cctx->prefixDict.dictContentType = dictContentType;
+    if (prefix != NULL && prefixSize > 0) {
+        cctx->prefixDict.dict = prefix;
+        cctx->prefixDict.dictSize = prefixSize;
+        cctx->prefixDict.dictContentType = dictContentType;
+    }
     return 0;
 }
 
@@ -949,7 +968,8 @@ size_t ZSTD_CCtx_reset(ZSTD_CCtx* cctx, ZSTD_ResetDirective reset)
     }
     if ( (reset == ZSTD_reset_parameters)
       || (reset == ZSTD_reset_session_and_parameters) ) {
-        RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong);
+        RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong,
+                        "Can't reset parameters only when not in init stage.");
         ZSTD_clearAllDicts(cctx);
         return ZSTD_CCtxParams_reset(&cctx->requestedParams);
     }
@@ -996,7 +1016,7 @@ ZSTD_clampCParams(ZSTD_compressionParameters cParams)
 
 /** ZSTD_cycleLog() :
  *  condition for correct operation : hashLog > 1 */
-static U32 ZSTD_cycleLog(U32 hashLog, ZSTD_strategy strat)
+U32 ZSTD_cycleLog(U32 hashLog, ZSTD_strategy strat)
 {
     U32 const btScale = ((U32)strat >= (U32)ZSTD_btlazy2);
     return hashLog - btScale;
@@ -1006,7 +1026,7 @@ static U32 ZSTD_cycleLog(U32 hashLog, ZSTD_strategy strat)
  *  optimize `cPar` for a specified input (`srcSize` and `dictSize`).
  *  mostly downsize to reduce memory consumption and initialization latency.
  * `srcSize` can be ZSTD_CONTENTSIZE_UNKNOWN when not known.
- *  note : for the time being, `srcSize==0` means "unknown" too, for compatibility with older convention.
+ *  note : `srcSize==0` means 0!
  *  condition : cPar is presumed validated (can be checked using ZSTD_checkCParams()). */
 static ZSTD_compressionParameters
 ZSTD_adjustCParams_internal(ZSTD_compressionParameters cPar,
@@ -1017,10 +1037,8 @@ ZSTD_adjustCParams_internal(ZSTD_compressionParameters cPar,
     static const U64 maxWindowResize = 1ULL << (ZSTD_WINDOWLOG_MAX-1);
     assert(ZSTD_checkCParams(cPar)==0);
 
-    if (dictSize && (srcSize+1<2) /* ZSTD_CONTENTSIZE_UNKNOWN and 0 mean "unknown" */ )
-        srcSize = minSrcSize;  /* presumed small when there is a dictionary */
-    else if (srcSize == 0)
-        srcSize = ZSTD_CONTENTSIZE_UNKNOWN;  /* 0 == unknown : presumed large */
+    if (dictSize && srcSize == ZSTD_CONTENTSIZE_UNKNOWN)
+        srcSize = minSrcSize;
 
     /* resize windowLog if input is small enough, to use less memory */
     if ( (srcSize < maxWindowResize)
@@ -1049,9 +1067,13 @@ ZSTD_adjustCParams(ZSTD_compressionParameters cPar,
                    size_t dictSize)
 {
     cPar = ZSTD_clampCParams(cPar);   /* resulting cPar is necessarily valid (all parameters within range) */
+    if (srcSize == 0) srcSize = ZSTD_CONTENTSIZE_UNKNOWN;
     return ZSTD_adjustCParams_internal(cPar, srcSize, dictSize);
 }
 
+static ZSTD_compressionParameters ZSTD_getCParams_internal(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize);
+static ZSTD_parameters ZSTD_getParams_internal(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize);
+
 ZSTD_compressionParameters ZSTD_getCParamsFromCCtxParams(
         const ZSTD_CCtx_params* CCtxParams, U64 srcSizeHint, size_t dictSize)
 {
@@ -1059,7 +1081,7 @@ ZSTD_compressionParameters ZSTD_getCParamsFromCCtxParams(
     if (srcSizeHint == ZSTD_CONTENTSIZE_UNKNOWN && CCtxParams->srcSizeHint > 0) {
       srcSizeHint = CCtxParams->srcSizeHint;
     }
-    cParams = ZSTD_getCParams(CCtxParams->compressionLevel, srcSizeHint, dictSize);
+    cParams = ZSTD_getCParams_internal(CCtxParams->compressionLevel, srcSizeHint, dictSize);
     if (CCtxParams->ldmParams.enableLdm) cParams.windowLog = ZSTD_LDM_DEFAULT_WINDOW_LOG;
     if (CCtxParams->cParams.windowLog) cParams.windowLog = CCtxParams->cParams.windowLog;
     if (CCtxParams->cParams.hashLog) cParams.hashLog = CCtxParams->cParams.hashLog;
@@ -1069,6 +1091,7 @@ ZSTD_compressionParameters ZSTD_getCParamsFromCCtxParams(
     if (CCtxParams->cParams.targetLength) cParams.targetLength = CCtxParams->cParams.targetLength;
     if (CCtxParams->cParams.strategy) cParams.strategy = CCtxParams->cParams.strategy;
     assert(!ZSTD_checkCParams(cParams));
+    /* srcSizeHint == 0 means 0 */
     return ZSTD_adjustCParams_internal(cParams, srcSizeHint, dictSize);
 }
 
@@ -1104,7 +1127,7 @@ size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params)
 {
     RETURN_ERROR_IF(params->nbWorkers > 0, GENERIC, "Estimate CCtx size is supported for single-threaded compression only.");
     {   ZSTD_compressionParameters const cParams =
-                ZSTD_getCParamsFromCCtxParams(params, 0, 0);
+                ZSTD_getCParamsFromCCtxParams(params, ZSTD_CONTENTSIZE_UNKNOWN, 0);
         size_t const blockSize = MIN(ZSTD_BLOCKSIZE_MAX, (size_t)1 << cParams.windowLog);
         U32    const divider = (cParams.minMatch==3) ? 3 : 4;
         size_t const maxNbSeq = blockSize / divider;
@@ -1118,13 +1141,26 @@ size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params)
         size_t const ldmSpace = ZSTD_ldm_getTableSize(params->ldmParams);
         size_t const ldmSeqSpace = ZSTD_cwksp_alloc_size(ZSTD_ldm_getMaxNbSeq(params->ldmParams, blockSize) * sizeof(rawSeq));
 
-        size_t const neededSpace = entropySpace + blockStateSpace + tokenSpace +
-                                   matchStateSize + ldmSpace + ldmSeqSpace;
+        /* estimateCCtxSize is for one-shot compression. So no buffers should
+         * be needed. However, we still allocate two 0-sized buffers, which can
+         * take space under ASAN. */
+        size_t const bufferSpace = ZSTD_cwksp_alloc_size(0)
+                                 + ZSTD_cwksp_alloc_size(0);
+
         size_t const cctxSpace = ZSTD_cwksp_alloc_size(sizeof(ZSTD_CCtx));
 
-        DEBUGLOG(5, "sizeof(ZSTD_CCtx) : %u", (U32)cctxSpace);
+        size_t const neededSpace =
+            cctxSpace +
+            entropySpace +
+            blockStateSpace +
+            ldmSpace +
+            ldmSeqSpace +
+            matchStateSize +
+            tokenSpace +
+            bufferSpace;
+
         DEBUGLOG(5, "estimate workspace : %u", (U32)neededSpace);
-        return cctxSpace + neededSpace;
+        return neededSpace;
     }
 }
 
@@ -1136,7 +1172,7 @@ size_t ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams)
 
 static size_t ZSTD_estimateCCtxSize_internal(int compressionLevel)
 {
-    ZSTD_compressionParameters const cParams = ZSTD_getCParams(compressionLevel, 0, 0);
+    ZSTD_compressionParameters const cParams = ZSTD_getCParams_internal(compressionLevel, ZSTD_CONTENTSIZE_UNKNOWN, 0);
     return ZSTD_estimateCCtxSize_usingCParams(cParams);
 }
 
@@ -1155,7 +1191,7 @@ size_t ZSTD_estimateCStreamSize_usingCCtxParams(const ZSTD_CCtx_params* params)
 {
     RETURN_ERROR_IF(params->nbWorkers > 0, GENERIC, "Estimate CCtx size is supported for single-threaded compression only.");
     {   ZSTD_compressionParameters const cParams =
-                ZSTD_getCParamsFromCCtxParams(params, 0, 0);
+                ZSTD_getCParamsFromCCtxParams(params, ZSTD_CONTENTSIZE_UNKNOWN, 0);
         size_t const CCtxSize = ZSTD_estimateCCtxSize_usingCCtxParams(params);
         size_t const blockSize = MIN(ZSTD_BLOCKSIZE_MAX, (size_t)1 << cParams.windowLog);
         size_t const inBuffSize = ((size_t)1 << cParams.windowLog) + blockSize;
@@ -1175,7 +1211,7 @@ size_t ZSTD_estimateCStreamSize_usingCParams(ZSTD_compressionParameters cParams)
 
 static size_t ZSTD_estimateCStreamSize_internal(int compressionLevel)
 {
-    ZSTD_compressionParameters const cParams = ZSTD_getCParams(compressionLevel, 0, 0);
+    ZSTD_compressionParameters const cParams = ZSTD_getCParams_internal(compressionLevel, ZSTD_CONTENTSIZE_UNKNOWN, 0);
     return ZSTD_estimateCStreamSize_usingCParams(cParams);
 }
 
@@ -1243,7 +1279,7 @@ static void ZSTD_assertEqualCParams(ZSTD_compressionParameters cParams1,
     assert(cParams1.strategy     == cParams2.strategy);
 }
 
-static void ZSTD_reset_compressedBlockState(ZSTD_compressedBlockState_t* bs)
+void ZSTD_reset_compressedBlockState(ZSTD_compressedBlockState_t* bs)
 {
     int i;
     for (i = 0; i < ZSTD_REP_NUM; ++i)
@@ -1320,10 +1356,7 @@ ZSTD_reset_matchState(ZSTD_matchState_t* ms,
 
     DEBUGLOG(4, "reset indices : %u", forceResetIndex == ZSTDirp_reset);
     if (forceResetIndex == ZSTDirp_reset) {
-        memset(&ms->window, 0, sizeof(ms->window));
-        ms->window.dictLimit = 1;    /* start from 1, so that 1st position is valid */
-        ms->window.lowLimit = 1;     /* it ensures first and later CCtx usages compress the same */
-        ms->window.nextSrc = ms->window.base + 1;   /* see issue #1241 */
+        ZSTD_window_init(&ms->window);
         ZSTD_cwksp_mark_tables_dirty(ws);
     }
 
@@ -1416,13 +1449,13 @@ static size_t ZSTD_resetCCtx_internal(ZSTD_CCtx* zc,
         size_t const matchStateSize = ZSTD_sizeof_matchState(&params.cParams, /* forCCtx */ 1);
         size_t const maxNbLdmSeq = ZSTD_ldm_getMaxNbSeq(params.ldmParams, blockSize);
 
-        ZSTD_indexResetPolicy_e needsIndexReset = ZSTDirp_continue;
+        ZSTD_indexResetPolicy_e needsIndexReset = zc->initialized ? ZSTDirp_continue : ZSTDirp_reset;
 
         if (ZSTD_indexTooCloseToMax(zc->blockState.matchState.window)) {
             needsIndexReset = ZSTDirp_reset;
         }
 
-        ZSTD_cwksp_bump_oversized_duration(ws, 0);
+        if (!zc->staticSize) ZSTD_cwksp_bump_oversized_duration(ws, 0);
 
         /* Check if workspace is large enough, alloc a new one if needed */
         {   size_t const cctxSpace = zc->staticSize ? ZSTD_cwksp_alloc_size(sizeof(ZSTD_CCtx)) : 0;
@@ -1459,7 +1492,7 @@ static size_t ZSTD_resetCCtx_internal(ZSTD_CCtx* zc,
                 needsIndexReset = ZSTDirp_reset;
 
                 ZSTD_cwksp_free(ws, zc->customMem);
-                FORWARD_IF_ERROR(ZSTD_cwksp_create(ws, neededSpace, zc->customMem));
+                FORWARD_IF_ERROR(ZSTD_cwksp_create(ws, neededSpace, zc->customMem), "");
 
                 DEBUGLOG(5, "reserving object space");
                 /* Statically sized space.
@@ -1530,7 +1563,7 @@ static size_t ZSTD_resetCCtx_internal(ZSTD_CCtx* zc,
             &params.cParams,
             crp,
             needsIndexReset,
-            ZSTD_resetTarget_CCtx));
+            ZSTD_resetTarget_CCtx), "");
 
         /* ldm hash table */
         if (params.ldmParams.enableLdm) {
@@ -1541,11 +1574,13 @@ static size_t ZSTD_resetCCtx_internal(ZSTD_CCtx* zc,
             zc->ldmSequences = (rawSeq*)ZSTD_cwksp_reserve_aligned(ws, maxNbLdmSeq * sizeof(rawSeq));
             zc->maxNbLdmSequences = maxNbLdmSeq;
 
-            memset(&zc->ldmState.window, 0, sizeof(zc->ldmState.window));
+            ZSTD_window_init(&zc->ldmState.window);
             ZSTD_window_clear(&zc->ldmState.window);
+            zc->ldmState.loadedDictEnd = 0;
         }
 
         DEBUGLOG(3, "wksp: finished allocating, %zd bytes remain available", ZSTD_cwksp_available_space(ws));
+        zc->initialized = 1;
 
         return 0;
     }
@@ -1603,10 +1638,11 @@ ZSTD_resetCCtx_byAttachingCDict(ZSTD_CCtx* cctx,
         assert(windowLog != 0);
         /* Resize working context table params for input only, since the dict
          * has its own tables. */
+        /* pledgeSrcSize == 0 means 0! */
         params.cParams = ZSTD_adjustCParams_internal(*cdict_cParams, pledgedSrcSize, 0);
         params.cParams.windowLog = windowLog;
         FORWARD_IF_ERROR(ZSTD_resetCCtx_internal(cctx, params, pledgedSrcSize,
-                                                 ZSTDcrp_makeClean, zbuff));
+                                                 ZSTDcrp_makeClean, zbuff), "");
         assert(cctx->appliedParams.cParams.strategy == cdict_cParams->strategy);
     }
 
@@ -1655,7 +1691,7 @@ static size_t ZSTD_resetCCtx_byCopyingCDict(ZSTD_CCtx* cctx,
         params.cParams = *cdict_cParams;
         params.cParams.windowLog = windowLog;
         FORWARD_IF_ERROR(ZSTD_resetCCtx_internal(cctx, params, pledgedSrcSize,
-                                                 ZSTDcrp_leaveDirty, zbuff));
+                                                 ZSTDcrp_leaveDirty, zbuff), "");
         assert(cctx->appliedParams.cParams.strategy == cdict_cParams->strategy);
         assert(cctx->appliedParams.cParams.hashLog == cdict_cParams->hashLog);
         assert(cctx->appliedParams.cParams.chainLog == cdict_cParams->chainLog);
@@ -1736,7 +1772,8 @@ static size_t ZSTD_copyCCtx_internal(ZSTD_CCtx* dstCCtx,
                             ZSTD_buffered_policy_e zbuff)
 {
     DEBUGLOG(5, "ZSTD_copyCCtx_internal");
-    RETURN_ERROR_IF(srcCCtx->stage!=ZSTDcs_init, stage_wrong);
+    RETURN_ERROR_IF(srcCCtx->stage!=ZSTDcs_init, stage_wrong,
+                    "Can't copy a ctx that's not in init stage.");
 
     memcpy(&dstCCtx->customMem, &srcCCtx->customMem, sizeof(ZSTD_customMem));
     {   ZSTD_CCtx_params params = dstCCtx->requestedParams;
@@ -1889,16 +1926,6 @@ static void ZSTD_reduceIndex (ZSTD_matchState_t* ms, ZSTD_CCtx_params const* par
 
 /* See doc/zstd_compression_format.md for detailed format description */
 
-static size_t ZSTD_noCompressBlock (void* dst, size_t dstCapacity, const void* src, size_t srcSize, U32 lastBlock)
-{
-    U32 const cBlockHeader24 = lastBlock + (((U32)bt_raw)<<1) + (U32)(srcSize << 3);
-    RETURN_ERROR_IF(srcSize + ZSTD_blockHeaderSize > dstCapacity,
-                    dstSize_tooSmall);
-    MEM_writeLE24(dst, cBlockHeader24);
-    memcpy((BYTE*)dst + ZSTD_blockHeaderSize, src, srcSize);
-    return ZSTD_blockHeaderSize + srcSize;
-}
-
 void ZSTD_seqToCodes(const seqStore_t* seqStorePtr)
 {
     const seqDef* const sequences = seqStorePtr->sequencesStart;
@@ -1921,19 +1948,14 @@ void ZSTD_seqToCodes(const seqStore_t* seqStorePtr)
         mlCodeTable[seqStorePtr->longLengthPos] = MaxML;
 }
 
-static int ZSTD_disableLiteralsCompression(const ZSTD_CCtx_params* cctxParams)
+/* ZSTD_useTargetCBlockSize():
+ * Returns if target compressed block size param is being used.
+ * If used, compression will do best effort to make a compressed block size to be around targetCBlockSize.
+ * Returns 1 if true, 0 otherwise. */
+static int ZSTD_useTargetCBlockSize(const ZSTD_CCtx_params* cctxParams)
 {
-    switch (cctxParams->literalCompressionMode) {
-    case ZSTD_lcm_huffman:
-        return 0;
-    case ZSTD_lcm_uncompressed:
-        return 1;
-    default:
-        assert(0 /* impossible: pre-validated */);
-        /* fall-through */
-    case ZSTD_lcm_auto:
-        return (cctxParams->cParams.strategy == ZSTD_fast) && (cctxParams->cParams.targetLength > 0);
-    }
+    DEBUGLOG(5, "ZSTD_useTargetCBlockSize (targetCBlockSize=%zu)", cctxParams->targetCBlockSize);
+    return (cctxParams->targetCBlockSize != 0);
 }
 
 /* ZSTD_compressSequences_internal():
@@ -1979,14 +2001,14 @@ ZSTD_compressSequences_internal(seqStore_t* seqStorePtr,
                                     literals, litSize,
                                     entropyWorkspace, entropyWkspSize,
                                     bmi2);
-        FORWARD_IF_ERROR(cSize);
+        FORWARD_IF_ERROR(cSize, "ZSTD_compressLiterals failed");
         assert(cSize <= dstCapacity);
         op += cSize;
     }
 
     /* Sequences Header */
     RETURN_ERROR_IF((oend-op) < 3 /*max nbSeq Size*/ + 1 /*seqHead*/,
-                    dstSize_tooSmall);
+                    dstSize_tooSmall, "Can't fit seq hdr in output buf!");
     if (nbSeq < 128) {
         *op++ = (BYTE)nbSeq;
     } else if (nbSeq < LONGNBSEQ) {
@@ -2031,7 +2053,7 @@ ZSTD_compressSequences_internal(seqStore_t* seqStorePtr,
                 prevEntropy->fse.litlengthCTable,
                 sizeof(prevEntropy->fse.litlengthCTable),
                 entropyWorkspace, entropyWkspSize);
-            FORWARD_IF_ERROR(countSize);
+            FORWARD_IF_ERROR(countSize, "ZSTD_buildCTable for LitLens failed");
             if (LLtype == set_compressed)
                 lastNCount = op;
             op += countSize;
@@ -2059,7 +2081,7 @@ ZSTD_compressSequences_internal(seqStore_t* seqStorePtr,
                 prevEntropy->fse.offcodeCTable,
                 sizeof(prevEntropy->fse.offcodeCTable),
                 entropyWorkspace, entropyWkspSize);
-            FORWARD_IF_ERROR(countSize);
+            FORWARD_IF_ERROR(countSize, "ZSTD_buildCTable for Offsets failed");
             if (Offtype == set_compressed)
                 lastNCount = op;
             op += countSize;
@@ -2085,7 +2107,7 @@ ZSTD_compressSequences_internal(seqStore_t* seqStorePtr,
                 prevEntropy->fse.matchlengthCTable,
                 sizeof(prevEntropy->fse.matchlengthCTable),
                 entropyWorkspace, entropyWkspSize);
-            FORWARD_IF_ERROR(countSize);
+            FORWARD_IF_ERROR(countSize, "ZSTD_buildCTable for MatchLengths failed");
             if (MLtype == set_compressed)
                 lastNCount = op;
             op += countSize;
@@ -2101,7 +2123,7 @@ ZSTD_compressSequences_internal(seqStore_t* seqStorePtr,
                                         CTable_LitLength, llCodeTable,
                                         sequences, nbSeq,
                                         longOffsets, bmi2);
-        FORWARD_IF_ERROR(bitstreamSize);
+        FORWARD_IF_ERROR(bitstreamSize, "ZSTD_encodeSequences failed");
         op += bitstreamSize;
         assert(op <= oend);
         /* zstd versions <= 1.3.4 mistakenly report corruption when
@@ -2145,7 +2167,7 @@ ZSTD_compressSequences(seqStore_t* seqStorePtr,
      */
     if ((cSize == ERROR(dstSize_tooSmall)) & (srcSize <= dstCapacity))
         return 0;  /* block not compressed */
-    FORWARD_IF_ERROR(cSize);
+    FORWARD_IF_ERROR(cSize, "ZSTD_compressSequences_internal failed");
 
     /* Check compressibility */
     {   size_t const maxCSize = srcSize - ZSTD_minGain(srcSize, cctxParams->cParams.strategy);
@@ -2271,7 +2293,7 @@ static size_t ZSTD_buildSeqStore(ZSTD_CCtx* zc, const void* src, size_t srcSize)
             /* Updates ldmSeqStore.size */
             FORWARD_IF_ERROR(ZSTD_ldm_generateSequences(&zc->ldmState, &ldmSeqStore,
                                                &zc->appliedParams.ldmParams,
-                                               src, srcSize));
+                                               src, srcSize), "");
             /* Updates ldmSeqStore.pos */
             lastLLSize =
                 ZSTD_ldm_blockCompress(&ldmSeqStore,
@@ -2347,7 +2369,7 @@ size_t ZSTD_getSequences(ZSTD_CCtx* zc, ZSTD_Sequence* outSeqs,
     void* dst = ZSTD_malloc(dstCapacity, ZSTD_defaultCMem);
     SeqCollector seqCollector;
 
-    RETURN_ERROR_IF(dst == NULL, memory_allocation);
+    RETURN_ERROR_IF(dst == NULL, memory_allocation, "NULL pointer!");
 
     seqCollector.collectSequences = 1;
     seqCollector.seqStart = outSeqs;
@@ -2370,6 +2392,25 @@ static int ZSTD_isRLE(const BYTE *ip, size_t length) {
     return 1;
 }
 
+/* Returns true if the given block may be RLE.
+ * This is just a heuristic based on the compressibility.
+ * It may return both false positives and false negatives.
+ */
+static int ZSTD_maybeRLE(seqStore_t const* seqStore)
+{
+    size_t const nbSeqs = (size_t)(seqStore->sequences - seqStore->sequencesStart);
+    size_t const nbLits = (size_t)(seqStore->lit - seqStore->litStart);
+
+    return nbSeqs < 4 && nbLits < 10;
+}
+
+static void ZSTD_confirmRepcodesAndEntropyTables(ZSTD_CCtx* zc)
+{
+    ZSTD_compressedBlockState_t* const tmp = zc->blockState.prevCBlock;
+    zc->blockState.prevCBlock = zc->blockState.nextCBlock;
+    zc->blockState.nextCBlock = tmp;
+}
+
 static size_t ZSTD_compressBlock_internal(ZSTD_CCtx* zc,
                                         void* dst, size_t dstCapacity,
                                         const void* src, size_t srcSize, U32 frame)
@@ -2387,7 +2428,7 @@ static size_t ZSTD_compressBlock_internal(ZSTD_CCtx* zc,
                 (unsigned)zc->blockState.matchState.nextToUpdate);
 
     {   const size_t bss = ZSTD_buildSeqStore(zc, src, srcSize);
-        FORWARD_IF_ERROR(bss);
+        FORWARD_IF_ERROR(bss, "ZSTD_buildSeqStore failed");
         if (bss == ZSTDbss_noCompress) { cSize = 0; goto out; }
     }
 
@@ -2420,10 +2461,7 @@ static size_t ZSTD_compressBlock_internal(ZSTD_CCtx* zc,
 
 out:
     if (!ZSTD_isError(cSize) && cSize > 1) {
-        /* confirm repcodes and entropy tables when emitting a compressed block */
-        ZSTD_compressedBlockState_t* const tmp = zc->blockState.prevCBlock;
-        zc->blockState.prevCBlock = zc->blockState.nextCBlock;
-        zc->blockState.nextCBlock = tmp;
+        ZSTD_confirmRepcodesAndEntropyTables(zc);
     }
     /* We check that dictionaries have offset codes available for the first
      * block. After the first block, the offcode table might not have large
@@ -2435,6 +2473,80 @@ out:
     return cSize;
 }
 
+static size_t ZSTD_compressBlock_targetCBlockSize_body(ZSTD_CCtx* zc,
+                               void* dst, size_t dstCapacity,
+                               const void* src, size_t srcSize,
+                               const size_t bss, U32 lastBlock)
+{
+    DEBUGLOG(6, "Attempting ZSTD_compressSuperBlock()");
+    if (bss == ZSTDbss_compress) {
+        if (/* We don't want to emit our first block as a RLE even if it qualifies because
+            * doing so will cause the decoder (cli only) to throw a "should consume all input error."
+            * This is only an issue for zstd <= v1.4.3
+            */
+            !zc->isFirstBlock &&
+            ZSTD_maybeRLE(&zc->seqStore) &&
+            ZSTD_isRLE((BYTE const*)src, srcSize))
+        {
+            return ZSTD_rleCompressBlock(dst, dstCapacity, *(BYTE const*)src, srcSize, lastBlock);
+        }
+        /* Attempt superblock compression.
+         *
+         * Note that compressed size of ZSTD_compressSuperBlock() is not bound by the
+         * standard ZSTD_compressBound(). This is a problem, because even if we have
+         * space now, taking an extra byte now could cause us to run out of space later
+         * and violate ZSTD_compressBound().
+         *
+         * Define blockBound(blockSize) = blockSize + ZSTD_blockHeaderSize.
+         *
+         * In order to respect ZSTD_compressBound() we must attempt to emit a raw
+         * uncompressed block in these cases:
+         *   * cSize == 0: Return code for an uncompressed block.
+         *   * cSize == dstSize_tooSmall: We may have expanded beyond blockBound(srcSize).
+         *     ZSTD_noCompressBlock() will return dstSize_tooSmall if we are really out of
+         *     output space.
+         *   * cSize >= blockBound(srcSize): We have expanded the block too much so
+         *     emit an uncompressed block.
+         */
+        {
+            size_t const cSize = ZSTD_compressSuperBlock(zc, dst, dstCapacity, src, srcSize, lastBlock);
+            if (cSize != ERROR(dstSize_tooSmall)) {
+                size_t const maxCSize = srcSize - ZSTD_minGain(srcSize, zc->appliedParams.cParams.strategy);
+                FORWARD_IF_ERROR(cSize, "ZSTD_compressSuperBlock failed");
+                if (cSize != 0 && cSize < maxCSize + ZSTD_blockHeaderSize) {
+                    ZSTD_confirmRepcodesAndEntropyTables(zc);
+                    return cSize;
+                }
+            }
+        }
+    }
+
+    DEBUGLOG(6, "Resorting to ZSTD_noCompressBlock()");
+    /* Superblock compression failed, attempt to emit a single no compress block.
+     * The decoder will be able to stream this block since it is uncompressed.
+     */
+    return ZSTD_noCompressBlock(dst, dstCapacity, src, srcSize, lastBlock);
+}
+
+static size_t ZSTD_compressBlock_targetCBlockSize(ZSTD_CCtx* zc,
+                               void* dst, size_t dstCapacity,
+                               const void* src, size_t srcSize,
+                               U32 lastBlock)
+{
+    size_t cSize = 0;
+    const size_t bss = ZSTD_buildSeqStore(zc, src, srcSize);
+    DEBUGLOG(5, "ZSTD_compressBlock_targetCBlockSize (dstCapacity=%u, dictLimit=%u, nextToUpdate=%u, srcSize=%zu)",
+                (unsigned)dstCapacity, (unsigned)zc->blockState.matchState.window.dictLimit, (unsigned)zc->blockState.matchState.nextToUpdate, srcSize);
+    FORWARD_IF_ERROR(bss, "ZSTD_buildSeqStore failed");
+
+    cSize = ZSTD_compressBlock_targetCBlockSize_body(zc, dst, dstCapacity, src, srcSize, bss, lastBlock);
+    FORWARD_IF_ERROR(cSize, "ZSTD_compressBlock_targetCBlockSize_body failed");
+
+    if (zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode == FSE_repeat_valid)
+        zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode = FSE_repeat_check;
+
+    return cSize;
+}
 
 static void ZSTD_overflowCorrectIfNeeded(ZSTD_matchState_t* ms,
                                          ZSTD_cwksp* ws,
@@ -2478,6 +2590,7 @@ static size_t ZSTD_compress_frameChunk (ZSTD_CCtx* cctx,
     BYTE* const ostart = (BYTE*)dst;
     BYTE* op = ostart;
     U32 const maxDist = (U32)1 << cctx->appliedParams.cParams.windowLog;
+
     assert(cctx->appliedParams.cParams.windowLog <= ZSTD_WINDOWLOG_MAX);
 
     DEBUGLOG(5, "ZSTD_compress_frameChunk (blockSize=%u)", (unsigned)blockSize);
@@ -2500,21 +2613,31 @@ static size_t ZSTD_compress_frameChunk (ZSTD_CCtx* cctx,
         /* Ensure hash/chain table insertion resumes no sooner than lowlimit */
         if (ms->nextToUpdate < ms->window.lowLimit) ms->nextToUpdate = ms->window.lowLimit;
 
-        {   size_t cSize = ZSTD_compressBlock_internal(cctx,
-                                op+ZSTD_blockHeaderSize, dstCapacity-ZSTD_blockHeaderSize,
-                                ip, blockSize, 1 /* frame */);
-            FORWARD_IF_ERROR(cSize);
-            if (cSize == 0) {  /* block is not compressible */
-                cSize = ZSTD_noCompressBlock(op, dstCapacity, ip, blockSize, lastBlock);
-                FORWARD_IF_ERROR(cSize);
+        {   size_t cSize;
+            if (ZSTD_useTargetCBlockSize(&cctx->appliedParams)) {
+                cSize = ZSTD_compressBlock_targetCBlockSize(cctx, op, dstCapacity, ip, blockSize, lastBlock);
+                FORWARD_IF_ERROR(cSize, "ZSTD_compressBlock_targetCBlockSize failed");
+                assert(cSize > 0);
+                assert(cSize <= blockSize + ZSTD_blockHeaderSize);
             } else {
-                const U32 cBlockHeader = cSize == 1 ?
-                    lastBlock + (((U32)bt_rle)<<1) + (U32)(blockSize << 3) :
-                    lastBlock + (((U32)bt_compressed)<<1) + (U32)(cSize << 3);
-                MEM_writeLE24(op, cBlockHeader);
-                cSize += ZSTD_blockHeaderSize;
+                cSize = ZSTD_compressBlock_internal(cctx,
+                                        op+ZSTD_blockHeaderSize, dstCapacity-ZSTD_blockHeaderSize,
+                                        ip, blockSize, 1 /* frame */);
+                FORWARD_IF_ERROR(cSize, "ZSTD_compressBlock_internal failed");
+
+                if (cSize == 0) {  /* block is not compressible */
+                    cSize = ZSTD_noCompressBlock(op, dstCapacity, ip, blockSize, lastBlock);
+                    FORWARD_IF_ERROR(cSize, "ZSTD_noCompressBlock failed");
+                } else {
+                    U32 const cBlockHeader = cSize == 1 ?
+                        lastBlock + (((U32)bt_rle)<<1) + (U32)(blockSize << 3) :
+                        lastBlock + (((U32)bt_compressed)<<1) + (U32)(cSize << 3);
+                    MEM_writeLE24(op, cBlockHeader);
+                    cSize += ZSTD_blockHeaderSize;
+                }
             }
 
+
             ip += blockSize;
             assert(remaining >= blockSize);
             remaining -= blockSize;
@@ -2546,7 +2669,8 @@ static size_t ZSTD_writeFrameHeader(void* dst, size_t dstCapacity,
     size_t pos=0;
 
     assert(!(params->fParams.contentSizeFlag && pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN));
-    RETURN_ERROR_IF(dstCapacity < ZSTD_FRAMEHEADERSIZE_MAX, dstSize_tooSmall);
+    RETURN_ERROR_IF(dstCapacity < ZSTD_FRAMEHEADERSIZE_MAX, dstSize_tooSmall,
+                    "dst buf is too small to fit worst-case frame header size.");
     DEBUGLOG(4, "ZSTD_writeFrameHeader : dictIDFlag : %u ; dictID : %u ; dictIDSizeCode : %u",
                 !params->fParams.noDictIDFlag, (unsigned)dictID, (unsigned)dictIDSizeCode);
 
@@ -2582,7 +2706,8 @@ static size_t ZSTD_writeFrameHeader(void* dst, size_t dstCapacity,
  */
 size_t ZSTD_writeLastEmptyBlock(void* dst, size_t dstCapacity)
 {
-    RETURN_ERROR_IF(dstCapacity < ZSTD_blockHeaderSize, dstSize_tooSmall);
+    RETURN_ERROR_IF(dstCapacity < ZSTD_blockHeaderSize, dstSize_tooSmall,
+                    "dst buf is too small to write frame trailer empty block.");
     {   U32 const cBlockHeader24 = 1 /*lastBlock*/ + (((U32)bt_raw)<<1);  /* 0 size */
         MEM_writeLE24(dst, cBlockHeader24);
         return ZSTD_blockHeaderSize;
@@ -2591,9 +2716,11 @@ size_t ZSTD_writeLastEmptyBlock(void* dst, size_t dstCapacity)
 
 size_t ZSTD_referenceExternalSequences(ZSTD_CCtx* cctx, rawSeq* seq, size_t nbSeq)
 {
-    RETURN_ERROR_IF(cctx->stage != ZSTDcs_init, stage_wrong);
+    RETURN_ERROR_IF(cctx->stage != ZSTDcs_init, stage_wrong,
+                    "wrong cctx stage");
     RETURN_ERROR_IF(cctx->appliedParams.ldmParams.enableLdm,
-                    parameter_unsupported);
+                    parameter_unsupported,
+                    "incompatible with ldm");
     cctx->externSeqStore.seq = seq;
     cctx->externSeqStore.size = nbSeq;
     cctx->externSeqStore.capacity = nbSeq;
@@ -2618,7 +2745,7 @@ static size_t ZSTD_compressContinue_internal (ZSTD_CCtx* cctx,
     if (frame && (cctx->stage==ZSTDcs_init)) {
         fhSize = ZSTD_writeFrameHeader(dst, dstCapacity, &cctx->appliedParams,
                                        cctx->pledgedSrcSizePlusOne-1, cctx->dictID);
-        FORWARD_IF_ERROR(fhSize);
+        FORWARD_IF_ERROR(fhSize, "ZSTD_writeFrameHeader failed");
         assert(fhSize <= dstCapacity);
         dstCapacity -= fhSize;
         dst = (char*)dst + fhSize;
@@ -2645,7 +2772,7 @@ static size_t ZSTD_compressContinue_internal (ZSTD_CCtx* cctx,
     {   size_t const cSize = frame ?
                              ZSTD_compress_frameChunk (cctx, dst, dstCapacity, src, srcSize, lastFrameChunk) :
                              ZSTD_compressBlock_internal (cctx, dst, dstCapacity, src, srcSize, 0 /* frame */);
-        FORWARD_IF_ERROR(cSize);
+        FORWARD_IF_ERROR(cSize, "%s", frame ? "ZSTD_compress_frameChunk failed" : "ZSTD_compressBlock_internal failed");
         cctx->consumedSrcSize += srcSize;
         cctx->producedCSize += (cSize + fhSize);
         assert(!(cctx->appliedParams.fParams.contentSizeFlag && cctx->pledgedSrcSizePlusOne == 0));
@@ -2682,7 +2809,7 @@ size_t ZSTD_compressBlock(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const
 {
     DEBUGLOG(5, "ZSTD_compressBlock: srcSize = %u", (unsigned)srcSize);
     { size_t const blockSizeMax = ZSTD_getBlockSize(cctx);
-      RETURN_ERROR_IF(srcSize > blockSizeMax, srcSize_wrong); }
+      RETURN_ERROR_IF(srcSize > blockSizeMax, srcSize_wrong, "input is larger than a block"); }
 
     return ZSTD_compressContinue_internal(cctx, dst, dstCapacity, src, srcSize, 0 /* frame mode */, 0 /* last chunk */);
 }
@@ -2691,6 +2818,7 @@ size_t ZSTD_compressBlock(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const
  *  @return : 0, or an error code
  */
 static size_t ZSTD_loadDictionaryContent(ZSTD_matchState_t* ms,
+                                         ldmState_t* ls,
                                          ZSTD_cwksp* ws,
                                          ZSTD_CCtx_params const* params,
                                          const void* src, size_t srcSize,
@@ -2702,6 +2830,11 @@ static size_t ZSTD_loadDictionaryContent(ZSTD_matchState_t* ms,
     ZSTD_window_update(&ms->window, src, srcSize);
     ms->loadedDictEnd = params->forceWindow ? 0 : (U32)(iend - ms->window.base);
 
+    if (params->ldmParams.enableLdm && ls != NULL) {
+        ZSTD_window_update(&ls->window, src, srcSize);
+        ls->loadedDictEnd = params->forceWindow ? 0 : (U32)(iend - ls->window.base);
+    }
+
     /* Assert that we the ms params match the params we're being given */
     ZSTD_assertEqualCParams(params->cParams, ms->cParams);
 
@@ -2714,6 +2847,9 @@ static size_t ZSTD_loadDictionaryContent(ZSTD_matchState_t* ms,
 
         ZSTD_overflowCorrectIfNeeded(ms, ws, params, ip, ichunk);
 
+        if (params->ldmParams.enableLdm && ls != NULL)
+            ZSTD_ldm_fillHashTable(ls, (const BYTE*)src, (const BYTE*)src + srcSize, &params->ldmParams);
+
         switch(params->cParams.strategy)
         {
         case ZSTD_fast:
@@ -2756,102 +2892,123 @@ static size_t ZSTD_loadDictionaryContent(ZSTD_matchState_t* ms,
    NOTE: This behavior is not standard and could be improved in the future. */
 static size_t ZSTD_checkDictNCount(short* normalizedCounter, unsigned dictMaxSymbolValue, unsigned maxSymbolValue) {
     U32 s;
-    RETURN_ERROR_IF(dictMaxSymbolValue < maxSymbolValue, dictionary_corrupted);
+    RETURN_ERROR_IF(dictMaxSymbolValue < maxSymbolValue, dictionary_corrupted, "dict fse tables don't have all symbols");
     for (s = 0; s <= maxSymbolValue; ++s) {
-        RETURN_ERROR_IF(normalizedCounter[s] == 0, dictionary_corrupted);
+        RETURN_ERROR_IF(normalizedCounter[s] == 0, dictionary_corrupted, "dict fse tables don't have all symbols");
     }
     return 0;
 }
 
-
-/* Dictionary format :
- * See :
- * https://github.com/facebook/zstd/blob/master/doc/zstd_compression_format.md#dictionary-format
- */
-/*! ZSTD_loadZstdDictionary() :
- * @return : dictID, or an error code
- *  assumptions : magic number supposed already checked
- *                dictSize supposed >= 8
- */
-static size_t ZSTD_loadZstdDictionary(ZSTD_compressedBlockState_t* bs,
-                                      ZSTD_matchState_t* ms,
-                                      ZSTD_cwksp* ws,
-                                      ZSTD_CCtx_params const* params,
-                                      const void* dict, size_t dictSize,
-                                      ZSTD_dictTableLoadMethod_e dtlm,
-                                      void* workspace)
+size_t ZSTD_loadCEntropy(ZSTD_compressedBlockState_t* bs, void* workspace,
+                         short* offcodeNCount, unsigned* offcodeMaxValue,
+                         const void* const dict, size_t dictSize)
 {
-    const BYTE* dictPtr = (const BYTE*)dict;
+    const BYTE* dictPtr = (const BYTE*)dict;    /* skip magic num and dict ID */
     const BYTE* const dictEnd = dictPtr + dictSize;
-    short offcodeNCount[MaxOff+1];
-    unsigned offcodeMaxValue = MaxOff;
-    size_t dictID;
+    dictPtr += 8;
+    bs->entropy.huf.repeatMode = HUF_repeat_check;
 
-    ZSTD_STATIC_ASSERT(HUF_WORKSPACE_SIZE >= (1<<MAX(MLFSELog,LLFSELog)));
-    assert(dictSize >= 8);
-    assert(MEM_readLE32(dictPtr) == ZSTD_MAGIC_DICTIONARY);
+    {   unsigned maxSymbolValue = 255;
+        unsigned hasZeroWeights = 1;
+        size_t const hufHeaderSize = HUF_readCTable((HUF_CElt*)bs->entropy.huf.CTable, &maxSymbolValue, dictPtr,
+            dictEnd-dictPtr, &hasZeroWeights);
 
-    dictPtr += 4;   /* skip magic number */
-    dictID = params->fParams.noDictIDFlag ? 0 :  MEM_readLE32(dictPtr);
-    dictPtr += 4;
+        /* We only set the loaded table as valid if it contains all non-zero
+         * weights. Otherwise, we set it to check */
+        if (!hasZeroWeights)
+            bs->entropy.huf.repeatMode = HUF_repeat_valid;
 
-    {   unsigned maxSymbolValue = 255;
-        size_t const hufHeaderSize = HUF_readCTable((HUF_CElt*)bs->entropy.huf.CTable, &maxSymbolValue, dictPtr, dictEnd-dictPtr);
-        RETURN_ERROR_IF(HUF_isError(hufHeaderSize), dictionary_corrupted);
-        RETURN_ERROR_IF(maxSymbolValue < 255, dictionary_corrupted);
+        RETURN_ERROR_IF(HUF_isError(hufHeaderSize), dictionary_corrupted, "");
+        RETURN_ERROR_IF(maxSymbolValue < 255, dictionary_corrupted, "");
         dictPtr += hufHeaderSize;
     }
 
     {   unsigned offcodeLog;
-        size_t const offcodeHeaderSize = FSE_readNCount(offcodeNCount, &offcodeMaxValue, &offcodeLog, dictPtr, dictEnd-dictPtr);
-        RETURN_ERROR_IF(FSE_isError(offcodeHeaderSize), dictionary_corrupted);
-        RETURN_ERROR_IF(offcodeLog > OffFSELog, dictionary_corrupted);
+        size_t const offcodeHeaderSize = FSE_readNCount(offcodeNCount, offcodeMaxValue, &offcodeLog, dictPtr, dictEnd-dictPtr);
+        RETURN_ERROR_IF(FSE_isError(offcodeHeaderSize), dictionary_corrupted, "");
+        RETURN_ERROR_IF(offcodeLog > OffFSELog, dictionary_corrupted, "");
         /* Defer checking offcodeMaxValue because we need to know the size of the dictionary content */
         /* fill all offset symbols to avoid garbage at end of table */
         RETURN_ERROR_IF(FSE_isError(FSE_buildCTable_wksp(
                 bs->entropy.fse.offcodeCTable,
                 offcodeNCount, MaxOff, offcodeLog,
                 workspace, HUF_WORKSPACE_SIZE)),
-            dictionary_corrupted);
+            dictionary_corrupted, "");
         dictPtr += offcodeHeaderSize;
     }
 
     {   short matchlengthNCount[MaxML+1];
         unsigned matchlengthMaxValue = MaxML, matchlengthLog;
         size_t const matchlengthHeaderSize = FSE_readNCount(matchlengthNCount, &matchlengthMaxValue, &matchlengthLog, dictPtr, dictEnd-dictPtr);
-        RETURN_ERROR_IF(FSE_isError(matchlengthHeaderSize), dictionary_corrupted);
-        RETURN_ERROR_IF(matchlengthLog > MLFSELog, dictionary_corrupted);
+        RETURN_ERROR_IF(FSE_isError(matchlengthHeaderSize), dictionary_corrupted, "");
+        RETURN_ERROR_IF(matchlengthLog > MLFSELog, dictionary_corrupted, "");
         /* Every match length code must have non-zero probability */
-        FORWARD_IF_ERROR( ZSTD_checkDictNCount(matchlengthNCount, matchlengthMaxValue, MaxML));
+        FORWARD_IF_ERROR( ZSTD_checkDictNCount(matchlengthNCount, matchlengthMaxValue, MaxML), "");
         RETURN_ERROR_IF(FSE_isError(FSE_buildCTable_wksp(
                 bs->entropy.fse.matchlengthCTable,
                 matchlengthNCount, matchlengthMaxValue, matchlengthLog,
                 workspace, HUF_WORKSPACE_SIZE)),
-            dictionary_corrupted);
+            dictionary_corrupted, "");
         dictPtr += matchlengthHeaderSize;
     }
 
     {   short litlengthNCount[MaxLL+1];
         unsigned litlengthMaxValue = MaxLL, litlengthLog;
         size_t const litlengthHeaderSize = FSE_readNCount(litlengthNCount, &litlengthMaxValue, &litlengthLog, dictPtr, dictEnd-dictPtr);
-        RETURN_ERROR_IF(FSE_isError(litlengthHeaderSize), dictionary_corrupted);
-        RETURN_ERROR_IF(litlengthLog > LLFSELog, dictionary_corrupted);
+        RETURN_ERROR_IF(FSE_isError(litlengthHeaderSize), dictionary_corrupted, "");
+        RETURN_ERROR_IF(litlengthLog > LLFSELog, dictionary_corrupted, "");
         /* Every literal length code must have non-zero probability */
-        FORWARD_IF_ERROR( ZSTD_checkDictNCount(litlengthNCount, litlengthMaxValue, MaxLL));
+        FORWARD_IF_ERROR( ZSTD_checkDictNCount(litlengthNCount, litlengthMaxValue, MaxLL), "");
         RETURN_ERROR_IF(FSE_isError(FSE_buildCTable_wksp(
                 bs->entropy.fse.litlengthCTable,
                 litlengthNCount, litlengthMaxValue, litlengthLog,
                 workspace, HUF_WORKSPACE_SIZE)),
-            dictionary_corrupted);
+            dictionary_corrupted, "");
         dictPtr += litlengthHeaderSize;
     }
 
-    RETURN_ERROR_IF(dictPtr+12 > dictEnd, dictionary_corrupted);
+    RETURN_ERROR_IF(dictPtr+12 > dictEnd, dictionary_corrupted, "");
     bs->rep[0] = MEM_readLE32(dictPtr+0);
     bs->rep[1] = MEM_readLE32(dictPtr+4);
     bs->rep[2] = MEM_readLE32(dictPtr+8);
     dictPtr += 12;
 
+    return dictPtr - (const BYTE*)dict;
+}
+
+/* Dictionary format :
+ * See :
+ * https://github.com/facebook/zstd/blob/master/doc/zstd_compression_format.md#dictionary-format
+ */
+/*! ZSTD_loadZstdDictionary() :
+ * @return : dictID, or an error code
+ *  assumptions : magic number supposed already checked
+ *                dictSize supposed >= 8
+ */
+static size_t ZSTD_loadZstdDictionary(ZSTD_compressedBlockState_t* bs,
+                                      ZSTD_matchState_t* ms,
+                                      ZSTD_cwksp* ws,
+                                      ZSTD_CCtx_params const* params,
+                                      const void* dict, size_t dictSize,
+                                      ZSTD_dictTableLoadMethod_e dtlm,
+                                      void* workspace)
+{
+    const BYTE* dictPtr = (const BYTE*)dict;
+    const BYTE* const dictEnd = dictPtr + dictSize;
+    short offcodeNCount[MaxOff+1];
+    unsigned offcodeMaxValue = MaxOff;
+    size_t dictID;
+    size_t eSize;
+
+    ZSTD_STATIC_ASSERT(HUF_WORKSPACE_SIZE >= (1<<MAX(MLFSELog,LLFSELog)));
+    assert(dictSize >= 8);
+    assert(MEM_readLE32(dictPtr) == ZSTD_MAGIC_DICTIONARY);
+
+    dictID = params->fParams.noDictIDFlag ? 0 :  MEM_readLE32(dictPtr + 4 /* skip magic number */ );
+    eSize = ZSTD_loadCEntropy(bs, workspace, offcodeNCount, &offcodeMaxValue, dict, dictSize);
+    FORWARD_IF_ERROR(eSize, "ZSTD_loadCEntropy failed");
+    dictPtr += eSize;
+
     {   size_t const dictContentSize = (size_t)(dictEnd - dictPtr);
         U32 offcodeMax = MaxOff;
         if (dictContentSize <= ((U32)-1) - 128 KB) {
@@ -2859,20 +3016,19 @@ static size_t ZSTD_loadZstdDictionary(ZSTD_compressedBlockState_t* bs,
             offcodeMax = ZSTD_highbit32(maxOffset); /* Calculate minimum offset code required to represent maxOffset */
         }
         /* All offset values <= dictContentSize + 128 KB must be representable */
-        FORWARD_IF_ERROR(ZSTD_checkDictNCount(offcodeNCount, offcodeMaxValue, MIN(offcodeMax, MaxOff)));
+        FORWARD_IF_ERROR(ZSTD_checkDictNCount(offcodeNCount, offcodeMaxValue, MIN(offcodeMax, MaxOff)), "");
         /* All repCodes must be <= dictContentSize and != 0*/
         {   U32 u;
             for (u=0; u<3; u++) {
-                RETURN_ERROR_IF(bs->rep[u] == 0, dictionary_corrupted);
-                RETURN_ERROR_IF(bs->rep[u] > dictContentSize, dictionary_corrupted);
+                RETURN_ERROR_IF(bs->rep[u] == 0, dictionary_corrupted, "");
+                RETURN_ERROR_IF(bs->rep[u] > dictContentSize, dictionary_corrupted, "");
         }   }
 
-        bs->entropy.huf.repeatMode = HUF_repeat_valid;
         bs->entropy.fse.offcode_repeatMode = FSE_repeat_valid;
         bs->entropy.fse.matchlength_repeatMode = FSE_repeat_valid;
         bs->entropy.fse.litlength_repeatMode = FSE_repeat_valid;
         FORWARD_IF_ERROR(ZSTD_loadDictionaryContent(
-            ms, ws, params, dictPtr, dictContentSize, dtlm));
+            ms, NULL, ws, params, dictPtr, dictContentSize, dtlm), "");
         return dictID;
     }
 }
@@ -2882,6 +3038,7 @@ static size_t ZSTD_loadZstdDictionary(ZSTD_compressedBlockState_t* bs,
 static size_t
 ZSTD_compress_insertDictionary(ZSTD_compressedBlockState_t* bs,
                                ZSTD_matchState_t* ms,
+                               ldmState_t* ls,
                                ZSTD_cwksp* ws,
                          const ZSTD_CCtx_params* params,
                          const void* dict, size_t dictSize,
@@ -2891,7 +3048,7 @@ ZSTD_compress_insertDictionary(ZSTD_compressedBlockState_t* bs,
 {
     DEBUGLOG(4, "ZSTD_compress_insertDictionary (dictSize=%u)", (U32)dictSize);
     if ((dict==NULL) || (dictSize<8)) {
-        RETURN_ERROR_IF(dictContentType == ZSTD_dct_fullDict, dictionary_wrong);
+        RETURN_ERROR_IF(dictContentType == ZSTD_dct_fullDict, dictionary_wrong, "");
         return 0;
     }
 
@@ -2899,15 +3056,15 @@ ZSTD_compress_insertDictionary(ZSTD_compressedBlockState_t* bs,
 
     /* dict restricted modes */
     if (dictContentType == ZSTD_dct_rawContent)
-        return ZSTD_loadDictionaryContent(ms, ws, params, dict, dictSize, dtlm);
+        return ZSTD_loadDictionaryContent(ms, ls, ws, params, dict, dictSize, dtlm);
 
     if (MEM_readLE32(dict) != ZSTD_MAGIC_DICTIONARY) {
         if (dictContentType == ZSTD_dct_auto) {
             DEBUGLOG(4, "raw content dictionary detected");
             return ZSTD_loadDictionaryContent(
-                ms, ws, params, dict, dictSize, dtlm);
+                ms, ls, ws, params, dict, dictSize, dtlm);
         }
-        RETURN_ERROR_IF(dictContentType == ZSTD_dct_fullDict, dictionary_wrong);
+        RETURN_ERROR_IF(dictContentType == ZSTD_dct_fullDict, dictionary_wrong, "");
         assert(0);   /* impossible */
     }
 
@@ -2944,17 +3101,18 @@ static size_t ZSTD_compressBegin_internal(ZSTD_CCtx* cctx,
     }
 
     FORWARD_IF_ERROR( ZSTD_resetCCtx_internal(cctx, *params, pledgedSrcSize,
-                                     ZSTDcrp_makeClean, zbuff) );
+                                     ZSTDcrp_makeClean, zbuff) , "");
     {   size_t const dictID = cdict ?
                 ZSTD_compress_insertDictionary(
                         cctx->blockState.prevCBlock, &cctx->blockState.matchState,
-                        &cctx->workspace, params, cdict->dictContent, cdict->dictContentSize,
-                        dictContentType, dtlm, cctx->entropyWorkspace)
+                        &cctx->ldmState, &cctx->workspace, &cctx->appliedParams, cdict->dictContent,
+                        cdict->dictContentSize, dictContentType, dtlm,
+                        cctx->entropyWorkspace)
               : ZSTD_compress_insertDictionary(
                         cctx->blockState.prevCBlock, &cctx->blockState.matchState,
-                        &cctx->workspace, params, dict, dictSize,
+                        &cctx->ldmState, &cctx->workspace, &cctx->appliedParams, dict, dictSize,
                         dictContentType, dtlm, cctx->entropyWorkspace);
-        FORWARD_IF_ERROR(dictID);
+        FORWARD_IF_ERROR(dictID, "ZSTD_compress_insertDictionary failed");
         assert(dictID <= UINT_MAX);
         cctx->dictID = (U32)dictID;
     }
@@ -2971,7 +3129,7 @@ size_t ZSTD_compressBegin_advanced_internal(ZSTD_CCtx* cctx,
 {
     DEBUGLOG(4, "ZSTD_compressBegin_advanced_internal: wlog=%u", params->cParams.windowLog);
     /* compression parameters verification and optimization */
-    FORWARD_IF_ERROR( ZSTD_checkCParams(params->cParams) );
+    FORWARD_IF_ERROR( ZSTD_checkCParams(params->cParams) , "");
     return ZSTD_compressBegin_internal(cctx,
                                        dict, dictSize, dictContentType, dtlm,
                                        cdict,
@@ -2986,7 +3144,7 @@ size_t ZSTD_compressBegin_advanced(ZSTD_CCtx* cctx,
                                    ZSTD_parameters params, unsigned long long pledgedSrcSize)
 {
     ZSTD_CCtx_params const cctxParams =
-            ZSTD_assignParamsToCCtxParams(&cctx->requestedParams, params);
+            ZSTD_assignParamsToCCtxParams(&cctx->requestedParams, &params);
     return ZSTD_compressBegin_advanced_internal(cctx,
                                             dict, dictSize, ZSTD_dct_auto, ZSTD_dtlm_fast,
                                             NULL /*cdict*/,
@@ -2995,9 +3153,9 @@ size_t ZSTD_compressBegin_advanced(ZSTD_CCtx* cctx,
 
 size_t ZSTD_compressBegin_usingDict(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, int compressionLevel)
 {
-    ZSTD_parameters const params = ZSTD_getParams(compressionLevel, ZSTD_CONTENTSIZE_UNKNOWN, dictSize);
+    ZSTD_parameters const params = ZSTD_getParams_internal(compressionLevel, ZSTD_CONTENTSIZE_UNKNOWN, dictSize);
     ZSTD_CCtx_params const cctxParams =
-            ZSTD_assignParamsToCCtxParams(&cctx->requestedParams, params);
+            ZSTD_assignParamsToCCtxParams(&cctx->requestedParams, &params);
     DEBUGLOG(4, "ZSTD_compressBegin_usingDict (dictSize=%u)", (unsigned)dictSize);
     return ZSTD_compressBegin_internal(cctx, dict, dictSize, ZSTD_dct_auto, ZSTD_dtlm_fast, NULL,
                                        &cctxParams, ZSTD_CONTENTSIZE_UNKNOWN, ZSTDb_not_buffered);
@@ -3024,7 +3182,7 @@ static size_t ZSTD_writeEpilogue(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity)
     /* special case : empty frame */
     if (cctx->stage == ZSTDcs_init) {
         fhSize = ZSTD_writeFrameHeader(dst, dstCapacity, &cctx->appliedParams, 0, 0);
-        FORWARD_IF_ERROR(fhSize);
+        FORWARD_IF_ERROR(fhSize, "ZSTD_writeFrameHeader failed");
         dstCapacity -= fhSize;
         op += fhSize;
         cctx->stage = ZSTDcs_ongoing;
@@ -3033,7 +3191,7 @@ static size_t ZSTD_writeEpilogue(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity)
     if (cctx->stage != ZSTDcs_ending) {
         /* write one last empty block, make it the "last" block */
         U32 const cBlockHeader24 = 1 /* last block */ + (((U32)bt_raw)<<1) + 0;
-        RETURN_ERROR_IF(dstCapacity<4, dstSize_tooSmall);
+        RETURN_ERROR_IF(dstCapacity<4, dstSize_tooSmall, "no room for epilogue");
         MEM_writeLE32(op, cBlockHeader24);
         op += ZSTD_blockHeaderSize;
         dstCapacity -= ZSTD_blockHeaderSize;
@@ -3041,7 +3199,7 @@ static size_t ZSTD_writeEpilogue(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity)
 
     if (cctx->appliedParams.fParams.checksumFlag) {
         U32 const checksum = (U32) XXH64_digest(&cctx->xxhState);
-        RETURN_ERROR_IF(dstCapacity<4, dstSize_tooSmall);
+        RETURN_ERROR_IF(dstCapacity<4, dstSize_tooSmall, "no room for checksum");
         DEBUGLOG(4, "ZSTD_writeEpilogue: write checksum : %08X", (unsigned)checksum);
         MEM_writeLE32(op, checksum);
         op += 4;
@@ -3059,9 +3217,9 @@ size_t ZSTD_compressEnd (ZSTD_CCtx* cctx,
     size_t const cSize = ZSTD_compressContinue_internal(cctx,
                                 dst, dstCapacity, src, srcSize,
                                 1 /* frame mode */, 1 /* last chunk */);
-    FORWARD_IF_ERROR(cSize);
+    FORWARD_IF_ERROR(cSize, "ZSTD_compressContinue_internal failed");
     endResult = ZSTD_writeEpilogue(cctx, (char*)dst + cSize, dstCapacity-cSize);
-    FORWARD_IF_ERROR(endResult);
+    FORWARD_IF_ERROR(endResult, "ZSTD_writeEpilogue failed");
     assert(!(cctx->appliedParams.fParams.contentSizeFlag && cctx->pledgedSrcSizePlusOne == 0));
     if (cctx->pledgedSrcSizePlusOne != 0) {  /* control src size */
         ZSTD_STATIC_ASSERT(ZSTD_CONTENTSIZE_UNKNOWN == (unsigned long long)-1);
@@ -3081,7 +3239,7 @@ static size_t ZSTD_compress_internal (ZSTD_CCtx* cctx,
                                       void* dst, size_t dstCapacity,
                                 const void* src, size_t srcSize,
                                 const void* dict,size_t dictSize,
-                                      ZSTD_parameters params)
+                                const ZSTD_parameters* params)
 {
     ZSTD_CCtx_params const cctxParams =
             ZSTD_assignParamsToCCtxParams(&cctx->requestedParams, params);
@@ -3100,12 +3258,12 @@ size_t ZSTD_compress_advanced (ZSTD_CCtx* cctx,
                                ZSTD_parameters params)
 {
     DEBUGLOG(4, "ZSTD_compress_advanced");
-    FORWARD_IF_ERROR(ZSTD_checkCParams(params.cParams));
+    FORWARD_IF_ERROR(ZSTD_checkCParams(params.cParams), "");
     return ZSTD_compress_internal(cctx,
                                   dst, dstCapacity,
                                   src, srcSize,
                                   dict, dictSize,
-                                  params);
+                                  &params);
 }
 
 /* Internal */
@@ -3119,7 +3277,7 @@ size_t ZSTD_compress_advanced_internal(
     DEBUGLOG(4, "ZSTD_compress_advanced_internal (srcSize:%u)", (unsigned)srcSize);
     FORWARD_IF_ERROR( ZSTD_compressBegin_internal(cctx,
                          dict, dictSize, ZSTD_dct_auto, ZSTD_dtlm_fast, NULL,
-                         params, srcSize, ZSTDb_not_buffered) );
+                         params, srcSize, ZSTDb_not_buffered) , "");
     return ZSTD_compressEnd(cctx, dst, dstCapacity, src, srcSize);
 }
 
@@ -3129,8 +3287,9 @@ size_t ZSTD_compress_usingDict(ZSTD_CCtx* cctx,
                          const void* dict, size_t dictSize,
                                int compressionLevel)
 {
-    ZSTD_parameters const params = ZSTD_getParams(compressionLevel, srcSize + (!srcSize), dict ? dictSize : 0);
-    ZSTD_CCtx_params cctxParams = ZSTD_assignParamsToCCtxParams(&cctx->requestedParams, params);
+    ZSTD_parameters const params = ZSTD_getParams_internal(compressionLevel, srcSize, dict ? dictSize : 0);
+    ZSTD_CCtx_params cctxParams = ZSTD_assignParamsToCCtxParams(&cctx->requestedParams, &params);
+    DEBUGLOG(4, "ZSTD_compress_usingDict (srcSize=%u)", (unsigned)srcSize);
     assert(params.fParams.contentSizeFlag == 1);
     return ZSTD_compress_advanced_internal(cctx, dst, dstCapacity, src, srcSize, dict, dictSize, &cctxParams);
 }
@@ -3176,7 +3335,7 @@ size_t ZSTD_estimateCDictSize_advanced(
 
 size_t ZSTD_estimateCDictSize(size_t dictSize, int compressionLevel)
 {
-    ZSTD_compressionParameters const cParams = ZSTD_getCParams(compressionLevel, 0, dictSize);
+    ZSTD_compressionParameters const cParams = ZSTD_getCParams_internal(compressionLevel, ZSTD_CONTENTSIZE_UNKNOWN, dictSize);
     return ZSTD_estimateCDictSize_advanced(dictSize, cParams, ZSTD_dlm_byCopy);
 }
 
@@ -3203,7 +3362,7 @@ static size_t ZSTD_initCDict_internal(
         cdict->dictContent = dictBuffer;
     } else {
          void *internalBuffer = ZSTD_cwksp_reserve_object(&cdict->workspace, ZSTD_cwksp_align(dictSize, sizeof(void*)));
-        RETURN_ERROR_IF(!internalBuffer, memory_allocation);
+        RETURN_ERROR_IF(!internalBuffer, memory_allocation, "NULL pointer!");
         cdict->dictContent = internalBuffer;
         memcpy(internalBuffer, dictBuffer, dictSize);
     }
@@ -3220,7 +3379,7 @@ static size_t ZSTD_initCDict_internal(
         &cParams,
         ZSTDcrp_makeClean,
         ZSTDirp_reset,
-        ZSTD_resetTarget_CDict));
+        ZSTD_resetTarget_CDict), "");
     /* (Maybe) load the dictionary
      * Skips loading the dictionary if it is < 8 bytes.
      */
@@ -3230,10 +3389,10 @@ static size_t ZSTD_initCDict_internal(
         params.fParams.contentSizeFlag = 1;
         params.cParams = cParams;
         {   size_t const dictID = ZSTD_compress_insertDictionary(
-                    &cdict->cBlockState, &cdict->matchState, &cdict->workspace,
+                    &cdict->cBlockState, &cdict->matchState, NULL, &cdict->workspace,
                     &params, cdict->dictContent, cdict->dictContentSize,
                     dictContentType, ZSTD_dtlm_full, cdict->entropyWorkspace);
-            FORWARD_IF_ERROR(dictID);
+            FORWARD_IF_ERROR(dictID, "ZSTD_compress_insertDictionary failed");
             assert(dictID <= (size_t)(U32)-1);
             cdict->dictID = (U32)dictID;
         }
@@ -3287,7 +3446,7 @@ ZSTD_CDict* ZSTD_createCDict_advanced(const void* dictBuffer, size_t dictSize,
 
 ZSTD_CDict* ZSTD_createCDict(const void* dict, size_t dictSize, int compressionLevel)
 {
-    ZSTD_compressionParameters cParams = ZSTD_getCParams(compressionLevel, 0, dictSize);
+    ZSTD_compressionParameters cParams = ZSTD_getCParams_internal(compressionLevel, ZSTD_CONTENTSIZE_UNKNOWN, dictSize);
     ZSTD_CDict* cdict = ZSTD_createCDict_advanced(dict, dictSize,
                                                   ZSTD_dlm_byCopy, ZSTD_dct_auto,
                                                   cParams, ZSTD_defaultCMem);
@@ -3298,7 +3457,7 @@ ZSTD_CDict* ZSTD_createCDict(const void* dict, size_t dictSize, int compressionL
 
 ZSTD_CDict* ZSTD_createCDict_byReference(const void* dict, size_t dictSize, int compressionLevel)
 {
-    ZSTD_compressionParameters cParams = ZSTD_getCParams(compressionLevel, 0, dictSize);
+    ZSTD_compressionParameters cParams = ZSTD_getCParams_internal(compressionLevel, ZSTD_CONTENTSIZE_UNKNOWN, dictSize);
     return ZSTD_createCDict_advanced(dict, dictSize,
                                      ZSTD_dlm_byRef, ZSTD_dct_auto,
                                      cParams, ZSTD_defaultCMem);
@@ -3381,7 +3540,7 @@ size_t ZSTD_compressBegin_usingCDict_advanced(
     ZSTD_frameParameters const fParams, unsigned long long const pledgedSrcSize)
 {
     DEBUGLOG(4, "ZSTD_compressBegin_usingCDict_advanced");
-    RETURN_ERROR_IF(cdict==NULL, dictionary_wrong);
+    RETURN_ERROR_IF(cdict==NULL, dictionary_wrong, "NULL pointer!");
     {   ZSTD_CCtx_params params = cctx->requestedParams;
         params.cParams = ( pledgedSrcSize < ZSTD_USE_CDICT_PARAMS_SRCSIZE_CUTOFF
                         || pledgedSrcSize < cdict->dictContentSize * ZSTD_USE_CDICT_PARAMS_DICTSIZE_MULTIPLIER
@@ -3425,7 +3584,7 @@ size_t ZSTD_compress_usingCDict_advanced(ZSTD_CCtx* cctx,
                                 const void* src, size_t srcSize,
                                 const ZSTD_CDict* cdict, ZSTD_frameParameters fParams)
 {
-    FORWARD_IF_ERROR(ZSTD_compressBegin_usingCDict_advanced(cctx, cdict, fParams, srcSize));   /* will check if cdict != NULL */
+    FORWARD_IF_ERROR(ZSTD_compressBegin_usingCDict_advanced(cctx, cdict, fParams, srcSize), "");   /* will check if cdict != NULL */
     return ZSTD_compressEnd(cctx, dst, dstCapacity, src, srcSize);
 }
 
@@ -3497,7 +3656,7 @@ static size_t ZSTD_resetCStream_internal(ZSTD_CStream* cctx,
                                          dict, dictSize, dictContentType, ZSTD_dtlm_fast,
                                          cdict,
                                          &params, pledgedSrcSize,
-                                         ZSTDb_buffered) );
+                                         ZSTDb_buffered) , "");
 
     cctx->inToCompress = 0;
     cctx->inBuffPos = 0;
@@ -3519,8 +3678,8 @@ size_t ZSTD_resetCStream(ZSTD_CStream* zcs, unsigned long long pss)
      */
     U64 const pledgedSrcSize = (pss==0) ? ZSTD_CONTENTSIZE_UNKNOWN : pss;
     DEBUGLOG(4, "ZSTD_resetCStream: pledgedSrcSize = %u", (unsigned)pledgedSrcSize);
-    FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) );
-    FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) );
+    FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , "");
+    FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) , "");
     return 0;
 }
 
@@ -3534,16 +3693,16 @@ size_t ZSTD_initCStream_internal(ZSTD_CStream* zcs,
                     unsigned long long pledgedSrcSize)
 {
     DEBUGLOG(4, "ZSTD_initCStream_internal");
-    FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) );
-    FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) );
+    FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , "");
+    FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) , "");
     assert(!ZSTD_isError(ZSTD_checkCParams(params->cParams)));
     zcs->requestedParams = *params;
     assert(!((dict) && (cdict)));  /* either dict or cdict, not both */
     if (dict) {
-        FORWARD_IF_ERROR( ZSTD_CCtx_loadDictionary(zcs, dict, dictSize) );
+        FORWARD_IF_ERROR( ZSTD_CCtx_loadDictionary(zcs, dict, dictSize) , "");
     } else {
         /* Dictionary is cleared if !cdict */
-        FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, cdict) );
+        FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, cdict) , "");
     }
     return 0;
 }
@@ -3556,10 +3715,10 @@ size_t ZSTD_initCStream_usingCDict_advanced(ZSTD_CStream* zcs,
                                             unsigned long long pledgedSrcSize)
 {
     DEBUGLOG(4, "ZSTD_initCStream_usingCDict_advanced");
-    FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) );
-    FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) );
+    FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , "");
+    FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) , "");
     zcs->requestedParams.fParams = fParams;
-    FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, cdict) );
+    FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, cdict) , "");
     return 0;
 }
 
@@ -3567,8 +3726,8 @@ size_t ZSTD_initCStream_usingCDict_advanced(ZSTD_CStream* zcs,
 size_t ZSTD_initCStream_usingCDict(ZSTD_CStream* zcs, const ZSTD_CDict* cdict)
 {
     DEBUGLOG(4, "ZSTD_initCStream_usingCDict");
-    FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) );
-    FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, cdict) );
+    FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , "");
+    FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, cdict) , "");
     return 0;
 }
 
@@ -3587,20 +3746,20 @@ size_t ZSTD_initCStream_advanced(ZSTD_CStream* zcs,
      */
     U64 const pledgedSrcSize = (pss==0 && params.fParams.contentSizeFlag==0) ? ZSTD_CONTENTSIZE_UNKNOWN : pss;
     DEBUGLOG(4, "ZSTD_initCStream_advanced");
-    FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) );
-    FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) );
-    FORWARD_IF_ERROR( ZSTD_checkCParams(params.cParams) );
-    zcs->requestedParams = ZSTD_assignParamsToCCtxParams(&zcs->requestedParams, params);
-    FORWARD_IF_ERROR( ZSTD_CCtx_loadDictionary(zcs, dict, dictSize) );
+    FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , "");
+    FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) , "");
+    FORWARD_IF_ERROR( ZSTD_checkCParams(params.cParams) , "");
+    zcs->requestedParams = ZSTD_assignParamsToCCtxParams(&zcs->requestedParams, &params);
+    FORWARD_IF_ERROR( ZSTD_CCtx_loadDictionary(zcs, dict, dictSize) , "");
     return 0;
 }
 
 size_t ZSTD_initCStream_usingDict(ZSTD_CStream* zcs, const void* dict, size_t dictSize, int compressionLevel)
 {
     DEBUGLOG(4, "ZSTD_initCStream_usingDict");
-    FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) );
-    FORWARD_IF_ERROR( ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel) );
-    FORWARD_IF_ERROR( ZSTD_CCtx_loadDictionary(zcs, dict, dictSize) );
+    FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , "");
+    FORWARD_IF_ERROR( ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel) , "");
+    FORWARD_IF_ERROR( ZSTD_CCtx_loadDictionary(zcs, dict, dictSize) , "");
     return 0;
 }
 
@@ -3612,19 +3771,19 @@ size_t ZSTD_initCStream_srcSize(ZSTD_CStream* zcs, int compressionLevel, unsigne
      */
     U64 const pledgedSrcSize = (pss==0) ? ZSTD_CONTENTSIZE_UNKNOWN : pss;
     DEBUGLOG(4, "ZSTD_initCStream_srcSize");
-    FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) );
-    FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, NULL) );
-    FORWARD_IF_ERROR( ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel) );
-    FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) );
+    FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , "");
+    FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, NULL) , "");
+    FORWARD_IF_ERROR( ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel) , "");
+    FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) , "");
     return 0;
 }
 
 size_t ZSTD_initCStream(ZSTD_CStream* zcs, int compressionLevel)
 {
     DEBUGLOG(4, "ZSTD_initCStream");
-    FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) );
-    FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, NULL) );
-    FORWARD_IF_ERROR( ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel) );
+    FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , "");
+    FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, NULL) , "");
+    FORWARD_IF_ERROR( ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel) , "");
     return 0;
 }
 
@@ -3637,14 +3796,6 @@ static size_t ZSTD_nextInputSizeHint(const ZSTD_CCtx* cctx)
     return hintInSize;
 }
 
-static size_t ZSTD_limitCopy(void* dst, size_t dstCapacity,
-                       const void* src, size_t srcSize)
-{
-    size_t const length = MIN(dstCapacity, srcSize);
-    if (length) memcpy(dst, src, length);
-    return length;
-}
-
 /** ZSTD_compressStream_generic():
  *  internal function for all *compressStream*() variants
  *  non-static, because can be called from zstdmt_compress.c
@@ -3655,11 +3806,11 @@ static size_t ZSTD_compressStream_generic(ZSTD_CStream* zcs,
                                           ZSTD_EndDirective const flushMode)
 {
     const char* const istart = (const char*)input->src;
-    const char* const iend = istart + input->size;
-    const char* ip = istart + input->pos;
+    const char* const iend = input->size != 0 ? istart + input->size : istart;
+    const char* ip = input->pos != 0 ? istart + input->pos : istart;
     char* const ostart = (char*)output->dst;
-    char* const oend = ostart + output->size;
-    char* op = ostart + output->pos;
+    char* const oend = output->size != 0 ? ostart + output->size : ostart;
+    char* op = output->pos != 0 ? ostart + output->pos : ostart;
     U32 someMoreWork = 1;
 
     /* check expectations */
@@ -3685,7 +3836,7 @@ static size_t ZSTD_compressStream_generic(ZSTD_CStream* zcs,
                 size_t const cSize = ZSTD_compressEnd(zcs,
                                                 op, oend-op, ip, iend-ip);
                 DEBUGLOG(4, "ZSTD_compressEnd : cSize=%u", (unsigned)cSize);
-                FORWARD_IF_ERROR(cSize);
+                FORWARD_IF_ERROR(cSize, "ZSTD_compressEnd failed");
                 ip = iend;
                 op += cSize;
                 zcs->frameEnded = 1;
@@ -3698,7 +3849,8 @@ static size_t ZSTD_compressStream_generic(ZSTD_CStream* zcs,
                                         zcs->inBuff + zcs->inBuffPos, toLoad,
                                         ip, iend-ip);
                 zcs->inBuffPos += loaded;
-                ip += loaded;
+                if (loaded != 0)
+                    ip += loaded;
                 if ( (flushMode == ZSTD_e_continue)
                   && (zcs->inBuffPos < zcs->inBuffTarget) ) {
                     /* not enough input to fill full block : stop here */
@@ -3726,7 +3878,7 @@ static size_t ZSTD_compressStream_generic(ZSTD_CStream* zcs,
                                     zcs->inBuff + zcs->inToCompress, iSize) :
                         ZSTD_compressContinue(zcs, cDst, oSize,
                                     zcs->inBuff + zcs->inToCompress, iSize);
-                FORWARD_IF_ERROR(cSize);
+                FORWARD_IF_ERROR(cSize, "%s", lastBlock ? "ZSTD_compressEnd failed" : "ZSTD_compressContinue failed");
                 zcs->frameEnded = lastBlock;
                 /* prepare next block */
                 zcs->inBuffTarget = zcs->inBuffPos + zcs->blockSize;
@@ -3758,7 +3910,8 @@ static size_t ZSTD_compressStream_generic(ZSTD_CStream* zcs,
                             zcs->outBuff + zcs->outBuffFlushedSize, toFlush);
                 DEBUGLOG(5, "toFlush: %u into %u ==> flushed: %u",
                             (unsigned)toFlush, (unsigned)(oend-op), (unsigned)flushed);
-                op += flushed;
+                if (flushed)
+                    op += flushed;
                 zcs->outBuffFlushedSize += flushed;
                 if (toFlush!=flushed) {
                     /* flush not fully completed, presumably because dst is too small */
@@ -3802,7 +3955,7 @@ static size_t ZSTD_nextInputSizeHint_MTorST(const ZSTD_CCtx* cctx)
 
 size_t ZSTD_compressStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output, ZSTD_inBuffer* input)
 {
-    FORWARD_IF_ERROR( ZSTD_compressStream2(zcs, output, input, ZSTD_e_continue) );
+    FORWARD_IF_ERROR( ZSTD_compressStream2(zcs, output, input, ZSTD_e_continue) , "");
     return ZSTD_nextInputSizeHint_MTorST(zcs);
 }
 
@@ -3814,15 +3967,15 @@ size_t ZSTD_compressStream2( ZSTD_CCtx* cctx,
 {
     DEBUGLOG(5, "ZSTD_compressStream2, endOp=%u ", (unsigned)endOp);
     /* check conditions */
-    RETURN_ERROR_IF(output->pos > output->size, GENERIC);
-    RETURN_ERROR_IF(input->pos  > input->size, GENERIC);
+    RETURN_ERROR_IF(output->pos > output->size, GENERIC, "invalid buffer");
+    RETURN_ERROR_IF(input->pos  > input->size, GENERIC, "invalid buffer");
     assert(cctx!=NULL);
 
     /* transparent initialization stage */
     if (cctx->streamStage == zcss_init) {
         ZSTD_CCtx_params params = cctx->requestedParams;
         ZSTD_prefixDict const prefixDict = cctx->prefixDict;
-        FORWARD_IF_ERROR( ZSTD_initLocalDict(cctx) ); /* Init the local dict if present. */
+        FORWARD_IF_ERROR( ZSTD_initLocalDict(cctx) , ""); /* Init the local dict if present. */
         memset(&cctx->prefixDict, 0, sizeof(cctx->prefixDict));   /* single usage */
         assert(prefixDict.dict==NULL || cctx->cdict==NULL);    /* only one can be set */
         DEBUGLOG(4, "ZSTD_compressStream2 : transparent init stage");
@@ -3841,14 +3994,14 @@ size_t ZSTD_compressStream2( ZSTD_CCtx* cctx,
                 DEBUGLOG(4, "ZSTD_compressStream2: creating new mtctx for nbWorkers=%u",
                             params.nbWorkers);
                 cctx->mtctx = ZSTDMT_createCCtx_advanced((U32)params.nbWorkers, cctx->customMem);
-                RETURN_ERROR_IF(cctx->mtctx == NULL, memory_allocation);
+                RETURN_ERROR_IF(cctx->mtctx == NULL, memory_allocation, "NULL pointer!");
             }
             /* mt compression */
             DEBUGLOG(4, "call ZSTDMT_initCStream_internal as nbWorkers=%u", params.nbWorkers);
             FORWARD_IF_ERROR( ZSTDMT_initCStream_internal(
                         cctx->mtctx,
-                        prefixDict.dict, prefixDict.dictSize, ZSTD_dct_rawContent,
-                        cctx->cdict, params, cctx->pledgedSrcSizePlusOne-1) );
+                        prefixDict.dict, prefixDict.dictSize, prefixDict.dictContentType,
+                        cctx->cdict, params, cctx->pledgedSrcSizePlusOne-1) , "");
             cctx->streamStage = zcss_load;
             cctx->appliedParams.nbWorkers = params.nbWorkers;
         } else
@@ -3856,7 +4009,7 @@ size_t ZSTD_compressStream2( ZSTD_CCtx* cctx,
         {   FORWARD_IF_ERROR( ZSTD_resetCStream_internal(cctx,
                             prefixDict.dict, prefixDict.dictSize, prefixDict.dictContentType,
                             cctx->cdict,
-                            params, cctx->pledgedSrcSizePlusOne-1) );
+                            params, cctx->pledgedSrcSizePlusOne-1) , "");
             assert(cctx->streamStage == zcss_load);
             assert(cctx->appliedParams.nbWorkers == 0);
     }   }
@@ -3878,7 +4031,7 @@ size_t ZSTD_compressStream2( ZSTD_CCtx* cctx,
               || (endOp == ZSTD_e_end && flushMin == 0) ) { /* compression completed */
                 ZSTD_CCtx_reset(cctx, ZSTD_reset_session_only);
             }
-            FORWARD_IF_ERROR(flushMin);
+            FORWARD_IF_ERROR(flushMin, "ZSTDMT_compressStream_generic failed");
         } while (forceMaxProgress && flushMin != 0 && output->pos < output->size);
         DEBUGLOG(5, "completed ZSTD_compressStream2 delegating to ZSTDMT_compressStream_generic");
         /* Either we don't require maximum forward progress, we've finished the
@@ -3888,7 +4041,7 @@ size_t ZSTD_compressStream2( ZSTD_CCtx* cctx,
         return flushMin;
     }
 #endif
-    FORWARD_IF_ERROR( ZSTD_compressStream_generic(cctx, output, input, endOp) );
+    FORWARD_IF_ERROR( ZSTD_compressStream_generic(cctx, output, input, endOp) , "");
     DEBUGLOG(5, "completed ZSTD_compressStream2");
     return cctx->outBuffContentSize - cctx->outBuffFlushedSize; /* remaining to flush */
 }
@@ -3912,6 +4065,7 @@ size_t ZSTD_compress2(ZSTD_CCtx* cctx,
                       void* dst, size_t dstCapacity,
                       const void* src, size_t srcSize)
 {
+    DEBUGLOG(4, "ZSTD_compress2 (srcSize=%u)", (unsigned)srcSize);
     ZSTD_CCtx_reset(cctx, ZSTD_reset_session_only);
     {   size_t oPos = 0;
         size_t iPos = 0;
@@ -3919,10 +4073,10 @@ size_t ZSTD_compress2(ZSTD_CCtx* cctx,
                                         dst, dstCapacity, &oPos,
                                         src, srcSize, &iPos,
                                         ZSTD_e_end);
-        FORWARD_IF_ERROR(result);
+        FORWARD_IF_ERROR(result, "ZSTD_compressStream2_simpleArgs failed");
         if (result != 0) {  /* compression not completed, due to lack of output space */
             assert(oPos == dstCapacity);
-            RETURN_ERROR(dstSize_tooSmall);
+            RETURN_ERROR(dstSize_tooSmall, "");
         }
         assert(iPos == srcSize);   /* all input is expected consumed */
         return oPos;
@@ -3944,7 +4098,7 @@ size_t ZSTD_endStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output)
 {
     ZSTD_inBuffer input = { NULL, 0, 0 };
     size_t const remainingToFlush = ZSTD_compressStream2(zcs, output, &input, ZSTD_e_end);
-    FORWARD_IF_ERROR( remainingToFlush );
+    FORWARD_IF_ERROR( remainingToFlush , "ZSTD_compressStream2 failed");
     if (zcs->appliedParams.nbWorkers > 0) return remainingToFlush;   /* minimal estimation */
     /* single thread mode : attempt to calculate remaining to flush more precisely */
     {   size_t const lastBlockSize = zcs->frameEnded ? 0 : ZSTD_BLOCKHEADERSIZE;
@@ -4069,35 +4223,56 @@ static const ZSTD_compressionParameters ZSTD_defaultCParameters[4][ZSTD_MAX_CLEV
 },
 };
 
-/*! ZSTD_getCParams() :
+/*! ZSTD_getCParams_internal() :
  * @return ZSTD_compressionParameters structure for a selected compression level, srcSize and dictSize.
- *  Size values are optional, provide 0 if not known or unused */
-ZSTD_compressionParameters ZSTD_getCParams(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize)
+ *  Note: srcSizeHint 0 means 0, use ZSTD_CONTENTSIZE_UNKNOWN for unknown.
+ *        Use dictSize == 0 for unknown or unused. */
+static ZSTD_compressionParameters ZSTD_getCParams_internal(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize)
 {
-    size_t const addedSize = srcSizeHint ? 0 : 500;
-    U64 const rSize = srcSizeHint+dictSize ? srcSizeHint+dictSize+addedSize : ZSTD_CONTENTSIZE_UNKNOWN;  /* intentional overflow for srcSizeHint == ZSTD_CONTENTSIZE_UNKNOWN */
+    int const unknown = srcSizeHint == ZSTD_CONTENTSIZE_UNKNOWN;
+    size_t const addedSize = unknown && dictSize > 0 ? 500 : 0;
+    U64 const rSize = unknown && dictSize == 0 ? ZSTD_CONTENTSIZE_UNKNOWN : srcSizeHint+dictSize+addedSize;
     U32 const tableID = (rSize <= 256 KB) + (rSize <= 128 KB) + (rSize <= 16 KB);
     int row = compressionLevel;
-    DEBUGLOG(5, "ZSTD_getCParams (cLevel=%i)", compressionLevel);
+    DEBUGLOG(5, "ZSTD_getCParams_internal (cLevel=%i)", compressionLevel);
     if (compressionLevel == 0) row = ZSTD_CLEVEL_DEFAULT;   /* 0 == default */
     if (compressionLevel < 0) row = 0;   /* entry 0 is baseline for fast mode */
     if (compressionLevel > ZSTD_MAX_CLEVEL) row = ZSTD_MAX_CLEVEL;
     {   ZSTD_compressionParameters cp = ZSTD_defaultCParameters[tableID][row];
         if (compressionLevel < 0) cp.targetLength = (unsigned)(-compressionLevel);   /* acceleration factor */
-        return ZSTD_adjustCParams_internal(cp, srcSizeHint, dictSize);               /* refine parameters based on srcSize & dictSize */
+        /* refine parameters based on srcSize & dictSize */
+        return ZSTD_adjustCParams_internal(cp, srcSizeHint, dictSize);
     }
 }
 
+/*! ZSTD_getCParams() :
+ * @return ZSTD_compressionParameters structure for a selected compression level, srcSize and dictSize.
+ *  Size values are optional, provide 0 if not known or unused */
+ZSTD_compressionParameters ZSTD_getCParams(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize)
+{
+    if (srcSizeHint == 0) srcSizeHint = ZSTD_CONTENTSIZE_UNKNOWN;
+    return ZSTD_getCParams_internal(compressionLevel, srcSizeHint, dictSize);
+}
+
 /*! ZSTD_getParams() :
  *  same idea as ZSTD_getCParams()
  * @return a `ZSTD_parameters` structure (instead of `ZSTD_compressionParameters`).
  *  Fields of `ZSTD_frameParameters` are set to default values */
-ZSTD_parameters ZSTD_getParams(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize) {
+static ZSTD_parameters ZSTD_getParams_internal(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize) {
     ZSTD_parameters params;
-    ZSTD_compressionParameters const cParams = ZSTD_getCParams(compressionLevel, srcSizeHint, dictSize);
+    ZSTD_compressionParameters const cParams = ZSTD_getCParams_internal(compressionLevel, srcSizeHint, dictSize);
     DEBUGLOG(5, "ZSTD_getParams (cLevel=%i)", compressionLevel);
     memset(&params, 0, sizeof(params));
     params.cParams = cParams;
     params.fParams.contentSizeFlag = 1;
     return params;
 }
+
+/*! ZSTD_getParams() :
+ *  same idea as ZSTD_getCParams()
+ * @return a `ZSTD_parameters` structure (instead of `ZSTD_compressionParameters`).
+ *  Fields of `ZSTD_frameParameters` are set to default values */
+ZSTD_parameters ZSTD_getParams(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize) {
+    if (srcSizeHint == 0) srcSizeHint = ZSTD_CONTENTSIZE_UNKNOWN;
+    return ZSTD_getParams_internal(compressionLevel, srcSizeHint, dictSize);
+}
diff --git a/lib/compress/zstd_compress_internal.h b/lib/compress/zstd_compress_internal.h
index 14036f873f53..db73f6ce21f2 100644
--- a/lib/compress/zstd_compress_internal.h
+++ b/lib/compress/zstd_compress_internal.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -18,7 +18,7 @@
 /*-*************************************
 *  Dependencies
 ***************************************/
-#include "zstd_internal.h"
+#include "../common/zstd_internal.h"
 #include "zstd_cwksp.h"
 #ifdef ZSTD_MULTITHREAD
 #  include "zstdmt_compress.h"
@@ -166,6 +166,7 @@ typedef struct {
 typedef struct {
     ZSTD_window_t window;   /* State for the window round buffer management */
     ldmEntry_t* hashTable;
+    U32 loadedDictEnd;
     BYTE* bucketOffsets;    /* Next position in bucket to insert entry */
     U64 hashPower;          /* Used to compute the rolling hash.
                              * Depends on ldmParams.minMatchLength */
@@ -249,6 +250,7 @@ struct ZSTD_CCtx_s {
     size_t staticSize;
     SeqCollector seqCollector;
     int isFirstBlock;
+    int initialized;
 
     seqStore_t seqStore;      /* sequences storage ptrs */
     ldmState_t ldmState;      /* long distance matching state */
@@ -324,6 +326,31 @@ MEM_STATIC U32 ZSTD_MLcode(U32 mlBase)
     return (mlBase > 127) ? ZSTD_highbit32(mlBase) + ML_deltaCode : ML_Code[mlBase];
 }
 
+typedef struct repcodes_s {
+    U32 rep[3];
+} repcodes_t;
+
+MEM_STATIC repcodes_t ZSTD_updateRep(U32 const rep[3], U32 const offset, U32 const ll0)
+{
+    repcodes_t newReps;
+    if (offset >= ZSTD_REP_NUM) {  /* full offset */
+        newReps.rep[2] = rep[1];
+        newReps.rep[1] = rep[0];
+        newReps.rep[0] = offset - ZSTD_REP_MOVE;
+    } else {   /* repcode */
+        U32 const repCode = offset + ll0;
+        if (repCode > 0) {  /* note : if repCode==0, no change */
+            U32 const currentOffset = (repCode==ZSTD_REP_NUM) ? (rep[0] - 1) : rep[repCode];
+            newReps.rep[2] = (repCode >= 2) ? rep[1] : rep[2];
+            newReps.rep[1] = rep[0];
+            newReps.rep[0] = currentOffset;
+        } else {   /* repCode == 0 */
+            memcpy(&newReps, rep, sizeof(newReps));
+        }
+    }
+    return newReps;
+}
+
 /* ZSTD_cParam_withinBounds:
  * @return 1 if value is within cParam bounds,
  * 0 otherwise */
@@ -336,6 +363,30 @@ MEM_STATIC int ZSTD_cParam_withinBounds(ZSTD_cParameter cParam, int value)
     return 1;
 }
 
+/* ZSTD_noCompressBlock() :
+ * Writes uncompressed block to dst buffer from given src.
+ * Returns the size of the block */
+MEM_STATIC size_t ZSTD_noCompressBlock (void* dst, size_t dstCapacity, const void* src, size_t srcSize, U32 lastBlock)
+{
+    U32 const cBlockHeader24 = lastBlock + (((U32)bt_raw)<<1) + (U32)(srcSize << 3);
+    RETURN_ERROR_IF(srcSize + ZSTD_blockHeaderSize > dstCapacity,
+                    dstSize_tooSmall, "dst buf too small for uncompressed block");
+    MEM_writeLE24(dst, cBlockHeader24);
+    memcpy((BYTE*)dst + ZSTD_blockHeaderSize, src, srcSize);
+    return ZSTD_blockHeaderSize + srcSize;
+}
+
+MEM_STATIC size_t ZSTD_rleCompressBlock (void* dst, size_t dstCapacity, BYTE src, size_t srcSize, U32 lastBlock)
+{
+    BYTE* const op = (BYTE*)dst;
+    U32 const cBlockHeader = lastBlock + (((U32)bt_rle)<<1) + (U32)(srcSize << 3);
+    RETURN_ERROR_IF(dstCapacity < 4, dstSize_tooSmall, "");
+    MEM_writeLE24(op, cBlockHeader);
+    op[3] = src;
+    return 4;
+}
+
+
 /* ZSTD_minGain() :
  * minimum compression required
  * to generate a compress block or a compressed literals section.
@@ -348,6 +399,21 @@ MEM_STATIC size_t ZSTD_minGain(size_t srcSize, ZSTD_strategy strat)
     return (srcSize >> minlog) + 2;
 }
 
+MEM_STATIC int ZSTD_disableLiteralsCompression(const ZSTD_CCtx_params* cctxParams)
+{
+    switch (cctxParams->literalCompressionMode) {
+    case ZSTD_lcm_huffman:
+        return 0;
+    case ZSTD_lcm_uncompressed:
+        return 1;
+    default:
+        assert(0 /* impossible: pre-validated */);
+        /* fall-through */
+    case ZSTD_lcm_auto:
+        return (cctxParams->cParams.strategy == ZSTD_fast) && (cctxParams->cParams.targetLength > 0);
+    }
+}
+
 /*! ZSTD_safecopyLiterals() :
  *  memcpy() function that won't read beyond more than WILDCOPY_OVERLENGTH bytes past ilimit_w.
  *  Only called when the sequence ends past ilimit_w, so it only needs to be optimized for single
@@ -433,8 +499,7 @@ static unsigned ZSTD_NbCommonBytes (size_t val)
         if (MEM_64bits()) {
 #       if defined(_MSC_VER) && defined(_WIN64)
             unsigned long r = 0;
-            _BitScanForward64( &r, (U64)val );
-            return (unsigned)(r>>3);
+            return _BitScanForward64( &r, (U64)val ) ? (unsigned)(r >> 3) : 0;
 #       elif defined(__GNUC__) && (__GNUC__ >= 4)
             return (__builtin_ctzll((U64)val) >> 3);
 #       else
@@ -451,8 +516,7 @@ static unsigned ZSTD_NbCommonBytes (size_t val)
         } else { /* 32 bits */
 #       if defined(_MSC_VER)
             unsigned long r=0;
-            _BitScanForward( &r, (U32)val );
-            return (unsigned)(r>>3);
+            return _BitScanForward( &r, (U32)val ) ? (unsigned)(r >> 3) : 0;
 #       elif defined(__GNUC__) && (__GNUC__ >= 3)
             return (__builtin_ctz((U32)val) >> 3);
 #       else
@@ -467,8 +531,7 @@ static unsigned ZSTD_NbCommonBytes (size_t val)
         if (MEM_64bits()) {
 #       if defined(_MSC_VER) && defined(_WIN64)
             unsigned long r = 0;
-            _BitScanReverse64( &r, val );
-            return (unsigned)(r>>3);
+            return _BitScanReverse64( &r, val ) ? (unsigned)(r >> 3) : 0;
 #       elif defined(__GNUC__) && (__GNUC__ >= 4)
             return (__builtin_clzll(val) >> 3);
 #       else
@@ -482,8 +545,7 @@ static unsigned ZSTD_NbCommonBytes (size_t val)
         } else { /* 32 bits */
 #       if defined(_MSC_VER)
             unsigned long r = 0;
-            _BitScanReverse( &r, (unsigned long)val );
-            return (unsigned)(r>>3);
+            return _BitScanReverse( &r, (unsigned long)val ) ? (unsigned)(r >> 3) : 0;
 #       elif defined(__GNUC__) && (__GNUC__ >= 3)
             return (__builtin_clz((U32)val) >> 3);
 #       else
@@ -730,7 +792,10 @@ MEM_STATIC U32 ZSTD_window_correctOverflow(ZSTD_window_t* window, U32 cycleLog,
      */
     U32 const cycleMask = (1U << cycleLog) - 1;
     U32 const current = (U32)((BYTE const*)src - window->base);
-    U32 const newCurrent = (current & cycleMask) + maxDist;
+    U32 const currentCycle0 = current & cycleMask;
+    /* Exclude zero so that newCurrent - maxDist >= 1. */
+    U32 const currentCycle1 = currentCycle0 == 0 ? (1U << cycleLog) : currentCycle0;
+    U32 const newCurrent = currentCycle1 + maxDist;
     U32 const correction = current - newCurrent;
     assert((maxDist & cycleMask) == 0);
     assert(current > newCurrent);
@@ -739,8 +804,17 @@ MEM_STATIC U32 ZSTD_window_correctOverflow(ZSTD_window_t* window, U32 cycleLog,
 
     window->base += correction;
     window->dictBase += correction;
-    window->lowLimit -= correction;
-    window->dictLimit -= correction;
+    if (window->lowLimit <= correction) window->lowLimit = 1;
+    else window->lowLimit -= correction;
+    if (window->dictLimit <= correction) window->dictLimit = 1;
+    else window->dictLimit -= correction;
+
+    /* Ensure we can still reference the full window. */
+    assert(newCurrent >= maxDist);
+    assert(newCurrent - maxDist >= 1);
+    /* Ensure that lowLimit and dictLimit didn't underflow. */
+    assert(window->lowLimit <= newCurrent);
+    assert(window->dictLimit <= newCurrent);
 
     DEBUGLOG(4, "Correction of 0x%x bytes to lowLimit=0x%x", correction,
              window->lowLimit);
@@ -844,6 +918,15 @@ ZSTD_checkDictValidity(const ZSTD_window_t* window,
     }   }   }
 }
 
+MEM_STATIC void ZSTD_window_init(ZSTD_window_t* window) {
+    memset(window, 0, sizeof(*window));
+    window->base = (BYTE const*)"";
+    window->dictBase = (BYTE const*)"";
+    window->dictLimit = 1;    /* start from 1, so that 1st position is valid */
+    window->lowLimit = 1;     /* it ensures first and later CCtx usages compress the same */
+    window->nextSrc = window->base + 1;   /* see issue #1241 */
+}
+
 /**
  * ZSTD_window_update():
  * Updates the window by appending [src, src + srcSize) to the window.
@@ -857,6 +940,10 @@ MEM_STATIC U32 ZSTD_window_update(ZSTD_window_t* window,
     BYTE const* const ip = (BYTE const*)src;
     U32 contiguous = 1;
     DEBUGLOG(5, "ZSTD_window_update");
+    if (srcSize == 0)
+        return contiguous;
+    assert(window->base != NULL);
+    assert(window->dictBase != NULL);
     /* Check if blocks follow each other */
     if (src != window->nextSrc) {
         /* not contiguous */
@@ -867,7 +954,7 @@ MEM_STATIC U32 ZSTD_window_update(ZSTD_window_t* window,
         window->dictLimit = (U32)distanceFromBase;
         window->dictBase = window->base;
         window->base = ip - distanceFromBase;
-        // ms->nextToUpdate = window->dictLimit;
+        /* ms->nextToUpdate = window->dictLimit; */
         if (window->dictLimit - window->lowLimit < HASH_READ_SIZE) window->lowLimit = window->dictLimit;   /* too small extDict */
         contiguous = 0;
     }
@@ -883,6 +970,9 @@ MEM_STATIC U32 ZSTD_window_update(ZSTD_window_t* window,
     return contiguous;
 }
 
+/**
+ * Returns the lowest allowed match index. It may either be in the ext-dict or the prefix.
+ */
 MEM_STATIC U32 ZSTD_getLowestMatchIndex(const ZSTD_matchState_t* ms, U32 current, unsigned windowLog)
 {
     U32    const maxDistance = 1U << windowLog;
@@ -893,6 +983,19 @@ MEM_STATIC U32 ZSTD_getLowestMatchIndex(const ZSTD_matchState_t* ms, U32 current
     return matchLowest;
 }
 
+/**
+ * Returns the lowest allowed match index in the prefix.
+ */
+MEM_STATIC U32 ZSTD_getLowestPrefixIndex(const ZSTD_matchState_t* ms, U32 current, unsigned windowLog)
+{
+    U32    const maxDistance = 1U << windowLog;
+    U32    const lowestValid = ms->window.dictLimit;
+    U32    const withinWindow = (current - lowestValid > maxDistance) ? current - maxDistance : lowestValid;
+    U32    const isDictionary = (ms->loadedDictEnd != 0);
+    U32    const matchLowest = isDictionary ? lowestValid : withinWindow;
+    return matchLowest;
+}
+
 
 
 /* debug functions */
@@ -931,6 +1034,21 @@ MEM_STATIC void ZSTD_debugTable(const U32* table, U32 max)
 }
 #endif
 
+/* ===============================================================
+ * Shared internal declarations
+ * These prototypes may be called from sources not in lib/compress
+ * =============================================================== */
+
+/* ZSTD_loadCEntropy() :
+ * dict : must point at beginning of a valid zstd dictionary.
+ * return : size of dictionary header (size of magic number + dict ID + entropy tables)
+ * assumptions : magic number supposed already checked
+ *               and dictSize >= 8 */
+size_t ZSTD_loadCEntropy(ZSTD_compressedBlockState_t* bs, void* workspace,
+                         short* offcodeNCount, unsigned* offcodeMaxValue,
+                         const void* const dict, size_t dictSize);
+
+void ZSTD_reset_compressedBlockState(ZSTD_compressedBlockState_t* bs);
 
 /* ==============================================================
  * Private declarations
@@ -940,6 +1058,7 @@ MEM_STATIC void ZSTD_debugTable(const U32* table, U32 max)
 /* ZSTD_getCParamsFromCCtxParams() :
  * cParams are built depending on compressionLevel, src size hints,
  * LDM and manually set compression parameters.
+ * Note: srcSizeHint == 0 means 0!
  */
 ZSTD_compressionParameters ZSTD_getCParamsFromCCtxParams(
         const ZSTD_CCtx_params* CCtxParams, U64 srcSizeHint, size_t dictSize);
@@ -999,5 +1118,8 @@ size_t ZSTD_writeLastEmptyBlock(void* dst, size_t dstCapacity);
  */
 size_t ZSTD_referenceExternalSequences(ZSTD_CCtx* cctx, rawSeq* seq, size_t nbSeq);
 
+/** ZSTD_cycleLog() :
+ *  condition for correct operation : hashLog > 1 */
+U32 ZSTD_cycleLog(U32 hashLog, ZSTD_strategy strat);
 
 #endif /* ZSTD_COMPRESS_H */
diff --git a/lib/compress/zstd_compress_literals.c b/lib/compress/zstd_compress_literals.c
index 6c133311821e..17e7168d8936 100644
--- a/lib/compress/zstd_compress_literals.c
+++ b/lib/compress/zstd_compress_literals.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -18,7 +18,7 @@ size_t ZSTD_noCompressLiterals (void* dst, size_t dstCapacity, const void* src,
     BYTE* const ostart = (BYTE* const)dst;
     U32   const flSize = 1 + (srcSize>31) + (srcSize>4095);
 
-    RETURN_ERROR_IF(srcSize + flSize > dstCapacity, dstSize_tooSmall);
+    RETURN_ERROR_IF(srcSize + flSize > dstCapacity, dstSize_tooSmall, "");
 
     switch(flSize)
     {
@@ -36,6 +36,7 @@ size_t ZSTD_noCompressLiterals (void* dst, size_t dstCapacity, const void* src,
     }
 
     memcpy(ostart + flSize, src, srcSize);
+    DEBUGLOG(5, "Raw literals: %u -> %u", (U32)srcSize, (U32)(srcSize + flSize));
     return srcSize + flSize;
 }
 
@@ -62,6 +63,7 @@ size_t ZSTD_compressRleLiteralsBlock (void* dst, size_t dstCapacity, const void*
     }
 
     ostart[flSize] = *(const BYTE*)src;
+    DEBUGLOG(5, "RLE literals: %u -> %u", (U32)srcSize, (U32)flSize + 1);
     return flSize+1;
 }
 
@@ -80,8 +82,8 @@ size_t ZSTD_compressLiterals (ZSTD_hufCTables_t const* prevHuf,
     symbolEncodingType_e hType = set_compressed;
     size_t cLitSize;
 
-    DEBUGLOG(5,"ZSTD_compressLiterals (disableLiteralCompression=%i)",
-                disableLiteralCompression);
+    DEBUGLOG(5,"ZSTD_compressLiterals (disableLiteralCompression=%i srcSize=%u)",
+                disableLiteralCompression, (U32)srcSize);
 
     /* Prepare nextEntropy assuming reusing the existing table */
     memcpy(nextHuf, prevHuf, sizeof(*prevHuf));
@@ -102,14 +104,15 @@ size_t ZSTD_compressLiterals (ZSTD_hufCTables_t const* prevHuf,
         cLitSize = singleStream ?
             HUF_compress1X_repeat(
                 ostart+lhSize, dstCapacity-lhSize, src, srcSize,
-                255, 11, entropyWorkspace, entropyWorkspaceSize,
+                HUF_SYMBOLVALUE_MAX, HUF_TABLELOG_DEFAULT, entropyWorkspace, entropyWorkspaceSize,
                 (HUF_CElt*)nextHuf->CTable, &repeat, preferRepeat, bmi2) :
             HUF_compress4X_repeat(
                 ostart+lhSize, dstCapacity-lhSize, src, srcSize,
-                255, 11, entropyWorkspace, entropyWorkspaceSize,
+                HUF_SYMBOLVALUE_MAX, HUF_TABLELOG_DEFAULT, entropyWorkspace, entropyWorkspaceSize,
                 (HUF_CElt*)nextHuf->CTable, &repeat, preferRepeat, bmi2);
         if (repeat != HUF_repeat_none) {
             /* reused the existing table */
+            DEBUGLOG(5, "Reusing previous huffman table");
             hType = set_repeat;
         }
     }
@@ -150,5 +153,6 @@ size_t ZSTD_compressLiterals (ZSTD_hufCTables_t const* prevHuf,
     default:  /* not possible : lhSize is {3,4,5} */
         assert(0);
     }
+    DEBUGLOG(5, "Compressed literals: %u -> %u", (U32)srcSize, (U32)(lhSize+cLitSize));
     return lhSize+cLitSize;
 }
diff --git a/lib/compress/zstd_compress_literals.h b/lib/compress/zstd_compress_literals.h
index 97273d7cfdb5..8b0870574326 100644
--- a/lib/compress/zstd_compress_literals.h
+++ b/lib/compress/zstd_compress_literals.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/lib/compress/zstd_compress_sequences.c b/lib/compress/zstd_compress_sequences.c
index 0ff7a26823b0..f9f8097c839b 100644
--- a/lib/compress/zstd_compress_sequences.c
+++ b/lib/compress/zstd_compress_sequences.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -60,7 +60,7 @@ static size_t ZSTD_NCountCost(unsigned const* count, unsigned const max,
     BYTE wksp[FSE_NCOUNTBOUND];
     S16 norm[MaxSeq + 1];
     const U32 tableLog = FSE_optimalTableLog(FSELog, nbSeq, max);
-    FORWARD_IF_ERROR(FSE_normalizeCount(norm, tableLog, count, nbSeq, max));
+    FORWARD_IF_ERROR(FSE_normalizeCount(norm, tableLog, count, nbSeq, max), "");
     return FSE_writeNCount(wksp, sizeof(wksp), norm, max, tableLog);
 }
 
@@ -86,7 +86,7 @@ static size_t ZSTD_entropyCost(unsigned const* count, unsigned const max, size_t
  * Returns the cost in bits of encoding the distribution in count using ctable.
  * Returns an error if ctable cannot represent all the symbols in count.
  */
-static size_t ZSTD_fseBitCost(
+size_t ZSTD_fseBitCost(
     FSE_CTable const* ctable,
     unsigned const* count,
     unsigned const max)
@@ -96,18 +96,22 @@ static size_t ZSTD_fseBitCost(
     unsigned s;
     FSE_CState_t cstate;
     FSE_initCState(&cstate, ctable);
-    RETURN_ERROR_IF(ZSTD_getFSEMaxSymbolValue(ctable) < max, GENERIC,
-                    "Repeat FSE_CTable has maxSymbolValue %u < %u",
+    if (ZSTD_getFSEMaxSymbolValue(ctable) < max) {
+        DEBUGLOG(5, "Repeat FSE_CTable has maxSymbolValue %u < %u",
                     ZSTD_getFSEMaxSymbolValue(ctable), max);
+        return ERROR(GENERIC);
+    }
     for (s = 0; s <= max; ++s) {
         unsigned const tableLog = cstate.stateLog;
         unsigned const badCost = (tableLog + 1) << kAccuracyLog;
         unsigned const bitCost = FSE_bitCost(cstate.symbolTT, tableLog, s, kAccuracyLog);
         if (count[s] == 0)
             continue;
-        RETURN_ERROR_IF(bitCost >= badCost, GENERIC,
-                        "Repeat FSE_CTable has Prob[%u] == 0", s);
-        cost += count[s] * bitCost;
+        if (bitCost >= badCost) {
+            DEBUGLOG(5, "Repeat FSE_CTable has Prob[%u] == 0", s);
+            return ERROR(GENERIC);
+        }
+        cost += (size_t)count[s] * bitCost;
     }
     return cost >> kAccuracyLog;
 }
@@ -117,15 +121,15 @@ static size_t ZSTD_fseBitCost(
  * table described by norm. The max symbol support by norm is assumed >= max.
  * norm must be valid for every symbol with non-zero probability in count.
  */
-static size_t ZSTD_crossEntropyCost(short const* norm, unsigned accuracyLog,
-                                    unsigned const* count, unsigned const max)
+size_t ZSTD_crossEntropyCost(short const* norm, unsigned accuracyLog,
+                             unsigned const* count, unsigned const max)
 {
     unsigned const shift = 8 - accuracyLog;
     size_t cost = 0;
     unsigned s;
     assert(accuracyLog <= 8);
     for (s = 0; s <= max; ++s) {
-        unsigned const normAcc = norm[s] != -1 ? norm[s] : 1;
+        unsigned const normAcc = (norm[s] != -1) ? (unsigned)norm[s] : 1;
         unsigned const norm256 = normAcc << shift;
         assert(norm256 > 0);
         assert(norm256 < 256);
@@ -230,15 +234,15 @@ ZSTD_buildCTable(void* dst, size_t dstCapacity,
 
     switch (type) {
     case set_rle:
-        FORWARD_IF_ERROR(FSE_buildCTable_rle(nextCTable, (BYTE)max));
-        RETURN_ERROR_IF(dstCapacity==0, dstSize_tooSmall);
+        FORWARD_IF_ERROR(FSE_buildCTable_rle(nextCTable, (BYTE)max), "");
+        RETURN_ERROR_IF(dstCapacity==0, dstSize_tooSmall, "not enough space");
         *op = codeTable[0];
         return 1;
     case set_repeat:
         memcpy(nextCTable, prevCTable, prevCTableSize);
         return 0;
     case set_basic:
-        FORWARD_IF_ERROR(FSE_buildCTable_wksp(nextCTable, defaultNorm, defaultMax, defaultNormLog, entropyWorkspace, entropyWorkspaceSize));  /* note : could be pre-calculated */
+        FORWARD_IF_ERROR(FSE_buildCTable_wksp(nextCTable, defaultNorm, defaultMax, defaultNormLog, entropyWorkspace, entropyWorkspaceSize), "");  /* note : could be pre-calculated */
         return 0;
     case set_compressed: {
         S16 norm[MaxSeq + 1];
@@ -249,14 +253,14 @@ ZSTD_buildCTable(void* dst, size_t dstCapacity,
             nbSeq_1--;
         }
         assert(nbSeq_1 > 1);
-        FORWARD_IF_ERROR(FSE_normalizeCount(norm, tableLog, count, nbSeq_1, max));
+        FORWARD_IF_ERROR(FSE_normalizeCount(norm, tableLog, count, nbSeq_1, max), "");
         {   size_t const NCountSize = FSE_writeNCount(op, oend - op, norm, max, tableLog);   /* overflow protected */
-            FORWARD_IF_ERROR(NCountSize);
-            FORWARD_IF_ERROR(FSE_buildCTable_wksp(nextCTable, norm, max, tableLog, entropyWorkspace, entropyWorkspaceSize));
+            FORWARD_IF_ERROR(NCountSize, "FSE_writeNCount failed");
+            FORWARD_IF_ERROR(FSE_buildCTable_wksp(nextCTable, norm, max, tableLog, entropyWorkspace, entropyWorkspaceSize), "");
             return NCountSize;
         }
     }
-    default: assert(0); RETURN_ERROR(GENERIC);
+    default: assert(0); RETURN_ERROR(GENERIC, "impossible to reach");
     }
 }
 
@@ -290,7 +294,7 @@ ZSTD_encodeSequences_body(
     if (MEM_32bits()) BIT_flushBits(&blockStream);
     if (longOffsets) {
         U32 const ofBits = ofCodeTable[nbSeq-1];
-        int const extraBits = ofBits - MIN(ofBits, STREAM_ACCUMULATOR_MIN-1);
+        unsigned const extraBits = ofBits - MIN(ofBits, STREAM_ACCUMULATOR_MIN-1);
         if (extraBits) {
             BIT_addBits(&blockStream, sequences[nbSeq-1].offset, extraBits);
             BIT_flushBits(&blockStream);
@@ -327,7 +331,7 @@ ZSTD_encodeSequences_body(
             BIT_addBits(&blockStream, sequences[n].matchLength, mlBits);
             if (MEM_32bits() || (ofBits+mlBits+llBits > 56)) BIT_flushBits(&blockStream);
             if (longOffsets) {
-                int const extraBits = ofBits - MIN(ofBits, STREAM_ACCUMULATOR_MIN-1);
+                unsigned const extraBits = ofBits - MIN(ofBits, STREAM_ACCUMULATOR_MIN-1);
                 if (extraBits) {
                     BIT_addBits(&blockStream, sequences[n].offset, extraBits);
                     BIT_flushBits(&blockStream);                            /* (7)*/
diff --git a/lib/compress/zstd_compress_sequences.h b/lib/compress/zstd_compress_sequences.h
index 57e8e367b093..68c6f9a5acd8 100644
--- a/lib/compress/zstd_compress_sequences.h
+++ b/lib/compress/zstd_compress_sequences.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -11,8 +11,8 @@
 #ifndef ZSTD_COMPRESS_SEQUENCES_H
 #define ZSTD_COMPRESS_SEQUENCES_H
 
-#include "fse.h" /* FSE_repeat, FSE_CTable */
-#include "zstd_internal.h" /* symbolEncodingType_e, ZSTD_strategy */
+#include "../common/fse.h" /* FSE_repeat, FSE_CTable */
+#include "../common/zstd_internal.h" /* symbolEncodingType_e, ZSTD_strategy */
 
 typedef enum {
     ZSTD_defaultDisallowed = 0,
@@ -44,4 +44,11 @@ size_t ZSTD_encodeSequences(
             FSE_CTable const* CTable_LitLength, BYTE const* llCodeTable,
             seqDef const* sequences, size_t nbSeq, int longOffsets, int bmi2);
 
+size_t ZSTD_fseBitCost(
+    FSE_CTable const* ctable,
+    unsigned const* count,
+    unsigned const max);
+
+size_t ZSTD_crossEntropyCost(short const* norm, unsigned accuracyLog,
+                             unsigned const* count, unsigned const max);
 #endif /* ZSTD_COMPRESS_SEQUENCES_H */
diff --git a/lib/compress/zstd_compress_superblock.c b/lib/compress/zstd_compress_superblock.c
new file mode 100644
index 000000000000..b693866c0ac1
--- /dev/null
+++ b/lib/compress/zstd_compress_superblock.c
@@ -0,0 +1,845 @@
+/*
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+ /*-*************************************
+ *  Dependencies
+ ***************************************/
+#include "zstd_compress_superblock.h"
+
+#include "../common/zstd_internal.h"  /* ZSTD_getSequenceLength */
+#include "hist.h"                     /* HIST_countFast_wksp */
+#include "zstd_compress_internal.h"
+#include "zstd_compress_sequences.h"
+#include "zstd_compress_literals.h"
+
+/*-*************************************
+*  Superblock entropy buffer structs
+***************************************/
+/** ZSTD_hufCTablesMetadata_t :
+ *  Stores Literals Block Type for a super-block in hType, and
+ *  huffman tree description in hufDesBuffer.
+ *  hufDesSize refers to the size of huffman tree description in bytes.
+ *  This metadata is populated in ZSTD_buildSuperBlockEntropy_literal() */
+typedef struct {
+    symbolEncodingType_e hType;
+    BYTE hufDesBuffer[500]; /* TODO give name to this value */
+    size_t hufDesSize;
+} ZSTD_hufCTablesMetadata_t;
+
+/** ZSTD_fseCTablesMetadata_t :
+ *  Stores symbol compression modes for a super-block in {ll, ol, ml}Type, and
+ *  fse tables in fseTablesBuffer.
+ *  fseTablesSize refers to the size of fse tables in bytes.
+ *  This metadata is populated in ZSTD_buildSuperBlockEntropy_sequences() */
+typedef struct {
+    symbolEncodingType_e llType;
+    symbolEncodingType_e ofType;
+    symbolEncodingType_e mlType;
+    BYTE fseTablesBuffer[500]; /* TODO give name to this value */
+    size_t fseTablesSize;
+    size_t lastCountSize; /* This is to account for bug in 1.3.4. More detail in ZSTD_compressSubBlock_sequences() */
+} ZSTD_fseCTablesMetadata_t;
+
+typedef struct {
+    ZSTD_hufCTablesMetadata_t hufMetadata;
+    ZSTD_fseCTablesMetadata_t fseMetadata;
+} ZSTD_entropyCTablesMetadata_t;
+
+
+/** ZSTD_buildSuperBlockEntropy_literal() :
+ *  Builds entropy for the super-block literals.
+ *  Stores literals block type (raw, rle, compressed, repeat) and
+ *  huffman description table to hufMetadata.
+ *  @return : size of huffman description table or error code */
+static size_t ZSTD_buildSuperBlockEntropy_literal(void* const src, size_t srcSize,
+                                            const ZSTD_hufCTables_t* prevHuf,
+                                                  ZSTD_hufCTables_t* nextHuf,
+                                                  ZSTD_hufCTablesMetadata_t* hufMetadata,
+                                                  const int disableLiteralsCompression,
+                                                  void* workspace, size_t wkspSize)
+{
+    BYTE* const wkspStart = (BYTE*)workspace;
+    BYTE* const wkspEnd = wkspStart + wkspSize;
+    BYTE* const countWkspStart = wkspStart;
+    unsigned* const countWksp = (unsigned*)workspace;
+    const size_t countWkspSize = (HUF_SYMBOLVALUE_MAX + 1) * sizeof(unsigned);
+    BYTE* const nodeWksp = countWkspStart + countWkspSize;
+    const size_t nodeWkspSize = wkspEnd-nodeWksp;
+    unsigned maxSymbolValue = 255;
+    unsigned huffLog = HUF_TABLELOG_DEFAULT;
+    HUF_repeat repeat = prevHuf->repeatMode;
+
+    DEBUGLOG(5, "ZSTD_buildSuperBlockEntropy_literal (srcSize=%zu)", srcSize);
+
+    /* Prepare nextEntropy assuming reusing the existing table */
+    memcpy(nextHuf, prevHuf, sizeof(*prevHuf));
+
+    if (disableLiteralsCompression) {
+        DEBUGLOG(5, "set_basic - disabled");
+        hufMetadata->hType = set_basic;
+        return 0;
+    }
+
+    /* small ? don't even attempt compression (speed opt) */
+#   define COMPRESS_LITERALS_SIZE_MIN 63
+    {   size_t const minLitSize = (prevHuf->repeatMode == HUF_repeat_valid) ? 6 : COMPRESS_LITERALS_SIZE_MIN;
+        if (srcSize <= minLitSize) {
+            DEBUGLOG(5, "set_basic - too small");
+            hufMetadata->hType = set_basic;
+            return 0;
+        }
+    }
+
+    /* Scan input and build symbol stats */
+    {   size_t const largest = HIST_count_wksp (countWksp, &maxSymbolValue, (const BYTE*)src, srcSize, workspace, wkspSize);
+        FORWARD_IF_ERROR(largest, "HIST_count_wksp failed");
+        if (largest == srcSize) {
+            DEBUGLOG(5, "set_rle");
+            hufMetadata->hType = set_rle;
+            return 0;
+        }
+        if (largest <= (srcSize >> 7)+4) {
+            DEBUGLOG(5, "set_basic - no gain");
+            hufMetadata->hType = set_basic;
+            return 0;
+        }
+    }
+
+    /* Validate the previous Huffman table */
+    if (repeat == HUF_repeat_check && !HUF_validateCTable((HUF_CElt const*)prevHuf->CTable, countWksp, maxSymbolValue)) {
+        repeat = HUF_repeat_none;
+    }
+
+    /* Build Huffman Tree */
+    memset(nextHuf->CTable, 0, sizeof(nextHuf->CTable));
+    huffLog = HUF_optimalTableLog(huffLog, srcSize, maxSymbolValue);
+    {   size_t const maxBits = HUF_buildCTable_wksp((HUF_CElt*)nextHuf->CTable, countWksp,
+                                                    maxSymbolValue, huffLog,
+                                                    nodeWksp, nodeWkspSize);
+        FORWARD_IF_ERROR(maxBits, "HUF_buildCTable_wksp");
+        huffLog = (U32)maxBits;
+        {   /* Build and write the CTable */
+            size_t const newCSize = HUF_estimateCompressedSize(
+                    (HUF_CElt*)nextHuf->CTable, countWksp, maxSymbolValue);
+            size_t const hSize = HUF_writeCTable(
+                    hufMetadata->hufDesBuffer, sizeof(hufMetadata->hufDesBuffer),
+                    (HUF_CElt*)nextHuf->CTable, maxSymbolValue, huffLog);
+            /* Check against repeating the previous CTable */
+            if (repeat != HUF_repeat_none) {
+                size_t const oldCSize = HUF_estimateCompressedSize(
+                        (HUF_CElt const*)prevHuf->CTable, countWksp, maxSymbolValue);
+                if (oldCSize < srcSize && (oldCSize <= hSize + newCSize || hSize + 12 >= srcSize)) {
+                    DEBUGLOG(5, "set_repeat - smaller");
+                    memcpy(nextHuf, prevHuf, sizeof(*prevHuf));
+                    hufMetadata->hType = set_repeat;
+                    return 0;
+                }
+            }
+            if (newCSize + hSize >= srcSize) {
+                DEBUGLOG(5, "set_basic - no gains");
+                memcpy(nextHuf, prevHuf, sizeof(*prevHuf));
+                hufMetadata->hType = set_basic;
+                return 0;
+            }
+            DEBUGLOG(5, "set_compressed (hSize=%u)", (U32)hSize);
+            hufMetadata->hType = set_compressed;
+            nextHuf->repeatMode = HUF_repeat_check;
+            return hSize;
+        }
+    }
+}
+
+/** ZSTD_buildSuperBlockEntropy_sequences() :
+ *  Builds entropy for the super-block sequences.
+ *  Stores symbol compression modes and fse table to fseMetadata.
+ *  @return : size of fse tables or error code */
+static size_t ZSTD_buildSuperBlockEntropy_sequences(seqStore_t* seqStorePtr,
+                                              const ZSTD_fseCTables_t* prevEntropy,
+                                                    ZSTD_fseCTables_t* nextEntropy,
+                                              const ZSTD_CCtx_params* cctxParams,
+                                                    ZSTD_fseCTablesMetadata_t* fseMetadata,
+                                                    void* workspace, size_t wkspSize)
+{
+    BYTE* const wkspStart = (BYTE*)workspace;
+    BYTE* const wkspEnd = wkspStart + wkspSize;
+    BYTE* const countWkspStart = wkspStart;
+    unsigned* const countWksp = (unsigned*)workspace;
+    const size_t countWkspSize = (MaxSeq + 1) * sizeof(unsigned);
+    BYTE* const cTableWksp = countWkspStart + countWkspSize;
+    const size_t cTableWkspSize = wkspEnd-cTableWksp;
+    ZSTD_strategy const strategy = cctxParams->cParams.strategy;
+    FSE_CTable* CTable_LitLength = nextEntropy->litlengthCTable;
+    FSE_CTable* CTable_OffsetBits = nextEntropy->offcodeCTable;
+    FSE_CTable* CTable_MatchLength = nextEntropy->matchlengthCTable;
+    const BYTE* const ofCodeTable = seqStorePtr->ofCode;
+    const BYTE* const llCodeTable = seqStorePtr->llCode;
+    const BYTE* const mlCodeTable = seqStorePtr->mlCode;
+    size_t const nbSeq = seqStorePtr->sequences - seqStorePtr->sequencesStart;
+    BYTE* const ostart = fseMetadata->fseTablesBuffer;
+    BYTE* const oend = ostart + sizeof(fseMetadata->fseTablesBuffer);
+    BYTE* op = ostart;
+
+    assert(cTableWkspSize >= (1 << MaxFSELog) * sizeof(FSE_FUNCTION_TYPE));
+    DEBUGLOG(5, "ZSTD_buildSuperBlockEntropy_sequences (nbSeq=%zu)", nbSeq);
+    memset(workspace, 0, wkspSize);
+
+    fseMetadata->lastCountSize = 0;
+    /* convert length/distances into codes */
+    ZSTD_seqToCodes(seqStorePtr);
+    /* build CTable for Literal Lengths */
+    {   U32 LLtype;
+        unsigned max = MaxLL;
+        size_t const mostFrequent = HIST_countFast_wksp(countWksp, &max, llCodeTable, nbSeq, workspace, wkspSize);  /* can't fail */
+        DEBUGLOG(5, "Building LL table");
+        nextEntropy->litlength_repeatMode = prevEntropy->litlength_repeatMode;
+        LLtype = ZSTD_selectEncodingType(&nextEntropy->litlength_repeatMode,
+                                        countWksp, max, mostFrequent, nbSeq,
+                                        LLFSELog, prevEntropy->litlengthCTable,
+                                        LL_defaultNorm, LL_defaultNormLog,
+                                        ZSTD_defaultAllowed, strategy);
+        assert(set_basic < set_compressed && set_rle < set_compressed);
+        assert(!(LLtype < set_compressed && nextEntropy->litlength_repeatMode != FSE_repeat_none)); /* We don't copy tables */
+        {   size_t const countSize = ZSTD_buildCTable(op, oend - op, CTable_LitLength, LLFSELog, (symbolEncodingType_e)LLtype,
+                                                    countWksp, max, llCodeTable, nbSeq, LL_defaultNorm, LL_defaultNormLog, MaxLL,
+                                                    prevEntropy->litlengthCTable, sizeof(prevEntropy->litlengthCTable),
+                                                    cTableWksp, cTableWkspSize);
+            FORWARD_IF_ERROR(countSize, "ZSTD_buildCTable for LitLens failed");
+            if (LLtype == set_compressed)
+                fseMetadata->lastCountSize = countSize;
+            op += countSize;
+            fseMetadata->llType = (symbolEncodingType_e) LLtype;
+    }   }
+    /* build CTable for Offsets */
+    {   U32 Offtype;
+        unsigned max = MaxOff;
+        size_t const mostFrequent = HIST_countFast_wksp(countWksp, &max, ofCodeTable, nbSeq, workspace, wkspSize);  /* can't fail */
+        /* We can only use the basic table if max <= DefaultMaxOff, otherwise the offsets are too large */
+        ZSTD_defaultPolicy_e const defaultPolicy = (max <= DefaultMaxOff) ? ZSTD_defaultAllowed : ZSTD_defaultDisallowed;
+        DEBUGLOG(5, "Building OF table");
+        nextEntropy->offcode_repeatMode = prevEntropy->offcode_repeatMode;
+        Offtype = ZSTD_selectEncodingType(&nextEntropy->offcode_repeatMode,
+                                        countWksp, max, mostFrequent, nbSeq,
+                                        OffFSELog, prevEntropy->offcodeCTable,
+                                        OF_defaultNorm, OF_defaultNormLog,
+                                        defaultPolicy, strategy);
+        assert(!(Offtype < set_compressed && nextEntropy->offcode_repeatMode != FSE_repeat_none)); /* We don't copy tables */
+        {   size_t const countSize = ZSTD_buildCTable(op, oend - op, CTable_OffsetBits, OffFSELog, (symbolEncodingType_e)Offtype,
+                                                    countWksp, max, ofCodeTable, nbSeq, OF_defaultNorm, OF_defaultNormLog, DefaultMaxOff,
+                                                    prevEntropy->offcodeCTable, sizeof(prevEntropy->offcodeCTable),
+                                                    cTableWksp, cTableWkspSize);
+            FORWARD_IF_ERROR(countSize, "ZSTD_buildCTable for Offsets failed");
+            if (Offtype == set_compressed)
+                fseMetadata->lastCountSize = countSize;
+            op += countSize;
+            fseMetadata->ofType = (symbolEncodingType_e) Offtype;
+    }   }
+    /* build CTable for MatchLengths */
+    {   U32 MLtype;
+        unsigned max = MaxML;
+        size_t const mostFrequent = HIST_countFast_wksp(countWksp, &max, mlCodeTable, nbSeq, workspace, wkspSize);   /* can't fail */
+        DEBUGLOG(5, "Building ML table (remaining space : %i)", (int)(oend-op));
+        nextEntropy->matchlength_repeatMode = prevEntropy->matchlength_repeatMode;
+        MLtype = ZSTD_selectEncodingType(&nextEntropy->matchlength_repeatMode,
+                                        countWksp, max, mostFrequent, nbSeq,
+                                        MLFSELog, prevEntropy->matchlengthCTable,
+                                        ML_defaultNorm, ML_defaultNormLog,
+                                        ZSTD_defaultAllowed, strategy);
+        assert(!(MLtype < set_compressed && nextEntropy->matchlength_repeatMode != FSE_repeat_none)); /* We don't copy tables */
+        {   size_t const countSize = ZSTD_buildCTable(op, oend - op, CTable_MatchLength, MLFSELog, (symbolEncodingType_e)MLtype,
+                                                    countWksp, max, mlCodeTable, nbSeq, ML_defaultNorm, ML_defaultNormLog, MaxML,
+                                                    prevEntropy->matchlengthCTable, sizeof(prevEntropy->matchlengthCTable),
+                                                    cTableWksp, cTableWkspSize);
+            FORWARD_IF_ERROR(countSize, "ZSTD_buildCTable for MatchLengths failed");
+            if (MLtype == set_compressed)
+                fseMetadata->lastCountSize = countSize;
+            op += countSize;
+            fseMetadata->mlType = (symbolEncodingType_e) MLtype;
+    }   }
+    assert((size_t) (op-ostart) <= sizeof(fseMetadata->fseTablesBuffer));
+    return op-ostart;
+}
+
+
+/** ZSTD_buildSuperBlockEntropy() :
+ *  Builds entropy for the super-block.
+ *  @return : 0 on success or error code */
+static size_t
+ZSTD_buildSuperBlockEntropy(seqStore_t* seqStorePtr,
+                      const ZSTD_entropyCTables_t* prevEntropy,
+                            ZSTD_entropyCTables_t* nextEntropy,
+                      const ZSTD_CCtx_params* cctxParams,
+                            ZSTD_entropyCTablesMetadata_t* entropyMetadata,
+                            void* workspace, size_t wkspSize)
+{
+    size_t const litSize = seqStorePtr->lit - seqStorePtr->litStart;
+    DEBUGLOG(5, "ZSTD_buildSuperBlockEntropy");
+    entropyMetadata->hufMetadata.hufDesSize =
+        ZSTD_buildSuperBlockEntropy_literal(seqStorePtr->litStart, litSize,
+                                            &prevEntropy->huf, &nextEntropy->huf,
+                                            &entropyMetadata->hufMetadata,
+                                            ZSTD_disableLiteralsCompression(cctxParams),
+                                            workspace, wkspSize);
+    FORWARD_IF_ERROR(entropyMetadata->hufMetadata.hufDesSize, "ZSTD_buildSuperBlockEntropy_literal failed");
+    entropyMetadata->fseMetadata.fseTablesSize =
+        ZSTD_buildSuperBlockEntropy_sequences(seqStorePtr,
+                                              &prevEntropy->fse, &nextEntropy->fse,
+                                              cctxParams,
+                                              &entropyMetadata->fseMetadata,
+                                              workspace, wkspSize);
+    FORWARD_IF_ERROR(entropyMetadata->fseMetadata.fseTablesSize, "ZSTD_buildSuperBlockEntropy_sequences failed");
+    return 0;
+}
+
+/** ZSTD_compressSubBlock_literal() :
+ *  Compresses literals section for a sub-block.
+ *  When we have to write the Huffman table we will sometimes choose a header
+ *  size larger than necessary. This is because we have to pick the header size
+ *  before we know the table size + compressed size, so we have a bound on the
+ *  table size. If we guessed incorrectly, we fall back to uncompressed literals.
+ *
+ *  We write the header when writeEntropy=1 and set entropyWrriten=1 when we succeeded
+ *  in writing the header, otherwise it is set to 0.
+ *
+ *  hufMetadata->hType has literals block type info.
+ *      If it is set_basic, all sub-blocks literals section will be Raw_Literals_Block.
+ *      If it is set_rle, all sub-blocks literals section will be RLE_Literals_Block.
+ *      If it is set_compressed, first sub-block's literals section will be Compressed_Literals_Block
+ *      If it is set_compressed, first sub-block's literals section will be Treeless_Literals_Block
+ *      and the following sub-blocks' literals sections will be Treeless_Literals_Block.
+ *  @return : compressed size of literals section of a sub-block
+ *            Or 0 if it unable to compress.
+ *            Or error code */
+static size_t ZSTD_compressSubBlock_literal(const HUF_CElt* hufTable,
+                                    const ZSTD_hufCTablesMetadata_t* hufMetadata,
+                                    const BYTE* literals, size_t litSize,
+                                    void* dst, size_t dstSize,
+                                    const int bmi2, int writeEntropy, int* entropyWritten)
+{
+    size_t const header = writeEntropy ? 200 : 0;
+    size_t const lhSize = 3 + (litSize >= (1 KB - header)) + (litSize >= (16 KB - header));
+    BYTE* const ostart = (BYTE*)dst;
+    BYTE* const oend = ostart + dstSize;
+    BYTE* op = ostart + lhSize;
+    U32 const singleStream = lhSize == 3;
+    symbolEncodingType_e hType = writeEntropy ? hufMetadata->hType : set_repeat;
+    size_t cLitSize = 0;
+
+    (void)bmi2; /* TODO bmi2... */
+
+    DEBUGLOG(5, "ZSTD_compressSubBlock_literal (litSize=%zu, lhSize=%zu, writeEntropy=%d)", litSize, lhSize, writeEntropy);
+
+    *entropyWritten = 0;
+    if (litSize == 0 || hufMetadata->hType == set_basic) {
+      DEBUGLOG(5, "ZSTD_compressSubBlock_literal using raw literal");
+      return ZSTD_noCompressLiterals(dst, dstSize, literals, litSize);
+    } else if (hufMetadata->hType == set_rle) {
+      DEBUGLOG(5, "ZSTD_compressSubBlock_literal using rle literal");
+      return ZSTD_compressRleLiteralsBlock(dst, dstSize, literals, litSize);
+    }
+
+    assert(litSize > 0);
+    assert(hufMetadata->hType == set_compressed || hufMetadata->hType == set_repeat);
+
+    if (writeEntropy && hufMetadata->hType == set_compressed) {
+        memcpy(op, hufMetadata->hufDesBuffer, hufMetadata->hufDesSize);
+        op += hufMetadata->hufDesSize;
+        cLitSize += hufMetadata->hufDesSize;
+        DEBUGLOG(5, "ZSTD_compressSubBlock_literal (hSize=%zu)", hufMetadata->hufDesSize);
+    }
+
+    /* TODO bmi2 */
+    {   const size_t cSize = singleStream ? HUF_compress1X_usingCTable(op, oend-op, literals, litSize, hufTable)
+                                          : HUF_compress4X_usingCTable(op, oend-op, literals, litSize, hufTable);
+        op += cSize;
+        cLitSize += cSize;
+        if (cSize == 0 || ERR_isError(cSize)) {
+            DEBUGLOG(5, "Failed to write entropy tables %s", ZSTD_getErrorName(cSize));
+            return 0;
+        }
+        /* If we expand and we aren't writing a header then emit uncompressed */
+        if (!writeEntropy && cLitSize >= litSize) {
+            DEBUGLOG(5, "ZSTD_compressSubBlock_literal using raw literal because uncompressible");
+            return ZSTD_noCompressLiterals(dst, dstSize, literals, litSize);
+        }
+        /* If we are writing headers then allow expansion that doesn't change our header size. */
+        if (lhSize < (size_t)(3 + (cLitSize >= 1 KB) + (cLitSize >= 16 KB))) {
+            assert(cLitSize > litSize);
+            DEBUGLOG(5, "Literals expanded beyond allowed header size");
+            return ZSTD_noCompressLiterals(dst, dstSize, literals, litSize);
+        }
+        DEBUGLOG(5, "ZSTD_compressSubBlock_literal (cSize=%zu)", cSize);
+    }
+
+    /* Build header */
+    switch(lhSize)
+    {
+    case 3: /* 2 - 2 - 10 - 10 */
+        {   U32 const lhc = hType + ((!singleStream) << 2) + ((U32)litSize<<4) + ((U32)cLitSize<<14);
+            MEM_writeLE24(ostart, lhc);
+            break;
+        }
+    case 4: /* 2 - 2 - 14 - 14 */
+        {   U32 const lhc = hType + (2 << 2) + ((U32)litSize<<4) + ((U32)cLitSize<<18);
+            MEM_writeLE32(ostart, lhc);
+            break;
+        }
+    case 5: /* 2 - 2 - 18 - 18 */
+        {   U32 const lhc = hType + (3 << 2) + ((U32)litSize<<4) + ((U32)cLitSize<<22);
+            MEM_writeLE32(ostart, lhc);
+            ostart[4] = (BYTE)(cLitSize >> 10);
+            break;
+        }
+    default:  /* not possible : lhSize is {3,4,5} */
+        assert(0);
+    }
+    *entropyWritten = 1;
+    DEBUGLOG(5, "Compressed literals: %u -> %u", (U32)litSize, (U32)(op-ostart));
+    return op-ostart;
+}
+
+static size_t ZSTD_seqDecompressedSize(seqStore_t const* seqStore, const seqDef* sequences, size_t nbSeq, size_t litSize, int lastSequence) {
+    const seqDef* const sstart = sequences;
+    const seqDef* const send = sequences + nbSeq;
+    const seqDef* sp = sstart;
+    size_t matchLengthSum = 0;
+    size_t litLengthSum = 0;
+    while (send-sp > 0) {
+        ZSTD_sequenceLength const seqLen = ZSTD_getSequenceLength(seqStore, sp);
+        litLengthSum += seqLen.litLength;
+        matchLengthSum += seqLen.matchLength;
+        sp++;
+    }
+    assert(litLengthSum <= litSize);
+    if (!lastSequence) {
+        assert(litLengthSum == litSize);
+    }
+    return matchLengthSum + litSize;
+}
+
+/** ZSTD_compressSubBlock_sequences() :
+ *  Compresses sequences section for a sub-block.
+ *  fseMetadata->llType, fseMetadata->ofType, and fseMetadata->mlType have
+ *  symbol compression modes for the super-block.
+ *  The first successfully compressed block will have these in its header.
+ *  We set entropyWritten=1 when we succeed in compressing the sequences.
+ *  The following sub-blocks will always have repeat mode.
+ *  @return : compressed size of sequences section of a sub-block
+ *            Or 0 if it is unable to compress
+ *            Or error code. */
+static size_t ZSTD_compressSubBlock_sequences(const ZSTD_fseCTables_t* fseTables,
+                                              const ZSTD_fseCTablesMetadata_t* fseMetadata,
+                                              const seqDef* sequences, size_t nbSeq,
+                                              const BYTE* llCode, const BYTE* mlCode, const BYTE* ofCode,
+                                              const ZSTD_CCtx_params* cctxParams,
+                                              void* dst, size_t dstCapacity,
+                                              const int bmi2, int writeEntropy, int* entropyWritten)
+{
+    const int longOffsets = cctxParams->cParams.windowLog > STREAM_ACCUMULATOR_MIN;
+    BYTE* const ostart = (BYTE*)dst;
+    BYTE* const oend = ostart + dstCapacity;
+    BYTE* op = ostart;
+    BYTE* seqHead;
+
+    DEBUGLOG(5, "ZSTD_compressSubBlock_sequences (nbSeq=%zu, writeEntropy=%d, longOffsets=%d)", nbSeq, writeEntropy, longOffsets);
+
+    *entropyWritten = 0;
+    /* Sequences Header */
+    RETURN_ERROR_IF((oend-op) < 3 /*max nbSeq Size*/ + 1 /*seqHead*/,
+                    dstSize_tooSmall, "");
+    if (nbSeq < 0x7F)
+        *op++ = (BYTE)nbSeq;
+    else if (nbSeq < LONGNBSEQ)
+        op[0] = (BYTE)((nbSeq>>8) + 0x80), op[1] = (BYTE)nbSeq, op+=2;
+    else
+        op[0]=0xFF, MEM_writeLE16(op+1, (U16)(nbSeq - LONGNBSEQ)), op+=3;
+    if (nbSeq==0) {
+        return op - ostart;
+    }
+
+    /* seqHead : flags for FSE encoding type */
+    seqHead = op++;
+
+    DEBUGLOG(5, "ZSTD_compressSubBlock_sequences (seqHeadSize=%u)", (unsigned)(op-ostart));
+
+    if (writeEntropy) {
+        const U32 LLtype = fseMetadata->llType;
+        const U32 Offtype = fseMetadata->ofType;
+        const U32 MLtype = fseMetadata->mlType;
+        DEBUGLOG(5, "ZSTD_compressSubBlock_sequences (fseTablesSize=%zu)", fseMetadata->fseTablesSize);
+        *seqHead = (BYTE)((LLtype<<6) + (Offtype<<4) + (MLtype<<2));
+        memcpy(op, fseMetadata->fseTablesBuffer, fseMetadata->fseTablesSize);
+        op += fseMetadata->fseTablesSize;
+    } else {
+        const U32 repeat = set_repeat;
+        *seqHead = (BYTE)((repeat<<6) + (repeat<<4) + (repeat<<2));
+    }
+
+    {   size_t const bitstreamSize = ZSTD_encodeSequences(
+                                        op, oend - op,
+                                        fseTables->matchlengthCTable, mlCode,
+                                        fseTables->offcodeCTable, ofCode,
+                                        fseTables->litlengthCTable, llCode,
+                                        sequences, nbSeq,
+                                        longOffsets, bmi2);
+        FORWARD_IF_ERROR(bitstreamSize, "ZSTD_encodeSequences failed");
+        op += bitstreamSize;
+        /* zstd versions <= 1.3.4 mistakenly report corruption when
+         * FSE_readNCount() receives a buffer < 4 bytes.
+         * Fixed by https://github.com/facebook/zstd/pull/1146.
+         * This can happen when the last set_compressed table present is 2
+         * bytes and the bitstream is only one byte.
+         * In this exceedingly rare case, we will simply emit an uncompressed
+         * block, since it isn't worth optimizing.
+         */
+#ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
+        if (writeEntropy && fseMetadata->lastCountSize && fseMetadata->lastCountSize + bitstreamSize < 4) {
+            /* NCountSize >= 2 && bitstreamSize > 0 ==> lastCountSize == 3 */
+            assert(fseMetadata->lastCountSize + bitstreamSize == 3);
+            DEBUGLOG(5, "Avoiding bug in zstd decoder in versions <= 1.3.4 by "
+                        "emitting an uncompressed block.");
+            return 0;
+        }
+#endif
+        DEBUGLOG(5, "ZSTD_compressSubBlock_sequences (bitstreamSize=%zu)", bitstreamSize);
+    }
+
+    /* zstd versions <= 1.4.0 mistakenly report error when
+     * sequences section body size is less than 3 bytes.
+     * Fixed by https://github.com/facebook/zstd/pull/1664.
+     * This can happen when the previous sequences section block is compressed
+     * with rle mode and the current block's sequences section is compressed
+     * with repeat mode where sequences section body size can be 1 byte.
+     */
+#ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
+    if (op-seqHead < 4) {
+        DEBUGLOG(5, "Avoiding bug in zstd decoder in versions <= 1.4.0 by emitting "
+                    "an uncompressed block when sequences are < 4 bytes");
+        return 0;
+    }
+#endif
+
+    *entropyWritten = 1;
+    return op - ostart;
+}
+
+/** ZSTD_compressSubBlock() :
+ *  Compresses a single sub-block.
+ *  @return : compressed size of the sub-block
+ *            Or 0 if it failed to compress. */
+static size_t ZSTD_compressSubBlock(const ZSTD_entropyCTables_t* entropy,
+                                    const ZSTD_entropyCTablesMetadata_t* entropyMetadata,
+                                    const seqDef* sequences, size_t nbSeq,
+                                    const BYTE* literals, size_t litSize,
+                                    const BYTE* llCode, const BYTE* mlCode, const BYTE* ofCode,
+                                    const ZSTD_CCtx_params* cctxParams,
+                                    void* dst, size_t dstCapacity,
+                                    const int bmi2,
+                                    int writeLitEntropy, int writeSeqEntropy,
+                                    int* litEntropyWritten, int* seqEntropyWritten,
+                                    U32 lastBlock)
+{
+    BYTE* const ostart = (BYTE*)dst;
+    BYTE* const oend = ostart + dstCapacity;
+    BYTE* op = ostart + ZSTD_blockHeaderSize;
+    DEBUGLOG(5, "ZSTD_compressSubBlock (litSize=%zu, nbSeq=%zu, writeLitEntropy=%d, writeSeqEntropy=%d, lastBlock=%d)",
+                litSize, nbSeq, writeLitEntropy, writeSeqEntropy, lastBlock);
+    {   size_t cLitSize = ZSTD_compressSubBlock_literal((const HUF_CElt*)entropy->huf.CTable,
+                                                        &entropyMetadata->hufMetadata, literals, litSize,
+                                                        op, oend-op, bmi2, writeLitEntropy, litEntropyWritten);
+        FORWARD_IF_ERROR(cLitSize, "ZSTD_compressSubBlock_literal failed");
+        if (cLitSize == 0) return 0;
+        op += cLitSize;
+    }
+    {   size_t cSeqSize = ZSTD_compressSubBlock_sequences(&entropy->fse,
+                                                  &entropyMetadata->fseMetadata,
+                                                  sequences, nbSeq,
+                                                  llCode, mlCode, ofCode,
+                                                  cctxParams,
+                                                  op, oend-op,
+                                                  bmi2, writeSeqEntropy, seqEntropyWritten);
+        FORWARD_IF_ERROR(cSeqSize, "ZSTD_compressSubBlock_sequences failed");
+        if (cSeqSize == 0) return 0;
+        op += cSeqSize;
+    }
+    /* Write block header */
+    {   size_t cSize = (op-ostart)-ZSTD_blockHeaderSize;
+        U32 const cBlockHeader24 = lastBlock + (((U32)bt_compressed)<<1) + (U32)(cSize << 3);
+        MEM_writeLE24(ostart, cBlockHeader24);
+    }
+    return op-ostart;
+}
+
+static size_t ZSTD_estimateSubBlockSize_literal(const BYTE* literals, size_t litSize,
+                                                const ZSTD_hufCTables_t* huf,
+                                                const ZSTD_hufCTablesMetadata_t* hufMetadata,
+                                                void* workspace, size_t wkspSize,
+                                                int writeEntropy)
+{
+    unsigned* const countWksp = (unsigned*)workspace;
+    unsigned maxSymbolValue = 255;
+    size_t literalSectionHeaderSize = 3; /* Use hard coded size of 3 bytes */
+
+    if (hufMetadata->hType == set_basic) return litSize;
+    else if (hufMetadata->hType == set_rle) return 1;
+    else if (hufMetadata->hType == set_compressed || hufMetadata->hType == set_repeat) {
+        size_t const largest = HIST_count_wksp (countWksp, &maxSymbolValue, (const BYTE*)literals, litSize, workspace, wkspSize);
+        if (ZSTD_isError(largest)) return litSize;
+        {   size_t cLitSizeEstimate = HUF_estimateCompressedSize((const HUF_CElt*)huf->CTable, countWksp, maxSymbolValue);
+            if (writeEntropy) cLitSizeEstimate += hufMetadata->hufDesSize;
+            return cLitSizeEstimate + literalSectionHeaderSize;
+    }   }
+    assert(0); /* impossible */
+    return 0;
+}
+
+static size_t ZSTD_estimateSubBlockSize_symbolType(symbolEncodingType_e type,
+                        const BYTE* codeTable, unsigned maxCode,
+                        size_t nbSeq, const FSE_CTable* fseCTable,
+                        const U32* additionalBits,
+                        short const* defaultNorm, U32 defaultNormLog,
+                        void* workspace, size_t wkspSize)
+{
+    unsigned* const countWksp = (unsigned*)workspace;
+    const BYTE* ctp = codeTable;
+    const BYTE* const ctStart = ctp;
+    const BYTE* const ctEnd = ctStart + nbSeq;
+    size_t cSymbolTypeSizeEstimateInBits = 0;
+    unsigned max = maxCode;
+
+    HIST_countFast_wksp(countWksp, &max, codeTable, nbSeq, workspace, wkspSize);  /* can't fail */
+    if (type == set_basic) {
+        cSymbolTypeSizeEstimateInBits = ZSTD_crossEntropyCost(defaultNorm, defaultNormLog, countWksp, max);
+    } else if (type == set_rle) {
+        cSymbolTypeSizeEstimateInBits = 0;
+    } else if (type == set_compressed || type == set_repeat) {
+        cSymbolTypeSizeEstimateInBits = ZSTD_fseBitCost(fseCTable, countWksp, max);
+    }
+    if (ZSTD_isError(cSymbolTypeSizeEstimateInBits)) return nbSeq * 10;
+    while (ctp < ctEnd) {
+        if (additionalBits) cSymbolTypeSizeEstimateInBits += additionalBits[*ctp];
+        else cSymbolTypeSizeEstimateInBits += *ctp; /* for offset, offset code is also the number of additional bits */
+        ctp++;
+    }
+    return cSymbolTypeSizeEstimateInBits / 8;
+}
+
+static size_t ZSTD_estimateSubBlockSize_sequences(const BYTE* ofCodeTable,
+                                                  const BYTE* llCodeTable,
+                                                  const BYTE* mlCodeTable,
+                                                  size_t nbSeq,
+                                                  const ZSTD_fseCTables_t* fseTables,
+                                                  const ZSTD_fseCTablesMetadata_t* fseMetadata,
+                                                  void* workspace, size_t wkspSize,
+                                                  int writeEntropy)
+{
+    size_t sequencesSectionHeaderSize = 3; /* Use hard coded size of 3 bytes */
+    size_t cSeqSizeEstimate = 0;
+    cSeqSizeEstimate += ZSTD_estimateSubBlockSize_symbolType(fseMetadata->ofType, ofCodeTable, MaxOff,
+                                         nbSeq, fseTables->offcodeCTable, NULL,
+                                         OF_defaultNorm, OF_defaultNormLog,
+                                         workspace, wkspSize);
+    cSeqSizeEstimate += ZSTD_estimateSubBlockSize_symbolType(fseMetadata->llType, llCodeTable, MaxLL,
+                                         nbSeq, fseTables->litlengthCTable, LL_bits,
+                                         LL_defaultNorm, LL_defaultNormLog,
+                                         workspace, wkspSize);
+    cSeqSizeEstimate += ZSTD_estimateSubBlockSize_symbolType(fseMetadata->mlType, mlCodeTable, MaxML,
+                                         nbSeq, fseTables->matchlengthCTable, ML_bits,
+                                         ML_defaultNorm, ML_defaultNormLog,
+                                         workspace, wkspSize);
+    if (writeEntropy) cSeqSizeEstimate += fseMetadata->fseTablesSize;
+    return cSeqSizeEstimate + sequencesSectionHeaderSize;
+}
+
+static size_t ZSTD_estimateSubBlockSize(const BYTE* literals, size_t litSize,
+                                        const BYTE* ofCodeTable,
+                                        const BYTE* llCodeTable,
+                                        const BYTE* mlCodeTable,
+                                        size_t nbSeq,
+                                        const ZSTD_entropyCTables_t* entropy,
+                                        const ZSTD_entropyCTablesMetadata_t* entropyMetadata,
+                                        void* workspace, size_t wkspSize,
+                                        int writeLitEntropy, int writeSeqEntropy) {
+    size_t cSizeEstimate = 0;
+    cSizeEstimate += ZSTD_estimateSubBlockSize_literal(literals, litSize,
+                                                         &entropy->huf, &entropyMetadata->hufMetadata,
+                                                         workspace, wkspSize, writeLitEntropy);
+    cSizeEstimate += ZSTD_estimateSubBlockSize_sequences(ofCodeTable, llCodeTable, mlCodeTable,
+                                                         nbSeq, &entropy->fse, &entropyMetadata->fseMetadata,
+                                                         workspace, wkspSize, writeSeqEntropy);
+    return cSizeEstimate + ZSTD_blockHeaderSize;
+}
+
+static int ZSTD_needSequenceEntropyTables(ZSTD_fseCTablesMetadata_t const* fseMetadata)
+{
+    if (fseMetadata->llType == set_compressed || fseMetadata->llType == set_rle)
+        return 1;
+    if (fseMetadata->mlType == set_compressed || fseMetadata->mlType == set_rle)
+        return 1;
+    if (fseMetadata->ofType == set_compressed || fseMetadata->ofType == set_rle)
+        return 1;
+    return 0;
+}
+
+/** ZSTD_compressSubBlock_multi() :
+ *  Breaks super-block into multiple sub-blocks and compresses them.
+ *  Entropy will be written to the first block.
+ *  The following blocks will use repeat mode to compress.
+ *  All sub-blocks are compressed blocks (no raw or rle blocks).
+ *  @return : compressed size of the super block (which is multiple ZSTD blocks)
+ *            Or 0 if it failed to compress. */
+static size_t ZSTD_compressSubBlock_multi(const seqStore_t* seqStorePtr,
+                            const ZSTD_compressedBlockState_t* prevCBlock,
+                            ZSTD_compressedBlockState_t* nextCBlock,
+                            const ZSTD_entropyCTablesMetadata_t* entropyMetadata,
+                            const ZSTD_CCtx_params* cctxParams,
+                                  void* dst, size_t dstCapacity,
+                            const void* src, size_t srcSize,
+                            const int bmi2, U32 lastBlock,
+                            void* workspace, size_t wkspSize)
+{
+    const seqDef* const sstart = seqStorePtr->sequencesStart;
+    const seqDef* const send = seqStorePtr->sequences;
+    const seqDef* sp = sstart;
+    const BYTE* const lstart = seqStorePtr->litStart;
+    const BYTE* const lend = seqStorePtr->lit;
+    const BYTE* lp = lstart;
+    BYTE const* ip = (BYTE const*)src;
+    BYTE const* const iend = ip + srcSize;
+    BYTE* const ostart = (BYTE*)dst;
+    BYTE* const oend = ostart + dstCapacity;
+    BYTE* op = ostart;
+    const BYTE* llCodePtr = seqStorePtr->llCode;
+    const BYTE* mlCodePtr = seqStorePtr->mlCode;
+    const BYTE* ofCodePtr = seqStorePtr->ofCode;
+    size_t targetCBlockSize = cctxParams->targetCBlockSize;
+    size_t litSize, seqCount;
+    int writeLitEntropy = entropyMetadata->hufMetadata.hType == set_compressed;
+    int writeSeqEntropy = 1;
+    int lastSequence = 0;
+
+    DEBUGLOG(5, "ZSTD_compressSubBlock_multi (litSize=%u, nbSeq=%u)",
+                (unsigned)(lend-lp), (unsigned)(send-sstart));
+
+    litSize = 0;
+    seqCount = 0;
+    do {
+        size_t cBlockSizeEstimate = 0;
+        if (sstart == send) {
+            lastSequence = 1;
+        } else {
+            const seqDef* const sequence = sp + seqCount;
+            lastSequence = sequence == send - 1;
+            litSize += ZSTD_getSequenceLength(seqStorePtr, sequence).litLength;
+            seqCount++;
+        }
+        if (lastSequence) {
+            assert(lp <= lend);
+            assert(litSize <= (size_t)(lend - lp));
+            litSize = (size_t)(lend - lp);
+        }
+        /* I think there is an optimization opportunity here.
+         * Calling ZSTD_estimateSubBlockSize for every sequence can be wasteful
+         * since it recalculates estimate from scratch.
+         * For example, it would recount literal distribution and symbol codes everytime.
+         */
+        cBlockSizeEstimate = ZSTD_estimateSubBlockSize(lp, litSize, ofCodePtr, llCodePtr, mlCodePtr, seqCount,
+                                                       &nextCBlock->entropy, entropyMetadata,
+                                                       workspace, wkspSize, writeLitEntropy, writeSeqEntropy);
+        if (cBlockSizeEstimate > targetCBlockSize || lastSequence) {
+            int litEntropyWritten = 0;
+            int seqEntropyWritten = 0;
+            const size_t decompressedSize = ZSTD_seqDecompressedSize(seqStorePtr, sp, seqCount, litSize, lastSequence);
+            const size_t cSize = ZSTD_compressSubBlock(&nextCBlock->entropy, entropyMetadata,
+                                                       sp, seqCount,
+                                                       lp, litSize,
+                                                       llCodePtr, mlCodePtr, ofCodePtr,
+                                                       cctxParams,
+                                                       op, oend-op,
+                                                       bmi2, writeLitEntropy, writeSeqEntropy,
+                                                       &litEntropyWritten, &seqEntropyWritten,
+                                                       lastBlock && lastSequence);
+            FORWARD_IF_ERROR(cSize, "ZSTD_compressSubBlock failed");
+            if (cSize > 0 && cSize < decompressedSize) {
+                DEBUGLOG(5, "Committed the sub-block");
+                assert(ip + decompressedSize <= iend);
+                ip += decompressedSize;
+                sp += seqCount;
+                lp += litSize;
+                op += cSize;
+                llCodePtr += seqCount;
+                mlCodePtr += seqCount;
+                ofCodePtr += seqCount;
+                litSize = 0;
+                seqCount = 0;
+                /* Entropy only needs to be written once */
+                if (litEntropyWritten) {
+                    writeLitEntropy = 0;
+                }
+                if (seqEntropyWritten) {
+                    writeSeqEntropy = 0;
+                }
+            }
+        }
+    } while (!lastSequence);
+    if (writeLitEntropy) {
+        DEBUGLOG(5, "ZSTD_compressSubBlock_multi has literal entropy tables unwritten");
+        memcpy(&nextCBlock->entropy.huf, &prevCBlock->entropy.huf, sizeof(prevCBlock->entropy.huf));
+    }
+    if (writeSeqEntropy && ZSTD_needSequenceEntropyTables(&entropyMetadata->fseMetadata)) {
+        /* If we haven't written our entropy tables, then we've violated our contract and
+         * must emit an uncompressed block.
+         */
+        DEBUGLOG(5, "ZSTD_compressSubBlock_multi has sequence entropy tables unwritten");
+        return 0;
+    }
+    if (ip < iend) {
+        size_t const cSize = ZSTD_noCompressBlock(op, oend - op, ip, iend - ip, lastBlock);
+        DEBUGLOG(5, "ZSTD_compressSubBlock_multi last sub-block uncompressed, %zu bytes", (size_t)(iend - ip));
+        FORWARD_IF_ERROR(cSize, "ZSTD_noCompressBlock failed");
+        assert(cSize != 0);
+        op += cSize;
+        /* We have to regenerate the repcodes because we've skipped some sequences */
+        if (sp < send) {
+            seqDef const* seq;
+            repcodes_t rep;
+            memcpy(&rep, prevCBlock->rep, sizeof(rep)); 
+            for (seq = sstart; seq < sp; ++seq) {
+                rep = ZSTD_updateRep(rep.rep, seq->offset - 1, ZSTD_getSequenceLength(seqStorePtr, seq).litLength == 0);
+            }
+            memcpy(nextCBlock->rep, &rep, sizeof(rep));
+        }
+    }
+    DEBUGLOG(5, "ZSTD_compressSubBlock_multi compressed");
+    return op-ostart;
+}
+
+size_t ZSTD_compressSuperBlock(ZSTD_CCtx* zc,
+                               void* dst, size_t dstCapacity,
+                               void const* src, size_t srcSize,
+                               unsigned lastBlock) {
+    ZSTD_entropyCTablesMetadata_t entropyMetadata;
+
+    FORWARD_IF_ERROR(ZSTD_buildSuperBlockEntropy(&zc->seqStore,
+          &zc->blockState.prevCBlock->entropy,
+          &zc->blockState.nextCBlock->entropy,
+          &zc->appliedParams,
+          &entropyMetadata,
+          zc->entropyWorkspace, HUF_WORKSPACE_SIZE /* statically allocated in resetCCtx */), "");
+
+    return ZSTD_compressSubBlock_multi(&zc->seqStore,
+            zc->blockState.prevCBlock,
+            zc->blockState.nextCBlock,
+            &entropyMetadata,
+            &zc->appliedParams,
+            dst, dstCapacity,
+            src, srcSize,
+            zc->bmi2, lastBlock,
+            zc->entropyWorkspace, HUF_WORKSPACE_SIZE /* statically allocated in resetCCtx */);
+}
diff --git a/lib/compress/zstd_compress_superblock.h b/lib/compress/zstd_compress_superblock.h
new file mode 100644
index 000000000000..07f4cb1dc646
--- /dev/null
+++ b/lib/compress/zstd_compress_superblock.h
@@ -0,0 +1,32 @@
+/*
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#ifndef ZSTD_COMPRESS_ADVANCED_H
+#define ZSTD_COMPRESS_ADVANCED_H
+
+/*-*************************************
+*  Dependencies
+***************************************/
+
+#include "../zstd.h" /* ZSTD_CCtx */
+
+/*-*************************************
+*  Target Compressed Block Size
+***************************************/
+
+/* ZSTD_compressSuperBlock() :
+ * Used to compress a super block when targetCBlockSize is being used.
+ * The given block will be compressed into multiple sub blocks that are around targetCBlockSize. */
+size_t ZSTD_compressSuperBlock(ZSTD_CCtx* zc,
+                               void* dst, size_t dstCapacity,
+                               void const* src, size_t srcSize,
+                               unsigned lastBlock);
+
+#endif /* ZSTD_COMPRESS_ADVANCED_H */
diff --git a/lib/compress/zstd_cwksp.h b/lib/compress/zstd_cwksp.h
index fc9765bd3f37..a25c9263b7d7 100644
--- a/lib/compress/zstd_cwksp.h
+++ b/lib/compress/zstd_cwksp.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -14,7 +14,7 @@
 /*-*************************************
 *  Dependencies
 ***************************************/
-#include "zstd_internal.h"
+#include "../common/zstd_internal.h"
 
 #if defined (__cplusplus)
 extern "C" {
@@ -24,16 +24,6 @@ extern "C" {
 *  Constants
 ***************************************/
 
-/* define "workspace is too large" as this number of times larger than needed */
-#define ZSTD_WORKSPACETOOLARGE_FACTOR 3
-
-/* when workspace is continuously too large
- * during at least this number of times,
- * context's memory usage is considered wasteful,
- * because it's sized to handle a worst case scenario which rarely happens.
- * In which case, resize it down to free some memory */
-#define ZSTD_WORKSPACETOOLARGE_MAXDURATION 128
-
 /* Since the workspace is effectively its own little malloc implementation /
  * arena, when we run under ASAN, we should similarly insert redzones between
  * each internal element of the workspace, so ASAN will catch overruns that
@@ -468,7 +458,7 @@ MEM_STATIC void ZSTD_cwksp_init(ZSTD_cwksp* ws, void* start, size_t size) {
 MEM_STATIC size_t ZSTD_cwksp_create(ZSTD_cwksp* ws, size_t size, ZSTD_customMem customMem) {
     void* workspace = ZSTD_malloc(size, customMem);
     DEBUGLOG(4, "cwksp: creating new workspace with %zd bytes", size);
-    RETURN_ERROR_IF(workspace == NULL, memory_allocation);
+    RETURN_ERROR_IF(workspace == NULL, memory_allocation, "NULL pointer!");
     ZSTD_cwksp_init(ws, workspace, size);
     return 0;
 }
diff --git a/lib/compress/zstd_double_fast.c b/lib/compress/zstd_double_fast.c
index a661a48534da..27eed66cfedd 100644
--- a/lib/compress/zstd_double_fast.c
+++ b/lib/compress/zstd_double_fast.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -63,10 +63,8 @@ size_t ZSTD_compressBlock_doubleFast_generic(
     const BYTE* ip = istart;
     const BYTE* anchor = istart;
     const U32 endIndex = (U32)((size_t)(istart - base) + srcSize);
-    const U32 lowestValid = ms->window.dictLimit;
-    const U32 maxDistance = 1U << cParams->windowLog;
     /* presumes that, if there is a dictionary, it must be using Attach mode */
-    const U32 prefixLowestIndex = (endIndex - lowestValid > maxDistance) ? endIndex - maxDistance : lowestValid;
+    const U32 prefixLowestIndex = ZSTD_getLowestPrefixIndex(ms, endIndex, cParams->windowLog);
     const BYTE* const prefixLowest = base + prefixLowestIndex;
     const BYTE* const iend = istart + srcSize;
     const BYTE* const ilimit = iend - HASH_READ_SIZE;
@@ -96,7 +94,7 @@ size_t ZSTD_compressBlock_doubleFast_generic(
                                      dictCParams->hashLog : hBitsL;
     const U32 dictHBitsS           = dictMode == ZSTD_dictMatchState ?
                                      dictCParams->chainLog : hBitsS;
-    const U32 dictAndPrefixLength  = (U32)(ip - prefixLowest + dictEnd - dictStart);
+    const U32 dictAndPrefixLength  = (U32)((ip - prefixLowest) + (dictEnd - dictStart));
 
     DEBUGLOG(5, "ZSTD_compressBlock_doubleFast_generic");
 
@@ -104,13 +102,15 @@ size_t ZSTD_compressBlock_doubleFast_generic(
 
     /* if a dictionary is attached, it must be within window range */
     if (dictMode == ZSTD_dictMatchState) {
-        assert(lowestValid + maxDistance >= endIndex);
+        assert(ms->window.dictLimit + (1U << cParams->windowLog) >= endIndex);
     }
 
     /* init */
     ip += (dictAndPrefixLength == 0);
     if (dictMode == ZSTD_noDict) {
-        U32 const maxRep = (U32)(ip - prefixLowest);
+        U32 const current = (U32)(ip - base);
+        U32 const windowLow = ZSTD_getLowestPrefixIndex(ms, current, cParams->windowLog);
+        U32 const maxRep = current - windowLow;
         if (offset_2 > maxRep) offsetSaved = offset_2, offset_2 = 0;
         if (offset_1 > maxRep) offsetSaved = offset_1, offset_1 = 0;
     }
@@ -198,6 +198,9 @@ size_t ZSTD_compressBlock_doubleFast_generic(
         }   }
 
         ip += ((ip-anchor) >> kSearchStrength) + 1;
+#if defined(__aarch64__)
+        PREFETCH_L1(ip+256);
+#endif
         continue;
 
 _search_next_long:
@@ -271,7 +274,7 @@ _match_stored:
                     U32 const repIndex2 = current2 - offset_2;
                     const BYTE* repMatch2 = dictMode == ZSTD_dictMatchState
                         && repIndex2 < prefixLowestIndex ?
-                            dictBase - dictIndexDelta + repIndex2 :
+                            dictBase + repIndex2 - dictIndexDelta :
                             base + repIndex2;
                     if ( ((U32)((prefixLowestIndex-1) - (U32)repIndex2) >= 3 /* intentional overflow */)
                        && (MEM_read32(repMatch2) == MEM_read32(ip)) ) {
diff --git a/lib/compress/zstd_double_fast.h b/lib/compress/zstd_double_fast.h
index 4fa31acfc0d6..14d944d69bc1 100644
--- a/lib/compress/zstd_double_fast.h
+++ b/lib/compress/zstd_double_fast.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -15,7 +15,7 @@
 extern "C" {
 #endif
 
-#include "mem.h"      /* U32 */
+#include "../common/mem.h"      /* U32 */
 #include "zstd_compress_internal.h"     /* ZSTD_CCtx, size_t */
 
 void ZSTD_fillDoubleHashTable(ZSTD_matchState_t* ms,
diff --git a/lib/compress/zstd_fast.c b/lib/compress/zstd_fast.c
index 6dbefee6b7fc..85a3a7a91e49 100644
--- a/lib/compress/zstd_fast.c
+++ b/lib/compress/zstd_fast.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -61,9 +61,7 @@ ZSTD_compressBlock_fast_generic(
     const BYTE* ip1;
     const BYTE* anchor = istart;
     const U32   endIndex = (U32)((size_t)(istart - base) + srcSize);
-    const U32   maxDistance = 1U << cParams->windowLog;
-    const U32   validStartIndex = ms->window.dictLimit;
-    const U32   prefixStartIndex = (endIndex - validStartIndex > maxDistance) ? endIndex - maxDistance : validStartIndex;
+    const U32   prefixStartIndex = ZSTD_getLowestPrefixIndex(ms, endIndex, cParams->windowLog);
     const BYTE* const prefixStart = base + prefixStartIndex;
     const BYTE* const iend = istart + srcSize;
     const BYTE* const ilimit = iend - HASH_READ_SIZE;
@@ -74,12 +72,21 @@ ZSTD_compressBlock_fast_generic(
     DEBUGLOG(5, "ZSTD_compressBlock_fast_generic");
     ip0 += (ip0 == prefixStart);
     ip1 = ip0 + 1;
-    {   U32 const maxRep = (U32)(ip0 - prefixStart);
+    {   U32 const current = (U32)(ip0 - base);
+        U32 const windowLow = ZSTD_getLowestPrefixIndex(ms, current, cParams->windowLog);
+        U32 const maxRep = current - windowLow;
         if (offset_2 > maxRep) offsetSaved = offset_2, offset_2 = 0;
         if (offset_1 > maxRep) offsetSaved = offset_1, offset_1 = 0;
     }
 
     /* Main Search Loop */
+#ifdef __INTEL_COMPILER
+    /* From intel 'The vector pragma indicates that the loop should be
+     * vectorized if it is legal to do so'. Can be used together with
+     * #pragma ivdep (but have opted to exclude that because intel
+     * warns against using it).*/
+    #pragma vector always
+#endif
     while (ip1 < ilimit) {   /* < instead of <=, because check at ip0+2 */
         size_t mLength;
         BYTE const* ip2 = ip0 + 2;
@@ -91,19 +98,25 @@ ZSTD_compressBlock_fast_generic(
         U32 const current1 = (U32)(ip1-base);
         U32 const matchIndex0 = hashTable[h0];
         U32 const matchIndex1 = hashTable[h1];
-        BYTE const* repMatch = ip2-offset_1;
+        BYTE const* repMatch = ip2 - offset_1;
         const BYTE* match0 = base + matchIndex0;
         const BYTE* match1 = base + matchIndex1;
         U32 offcode;
+
+#if defined(__aarch64__)
+        PREFETCH_L1(ip0+256);
+#endif
+
         hashTable[h0] = current0;   /* update hash table */
         hashTable[h1] = current1;   /* update hash table */
 
         assert(ip0 + 1 == ip1);
 
         if ((offset_1 > 0) & (MEM_read32(repMatch) == MEM_read32(ip2))) {
-            mLength = ip2[-1] == repMatch[-1] ? 1 : 0;
+            mLength = (ip2[-1] == repMatch[-1]) ? 1 : 0;
             ip0 = ip2 - mLength;
             match0 = repMatch - mLength;
+            mLength += 4;
             offcode = 0;
             goto _match;
         }
@@ -128,19 +141,18 @@ _offset: /* Requires: ip0, match0 */
         offset_2 = offset_1;
         offset_1 = (U32)(ip0-match0);
         offcode = offset_1 + ZSTD_REP_MOVE;
-        mLength = 0;
+        mLength = 4;
         /* Count the backwards match length */
         while (((ip0>anchor) & (match0>prefixStart))
              && (ip0[-1] == match0[-1])) { ip0--; match0--; mLength++; } /* catch up */
 
 _match: /* Requires: ip0, match0, offcode */
         /* Count the forward length */
-        mLength += ZSTD_count(ip0+mLength+4, match0+mLength+4, iend) + 4;
+        mLength += ZSTD_count(ip0+mLength, match0+mLength, iend);
         ZSTD_storeSeq(seqStore, (size_t)(ip0-anchor), anchor, iend, offcode, mLength-MINMATCH);
         /* match found */
         ip0 += mLength;
         anchor = ip0;
-        ip1 = ip0 + 1;
 
         if (ip0 <= ilimit) {
             /* Fill Table */
@@ -148,19 +160,18 @@ _match: /* Requires: ip0, match0, offcode */
             hashTable[ZSTD_hashPtr(base+current0+2, hlog, mls)] = current0+2;  /* here because current+2 could be > iend-8 */
             hashTable[ZSTD_hashPtr(ip0-2, hlog, mls)] = (U32)(ip0-2-base);
 
-            while ( ((ip0 <= ilimit) & (offset_2>0))  /* offset_2==0 means offset_2 is invalidated */
-                 && (MEM_read32(ip0) == MEM_read32(ip0 - offset_2)) ) {
-                /* store sequence */
-                size_t const rLength = ZSTD_count(ip0+4, ip0+4-offset_2, iend) + 4;
-                { U32 const tmpOff = offset_2; offset_2 = offset_1; offset_1 = tmpOff; } /* swap offset_2 <=> offset_1 */
-                hashTable[ZSTD_hashPtr(ip0, hlog, mls)] = (U32)(ip0-base);
-                ip0 += rLength;
-                ip1 = ip0 + 1;
-                ZSTD_storeSeq(seqStore, 0 /*litLen*/, anchor, iend, 0 /*offCode*/, rLength-MINMATCH);
-                anchor = ip0;
-                continue;   /* faster when present (confirmed on gcc-8) ... (?) */
-            }
-        }
+            if (offset_2 > 0) { /* offset_2==0 means offset_2 is invalidated */
+                while ( (ip0 <= ilimit) && (MEM_read32(ip0) == MEM_read32(ip0 - offset_2)) ) {
+                    /* store sequence */
+                    size_t const rLength = ZSTD_count(ip0+4, ip0+4-offset_2, iend) + 4;
+                    { U32 const tmpOff = offset_2; offset_2 = offset_1; offset_1 = tmpOff; } /* swap offset_2 <=> offset_1 */
+                    hashTable[ZSTD_hashPtr(ip0, hlog, mls)] = (U32)(ip0-base);
+                    ip0 += rLength;
+                    ZSTD_storeSeq(seqStore, 0 /*litLen*/, anchor, iend, 0 /*offCode*/, rLength-MINMATCH);
+                    anchor = ip0;
+                    continue;   /* faster when present (confirmed on gcc-8) ... (?) */
+        }   }   }
+        ip1 = ip0 + 1;
     }
 
     /* save reps for next block */
@@ -387,7 +398,7 @@ static size_t ZSTD_compressBlock_fast_extDict_generic(
     const BYTE* const ilimit = iend - 8;
     U32 offset_1=rep[0], offset_2=rep[1];
 
-    DEBUGLOG(5, "ZSTD_compressBlock_fast_extDict_generic");
+    DEBUGLOG(5, "ZSTD_compressBlock_fast_extDict_generic (offset_1=%u)", offset_1);
 
     /* switch to "regular" variant if extDict is invalidated due to maxDistance */
     if (prefixStartIndex == dictStartIndex)
@@ -404,6 +415,7 @@ static size_t ZSTD_compressBlock_fast_extDict_generic(
         const BYTE* const repBase = repIndex < prefixStartIndex ? dictBase : base;
         const BYTE* const repMatch = repBase + repIndex;
         hashTable[h] = current;   /* update hash table */
+        DEBUGLOG(7, "offset_1 = %u , current = %u", offset_1, current);
         assert(offset_1 <= current +1);   /* check repIndex */
 
         if ( (((U32)((prefixStartIndex-1) - repIndex) >= 3) /* intentional underflow */ & (repIndex > dictStartIndex))
diff --git a/lib/compress/zstd_fast.h b/lib/compress/zstd_fast.h
index b74a88c57c81..cf6aaa8e6750 100644
--- a/lib/compress/zstd_fast.h
+++ b/lib/compress/zstd_fast.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -15,7 +15,7 @@
 extern "C" {
 #endif
 
-#include "mem.h"      /* U32 */
+#include "../common/mem.h"      /* U32 */
 #include "zstd_compress_internal.h"
 
 void ZSTD_fillHashTable(ZSTD_matchState_t* ms,
diff --git a/lib/compress/zstd_lazy.c b/lib/compress/zstd_lazy.c
index 9ad7e03b54ca..4cf5c88b5325 100644
--- a/lib/compress/zstd_lazy.c
+++ b/lib/compress/zstd_lazy.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -660,12 +660,16 @@ ZSTD_compressBlock_lazy_generic(
     const U32 dictIndexDelta       = dictMode == ZSTD_dictMatchState ?
                                      prefixLowestIndex - (U32)(dictEnd - dictBase) :
                                      0;
-    const U32 dictAndPrefixLength = (U32)(ip - prefixLowest + dictEnd - dictLowest);
+    const U32 dictAndPrefixLength = (U32)((ip - prefixLowest) + (dictEnd - dictLowest));
+
+    DEBUGLOG(5, "ZSTD_compressBlock_lazy_generic (dictMode=%u)", (U32)dictMode);
 
     /* init */
     ip += (dictAndPrefixLength == 0);
     if (dictMode == ZSTD_noDict) {
-        U32 const maxRep = (U32)(ip - prefixLowest);
+        U32 const current = (U32)(ip - base);
+        U32 const windowLow = ZSTD_getLowestPrefixIndex(ms, current, ms->cParams.windowLog);
+        U32 const maxRep = current - windowLow;
         if (offset_2 > maxRep) savedOffset = offset_2, offset_2 = 0;
         if (offset_1 > maxRep) savedOffset = offset_1, offset_1 = 0;
     }
@@ -677,6 +681,12 @@ ZSTD_compressBlock_lazy_generic(
     }
 
     /* Match Loop */
+#if defined(__GNUC__) && defined(__x86_64__)
+    /* I've measured random a 5% speed loss on levels 5 & 6 (greedy) when the
+     * code alignment is perturbed. To fix the instability align the loop on 32-bytes.
+     */
+    __asm__(".p2align 5");
+#endif
     while (ip < ilimit) {
         size_t matchLength=0;
         size_t offset=0;
@@ -929,11 +939,11 @@ size_t ZSTD_compressBlock_lazy_extDict_generic(
     const BYTE* const ilimit = iend - 8;
     const BYTE* const base = ms->window.base;
     const U32 dictLimit = ms->window.dictLimit;
-    const U32 lowestIndex = ms->window.lowLimit;
     const BYTE* const prefixStart = base + dictLimit;
     const BYTE* const dictBase = ms->window.dictBase;
     const BYTE* const dictEnd  = dictBase + dictLimit;
-    const BYTE* const dictStart  = dictBase + lowestIndex;
+    const BYTE* const dictStart  = dictBase + ms->window.lowLimit;
+    const U32 windowLog = ms->cParams.windowLog;
 
     typedef size_t (*searchMax_f)(
                         ZSTD_matchState_t* ms,
@@ -942,10 +952,18 @@ size_t ZSTD_compressBlock_lazy_extDict_generic(
 
     U32 offset_1 = rep[0], offset_2 = rep[1];
 
+    DEBUGLOG(5, "ZSTD_compressBlock_lazy_extDict_generic");
+
     /* init */
     ip += (ip == prefixStart);
 
     /* Match Loop */
+#if defined(__GNUC__) && defined(__x86_64__)
+    /* I've measured random a 5% speed loss on levels 5 & 6 (greedy) when the
+     * code alignment is perturbed. To fix the instability align the loop on 32-bytes.
+     */
+    __asm__(".p2align 5");
+#endif
     while (ip < ilimit) {
         size_t matchLength=0;
         size_t offset=0;
@@ -953,10 +971,11 @@ size_t ZSTD_compressBlock_lazy_extDict_generic(
         U32 current = (U32)(ip-base);
 
         /* check repCode */
-        {   const U32 repIndex = (U32)(current+1 - offset_1);
+        {   const U32 windowLow = ZSTD_getLowestMatchIndex(ms, current+1, windowLog);
+            const U32 repIndex = (U32)(current+1 - offset_1);
             const BYTE* const repBase = repIndex < dictLimit ? dictBase : base;
             const BYTE* const repMatch = repBase + repIndex;
-            if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > lowestIndex))   /* intentional overflow */
+            if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > windowLow))   /* intentional overflow */
             if (MEM_read32(ip+1) == MEM_read32(repMatch)) {
                 /* repcode detected we should take it */
                 const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend;
@@ -983,10 +1002,11 @@ size_t ZSTD_compressBlock_lazy_extDict_generic(
             current++;
             /* check repCode */
             if (offset) {
+                const U32 windowLow = ZSTD_getLowestMatchIndex(ms, current, windowLog);
                 const U32 repIndex = (U32)(current - offset_1);
                 const BYTE* const repBase = repIndex < dictLimit ? dictBase : base;
                 const BYTE* const repMatch = repBase + repIndex;
-                if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > lowestIndex))  /* intentional overflow */
+                if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > windowLow))  /* intentional overflow */
                 if (MEM_read32(ip) == MEM_read32(repMatch)) {
                     /* repcode detected */
                     const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend;
@@ -1013,10 +1033,11 @@ size_t ZSTD_compressBlock_lazy_extDict_generic(
                 current++;
                 /* check repCode */
                 if (offset) {
+                    const U32 windowLow = ZSTD_getLowestMatchIndex(ms, current, windowLog);
                     const U32 repIndex = (U32)(current - offset_1);
                     const BYTE* const repBase = repIndex < dictLimit ? dictBase : base;
                     const BYTE* const repMatch = repBase + repIndex;
-                    if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > lowestIndex))  /* intentional overflow */
+                    if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > windowLow))  /* intentional overflow */
                     if (MEM_read32(ip) == MEM_read32(repMatch)) {
                         /* repcode detected */
                         const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend;
@@ -1057,10 +1078,12 @@ _storeSequence:
 
         /* check immediate repcode */
         while (ip <= ilimit) {
-            const U32 repIndex = (U32)((ip-base) - offset_2);
+            const U32 repCurrent = (U32)(ip-base);
+            const U32 windowLow = ZSTD_getLowestMatchIndex(ms, repCurrent, windowLog);
+            const U32 repIndex = repCurrent - offset_2;
             const BYTE* const repBase = repIndex < dictLimit ? dictBase : base;
             const BYTE* const repMatch = repBase + repIndex;
-            if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > lowestIndex))  /* intentional overflow */
+            if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > windowLow))  /* intentional overflow */
             if (MEM_read32(ip) == MEM_read32(repMatch)) {
                 /* repcode detected we should take it */
                 const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend;
diff --git a/lib/compress/zstd_lazy.h b/lib/compress/zstd_lazy.h
index bb1763069f3b..581936f03bd4 100644
--- a/lib/compress/zstd_lazy.h
+++ b/lib/compress/zstd_lazy.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/lib/compress/zstd_ldm.c b/lib/compress/zstd_ldm.c
index c3312ad3e3d2..8c479483581c 100644
--- a/lib/compress/zstd_ldm.c
+++ b/lib/compress/zstd_ldm.c
@@ -1,15 +1,16 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
  * LICENSE file in the root directory of this source tree) and the GPLv2 (found
  * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
  */
 
 #include "zstd_ldm.h"
 
-#include "debug.h"
+#include "../common/debug.h"
 #include "zstd_fast.h"          /* ZSTD_fillHashTable() */
 #include "zstd_double_fast.h"   /* ZSTD_fillDoubleHashTable() */
 
@@ -223,6 +224,20 @@ static U64 ZSTD_ldm_fillLdmHashTable(ldmState_t* state,
     return rollingHash;
 }
 
+void ZSTD_ldm_fillHashTable(
+            ldmState_t* state, const BYTE* ip,
+            const BYTE* iend, ldmParams_t const* params)
+{
+    DEBUGLOG(5, "ZSTD_ldm_fillHashTable");
+    if ((size_t)(iend - ip) >= params->minMatchLength) {
+        U64 startingHash = ZSTD_rollingHash_compute(ip, params->minMatchLength);
+        ZSTD_ldm_fillLdmHashTable(
+            state, startingHash, ip, iend - params->minMatchLength, state->window.base,
+            params->hashLog - params->bucketSizeLog,
+            *params);
+    }
+}
+
 
 /** ZSTD_ldm_limitTableUpdate() :
  *
@@ -449,6 +464,8 @@ size_t ZSTD_ldm_generateSequences(
             U32 const correction = ZSTD_window_correctOverflow(
                 &ldmState->window, /* cycleLog */ 0, maxDist, chunkStart);
             ZSTD_ldm_reduceTable(ldmState->hashTable, ldmHSize, correction);
+            /* invalidate dictionaries on overflow correction */
+            ldmState->loadedDictEnd = 0;
         }
         /* 2. We enforce the maximum offset allowed.
          *
@@ -457,8 +474,14 @@ size_t ZSTD_ldm_generateSequences(
          * TODO: * Test the chunk size.
          *       * Try invalidation after the sequence generation and test the
          *         the offset against maxDist directly.
+         *
+         * NOTE: Because of dictionaries + sequence splitting we MUST make sure
+         * that any offset used is valid at the END of the sequence, since it may
+         * be split into two sequences. This condition holds when using
+         * ZSTD_window_enforceMaxDist(), but if we move to checking offsets
+         * against maxDist directly, we'll have to carefully handle that case.
          */
-        ZSTD_window_enforceMaxDist(&ldmState->window, chunkEnd, maxDist, NULL, NULL);
+        ZSTD_window_enforceMaxDist(&ldmState->window, chunkEnd, maxDist, &ldmState->loadedDictEnd, NULL);
         /* 3. Generate the sequences for the chunk, and get newLeftoverSize. */
         newLeftoverSize = ZSTD_ldm_generateSequences_internal(
             ldmState, sequences, params, chunkStart, chunkSize);
@@ -566,14 +589,13 @@ size_t ZSTD_ldm_blockCompress(rawSeqStore_t* rawSeqStore,
         if (sequence.offset == 0)
             break;
 
-        assert(sequence.offset <= (1U << cParams->windowLog));
         assert(ip + sequence.litLength + sequence.matchLength <= iend);
 
         /* Fill tables for block compressor */
         ZSTD_ldm_limitTableUpdate(ms, ip);
         ZSTD_ldm_fillFastTables(ms, ip);
         /* Run the block compressor */
-        DEBUGLOG(5, "calling block compressor on segment of size %u", sequence.litLength);
+        DEBUGLOG(5, "pos %u : calling block compressor on segment of size %u", (unsigned)(ip-istart), sequence.litLength);
         {
             size_t const newLitLength =
                 blockCompressor(ms, seqStore, rep, ip, sequence.litLength);
diff --git a/lib/compress/zstd_ldm.h b/lib/compress/zstd_ldm.h
index a47846128b20..229ea05a9e1e 100644
--- a/lib/compress/zstd_ldm.h
+++ b/lib/compress/zstd_ldm.h
@@ -1,10 +1,11 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
  * LICENSE file in the root directory of this source tree) and the GPLv2 (found
  * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
  */
 
 #ifndef ZSTD_LDM_H
@@ -15,7 +16,7 @@ extern "C" {
 #endif
 
 #include "zstd_compress_internal.h"   /* ldmParams_t, U32 */
-#include "zstd.h"   /* ZSTD_CCtx, size_t */
+#include "../zstd.h"   /* ZSTD_CCtx, size_t */
 
 /*-*************************************
 *  Long distance matching
@@ -23,6 +24,10 @@ extern "C" {
 
 #define ZSTD_LDM_DEFAULT_WINDOW_LOG ZSTD_WINDOWLOG_LIMIT_DEFAULT
 
+void ZSTD_ldm_fillHashTable(
+            ldmState_t* state, const BYTE* ip,
+            const BYTE* iend, ldmParams_t const* params);
+
 /**
  * ZSTD_ldm_generateSequences():
  *
diff --git a/lib/compress/zstd_opt.c b/lib/compress/zstd_opt.c
index 2e50fca6ff53..36fff050cf5a 100644
--- a/lib/compress/zstd_opt.c
+++ b/lib/compress/zstd_opt.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -249,40 +249,6 @@ static U32 ZSTD_litLengthPrice(U32 const litLength, const optState_t* const optP
     }
 }
 
-/* ZSTD_litLengthContribution() :
- * @return ( cost(litlength) - cost(0) )
- * this value can then be added to rawLiteralsCost()
- * to provide a cost which is directly comparable to a match ending at same position */
-static int ZSTD_litLengthContribution(U32 const litLength, const optState_t* const optPtr, int optLevel)
-{
-    if (optPtr->priceType >= zop_predef) return (int)WEIGHT(litLength, optLevel);
-
-    /* dynamic statistics */
-    {   U32 const llCode = ZSTD_LLcode(litLength);
-        int const contribution = (int)(LL_bits[llCode] * BITCOST_MULTIPLIER)
-                               + (int)WEIGHT(optPtr->litLengthFreq[0], optLevel)   /* note: log2litLengthSum cancel out */
-                               - (int)WEIGHT(optPtr->litLengthFreq[llCode], optLevel);
-#if 1
-        return contribution;
-#else
-        return MAX(0, contribution); /* sometimes better, sometimes not ... */
-#endif
-    }
-}
-
-/* ZSTD_literalsContribution() :
- * creates a fake cost for the literals part of a sequence
- * which can be compared to the ending cost of a match
- * should a new match start at this position */
-static int ZSTD_literalsContribution(const BYTE* const literals, U32 const litLength,
-                                     const optState_t* const optPtr,
-                                     int optLevel)
-{
-    int const contribution = (int)ZSTD_rawLiteralsCost(literals, litLength, optPtr, optLevel)
-                           + ZSTD_litLengthContribution(litLength, optPtr, optLevel);
-    return contribution;
-}
-
 /* ZSTD_getMatchPrice() :
  * Provides the cost of the match part (offset + matchLength) of a sequence
  * Must be combined with ZSTD_fullLiteralsCost() to get the full cost of a sequence.
@@ -603,7 +569,10 @@ U32 ZSTD_insertBtAndGetAllMatches (
             U32 repLen = 0;
             assert(current >= dictLimit);
             if (repOffset-1 /* intentional overflow, discards 0 and -1 */ < current-dictLimit) {  /* equivalent to `current > repIndex >= dictLimit` */
-                if (ZSTD_readMINMATCH(ip, minMatch) == ZSTD_readMINMATCH(ip - repOffset, minMatch)) {
+                /* We must validate the repcode offset because when we're using a dictionary the
+                 * valid offset range shrinks when the dictionary goes out of bounds.
+                 */
+                if ((repIndex >= windowLow) & (ZSTD_readMINMATCH(ip, minMatch) == ZSTD_readMINMATCH(ip - repOffset, minMatch))) {
                     repLen = (U32)ZSTD_count(ip+minMatch, ip+minMatch-repOffset, iLimit) + minMatch;
                 }
             } else {  /* repIndex < dictLimit || repIndex >= current */
@@ -799,30 +768,6 @@ FORCE_INLINE_TEMPLATE U32 ZSTD_BtGetAllMatches (
 /*-*******************************
 *  Optimal parser
 *********************************/
-typedef struct repcodes_s {
-    U32 rep[3];
-} repcodes_t;
-
-static repcodes_t ZSTD_updateRep(U32 const rep[3], U32 const offset, U32 const ll0)
-{
-    repcodes_t newReps;
-    if (offset >= ZSTD_REP_NUM) {  /* full offset */
-        newReps.rep[2] = rep[1];
-        newReps.rep[1] = rep[0];
-        newReps.rep[0] = offset - ZSTD_REP_MOVE;
-    } else {   /* repcode */
-        U32 const repCode = offset + ll0;
-        if (repCode > 0) {  /* note : if repCode==0, no change */
-            U32 const currentOffset = (repCode==ZSTD_REP_NUM) ? (rep[0] - 1) : rep[repCode];
-            newReps.rep[2] = (repCode >= 2) ? rep[1] : rep[2];
-            newReps.rep[1] = rep[0];
-            newReps.rep[0] = currentOffset;
-        } else {   /* repCode == 0 */
-            memcpy(&newReps, rep, sizeof(newReps));
-        }
-    }
-    return newReps;
-}
 
 
 static U32 ZSTD_totalLen(ZSTD_optimal_t sol)
@@ -839,7 +784,7 @@ listStats(const U32* table, int lastEltID)
     int enb;
     for (enb=0; enb < nbElts; enb++) {
         (void)table;
-        //RAWLOG(2, "%3i:%3i,  ", enb, table[enb]);
+        /* RAWLOG(2, "%3i:%3i,  ", enb, table[enb]); */
         RAWLOG(2, "%4i,", table[enb]);
     }
     RAWLOG(2, " \n");
@@ -894,7 +839,12 @@ ZSTD_compressBlock_opt_generic(ZSTD_matchState_t* ms,
             { U32 i ; for (i=0; i<ZSTD_REP_NUM; i++) opt[0].rep[i] = rep[i]; }
             opt[0].mlen = 0;  /* means is_a_literal */
             opt[0].litlen = litlen;
-            opt[0].price = ZSTD_literalsContribution(anchor, litlen, optStatePtr, optLevel);
+            /* We don't need to include the actual price of the literals because
+             * it is static for the duration of the forward pass, and is included
+             * in every price. We include the literal length to avoid negative
+             * prices when we subtract the previous literal length.
+             */
+            opt[0].price = ZSTD_litLengthPrice(litlen, optStatePtr, optLevel);
 
             /* large match -> immediate encoding */
             {   U32 const maxML = matches[nbMatches-1].len;
@@ -923,7 +873,6 @@ ZSTD_compressBlock_opt_generic(ZSTD_matchState_t* ms,
                 for (matchNb = 0; matchNb < nbMatches; matchNb++) {
                     U32 const offset = matches[matchNb].off;
                     U32 const end = matches[matchNb].len;
-                    repcodes_t const repHistory = ZSTD_updateRep(rep, offset, ll0);
                     for ( ; pos <= end ; pos++ ) {
                         U32 const matchPrice = ZSTD_getMatchPrice(offset, pos, optStatePtr, optLevel);
                         U32 const sequencePrice = literalsPrice + matchPrice;
@@ -933,8 +882,6 @@ ZSTD_compressBlock_opt_generic(ZSTD_matchState_t* ms,
                         opt[pos].off = offset;
                         opt[pos].litlen = litlen;
                         opt[pos].price = sequencePrice;
-                        ZSTD_STATIC_ASSERT(sizeof(opt[pos].rep) == sizeof(repHistory));
-                        memcpy(opt[pos].rep, &repHistory, sizeof(repHistory));
                 }   }
                 last_pos = pos-1;
             }
@@ -961,7 +908,6 @@ ZSTD_compressBlock_opt_generic(ZSTD_matchState_t* ms,
                     opt[cur].off = 0;
                     opt[cur].litlen = litlen;
                     opt[cur].price = price;
-                    memcpy(opt[cur].rep, opt[cur-1].rep, sizeof(opt[cur].rep));
                 } else {
                     DEBUGLOG(7, "cPos:%zi==rPos:%u : literal would cost more (%.2f>%.2f) (hist:%u,%u,%u)",
                                 inr-istart, cur, ZSTD_fCost(price), ZSTD_fCost(opt[cur].price),
@@ -969,6 +915,21 @@ ZSTD_compressBlock_opt_generic(ZSTD_matchState_t* ms,
                 }
             }
 
+            /* Set the repcodes of the current position. We must do it here
+             * because we rely on the repcodes of the 2nd to last sequence being
+             * correct to set the next chunks repcodes during the backward
+             * traversal.
+             */
+            ZSTD_STATIC_ASSERT(sizeof(opt[cur].rep) == sizeof(repcodes_t));
+            assert(cur >= opt[cur].mlen);
+            if (opt[cur].mlen != 0) {
+                U32 const prev = cur - opt[cur].mlen;
+                repcodes_t newReps = ZSTD_updateRep(opt[prev].rep, opt[cur].off, opt[cur].litlen==0);
+                memcpy(opt[cur].rep, &newReps, sizeof(repcodes_t));
+            } else {
+                memcpy(opt[cur].rep, opt[cur - 1].rep, sizeof(repcodes_t));
+            }
+
             /* last match must start at a minimum distance of 8 from oend */
             if (inr > ilimit) continue;
 
@@ -1009,7 +970,6 @@ ZSTD_compressBlock_opt_generic(ZSTD_matchState_t* ms,
                 /* set prices using matches found at position == cur */
                 for (matchNb = 0; matchNb < nbMatches; matchNb++) {
                     U32 const offset = matches[matchNb].off;
-                    repcodes_t const repHistory = ZSTD_updateRep(opt[cur].rep, offset, ll0);
                     U32 const lastML = matches[matchNb].len;
                     U32 const startML = (matchNb>0) ? matches[matchNb-1].len+1 : minMatch;
                     U32 mlen;
@@ -1029,8 +989,6 @@ ZSTD_compressBlock_opt_generic(ZSTD_matchState_t* ms,
                             opt[pos].off = offset;
                             opt[pos].litlen = litlen;
                             opt[pos].price = price;
-                            ZSTD_STATIC_ASSERT(sizeof(opt[pos].rep) == sizeof(repHistory));
-                            memcpy(opt[pos].rep, &repHistory, sizeof(repHistory));
                         } else {
                             DEBUGLOG(7, "rPos:%u (ml=%2u) => new price is worse (%.2f>=%.2f)",
                                         pos, mlen, ZSTD_fCost(price), ZSTD_fCost(opt[pos].price));
@@ -1046,6 +1004,17 @@ ZSTD_compressBlock_opt_generic(ZSTD_matchState_t* ms,
 _shortestPath:   /* cur, last_pos, best_mlen, best_off have to be set */
         assert(opt[0].mlen == 0);
 
+        /* Set the next chunk's repcodes based on the repcodes of the beginning
+         * of the last match, and the last sequence. This avoids us having to
+         * update them while traversing the sequences.
+         */
+        if (lastSequence.mlen != 0) {
+            repcodes_t reps = ZSTD_updateRep(opt[cur].rep, lastSequence.off, lastSequence.litlen==0);
+            memcpy(rep, &reps, sizeof(reps));
+        } else {
+            memcpy(rep, opt[cur].rep, sizeof(repcodes_t));
+        }
+
         {   U32 const storeEnd = cur + 1;
             U32 storeStart = storeEnd;
             U32 seqPos = cur;
@@ -1082,20 +1051,6 @@ _shortestPath:   /* cur, last_pos, best_mlen, best_off have to be set */
                         continue;   /* will finish */
                     }
 
-                    /* repcodes update : like ZSTD_updateRep(), but update in place */
-                    if (offCode >= ZSTD_REP_NUM) {  /* full offset */
-                        rep[2] = rep[1];
-                        rep[1] = rep[0];
-                        rep[0] = offCode - ZSTD_REP_MOVE;
-                    } else {   /* repcode */
-                        U32 const repCode = offCode + (llen==0);
-                        if (repCode) {  /* note : if repCode==0, no change */
-                            U32 const currentOffset = (repCode==ZSTD_REP_NUM) ? (rep[0] - 1) : rep[repCode];
-                            if (repCode >= 2) rep[2] = rep[1];
-                            rep[1] = rep[0];
-                            rep[0] = currentOffset;
-                    }   }
-
                     assert(anchor + llen <= iend);
                     ZSTD_updateStats(optStatePtr, llen, anchor, offCode, mlen);
                     ZSTD_storeSeq(seqStore, llen, anchor, iend, offCode, mlen-MINMATCH);
@@ -1104,7 +1059,6 @@ _shortestPath:   /* cur, last_pos, best_mlen, best_off have to be set */
             }   }
             ZSTD_setBasePrices(optStatePtr, optLevel);
         }
-
     }   /* while (ip < ilimit) */
 
     /* Return the last literals size */
diff --git a/lib/compress/zstd_opt.h b/lib/compress/zstd_opt.h
index 094f7476650e..9aba8a9018c5 100644
--- a/lib/compress/zstd_opt.h
+++ b/lib/compress/zstd_opt.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/lib/compress/zstdmt_compress.c b/lib/compress/zstdmt_compress.c
index bc3062b5309b..1e3c8fdbee28 100644
--- a/lib/compress/zstdmt_compress.c
+++ b/lib/compress/zstdmt_compress.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -22,9 +22,9 @@
 /* ======   Dependencies   ====== */
 #include <string.h>      /* memcpy, memset */
 #include <limits.h>      /* INT_MAX, UINT_MAX */
-#include "mem.h"         /* MEM_STATIC */
-#include "pool.h"        /* threadpool */
-#include "threading.h"   /* mutex */
+#include "../common/mem.h"         /* MEM_STATIC */
+#include "../common/pool.h"        /* threadpool */
+#include "../common/threading.h"   /* mutex */
 #include "zstd_compress_internal.h"  /* MIN, ERROR, ZSTD_*, ZSTD_highbit32 */
 #include "zstd_ldm.h"
 #include "zstdmt_compress.h"
@@ -461,7 +461,13 @@ typedef struct {
     ZSTD_window_t ldmWindow;  /* A thread-safe copy of ldmState.window */
 } serialState_t;
 
-static int ZSTDMT_serialState_reset(serialState_t* serialState, ZSTDMT_seqPool* seqPool, ZSTD_CCtx_params params, size_t jobSize)
+static int
+ZSTDMT_serialState_reset(serialState_t* serialState,
+                         ZSTDMT_seqPool* seqPool,
+                         ZSTD_CCtx_params params,
+                         size_t jobSize,
+                         const void* dict, size_t const dictSize,
+                         ZSTD_dictContentType_e dictContentType)
 {
     /* Adjust parameters */
     if (params.ldmParams.enableLdm) {
@@ -490,8 +496,7 @@ static int ZSTDMT_serialState_reset(serialState_t* serialState, ZSTDMT_seqPool*
         /* Size the seq pool tables */
         ZSTDMT_setNbSeq(seqPool, ZSTD_ldm_getMaxNbSeq(params.ldmParams, jobSize));
         /* Reset the window */
-        ZSTD_window_clear(&serialState->ldmState.window);
-        serialState->ldmWindow = serialState->ldmState.window;
+        ZSTD_window_init(&serialState->ldmState.window);
         /* Resize tables and output space if necessary. */
         if (serialState->ldmState.hashTable == NULL || serialState->params.ldmParams.hashLog < hashLog) {
             ZSTD_free(serialState->ldmState.hashTable, cMem);
@@ -506,7 +511,24 @@ static int ZSTDMT_serialState_reset(serialState_t* serialState, ZSTDMT_seqPool*
         /* Zero the tables */
         memset(serialState->ldmState.hashTable, 0, hashSize);
         memset(serialState->ldmState.bucketOffsets, 0, bucketSize);
+
+        /* Update window state and fill hash table with dict */
+        serialState->ldmState.loadedDictEnd = 0;
+        if (dictSize > 0) {
+            if (dictContentType == ZSTD_dct_rawContent) {
+                BYTE const* const dictEnd = (const BYTE*)dict + dictSize;
+                ZSTD_window_update(&serialState->ldmState.window, dict, dictSize);
+                ZSTD_ldm_fillHashTable(&serialState->ldmState, (const BYTE*)dict, dictEnd, &params.ldmParams);
+                serialState->ldmState.loadedDictEnd = params.forceWindow ? 0 : (U32)(dictEnd - serialState->ldmState.window.base);
+            } else {
+                /* don't even load anything */
+            }
+        }
+
+        /* Initialize serialState's copy of ldmWindow. */
+        serialState->ldmWindow = serialState->ldmState.window;
     }
+
     serialState->params = params;
     serialState->params.jobSize = (U32)jobSize;
     return 0;
@@ -1054,7 +1076,7 @@ static ZSTD_CCtx_params ZSTDMT_initJobCCtxParams(const ZSTD_CCtx_params* params)
 static size_t ZSTDMT_resize(ZSTDMT_CCtx* mtctx, unsigned nbWorkers)
 {
     if (POOL_resize(mtctx->factory, nbWorkers)) return ERROR(memory_allocation);
-    FORWARD_IF_ERROR( ZSTDMT_expandJobsTable(mtctx, nbWorkers) );
+    FORWARD_IF_ERROR( ZSTDMT_expandJobsTable(mtctx, nbWorkers) , "");
     mtctx->bufPool = ZSTDMT_expandBufferPool(mtctx->bufPool, nbWorkers);
     if (mtctx->bufPool == NULL) return ERROR(memory_allocation);
     mtctx->cctxPool = ZSTDMT_expandCCtxPool(mtctx->cctxPool, nbWorkers);
@@ -1076,7 +1098,7 @@ void ZSTDMT_updateCParams_whileCompressing(ZSTDMT_CCtx* mtctx, const ZSTD_CCtx_p
     DEBUGLOG(5, "ZSTDMT_updateCParams_whileCompressing (level:%i)",
                 compressionLevel);
     mtctx->params.compressionLevel = compressionLevel;
-    {   ZSTD_compressionParameters cParams = ZSTD_getCParamsFromCCtxParams(cctxParams, 0, 0);
+    {   ZSTD_compressionParameters cParams = ZSTD_getCParamsFromCCtxParams(cctxParams, ZSTD_CONTENTSIZE_UNKNOWN, 0);
         cParams.windowLog = saved_wlog;
         mtctx->params.cParams = cParams;
     }
@@ -1235,7 +1257,8 @@ ZSTDMT_computeNbJobs(const ZSTD_CCtx_params* params, size_t srcSize, unsigned nb
 /* ZSTDMT_compress_advanced_internal() :
  * This is a blocking function : it will only give back control to caller after finishing its compression job.
  */
-static size_t ZSTDMT_compress_advanced_internal(
+static size_t
+ZSTDMT_compress_advanced_internal(
                 ZSTDMT_CCtx* mtctx,
                 void* dst, size_t dstCapacity,
           const void* src, size_t srcSize,
@@ -1267,10 +1290,11 @@ static size_t ZSTDMT_compress_advanced_internal(
 
     assert(avgJobSize >= 256 KB);  /* condition for ZSTD_compressBound(A) + ZSTD_compressBound(B) <= ZSTD_compressBound(A+B), required to compress directly into Dst (no additional buffer) */
     ZSTDMT_setBufferSize(mtctx->bufPool, ZSTD_compressBound(avgJobSize) );
-    if (ZSTDMT_serialState_reset(&mtctx->serial, mtctx->seqPool, params, avgJobSize))
+    /* LDM doesn't even try to load the dictionary in single-ingestion mode */
+    if (ZSTDMT_serialState_reset(&mtctx->serial, mtctx->seqPool, params, avgJobSize, NULL, 0, ZSTD_dct_auto))
         return ERROR(memory_allocation);
 
-    FORWARD_IF_ERROR( ZSTDMT_expandJobsTable(mtctx, nbJobs) );  /* only expands if necessary */
+    FORWARD_IF_ERROR( ZSTDMT_expandJobsTable(mtctx, nbJobs) , "");  /* only expands if necessary */
 
     {   unsigned u;
         for (u=0; u<nbJobs; u++) {
@@ -1403,7 +1427,7 @@ size_t ZSTDMT_initCStream_internal(
 
     /* init */
     if (params.nbWorkers != mtctx->params.nbWorkers)
-        FORWARD_IF_ERROR( ZSTDMT_resize(mtctx, params.nbWorkers) );
+        FORWARD_IF_ERROR( ZSTDMT_resize(mtctx, params.nbWorkers) , "");
 
     if (params.jobSize != 0 && params.jobSize < ZSTDMT_JOBSIZE_MIN) params.jobSize = ZSTDMT_JOBSIZE_MIN;
     if (params.jobSize > (size_t)ZSTDMT_JOBSIZE_MAX) params.jobSize = (size_t)ZSTDMT_JOBSIZE_MAX;
@@ -1500,7 +1524,8 @@ size_t ZSTDMT_initCStream_internal(
     mtctx->allJobsCompleted = 0;
     mtctx->consumed = 0;
     mtctx->produced = 0;
-    if (ZSTDMT_serialState_reset(&mtctx->serial, mtctx->seqPool, params, mtctx->targetSectionSize))
+    if (ZSTDMT_serialState_reset(&mtctx->serial, mtctx->seqPool, params, mtctx->targetSectionSize,
+                                 dict, dictSize, dictContentType))
         return ERROR(memory_allocation);
     return 0;
 }
@@ -1714,9 +1739,11 @@ static size_t ZSTDMT_flushProduced(ZSTDMT_CCtx* mtctx, ZSTD_outBuffer* output, u
             assert(mtctx->doneJobID < mtctx->nextJobID);
             assert(cSize >= mtctx->jobs[wJobID].dstFlushed);
             assert(mtctx->jobs[wJobID].dstBuff.start != NULL);
-            memcpy((char*)output->dst + output->pos,
-                   (const char*)mtctx->jobs[wJobID].dstBuff.start + mtctx->jobs[wJobID].dstFlushed,
-                   toFlush);
+            if (toFlush > 0) {
+                memcpy((char*)output->dst + output->pos,
+                    (const char*)mtctx->jobs[wJobID].dstBuff.start + mtctx->jobs[wJobID].dstFlushed,
+                    toFlush);
+            }
             output->pos += toFlush;
             mtctx->jobs[wJobID].dstFlushed += toFlush;  /* can write : this value is only used by mtctx */
 
@@ -1786,7 +1813,7 @@ static int ZSTDMT_isOverlapped(buffer_t buffer, range_t range)
     BYTE const* const bufferStart = (BYTE const*)buffer.start;
     BYTE const* const bufferEnd = bufferStart + buffer.capacity;
     BYTE const* const rangeStart = (BYTE const*)range.start;
-    BYTE const* const rangeEnd = rangeStart + range.size;
+    BYTE const* const rangeEnd = range.size != 0 ? rangeStart + range.size : rangeStart;
 
     if (rangeStart == NULL || bufferStart == NULL)
         return 0;
@@ -2060,7 +2087,7 @@ size_t ZSTDMT_compressStream_generic(ZSTDMT_CCtx* mtctx,
       || ((endOp == ZSTD_e_end) && (!mtctx->frameEnded)) ) {   /* must finish the frame with a zero-size block */
         size_t const jobSize = mtctx->inBuff.filled;
         assert(mtctx->inBuff.filled <= mtctx->targetSectionSize);
-        FORWARD_IF_ERROR( ZSTDMT_createCompressionJob(mtctx, jobSize, endOp) );
+        FORWARD_IF_ERROR( ZSTDMT_createCompressionJob(mtctx, jobSize, endOp) , "");
     }
 
     /* check for potential compressed data ready to be flushed */
@@ -2074,7 +2101,7 @@ size_t ZSTDMT_compressStream_generic(ZSTDMT_CCtx* mtctx,
 
 size_t ZSTDMT_compressStream(ZSTDMT_CCtx* mtctx, ZSTD_outBuffer* output, ZSTD_inBuffer* input)
 {
-    FORWARD_IF_ERROR( ZSTDMT_compressStream_generic(mtctx, output, input, ZSTD_e_continue) );
+    FORWARD_IF_ERROR( ZSTDMT_compressStream_generic(mtctx, output, input, ZSTD_e_continue) , "");
 
     /* recommended next input size : fill current input buffer */
     return mtctx->targetSectionSize - mtctx->inBuff.filled;   /* note : could be zero when input buffer is fully filled and no more availability to create new job */
@@ -2091,7 +2118,7 @@ static size_t ZSTDMT_flushStream_internal(ZSTDMT_CCtx* mtctx, ZSTD_outBuffer* ou
       || ((endFrame==ZSTD_e_end) && !mtctx->frameEnded)) {  /* need a last 0-size block to end frame */
            DEBUGLOG(5, "ZSTDMT_flushStream_internal : create a new job (%u bytes, end:%u)",
                         (U32)srcSize, (U32)endFrame);
-        FORWARD_IF_ERROR( ZSTDMT_createCompressionJob(mtctx, srcSize, endFrame) );
+        FORWARD_IF_ERROR( ZSTDMT_createCompressionJob(mtctx, srcSize, endFrame) , "");
     }
 
     /* check if there is any data available to flush */
diff --git a/lib/compress/zstdmt_compress.h b/lib/compress/zstdmt_compress.h
index 12a526087db4..89914eb7f883 100644
--- a/lib/compress/zstdmt_compress.h
+++ b/lib/compress/zstdmt_compress.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -40,7 +40,7 @@
 /* ===   Dependencies   === */
 #include <stddef.h>                /* size_t */
 #define ZSTD_STATIC_LINKING_ONLY   /* ZSTD_parameters */
-#include "zstd.h"            /* ZSTD_inBuffer, ZSTD_outBuffer, ZSTDLIB_API */
+#include "../zstd.h"            /* ZSTD_inBuffer, ZSTD_outBuffer, ZSTDLIB_API */
 
 
 /* ===   Constants   === */
diff --git a/lib/decompress/huf_decompress.c b/lib/decompress/huf_decompress.c
index bb2d0a96bc1e..68293a13096a 100644
--- a/lib/decompress/huf_decompress.c
+++ b/lib/decompress/huf_decompress.c
@@ -1,47 +1,27 @@
 /* ******************************************************************
-   huff0 huffman decoder,
-   part of Finite State Entropy library
-   Copyright (C) 2013-present, Yann Collet.
-
-   BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
-
-   Redistribution and use in source and binary forms, with or without
-   modification, are permitted provided that the following conditions are
-   met:
-
-       * Redistributions of source code must retain the above copyright
-   notice, this list of conditions and the following disclaimer.
-       * Redistributions in binary form must reproduce the above
-   copyright notice, this list of conditions and the following disclaimer
-   in the documentation and/or other materials provided with the
-   distribution.
-
-   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-    You can contact the author at :
-    - FSE+HUF source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ * huff0 huffman decoder,
+ * part of Finite State Entropy library
+ * Copyright (c) 2013-2020, Yann Collet, Facebook, Inc.
+ *
+ *  You can contact the author at :
+ *  - FSE+HUF source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
 ****************************************************************** */
 
 /* **************************************************************
 *  Dependencies
 ****************************************************************/
 #include <string.h>     /* memcpy, memset */
-#include "compiler.h"
-#include "bitstream.h"  /* BIT_* */
-#include "fse.h"        /* to compress headers */
+#include "../common/compiler.h"
+#include "../common/bitstream.h"  /* BIT_* */
+#include "../common/fse.h"        /* to compress headers */
 #define HUF_STATIC_LINKING_ONLY
-#include "huf.h"
-#include "error_private.h"
+#include "../common/huf.h"
+#include "../common/error_private.h"
 
 /* **************************************************************
 *  Macros
@@ -61,9 +41,6 @@
 *  Error Management
 ****************************************************************/
 #define HUF_isError ERR_isError
-#ifndef CHECK_F
-#define CHECK_F(f) { size_t const err_ = (f); if (HUF_isError(err_)) return err_; }
-#endif
 
 
 /* **************************************************************
@@ -181,17 +158,29 @@ size_t HUF_readDTableX1_wksp(HUF_DTable* DTable, const void* src, size_t srcSize
 
     /* fill DTable */
     {   U32 n;
-        for (n=0; n<nbSymbols; n++) {
-            U32 const w = huffWeight[n];
-            U32 const length = (1 << w) >> 1;
-            U32 u;
+        size_t const nEnd = nbSymbols;
+        for (n=0; n<nEnd; n++) {
+            size_t const w = huffWeight[n];
+            size_t const length = (1 << w) >> 1;
+            size_t const uStart = rankVal[w];
+            size_t const uEnd = uStart + length;
+            size_t u;
             HUF_DEltX1 D;
-            D.byte = (BYTE)n; D.nbBits = (BYTE)(tableLog + 1 - w);
-            for (u = rankVal[w]; u < rankVal[w] + length; u++)
-                dt[u] = D;
-            rankVal[w] += length;
-    }   }
-
+            D.byte = (BYTE)n;
+            D.nbBits = (BYTE)(tableLog + 1 - w);
+            rankVal[w] = (U32)uEnd;
+            if (length < 4) {
+                /* Use length in the loop bound so the compiler knows it is short. */
+                for (u = 0; u < length; ++u)
+                    dt[uStart + u] = D;
+            } else {
+                /* Unroll the loop 4 times, we know it is a power of 2. */
+                for (u = uStart; u < uEnd; u += 4) {
+                    dt[u + 0] = D;
+                    dt[u + 1] = D;
+                    dt[u + 2] = D;
+                    dt[u + 3] = D;
+    }   }   }   }
     return iSize;
 }
 
@@ -282,6 +271,7 @@ HUF_decompress4X1_usingDTable_internal_body(
     {   const BYTE* const istart = (const BYTE*) cSrc;
         BYTE* const ostart = (BYTE*) dst;
         BYTE* const oend = ostart + dstSize;
+        BYTE* const olimit = oend - 3;
         const void* const dtPtr = DTable + 1;
         const HUF_DEltX1* const dt = (const HUF_DEltX1*)dtPtr;
 
@@ -306,9 +296,9 @@ HUF_decompress4X1_usingDTable_internal_body(
         BYTE* op2 = opStart2;
         BYTE* op3 = opStart3;
         BYTE* op4 = opStart4;
-        U32 endSignal = BIT_DStream_unfinished;
         DTableDesc const dtd = HUF_getDTableDesc(DTable);
         U32 const dtLog = dtd.tableLog;
+        U32 endSignal = 1;
 
         if (length4 > cSrcSize) return ERROR(corruption_detected);   /* overflow */
         CHECK_F( BIT_initDStream(&bitD1, istart1, length1) );
@@ -317,8 +307,7 @@ HUF_decompress4X1_usingDTable_internal_body(
         CHECK_F( BIT_initDStream(&bitD4, istart4, length4) );
 
         /* up to 16 symbols per loop (4 symbols per stream) in 64-bit mode */
-        endSignal = BIT_reloadDStream(&bitD1) | BIT_reloadDStream(&bitD2) | BIT_reloadDStream(&bitD3) | BIT_reloadDStream(&bitD4);
-        while ( (endSignal==BIT_DStream_unfinished) && (op4<(oend-3)) ) {
+        for ( ; (endSignal) & (op4 < olimit) ; ) {
             HUF_DECODE_SYMBOLX1_2(op1, &bitD1);
             HUF_DECODE_SYMBOLX1_2(op2, &bitD2);
             HUF_DECODE_SYMBOLX1_2(op3, &bitD3);
@@ -335,10 +324,10 @@ HUF_decompress4X1_usingDTable_internal_body(
             HUF_DECODE_SYMBOLX1_0(op2, &bitD2);
             HUF_DECODE_SYMBOLX1_0(op3, &bitD3);
             HUF_DECODE_SYMBOLX1_0(op4, &bitD4);
-            BIT_reloadDStream(&bitD1);
-            BIT_reloadDStream(&bitD2);
-            BIT_reloadDStream(&bitD3);
-            BIT_reloadDStream(&bitD4);
+            endSignal &= BIT_reloadDStreamFast(&bitD1) == BIT_DStream_unfinished;
+            endSignal &= BIT_reloadDStreamFast(&bitD2) == BIT_DStream_unfinished;
+            endSignal &= BIT_reloadDStreamFast(&bitD3) == BIT_DStream_unfinished;
+            endSignal &= BIT_reloadDStreamFast(&bitD4) == BIT_DStream_unfinished;
         }
 
         /* check corruption */
@@ -757,7 +746,6 @@ HUF_decompress1X2_usingDTable_internal_body(
     return dstSize;
 }
 
-
 FORCE_INLINE_TEMPLATE size_t
 HUF_decompress4X2_usingDTable_internal_body(
           void* dst,  size_t dstSize,
@@ -769,6 +757,7 @@ HUF_decompress4X2_usingDTable_internal_body(
     {   const BYTE* const istart = (const BYTE*) cSrc;
         BYTE* const ostart = (BYTE*) dst;
         BYTE* const oend = ostart + dstSize;
+        BYTE* const olimit = oend - (sizeof(size_t)-1);
         const void* const dtPtr = DTable+1;
         const HUF_DEltX2* const dt = (const HUF_DEltX2*)dtPtr;
 
@@ -793,7 +782,7 @@ HUF_decompress4X2_usingDTable_internal_body(
         BYTE* op2 = opStart2;
         BYTE* op3 = opStart3;
         BYTE* op4 = opStart4;
-        U32 endSignal;
+        U32 endSignal = 1;
         DTableDesc const dtd = HUF_getDTableDesc(DTable);
         U32 const dtLog = dtd.tableLog;
 
@@ -804,8 +793,29 @@ HUF_decompress4X2_usingDTable_internal_body(
         CHECK_F( BIT_initDStream(&bitD4, istart4, length4) );
 
         /* 16-32 symbols per loop (4-8 symbols per stream) */
-        endSignal = BIT_reloadDStream(&bitD1) | BIT_reloadDStream(&bitD2) | BIT_reloadDStream(&bitD3) | BIT_reloadDStream(&bitD4);
-        for ( ; (endSignal==BIT_DStream_unfinished) & (op4<(oend-(sizeof(bitD4.bitContainer)-1))) ; ) {
+        for ( ; (endSignal) & (op4 < olimit); ) {
+#if defined(__clang__) && (defined(__x86_64__) || defined(__i386__))
+            HUF_DECODE_SYMBOLX2_2(op1, &bitD1);
+            HUF_DECODE_SYMBOLX2_1(op1, &bitD1);
+            HUF_DECODE_SYMBOLX2_2(op1, &bitD1);
+            HUF_DECODE_SYMBOLX2_0(op1, &bitD1);
+            HUF_DECODE_SYMBOLX2_2(op2, &bitD2);
+            HUF_DECODE_SYMBOLX2_1(op2, &bitD2);
+            HUF_DECODE_SYMBOLX2_2(op2, &bitD2);
+            HUF_DECODE_SYMBOLX2_0(op2, &bitD2);
+            endSignal &= BIT_reloadDStreamFast(&bitD1) == BIT_DStream_unfinished;
+            endSignal &= BIT_reloadDStreamFast(&bitD2) == BIT_DStream_unfinished;
+            HUF_DECODE_SYMBOLX2_2(op3, &bitD3);
+            HUF_DECODE_SYMBOLX2_1(op3, &bitD3);
+            HUF_DECODE_SYMBOLX2_2(op3, &bitD3);
+            HUF_DECODE_SYMBOLX2_0(op3, &bitD3);
+            HUF_DECODE_SYMBOLX2_2(op4, &bitD4);
+            HUF_DECODE_SYMBOLX2_1(op4, &bitD4);
+            HUF_DECODE_SYMBOLX2_2(op4, &bitD4);
+            HUF_DECODE_SYMBOLX2_0(op4, &bitD4);
+            endSignal &= BIT_reloadDStreamFast(&bitD3) == BIT_DStream_unfinished;
+            endSignal &= BIT_reloadDStreamFast(&bitD4) == BIT_DStream_unfinished;
+#else
             HUF_DECODE_SYMBOLX2_2(op1, &bitD1);
             HUF_DECODE_SYMBOLX2_2(op2, &bitD2);
             HUF_DECODE_SYMBOLX2_2(op3, &bitD3);
@@ -822,8 +832,12 @@ HUF_decompress4X2_usingDTable_internal_body(
             HUF_DECODE_SYMBOLX2_0(op2, &bitD2);
             HUF_DECODE_SYMBOLX2_0(op3, &bitD3);
             HUF_DECODE_SYMBOLX2_0(op4, &bitD4);
-
-            endSignal = BIT_reloadDStream(&bitD1) | BIT_reloadDStream(&bitD2) | BIT_reloadDStream(&bitD3) | BIT_reloadDStream(&bitD4);
+            endSignal = (U32)LIKELY(
+                        (BIT_reloadDStreamFast(&bitD1) == BIT_DStream_unfinished)
+                      & (BIT_reloadDStreamFast(&bitD2) == BIT_DStream_unfinished)
+                      & (BIT_reloadDStreamFast(&bitD3) == BIT_DStream_unfinished)
+                      & (BIT_reloadDStreamFast(&bitD4) == BIT_DStream_unfinished));
+#endif
         }
 
         /* check corruption */
diff --git a/lib/decompress/zstd_ddict.c b/lib/decompress/zstd_ddict.c
index 0af3d23bfe2d..c8cb8ecc9524 100644
--- a/lib/decompress/zstd_ddict.c
+++ b/lib/decompress/zstd_ddict.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -15,17 +15,17 @@
 *  Dependencies
 *********************************************************/
 #include <string.h>      /* memcpy, memmove, memset */
-#include "cpu.h"         /* bmi2 */
-#include "mem.h"         /* low level memory routines */
+#include "../common/cpu.h"         /* bmi2 */
+#include "../common/mem.h"         /* low level memory routines */
 #define FSE_STATIC_LINKING_ONLY
-#include "fse.h"
+#include "../common/fse.h"
 #define HUF_STATIC_LINKING_ONLY
-#include "huf.h"
+#include "../common/huf.h"
 #include "zstd_decompress_internal.h"
 #include "zstd_ddict.h"
 
 #if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT>=1)
-#  include "zstd_legacy.h"
+#  include "../legacy/zstd_legacy.h"
 #endif
 
 
@@ -65,6 +65,10 @@ void ZSTD_copyDDictParameters(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict)
     dctx->virtualStart = ddict->dictContent;
     dctx->dictEnd = (const BYTE*)ddict->dictContent + ddict->dictSize;
     dctx->previousDstEnd = dctx->dictEnd;
+#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
+    dctx->dictContentBeginForFuzzing = dctx->prefixStart;
+    dctx->dictContentEndForFuzzing = dctx->previousDstEnd;
+#endif
     if (ddict->entropyPresent) {
         dctx->litEntropy = 1;
         dctx->fseEntropy = 1;
@@ -107,7 +111,7 @@ ZSTD_loadEntropy_intoDDict(ZSTD_DDict* ddict,
     /* load entropy tables */
     RETURN_ERROR_IF(ZSTD_isError(ZSTD_loadDEntropy(
             &ddict->entropy, ddict->dictContent, ddict->dictSize)),
-        dictionary_corrupted);
+        dictionary_corrupted, "");
     ddict->entropyPresent = 1;
     return 0;
 }
@@ -133,7 +137,7 @@ static size_t ZSTD_initDDict_internal(ZSTD_DDict* ddict,
     ddict->entropy.hufTable[0] = (HUF_DTable)((HufLog)*0x1000001);  /* cover both little and big endian */
 
     /* parse dictionary content */
-    FORWARD_IF_ERROR( ZSTD_loadEntropy_intoDDict(ddict, dictContentType) );
+    FORWARD_IF_ERROR( ZSTD_loadEntropy_intoDDict(ddict, dictContentType) , "");
 
     return 0;
 }
diff --git a/lib/decompress/zstd_ddict.h b/lib/decompress/zstd_ddict.h
index 0479d11bb032..af307efd3d76 100644
--- a/lib/decompress/zstd_ddict.h
+++ b/lib/decompress/zstd_ddict.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -16,7 +16,7 @@
  *  Dependencies
  *********************************************************/
 #include <stddef.h>   /* size_t */
-#include "zstd.h"     /* ZSTD_DDict, and several public functions */
+#include "../zstd.h"     /* ZSTD_DDict, and several public functions */
 
 
 /*-*******************************************************
diff --git a/lib/decompress/zstd_decompress.c b/lib/decompress/zstd_decompress.c
index dd4591b7be9a..be5c7cfc3347 100644
--- a/lib/decompress/zstd_decompress.c
+++ b/lib/decompress/zstd_decompress.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -56,19 +56,19 @@
 *  Dependencies
 *********************************************************/
 #include <string.h>      /* memcpy, memmove, memset */
-#include "cpu.h"         /* bmi2 */
-#include "mem.h"         /* low level memory routines */
+#include "../common/cpu.h"         /* bmi2 */
+#include "../common/mem.h"         /* low level memory routines */
 #define FSE_STATIC_LINKING_ONLY
-#include "fse.h"
+#include "../common/fse.h"
 #define HUF_STATIC_LINKING_ONLY
-#include "huf.h"
-#include "zstd_internal.h"  /* blockProperties_t */
+#include "../common/huf.h"
+#include "../common/zstd_internal.h"  /* blockProperties_t */
 #include "zstd_decompress_internal.h"   /* ZSTD_DCtx */
 #include "zstd_ddict.h"  /* ZSTD_DDictDictContent */
 #include "zstd_decompress_block.h"   /* ZSTD_decompressBlock_internal */
 
 #if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT>=1)
-#  include "zstd_legacy.h"
+#  include "../legacy/zstd_legacy.h"
 #endif
 
 
@@ -111,7 +111,12 @@ static void ZSTD_initDCtx_internal(ZSTD_DCtx* dctx)
     dctx->legacyContext = NULL;
     dctx->previousLegacyVersion = 0;
     dctx->noForwardProgress = 0;
+    dctx->oversizedDuration = 0;
     dctx->bmi2 = ZSTD_cpuid_bmi2(ZSTD_cpuid());
+    dctx->outBufferMode = ZSTD_obm_buffered;
+#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
+    dctx->dictContentEndForFuzzing = NULL;
+#endif
 }
 
 ZSTD_DCtx* ZSTD_initStaticDCtx(void *workspace, size_t workspaceSize)
@@ -208,7 +213,7 @@ unsigned ZSTD_isFrame(const void* buffer, size_t size)
 static size_t ZSTD_frameHeaderSize_internal(const void* src, size_t srcSize, ZSTD_format_e format)
 {
     size_t const minInputSize = ZSTD_startingInputLength(format);
-    RETURN_ERROR_IF(srcSize < minInputSize, srcSize_wrong);
+    RETURN_ERROR_IF(srcSize < minInputSize, srcSize_wrong, "");
 
     {   BYTE const fhd = ((const BYTE*)src)[minInputSize-1];
         U32 const dictID= fhd & 3;
@@ -256,7 +261,7 @@ size_t ZSTD_getFrameHeader_advanced(ZSTD_frameHeader* zfhPtr, const void* src, s
             zfhPtr->frameType = ZSTD_skippableFrame;
             return 0;
         }
-        RETURN_ERROR(prefix_unknown);
+        RETURN_ERROR(prefix_unknown, "");
     }
 
     /* ensure there is enough `srcSize` to fully read/decode frame header */
@@ -280,7 +285,7 @@ size_t ZSTD_getFrameHeader_advanced(ZSTD_frameHeader* zfhPtr, const void* src, s
         if (!singleSegment) {
             BYTE const wlByte = ip[pos++];
             U32 const windowLog = (wlByte >> 3) + ZSTD_WINDOWLOG_ABSOLUTEMIN;
-            RETURN_ERROR_IF(windowLog > ZSTD_WINDOWLOG_MAX, frameParameter_windowTooLarge);
+            RETURN_ERROR_IF(windowLog > ZSTD_WINDOWLOG_MAX, frameParameter_windowTooLarge, "");
             windowSize = (1ULL << windowLog);
             windowSize += (windowSize >> 3) * (wlByte&7);
         }
@@ -352,14 +357,14 @@ static size_t readSkippableFrameSize(void const* src, size_t srcSize)
     size_t const skippableHeaderSize = ZSTD_SKIPPABLEHEADERSIZE;
     U32 sizeU32;
 
-    RETURN_ERROR_IF(srcSize < ZSTD_SKIPPABLEHEADERSIZE, srcSize_wrong);
+    RETURN_ERROR_IF(srcSize < ZSTD_SKIPPABLEHEADERSIZE, srcSize_wrong, "");
 
     sizeU32 = MEM_readLE32((BYTE const*)src + ZSTD_FRAMEIDSIZE);
     RETURN_ERROR_IF((U32)(sizeU32 + ZSTD_SKIPPABLEHEADERSIZE) < sizeU32,
-                    frameParameter_unsupported);
+                    frameParameter_unsupported, "");
     {
         size_t const skippableSize = skippableHeaderSize + sizeU32;
-        RETURN_ERROR_IF(skippableSize > srcSize, srcSize_wrong);
+        RETURN_ERROR_IF(skippableSize > srcSize, srcSize_wrong, "");
         return skippableSize;
     }
 }
@@ -439,7 +444,7 @@ static size_t ZSTD_decodeFrameHeader(ZSTD_DCtx* dctx, const void* src, size_t he
      * harder.
      */
     RETURN_ERROR_IF(dctx->fParams.dictID && (dctx->dictID != dctx->fParams.dictID),
-                    dictionary_wrong);
+                    dictionary_wrong, "");
 #endif
     if (dctx->fParams.checksumFlag) XXH64_reset(&dctx->xxhState, 0);
     return 0;
@@ -559,17 +564,6 @@ unsigned long long ZSTD_decompressBound(const void* src, size_t srcSize)
  *   Frame decoding
  ***************************************************************/
 
-
-void ZSTD_checkContinuity(ZSTD_DCtx* dctx, const void* dst)
-{
-    if (dst != dctx->previousDstEnd) {   /* not contiguous */
-        dctx->dictEnd = dctx->previousDstEnd;
-        dctx->virtualStart = (const char*)dst - ((const char*)(dctx->previousDstEnd) - (const char*)(dctx->prefixStart));
-        dctx->prefixStart = dst;
-        dctx->previousDstEnd = dst;
-    }
-}
-
 /** ZSTD_insertBlock() :
  *  insert `src` block into `dctx` history. Useful to track uncompressed blocks. */
 size_t ZSTD_insertBlock(ZSTD_DCtx* dctx, const void* blockStart, size_t blockSize)
@@ -587,9 +581,9 @@ static size_t ZSTD_copyRawBlock(void* dst, size_t dstCapacity,
     DEBUGLOG(5, "ZSTD_copyRawBlock");
     if (dst == NULL) {
         if (srcSize == 0) return 0;
-        RETURN_ERROR(dstBuffer_null);
+        RETURN_ERROR(dstBuffer_null, "");
     }
-    RETURN_ERROR_IF(srcSize > dstCapacity, dstSize_tooSmall);
+    RETURN_ERROR_IF(srcSize > dstCapacity, dstSize_tooSmall, "");
     memcpy(dst, src, srcSize);
     return srcSize;
 }
@@ -600,9 +594,9 @@ static size_t ZSTD_setRleBlock(void* dst, size_t dstCapacity,
 {
     if (dst == NULL) {
         if (regenSize == 0) return 0;
-        RETURN_ERROR(dstBuffer_null);
+        RETURN_ERROR(dstBuffer_null, "");
     }
-    RETURN_ERROR_IF(regenSize > dstCapacity, dstSize_tooSmall);
+    RETURN_ERROR_IF(regenSize > dstCapacity, dstSize_tooSmall, "");
     memset(dst, b, regenSize);
     return regenSize;
 }
@@ -618,7 +612,7 @@ static size_t ZSTD_decompressFrame(ZSTD_DCtx* dctx,
 {
     const BYTE* ip = (const BYTE*)(*srcPtr);
     BYTE* const ostart = (BYTE* const)dst;
-    BYTE* const oend = ostart + dstCapacity;
+    BYTE* const oend = dstCapacity != 0 ? ostart + dstCapacity : ostart;
     BYTE* op = ostart;
     size_t remainingSrcSize = *srcSizePtr;
 
@@ -627,15 +621,15 @@ static size_t ZSTD_decompressFrame(ZSTD_DCtx* dctx,
     /* check */
     RETURN_ERROR_IF(
         remainingSrcSize < ZSTD_FRAMEHEADERSIZE_MIN(dctx->format)+ZSTD_blockHeaderSize,
-        srcSize_wrong);
+        srcSize_wrong, "");
 
     /* Frame Header */
     {   size_t const frameHeaderSize = ZSTD_frameHeaderSize_internal(
                 ip, ZSTD_FRAMEHEADERSIZE_PREFIX(dctx->format), dctx->format);
         if (ZSTD_isError(frameHeaderSize)) return frameHeaderSize;
         RETURN_ERROR_IF(remainingSrcSize < frameHeaderSize+ZSTD_blockHeaderSize,
-                        srcSize_wrong);
-        FORWARD_IF_ERROR( ZSTD_decodeFrameHeader(dctx, ip, frameHeaderSize) );
+                        srcSize_wrong, "");
+        FORWARD_IF_ERROR( ZSTD_decodeFrameHeader(dctx, ip, frameHeaderSize) , "");
         ip += frameHeaderSize; remainingSrcSize -= frameHeaderSize;
     }
 
@@ -648,7 +642,7 @@ static size_t ZSTD_decompressFrame(ZSTD_DCtx* dctx,
 
         ip += ZSTD_blockHeaderSize;
         remainingSrcSize -= ZSTD_blockHeaderSize;
-        RETURN_ERROR_IF(cBlockSize > remainingSrcSize, srcSize_wrong);
+        RETURN_ERROR_IF(cBlockSize > remainingSrcSize, srcSize_wrong, "");
 
         switch(blockProperties.blockType)
         {
@@ -663,13 +657,15 @@ static size_t ZSTD_decompressFrame(ZSTD_DCtx* dctx,
             break;
         case bt_reserved :
         default:
-            RETURN_ERROR(corruption_detected);
+            RETURN_ERROR(corruption_detected, "invalid block type");
         }
 
         if (ZSTD_isError(decodedSize)) return decodedSize;
         if (dctx->fParams.checksumFlag)
             XXH64_update(&dctx->xxhState, op, decodedSize);
-        op += decodedSize;
+        if (decodedSize != 0)
+            op += decodedSize;
+        assert(ip != NULL);
         ip += cBlockSize;
         remainingSrcSize -= cBlockSize;
         if (blockProperties.lastBlock) break;
@@ -677,14 +673,14 @@ static size_t ZSTD_decompressFrame(ZSTD_DCtx* dctx,
 
     if (dctx->fParams.frameContentSize != ZSTD_CONTENTSIZE_UNKNOWN) {
         RETURN_ERROR_IF((U64)(op-ostart) != dctx->fParams.frameContentSize,
-                        corruption_detected);
+                        corruption_detected, "");
     }
     if (dctx->fParams.checksumFlag) { /* Frame content checksum verification */
         U32 const checkCalc = (U32)XXH64_digest(&dctx->xxhState);
         U32 checkRead;
-        RETURN_ERROR_IF(remainingSrcSize<4, checksum_wrong);
+        RETURN_ERROR_IF(remainingSrcSize<4, checksum_wrong, "");
         checkRead = MEM_readLE32(ip);
-        RETURN_ERROR_IF(checkRead != checkCalc, checksum_wrong);
+        RETURN_ERROR_IF(checkRead != checkCalc, checksum_wrong, "");
         ip += 4;
         remainingSrcSize -= 4;
     }
@@ -741,7 +737,7 @@ static size_t ZSTD_decompressMultiFrame(ZSTD_DCtx* dctx,
                         (unsigned)magicNumber, ZSTD_MAGICNUMBER);
             if ((magicNumber & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) {
                 size_t const skippableSize = readSkippableFrameSize(src, srcSize);
-                FORWARD_IF_ERROR(skippableSize);
+                FORWARD_IF_ERROR(skippableSize, "readSkippableFrameSize failed");
                 assert(skippableSize <= srcSize);
 
                 src = (const BYTE *)src + skippableSize;
@@ -751,11 +747,11 @@ static size_t ZSTD_decompressMultiFrame(ZSTD_DCtx* dctx,
 
         if (ddict) {
             /* we were called from ZSTD_decompress_usingDDict */
-            FORWARD_IF_ERROR(ZSTD_decompressBegin_usingDDict(dctx, ddict));
+            FORWARD_IF_ERROR(ZSTD_decompressBegin_usingDDict(dctx, ddict), "");
         } else {
             /* this will initialize correctly with no dict if dict == NULL, so
              * use this in all cases but ddict */
-            FORWARD_IF_ERROR(ZSTD_decompressBegin_usingDict(dctx, dict, dictSize));
+            FORWARD_IF_ERROR(ZSTD_decompressBegin_usingDict(dctx, dict, dictSize), "");
         }
         ZSTD_checkContinuity(dctx, dst);
 
@@ -776,7 +772,8 @@ static size_t ZSTD_decompressMultiFrame(ZSTD_DCtx* dctx,
                 "error.");
             if (ZSTD_isError(res)) return res;
             assert(res <= dstCapacity);
-            dst = (BYTE*)dst + res;
+            if (res != 0)
+                dst = (BYTE*)dst + res;
             dstCapacity -= res;
         }
         moreThan1Frame = 1;
@@ -824,7 +821,7 @@ size_t ZSTD_decompress(void* dst, size_t dstCapacity, const void* src, size_t sr
 #if defined(ZSTD_HEAPMODE) && (ZSTD_HEAPMODE>=1)
     size_t regenSize;
     ZSTD_DCtx* const dctx = ZSTD_createDCtx();
-    RETURN_ERROR_IF(dctx==NULL, memory_allocation);
+    RETURN_ERROR_IF(dctx==NULL, memory_allocation, "NULL pointer!");
     regenSize = ZSTD_decompressDCtx(dctx, dst, dstCapacity, src, srcSize);
     ZSTD_freeDCtx(dctx);
     return regenSize;
@@ -842,6 +839,24 @@ size_t ZSTD_decompress(void* dst, size_t dstCapacity, const void* src, size_t sr
 ****************************************/
 size_t ZSTD_nextSrcSizeToDecompress(ZSTD_DCtx* dctx) { return dctx->expected; }
 
+/**
+ * Similar to ZSTD_nextSrcSizeToDecompress(), but when when a block input can be streamed,
+ * we allow taking a partial block as the input. Currently only raw uncompressed blocks can
+ * be streamed.
+ *
+ * For blocks that can be streamed, this allows us to reduce the latency until we produce
+ * output, and avoid copying the input.
+ *
+ * @param inputSize - The total amount of input that the caller currently has.
+ */
+static size_t ZSTD_nextSrcSizeToDecompressWithInputSize(ZSTD_DCtx* dctx, size_t inputSize) {
+    if (!(dctx->stage == ZSTDds_decompressBlock || dctx->stage == ZSTDds_decompressLastBlock))
+        return dctx->expected;
+    if (dctx->bType != bt_raw)
+        return dctx->expected;
+    return MIN(MAX(inputSize, 1), dctx->expected);
+}
+
 ZSTD_nextInputType_e ZSTD_nextInputType(ZSTD_DCtx* dctx) {
     switch(dctx->stage)
     {
@@ -874,7 +889,7 @@ size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, c
 {
     DEBUGLOG(5, "ZSTD_decompressContinue (srcSize:%u)", (unsigned)srcSize);
     /* Sanity check */
-    RETURN_ERROR_IF(srcSize != dctx->expected, srcSize_wrong, "not allowed");
+    RETURN_ERROR_IF(srcSize != ZSTD_nextSrcSizeToDecompressWithInputSize(dctx, srcSize), srcSize_wrong, "not allowed");
     if (dstCapacity) ZSTD_checkContinuity(dctx, dst);
 
     switch (dctx->stage)
@@ -899,7 +914,7 @@ size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, c
     case ZSTDds_decodeFrameHeader:
         assert(src != NULL);
         memcpy(dctx->headerBuffer + (dctx->headerSize - srcSize), src, srcSize);
-        FORWARD_IF_ERROR(ZSTD_decodeFrameHeader(dctx, dctx->headerBuffer, dctx->headerSize));
+        FORWARD_IF_ERROR(ZSTD_decodeFrameHeader(dctx, dctx->headerBuffer, dctx->headerSize), "");
         dctx->expected = ZSTD_blockHeaderSize;
         dctx->stage = ZSTDds_decodeBlockHeader;
         return 0;
@@ -941,29 +956,41 @@ size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, c
             case bt_compressed:
                 DEBUGLOG(5, "ZSTD_decompressContinue: case bt_compressed");
                 rSize = ZSTD_decompressBlock_internal(dctx, dst, dstCapacity, src, srcSize, /* frame */ 1);
+                dctx->expected = 0;  /* Streaming not supported */
                 break;
             case bt_raw :
+                assert(srcSize <= dctx->expected);
                 rSize = ZSTD_copyRawBlock(dst, dstCapacity, src, srcSize);
+                FORWARD_IF_ERROR(rSize, "ZSTD_copyRawBlock failed");
+                assert(rSize == srcSize);
+                dctx->expected -= rSize;
                 break;
             case bt_rle :
                 rSize = ZSTD_setRleBlock(dst, dstCapacity, *(const BYTE*)src, dctx->rleSize);
+                dctx->expected = 0;  /* Streaming not supported */
                 break;
             case bt_reserved :   /* should never happen */
             default:
-                RETURN_ERROR(corruption_detected);
+                RETURN_ERROR(corruption_detected, "invalid block type");
             }
-            if (ZSTD_isError(rSize)) return rSize;
+            FORWARD_IF_ERROR(rSize, "");
             RETURN_ERROR_IF(rSize > dctx->fParams.blockSizeMax, corruption_detected, "Decompressed Block Size Exceeds Maximum");
             DEBUGLOG(5, "ZSTD_decompressContinue: decoded size from block : %u", (unsigned)rSize);
             dctx->decodedSize += rSize;
             if (dctx->fParams.checksumFlag) XXH64_update(&dctx->xxhState, dst, rSize);
+            dctx->previousDstEnd = (char*)dst + rSize;
+
+            /* Stay on the same stage until we are finished streaming the block. */
+            if (dctx->expected > 0) {
+                return rSize;
+            }
 
             if (dctx->stage == ZSTDds_decompressLastBlock) {   /* end of frame */
                 DEBUGLOG(4, "ZSTD_decompressContinue: decoded size from frame : %u", (unsigned)dctx->decodedSize);
                 RETURN_ERROR_IF(
                     dctx->fParams.frameContentSize != ZSTD_CONTENTSIZE_UNKNOWN
                  && dctx->decodedSize != dctx->fParams.frameContentSize,
-                    corruption_detected);
+                    corruption_detected, "");
                 if (dctx->fParams.checksumFlag) {  /* another round for frame checksum */
                     dctx->expected = 4;
                     dctx->stage = ZSTDds_checkChecksum;
@@ -974,7 +1001,6 @@ size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, c
             } else {
                 dctx->stage = ZSTDds_decodeBlockHeader;
                 dctx->expected = ZSTD_blockHeaderSize;
-                dctx->previousDstEnd = (char*)dst + rSize;
             }
             return rSize;
         }
@@ -984,7 +1010,7 @@ size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, c
         {   U32 const h32 = (U32)XXH64_digest(&dctx->xxhState);
             U32 const check32 = MEM_readLE32(src);
             DEBUGLOG(4, "ZSTD_decompressContinue: checksum : calculated %08X :: %08X read", (unsigned)h32, (unsigned)check32);
-            RETURN_ERROR_IF(check32 != h32, checksum_wrong);
+            RETURN_ERROR_IF(check32 != h32, checksum_wrong, "");
             dctx->expected = 0;
             dctx->stage = ZSTDds_getFrameHeaderSize;
             return 0;
@@ -1005,7 +1031,7 @@ size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, c
 
     default:
         assert(0);   /* impossible */
-        RETURN_ERROR(GENERIC);   /* some compiler require default to do something */
+        RETURN_ERROR(GENERIC, "impossible to reach");   /* some compiler require default to do something */
     }
 }
 
@@ -1016,6 +1042,10 @@ static size_t ZSTD_refDictContent(ZSTD_DCtx* dctx, const void* dict, size_t dict
     dctx->virtualStart = (const char*)dict - ((const char*)(dctx->previousDstEnd) - (const char*)(dctx->prefixStart));
     dctx->prefixStart = dict;
     dctx->previousDstEnd = (const char*)dict + dictSize;
+#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
+    dctx->dictContentBeginForFuzzing = dctx->prefixStart;
+    dctx->dictContentEndForFuzzing = dctx->previousDstEnd;
+#endif
     return 0;
 }
 
@@ -1029,7 +1059,7 @@ ZSTD_loadDEntropy(ZSTD_entropyDTables_t* entropy,
     const BYTE* dictPtr = (const BYTE*)dict;
     const BYTE* const dictEnd = dictPtr + dictSize;
 
-    RETURN_ERROR_IF(dictSize <= 8, dictionary_corrupted);
+    RETURN_ERROR_IF(dictSize <= 8, dictionary_corrupted, "dict is too small");
     assert(MEM_readLE32(dict) == ZSTD_MAGIC_DICTIONARY);   /* dict must be valid */
     dictPtr += 8;   /* skip header = magic + dictID */
 
@@ -1048,16 +1078,16 @@ ZSTD_loadDEntropy(ZSTD_entropyDTables_t* entropy,
                                                 dictPtr, dictEnd - dictPtr,
                                                 workspace, workspaceSize);
 #endif
-        RETURN_ERROR_IF(HUF_isError(hSize), dictionary_corrupted);
+        RETURN_ERROR_IF(HUF_isError(hSize), dictionary_corrupted, "");
         dictPtr += hSize;
     }
 
     {   short offcodeNCount[MaxOff+1];
         unsigned offcodeMaxValue = MaxOff, offcodeLog;
         size_t const offcodeHeaderSize = FSE_readNCount(offcodeNCount, &offcodeMaxValue, &offcodeLog, dictPtr, dictEnd-dictPtr);
-        RETURN_ERROR_IF(FSE_isError(offcodeHeaderSize), dictionary_corrupted);
-        RETURN_ERROR_IF(offcodeMaxValue > MaxOff, dictionary_corrupted);
-        RETURN_ERROR_IF(offcodeLog > OffFSELog, dictionary_corrupted);
+        RETURN_ERROR_IF(FSE_isError(offcodeHeaderSize), dictionary_corrupted, "");
+        RETURN_ERROR_IF(offcodeMaxValue > MaxOff, dictionary_corrupted, "");
+        RETURN_ERROR_IF(offcodeLog > OffFSELog, dictionary_corrupted, "");
         ZSTD_buildFSETable( entropy->OFTable,
                             offcodeNCount, offcodeMaxValue,
                             OF_base, OF_bits,
@@ -1068,9 +1098,9 @@ ZSTD_loadDEntropy(ZSTD_entropyDTables_t* entropy,
     {   short matchlengthNCount[MaxML+1];
         unsigned matchlengthMaxValue = MaxML, matchlengthLog;
         size_t const matchlengthHeaderSize = FSE_readNCount(matchlengthNCount, &matchlengthMaxValue, &matchlengthLog, dictPtr, dictEnd-dictPtr);
-        RETURN_ERROR_IF(FSE_isError(matchlengthHeaderSize), dictionary_corrupted);
-        RETURN_ERROR_IF(matchlengthMaxValue > MaxML, dictionary_corrupted);
-        RETURN_ERROR_IF(matchlengthLog > MLFSELog, dictionary_corrupted);
+        RETURN_ERROR_IF(FSE_isError(matchlengthHeaderSize), dictionary_corrupted, "");
+        RETURN_ERROR_IF(matchlengthMaxValue > MaxML, dictionary_corrupted, "");
+        RETURN_ERROR_IF(matchlengthLog > MLFSELog, dictionary_corrupted, "");
         ZSTD_buildFSETable( entropy->MLTable,
                             matchlengthNCount, matchlengthMaxValue,
                             ML_base, ML_bits,
@@ -1081,9 +1111,9 @@ ZSTD_loadDEntropy(ZSTD_entropyDTables_t* entropy,
     {   short litlengthNCount[MaxLL+1];
         unsigned litlengthMaxValue = MaxLL, litlengthLog;
         size_t const litlengthHeaderSize = FSE_readNCount(litlengthNCount, &litlengthMaxValue, &litlengthLog, dictPtr, dictEnd-dictPtr);
-        RETURN_ERROR_IF(FSE_isError(litlengthHeaderSize), dictionary_corrupted);
-        RETURN_ERROR_IF(litlengthMaxValue > MaxLL, dictionary_corrupted);
-        RETURN_ERROR_IF(litlengthLog > LLFSELog, dictionary_corrupted);
+        RETURN_ERROR_IF(FSE_isError(litlengthHeaderSize), dictionary_corrupted, "");
+        RETURN_ERROR_IF(litlengthMaxValue > MaxLL, dictionary_corrupted, "");
+        RETURN_ERROR_IF(litlengthLog > LLFSELog, dictionary_corrupted, "");
         ZSTD_buildFSETable( entropy->LLTable,
                             litlengthNCount, litlengthMaxValue,
                             LL_base, LL_bits,
@@ -1091,13 +1121,13 @@ ZSTD_loadDEntropy(ZSTD_entropyDTables_t* entropy,
         dictPtr += litlengthHeaderSize;
     }
 
-    RETURN_ERROR_IF(dictPtr+12 > dictEnd, dictionary_corrupted);
+    RETURN_ERROR_IF(dictPtr+12 > dictEnd, dictionary_corrupted, "");
     {   int i;
         size_t const dictContentSize = (size_t)(dictEnd - (dictPtr+12));
         for (i=0; i<3; i++) {
             U32 const rep = MEM_readLE32(dictPtr); dictPtr += 4;
             RETURN_ERROR_IF(rep==0 || rep > dictContentSize,
-                            dictionary_corrupted);
+                            dictionary_corrupted, "");
             entropy->rep[i] = rep;
     }   }
 
@@ -1115,7 +1145,7 @@ static size_t ZSTD_decompress_insertDictionary(ZSTD_DCtx* dctx, const void* dict
 
     /* load entropy tables */
     {   size_t const eSize = ZSTD_loadDEntropy(&dctx->entropy, dict, dictSize);
-        RETURN_ERROR_IF(ZSTD_isError(eSize), dictionary_corrupted);
+        RETURN_ERROR_IF(ZSTD_isError(eSize), dictionary_corrupted, "");
         dict = (const char*)dict + eSize;
         dictSize -= eSize;
     }
@@ -1138,6 +1168,7 @@ size_t ZSTD_decompressBegin(ZSTD_DCtx* dctx)
     dctx->entropy.hufTable[0] = (HUF_DTable)((HufLog)*0x1000001);  /* cover both little and big endian */
     dctx->litEntropy = dctx->fseEntropy = 0;
     dctx->dictID = 0;
+    dctx->bType = bt_reserved;
     ZSTD_STATIC_ASSERT(sizeof(dctx->entropy.rep) == sizeof(repStartValue));
     memcpy(dctx->entropy.rep, repStartValue, sizeof(repStartValue));  /* initial repcodes */
     dctx->LLTptr = dctx->entropy.LLTable;
@@ -1149,11 +1180,11 @@ size_t ZSTD_decompressBegin(ZSTD_DCtx* dctx)
 
 size_t ZSTD_decompressBegin_usingDict(ZSTD_DCtx* dctx, const void* dict, size_t dictSize)
 {
-    FORWARD_IF_ERROR( ZSTD_decompressBegin(dctx) );
+    FORWARD_IF_ERROR( ZSTD_decompressBegin(dctx) , "");
     if (dict && dictSize)
         RETURN_ERROR_IF(
             ZSTD_isError(ZSTD_decompress_insertDictionary(dctx, dict, dictSize)),
-            dictionary_corrupted);
+            dictionary_corrupted, "");
     return 0;
 }
 
@@ -1172,7 +1203,7 @@ size_t ZSTD_decompressBegin_usingDDict(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict)
         DEBUGLOG(4, "DDict is %s",
                     dctx->ddictIsCold ? "~cold~" : "hot!");
     }
-    FORWARD_IF_ERROR( ZSTD_decompressBegin(dctx) );
+    FORWARD_IF_ERROR( ZSTD_decompressBegin(dctx) , "");
     if (ddict) {   /* NULL ddict is equivalent to no dictionary */
         ZSTD_copyDDictParameters(dctx, ddict);
     }
@@ -1263,11 +1294,11 @@ size_t ZSTD_DCtx_loadDictionary_advanced(ZSTD_DCtx* dctx,
                                          ZSTD_dictLoadMethod_e dictLoadMethod,
                                          ZSTD_dictContentType_e dictContentType)
 {
-    RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong);
+    RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong, "");
     ZSTD_clearDict(dctx);
     if (dict && dictSize != 0) {
         dctx->ddictLocal = ZSTD_createDDict_advanced(dict, dictSize, dictLoadMethod, dictContentType, dctx->customMem);
-        RETURN_ERROR_IF(dctx->ddictLocal == NULL, memory_allocation);
+        RETURN_ERROR_IF(dctx->ddictLocal == NULL, memory_allocation, "NULL pointer!");
         dctx->ddict = dctx->ddictLocal;
         dctx->dictUses = ZSTD_use_indefinitely;
     }
@@ -1286,7 +1317,7 @@ size_t ZSTD_DCtx_loadDictionary(ZSTD_DCtx* dctx, const void* dict, size_t dictSi
 
 size_t ZSTD_DCtx_refPrefix_advanced(ZSTD_DCtx* dctx, const void* prefix, size_t prefixSize, ZSTD_dictContentType_e dictContentType)
 {
-    FORWARD_IF_ERROR(ZSTD_DCtx_loadDictionary_advanced(dctx, prefix, prefixSize, ZSTD_dlm_byRef, dictContentType));
+    FORWARD_IF_ERROR(ZSTD_DCtx_loadDictionary_advanced(dctx, prefix, prefixSize, ZSTD_dlm_byRef, dictContentType), "");
     dctx->dictUses = ZSTD_use_once;
     return 0;
 }
@@ -1303,8 +1334,8 @@ size_t ZSTD_DCtx_refPrefix(ZSTD_DCtx* dctx, const void* prefix, size_t prefixSiz
 size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dict, size_t dictSize)
 {
     DEBUGLOG(4, "ZSTD_initDStream_usingDict");
-    FORWARD_IF_ERROR( ZSTD_DCtx_reset(zds, ZSTD_reset_session_only) );
-    FORWARD_IF_ERROR( ZSTD_DCtx_loadDictionary(zds, dict, dictSize) );
+    FORWARD_IF_ERROR( ZSTD_DCtx_reset(zds, ZSTD_reset_session_only) , "");
+    FORWARD_IF_ERROR( ZSTD_DCtx_loadDictionary(zds, dict, dictSize) , "");
     return ZSTD_startingInputLength(zds->format);
 }
 
@@ -1320,8 +1351,8 @@ size_t ZSTD_initDStream(ZSTD_DStream* zds)
  * this function cannot fail */
 size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* dctx, const ZSTD_DDict* ddict)
 {
-    FORWARD_IF_ERROR( ZSTD_DCtx_reset(dctx, ZSTD_reset_session_only) );
-    FORWARD_IF_ERROR( ZSTD_DCtx_refDDict(dctx, ddict) );
+    FORWARD_IF_ERROR( ZSTD_DCtx_reset(dctx, ZSTD_reset_session_only) , "");
+    FORWARD_IF_ERROR( ZSTD_DCtx_refDDict(dctx, ddict) , "");
     return ZSTD_startingInputLength(dctx->format);
 }
 
@@ -1330,14 +1361,14 @@ size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* dctx, const ZSTD_DDict* ddict)
  * this function cannot fail */
 size_t ZSTD_resetDStream(ZSTD_DStream* dctx)
 {
-    FORWARD_IF_ERROR(ZSTD_DCtx_reset(dctx, ZSTD_reset_session_only));
+    FORWARD_IF_ERROR(ZSTD_DCtx_reset(dctx, ZSTD_reset_session_only), "");
     return ZSTD_startingInputLength(dctx->format);
 }
 
 
 size_t ZSTD_DCtx_refDDict(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict)
 {
-    RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong);
+    RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong, "");
     ZSTD_clearDict(dctx);
     if (ddict) {
         dctx->ddict = ddict;
@@ -1354,9 +1385,9 @@ size_t ZSTD_DCtx_setMaxWindowSize(ZSTD_DCtx* dctx, size_t maxWindowSize)
     ZSTD_bounds const bounds = ZSTD_dParam_getBounds(ZSTD_d_windowLogMax);
     size_t const min = (size_t)1 << bounds.lowerBound;
     size_t const max = (size_t)1 << bounds.upperBound;
-    RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong);
-    RETURN_ERROR_IF(maxWindowSize < min, parameter_outOfBound);
-    RETURN_ERROR_IF(maxWindowSize > max, parameter_outOfBound);
+    RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong, "");
+    RETURN_ERROR_IF(maxWindowSize < min, parameter_outOfBound, "");
+    RETURN_ERROR_IF(maxWindowSize > max, parameter_outOfBound, "");
     dctx->maxWindowSize = maxWindowSize;
     return 0;
 }
@@ -1379,6 +1410,10 @@ ZSTD_bounds ZSTD_dParam_getBounds(ZSTD_dParameter dParam)
             bounds.upperBound = (int)ZSTD_f_zstd1_magicless;
             ZSTD_STATIC_ASSERT(ZSTD_f_zstd1 < ZSTD_f_zstd1_magicless);
             return bounds;
+        case ZSTD_d_stableOutBuffer:
+            bounds.lowerBound = (int)ZSTD_obm_buffered;
+            bounds.upperBound = (int)ZSTD_obm_stable;
+            return bounds;
         default:;
     }
     bounds.error = ERROR(parameter_unsupported);
@@ -1398,12 +1433,12 @@ static int ZSTD_dParam_withinBounds(ZSTD_dParameter dParam, int value)
 }
 
 #define CHECK_DBOUNDS(p,v) {                \
-    RETURN_ERROR_IF(!ZSTD_dParam_withinBounds(p, v), parameter_outOfBound); \
+    RETURN_ERROR_IF(!ZSTD_dParam_withinBounds(p, v), parameter_outOfBound, ""); \
 }
 
 size_t ZSTD_DCtx_setParameter(ZSTD_DCtx* dctx, ZSTD_dParameter dParam, int value)
 {
-    RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong);
+    RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong, "");
     switch(dParam) {
         case ZSTD_d_windowLogMax:
             if (value == 0) value = ZSTD_WINDOWLOG_LIMIT_DEFAULT;
@@ -1414,9 +1449,13 @@ size_t ZSTD_DCtx_setParameter(ZSTD_DCtx* dctx, ZSTD_dParameter dParam, int value
             CHECK_DBOUNDS(ZSTD_d_format, value);
             dctx->format = (ZSTD_format_e)value;
             return 0;
+        case ZSTD_d_stableOutBuffer:
+            CHECK_DBOUNDS(ZSTD_d_stableOutBuffer, value);
+            dctx->outBufferMode = (ZSTD_outBufferMode_e)value;
+            return 0;
         default:;
     }
-    RETURN_ERROR(parameter_unsupported);
+    RETURN_ERROR(parameter_unsupported, "");
 }
 
 size_t ZSTD_DCtx_reset(ZSTD_DCtx* dctx, ZSTD_ResetDirective reset)
@@ -1428,7 +1467,7 @@ size_t ZSTD_DCtx_reset(ZSTD_DCtx* dctx, ZSTD_ResetDirective reset)
     }
     if ( (reset == ZSTD_reset_parameters)
       || (reset == ZSTD_reset_session_and_parameters) ) {
-        RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong);
+        RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong, "");
         ZSTD_clearDict(dctx);
         dctx->format = ZSTD_f_zstd1;
         dctx->maxWindowSize = ZSTD_MAXWINDOWSIZE_DEFAULT;
@@ -1449,7 +1488,7 @@ size_t ZSTD_decodingBufferSize_min(unsigned long long windowSize, unsigned long
     unsigned long long const neededSize = MIN(frameContentSize, neededRBSize);
     size_t const minRBSize = (size_t) neededSize;
     RETURN_ERROR_IF((unsigned long long)minRBSize != neededSize,
-                    frameParameter_windowTooLarge);
+                    frameParameter_windowTooLarge, "");
     return minRBSize;
 }
 
@@ -1467,30 +1506,94 @@ size_t ZSTD_estimateDStreamSize_fromFrame(const void* src, size_t srcSize)
     ZSTD_frameHeader zfh;
     size_t const err = ZSTD_getFrameHeader(&zfh, src, srcSize);
     if (ZSTD_isError(err)) return err;
-    RETURN_ERROR_IF(err>0, srcSize_wrong);
+    RETURN_ERROR_IF(err>0, srcSize_wrong, "");
     RETURN_ERROR_IF(zfh.windowSize > windowSizeMax,
-                    frameParameter_windowTooLarge);
+                    frameParameter_windowTooLarge, "");
     return ZSTD_estimateDStreamSize((size_t)zfh.windowSize);
 }
 
 
 /* *****   Decompression   ***** */
 
-MEM_STATIC size_t ZSTD_limitCopy(void* dst, size_t dstCapacity, const void* src, size_t srcSize)
+static int ZSTD_DCtx_isOverflow(ZSTD_DStream* zds, size_t const neededInBuffSize, size_t const neededOutBuffSize)
 {
-    size_t const length = MIN(dstCapacity, srcSize);
-    memcpy(dst, src, length);
-    return length;
+    return (zds->inBuffSize + zds->outBuffSize) >= (neededInBuffSize + neededOutBuffSize) * ZSTD_WORKSPACETOOLARGE_FACTOR;
+}
+
+static void ZSTD_DCtx_updateOversizedDuration(ZSTD_DStream* zds, size_t const neededInBuffSize, size_t const neededOutBuffSize)
+{
+    if (ZSTD_DCtx_isOverflow(zds, neededInBuffSize, neededOutBuffSize))
+        zds->oversizedDuration++;
+    else 
+        zds->oversizedDuration = 0;
+}
+
+static int ZSTD_DCtx_isOversizedTooLong(ZSTD_DStream* zds)
+{
+    return zds->oversizedDuration >= ZSTD_WORKSPACETOOLARGE_MAXDURATION;
+}
+
+/* Checks that the output buffer hasn't changed if ZSTD_obm_stable is used. */
+static size_t ZSTD_checkOutBuffer(ZSTD_DStream const* zds, ZSTD_outBuffer const* output)
+{
+    ZSTD_outBuffer const expect = zds->expectedOutBuffer;
+    /* No requirement when ZSTD_obm_stable is not enabled. */
+    if (zds->outBufferMode != ZSTD_obm_stable)
+        return 0;
+    /* Any buffer is allowed in zdss_init, this must be the same for every other call until
+     * the context is reset.
+     */
+    if (zds->streamStage == zdss_init)
+        return 0;
+    /* The buffer must match our expectation exactly. */
+    if (expect.dst == output->dst && expect.pos == output->pos && expect.size == output->size)
+        return 0;
+    RETURN_ERROR(dstBuffer_wrong, "ZSTD_obm_stable enabled but output differs!");
 }
 
+/* Calls ZSTD_decompressContinue() with the right parameters for ZSTD_decompressStream()
+ * and updates the stage and the output buffer state. This call is extracted so it can be
+ * used both when reading directly from the ZSTD_inBuffer, and in buffered input mode.
+ * NOTE: You must break after calling this function since the streamStage is modified.
+ */
+static size_t ZSTD_decompressContinueStream(
+            ZSTD_DStream* zds, char** op, char* oend,
+            void const* src, size_t srcSize) {
+    int const isSkipFrame = ZSTD_isSkipFrame(zds);
+    if (zds->outBufferMode == ZSTD_obm_buffered) {
+        size_t const dstSize = isSkipFrame ? 0 : zds->outBuffSize - zds->outStart;
+        size_t const decodedSize = ZSTD_decompressContinue(zds,
+                zds->outBuff + zds->outStart, dstSize, src, srcSize);
+        FORWARD_IF_ERROR(decodedSize, "");
+        if (!decodedSize && !isSkipFrame) {
+            zds->streamStage = zdss_read;
+        } else {
+            zds->outEnd = zds->outStart + decodedSize;
+            zds->streamStage = zdss_flush;
+        }
+    } else {
+        /* Write directly into the output buffer */
+        size_t const dstSize = isSkipFrame ? 0 : oend - *op;
+        size_t const decodedSize = ZSTD_decompressContinue(zds, *op, dstSize, src, srcSize);
+        FORWARD_IF_ERROR(decodedSize, "");
+        *op += decodedSize;
+        /* Flushing is not needed. */
+        zds->streamStage = zdss_read;
+        assert(*op <= oend);
+        assert(zds->outBufferMode == ZSTD_obm_stable);
+    }
+    return 0;
+}
 
 size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inBuffer* input)
 {
-    const char* const istart = (const char*)(input->src) + input->pos;
-    const char* const iend = (const char*)(input->src) + input->size;
+    const char* const src = (const char*)input->src;
+    const char* const istart = input->pos != 0 ? src + input->pos : src;
+    const char* const iend = input->size != 0 ? src + input->size : src;
     const char* ip = istart;
-    char* const ostart = (char*)(output->dst) + output->pos;
-    char* const oend = (char*)(output->dst) + output->size;
+    char* const dst = (char*)output->dst;
+    char* const ostart = output->pos != 0 ? dst + output->pos : dst;
+    char* const oend = output->size != 0 ? dst + output->size : dst;
     char* op = ostart;
     U32 someMoreWork = 1;
 
@@ -1506,6 +1609,7 @@ size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inB
         "forbidden. out: pos: %u   vs size: %u",
         (U32)output->pos, (U32)output->size);
     DEBUGLOG(5, "input size : %u", (U32)(input->size - input->pos));
+    FORWARD_IF_ERROR(ZSTD_checkOutBuffer(zds, output), "");
 
     while (someMoreWork) {
         switch(zds->streamStage)
@@ -1516,6 +1620,7 @@ size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inB
             zds->lhSize = zds->inPos = zds->outStart = zds->outEnd = 0;
             zds->legacyVersion = 0;
             zds->hostageByte = 0;
+            zds->expectedOutBuffer = *output;
             /* fall-through */
 
         case zdss_loadHeader :
@@ -1543,7 +1648,7 @@ size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inB
                             "legacy support is incompatible with static dctx");
                         FORWARD_IF_ERROR(ZSTD_initLegacyStream(&zds->legacyContext,
                                     zds->previousLegacyVersion, legacyVersion,
-                                    dict, dictSize));
+                                    dict, dictSize), "");
                         zds->legacyVersion = zds->previousLegacyVersion = legacyVersion;
                         {   size_t const hint = ZSTD_decompressLegacyStream(zds->legacyContext, legacyVersion, output, input);
                             if (hint==0) zds->streamStage = zdss_init;   /* or stay in stage zdss_loadHeader */
@@ -1570,7 +1675,8 @@ size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inB
             }   }
 
             /* check for single-pass mode opportunity */
-            if (zds->fParams.frameContentSize && zds->fParams.windowSize /* skippable frame if == 0 */
+            if (zds->fParams.frameContentSize != ZSTD_CONTENTSIZE_UNKNOWN
+                && zds->fParams.frameType != ZSTD_skippableFrame
                 && (U64)(size_t)(oend-op) >= zds->fParams.frameContentSize) {
                 size_t const cSize = ZSTD_findFrameCompressedSize(istart, iend-istart);
                 if (cSize <= (size_t)(iend-istart)) {
@@ -1586,15 +1692,23 @@ size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inB
                     break;
             }   }
 
+            /* Check output buffer is large enough for ZSTD_odm_stable. */
+            if (zds->outBufferMode == ZSTD_obm_stable
+                && zds->fParams.frameType != ZSTD_skippableFrame
+                && zds->fParams.frameContentSize != ZSTD_CONTENTSIZE_UNKNOWN
+                && (U64)(size_t)(oend-op) < zds->fParams.frameContentSize) {
+                RETURN_ERROR(dstSize_tooSmall, "ZSTD_obm_stable passed but ZSTD_outBuffer is too small");
+            }
+
             /* Consume header (see ZSTDds_decodeFrameHeader) */
             DEBUGLOG(4, "Consume header");
-            FORWARD_IF_ERROR(ZSTD_decompressBegin_usingDDict(zds, ZSTD_getDDict(zds)));
+            FORWARD_IF_ERROR(ZSTD_decompressBegin_usingDDict(zds, ZSTD_getDDict(zds)), "");
 
             if ((MEM_readLE32(zds->headerBuffer) & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) {  /* skippable frame */
                 zds->expected = MEM_readLE32(zds->headerBuffer + ZSTD_FRAMEIDSIZE);
                 zds->stage = ZSTDds_skipFrame;
             } else {
-                FORWARD_IF_ERROR(ZSTD_decodeFrameHeader(zds, zds->headerBuffer, zds->lhSize));
+                FORWARD_IF_ERROR(ZSTD_decodeFrameHeader(zds, zds->headerBuffer, zds->lhSize), "");
                 zds->expected = ZSTD_blockHeaderSize;
                 zds->stage = ZSTDds_decodeBlockHeader;
             }
@@ -1605,40 +1719,48 @@ size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inB
                         (U32)(zds->maxWindowSize >> 10) );
             zds->fParams.windowSize = MAX(zds->fParams.windowSize, 1U << ZSTD_WINDOWLOG_ABSOLUTEMIN);
             RETURN_ERROR_IF(zds->fParams.windowSize > zds->maxWindowSize,
-                            frameParameter_windowTooLarge);
+                            frameParameter_windowTooLarge, "");
 
             /* Adapt buffer sizes to frame header instructions */
             {   size_t const neededInBuffSize = MAX(zds->fParams.blockSizeMax, 4 /* frame checksum */);
-                size_t const neededOutBuffSize = ZSTD_decodingBufferSize_min(zds->fParams.windowSize, zds->fParams.frameContentSize);
-                if ((zds->inBuffSize < neededInBuffSize) || (zds->outBuffSize < neededOutBuffSize)) {
-                    size_t const bufferSize = neededInBuffSize + neededOutBuffSize;
-                    DEBUGLOG(4, "inBuff  : from %u to %u",
-                                (U32)zds->inBuffSize, (U32)neededInBuffSize);
-                    DEBUGLOG(4, "outBuff : from %u to %u",
-                                (U32)zds->outBuffSize, (U32)neededOutBuffSize);
-                    if (zds->staticSize) {  /* static DCtx */
-                        DEBUGLOG(4, "staticSize : %u", (U32)zds->staticSize);
-                        assert(zds->staticSize >= sizeof(ZSTD_DCtx));  /* controlled at init */
-                        RETURN_ERROR_IF(
-                            bufferSize > zds->staticSize - sizeof(ZSTD_DCtx),
-                            memory_allocation);
-                    } else {
-                        ZSTD_free(zds->inBuff, zds->customMem);
-                        zds->inBuffSize = 0;
-                        zds->outBuffSize = 0;
-                        zds->inBuff = (char*)ZSTD_malloc(bufferSize, zds->customMem);
-                        RETURN_ERROR_IF(zds->inBuff == NULL, memory_allocation);
-                    }
-                    zds->inBuffSize = neededInBuffSize;
-                    zds->outBuff = zds->inBuff + zds->inBuffSize;
-                    zds->outBuffSize = neededOutBuffSize;
-            }   }
+                size_t const neededOutBuffSize = zds->outBufferMode == ZSTD_obm_buffered
+                        ? ZSTD_decodingBufferSize_min(zds->fParams.windowSize, zds->fParams.frameContentSize)
+                        : 0;
+
+                ZSTD_DCtx_updateOversizedDuration(zds, neededInBuffSize, neededOutBuffSize);
+
+                {   int const tooSmall = (zds->inBuffSize < neededInBuffSize) || (zds->outBuffSize < neededOutBuffSize);
+                    int const tooLarge = ZSTD_DCtx_isOversizedTooLong(zds);
+                    
+                    if (tooSmall || tooLarge) {
+                        size_t const bufferSize = neededInBuffSize + neededOutBuffSize;
+                        DEBUGLOG(4, "inBuff  : from %u to %u",
+                                    (U32)zds->inBuffSize, (U32)neededInBuffSize);
+                        DEBUGLOG(4, "outBuff : from %u to %u",
+                                    (U32)zds->outBuffSize, (U32)neededOutBuffSize);
+                        if (zds->staticSize) {  /* static DCtx */
+                            DEBUGLOG(4, "staticSize : %u", (U32)zds->staticSize);
+                            assert(zds->staticSize >= sizeof(ZSTD_DCtx));  /* controlled at init */
+                            RETURN_ERROR_IF(
+                                bufferSize > zds->staticSize - sizeof(ZSTD_DCtx),
+                                memory_allocation, "");
+                        } else {
+                            ZSTD_free(zds->inBuff, zds->customMem);
+                            zds->inBuffSize = 0;
+                            zds->outBuffSize = 0;
+                            zds->inBuff = (char*)ZSTD_malloc(bufferSize, zds->customMem);
+                            RETURN_ERROR_IF(zds->inBuff == NULL, memory_allocation, "");
+                        }
+                        zds->inBuffSize = neededInBuffSize;
+                        zds->outBuff = zds->inBuff + zds->inBuffSize;
+                        zds->outBuffSize = neededOutBuffSize;
+            }   }   }
             zds->streamStage = zdss_read;
             /* fall-through */
 
         case zdss_read:
             DEBUGLOG(5, "stage zdss_read");
-            {   size_t const neededInSize = ZSTD_nextSrcSizeToDecompress(zds);
+            {   size_t const neededInSize = ZSTD_nextSrcSizeToDecompressWithInputSize(zds, iend - ip);
                 DEBUGLOG(5, "neededInSize = %u", (U32)neededInSize);
                 if (neededInSize==0) {  /* end of frame */
                     zds->streamStage = zdss_init;
@@ -1646,15 +1768,9 @@ size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inB
                     break;
                 }
                 if ((size_t)(iend-ip) >= neededInSize) {  /* decode directly from src */
-                    int const isSkipFrame = ZSTD_isSkipFrame(zds);
-                    size_t const decodedSize = ZSTD_decompressContinue(zds,
-                        zds->outBuff + zds->outStart, (isSkipFrame ? 0 : zds->outBuffSize - zds->outStart),
-                        ip, neededInSize);
-                    if (ZSTD_isError(decodedSize)) return decodedSize;
+                    FORWARD_IF_ERROR(ZSTD_decompressContinueStream(zds, &op, oend, ip, neededInSize), "");
                     ip += neededInSize;
-                    if (!decodedSize && !isSkipFrame) break;   /* this was just a header */
-                    zds->outEnd = zds->outStart + decodedSize;
-                    zds->streamStage = zdss_flush;
+                    /* Function modifies the stage so we must break */
                     break;
             }   }
             if (ip==iend) { someMoreWork = 0; break; }   /* no more input */
@@ -1666,6 +1782,8 @@ size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inB
                 size_t const toLoad = neededInSize - zds->inPos;
                 int const isSkipFrame = ZSTD_isSkipFrame(zds);
                 size_t loadedSize;
+                /* At this point we shouldn't be decompressing a block that we can stream. */
+                assert(neededInSize == ZSTD_nextSrcSizeToDecompressWithInputSize(zds, iend - ip));
                 if (isSkipFrame) {
                     loadedSize = MIN(toLoad, (size_t)(iend-ip));
                 } else {
@@ -1679,17 +1797,11 @@ size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inB
                 if (loadedSize < toLoad) { someMoreWork = 0; break; }   /* not enough input, wait for more */
 
                 /* decode loaded input */
-                {   size_t const decodedSize = ZSTD_decompressContinue(zds,
-                        zds->outBuff + zds->outStart, zds->outBuffSize - zds->outStart,
-                        zds->inBuff, neededInSize);
-                    if (ZSTD_isError(decodedSize)) return decodedSize;
-                    zds->inPos = 0;   /* input is consumed */
-                    if (!decodedSize && !isSkipFrame) { zds->streamStage = zdss_read; break; }   /* this was just a header */
-                    zds->outEnd = zds->outStart +  decodedSize;
-            }   }
-            zds->streamStage = zdss_flush;
-            /* fall-through */
-
+                zds->inPos = 0;   /* input is consumed */
+                FORWARD_IF_ERROR(ZSTD_decompressContinueStream(zds, &op, oend, zds->inBuff, neededInSize), "");
+                /* Function modifies the stage so we must break */
+                break;
+            }
         case zdss_flush:
             {   size_t const toFlushSize = zds->outEnd - zds->outStart;
                 size_t const flushedSize = ZSTD_limitCopy(op, oend-op, zds->outBuff + zds->outStart, toFlushSize);
@@ -1712,17 +1824,21 @@ size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inB
 
         default:
             assert(0);    /* impossible */
-            RETURN_ERROR(GENERIC);   /* some compiler require default to do something */
+            RETURN_ERROR(GENERIC, "impossible to reach");   /* some compiler require default to do something */
     }   }
 
     /* result */
     input->pos = (size_t)(ip - (const char*)(input->src));
     output->pos = (size_t)(op - (char*)(output->dst));
+
+    /* Update the expected output buffer for ZSTD_obm_stable. */
+    zds->expectedOutBuffer = *output;
+
     if ((ip==istart) && (op==ostart)) {  /* no forward progress */
         zds->noForwardProgress ++;
         if (zds->noForwardProgress >= ZSTD_NO_FORWARD_PROGRESS_MAX) {
-            RETURN_ERROR_IF(op==oend, dstSize_tooSmall);
-            RETURN_ERROR_IF(ip==iend, srcSize_wrong);
+            RETURN_ERROR_IF(op==oend, dstSize_tooSmall, "");
+            RETURN_ERROR_IF(ip==iend, srcSize_wrong, "");
             assert(0);
         }
     } else {
diff --git a/lib/decompress/zstd_decompress_block.c b/lib/decompress/zstd_decompress_block.c
index 767e5f9a0b08..ad3b3d8dbd0a 100644
--- a/lib/decompress/zstd_decompress_block.c
+++ b/lib/decompress/zstd_decompress_block.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -15,14 +15,14 @@
 *  Dependencies
 *********************************************************/
 #include <string.h>      /* memcpy, memmove, memset */
-#include "compiler.h"    /* prefetch */
-#include "cpu.h"         /* bmi2 */
-#include "mem.h"         /* low level memory routines */
+#include "../common/compiler.h"    /* prefetch */
+#include "../common/cpu.h"         /* bmi2 */
+#include "../common/mem.h"         /* low level memory routines */
 #define FSE_STATIC_LINKING_ONLY
-#include "fse.h"
+#include "../common/fse.h"
 #define HUF_STATIC_LINKING_ONLY
-#include "huf.h"
-#include "zstd_internal.h"
+#include "../common/huf.h"
+#include "../common/zstd_internal.h"
 #include "zstd_decompress_internal.h"   /* ZSTD_DCtx */
 #include "zstd_ddict.h"  /* ZSTD_DDictDictContent */
 #include "zstd_decompress_block.h"
@@ -56,7 +56,7 @@ static void ZSTD_copy4(void* dst, const void* src) { memcpy(dst, src, 4); }
 size_t ZSTD_getcBlockSize(const void* src, size_t srcSize,
                           blockProperties_t* bpPtr)
 {
-    RETURN_ERROR_IF(srcSize < ZSTD_blockHeaderSize, srcSize_wrong);
+    RETURN_ERROR_IF(srcSize < ZSTD_blockHeaderSize, srcSize_wrong, "");
 
     {   U32 const cBlockHeader = MEM_readLE24(src);
         U32 const cSize = cBlockHeader >> 3;
@@ -64,7 +64,7 @@ size_t ZSTD_getcBlockSize(const void* src, size_t srcSize,
         bpPtr->blockType = (blockType_e)((cBlockHeader >> 1) & 3);
         bpPtr->origSize = cSize;   /* only useful for RLE */
         if (bpPtr->blockType == bt_rle) return 1;
-        RETURN_ERROR_IF(bpPtr->blockType == bt_reserved, corruption_detected);
+        RETURN_ERROR_IF(bpPtr->blockType == bt_reserved, corruption_detected, "");
         return cSize;
     }
 }
@@ -80,7 +80,7 @@ size_t ZSTD_decodeLiteralsBlock(ZSTD_DCtx* dctx,
                           const void* src, size_t srcSize)   /* note : srcSize < BLOCKSIZE */
 {
     DEBUGLOG(5, "ZSTD_decodeLiteralsBlock");
-    RETURN_ERROR_IF(srcSize < MIN_CBLOCK_SIZE, corruption_detected);
+    RETURN_ERROR_IF(srcSize < MIN_CBLOCK_SIZE, corruption_detected, "");
 
     {   const BYTE* const istart = (const BYTE*) src;
         symbolEncodingType_e const litEncType = (symbolEncodingType_e)(istart[0] & 3);
@@ -89,7 +89,7 @@ size_t ZSTD_decodeLiteralsBlock(ZSTD_DCtx* dctx,
         {
         case set_repeat:
             DEBUGLOG(5, "set_repeat flag : re-using stats from previous compressed literals block");
-            RETURN_ERROR_IF(dctx->litEntropy==0, dictionary_corrupted);
+            RETURN_ERROR_IF(dctx->litEntropy==0, dictionary_corrupted, "");
             /* fall-through */
 
         case set_compressed:
@@ -121,8 +121,8 @@ size_t ZSTD_decodeLiteralsBlock(ZSTD_DCtx* dctx,
                     litCSize = (lhc >> 22) + ((size_t)istart[4] << 10);
                     break;
                 }
-                RETURN_ERROR_IF(litSize > ZSTD_BLOCKSIZE_MAX, corruption_detected);
-                RETURN_ERROR_IF(litCSize + lhSize > srcSize, corruption_detected);
+                RETURN_ERROR_IF(litSize > ZSTD_BLOCKSIZE_MAX, corruption_detected, "");
+                RETURN_ERROR_IF(litCSize + lhSize > srcSize, corruption_detected, "");
 
                 /* prefetch huffman table if cold */
                 if (dctx->ddictIsCold && (litSize > 768 /* heuristic */)) {
@@ -160,7 +160,7 @@ size_t ZSTD_decodeLiteralsBlock(ZSTD_DCtx* dctx,
                     }
                 }
 
-                RETURN_ERROR_IF(HUF_isError(hufSuccess), corruption_detected);
+                RETURN_ERROR_IF(HUF_isError(hufSuccess), corruption_detected, "");
 
                 dctx->litPtr = dctx->litBuffer;
                 dctx->litSize = litSize;
@@ -190,7 +190,7 @@ size_t ZSTD_decodeLiteralsBlock(ZSTD_DCtx* dctx,
                 }
 
                 if (lhSize+litSize+WILDCOPY_OVERLENGTH > srcSize) {  /* risk reading beyond src buffer with wildcopy */
-                    RETURN_ERROR_IF(litSize+lhSize > srcSize, corruption_detected);
+                    RETURN_ERROR_IF(litSize+lhSize > srcSize, corruption_detected, "");
                     memcpy(dctx->litBuffer, istart+lhSize, litSize);
                     dctx->litPtr = dctx->litBuffer;
                     dctx->litSize = litSize;
@@ -222,7 +222,7 @@ size_t ZSTD_decodeLiteralsBlock(ZSTD_DCtx* dctx,
                     RETURN_ERROR_IF(srcSize<4, corruption_detected, "srcSize >= MIN_CBLOCK_SIZE == 3; here we need lhSize+1 = 4");
                     break;
                 }
-                RETURN_ERROR_IF(litSize > ZSTD_BLOCKSIZE_MAX, corruption_detected);
+                RETURN_ERROR_IF(litSize > ZSTD_BLOCKSIZE_MAX, corruption_detected, "");
                 memset(dctx->litBuffer, istart[lhSize], litSize + WILDCOPY_OVERLENGTH);
                 dctx->litPtr = dctx->litBuffer;
                 dctx->litSize = litSize;
@@ -440,8 +440,8 @@ static size_t ZSTD_buildSeqTable(ZSTD_seqSymbol* DTableSpace, const ZSTD_seqSymb
     switch(type)
     {
     case set_rle :
-        RETURN_ERROR_IF(!srcSize, srcSize_wrong);
-        RETURN_ERROR_IF((*(const BYTE*)src) > max, corruption_detected);
+        RETURN_ERROR_IF(!srcSize, srcSize_wrong, "");
+        RETURN_ERROR_IF((*(const BYTE*)src) > max, corruption_detected, "");
         {   U32 const symbol = *(const BYTE*)src;
             U32 const baseline = baseValue[symbol];
             U32 const nbBits = nbAdditionalBits[symbol];
@@ -453,7 +453,7 @@ static size_t ZSTD_buildSeqTable(ZSTD_seqSymbol* DTableSpace, const ZSTD_seqSymb
         *DTablePtr = defaultTable;
         return 0;
     case set_repeat:
-        RETURN_ERROR_IF(!flagRepeatTable, corruption_detected);
+        RETURN_ERROR_IF(!flagRepeatTable, corruption_detected, "");
         /* prefetch FSE table if used */
         if (ddictIsCold && (nbSeq > 24 /* heuristic */)) {
             const void* const pStart = *DTablePtr;
@@ -465,8 +465,8 @@ static size_t ZSTD_buildSeqTable(ZSTD_seqSymbol* DTableSpace, const ZSTD_seqSymb
         {   unsigned tableLog;
             S16 norm[MaxSeq+1];
             size_t const headerSize = FSE_readNCount(norm, &max, &tableLog, src, srcSize);
-            RETURN_ERROR_IF(FSE_isError(headerSize), corruption_detected);
-            RETURN_ERROR_IF(tableLog > maxLog, corruption_detected);
+            RETURN_ERROR_IF(FSE_isError(headerSize), corruption_detected, "");
+            RETURN_ERROR_IF(tableLog > maxLog, corruption_detected, "");
             ZSTD_buildFSETable(DTableSpace, norm, max, baseValue, nbAdditionalBits, tableLog);
             *DTablePtr = DTableSpace;
             return headerSize;
@@ -487,28 +487,28 @@ size_t ZSTD_decodeSeqHeaders(ZSTD_DCtx* dctx, int* nbSeqPtr,
     DEBUGLOG(5, "ZSTD_decodeSeqHeaders");
 
     /* check */
-    RETURN_ERROR_IF(srcSize < MIN_SEQUENCES_SIZE, srcSize_wrong);
+    RETURN_ERROR_IF(srcSize < MIN_SEQUENCES_SIZE, srcSize_wrong, "");
 
     /* SeqHead */
     nbSeq = *ip++;
     if (!nbSeq) {
         *nbSeqPtr=0;
-        RETURN_ERROR_IF(srcSize != 1, srcSize_wrong);
+        RETURN_ERROR_IF(srcSize != 1, srcSize_wrong, "");
         return 1;
     }
     if (nbSeq > 0x7F) {
         if (nbSeq == 0xFF) {
-            RETURN_ERROR_IF(ip+2 > iend, srcSize_wrong);
+            RETURN_ERROR_IF(ip+2 > iend, srcSize_wrong, "");
             nbSeq = MEM_readLE16(ip) + LONGNBSEQ, ip+=2;
         } else {
-            RETURN_ERROR_IF(ip >= iend, srcSize_wrong);
+            RETURN_ERROR_IF(ip >= iend, srcSize_wrong, "");
             nbSeq = ((nbSeq-0x80)<<8) + *ip++;
         }
     }
     *nbSeqPtr = nbSeq;
 
     /* FSE table descriptors */
-    RETURN_ERROR_IF(ip+1 > iend, srcSize_wrong); /* minimum possible size: 1 byte for symbol encoding types */
+    RETURN_ERROR_IF(ip+1 > iend, srcSize_wrong, ""); /* minimum possible size: 1 byte for symbol encoding types */
     {   symbolEncodingType_e const LLtype = (symbolEncodingType_e)(*ip >> 6);
         symbolEncodingType_e const OFtype = (symbolEncodingType_e)((*ip >> 4) & 3);
         symbolEncodingType_e const MLtype = (symbolEncodingType_e)((*ip >> 2) & 3);
@@ -521,7 +521,7 @@ size_t ZSTD_decodeSeqHeaders(ZSTD_DCtx* dctx, int* nbSeqPtr,
                                                       LL_base, LL_bits,
                                                       LL_defaultDTable, dctx->fseEntropy,
                                                       dctx->ddictIsCold, nbSeq);
-            RETURN_ERROR_IF(ZSTD_isError(llhSize), corruption_detected);
+            RETURN_ERROR_IF(ZSTD_isError(llhSize), corruption_detected, "ZSTD_buildSeqTable failed");
             ip += llhSize;
         }
 
@@ -531,7 +531,7 @@ size_t ZSTD_decodeSeqHeaders(ZSTD_DCtx* dctx, int* nbSeqPtr,
                                                       OF_base, OF_bits,
                                                       OF_defaultDTable, dctx->fseEntropy,
                                                       dctx->ddictIsCold, nbSeq);
-            RETURN_ERROR_IF(ZSTD_isError(ofhSize), corruption_detected);
+            RETURN_ERROR_IF(ZSTD_isError(ofhSize), corruption_detected, "ZSTD_buildSeqTable failed");
             ip += ofhSize;
         }
 
@@ -541,7 +541,7 @@ size_t ZSTD_decodeSeqHeaders(ZSTD_DCtx* dctx, int* nbSeqPtr,
                                                       ML_base, ML_bits,
                                                       ML_defaultDTable, dctx->fseEntropy,
                                                       dctx->ddictIsCold, nbSeq);
-            RETURN_ERROR_IF(ZSTD_isError(mlhSize), corruption_detected);
+            RETURN_ERROR_IF(ZSTD_isError(mlhSize), corruption_detected, "ZSTD_buildSeqTable failed");
             ip += mlhSize;
         }
     }
@@ -580,7 +580,7 @@ typedef struct {
  *  Precondition: *ip <= *op
  *  Postcondition: *op - *op >= 8
  */
-static void ZSTD_overlapCopy8(BYTE** op, BYTE const** ip, size_t offset) {
+HINT_INLINE void ZSTD_overlapCopy8(BYTE** op, BYTE const** ip, size_t offset) {
     assert(*ip <= *op);
     if (offset < 8) {
         /* close range match, overlap */
@@ -665,15 +665,15 @@ size_t ZSTD_execSequenceEnd(BYTE* op,
 {
     BYTE* const oLitEnd = op + sequence.litLength;
     size_t const sequenceLength = sequence.litLength + sequence.matchLength;
-    BYTE* const oMatchEnd = op + sequenceLength;   /* risk : address space overflow (32-bits) */
     const BYTE* const iLitEnd = *litPtr + sequence.litLength;
     const BYTE* match = oLitEnd - sequence.offset;
     BYTE* const oend_w = oend - WILDCOPY_OVERLENGTH;
 
-    /* bounds checks */
-    assert(oLitEnd < oMatchEnd);
-    RETURN_ERROR_IF(oMatchEnd > oend, dstSize_tooSmall, "last match must fit within dstBuffer");
-    RETURN_ERROR_IF(iLitEnd > litLimit, corruption_detected, "try to read beyond literal buffer");
+    /* bounds checks : careful of address space overflow in 32-bit mode */
+    RETURN_ERROR_IF(sequenceLength > (size_t)(oend - op), dstSize_tooSmall, "last match must fit within dstBuffer");
+    RETURN_ERROR_IF(sequence.litLength > (size_t)(litLimit - *litPtr), corruption_detected, "try to read beyond literal buffer");
+    assert(op < op + sequenceLength);
+    assert(oLitEnd < op + sequenceLength);
 
     /* copy literals */
     ZSTD_safecopy(op, oend_w, *litPtr, sequence.litLength, ZSTD_no_overlap);
@@ -683,7 +683,7 @@ size_t ZSTD_execSequenceEnd(BYTE* op,
     /* copy Match */
     if (sequence.offset > (size_t)(oLitEnd - prefixStart)) {
         /* offset beyond prefix */
-        RETURN_ERROR_IF(sequence.offset > (size_t)(oLitEnd - virtualStart), corruption_detected);
+        RETURN_ERROR_IF(sequence.offset > (size_t)(oLitEnd - virtualStart), corruption_detected, "");
         match = dictEnd - (prefixStart-match);
         if (match + sequence.matchLength <= dictEnd) {
             memmove(oLitEnd, match, sequence.matchLength);
@@ -709,16 +709,27 @@ size_t ZSTD_execSequence(BYTE* op,
     BYTE* const oLitEnd = op + sequence.litLength;
     size_t const sequenceLength = sequence.litLength + sequence.matchLength;
     BYTE* const oMatchEnd = op + sequenceLength;   /* risk : address space overflow (32-bits) */
-    BYTE* const oend_w = oend - WILDCOPY_OVERLENGTH;
+    BYTE* const oend_w = oend - WILDCOPY_OVERLENGTH;   /* risk : address space underflow on oend=NULL */
     const BYTE* const iLitEnd = *litPtr + sequence.litLength;
     const BYTE* match = oLitEnd - sequence.offset;
 
-    /* Errors and uncommon cases handled here. */
-    assert(oLitEnd < oMatchEnd);
-    if (iLitEnd > litLimit || oMatchEnd > oend_w)
+    assert(op != NULL /* Precondition */);
+    assert(oend_w < oend /* No underflow */);
+    /* Handle edge cases in a slow path:
+     *   - Read beyond end of literals
+     *   - Match end is within WILDCOPY_OVERLIMIT of oend
+     *   - 32-bit mode and the match length overflows
+     */
+    if (UNLIKELY(
+            iLitEnd > litLimit ||
+            oMatchEnd > oend_w ||
+            (MEM_32bits() && (size_t)(oend - op) < sequenceLength + WILDCOPY_OVERLENGTH)))
         return ZSTD_execSequenceEnd(op, oend, sequence, litPtr, litLimit, prefixStart, virtualStart, dictEnd);
 
     /* Assumptions (everything else goes into ZSTD_execSequenceEnd()) */
+    assert(op <= oLitEnd /* No overflow */);
+    assert(oLitEnd < oMatchEnd /* Non-zero match & no overflow */);
+    assert(oMatchEnd <= oend /* No underflow */);
     assert(iLitEnd <= litLimit /* Literal length is in bounds */);
     assert(oLitEnd <= oend_w /* Can wildcopy literals */);
     assert(oMatchEnd <= oend_w /* Can wildcopy matches */);
@@ -729,7 +740,7 @@ size_t ZSTD_execSequence(BYTE* op,
      */
     assert(WILDCOPY_OVERLENGTH >= 16);
     ZSTD_copy16(op, (*litPtr));
-    if (sequence.litLength > 16) {
+    if (UNLIKELY(sequence.litLength > 16)) {
         ZSTD_wildcopy(op+16, (*litPtr)+16, sequence.litLength-16, ZSTD_no_overlap);
     }
     op = oLitEnd;
@@ -738,7 +749,7 @@ size_t ZSTD_execSequence(BYTE* op,
     /* Copy Match */
     if (sequence.offset > (size_t)(oLitEnd - prefixStart)) {
         /* offset beyond prefix -> go into extDict */
-        RETURN_ERROR_IF(sequence.offset > (size_t)(oLitEnd - virtualStart), corruption_detected);
+        RETURN_ERROR_IF(UNLIKELY(sequence.offset > (size_t)(oLitEnd - virtualStart)), corruption_detected, "");
         match = dictEnd + (match - prefixStart);
         if (match + sequence.matchLength <= dictEnd) {
             memmove(oLitEnd, match, sequence.matchLength);
@@ -760,7 +771,7 @@ size_t ZSTD_execSequence(BYTE* op,
     /* Nearly all offsets are >= WILDCOPY_VECLEN bytes, which means we can use wildcopy
      * without overlap checking.
      */
-    if (sequence.offset >= WILDCOPY_VECLEN) {
+    if (LIKELY(sequence.offset >= WILDCOPY_VECLEN)) {
         /* We bet on a full wildcopy for matches, since we expect matches to be
          * longer than literals (in general). In silesia, ~10% of matches are longer
          * than 16 bytes.
@@ -802,6 +813,14 @@ ZSTD_updateFseState(ZSTD_fseState* DStatePtr, BIT_DStream_t* bitD)
     DStatePtr->state = DInfo.nextState + lowBits;
 }
 
+FORCE_INLINE_TEMPLATE void
+ZSTD_updateFseStateWithDInfo(ZSTD_fseState* DStatePtr, BIT_DStream_t* bitD, ZSTD_seqSymbol const DInfo)
+{
+    U32 const nbBits = DInfo.nbBits;
+    size_t const lowBits = BIT_readBits(bitD, nbBits);
+    DStatePtr->state = DInfo.nextState + lowBits;
+}
+
 /* We need to add at most (ZSTD_WINDOWLOG_MAX_32 - 1) bits to read the maximum
  * offset bits. But we can only read at most (STREAM_ACCUMULATOR_MIN_32 - 1)
  * bits before reloading. This value is the maximum number of bytes we read
@@ -813,25 +832,26 @@ ZSTD_updateFseState(ZSTD_fseState* DStatePtr, BIT_DStream_t* bitD)
         : 0)
 
 typedef enum { ZSTD_lo_isRegularOffset, ZSTD_lo_isLongOffset=1 } ZSTD_longOffset_e;
+typedef enum { ZSTD_p_noPrefetch=0, ZSTD_p_prefetch=1 } ZSTD_prefetch_e;
 
-#ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG
 FORCE_INLINE_TEMPLATE seq_t
-ZSTD_decodeSequence(seqState_t* seqState, const ZSTD_longOffset_e longOffsets)
+ZSTD_decodeSequence(seqState_t* seqState, const ZSTD_longOffset_e longOffsets, const ZSTD_prefetch_e prefetch)
 {
     seq_t seq;
-    U32 const llBits = seqState->stateLL.table[seqState->stateLL.state].nbAdditionalBits;
-    U32 const mlBits = seqState->stateML.table[seqState->stateML.state].nbAdditionalBits;
-    U32 const ofBits = seqState->stateOffb.table[seqState->stateOffb.state].nbAdditionalBits;
-    U32 const totalBits = llBits+mlBits+ofBits;
-    U32 const llBase = seqState->stateLL.table[seqState->stateLL.state].baseValue;
-    U32 const mlBase = seqState->stateML.table[seqState->stateML.state].baseValue;
-    U32 const ofBase = seqState->stateOffb.table[seqState->stateOffb.state].baseValue;
+    ZSTD_seqSymbol const llDInfo = seqState->stateLL.table[seqState->stateLL.state];
+    ZSTD_seqSymbol const mlDInfo = seqState->stateML.table[seqState->stateML.state];
+    ZSTD_seqSymbol const ofDInfo = seqState->stateOffb.table[seqState->stateOffb.state];
+    U32 const llBase = llDInfo.baseValue;
+    U32 const mlBase = mlDInfo.baseValue;
+    U32 const ofBase = ofDInfo.baseValue;
+    BYTE const llBits = llDInfo.nbAdditionalBits;
+    BYTE const mlBits = mlDInfo.nbAdditionalBits;
+    BYTE const ofBits = ofDInfo.nbAdditionalBits;
+    BYTE const totalBits = llBits+mlBits+ofBits;
 
     /* sequence */
     {   size_t offset;
-        if (!ofBits)
-            offset = 0;
-        else {
+        if (ofBits > 1) {
             ZSTD_STATIC_ASSERT(ZSTD_lo_isLongOffset == 1);
             ZSTD_STATIC_ASSERT(LONG_OFFSETS_MAX_EXTRA_BITS_32 == 5);
             assert(ofBits <= MaxOff);
@@ -845,59 +865,138 @@ ZSTD_decodeSequence(seqState_t* seqState, const ZSTD_longOffset_e longOffsets)
                 offset = ofBase + BIT_readBitsFast(&seqState->DStream, ofBits/*>0*/);   /* <=  (ZSTD_WINDOWLOG_MAX-1) bits */
                 if (MEM_32bits()) BIT_reloadDStream(&seqState->DStream);
             }
-        }
-
-        if (ofBits <= 1) {
-            offset += (llBase==0);
-            if (offset) {
-                size_t temp = (offset==3) ? seqState->prevOffset[0] - 1 : seqState->prevOffset[offset];
-                temp += !temp;   /* 0 is not valid; input is corrupted; force offset to 1 */
-                if (offset != 1) seqState->prevOffset[2] = seqState->prevOffset[1];
-                seqState->prevOffset[1] = seqState->prevOffset[0];
-                seqState->prevOffset[0] = offset = temp;
-            } else {  /* offset == 0 */
-                offset = seqState->prevOffset[0];
-            }
-        } else {
             seqState->prevOffset[2] = seqState->prevOffset[1];
             seqState->prevOffset[1] = seqState->prevOffset[0];
             seqState->prevOffset[0] = offset;
-        }
+        } else {
+            U32 const ll0 = (llBase == 0);
+            if (LIKELY((ofBits == 0))) {
+                if (LIKELY(!ll0))
+                    offset = seqState->prevOffset[0];
+                else {
+                    offset = seqState->prevOffset[1];
+                    seqState->prevOffset[1] = seqState->prevOffset[0];
+                    seqState->prevOffset[0] = offset;
+                }
+            } else {
+                offset = ofBase + ll0 + BIT_readBitsFast(&seqState->DStream, 1);
+                {   size_t temp = (offset==3) ? seqState->prevOffset[0] - 1 : seqState->prevOffset[offset];
+                    temp += !temp;   /* 0 is not valid; input is corrupted; force offset to 1 */
+                    if (offset != 1) seqState->prevOffset[2] = seqState->prevOffset[1];
+                    seqState->prevOffset[1] = seqState->prevOffset[0];
+                    seqState->prevOffset[0] = offset = temp;
+        }   }   }
         seq.offset = offset;
     }
 
-    seq.matchLength = mlBase
-                    + ((mlBits>0) ? BIT_readBitsFast(&seqState->DStream, mlBits/*>0*/) : 0);  /* <=  16 bits */
+    seq.matchLength = mlBase;
+    if (mlBits > 0)
+        seq.matchLength += BIT_readBitsFast(&seqState->DStream, mlBits/*>0*/);
+
     if (MEM_32bits() && (mlBits+llBits >= STREAM_ACCUMULATOR_MIN_32-LONG_OFFSETS_MAX_EXTRA_BITS_32))
         BIT_reloadDStream(&seqState->DStream);
-    if (MEM_64bits() && (totalBits >= STREAM_ACCUMULATOR_MIN_64-(LLFSELog+MLFSELog+OffFSELog)))
+    if (MEM_64bits() && UNLIKELY(totalBits >= STREAM_ACCUMULATOR_MIN_64-(LLFSELog+MLFSELog+OffFSELog)))
         BIT_reloadDStream(&seqState->DStream);
     /* Ensure there are enough bits to read the rest of data in 64-bit mode. */
     ZSTD_STATIC_ASSERT(16+LLFSELog+MLFSELog+OffFSELog < STREAM_ACCUMULATOR_MIN_64);
 
-    seq.litLength = llBase
-                  + ((llBits>0) ? BIT_readBitsFast(&seqState->DStream, llBits/*>0*/) : 0);    /* <=  16 bits */
+    seq.litLength = llBase;
+    if (llBits > 0)
+        seq.litLength += BIT_readBitsFast(&seqState->DStream, llBits/*>0*/);
+
     if (MEM_32bits())
         BIT_reloadDStream(&seqState->DStream);
 
     DEBUGLOG(6, "seq: litL=%u, matchL=%u, offset=%u",
                 (U32)seq.litLength, (U32)seq.matchLength, (U32)seq.offset);
 
-    /* ANS state update */
-    ZSTD_updateFseState(&seqState->stateLL, &seqState->DStream);    /* <=  9 bits */
-    ZSTD_updateFseState(&seqState->stateML, &seqState->DStream);    /* <=  9 bits */
-    if (MEM_32bits()) BIT_reloadDStream(&seqState->DStream);    /* <= 18 bits */
-    ZSTD_updateFseState(&seqState->stateOffb, &seqState->DStream);  /* <=  8 bits */
+    if (prefetch == ZSTD_p_prefetch) {
+        size_t const pos = seqState->pos + seq.litLength;
+        const BYTE* const matchBase = (seq.offset > pos) ? seqState->dictEnd : seqState->prefixStart;
+        seq.match = matchBase + pos - seq.offset;  /* note : this operation can overflow when seq.offset is really too large, which can only happen when input is corrupted.
+                                                    * No consequence though : no memory access will occur, offset is only used for prefetching */
+        seqState->pos = pos + seq.matchLength;
+    }
+
+    /* ANS state update
+     * gcc-9.0.0 does 2.5% worse with ZSTD_updateFseStateWithDInfo().
+     * clang-9.2.0 does 7% worse with ZSTD_updateFseState().
+     * Naturally it seems like ZSTD_updateFseStateWithDInfo() should be the
+     * better option, so it is the default for other compilers. But, if you
+     * measure that it is worse, please put up a pull request.
+     */
+    {
+#if defined(__GNUC__) && !defined(__clang__)
+        const int kUseUpdateFseState = 1;
+#else
+        const int kUseUpdateFseState = 0;
+#endif
+        if (kUseUpdateFseState) {
+            ZSTD_updateFseState(&seqState->stateLL, &seqState->DStream);    /* <=  9 bits */
+            ZSTD_updateFseState(&seqState->stateML, &seqState->DStream);    /* <=  9 bits */
+            if (MEM_32bits()) BIT_reloadDStream(&seqState->DStream);    /* <= 18 bits */
+            ZSTD_updateFseState(&seqState->stateOffb, &seqState->DStream);  /* <=  8 bits */
+        } else {
+            ZSTD_updateFseStateWithDInfo(&seqState->stateLL, &seqState->DStream, llDInfo);    /* <=  9 bits */
+            ZSTD_updateFseStateWithDInfo(&seqState->stateML, &seqState->DStream, mlDInfo);    /* <=  9 bits */
+            if (MEM_32bits()) BIT_reloadDStream(&seqState->DStream);    /* <= 18 bits */
+            ZSTD_updateFseStateWithDInfo(&seqState->stateOffb, &seqState->DStream, ofDInfo);  /* <=  8 bits */
+        }
+    }
 
     return seq;
 }
 
+#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
+static int ZSTD_dictionaryIsActive(ZSTD_DCtx const* dctx, BYTE const* prefixStart, BYTE const* oLitEnd)
+{
+    size_t const windowSize = dctx->fParams.windowSize;
+    /* No dictionary used. */
+    if (dctx->dictContentEndForFuzzing == NULL) return 0;
+    /* Dictionary is our prefix. */
+    if (prefixStart == dctx->dictContentBeginForFuzzing) return 1;
+    /* Dictionary is not our ext-dict. */
+    if (dctx->dictEnd != dctx->dictContentEndForFuzzing) return 0;
+    /* Dictionary is not within our window size. */
+    if ((size_t)(oLitEnd - prefixStart) >= windowSize) return 0;
+    /* Dictionary is active. */
+    return 1;
+}
+
+MEM_STATIC void ZSTD_assertValidSequence(
+        ZSTD_DCtx const* dctx,
+        BYTE const* op, BYTE const* oend,
+        seq_t const seq,
+        BYTE const* prefixStart, BYTE const* virtualStart)
+{
+    size_t const windowSize = dctx->fParams.windowSize;
+    size_t const sequenceSize = seq.litLength + seq.matchLength;
+    BYTE const* const oLitEnd = op + seq.litLength;
+    DEBUGLOG(6, "Checking sequence: litL=%u matchL=%u offset=%u",
+            (U32)seq.litLength, (U32)seq.matchLength, (U32)seq.offset);
+    assert(op <= oend);
+    assert((size_t)(oend - op) >= sequenceSize);
+    assert(sequenceSize <= ZSTD_BLOCKSIZE_MAX);
+    if (ZSTD_dictionaryIsActive(dctx, prefixStart, oLitEnd)) {
+        size_t const dictSize = (size_t)((char const*)dctx->dictContentEndForFuzzing - (char const*)dctx->dictContentBeginForFuzzing);
+        /* Offset must be within the dictionary. */
+        assert(seq.offset <= (size_t)(oLitEnd - virtualStart));
+        assert(seq.offset <= windowSize + dictSize);
+    } else {
+        /* Offset must be within our window. */
+        assert(seq.offset <= windowSize);
+    }
+}
+#endif
+
+#ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG
 FORCE_INLINE_TEMPLATE size_t
 DONT_VECTORIZE
 ZSTD_decompressSequences_body( ZSTD_DCtx* dctx,
                                void* dst, size_t maxDstSize,
                          const void* seqStart, size_t seqSize, int nbSeq,
-                         const ZSTD_longOffset_e isLongOffset)
+                         const ZSTD_longOffset_e isLongOffset,
+                         const int frame)
 {
     const BYTE* ip = (const BYTE*)seqStart;
     const BYTE* const iend = ip + seqSize;
@@ -910,46 +1009,104 @@ ZSTD_decompressSequences_body( ZSTD_DCtx* dctx,
     const BYTE* const vBase = (const BYTE*) (dctx->virtualStart);
     const BYTE* const dictEnd = (const BYTE*) (dctx->dictEnd);
     DEBUGLOG(5, "ZSTD_decompressSequences_body");
+    (void)frame;
 
     /* Regen sequences */
     if (nbSeq) {
         seqState_t seqState;
+        size_t error = 0;
         dctx->fseEntropy = 1;
         { U32 i; for (i=0; i<ZSTD_REP_NUM; i++) seqState.prevOffset[i] = dctx->entropy.rep[i]; }
         RETURN_ERROR_IF(
             ERR_isError(BIT_initDStream(&seqState.DStream, ip, iend-ip)),
-            corruption_detected);
+            corruption_detected, "");
         ZSTD_initFseState(&seqState.stateLL, &seqState.DStream, dctx->LLTptr);
         ZSTD_initFseState(&seqState.stateOffb, &seqState.DStream, dctx->OFTptr);
         ZSTD_initFseState(&seqState.stateML, &seqState.DStream, dctx->MLTptr);
+        assert(dst != NULL);
 
         ZSTD_STATIC_ASSERT(
                 BIT_DStream_unfinished < BIT_DStream_completed &&
                 BIT_DStream_endOfBuffer < BIT_DStream_completed &&
                 BIT_DStream_completed < BIT_DStream_overflow);
 
-        for ( ; (BIT_reloadDStream(&(seqState.DStream)) <= BIT_DStream_completed) && nbSeq ; ) {
-            nbSeq--;
-            {   seq_t const sequence = ZSTD_decodeSequence(&seqState, isLongOffset);
-                size_t const oneSeqSize = ZSTD_execSequence(op, oend, sequence, &litPtr, litEnd, prefixStart, vBase, dictEnd);
-                DEBUGLOG(6, "regenerated sequence size : %u", (U32)oneSeqSize);
-                if (ZSTD_isError(oneSeqSize)) return oneSeqSize;
-                op += oneSeqSize;
-        }   }
+#if defined(__GNUC__) && defined(__x86_64__)
+        /* Align the decompression loop to 32 + 16 bytes.
+         *
+         * zstd compiled with gcc-9 on an Intel i9-9900k shows 10% decompression
+         * speed swings based on the alignment of the decompression loop. This
+         * performance swing is caused by parts of the decompression loop falling
+         * out of the DSB. The entire decompression loop should fit in the DSB,
+         * when it can't we get much worse performance. You can measure if you've
+         * hit the good case or the bad case with this perf command for some
+         * compressed file test.zst:
+         *
+         *   perf stat -e cycles -e instructions -e idq.all_dsb_cycles_any_uops \
+         *             -e idq.all_mite_cycles_any_uops -- ./zstd -tq test.zst
+         *
+         * If you see most cycles served out of the MITE you've hit the bad case.
+         * If you see most cycles served out of the DSB you've hit the good case.
+         * If it is pretty even then you may be in an okay case.
+         *
+         * I've been able to reproduce this issue on the following CPUs:
+         *   - Kabylake: Macbook Pro (15-inch, 2019) 2.4 GHz Intel Core i9
+         *               Use Instruments->Counters to get DSB/MITE cycles.
+         *               I never got performance swings, but I was able to
+         *               go from the good case of mostly DSB to half of the
+         *               cycles served from MITE.
+         *   - Coffeelake: Intel i9-9900k
+         *
+         * I haven't been able to reproduce the instability or DSB misses on any
+         * of the following CPUS:
+         *   - Haswell
+         *   - Broadwell: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GH
+         *   - Skylake
+         *
+         * If you are seeing performance stability this script can help test.
+         * It tests on 4 commits in zstd where I saw performance change.
+         *
+         *   https://gist.github.com/terrelln/9889fc06a423fd5ca6e99351564473f4
+         */
+        __asm__(".p2align 5");
+        __asm__("nop");
+        __asm__(".p2align 4");
+#endif
+        for ( ; ; ) {
+            seq_t const sequence = ZSTD_decodeSequence(&seqState, isLongOffset, ZSTD_p_noPrefetch);
+            size_t const oneSeqSize = ZSTD_execSequence(op, oend, sequence, &litPtr, litEnd, prefixStart, vBase, dictEnd);
+#if defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) && defined(FUZZING_ASSERT_VALID_SEQUENCE)
+            assert(!ZSTD_isError(oneSeqSize));
+            if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequence, prefixStart, vBase);
+#endif
+            DEBUGLOG(6, "regenerated sequence size : %u", (U32)oneSeqSize);
+            BIT_reloadDStream(&(seqState.DStream));
+            /* gcc and clang both don't like early returns in this loop.
+             * gcc doesn't like early breaks either.
+             * Instead save an error and report it at the end.
+             * When there is an error, don't increment op, so we don't
+             * overwrite.
+             */
+            if (UNLIKELY(ZSTD_isError(oneSeqSize))) error = oneSeqSize;
+            else op += oneSeqSize;
+            if (UNLIKELY(!--nbSeq)) break;
+        }
 
         /* check if reached exact end */
         DEBUGLOG(5, "ZSTD_decompressSequences_body: after decode loop, remaining nbSeq : %i", nbSeq);
-        RETURN_ERROR_IF(nbSeq, corruption_detected);
-        RETURN_ERROR_IF(BIT_reloadDStream(&seqState.DStream) < BIT_DStream_completed, corruption_detected);
+        if (ZSTD_isError(error)) return error;
+        RETURN_ERROR_IF(nbSeq, corruption_detected, "");
+        RETURN_ERROR_IF(BIT_reloadDStream(&seqState.DStream) < BIT_DStream_completed, corruption_detected, "");
         /* save reps for next block */
         { U32 i; for (i=0; i<ZSTD_REP_NUM; i++) dctx->entropy.rep[i] = (U32)(seqState.prevOffset[i]); }
     }
 
     /* last literal segment */
     {   size_t const lastLLSize = litEnd - litPtr;
-        RETURN_ERROR_IF(lastLLSize > (size_t)(oend-op), dstSize_tooSmall);
-        memcpy(op, litPtr, lastLLSize);
-        op += lastLLSize;
+        RETURN_ERROR_IF(lastLLSize > (size_t)(oend-op), dstSize_tooSmall, "");
+        if (op != NULL) {
+            memcpy(op, litPtr, lastLLSize);
+            op += lastLLSize;
+        }
     }
 
     return op-ostart;
@@ -959,99 +1116,21 @@ static size_t
 ZSTD_decompressSequences_default(ZSTD_DCtx* dctx,
                                  void* dst, size_t maxDstSize,
                            const void* seqStart, size_t seqSize, int nbSeq,
-                           const ZSTD_longOffset_e isLongOffset)
+                           const ZSTD_longOffset_e isLongOffset,
+                           const int frame)
 {
-    return ZSTD_decompressSequences_body(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset);
+    return ZSTD_decompressSequences_body(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
 }
 #endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG */
 
-
-
 #ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT
-FORCE_INLINE_TEMPLATE seq_t
-ZSTD_decodeSequenceLong(seqState_t* seqState, ZSTD_longOffset_e const longOffsets)
-{
-    seq_t seq;
-    U32 const llBits = seqState->stateLL.table[seqState->stateLL.state].nbAdditionalBits;
-    U32 const mlBits = seqState->stateML.table[seqState->stateML.state].nbAdditionalBits;
-    U32 const ofBits = seqState->stateOffb.table[seqState->stateOffb.state].nbAdditionalBits;
-    U32 const totalBits = llBits+mlBits+ofBits;
-    U32 const llBase = seqState->stateLL.table[seqState->stateLL.state].baseValue;
-    U32 const mlBase = seqState->stateML.table[seqState->stateML.state].baseValue;
-    U32 const ofBase = seqState->stateOffb.table[seqState->stateOffb.state].baseValue;
-
-    /* sequence */
-    {   size_t offset;
-        if (!ofBits)
-            offset = 0;
-        else {
-            ZSTD_STATIC_ASSERT(ZSTD_lo_isLongOffset == 1);
-            ZSTD_STATIC_ASSERT(LONG_OFFSETS_MAX_EXTRA_BITS_32 == 5);
-            assert(ofBits <= MaxOff);
-            if (MEM_32bits() && longOffsets) {
-                U32 const extraBits = ofBits - MIN(ofBits, STREAM_ACCUMULATOR_MIN_32-1);
-                offset = ofBase + (BIT_readBitsFast(&seqState->DStream, ofBits - extraBits) << extraBits);
-                if (MEM_32bits() || extraBits) BIT_reloadDStream(&seqState->DStream);
-                if (extraBits) offset += BIT_readBitsFast(&seqState->DStream, extraBits);
-            } else {
-                offset = ofBase + BIT_readBitsFast(&seqState->DStream, ofBits);   /* <=  (ZSTD_WINDOWLOG_MAX-1) bits */
-                if (MEM_32bits()) BIT_reloadDStream(&seqState->DStream);
-            }
-        }
-
-        if (ofBits <= 1) {
-            offset += (llBase==0);
-            if (offset) {
-                size_t temp = (offset==3) ? seqState->prevOffset[0] - 1 : seqState->prevOffset[offset];
-                temp += !temp;   /* 0 is not valid; input is corrupted; force offset to 1 */
-                if (offset != 1) seqState->prevOffset[2] = seqState->prevOffset[1];
-                seqState->prevOffset[1] = seqState->prevOffset[0];
-                seqState->prevOffset[0] = offset = temp;
-            } else {
-                offset = seqState->prevOffset[0];
-            }
-        } else {
-            seqState->prevOffset[2] = seqState->prevOffset[1];
-            seqState->prevOffset[1] = seqState->prevOffset[0];
-            seqState->prevOffset[0] = offset;
-        }
-        seq.offset = offset;
-    }
-
-    seq.matchLength = mlBase + ((mlBits>0) ? BIT_readBitsFast(&seqState->DStream, mlBits) : 0);  /* <=  16 bits */
-    if (MEM_32bits() && (mlBits+llBits >= STREAM_ACCUMULATOR_MIN_32-LONG_OFFSETS_MAX_EXTRA_BITS_32))
-        BIT_reloadDStream(&seqState->DStream);
-    if (MEM_64bits() && (totalBits >= STREAM_ACCUMULATOR_MIN_64-(LLFSELog+MLFSELog+OffFSELog)))
-        BIT_reloadDStream(&seqState->DStream);
-    /* Verify that there is enough bits to read the rest of the data in 64-bit mode. */
-    ZSTD_STATIC_ASSERT(16+LLFSELog+MLFSELog+OffFSELog < STREAM_ACCUMULATOR_MIN_64);
-
-    seq.litLength = llBase + ((llBits>0) ? BIT_readBitsFast(&seqState->DStream, llBits) : 0);    /* <=  16 bits */
-    if (MEM_32bits())
-        BIT_reloadDStream(&seqState->DStream);
-
-    {   size_t const pos = seqState->pos + seq.litLength;
-        const BYTE* const matchBase = (seq.offset > pos) ? seqState->dictEnd : seqState->prefixStart;
-        seq.match = matchBase + pos - seq.offset;  /* note : this operation can overflow when seq.offset is really too large, which can only happen when input is corrupted.
-                                                    * No consequence though : no memory access will occur, overly large offset will be detected in ZSTD_execSequenceLong() */
-        seqState->pos = pos + seq.matchLength;
-    }
-
-    /* ANS state update */
-    ZSTD_updateFseState(&seqState->stateLL, &seqState->DStream);    /* <=  9 bits */
-    ZSTD_updateFseState(&seqState->stateML, &seqState->DStream);    /* <=  9 bits */
-    if (MEM_32bits()) BIT_reloadDStream(&seqState->DStream);    /* <= 18 bits */
-    ZSTD_updateFseState(&seqState->stateOffb, &seqState->DStream);  /* <=  8 bits */
-
-    return seq;
-}
-
 FORCE_INLINE_TEMPLATE size_t
 ZSTD_decompressSequencesLong_body(
                                ZSTD_DCtx* dctx,
                                void* dst, size_t maxDstSize,
                          const void* seqStart, size_t seqSize, int nbSeq,
-                         const ZSTD_longOffset_e isLongOffset)
+                         const ZSTD_longOffset_e isLongOffset,
+                         const int frame)
 {
     const BYTE* ip = (const BYTE*)seqStart;
     const BYTE* const iend = ip + seqSize;
@@ -1063,6 +1142,7 @@ ZSTD_decompressSequencesLong_body(
     const BYTE* const prefixStart = (const BYTE*) (dctx->prefixStart);
     const BYTE* const dictStart = (const BYTE*) (dctx->virtualStart);
     const BYTE* const dictEnd = (const BYTE*) (dctx->dictEnd);
+    (void)frame;
 
     /* Regen sequences */
     if (nbSeq) {
@@ -1078,36 +1158,45 @@ ZSTD_decompressSequencesLong_body(
         seqState.prefixStart = prefixStart;
         seqState.pos = (size_t)(op-prefixStart);
         seqState.dictEnd = dictEnd;
+        assert(dst != NULL);
         assert(iend >= ip);
         RETURN_ERROR_IF(
             ERR_isError(BIT_initDStream(&seqState.DStream, ip, iend-ip)),
-            corruption_detected);
+            corruption_detected, "");
         ZSTD_initFseState(&seqState.stateLL, &seqState.DStream, dctx->LLTptr);
         ZSTD_initFseState(&seqState.stateOffb, &seqState.DStream, dctx->OFTptr);
         ZSTD_initFseState(&seqState.stateML, &seqState.DStream, dctx->MLTptr);
 
         /* prepare in advance */
         for (seqNb=0; (BIT_reloadDStream(&seqState.DStream) <= BIT_DStream_completed) && (seqNb<seqAdvance); seqNb++) {
-            sequences[seqNb] = ZSTD_decodeSequenceLong(&seqState, isLongOffset);
+            sequences[seqNb] = ZSTD_decodeSequence(&seqState, isLongOffset, ZSTD_p_prefetch);
             PREFETCH_L1(sequences[seqNb].match); PREFETCH_L1(sequences[seqNb].match + sequences[seqNb].matchLength - 1); /* note : it's safe to invoke PREFETCH() on any memory address, including invalid ones */
         }
-        RETURN_ERROR_IF(seqNb<seqAdvance, corruption_detected);
+        RETURN_ERROR_IF(seqNb<seqAdvance, corruption_detected, "");
 
         /* decode and decompress */
         for ( ; (BIT_reloadDStream(&(seqState.DStream)) <= BIT_DStream_completed) && (seqNb<nbSeq) ; seqNb++) {
-            seq_t const sequence = ZSTD_decodeSequenceLong(&seqState, isLongOffset);
+            seq_t const sequence = ZSTD_decodeSequence(&seqState, isLongOffset, ZSTD_p_prefetch);
             size_t const oneSeqSize = ZSTD_execSequence(op, oend, sequences[(seqNb-ADVANCED_SEQS) & STORED_SEQS_MASK], &litPtr, litEnd, prefixStart, dictStart, dictEnd);
+#if defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) && defined(FUZZING_ASSERT_VALID_SEQUENCE)
+            assert(!ZSTD_isError(oneSeqSize));
+            if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequences[(seqNb-ADVANCED_SEQS) & STORED_SEQS_MASK], prefixStart, dictStart);
+#endif
             if (ZSTD_isError(oneSeqSize)) return oneSeqSize;
             PREFETCH_L1(sequence.match); PREFETCH_L1(sequence.match + sequence.matchLength - 1); /* note : it's safe to invoke PREFETCH() on any memory address, including invalid ones */
             sequences[seqNb & STORED_SEQS_MASK] = sequence;
             op += oneSeqSize;
         }
-        RETURN_ERROR_IF(seqNb<nbSeq, corruption_detected);
+        RETURN_ERROR_IF(seqNb<nbSeq, corruption_detected, "");
 
         /* finish queue */
         seqNb -= seqAdvance;
         for ( ; seqNb<nbSeq ; seqNb++) {
             size_t const oneSeqSize = ZSTD_execSequence(op, oend, sequences[seqNb&STORED_SEQS_MASK], &litPtr, litEnd, prefixStart, dictStart, dictEnd);
+#if defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) && defined(FUZZING_ASSERT_VALID_SEQUENCE)
+            assert(!ZSTD_isError(oneSeqSize));
+            if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequences[seqNb&STORED_SEQS_MASK], prefixStart, dictStart);
+#endif
             if (ZSTD_isError(oneSeqSize)) return oneSeqSize;
             op += oneSeqSize;
         }
@@ -1118,9 +1207,11 @@ ZSTD_decompressSequencesLong_body(
 
     /* last literal segment */
     {   size_t const lastLLSize = litEnd - litPtr;
-        RETURN_ERROR_IF(lastLLSize > (size_t)(oend-op), dstSize_tooSmall);
-        memcpy(op, litPtr, lastLLSize);
-        op += lastLLSize;
+        RETURN_ERROR_IF(lastLLSize > (size_t)(oend-op), dstSize_tooSmall, "");
+        if (op != NULL) {
+            memcpy(op, litPtr, lastLLSize);
+            op += lastLLSize;
+        }
     }
 
     return op-ostart;
@@ -1130,9 +1221,10 @@ static size_t
 ZSTD_decompressSequencesLong_default(ZSTD_DCtx* dctx,
                                  void* dst, size_t maxDstSize,
                            const void* seqStart, size_t seqSize, int nbSeq,
-                           const ZSTD_longOffset_e isLongOffset)
+                           const ZSTD_longOffset_e isLongOffset,
+                           const int frame)
 {
-    return ZSTD_decompressSequencesLong_body(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset);
+    return ZSTD_decompressSequencesLong_body(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
 }
 #endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT */
 
@@ -1146,9 +1238,10 @@ DONT_VECTORIZE
 ZSTD_decompressSequences_bmi2(ZSTD_DCtx* dctx,
                                  void* dst, size_t maxDstSize,
                            const void* seqStart, size_t seqSize, int nbSeq,
-                           const ZSTD_longOffset_e isLongOffset)
+                           const ZSTD_longOffset_e isLongOffset,
+                           const int frame)
 {
-    return ZSTD_decompressSequences_body(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset);
+    return ZSTD_decompressSequences_body(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
 }
 #endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG */
 
@@ -1157,9 +1250,10 @@ static TARGET_ATTRIBUTE("bmi2") size_t
 ZSTD_decompressSequencesLong_bmi2(ZSTD_DCtx* dctx,
                                  void* dst, size_t maxDstSize,
                            const void* seqStart, size_t seqSize, int nbSeq,
-                           const ZSTD_longOffset_e isLongOffset)
+                           const ZSTD_longOffset_e isLongOffset,
+                           const int frame)
 {
-    return ZSTD_decompressSequencesLong_body(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset);
+    return ZSTD_decompressSequencesLong_body(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
 }
 #endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT */
 
@@ -1169,21 +1263,23 @@ typedef size_t (*ZSTD_decompressSequences_t)(
                             ZSTD_DCtx* dctx,
                             void* dst, size_t maxDstSize,
                             const void* seqStart, size_t seqSize, int nbSeq,
-                            const ZSTD_longOffset_e isLongOffset);
+                            const ZSTD_longOffset_e isLongOffset,
+                            const int frame);
 
 #ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG
 static size_t
 ZSTD_decompressSequences(ZSTD_DCtx* dctx, void* dst, size_t maxDstSize,
                    const void* seqStart, size_t seqSize, int nbSeq,
-                   const ZSTD_longOffset_e isLongOffset)
+                   const ZSTD_longOffset_e isLongOffset,
+                   const int frame)
 {
     DEBUGLOG(5, "ZSTD_decompressSequences");
 #if DYNAMIC_BMI2
     if (dctx->bmi2) {
-        return ZSTD_decompressSequences_bmi2(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset);
+        return ZSTD_decompressSequences_bmi2(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
     }
 #endif
-  return ZSTD_decompressSequences_default(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset);
+  return ZSTD_decompressSequences_default(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
 }
 #endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG */
 
@@ -1198,15 +1294,16 @@ static size_t
 ZSTD_decompressSequencesLong(ZSTD_DCtx* dctx,
                              void* dst, size_t maxDstSize,
                              const void* seqStart, size_t seqSize, int nbSeq,
-                             const ZSTD_longOffset_e isLongOffset)
+                             const ZSTD_longOffset_e isLongOffset,
+                             const int frame)
 {
     DEBUGLOG(5, "ZSTD_decompressSequencesLong");
 #if DYNAMIC_BMI2
     if (dctx->bmi2) {
-        return ZSTD_decompressSequencesLong_bmi2(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset);
+        return ZSTD_decompressSequencesLong_bmi2(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
     }
 #endif
-  return ZSTD_decompressSequencesLong_default(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset);
+  return ZSTD_decompressSequencesLong_default(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
 }
 #endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT */
 
@@ -1240,7 +1337,6 @@ ZSTD_getLongOffsetsShare(const ZSTD_seqSymbol* offTable)
 }
 #endif
 
-
 size_t
 ZSTD_decompressBlock_internal(ZSTD_DCtx* dctx,
                               void* dst, size_t dstCapacity,
@@ -1256,7 +1352,7 @@ ZSTD_decompressBlock_internal(ZSTD_DCtx* dctx,
     ZSTD_longOffset_e const isLongOffset = (ZSTD_longOffset_e)(MEM_32bits() && (!frame || (dctx->fParams.windowSize > (1ULL << STREAM_ACCUMULATOR_MIN))));
     DEBUGLOG(5, "ZSTD_decompressBlock_internal (size : %u)", (U32)srcSize);
 
-    RETURN_ERROR_IF(srcSize >= ZSTD_BLOCKSIZE_MAX, srcSize_wrong);
+    RETURN_ERROR_IF(srcSize >= ZSTD_BLOCKSIZE_MAX, srcSize_wrong, "");
 
     /* Decode literals section */
     {   size_t const litCSize = ZSTD_decodeLiteralsBlock(dctx, src, srcSize);
@@ -1282,6 +1378,8 @@ ZSTD_decompressBlock_internal(ZSTD_DCtx* dctx,
         ip += seqHSize;
         srcSize -= seqHSize;
 
+        RETURN_ERROR_IF(dst == NULL && nbSeq > 0, dstSize_tooSmall, "NULL not handled");
+
 #if !defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT) && \
     !defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG)
         if ( !usePrefetchDecoder
@@ -1300,17 +1398,28 @@ ZSTD_decompressBlock_internal(ZSTD_DCtx* dctx,
         if (usePrefetchDecoder)
 #endif
 #ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT
-            return ZSTD_decompressSequencesLong(dctx, dst, dstCapacity, ip, srcSize, nbSeq, isLongOffset);
+            return ZSTD_decompressSequencesLong(dctx, dst, dstCapacity, ip, srcSize, nbSeq, isLongOffset, frame);
 #endif
 
 #ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG
         /* else */
-        return ZSTD_decompressSequences(dctx, dst, dstCapacity, ip, srcSize, nbSeq, isLongOffset);
+        return ZSTD_decompressSequences(dctx, dst, dstCapacity, ip, srcSize, nbSeq, isLongOffset, frame);
 #endif
     }
 }
 
 
+void ZSTD_checkContinuity(ZSTD_DCtx* dctx, const void* dst)
+{
+    if (dst != dctx->previousDstEnd) {   /* not contiguous */
+        dctx->dictEnd = dctx->previousDstEnd;
+        dctx->virtualStart = (const char*)dst - ((const char*)(dctx->previousDstEnd) - (const char*)(dctx->prefixStart));
+        dctx->prefixStart = dst;
+        dctx->previousDstEnd = dst;
+    }
+}
+
+
 size_t ZSTD_decompressBlock(ZSTD_DCtx* dctx,
                             void* dst, size_t dstCapacity,
                       const void* src, size_t srcSize)
diff --git a/lib/decompress/zstd_decompress_block.h b/lib/decompress/zstd_decompress_block.h
index 7e9296041023..bf39b7350cf9 100644
--- a/lib/decompress/zstd_decompress_block.h
+++ b/lib/decompress/zstd_decompress_block.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -16,8 +16,8 @@
  *  Dependencies
  *********************************************************/
 #include <stddef.h>   /* size_t */
-#include "zstd.h"    /* DCtx, and some public functions */
-#include "zstd_internal.h"  /* blockProperties_t, and some public functions */
+#include "../zstd.h"    /* DCtx, and some public functions */
+#include "../common/zstd_internal.h"  /* blockProperties_t, and some public functions */
 #include "zstd_decompress_internal.h"  /* ZSTD_seqSymbol */
 
 
diff --git a/lib/decompress/zstd_decompress_internal.h b/lib/decompress/zstd_decompress_internal.h
index ccbdfa090faa..9ad96c554885 100644
--- a/lib/decompress/zstd_decompress_internal.h
+++ b/lib/decompress/zstd_decompress_internal.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -19,8 +19,8 @@
 /*-*******************************************************
  *  Dependencies
  *********************************************************/
-#include "mem.h"             /* BYTE, U16, U32 */
-#include "zstd_internal.h"   /* ZSTD_seqSymbol */
+#include "../common/mem.h"             /* BYTE, U16, U32 */
+#include "../common/zstd_internal.h"   /* ZSTD_seqSymbol */
 
 
 
@@ -95,6 +95,11 @@ typedef enum {
     ZSTD_use_once = 1            /* Use the dictionary once and set to ZSTD_dont_use */
 } ZSTD_dictUses_e;
 
+typedef enum {
+    ZSTD_obm_buffered = 0,  /* Buffer the output */
+    ZSTD_obm_stable = 1     /* ZSTD_outBuffer is stable */
+} ZSTD_outBufferMode_e;
+
 struct ZSTD_DCtx_s
 {
     const ZSTD_seqSymbol* LLTptr;
@@ -147,10 +152,19 @@ struct ZSTD_DCtx_s
     U32 legacyVersion;
     U32 hostageByte;
     int noForwardProgress;
+    ZSTD_outBufferMode_e outBufferMode;
+    ZSTD_outBuffer expectedOutBuffer;
 
     /* workspace */
     BYTE litBuffer[ZSTD_BLOCKSIZE_MAX + WILDCOPY_OVERLENGTH];
     BYTE headerBuffer[ZSTD_FRAMEHEADERSIZE_MAX];
+
+    size_t oversizedDuration;
+
+#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
+    void const* dictContentBeginForFuzzing;
+    void const* dictContentEndForFuzzing;
+#endif
 };  /* typedef'd to ZSTD_DCtx within "zstd.h" */
 
 
@@ -160,7 +174,7 @@ struct ZSTD_DCtx_s
 
 /*! ZSTD_loadDEntropy() :
  *  dict : must point at beginning of a valid zstd dictionary.
- * @return : size of entropy tables read */
+ * @return : size of dictionary header (size of magic number + dict ID + entropy tables) */
 size_t ZSTD_loadDEntropy(ZSTD_entropyDTables_t* entropy,
                    const void* const dict, size_t const dictSize);
 
diff --git a/lib/deprecated/zbuff.h b/lib/deprecated/zbuff.h
index 04183eab98f6..03cb14a039de 100644
--- a/lib/deprecated/zbuff.h
+++ b/lib/deprecated/zbuff.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -28,7 +28,7 @@ extern "C" {
 *  Dependencies
 ***************************************/
 #include <stddef.h>      /* size_t */
-#include "zstd.h"        /* ZSTD_CStream, ZSTD_DStream, ZSTDLIB_API */
+#include "../zstd.h"        /* ZSTD_CStream, ZSTD_DStream, ZSTDLIB_API */
 
 
 /* ***************************************************************
@@ -186,7 +186,7 @@ ZBUFF_DEPRECATED("use ZSTD_DStreamOutSize") size_t ZBUFF_recommendedDOutSize(voi
 
 /*--- Dependency ---*/
 #define ZSTD_STATIC_LINKING_ONLY   /* ZSTD_parameters, ZSTD_customMem */
-#include "zstd.h"
+#include "../zstd.h"
 
 
 /*--- Custom memory allocator ---*/
diff --git a/lib/deprecated/zbuff_common.c b/lib/deprecated/zbuff_common.c
index 661b9b0e18c5..579bc4df14a2 100644
--- a/lib/deprecated/zbuff_common.c
+++ b/lib/deprecated/zbuff_common.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -11,7 +11,7 @@
 /*-*************************************
 *  Dependencies
 ***************************************/
-#include "error_private.h"
+#include "../common/error_private.h"
 #include "zbuff.h"
 
 /*-****************************************
diff --git a/lib/deprecated/zbuff_compress.c b/lib/deprecated/zbuff_compress.c
index f39c60d89f60..2d20b1377591 100644
--- a/lib/deprecated/zbuff_compress.c
+++ b/lib/deprecated/zbuff_compress.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/lib/deprecated/zbuff_decompress.c b/lib/deprecated/zbuff_decompress.c
index 923c22b73c57..d3c49e84b815 100644
--- a/lib/deprecated/zbuff_decompress.c
+++ b/lib/deprecated/zbuff_decompress.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/lib/dictBuilder/cover.c b/lib/dictBuilder/cover.c
index 2e129dd9183a..da54ef15f2b8 100644
--- a/lib/dictBuilder/cover.c
+++ b/lib/dictBuilder/cover.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -26,11 +26,11 @@
 #include <string.h> /* memset */
 #include <time.h>   /* clock */
 
-#include "mem.h" /* read */
-#include "pool.h"
-#include "threading.h"
+#include "../common/mem.h" /* read */
+#include "../common/pool.h"
+#include "../common/threading.h"
 #include "cover.h"
-#include "zstd_internal.h" /* includes zstd.h */
+#include "../common/zstd_internal.h" /* includes zstd.h */
 #ifndef ZDICT_STATIC_LINKING_ONLY
 #define ZDICT_STATIC_LINKING_ONLY
 #endif
diff --git a/lib/dictBuilder/cover.h b/lib/dictBuilder/cover.h
index d9e0636a6598..f2aa0e35e3aa 100644
--- a/lib/dictBuilder/cover.h
+++ b/lib/dictBuilder/cover.h
@@ -1,11 +1,21 @@
+/*
+ * Copyright (c) 2017-2020, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
 #include <stdio.h>  /* fprintf */
 #include <stdlib.h> /* malloc, free, qsort */
 #include <string.h> /* memset */
 #include <time.h>   /* clock */
-#include "mem.h" /* read */
-#include "pool.h"
-#include "threading.h"
-#include "zstd_internal.h" /* includes zstd.h */
+#include "../common/mem.h" /* read */
+#include "../common/pool.h"
+#include "../common/threading.h"
+#include "../common/zstd_internal.h" /* includes zstd.h */
 #ifndef ZDICT_STATIC_LINKING_ONLY
 #define ZDICT_STATIC_LINKING_ONLY
 #endif
diff --git a/lib/dictBuilder/fastcover.c b/lib/dictBuilder/fastcover.c
index 941bb5a26ae6..485c333b54a8 100644
--- a/lib/dictBuilder/fastcover.c
+++ b/lib/dictBuilder/fastcover.c
@@ -1,3 +1,13 @@
+/*
+ * Copyright (c) 2018-2020, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
 /*-*************************************
 *  Dependencies
 ***************************************/
@@ -6,11 +16,11 @@
 #include <string.h> /* memset */
 #include <time.h>   /* clock */
 
-#include "mem.h" /* read */
-#include "pool.h"
-#include "threading.h"
+#include "../common/mem.h" /* read */
+#include "../common/pool.h"
+#include "../common/threading.h"
 #include "cover.h"
-#include "zstd_internal.h" /* includes zstd.h */
+#include "../common/zstd_internal.h" /* includes zstd.h */
 #ifndef ZDICT_STATIC_LINKING_ONLY
 #define ZDICT_STATIC_LINKING_ONLY
 #endif
diff --git a/lib/dictBuilder/zdict.c b/lib/dictBuilder/zdict.c
index 4a263d822d85..6d0b04231bb5 100644
--- a/lib/dictBuilder/zdict.c
+++ b/lib/dictBuilder/zdict.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -37,17 +37,18 @@
 #include <stdio.h>         /* fprintf, fopen, ftello64 */
 #include <time.h>          /* clock */
 
-#include "mem.h"           /* read */
-#include "fse.h"           /* FSE_normalizeCount, FSE_writeNCount */
+#include "../common/mem.h"           /* read */
+#include "../common/fse.h"           /* FSE_normalizeCount, FSE_writeNCount */
 #define HUF_STATIC_LINKING_ONLY
-#include "huf.h"           /* HUF_buildCTable, HUF_writeCTable */
-#include "zstd_internal.h" /* includes zstd.h */
-#include "xxhash.h"        /* XXH64 */
+#include "../common/huf.h"           /* HUF_buildCTable, HUF_writeCTable */
+#include "../common/zstd_internal.h" /* includes zstd.h */
+#include "../common/xxhash.h"        /* XXH64 */
 #include "divsufsort.h"
 #ifndef ZDICT_STATIC_LINKING_ONLY
 #  define ZDICT_STATIC_LINKING_ONLY
 #endif
 #include "zdict.h"
+#include "../compress/zstd_compress_internal.h" /* ZSTD_loadCEntropy() */
 
 
 /*-*************************************
@@ -99,6 +100,29 @@ unsigned ZDICT_getDictID(const void* dictBuffer, size_t dictSize)
     return MEM_readLE32((const char*)dictBuffer + 4);
 }
 
+size_t ZDICT_getDictHeaderSize(const void* dictBuffer, size_t dictSize)
+{
+    size_t headerSize;
+    if (dictSize <= 8 || MEM_readLE32(dictBuffer) != ZSTD_MAGIC_DICTIONARY) return ERROR(dictionary_corrupted);
+
+    {   unsigned offcodeMaxValue = MaxOff;
+        ZSTD_compressedBlockState_t* bs = (ZSTD_compressedBlockState_t*)malloc(sizeof(ZSTD_compressedBlockState_t));
+        U32* wksp = (U32*)malloc(HUF_WORKSPACE_SIZE);
+        short* offcodeNCount = (short*)malloc((MaxOff+1)*sizeof(short));
+        if (!bs || !wksp || !offcodeNCount) {
+            headerSize = ERROR(memory_allocation);
+        } else {
+            ZSTD_reset_compressedBlockState(bs);
+            headerSize = ZSTD_loadCEntropy(bs, wksp, offcodeNCount, &offcodeMaxValue, dictBuffer, dictSize);
+        }
+
+        free(bs);
+        free(wksp);
+        free(offcodeNCount);
+    }
+
+    return headerSize;
+}
 
 /*-********************************************************
 *  Dictionary training functions
@@ -588,12 +612,12 @@ typedef struct
 
 #define MAXREPOFFSET 1024
 
-static void ZDICT_countEStats(EStats_ress_t esr, ZSTD_parameters params,
+static void ZDICT_countEStats(EStats_ress_t esr, const ZSTD_parameters* params,
                               unsigned* countLit, unsigned* offsetcodeCount, unsigned* matchlengthCount, unsigned* litlengthCount, U32* repOffsets,
                               const void* src, size_t srcSize,
                               U32 notificationLevel)
 {
-    size_t const blockSizeMax = MIN (ZSTD_BLOCKSIZE_MAX, 1 << params.cParams.windowLog);
+    size_t const blockSizeMax = MIN (ZSTD_BLOCKSIZE_MAX, 1 << params->cParams.windowLog);
     size_t cSize;
 
     if (srcSize > blockSizeMax) srcSize = blockSizeMax;   /* protection vs large samples */
@@ -731,7 +755,7 @@ static size_t ZDICT_analyzeEntropy(void*  dstBuffer, size_t maxDstSize,
 
     /* collect stats on all samples */
     for (u=0; u<nbFiles; u++) {
-        ZDICT_countEStats(esr, params,
+        ZDICT_countEStats(esr, &params,
                           countLit, offcodeCount, matchLengthCount, litLengthCount, repOffset,
                          (const char*)srcBuffer + pos, fileSizes[u],
                           notificationLevel);
diff --git a/lib/dictBuilder/zdict.h b/lib/dictBuilder/zdict.h
index 37978ecdfb81..ff2e77faf773 100644
--- a/lib/dictBuilder/zdict.h
+++ b/lib/dictBuilder/zdict.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -61,9 +61,57 @@ ZDICTLIB_API size_t ZDICT_trainFromBuffer(void* dictBuffer, size_t dictBufferCap
                                     const void* samplesBuffer,
                                     const size_t* samplesSizes, unsigned nbSamples);
 
+typedef struct {
+    int      compressionLevel;   /*< optimize for a specific zstd compression level; 0 means default */
+    unsigned notificationLevel;  /*< Write log to stderr; 0 = none (default); 1 = errors; 2 = progression; 3 = details; 4 = debug; */
+    unsigned dictID;             /*< force dictID value; 0 means auto mode (32-bits random value) */
+} ZDICT_params_t;
+
+/*! ZDICT_finalizeDictionary():
+ * Given a custom content as a basis for dictionary, and a set of samples,
+ * finalize dictionary by adding headers and statistics according to the zstd
+ * dictionary format.
+ *
+ * Samples must be stored concatenated in a flat buffer `samplesBuffer`,
+ * supplied with an array of sizes `samplesSizes`, providing the size of each
+ * sample in order. The samples are used to construct the statistics, so they
+ * should be representative of what you will compress with this dictionary.
+ *
+ * The compression level can be set in `parameters`. You should pass the
+ * compression level you expect to use in production. The statistics for each
+ * compression level differ, so tuning the dictionary for the compression level
+ * can help quite a bit.
+ *
+ * You can set an explicit dictionary ID in `parameters`, or allow us to pick
+ * a random dictionary ID for you, but we can't guarantee no collisions.
+ *
+ * The dstDictBuffer and the dictContent may overlap, and the content will be
+ * appended to the end of the header. If the header + the content doesn't fit in
+ * maxDictSize the beginning of the content is truncated to make room, since it
+ * is presumed that the most profitable content is at the end of the dictionary,
+ * since that is the cheapest to reference.
+ *
+ * `dictContentSize` must be >= ZDICT_CONTENTSIZE_MIN bytes.
+ * `maxDictSize` must be >= max(dictContentSize, ZSTD_DICTSIZE_MIN).
+ *
+ * @return: size of dictionary stored into `dstDictBuffer` (<= `maxDictSize`),
+ *          or an error code, which can be tested by ZDICT_isError().
+ * Note: ZDICT_finalizeDictionary() will push notifications into stderr if
+ *       instructed to, using notificationLevel>0.
+ * NOTE: This function currently may fail in several edge cases including:
+ *         * Not enough samples
+ *         * Samples are uncompressible
+ *         * Samples are all exactly the same
+ */
+ZDICTLIB_API size_t ZDICT_finalizeDictionary(void* dstDictBuffer, size_t maxDictSize,
+                                const void* dictContent, size_t dictContentSize,
+                                const void* samplesBuffer, const size_t* samplesSizes, unsigned nbSamples,
+                                ZDICT_params_t parameters);
+
 
 /*======   Helper functions   ======*/
 ZDICTLIB_API unsigned ZDICT_getDictID(const void* dictBuffer, size_t dictSize);  /**< extracts dictID; @return zero if error (not a valid dictionary) */
+ZDICTLIB_API size_t ZDICT_getDictHeaderSize(const void* dictBuffer, size_t dictSize);  /* returns dict header size; returns a ZSTD error code on failure */
 ZDICTLIB_API unsigned ZDICT_isError(size_t errorCode);
 ZDICTLIB_API const char* ZDICT_getErrorName(size_t errorCode);
 
@@ -78,11 +126,8 @@ ZDICTLIB_API const char* ZDICT_getErrorName(size_t errorCode);
  * Use them only in association with static linking.
  * ==================================================================================== */
 
-typedef struct {
-    int      compressionLevel;   /* optimize for a specific zstd compression level; 0 means default */
-    unsigned notificationLevel;  /* Write log to stderr; 0 = none (default); 1 = errors; 2 = progression; 3 = details; 4 = debug; */
-    unsigned dictID;             /* force dictID value; 0 means auto mode (32-bits random value) */
-} ZDICT_params_t;
+#define ZDICT_CONTENTSIZE_MIN 128
+#define ZDICT_DICTSIZE_MIN    256
 
 /*! ZDICT_cover_params_t:
  *  k and d are the only required parameters.
@@ -198,28 +243,6 @@ ZDICTLIB_API size_t ZDICT_optimizeTrainFromBuffer_fastCover(void* dictBuffer,
                     const size_t* samplesSizes, unsigned nbSamples,
                     ZDICT_fastCover_params_t* parameters);
 
-/*! ZDICT_finalizeDictionary():
- * Given a custom content as a basis for dictionary, and a set of samples,
- * finalize dictionary by adding headers and statistics.
- *
- * Samples must be stored concatenated in a flat buffer `samplesBuffer`,
- * supplied with an array of sizes `samplesSizes`, providing the size of each sample in order.
- *
- * dictContentSize must be >= ZDICT_CONTENTSIZE_MIN bytes.
- * maxDictSize must be >= dictContentSize, and must be >= ZDICT_DICTSIZE_MIN bytes.
- *
- * @return: size of dictionary stored into `dictBuffer` (<= `dictBufferCapacity`),
- *          or an error code, which can be tested by ZDICT_isError().
- * Note: ZDICT_finalizeDictionary() will push notifications into stderr if instructed to, using notificationLevel>0.
- * Note 2: dictBuffer and dictContent can overlap
- */
-#define ZDICT_CONTENTSIZE_MIN 128
-#define ZDICT_DICTSIZE_MIN    256
-ZDICTLIB_API size_t ZDICT_finalizeDictionary(void* dictBuffer, size_t dictBufferCapacity,
-                                const void* dictContent, size_t dictContentSize,
-                                const void* samplesBuffer, const size_t* samplesSizes, unsigned nbSamples,
-                                ZDICT_params_t parameters);
-
 typedef struct {
     unsigned selectivityLevel;   /* 0 means default; larger => select more => larger dictionary */
     ZDICT_params_t zParams;
diff --git a/lib/legacy/zstd_legacy.h b/lib/legacy/zstd_legacy.h
index 0dbd3c7a40f5..6bea6a519aba 100644
--- a/lib/legacy/zstd_legacy.h
+++ b/lib/legacy/zstd_legacy.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -18,9 +18,9 @@ extern "C" {
 /* *************************************
 *  Includes
 ***************************************/
-#include "mem.h"            /* MEM_STATIC */
-#include "error_private.h"  /* ERROR */
-#include "zstd_internal.h"  /* ZSTD_inBuffer, ZSTD_outBuffer, ZSTD_frameSizeInfo */
+#include "../common/mem.h"            /* MEM_STATIC */
+#include "../common/error_private.h"  /* ERROR */
+#include "../common/zstd_internal.h"  /* ZSTD_inBuffer, ZSTD_outBuffer, ZSTD_frameSizeInfo */
 
 #if !defined (ZSTD_LEGACY_SUPPORT) || (ZSTD_LEGACY_SUPPORT == 0)
 #  undef ZSTD_LEGACY_SUPPORT
diff --git a/lib/legacy/zstd_v01.c b/lib/legacy/zstd_v01.c
index 8112527f058b..eb23628330e7 100644
--- a/lib/legacy/zstd_v01.c
+++ b/lib/legacy/zstd_v01.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -14,7 +14,7 @@
 ******************************************/
 #include <stddef.h>    /* size_t, ptrdiff_t */
 #include "zstd_v01.h"
-#include "error_private.h"
+#include "../common/error_private.h"
 
 
 /******************************************
@@ -257,7 +257,7 @@ static U64 FSE_read64(const void* memPtr)
     U64 val; memcpy(&val, memPtr, sizeof(val)); return val;
 }
 
-#endif // FSE_FORCE_MEMORY_ACCESS
+#endif /* FSE_FORCE_MEMORY_ACCESS */
 
 static U16 FSE_readLE16(const void* memPtr)
 {
@@ -1078,7 +1078,7 @@ static size_t HUF_decompress_usingDTable(   /* -3% slower when non static */
         BYTE* const ostart = (BYTE*) dst;
         BYTE* op = ostart;
         BYTE* const omax = op + maxDstSize;
-        BYTE* const olimit = omax-15;
+        BYTE* const olimit = maxDstSize < 15 ? op : omax-15;
 
         const void* ptr = DTable;
         const HUF_DElt* const dt = (const HUF_DElt*)(ptr)+1;
@@ -1092,7 +1092,7 @@ static size_t HUF_decompress_usingDTable(   /* -3% slower when non static */
         const size_t length1 = FSE_readLE16(jumpTable);
         const size_t length2 = FSE_readLE16(jumpTable+1);
         const size_t length3 = FSE_readLE16(jumpTable+2);
-        const size_t length4 = cSrcSize - 6 - length1 - length2 - length3;   // check coherency !!
+        const size_t length4 = cSrcSize - 6 - length1 - length2 - length3;   /* check coherency !! */
         const char* const start1 = (const char*)(cSrc) + 6;
         const char* const start2 = start1 + length1;
         const char* const start3 = start2 + length2;
@@ -1150,11 +1150,11 @@ static size_t HUF_decompress_usingDTable(   /* -3% slower when non static */
 
         /* tail */
         {
-            // bitTail = bitD1;   // *much* slower : -20% !??!
+            /* bitTail = bitD1; */   /* *much* slower : -20% !??! */
             FSE_DStream_t bitTail;
             bitTail.ptr = bitD1.ptr;
             bitTail.bitsConsumed = bitD1.bitsConsumed;
-            bitTail.bitContainer = bitD1.bitContainer;   // required in case of FSE_DStream_endOfBuffer
+            bitTail.bitContainer = bitD1.bitContainer;   /* required in case of FSE_DStream_endOfBuffer */
             bitTail.start = start1;
             for ( ; (FSE_reloadDStream(&bitTail) < FSE_DStream_completed) && (op<omax) ; op++)
             {
@@ -1483,7 +1483,9 @@ static size_t ZSTDv01_getcBlockSize(const void* src, size_t srcSize, blockProper
 static size_t ZSTD_copyUncompressedBlock(void* dst, size_t maxDstSize, const void* src, size_t srcSize)
 {
     if (srcSize > maxDstSize) return ERROR(dstSize_tooSmall);
-    memcpy(dst, src, srcSize);
+    if (srcSize > 0) {
+        memcpy(dst, src, srcSize);
+    }
     return srcSize;
 }
 
@@ -1502,7 +1504,7 @@ static size_t ZSTD_decompressLiterals(void* ctx,
     if (srcSize <= 3) return ERROR(corruption_detected);
 
     litSize = ip[1] + (ip[0]<<8);
-    litSize += ((ip[-3] >> 3) & 7) << 16;   // mmmmh....
+    litSize += ((ip[-3] >> 3) & 7) << 16;   /* mmmmh.... */
     op = oend - litSize;
 
     (void)ctx;
@@ -1541,7 +1543,9 @@ static size_t ZSTDv01_decodeLiteralsBlock(void* ctx,
             size_t rleSize = litbp.origSize;
             if (rleSize>maxDstSize) return ERROR(dstSize_tooSmall);
             if (!srcSize) return ERROR(srcSize_wrong);
-            memset(oend - rleSize, *ip, rleSize);
+            if (rleSize > 0) {
+                memset(oend - rleSize, *ip, rleSize);
+            }
             *litStart = oend - rleSize;
             *litSize = rleSize;
             ip++;
@@ -1901,8 +1905,10 @@ static size_t ZSTD_decompressSequences(
         {
             size_t lastLLSize = litEnd - litPtr;
             if (op+lastLLSize > oend) return ERROR(dstSize_tooSmall);
-            if (op != litPtr) memmove(op, litPtr, lastLLSize);
-            op += lastLLSize;
+            if (lastLLSize > 0) {
+                if (op != litPtr) memmove(op, litPtr, lastLLSize);
+                op += lastLLSize;
+            }
         }
     }
 
diff --git a/lib/legacy/zstd_v01.h b/lib/legacy/zstd_v01.h
index 245f9dd31468..7910351726c4 100644
--- a/lib/legacy/zstd_v01.h
+++ b/lib/legacy/zstd_v01.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/lib/legacy/zstd_v02.c b/lib/legacy/zstd_v02.c
index c8783799b9c8..32d45a6d4bd3 100644
--- a/lib/legacy/zstd_v02.c
+++ b/lib/legacy/zstd_v02.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -11,7 +11,7 @@
 
 #include <stddef.h>    /* size_t, ptrdiff_t */
 #include "zstd_v02.h"
-#include "error_private.h"
+#include "../common/error_private.h"
 
 
 /******************************************
@@ -189,7 +189,7 @@ MEM_STATIC void MEM_write16(void* memPtr, U16 value)
     memcpy(memPtr, &value, sizeof(value));
 }
 
-#endif // MEM_FORCE_MEMORY_ACCESS
+#endif /* MEM_FORCE_MEMORY_ACCESS */
 
 
 MEM_STATIC U16 MEM_readLE16(const void* memPtr)
@@ -2836,7 +2836,9 @@ static size_t ZSTD_getcBlockSize(const void* src, size_t srcSize, blockPropertie
 static size_t ZSTD_copyUncompressedBlock(void* dst, size_t maxDstSize, const void* src, size_t srcSize)
 {
     if (srcSize > maxDstSize) return ERROR(dstSize_tooSmall);
-    memcpy(dst, src, srcSize);
+    if (srcSize > 0) {
+        memcpy(dst, src, srcSize);
+    }
     return srcSize;
 }
 
@@ -3229,8 +3231,10 @@ static size_t ZSTD_decompressSequences(
             size_t lastLLSize = litEnd - litPtr;
             if (litPtr > litEnd) return ERROR(corruption_detected);
             if (op+lastLLSize > oend) return ERROR(dstSize_tooSmall);
-            if (op != litPtr) memmove(op, litPtr, lastLLSize);
-            op += lastLLSize;
+            if (lastLLSize > 0) {
+                if (op != litPtr) memmove(op, litPtr, lastLLSize);
+                op += lastLLSize;
+            }
         }
     }
 
diff --git a/lib/legacy/zstd_v02.h b/lib/legacy/zstd_v02.h
index 9d7d8d9b5bc4..5f8f6cd60cc5 100644
--- a/lib/legacy/zstd_v02.h
+++ b/lib/legacy/zstd_v02.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/lib/legacy/zstd_v03.c b/lib/legacy/zstd_v03.c
index 162bd63024d7..b541eae2abb7 100644
--- a/lib/legacy/zstd_v03.c
+++ b/lib/legacy/zstd_v03.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -11,7 +11,7 @@
 
 #include <stddef.h>    /* size_t, ptrdiff_t */
 #include "zstd_v03.h"
-#include "error_private.h"
+#include "../common/error_private.h"
 
 
 /******************************************
@@ -191,7 +191,7 @@ MEM_STATIC void MEM_write16(void* memPtr, U16 value)
 }
 
 
-#endif // MEM_FORCE_MEMORY_ACCESS
+#endif /* MEM_FORCE_MEMORY_ACCESS */
 
 
 MEM_STATIC U16 MEM_readLE16(const void* memPtr)
@@ -2477,7 +2477,9 @@ static size_t ZSTD_getcBlockSize(const void* src, size_t srcSize, blockPropertie
 static size_t ZSTD_copyUncompressedBlock(void* dst, size_t maxDstSize, const void* src, size_t srcSize)
 {
     if (srcSize > maxDstSize) return ERROR(dstSize_tooSmall);
-    memcpy(dst, src, srcSize);
+    if (srcSize > 0) {
+        memcpy(dst, src, srcSize);
+    }
     return srcSize;
 }
 
@@ -2870,8 +2872,10 @@ static size_t ZSTD_decompressSequences(
             size_t lastLLSize = litEnd - litPtr;
             if (litPtr > litEnd) return ERROR(corruption_detected);
             if (op+lastLLSize > oend) return ERROR(dstSize_tooSmall);
-            if (op != litPtr) memmove(op, litPtr, lastLLSize);
-            op += lastLLSize;
+            if (lastLLSize > 0) {
+                if (op != litPtr) memmove(op, litPtr, lastLLSize);
+                op += lastLLSize;
+            }
         }
     }
 
diff --git a/lib/legacy/zstd_v03.h b/lib/legacy/zstd_v03.h
index efd8c2b9241a..5fc72730ce99 100644
--- a/lib/legacy/zstd_v03.h
+++ b/lib/legacy/zstd_v03.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/lib/legacy/zstd_v04.c b/lib/legacy/zstd_v04.c
index 4dec308168ef..56bf45222498 100644
--- a/lib/legacy/zstd_v04.c
+++ b/lib/legacy/zstd_v04.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -16,7 +16,7 @@
 #include <string.h>    /* memcpy */
 
 #include "zstd_v04.h"
-#include "error_private.h"
+#include "../common/error_private.h"
 
 
 /* ******************************************************************
@@ -161,7 +161,7 @@ MEM_STATIC void MEM_write16(void* memPtr, U16 value)
     memcpy(memPtr, &value, sizeof(value));
 }
 
-#endif // MEM_FORCE_MEMORY_ACCESS
+#endif /* MEM_FORCE_MEMORY_ACCESS */
 
 
 MEM_STATIC U16 MEM_readLE16(const void* memPtr)
@@ -2603,7 +2603,9 @@ static size_t ZSTD_getcBlockSize(const void* src, size_t srcSize, blockPropertie
 static size_t ZSTD_copyRawBlock(void* dst, size_t maxDstSize, const void* src, size_t srcSize)
 {
     if (srcSize > maxDstSize) return ERROR(dstSize_tooSmall);
-    memcpy(dst, src, srcSize);
+    if (srcSize > 0) {
+        memcpy(dst, src, srcSize);
+    }
     return srcSize;
 }
 
@@ -3008,8 +3010,10 @@ static size_t ZSTD_decompressSequences(
             size_t lastLLSize = litEnd - litPtr;
             if (litPtr > litEnd) return ERROR(corruption_detected);
             if (op+lastLLSize > oend) return ERROR(dstSize_tooSmall);
-            if (op != litPtr) memcpy(op, litPtr, lastLLSize);
-            op += lastLLSize;
+            if (lastLLSize > 0) {
+                if (op != litPtr) memcpy(op, litPtr, lastLLSize);
+                op += lastLLSize;
+            }
         }
     }
 
@@ -3407,7 +3411,9 @@ static size_t ZBUFF_decompressWithDictionary(ZBUFF_DCtx* zbc, const void* src, s
 static size_t ZBUFF_limitCopy(void* dst, size_t maxDstSize, const void* src, size_t srcSize)
 {
     size_t length = MIN(maxDstSize, srcSize);
-    memcpy(dst, src, length);
+    if (length > 0) {
+        memcpy(dst, src, length);
+    }
     return length;
 }
 
diff --git a/lib/legacy/zstd_v04.h b/lib/legacy/zstd_v04.h
index bb5f3b7d0b86..15fce0d487ff 100644
--- a/lib/legacy/zstd_v04.h
+++ b/lib/legacy/zstd_v04.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/lib/legacy/zstd_v05.c b/lib/legacy/zstd_v05.c
index 570e0ff86ee8..243d222562e3 100644
--- a/lib/legacy/zstd_v05.c
+++ b/lib/legacy/zstd_v05.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -11,7 +11,7 @@
 
 /*- Dependencies -*/
 #include "zstd_v05.h"
-#include "error_private.h"
+#include "../common/error_private.h"
 
 
 /* ******************************************************************
@@ -1804,7 +1804,7 @@ static size_t HUFv05_readStats(BYTE* huffWeight, size_t hwSize, U32* rankStats,
 
     if (!srcSize) return ERROR(srcSize_wrong);
     iSize = ip[0];
-    //memset(huffWeight, 0, hwSize);   /* is not necessary, even though some analyzer complain ... */
+    /* memset(huffWeight, 0, hwSize); */   /* is not necessary, even though some analyzer complain ... */
 
     if (iSize >= 128)  { /* special header */
         if (iSize >= (242)) {  /* RLE */
@@ -1879,7 +1879,7 @@ size_t HUFv05_readDTableX2 (U16* DTable, const void* src, size_t srcSize)
     HUFv05_DEltX2* const dt = (HUFv05_DEltX2*)dtPtr;
 
     HUFv05_STATIC_ASSERT(sizeof(HUFv05_DEltX2) == sizeof(U16));   /* if compilation fails here, assertion is false */
-    //memset(huffWeight, 0, sizeof(huffWeight));   /* is not necessary, even though some analyzer complain ... */
+    /* memset(huffWeight, 0, sizeof(huffWeight)); */   /* is not necessary, even though some analyzer complain ... */
 
     iSize = HUFv05_readStats(huffWeight, HUFv05_MAX_SYMBOL_VALUE + 1, rankVal, &nbSymbols, &tableLog, src, srcSize);
     if (HUFv05_isError(iSize)) return iSize;
@@ -2210,7 +2210,7 @@ size_t HUFv05_readDTableX4 (unsigned* DTable, const void* src, size_t srcSize)
 
     HUFv05_STATIC_ASSERT(sizeof(HUFv05_DEltX4) == sizeof(unsigned));   /* if compilation fails here, assertion is false */
     if (memLog > HUFv05_ABSOLUTEMAX_TABLELOG) return ERROR(tableLog_tooLarge);
-    //memset(weightList, 0, sizeof(weightList));   /* is not necessary, even though some analyzer complain ... */
+    /* memset(weightList, 0, sizeof(weightList)); */   /* is not necessary, even though some analyzer complain ... */
 
     iSize = HUFv05_readStats(weightList, HUFv05_MAX_SYMBOL_VALUE + 1, rankStats, &nbSymbols, &tableLog, src, srcSize);
     if (HUFv05_isError(iSize)) return iSize;
@@ -2539,9 +2539,9 @@ size_t HUFv05_decompress (void* dst, size_t dstSize, const void* cSrc, size_t cS
 
     return decompress[algoNb](dst, dstSize, cSrc, cSrcSize);
 
-    //return HUFv05_decompress4X2(dst, dstSize, cSrc, cSrcSize);   /* multi-streams single-symbol decoding */
-    //return HUFv05_decompress4X4(dst, dstSize, cSrc, cSrcSize);   /* multi-streams double-symbols decoding */
-    //return HUFv05_decompress4X6(dst, dstSize, cSrc, cSrcSize);   /* multi-streams quad-symbols decoding */
+    /* return HUFv05_decompress4X2(dst, dstSize, cSrc, cSrcSize); */   /* multi-streams single-symbol decoding */
+    /* return HUFv05_decompress4X4(dst, dstSize, cSrc, cSrcSize); */   /* multi-streams double-symbols decoding */
+    /* return HUFv05_decompress4X6(dst, dstSize, cSrc, cSrcSize); */   /* multi-streams quad-symbols decoding */
 }
 /*
     zstd - standard compression library
@@ -3362,8 +3362,10 @@ static size_t ZSTDv05_decompressSequences(
         size_t lastLLSize = litEnd - litPtr;
         if (litPtr > litEnd) return ERROR(corruption_detected);   /* too many literals already used */
         if (op+lastLLSize > oend) return ERROR(dstSize_tooSmall);
-        memcpy(op, litPtr, lastLLSize);
-        op += lastLLSize;
+        if (lastLLSize > 0) {
+            memcpy(op, litPtr, lastLLSize);
+            op += lastLLSize;
+        }
     }
 
     return op-ostart;
@@ -3791,7 +3793,9 @@ static size_t ZBUFFv05_blockHeaderSize = 3;
 static size_t ZBUFFv05_limitCopy(void* dst, size_t maxDstSize, const void* src, size_t srcSize)
 {
     size_t length = MIN(maxDstSize, srcSize);
-    memcpy(dst, src, length);
+    if (length > 0) {
+        memcpy(dst, src, length);
+    }
     return length;
 }
 
@@ -3928,7 +3932,7 @@ size_t ZBUFFv05_decompressContinue(ZBUFFv05_DCtx* zbc, void* dst, size_t* maxDst
                     *maxDstSizePtr = 0;
                     return headerSize - zbc->hPos;
                 }
-                // zbc->stage = ZBUFFv05ds_decodeHeader; break;   /* useless : stage follows */
+                /* zbc->stage = ZBUFFv05ds_decodeHeader; break; */   /* useless : stage follows */
             }
 	    /* fall-through */
         case ZBUFFv05ds_decodeHeader:
@@ -4001,7 +4005,7 @@ size_t ZBUFFv05_decompressContinue(ZBUFFv05_DCtx* zbc, void* dst, size_t* maxDst
                     if (!decodedSize) { zbc->stage = ZBUFFv05ds_read; break; }   /* this was just a header */
                     zbc->outEnd = zbc->outStart +  decodedSize;
                     zbc->stage = ZBUFFv05ds_flush;
-                    // break; /* ZBUFFv05ds_flush follows */
+                    /* break; */  /* ZBUFFv05ds_flush follows */
                 }
 	    }
 	    /* fall-through */
diff --git a/lib/legacy/zstd_v05.h b/lib/legacy/zstd_v05.h
index 4a979854b36e..167d892e6655 100644
--- a/lib/legacy/zstd_v05.h
+++ b/lib/legacy/zstd_v05.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -19,7 +19,7 @@ extern "C" {
 *  Dependencies
 ***************************************/
 #include <stddef.h>   /* size_t */
-#include "mem.h"      /* U64, U32 */
+#include "../common/mem.h"      /* U64, U32 */
 
 
 /* *************************************
diff --git a/lib/legacy/zstd_v06.c b/lib/legacy/zstd_v06.c
index 2a08e8dee842..c56f582772f8 100644
--- a/lib/legacy/zstd_v06.c
+++ b/lib/legacy/zstd_v06.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -14,7 +14,7 @@
 #include <stddef.h>    /* size_t, ptrdiff_t */
 #include <string.h>    /* memcpy */
 #include <stdlib.h>    /* malloc, free, qsort */
-#include "error_private.h"
+#include "../common/error_private.h"
 
 
 
@@ -1862,7 +1862,7 @@ MEM_STATIC size_t HUFv06_readStats(BYTE* huffWeight, size_t hwSize, U32* rankSta
 
     if (!srcSize) return ERROR(srcSize_wrong);
     iSize = ip[0];
-    //memset(huffWeight, 0, hwSize);   /* is not necessary, even though some analyzer complain ... */
+    /* memset(huffWeight, 0, hwSize); */   /* is not necessary, even though some analyzer complain ... */
 
     if (iSize >= 128)  { /* special header */
         if (iSize >= (242)) {  /* RLE */
@@ -2014,7 +2014,7 @@ size_t HUFv06_readDTableX2 (U16* DTable, const void* src, size_t srcSize)
     HUFv06_DEltX2* const dt = (HUFv06_DEltX2*)dtPtr;
 
     HUFv06_STATIC_ASSERT(sizeof(HUFv06_DEltX2) == sizeof(U16));   /* if compilation fails here, assertion is false */
-    //memset(huffWeight, 0, sizeof(huffWeight));   /* is not necessary, even though some analyzer complain ... */
+    /* memset(huffWeight, 0, sizeof(huffWeight)); */   /* is not necessary, even though some analyzer complain ... */
 
     iSize = HUFv06_readStats(huffWeight, HUFv06_MAX_SYMBOL_VALUE + 1, rankVal, &nbSymbols, &tableLog, src, srcSize);
     if (HUFv06_isError(iSize)) return iSize;
@@ -2340,7 +2340,7 @@ size_t HUFv06_readDTableX4 (U32* DTable, const void* src, size_t srcSize)
 
     HUFv06_STATIC_ASSERT(sizeof(HUFv06_DEltX4) == sizeof(U32));   /* if compilation fails here, assertion is false */
     if (memLog > HUFv06_ABSOLUTEMAX_TABLELOG) return ERROR(tableLog_tooLarge);
-    //memset(weightList, 0, sizeof(weightList));   /* is not necessary, even though some analyzer complain ... */
+    /* memset(weightList, 0, sizeof(weightList)); */   /* is not necessary, even though some analyzer complain ... */
 
     iSize = HUFv06_readStats(weightList, HUFv06_MAX_SYMBOL_VALUE + 1, rankStats, &nbSymbols, &tableLog, src, srcSize);
     if (HUFv06_isError(iSize)) return iSize;
@@ -2664,13 +2664,13 @@ size_t HUFv06_decompress (void* dst, size_t dstSize, const void* cSrc, size_t cS
 
     {   U32 algoNb = 0;
         if (Dtime[1] < Dtime[0]) algoNb = 1;
-        // if (Dtime[2] < Dtime[algoNb]) algoNb = 2;   /* current speed of HUFv06_decompress4X6 is not good */
+        /* if (Dtime[2] < Dtime[algoNb]) algoNb = 2; */   /* current speed of HUFv06_decompress4X6 is not good */
         return decompress[algoNb](dst, dstSize, cSrc, cSrcSize);
     }
 
-    //return HUFv06_decompress4X2(dst, dstSize, cSrc, cSrcSize);   /* multi-streams single-symbol decoding */
-    //return HUFv06_decompress4X4(dst, dstSize, cSrc, cSrcSize);   /* multi-streams double-symbols decoding */
-    //return HUFv06_decompress4X6(dst, dstSize, cSrc, cSrcSize);   /* multi-streams quad-symbols decoding */
+    /* return HUFv06_decompress4X2(dst, dstSize, cSrc, cSrcSize); */   /* multi-streams single-symbol decoding */
+    /* return HUFv06_decompress4X4(dst, dstSize, cSrc, cSrcSize); */   /* multi-streams double-symbols decoding */
+    /* return HUFv06_decompress4X6(dst, dstSize, cSrc, cSrcSize); */   /* multi-streams quad-symbols decoding */
 }
 /*
     Common functions of Zstd compression library
@@ -3501,8 +3501,10 @@ static size_t ZSTDv06_decompressSequences(
     {   size_t const lastLLSize = litEnd - litPtr;
         if (litPtr > litEnd) return ERROR(corruption_detected);   /* too many literals already used */
         if (op+lastLLSize > oend) return ERROR(dstSize_tooSmall);
-        memcpy(op, litPtr, lastLLSize);
-        op += lastLLSize;
+        if (lastLLSize > 0) {
+            memcpy(op, litPtr, lastLLSize);
+            op += lastLLSize;
+        }
     }
 
     return op-ostart;
@@ -4000,7 +4002,9 @@ size_t ZBUFFv06_decompressInit(ZBUFFv06_DCtx* zbd)
 MEM_STATIC size_t ZBUFFv06_limitCopy(void* dst, size_t dstCapacity, const void* src, size_t srcSize)
 {
     size_t length = MIN(dstCapacity, srcSize);
-    memcpy(dst, src, length);
+    if (length > 0) {
+        memcpy(dst, src, length);
+    }
     return length;
 }
 
@@ -4109,7 +4113,7 @@ size_t ZBUFFv06_decompressContinue(ZBUFFv06_DCtx* zbd,
                     if (!decodedSize) { zbd->stage = ZBUFFds_read; break; }   /* this was just a header */
                     zbd->outEnd = zbd->outStart +  decodedSize;
                     zbd->stage = ZBUFFds_flush;
-                    // break; /* ZBUFFds_flush follows */
+                    /* break; */ /* ZBUFFds_flush follows */
                 }
 	    }
 	    /* fall-through */
diff --git a/lib/legacy/zstd_v06.h b/lib/legacy/zstd_v06.h
index 07818571dca0..2fd99e629d47 100644
--- a/lib/legacy/zstd_v06.h
+++ b/lib/legacy/zstd_v06.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/lib/legacy/zstd_v07.c b/lib/legacy/zstd_v07.c
index a2eeff808e0e..9f3a597f1254 100644
--- a/lib/legacy/zstd_v07.c
+++ b/lib/legacy/zstd_v07.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -17,14 +17,14 @@
 #ifndef XXH_STATIC_LINKING_ONLY
 #  define XXH_STATIC_LINKING_ONLY    /* XXH64_state_t */
 #endif
-#include "xxhash.h"                  /* XXH64_* */
+#include "../common/xxhash.h"                  /* XXH64_* */
 #include "zstd_v07.h"
 
 #define FSEv07_STATIC_LINKING_ONLY   /* FSEv07_MIN_TABLELOG */
 #define HUFv07_STATIC_LINKING_ONLY   /* HUFv07_TABLELOG_ABSOLUTEMAX */
 #define ZSTDv07_STATIC_LINKING_ONLY
 
-#include "error_private.h"
+#include "../common/error_private.h"
 
 
 #ifdef ZSTDv07_STATIC_LINKING_ONLY
@@ -1314,7 +1314,7 @@ size_t HUFv07_readStats(BYTE* huffWeight, size_t hwSize, U32* rankStats,
 
     if (!srcSize) return ERROR(srcSize_wrong);
     iSize = ip[0];
-    //memset(huffWeight, 0, hwSize);   /* is not necessary, even though some analyzer complain ... */
+    /* memset(huffWeight, 0, hwSize); */   /* is not necessary, even though some analyzer complain ... */
 
     if (iSize >= 128)  { /* special header */
         if (iSize >= (242)) {  /* RLE */
@@ -1784,7 +1784,7 @@ size_t HUFv07_readDTableX2 (HUFv07_DTable* DTable, const void* src, size_t srcSi
     HUFv07_DEltX2* const dt = (HUFv07_DEltX2*)dtPtr;
 
     HUFv07_STATIC_ASSERT(sizeof(DTableDesc) == sizeof(HUFv07_DTable));
-    //memset(huffWeight, 0, sizeof(huffWeight));   /* is not necessary, even though some analyzer complain ... */
+    /* memset(huffWeight, 0, sizeof(huffWeight)); */   /* is not necessary, even though some analyzer complain ... */
 
     iSize = HUFv07_readStats(huffWeight, HUFv07_SYMBOLVALUE_MAX + 1, rankVal, &nbSymbols, &tableLog, src, srcSize);
     if (HUFv07_isError(iSize)) return iSize;
@@ -2148,7 +2148,7 @@ size_t HUFv07_readDTableX4 (HUFv07_DTable* DTable, const void* src, size_t srcSi
 
     HUFv07_STATIC_ASSERT(sizeof(HUFv07_DEltX4) == sizeof(HUFv07_DTable));   /* if compilation fails here, assertion is false */
     if (maxTableLog > HUFv07_TABLELOG_ABSOLUTEMAX) return ERROR(tableLog_tooLarge);
-    //memset(weightList, 0, sizeof(weightList));   /* is not necessary, even though some analyzer complain ... */
+    /* memset(weightList, 0, sizeof(weightList)); */   /* is not necessary, even though some analyzer complain ... */
 
     iSize = HUFv07_readStats(weightList, HUFv07_SYMBOLVALUE_MAX + 1, rankStats, &nbSymbols, &tableLog, src, srcSize);
     if (HUFv07_isError(iSize)) return iSize;
@@ -2530,8 +2530,8 @@ size_t HUFv07_decompress (void* dst, size_t dstSize, const void* cSrc, size_t cS
         return decompress[algoNb](dst, dstSize, cSrc, cSrcSize);
     }
 
-    //return HUFv07_decompress4X2(dst, dstSize, cSrc, cSrcSize);   /* multi-streams single-symbol decoding */
-    //return HUFv07_decompress4X4(dst, dstSize, cSrc, cSrcSize);   /* multi-streams double-symbols decoding */
+    /* return HUFv07_decompress4X2(dst, dstSize, cSrc, cSrcSize); */   /* multi-streams single-symbol decoding */
+    /* return HUFv07_decompress4X4(dst, dstSize, cSrc, cSrcSize); */   /* multi-streams double-symbols decoding */
 }
 
 size_t HUFv07_decompress4X_DCtx (HUFv07_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize)
@@ -3272,7 +3272,9 @@ static size_t ZSTDv07_getcBlockSize(const void* src, size_t srcSize, blockProper
 static size_t ZSTDv07_copyRawBlock(void* dst, size_t dstCapacity, const void* src, size_t srcSize)
 {
     if (srcSize > dstCapacity) return ERROR(dstSize_tooSmall);
-    memcpy(dst, src, srcSize);
+    if (srcSize > 0) {
+        memcpy(dst, src, srcSize);
+    }
     return srcSize;
 }
 
@@ -3712,10 +3714,12 @@ static size_t ZSTDv07_decompressSequences(
 
     /* last literal segment */
     {   size_t const lastLLSize = litEnd - litPtr;
-        //if (litPtr > litEnd) return ERROR(corruption_detected);   /* too many literals already used */
+        /* if (litPtr > litEnd) return ERROR(corruption_detected); */   /* too many literals already used */
         if (lastLLSize > (size_t)(oend-op)) return ERROR(dstSize_tooSmall);
-        memcpy(op, litPtr, lastLLSize);
-        op += lastLLSize;
+        if (lastLLSize > 0) {
+            memcpy(op, litPtr, lastLLSize);
+            op += lastLLSize;
+        }
     }
 
     return op-ostart;
@@ -3776,7 +3780,9 @@ ZSTDLIBv07_API size_t ZSTDv07_insertBlock(ZSTDv07_DCtx* dctx, const void* blockS
 static size_t ZSTDv07_generateNxBytes(void* dst, size_t dstCapacity, BYTE byte, size_t length)
 {
     if (length > dstCapacity) return ERROR(dstSize_tooSmall);
-    memset(dst, byte, length);
+    if (length > 0) {
+        memset(dst, byte, length);
+    }
     return length;
 }
 
@@ -4378,7 +4384,9 @@ size_t ZBUFFv07_decompressInit(ZBUFFv07_DCtx* zbd)
 MEM_STATIC size_t ZBUFFv07_limitCopy(void* dst, size_t dstCapacity, const void* src, size_t srcSize)
 {
     size_t const length = MIN(dstCapacity, srcSize);
-    memcpy(dst, src, length);
+    if (length > 0) {
+        memcpy(dst, src, length);
+    }
     return length;
 }
 
diff --git a/lib/legacy/zstd_v07.h b/lib/legacy/zstd_v07.h
index a566c1d102ac..9da50c4e6418 100644
--- a/lib/legacy/zstd_v07.h
+++ b/lib/legacy/zstd_v07.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/lib/libzstd.pc.in b/lib/libzstd.pc.in
index e7880be475d0..8ec0235ad9b8 100644
--- a/lib/libzstd.pc.in
+++ b/lib/libzstd.pc.in
@@ -4,8 +4,8 @@
 
 prefix=@PREFIX@
 exec_prefix=${prefix}
-includedir=${prefix}/include
-libdir=${exec_prefix}/lib
+includedir=${prefix}/@INCLUDEDIR@
+libdir=${exec_prefix}/@LIBDIR@
 
 Name: zstd
 Description: fast lossless compression algorithm library
diff --git a/lib/zstd.h b/lib/zstd.h
index 72080ea87e84..8c6fc6ae90e6 100644
--- a/lib/zstd.h
+++ b/lib/zstd.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -72,7 +72,7 @@ extern "C" {
 /*------   Version   ------*/
 #define ZSTD_VERSION_MAJOR    1
 #define ZSTD_VERSION_MINOR    4
-#define ZSTD_VERSION_RELEASE  4
+#define ZSTD_VERSION_RELEASE  5
 
 #define ZSTD_VERSION_NUMBER  (ZSTD_VERSION_MAJOR *100*100 + ZSTD_VERSION_MINOR *100 + ZSTD_VERSION_RELEASE)
 ZSTDLIB_API unsigned ZSTD_versionNumber(void);   /**< to check runtime library version */
@@ -274,7 +274,10 @@ typedef enum {
                               * Default level is ZSTD_CLEVEL_DEFAULT==3.
                               * Special: value 0 means default, which is controlled by ZSTD_CLEVEL_DEFAULT.
                               * Note 1 : it's possible to pass a negative compression level.
-                              * Note 2 : setting a level resets all other compression parameters to default */
+                              * Note 2 : setting a level does not automatically set all other compression parameters
+                              *   to default. Setting this will however eventually dynamically impact the compression
+                              *   parameters which have not been manually set. The manually set
+                              *   ones will 'stick'. */
     /* Advanced compression parameters :
      * It's possible to pin down compression parameters to some specific values.
      * In which case, these values are no longer dynamically selected by the compressor */
@@ -519,11 +522,13 @@ typedef enum {
     /* note : additional experimental parameters are also available
      * within the experimental section of the API.
      * At the time of this writing, they include :
-     * ZSTD_c_format
+     * ZSTD_d_format
+     * ZSTD_d_stableOutBuffer
      * Because they are not stable, it's necessary to define ZSTD_STATIC_LINKING_ONLY to access them.
      * note : never ever use experimentalParam? names directly
      */
-     ZSTD_d_experimentalParam1=1000
+     ZSTD_d_experimentalParam1=1000,
+     ZSTD_d_experimentalParam2=1001
 
 } ZSTD_dParameter;
 
@@ -763,7 +768,7 @@ ZSTDLIB_API size_t ZSTD_freeDStream(ZSTD_DStream* zds);
 
 /* This function is redundant with the advanced API and equivalent to:
  *
- *     ZSTD_DCtx_reset(zds);
+ *     ZSTD_DCtx_reset(zds, ZSTD_reset_session_only);
  *     ZSTD_DCtx_refDDict(zds, NULL);
  */
 ZSTDLIB_API size_t ZSTD_initDStream(ZSTD_DStream* zds);
@@ -1263,23 +1268,28 @@ ZSTDLIB_API size_t ZSTD_getSequences(ZSTD_CCtx* zc, ZSTD_Sequence* outSeqs,
 ***************************************/
 
 /*! ZSTD_estimate*() :
- *  These functions make it possible to estimate memory usage of a future
- *  {D,C}Ctx, before its creation.
+ *  These functions make it possible to estimate memory usage
+ *  of a future {D,C}Ctx, before its creation.
+ *
+ *  ZSTD_estimateCCtxSize() will provide a memory budget large enough
+ *  for any compression level up to selected one.
+ *  Note : Unlike ZSTD_estimateCStreamSize*(), this estimate
+ *         does not include space for a window buffer.
+ *         Therefore, the estimation is only guaranteed for single-shot compressions, not streaming.
+ *  The estimate will assume the input may be arbitrarily large,
+ *  which is the worst case.
  *
- *  ZSTD_estimateCCtxSize() will provide a budget large enough for any
- *  compression level up to selected one. Unlike ZSTD_estimateCStreamSize*(),
- *  this estimate does not include space for a window buffer, so this estimate
- *  is guaranteed to be enough for single-shot compressions, but not streaming
- *  compressions. It will however assume the input may be arbitrarily large,
- *  which is the worst case. If srcSize is known to always be small,
- *  ZSTD_estimateCCtxSize_usingCParams() can provide a tighter estimation.
- *  ZSTD_estimateCCtxSize_usingCParams() can be used in tandem with
- *  ZSTD_getCParams() to create cParams from compressionLevel.
- *  ZSTD_estimateCCtxSize_usingCCtxParams() can be used in tandem with
- *  ZSTD_CCtxParams_setParameter().
+ *  When srcSize can be bound by a known and rather "small" value,
+ *  this fact can be used to provide a tighter estimation
+ *  because the CCtx compression context will need less memory.
+ *  This tighter estimation can be provided by more advanced functions
+ *  ZSTD_estimateCCtxSize_usingCParams(), which can be used in tandem with ZSTD_getCParams(),
+ *  and ZSTD_estimateCCtxSize_usingCCtxParams(), which can be used in tandem with ZSTD_CCtxParams_setParameter().
+ *  Both can be used to estimate memory using custom compression parameters and arbitrary srcSize limits.
  *
- *  Note: only single-threaded compression is supported. This function will
- *  return an error code if ZSTD_c_nbWorkers is >= 1. */
+ *  Note 2 : only single-threaded compression is supported.
+ *  ZSTD_estimateCCtxSize_usingCCtxParams() will return an error code if ZSTD_c_nbWorkers is >= 1.
+ */
 ZSTDLIB_API size_t ZSTD_estimateCCtxSize(int compressionLevel);
 ZSTDLIB_API size_t ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams);
 ZSTDLIB_API size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params);
@@ -1642,6 +1652,37 @@ ZSTDLIB_API size_t ZSTD_DCtx_setMaxWindowSize(ZSTD_DCtx* dctx, size_t maxWindowS
  * allowing selection between ZSTD_format_e input compression formats
  */
 #define ZSTD_d_format ZSTD_d_experimentalParam1
+/* ZSTD_d_stableOutBuffer
+ * Experimental parameter.
+ * Default is 0 == disabled. Set to 1 to enable.
+ *
+ * Tells the decompressor that the ZSTD_outBuffer will ALWAYS be the same
+ * between calls, except for the modifications that zstd makes to pos (the
+ * caller must not modify pos). This is checked by the decompressor, and
+ * decompression will fail if it ever changes. Therefore the ZSTD_outBuffer
+ * MUST be large enough to fit the entire decompressed frame. This will be
+ * checked when the frame content size is known. The data in the ZSTD_outBuffer
+ * in the range [dst, dst + pos) MUST not be modified during decompression
+ * or you will get data corruption.
+ *
+ * When this flags is enabled zstd won't allocate an output buffer, because
+ * it can write directly to the ZSTD_outBuffer, but it will still allocate
+ * an input buffer large enough to fit any compressed block. This will also
+ * avoid the memcpy() from the internal output buffer to the ZSTD_outBuffer.
+ * If you need to avoid the input buffer allocation use the buffer-less
+ * streaming API.
+ *
+ * NOTE: So long as the ZSTD_outBuffer always points to valid memory, using
+ * this flag is ALWAYS memory safe, and will never access out-of-bounds
+ * memory. However, decompression WILL fail if you violate the preconditions.
+ *
+ * WARNING: The data in the ZSTD_outBuffer in the range [dst, dst + pos) MUST
+ * not be modified during decompression or you will get data corruption. This
+ * is because zstd needs to reference data in the ZSTD_outBuffer to regenerate
+ * matches. Normally zstd maintains its own buffer for this purpose, but passing
+ * this flag tells zstd to use the user provided buffer.
+ */
+#define ZSTD_d_stableOutBuffer ZSTD_d_experimentalParam2
 
 /*! ZSTD_DCtx_setFormat() :
  *  Instruct the decoder context about what kind of data to decode next.
diff --git a/programs/Makefile b/programs/Makefile
index 64dcae0028e7..418ad4e6348e 100644
--- a/programs/Makefile
+++ b/programs/Makefile
@@ -1,10 +1,11 @@
 # ################################################################
-# Copyright (c) 2015-present, Yann Collet, Facebook, Inc.
+# Copyright (c) 2015-2020, Yann Collet, Facebook, Inc.
 # All rights reserved.
 #
 # This source code is licensed under both the BSD-style license (found in the
 # LICENSE file in the root directory of this source tree) and the GPLv2 (found
 # in the COPYING file in the root directory of this source tree).
+# You may select, at your option, one of the above-listed licenses.
 # ##########################################################################
 # zstd : Command Line Utility, supporting gzip-like arguments
 # zstd32 : Same as zstd, but forced to compile in 32-bits mode
@@ -42,9 +43,7 @@ else
 ALIGN_LOOP =
 endif
 
-CPPFLAGS+= -I$(ZSTDDIR) -I$(ZSTDDIR)/common -I$(ZSTDDIR)/compress \
-           -I$(ZSTDDIR)/dictBuilder \
-           -DXXH_NAMESPACE=ZSTD_
+CPPFLAGS+= -DXXH_NAMESPACE=ZSTD_
 ifeq ($(OS),Windows_NT)   # MinGW assumed
 CPPFLAGS   += -D__USE_MINGW_ANSI_STDIO   # compatibility with %zu formatting
 endif
@@ -71,13 +70,15 @@ ifneq ($(ZSTD_LEGACY_SUPPORT), 0)
 ifeq ($(shell test $(ZSTD_LEGACY_SUPPORT) -lt 8; echo $$?), 0)
 	ZSTDLEGACY_FILES += $(shell ls $(ZSTDDIR)/legacy/*.c | $(GREP) 'v0[$(ZSTD_LEGACY_SUPPORT)-7]')
 endif
-	CPPFLAGS += -I$(ZSTDDIR)/legacy
 else
 endif
 
 # Sort files in alphabetical order for reproducible builds
 ZSTDLIB_FILES := $(sort $(wildcard $(ZSTD_FILES)) $(wildcard $(ZSTDLEGACY_FILES)) $(wildcard $(ZDICT_FILES)))
 
+ZSTD_CLI_FILES := $(wildcard *.c)
+ZSTD_CLI_OBJ := $(patsubst %.c,%.o,$(ZSTD_CLI_FILES))
+
 # Define *.exe as extension for Windows systems
 ifneq (,$(filter Windows%,$(OS)))
 EXT =.exe
@@ -94,9 +95,12 @@ endif
 
 VOID = /dev/null
 
+# Make 4.3 doesn't support '\#' anymore (https://lwn.net/Articles/810071/)
+NUM_SYMBOL := \#
+
 # thread detection
 NO_THREAD_MSG := ==> no threads, building without multithreading support
-HAVE_PTHREAD := $(shell printf '\#include <pthread.h>\nint main(void) { return 0; }' > have_pthread.c && $(CC) $(FLAGS) -o have_pthread$(EXT) have_pthread.c -pthread 2> $(VOID) && rm have_pthread$(EXT) && echo 1 || echo 0; rm have_pthread.c)
+HAVE_PTHREAD := $(shell printf '$(NUM_SYMBOL)include <pthread.h>\nint main(void) { return 0; }' > have_pthread.c && $(CC) $(FLAGS) -o have_pthread$(EXT) have_pthread.c -pthread 2> $(VOID) && rm have_pthread$(EXT) && echo 1 || echo 0; rm have_pthread.c)
 HAVE_THREAD := $(shell [ "$(HAVE_PTHREAD)" -eq "1" -o -n "$(filter Windows%,$(OS))" ] && echo 1 || echo 0)
 ifeq ($(HAVE_THREAD), 1)
 THREAD_MSG := ==> building with threading support
@@ -108,7 +112,7 @@ endif
 
 # zlib detection
 NO_ZLIB_MSG := ==> no zlib, building zstd without .gz support
-HAVE_ZLIB := $(shell printf '\#include <zlib.h>\nint main(void) { return 0; }' > have_zlib.c && $(CC) $(FLAGS) -o have_zlib$(EXT) have_zlib.c -lz 2> $(VOID) && rm have_zlib$(EXT) && echo 1 || echo 0; rm have_zlib.c)
+HAVE_ZLIB := $(shell printf '$(NUM_SYMBOL)include <zlib.h>\nint main(void) { return 0; }' > have_zlib.c && $(CC) $(FLAGS) -o have_zlib$(EXT) have_zlib.c -lz 2> $(VOID) && rm have_zlib$(EXT) && echo 1 || echo 0; rm have_zlib.c)
 ifeq ($(HAVE_ZLIB), 1)
 ZLIB_MSG := ==> building zstd with .gz compression support
 ZLIBCPP = -DZSTD_GZCOMPRESS -DZSTD_GZDECOMPRESS
@@ -119,7 +123,7 @@ endif
 
 # lzma detection
 NO_LZMA_MSG := ==> no liblzma, building zstd without .xz/.lzma support
-HAVE_LZMA := $(shell printf '\#include <lzma.h>\nint main(void) { return 0; }' > have_lzma.c && $(CC) $(FLAGS) -o have_lzma$(EXT) have_lzma.c -llzma 2> $(VOID) && rm have_lzma$(EXT) && echo 1 || echo 0; rm have_lzma.c)
+HAVE_LZMA := $(shell printf '$(NUM_SYMBOL)include <lzma.h>\nint main(void) { return 0; }' > have_lzma.c && $(CC) $(FLAGS) -o have_lzma$(EXT) have_lzma.c -llzma 2> $(VOID) && rm have_lzma$(EXT) && echo 1 || echo 0; rm have_lzma.c)
 ifeq ($(HAVE_LZMA), 1)
 LZMA_MSG := ==> building zstd with .xz/.lzma compression support
 LZMACPP = -DZSTD_LZMACOMPRESS -DZSTD_LZMADECOMPRESS
@@ -130,7 +134,7 @@ endif
 
 # lz4 detection
 NO_LZ4_MSG := ==> no liblz4, building zstd without .lz4 support
-HAVE_LZ4 := $(shell printf '\#include <lz4frame.h>\n\#include <lz4.h>\nint main(void) { return 0; }' > have_lz4.c && $(CC) $(FLAGS) -o have_lz4$(EXT) have_lz4.c -llz4 2> $(VOID) && rm have_lz4$(EXT) && echo 1 || echo 0; rm have_lz4.c)
+HAVE_LZ4 := $(shell printf '$(NUM_SYMBOL)include <lz4frame.h>\n$(NUM_SYMBOL)include <lz4.h>\nint main(void) { return 0; }' > have_lz4.c && $(CC) $(FLAGS) -o have_lz4$(EXT) have_lz4.c -llz4 2> $(VOID) && rm have_lz4$(EXT) && echo 1 || echo 0; rm have_lz4.c)
 ifeq ($(HAVE_LZ4), 1)
 LZ4_MSG := ==> building zstd with .lz4 compression support
 LZ4CPP = -DZSTD_LZ4COMPRESS -DZSTD_LZ4DECOMPRESS
@@ -158,22 +162,22 @@ default: zstd-release
 all: zstd
 
 .PHONY: allVariants
-allVariants: zstd zstd-compress zstd-decompress zstd-small zstd-nolegacy
+allVariants: zstd zstd-compress zstd-decompress zstd-small zstd-nolegacy zstd-dictBuilder
 
 $(ZSTDDECOMP_O): CFLAGS += $(ALIGN_LOOP)
 
 zstd : CPPFLAGS += $(THREAD_CPP) $(ZLIBCPP) $(LZMACPP) $(LZ4CPP)
 zstd : LDFLAGS += $(THREAD_LD) $(ZLIBLD) $(LZMALD) $(LZ4LD) $(DEBUGFLAGS_LD)
 zstd : CPPFLAGS += -DZSTD_LEGACY_SUPPORT=$(ZSTD_LEGACY_SUPPORT)
-zstd : $(ZSTDLIB_FILES) zstdcli.o util.o timefn.o fileio.o benchfn.o benchzstd.o datagen.o dibio.o
+ifneq (,$(filter Windows%,$(OS)))
+zstd : $(RES_FILE)
+endif
+zstd : $(ZSTDLIB_FILES) $(ZSTD_CLI_OBJ)
 	@echo "$(THREAD_MSG)"
 	@echo "$(ZLIB_MSG)"
 	@echo "$(LZMA_MSG)"
 	@echo "$(LZ4_MSG)"
-ifneq (,$(filter Windows%,$(OS)))
-	windres/generate_res.bat
-endif
-	$(CC) $(FLAGS) $^ $(RES_FILE) -o $@$(EXT) $(LDFLAGS)
+	$(CC) $(FLAGS) $^ -o $@$(EXT) $(LDFLAGS)
 
 .PHONY: zstd-release
 zstd-release: DEBUGFLAGS := -DBACKTRACE_ENABLE=0
@@ -183,13 +187,15 @@ zstd-release: zstd
 zstd32 : CPPFLAGS += $(THREAD_CPP)
 zstd32 : LDFLAGS  += $(THREAD_LD)
 zstd32 : CPPFLAGS += -DZSTD_LEGACY_SUPPORT=$(ZSTD_LEGACY_SUPPORT)
-zstd32 : $(ZSTDLIB_FILES) zstdcli.c util.c timefn.c fileio.c benchfn.c benchzstd.c datagen.c dibio.c
 ifneq (,$(filter Windows%,$(OS)))
-	windres/generate_res.bat
+zstd32 : $(RES32_FILE)
 endif
-	$(CC) -m32 $(FLAGS) $^ $(RES32_FILE) -o $@$(EXT)
+zstd32 : $(ZSTDLIB_FILES) $(ZSTD_CLI_FILES)
+	$(CC) -m32 $(FLAGS) $^ -o $@$(EXT)
 
-zstd-nolegacy : $(ZSTD_FILES) $(ZDICT_FILES) zstdcli.o util.o fileio.c benchfn.o benchzstd.o timefn.o datagen.o dibio.o
+## zstd-nolegacy: same scope as zstd, with just support of legacy formats removed
+zstd-nolegacy : LDFLAGS += $(THREAD_LD) $(ZLIBLD) $(LZMALD) $(LZ4LD) $(DEBUGFLAGS_LD)
+zstd-nolegacy : $(ZSTD_FILES) $(ZDICT_FILES) $(ZSTD_CLI_OBJ)
 	$(CC) $(FLAGS) $^ -o $@$(EXT) $(LDFLAGS)
 
 zstd-nomt : THREAD_CPP :=
@@ -207,7 +213,21 @@ zstd-noxz : LZMALD  :=
 zstd-noxz : LZMA_MSG := - xz/lzma support is disabled
 zstd-noxz : zstd
 
+## zstd-dll: zstd executable linked to dynamic library libzstd (must already exist)
+# note : the following target doesn't link
+#        because zstd uses non-public symbols from libzstd
+#        such as XXH64 (for benchmark),
+#        ZDICT_trainFromBuffer_unsafe_legacy (for dictionary builder)
+#        and ZSTD_cycleLog (likely for --patch-from).
+#        It's unclear at this stage if this is a scenario that must be supported
+.PHONY: zstd-dll
+zstd-dll : LDFLAGS+= -L$(ZSTDDIR) -lzstd
+zstd-dll : ZSTDLIB_FILES =
+zstd-dll : $(ZSTD_CLI_OBJ)
+	$(CC) $(FLAGS) $^ -o $@$(EXT) $(LDFLAGS)
+
 
+## zstd-pgo: zstd executable optimized with pgo. `gcc` only.
 zstd-pgo :
 	$(MAKE) clean
 	$(MAKE) zstd MOREFLAGS=-fprofile-generate
@@ -218,9 +238,10 @@ zstd-pgo :
 	./zstd -b7i2 $(PROFILE_WITH)
 	./zstd -b5 $(PROFILE_WITH)
 	$(RM) zstd *.o $(ZSTDDECOMP_O) $(ZSTDDIR)/compress/*.o
+	case $(CC) in *clang*) if ! [ -e default.profdata ]; then llvm-profdata merge -output=default.profdata default*.profraw; fi ;; esac
 	$(MAKE) zstd MOREFLAGS=-fprofile-use
 
-# minimal target, with only zstd compression and decompression. no bench. no legacy.
+## zstd-small: minimal target, supporting only zstd compression and decompression. no bench. no legacy. no other format.
 zstd-small: CFLAGS = -Os -s
 zstd-frugal zstd-small: $(ZSTD_FILES) zstdcli.c util.c timefn.c fileio.c
 	$(CC) $(FLAGS) -DZSTD_NOBENCH -DZSTD_NODICT $^ -o $@$(EXT)
@@ -231,12 +252,25 @@ zstd-decompress: $(ZSTDCOMMON_FILES) $(ZSTDDECOMP_FILES) zstdcli.c util.c timefn
 zstd-compress: $(ZSTDCOMMON_FILES) $(ZSTDCOMP_FILES) zstdcli.c util.c timefn.c fileio.c
 	$(CC) $(FLAGS) -DZSTD_NOBENCH -DZSTD_NODICT -DZSTD_NODECOMPRESS $^ -o $@$(EXT)
 
+## zstd-dictBuilder: executable supporting dictionary creation and compression (only)
+zstd-dictBuilder: CPPFLAGS += -DZSTD_NOBENCH -DZSTD_NODECOMPRESS
+zstd-dictBuilder: $(ZSTDCOMMON_FILES) $(ZSTDCOMP_FILES) $(ZDICT_FILES) zstdcli.c util.c timefn.c fileio.c dibio.c
+	$(CC) $(FLAGS) $^ -o $@$(EXT)
+
 zstdmt: zstd
 	ln -sf zstd zstdmt
 
 .PHONY: generate_res
-generate_res:
-	windres/generate_res.bat
+generate_res: $(RES64_FILE) $(RES32_FILE)
+
+ifneq (,$(filter Windows%,$(OS)))
+RC ?= windres
+# http://stackoverflow.com/questions/708238/how-do-i-add-an-icon-to-a-mingw-gcc-compiled-executable
+$(RES64_FILE): windres/zstd.rc
+	$(RC) -o $@ -I ../lib -I windres -i $< -O coff -F pe-x86-64
+$(RES32_FILE): windres/zstd.rc
+	$(RC) -o $@ -I ../lib -I windres -i $< -O coff -F pe-i386
+endif
 
 .PHONY: clean
 clean:
@@ -245,7 +279,7 @@ clean:
 	@$(RM) core *.o tmp* result* *.gcda dictionary *.zst \
         zstd$(EXT) zstd32$(EXT) zstd-compress$(EXT) zstd-decompress$(EXT) \
         zstd-small$(EXT) zstd-frugal$(EXT) zstd-nolegacy$(EXT) zstd4$(EXT) \
-        *.gcda default.profraw have_zlib$(EXT)
+        zstd-dictBuilder$(EXT) *.gcda default*.profraw default.profdata have_zlib$(EXT)
 	@echo Cleaning completed
 
 MD2ROFF = ronn
@@ -286,6 +320,7 @@ ifeq ($HAVE_COLORNEVER, 1)
 EGREP_OPTIONS += --color=never
 endif
 EGREP = egrep $(EGREP_OPTIONS)
+AWK = awk
 
 # Print a two column output of targets and their description. To add a target description, put a
 # comment in the Makefile with the format "## <TARGET>: <DESCRIPTION>".  For example:
@@ -294,14 +329,14 @@ EGREP = egrep $(EGREP_OPTIONS)
 .PHONY: list
 list:
 	@TARGETS=$$($(MAKE) -pRrq -f $(lastword $(MAKEFILE_LIST)) : 2>/dev/null \
-		| awk -v RS= -F: '/^# File/,/^# Finished Make data base/ {if ($$1 !~ "^[#.]") {print $$1}}' \
+		| $(AWK) -v RS= -F: '/^# File/,/^# Finished Make data base/ {if ($$1 !~ "^[#.]") {print $$1}}' \
 		| $(EGREP) -v  -e '^[^[:alnum:]]' | sort); \
 	{ \
 	    printf "Target Name\tDescription\n"; \
 	    printf "%0.s-" {1..16}; printf "\t"; printf "%0.s-" {1..40}; printf "\n"; \
 	    for target in $$TARGETS; do \
 	        line=$$($(EGREP) "^##[[:space:]]+$$target:" $(lastword $(MAKEFILE_LIST))); \
-	        description=$$(echo $$line | awk '{i=index($$0,":"); print substr($$0,i+1)}' | xargs); \
+	        description=$$(echo $$line | $(AWK) '{i=index($$0,":"); print substr($$0,i+1)}' | xargs); \
 	        printf "$$target\t$$description\n"; \
 	    done \
 	} | column -t -s $$'\t'
@@ -342,10 +377,10 @@ INSTALL_MAN     ?= $(INSTALL_DATA)
 install: zstd
 	@echo Installing binaries
 	@$(INSTALL) -d -m 755 $(DESTDIR)$(BINDIR)/ $(DESTDIR)$(MAN1DIR)/
-	@$(INSTALL_PROGRAM) zstd $(DESTDIR)$(BINDIR)/zstd
-	@ln -sf zstd $(DESTDIR)$(BINDIR)/zstdcat
-	@ln -sf zstd $(DESTDIR)$(BINDIR)/unzstd
-	@ln -sf zstd $(DESTDIR)$(BINDIR)/zstdmt
+	@$(INSTALL_PROGRAM) zstd$(EXT) $(DESTDIR)$(BINDIR)/zstd$(EXT)
+	@ln -sf zstd$(EXT) $(DESTDIR)$(BINDIR)/zstdcat$(EXT)
+	@ln -sf zstd$(EXT) $(DESTDIR)$(BINDIR)/unzstd$(EXT)
+	@ln -sf zstd$(EXT) $(DESTDIR)$(BINDIR)/zstdmt$(EXT)
 	@$(INSTALL_SCRIPT) zstdless $(DESTDIR)$(BINDIR)/zstdless
 	@$(INSTALL_SCRIPT) zstdgrep $(DESTDIR)$(BINDIR)/zstdgrep
 	@echo Installing man pages
diff --git a/programs/README.md b/programs/README.md
index 7668d49a2073..53706de727b7 100644
--- a/programs/README.md
+++ b/programs/README.md
@@ -10,7 +10,7 @@ There are however other Makefile targets that create different variations of CLI
 - `zstd-decompress` : version of CLI which can only decompress zstd format
 
 
-#### Compilation variables
+### Compilation variables
 `zstd` scope can be altered by modifying the following `make` variables :
 
 - __HAVE_THREAD__ : multithreading is automatically enabled when `pthread` is detected.
@@ -61,6 +61,24 @@ There are however other Makefile targets that create different variations of CLI
   In which case, linking stage will fail if `lz4` library cannot be found.
   This is useful to prevent silent feature disabling.
 
+- __ZSTD_NOBENCH__ : `zstd` cli will be compiled without its integrated benchmark module.
+  This can be useful to produce smaller binaries.
+  In this case, the corresponding unit can also be excluded from compilation target.
+
+- __ZSTD_NODICT__ : `zstd` cli will be compiled without support for the integrated dictionary builder.
+  This can be useful to produce smaller binaries.
+  In this case, the corresponding unit can also be excluded from compilation target.
+
+- __ZSTD_NOCOMPRESS__ : `zstd` cli will be compiled without support for compression.
+  The resulting binary will only be able to decompress files.
+  This can be useful to produce smaller binaries.
+  A corresponding `Makefile` target using this ability is `zstd-decompress`.
+
+- __ZSTD_NODECOMPRESS__ : `zstd` cli will be compiled without support for decompression.
+  The resulting binary will only be able to compress files.
+  This can be useful to produce smaller binaries.
+  A corresponding `Makefile` target using this ability is `zstd-compress`.
+
 - __BACKTRACE__ : `zstd` can display a stack backtrace when execution
   generates a runtime exception. By default, this feature may be
   degraded/disabled on some platforms unless additional compiler directives are
@@ -69,11 +87,11 @@ There are however other Makefile targets that create different variations of CLI
   Example : `make zstd BACKTRACE=1`
 
 
-#### Aggregation of parameters
+### Aggregation of parameters
 CLI supports aggregation of parameters i.e. `-b1`, `-e18`, and `-i1` can be joined into `-b1e18i1`.
 
 
-#### Symlink shortcuts
+### Symlink shortcuts
 It's possible to invoke `zstd` through a symlink.
 When the name of the symlink has a specific value, it triggers an associated behavior.
 - `zstdmt` : compress using all cores available on local system.
@@ -86,7 +104,7 @@ When the name of the symlink has a specific value, it triggers an associated beh
 - `ungz`, `unxz` and `unlzma` will do the same, and will also remove source file by default (use `--keep` to preserve).
 
 
-#### Dictionary builder in Command Line Interface
+### Dictionary builder in Command Line Interface
 Zstd offers a training mode, which can be used to tune the algorithm for a selected
 type of data, by providing it with a few samples. The result of the training is stored
 in a file selected with the `-o` option (default name is `dictionary`),
@@ -106,7 +124,7 @@ Usage of the dictionary builder and created dictionaries with CLI:
 3. Decompress with the dictionary: `zstd --decompress FILE.zst -D dictionaryName`
 
 
-#### Benchmark in Command Line Interface
+### Benchmark in Command Line Interface
 CLI includes in-memory compression benchmark module for zstd.
 The benchmark is conducted using given filenames. The files are read into memory and joined together.
 It makes benchmark more precise as it eliminates I/O overhead.
@@ -118,7 +136,7 @@ One can select compression levels starting from `-b` and ending with `-e`.
 The `-i` parameter selects minimal time used for each of tested levels.
 
 
-#### Usage of Command Line Interface
+### Usage of Command Line Interface
 The full list of options can be obtained with `-h` or `-H` parameter:
 ```
 Usage :
@@ -142,23 +160,34 @@ Advanced arguments :
  -q     : suppress warnings; specify twice to suppress errors too
  -c     : force write to standard output, even if it is the console
  -l     : print information about zstd compressed files
+--exclude-compressed:  only compress files that are not previously compressed
 --ultra : enable levels beyond 19, up to 22 (requires more memory)
---long  : enable long distance matching (requires more memory)
+--long[=#]: enable long distance matching with given window log (default: 27)
+--fast[=#]: switch to very fast compression levels (default: 1)
+--adapt : dynamically adapt compression level to I/O conditions
+--stream-size=# : optimize compression parameters for streaming input of given number of bytes
+--size-hint=# optimize compression parameters for streaming input of approximately this size
+--target-compressed-block-size=# : make compressed block near targeted size
+ -T#    : spawns # compression threads (default: 1, 0==# cores)
+ -B#    : select size of each job (default: 0==automatic)
+--rsyncable : compress using a rsync-friendly method (-B sets block size)
 --no-dictID : don't write dictID into header (dictionary compression)
 --[no-]check : integrity check (default: enabled)
+--[no-]compress-literals : force (un)compressed literals
  -r     : operate recursively on directories
+--output-dir-flat[=directory]: all resulting files stored into `directory`.
+--format=zstd : compress files to the .zst format (default)
 --format=gzip : compress files to the .gz format
---format=xz : compress files to the .xz format
---format=lzma : compress files to the .lzma format
 --test  : test compressed file integrity
 --[no-]sparse : sparse mode (default: disabled)
  -M#    : Set a memory usage limit for decompression
+--no-progress : do not display the progress bar
 --      : All arguments after "--" are treated as files
 
 Dictionary builder :
 --train ## : create a dictionary from a training set of files
 --train-cover[=k=#,d=#,steps=#,split=#,shrink[=#]] : use the cover algorithm with optional args
---train-fastcover[=k=#,d=#,f=#,steps=#,split=#,shrink[=#],accel=#] : use the fastcover algorithm with optional args
+--train-fastcover[=k=#,d=#,f=#,steps=#,split=#,accel=#,shrink[=#]] : use the fast cover algorithm with optional args
 --train-legacy[=s=#] : use the legacy algorithm with selectivity (default: 9)
  -o file : `file` is dictionary name (default: dictionary)
 --maxdict=# : limit dictionary to specified size (default: 112640)
@@ -172,16 +201,19 @@ Benchmark arguments :
 --priority=rt : set process priority to real-time
 ```
 
-#### Restricted usage of Environment Variables
-Using environment variables to set parameters has security implications.
-Therefore, this avenue is intentionally restricted.
-Only `ZSTD_CLEVEL` is supported currently, for setting compression level.
-`ZSTD_CLEVEL` can be used to set the level between 1 and 19 (the "normal" range).
-If the value of `ZSTD_CLEVEL` is not a valid integer, it will be ignored with a warning message.
-`ZSTD_CLEVEL` just replaces the default compression level (`3`).
-It can be overridden by corresponding command line arguments.
+### Passing parameters through Environment Variables
+`ZSTD_CLEVEL` can be used to modify the default compression level of `zstd`
+(usually set to `3`) to another value between 1 and 19 (the "normal" range).
+This can be useful when `zstd` CLI is invoked in a way that doesn't allow passing arguments.
+One such scenario is `tar --zstd`.
+As `ZSTD_CLEVEL` only replaces the default compression level,
+it can then be overridden by corresponding command line arguments.
+
+There is no "generic" way to pass "any kind of parameter" to `zstd` in a pass-through manner.
+Using environment variables for this purpose has security implications.
+Therefore, this avenue is intentionally restricted and only supports `ZSTD_CLEVEL`.
 
-#### Long distance matching mode
+### Long distance matching mode
 The long distance matching mode, enabled with `--long`, is designed to improve
 the compression ratio for files with long matches at a large distance (up to the
 maximum window size, `128 MiB`) while still maintaining compression speed.
@@ -205,12 +237,12 @@ Compression Speed vs Ratio | Decompression Speed
 
 | Method | Compression ratio | Compression speed | Decompression speed  |
 |:-------|------------------:|-------------------------:|---------------------------:|
-| `zstd -1`   | `5.065`   | `284.8 MB/s`  | `759.3 MB/s`  |
+| `zstd -1`  | `5.065`    | `284.8 MB/s`  | `759.3 MB/s`  |
 | `zstd -5`  | `5.826`    | `124.9 MB/s`  | `674.0 MB/s`  |
 | `zstd -10` | `6.504`    | `29.5 MB/s`   | `771.3 MB/s`  |
 | `zstd -1 --long` | `17.426` | `220.6 MB/s` | `1638.4 MB/s` |
-| `zstd -5 --long` | `19.661` | `165.5 MB/s` | `1530.6 MB/s`|
-| `zstd -10 --long`| `21.949` | `75.6 MB/s` | `1632.6 MB/s`|
+| `zstd -5 --long` | `19.661` | `165.5 MB/s` | `1530.6 MB/s` |
+| `zstd -10 --long`| `21.949` |  `75.6 MB/s` | `1632.6 MB/s` |
 
 On this file, the compression ratio improves significantly with minimal impact
 on compression speed, and the decompression speed doubles.
@@ -233,7 +265,7 @@ The below table illustrates this on the [Silesia compression corpus].
 | `zstd -10 --long`| `3.566` | `16.2 MB/s`       | `415.7 MB/s`   |
 
 
-#### zstdgrep
+### zstdgrep
 
 `zstdgrep` is a utility which makes it possible to `grep` directly a `.zst` compressed file.
 It's used the same way as normal `grep`, for example :
diff --git a/programs/benchfn.c b/programs/benchfn.c
index 2a51a34ff110..ed7273afb6e5 100644
--- a/programs/benchfn.c
+++ b/programs/benchfn.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/programs/benchfn.h b/programs/benchfn.h
index 19e056581a5b..e555bbe6ae32 100644
--- a/programs/benchfn.h
+++ b/programs/benchfn.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/programs/benchzstd.c b/programs/benchzstd.c
index 7439677c7f3e..77056203d55e 100644
--- a/programs/benchzstd.c
+++ b/programs/benchzstd.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -30,13 +30,13 @@
 
 #include "timefn.h"      /* UTIL_time_t */
 #include "benchfn.h"
-#include "mem.h"
+#include "../lib/common/mem.h"
 #define ZSTD_STATIC_LINKING_ONLY
-#include "zstd.h"
+#include "../lib/zstd.h"
 #include "datagen.h"     /* RDG_genBuffer */
-#include "xxhash.h"
+#include "../lib/common/xxhash.h"
 #include "benchzstd.h"
-#include "zstd_errors.h"
+#include "../lib/common/zstd_errors.h"
 
 
 /* *************************************
@@ -375,7 +375,6 @@ BMK_benchMemAdvancedNoAlloc(
                 resPtr += thisBlockSize;
                 remaining -= thisBlockSize;
                 if (adv->mode == BMK_decodeOnly) {
-                    assert(nbBlocks==0);
                     cSizes[nbBlocks] = thisBlockSize;
                     benchResult.cSize = thisBlockSize;
                 }
@@ -483,7 +482,7 @@ BMK_benchMemAdvancedNoAlloc(
                 {   int const ratioAccuracy = (ratio < 10.) ? 3 : 2;
                     DISPLAYLEVEL(2, "%2s-%-17.17s :%10u ->%10u (%5.*f),%6.*f MB/s ,%6.1f MB/s \r",
                             marks[markNb], displayName,
-                            (unsigned)srcSize, (unsigned)benchResult.cSize,
+                            (unsigned)srcSize, (unsigned)cSize,
                             ratioAccuracy, ratio,
                             benchResult.cSpeed < (10 MB) ? 2 : 1, (double)benchResult.cSpeed / MB_UNIT,
                             (double)benchResult.dSpeed / MB_UNIT);
diff --git a/programs/benchzstd.h b/programs/benchzstd.h
index ef7d9fb11145..8c55b3c4f297 100644
--- a/programs/benchzstd.h
+++ b/programs/benchzstd.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -24,7 +24,7 @@ extern "C" {
 /* ===  Dependencies  === */
 #include <stddef.h>   /* size_t */
 #define ZSTD_STATIC_LINKING_ONLY   /* ZSTD_compressionParameters */
-#include "zstd.h"     /* ZSTD_compressionParameters */
+#include "../lib/zstd.h"     /* ZSTD_compressionParameters */
 
 
 /* ===  Constants  === */
diff --git a/programs/datagen.c b/programs/datagen.c
index ead9b2d2415a..4353b7ff9943 100644
--- a/programs/datagen.c
+++ b/programs/datagen.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -18,7 +18,7 @@
 #include <stdlib.h>    /* malloc, free */
 #include <stdio.h>     /* FILE, fwrite, fprintf */
 #include <string.h>    /* memcpy */
-#include "mem.h"       /* U32 */
+#include "../lib/common/mem.h"  /* U32 */
 
 
 /*-************************************
diff --git a/programs/datagen.h b/programs/datagen.h
index 2fcc980e5e76..5a2682d8f9f6 100644
--- a/programs/datagen.h
+++ b/programs/datagen.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/programs/dibio.c b/programs/dibio.c
index ea4bb4bf1f30..463095a8e813 100644
--- a/programs/dibio.c
+++ b/programs/dibio.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -30,8 +30,8 @@
 #include <assert.h>
 
 #include "timefn.h"         /* UTIL_time_t, UTIL_clockSpanMicro, UTIL_getTime */
-#include "mem.h"            /* read */
-#include "error_private.h"
+#include "../lib/common/mem.h"  /* read */
+#include "../lib/common/error_private.h"
 #include "dibio.h"
 
 
diff --git a/programs/dibio.h b/programs/dibio.h
index ea163fe6afd9..682723d6a54b 100644
--- a/programs/dibio.h
+++ b/programs/dibio.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -19,7 +19,7 @@
 *  Dependencies
 ***************************************/
 #define ZDICT_STATIC_LINKING_ONLY
-#include "zdict.h"     /* ZDICT_params_t */
+#include "../lib/dictBuilder/zdict.h"     /* ZDICT_params_t */
 
 
 /*-*************************************
diff --git a/programs/fileio.c b/programs/fileio.c
index 9833767282ee..d72879d64eae 100644
--- a/programs/fileio.c
+++ b/programs/fileio.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -39,12 +39,13 @@
 #  include <io.h>
 #endif
 
-#include "mem.h"       /* U32, U64 */
+#include "../lib/common/mem.h"     /* U32, U64 */
 #include "fileio.h"
 
 #define ZSTD_STATIC_LINKING_ONLY   /* ZSTD_magicNumber, ZSTD_frameHeaderSize_max */
-#include "zstd.h"
-#include "zstd_errors.h"           /* ZSTD_error_frameParameter_windowTooLarge */
+#include "../lib/zstd.h"
+#include "../lib/common/zstd_errors.h"  /* ZSTD_error_frameParameter_windowTooLarge */
+#include "../lib/compress/zstd_compress_internal.h"
 
 #if defined(ZSTD_GZCOMPRESS) || defined(ZSTD_GZDECOMPRESS)
 #  include <zlib.h>
@@ -68,16 +69,11 @@
 /*-*************************************
 *  Constants
 ***************************************/
-#define KB *(1<<10)
-#define MB *(1<<20)
-#define GB *(1U<<30)
-
 #define ADAPT_WINDOWLOG_DEFAULT 23   /* 8 MB */
 #define DICTSIZE_MAX (32 MB)   /* protection against large input (attack scenario) */
 
 #define FNSPACE 30
 
-
 /*-*************************************
 *  Macros
 ***************************************/
@@ -99,7 +95,7 @@ static UTIL_time_t g_displayClock = UTIL_TIME_INITIALIZER;
 #define READY_FOR_UPDATE() (!g_display_prefs.noProgress && UTIL_clockSpanMicro(g_displayClock) > g_refreshRate)
 #define DELAY_NEXT_UPDATE() { g_displayClock = UTIL_getTime(); }
 #define DISPLAYUPDATE(l, ...) {                              \
-        if (g_display_prefs.displayLevel>=l && !g_display_prefs.noProgress) {            \
+        if (g_display_prefs.displayLevel>=l && !g_display_prefs.noProgress) { \
             if (READY_FOR_UPDATE() || (g_display_prefs.displayLevel>=4)) { \
                 DELAY_NEXT_UPDATE();                         \
                 DISPLAY(__VA_ARGS__);                        \
@@ -321,6 +317,8 @@ struct FIO_prefs_s {
     int nbWorkers;
 
     int excludeCompressedFiles;
+    int patchFromMode;
+    int contentSize;
 };
 
 
@@ -487,6 +485,15 @@ void FIO_setLdmHashRateLog(FIO_prefs_t* const prefs, int ldmHashRateLog) {
     prefs->ldmHashRateLog = ldmHashRateLog;
 }
 
+void FIO_setPatchFromMode(FIO_prefs_t* const prefs, int value)
+{
+    prefs->patchFromMode = value != 0;
+}
+
+void FIO_setContentSize(FIO_prefs_t* const prefs, int value)
+{
+    prefs->contentSize = value != 0;
+}
 
 /*-*************************************
 *  Functions
@@ -502,7 +509,7 @@ static int FIO_remove(const char* path)
 #if defined(_WIN32) || defined(WIN32)
     /* windows doesn't allow remove read-only files,
      * so try to make it writable first */
-    chmod(path, _S_IWRITE);
+    UTIL_chmod(path, _S_IWRITE);
 #endif
     return remove(path);
 }
@@ -526,9 +533,7 @@ static FILE* FIO_openSrcFile(const char* srcFileName)
     }
 
     if (!UTIL_isRegularFile(srcFileName)
-#ifndef _MSC_VER
-        && !UTIL_isFIFO(srcFileName)
-#endif /* _MSC_VER */
+     && !UTIL_isFIFO(srcFileName)
     ) {
         DISPLAYLEVEL(1, "zstd: %s is not a regular file -- ignored \n",
                         srcFileName);
@@ -609,8 +614,11 @@ FIO_openDstFile(FIO_prefs_t* const prefs,
     {   FILE* const f = fopen( dstFileName, "wb" );
         if (f == NULL) {
             DISPLAYLEVEL(1, "zstd: %s: %s\n", dstFileName, strerror(errno));
-        } else if(srcFileName != NULL && strcmp (srcFileName, stdinmark)) {
-            chmod(dstFileName, 00600);
+        } else if (srcFileName != NULL
+               && strcmp (srcFileName, stdinmark)
+               && strcmp(dstFileName, nulmark) ) {
+            /* reduce rights on newly created dst file while compression is ongoing */
+            UTIL_chmod(dstFileName, 00600);
         }
         return f;
     }
@@ -623,7 +631,7 @@ FIO_openDstFile(FIO_prefs_t* const prefs,
  * @return : loaded size
  *  if fileName==NULL, returns 0 and a NULL pointer
  */
-static size_t FIO_createDictBuffer(void** bufferPtr, const char* fileName)
+static size_t FIO_createDictBuffer(void** bufferPtr, const char* fileName, FIO_prefs_t* const prefs)
 {
     FILE* fileHandle;
     U64 fileSize;
@@ -637,9 +645,12 @@ static size_t FIO_createDictBuffer(void** bufferPtr, const char* fileName)
     if (fileHandle==NULL) EXM_THROW(31, "%s: %s", fileName, strerror(errno));
 
     fileSize = UTIL_getFileSize(fileName);
-    if (fileSize > DICTSIZE_MAX) {
-        EXM_THROW(32, "Dictionary file %s is too large (> %u MB)",
-                        fileName, DICTSIZE_MAX >> 20);   /* avoid extreme cases */
+    {
+        size_t const dictSizeMax = prefs->patchFromMode ? prefs->memLimit : DICTSIZE_MAX;
+        if (fileSize >  dictSizeMax) {
+            EXM_THROW(32, "Dictionary file %s is too large (> %u bytes)",
+                            fileName,  (unsigned)dictSizeMax);   /* avoid extreme cases */
+        }
     }
     *bufferPtr = malloc((size_t)fileSize);
     if (*bufferPtr==NULL) EXM_THROW(34, "%s", strerror(errno));
@@ -742,6 +753,30 @@ FIO_createFilename_fromOutDir(const char* path, const char* outDirName, const si
     return result;
 }
 
+/* FIO_highbit64() :
+ * gives position of highest bit.
+ * note : only works for v > 0 !
+ */
+static unsigned FIO_highbit64(unsigned long long v)
+{
+    unsigned count = 0;
+    assert(v != 0);
+    v >>= 1;
+    while (v) { v >>= 1; count++; }
+    return count;
+}
+
+static void FIO_adjustMemLimitForPatchFromMode(FIO_prefs_t* const prefs,
+                                    unsigned long long const dictSize,
+                                    unsigned long long const maxSrcFileSize)
+{
+    unsigned long long maxSize = MAX(prefs->memLimit, MAX(dictSize, maxSrcFileSize));
+    assert(maxSize != UTIL_FILESIZE_UNKNOWN);
+    if (maxSize > UINT_MAX)
+        EXM_THROW(42, "Can't handle files larger than %u GB\n", UINT_MAX/(1 GB) + 1);
+    FIO_setMemLimit(prefs, (unsigned)maxSize);
+}
+
 #ifndef ZSTD_NOCOMPRESS
 
 /* **********************************************************************
@@ -754,13 +789,41 @@ typedef struct {
     size_t srcBufferSize;
     void*  dstBuffer;
     size_t dstBufferSize;
+    void* dictBuffer;
+    size_t dictBufferSize;
     const char* dictFileName;
     ZSTD_CStream* cctx;
 } cRess_t;
 
+static void FIO_adjustParamsForPatchFromMode(FIO_prefs_t* const prefs,
+                                    ZSTD_compressionParameters* comprParams,
+                                    unsigned long long const dictSize,
+                                    unsigned long long const maxSrcFileSize,
+                                    int cLevel)
+{
+    unsigned const fileWindowLog = FIO_highbit64(maxSrcFileSize) + 1;
+    ZSTD_compressionParameters const cParams = ZSTD_getCParams(cLevel, (size_t)maxSrcFileSize, (size_t)dictSize);
+    FIO_adjustMemLimitForPatchFromMode(prefs, dictSize, maxSrcFileSize);
+    if (fileWindowLog > ZSTD_WINDOWLOG_MAX)
+        DISPLAYLEVEL(1, "Max window log exceeded by file (compression ratio will suffer)\n");
+    comprParams->windowLog = MIN(ZSTD_WINDOWLOG_MAX, fileWindowLog);
+    if (fileWindowLog > ZSTD_cycleLog(cParams.hashLog, cParams.strategy)) {
+        if (!prefs->ldmFlag)
+            DISPLAYLEVEL(1, "long mode automaticaly triggered\n");
+        FIO_setLdmFlag(prefs, 1);
+    }
+    if (cParams.strategy >= ZSTD_btopt) {
+        DISPLAYLEVEL(1, "[Optimal parser notes] Consider the following to improve patch size at the cost of speed:\n");
+        DISPLAYLEVEL(1, "- Use --single-thread mode in the zstd cli\n");
+        DISPLAYLEVEL(1, "- Set a larger targetLength (eg. --zstd=targetLength=4096)\n");
+        DISPLAYLEVEL(1, "- Set a larger chainLog (eg. --zstd=chainLog=%u)\n", ZSTD_CHAINLOG_MAX);
+        DISPLAYLEVEL(1, "Also consdier playing around with searchLog and hashLog\n");
+    }
+}
+
 static cRess_t FIO_createCResources(FIO_prefs_t* const prefs,
-                                    const char* dictFileName, int cLevel,
-                                    ZSTD_compressionParameters comprParams) {
+                                    const char* dictFileName, unsigned long long const maxSrcFileSize,
+                                    int cLevel, ZSTD_compressionParameters comprParams) {
     cRess_t ress;
     memset(&ress, 0, sizeof(ress));
 
@@ -772,62 +835,68 @@ static cRess_t FIO_createCResources(FIO_prefs_t* const prefs,
     ress.srcBufferSize = ZSTD_CStreamInSize();
     ress.srcBuffer = malloc(ress.srcBufferSize);
     ress.dstBufferSize = ZSTD_CStreamOutSize();
+
+    /* need to update memLimit before calling createDictBuffer
+     * because of memLimit check inside it */
+    if (prefs->patchFromMode)
+        FIO_adjustParamsForPatchFromMode(prefs, &comprParams, UTIL_getFileSize(dictFileName), maxSrcFileSize, cLevel);
     ress.dstBuffer = malloc(ress.dstBufferSize);
+    ress.dictBufferSize = FIO_createDictBuffer(&ress.dictBuffer, dictFileName, prefs);   /* works with dictFileName==NULL */
     if (!ress.srcBuffer || !ress.dstBuffer)
         EXM_THROW(31, "allocation error : not enough memory");
 
     /* Advanced parameters, including dictionary */
-    {   void* dictBuffer;
-        size_t const dictBuffSize = FIO_createDictBuffer(&dictBuffer, dictFileName);   /* works with dictFileName==NULL */
-        if (dictFileName && (dictBuffer==NULL))
-            EXM_THROW(32, "allocation error : can't create dictBuffer");
-        ress.dictFileName = dictFileName;
-
-        if (prefs->adaptiveMode && !prefs->ldmFlag && !comprParams.windowLog)
-            comprParams.windowLog = ADAPT_WINDOWLOG_DEFAULT;
-
-        CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_contentSizeFlag, 1) );  /* always enable content size when available (note: supposed to be default) */
-        CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_dictIDFlag, prefs->dictIDFlag) );
-        CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_checksumFlag, prefs->checksumFlag) );
-        /* compression level */
-        CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_compressionLevel, cLevel) );
-        /* max compressed block size */
-        CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_targetCBlockSize, (int)prefs->targetCBlockSize) );
-        /* source size hint */
-        CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_srcSizeHint, (int)prefs->srcSizeHint) );
-        /* long distance matching */
-        CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_enableLongDistanceMatching, prefs->ldmFlag) );
-        CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_ldmHashLog, prefs->ldmHashLog) );
-        CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_ldmMinMatch, prefs->ldmMinMatch) );
-        if (prefs->ldmBucketSizeLog != FIO_LDM_PARAM_NOTSET) {
-            CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_ldmBucketSizeLog, prefs->ldmBucketSizeLog) );
-        }
-        if (prefs->ldmHashRateLog != FIO_LDM_PARAM_NOTSET) {
-            CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_ldmHashRateLog, prefs->ldmHashRateLog) );
-        }
-        /* compression parameters */
-        CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_windowLog, (int)comprParams.windowLog) );
-        CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_chainLog, (int)comprParams.chainLog) );
-        CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_hashLog, (int)comprParams.hashLog) );
-        CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_searchLog, (int)comprParams.searchLog) );
-        CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_minMatch, (int)comprParams.minMatch) );
-        CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_targetLength, (int)comprParams.targetLength) );
-        CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_strategy, comprParams.strategy) );
-        CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_literalCompressionMode, (int)prefs->literalCompressionMode) );
-        /* multi-threading */
+    if (dictFileName && (ress.dictBuffer==NULL))
+        EXM_THROW(32, "allocation error : can't create dictBuffer");
+    ress.dictFileName = dictFileName;
+
+    if (prefs->adaptiveMode && !prefs->ldmFlag && !comprParams.windowLog)
+        comprParams.windowLog = ADAPT_WINDOWLOG_DEFAULT;
+
+    CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_contentSizeFlag, prefs->contentSize) );  /* always enable content size when available (note: supposed to be default) */
+    CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_dictIDFlag, prefs->dictIDFlag) );
+    CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_checksumFlag, prefs->checksumFlag) );
+    /* compression level */
+    CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_compressionLevel, cLevel) );
+    /* max compressed block size */
+    CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_targetCBlockSize, (int)prefs->targetCBlockSize) );
+    /* source size hint */
+    CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_srcSizeHint, (int)prefs->srcSizeHint) );
+    /* long distance matching */
+    CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_enableLongDistanceMatching, prefs->ldmFlag) );
+    CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_ldmHashLog, prefs->ldmHashLog) );
+    CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_ldmMinMatch, prefs->ldmMinMatch) );
+    if (prefs->ldmBucketSizeLog != FIO_LDM_PARAM_NOTSET) {
+        CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_ldmBucketSizeLog, prefs->ldmBucketSizeLog) );
+    }
+    if (prefs->ldmHashRateLog != FIO_LDM_PARAM_NOTSET) {
+        CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_ldmHashRateLog, prefs->ldmHashRateLog) );
+    }
+    /* compression parameters */
+    CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_windowLog, (int)comprParams.windowLog) );
+    CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_chainLog, (int)comprParams.chainLog) );
+    CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_hashLog, (int)comprParams.hashLog) );
+    CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_searchLog, (int)comprParams.searchLog) );
+    CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_minMatch, (int)comprParams.minMatch) );
+    CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_targetLength, (int)comprParams.targetLength) );
+    CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_strategy, comprParams.strategy) );
+    CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_literalCompressionMode, (int)prefs->literalCompressionMode) );
+    /* multi-threading */
 #ifdef ZSTD_MULTITHREAD
-        DISPLAYLEVEL(5,"set nb workers = %u \n", prefs->nbWorkers);
-        CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_nbWorkers, prefs->nbWorkers) );
-        CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_jobSize, prefs->blockSize) );
-        if (prefs->overlapLog != FIO_OVERLAP_LOG_NOTSET) {
-            DISPLAYLEVEL(3,"set overlapLog = %u \n", prefs->overlapLog);
-            CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_overlapLog, prefs->overlapLog) );
-        }
-        CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_rsyncable, prefs->rsyncable) );
+    DISPLAYLEVEL(5,"set nb workers = %u \n", prefs->nbWorkers);
+    CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_nbWorkers, prefs->nbWorkers) );
+    CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_jobSize, prefs->blockSize) );
+    if (prefs->overlapLog != FIO_OVERLAP_LOG_NOTSET) {
+        DISPLAYLEVEL(3,"set overlapLog = %u \n", prefs->overlapLog);
+        CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_overlapLog, prefs->overlapLog) );
+    }
+    CHECK( ZSTD_CCtx_setParameter(ress.cctx, ZSTD_c_rsyncable, prefs->rsyncable) );
 #endif
-        /* dictionary */
-        CHECK( ZSTD_CCtx_loadDictionary(ress.cctx, dictBuffer, dictBuffSize) );
-        free(dictBuffer);
+    /* dictionary */
+    if (prefs->patchFromMode) {
+        CHECK( ZSTD_CCtx_refPrefix(ress.cctx, ress.dictBuffer, ress.dictBufferSize) );
+    } else {
+        CHECK( ZSTD_CCtx_loadDictionary(ress.cctx, ress.dictBuffer, ress.dictBufferSize) );
     }
 
     return ress;
@@ -837,6 +906,7 @@ static void FIO_freeCResources(cRess_t ress)
 {
     free(ress.srcBuffer);
     free(ress.dstBuffer);
+    free(ress.dictBuffer);
     ZSTD_freeCStream(ress.cctx);   /* never fails */
 }
 
@@ -1352,11 +1422,18 @@ FIO_compressFilename_internal(FIO_prefs_t* const prefs,
 
     /* Status */
     DISPLAYLEVEL(2, "\r%79s\r", "");
-    DISPLAYLEVEL(2,"%-20s :%6.2f%%   (%6llu => %6llu bytes, %s) \n",
-        srcFileName,
-        (double)compressedfilesize / (readsize+(!readsize)/*avoid div by zero*/) * 100,
-        (unsigned long long)readsize, (unsigned long long) compressedfilesize,
-         dstFileName);
+    if (readsize == 0) {
+        DISPLAYLEVEL(2,"%-20s :  (%6llu => %6llu bytes, %s) \n",
+            srcFileName,
+            (unsigned long long)readsize, (unsigned long long) compressedfilesize,
+            dstFileName);
+    } else {
+        DISPLAYLEVEL(2,"%-20s :%6.2f%%   (%6llu => %6llu bytes, %s) \n",
+            srcFileName,
+            (double)compressedfilesize / readsize * 100,
+            (unsigned long long)readsize, (unsigned long long) compressedfilesize,
+            dstFileName);
+    }
 
     /* Elapsed Time and CPU Load */
     {   clock_t const cpuEnd = clock();
@@ -1393,7 +1470,7 @@ static int FIO_compressFilename_dstFile(FIO_prefs_t* const prefs,
     assert(ress.srcFile != NULL);
     if (ress.dstFile == NULL) {
         closeDstFile = 1;
-        DISPLAYLEVEL(6, "FIO_compressFilename_dstFile: opening dst: %s", dstFileName);
+        DISPLAYLEVEL(6, "FIO_compressFilename_dstFile: opening dst: %s \n", dstFileName);
         ress.dstFile = FIO_openDstFile(prefs, srcFileName, dstFileName);
         if (ress.dstFile==NULL) return 1;  /* could not open dstFileName */
         /* Must only be added after FIO_openDstFile() succeeds.
@@ -1415,6 +1492,7 @@ static int FIO_compressFilename_dstFile(FIO_prefs_t* const prefs,
 
         clearHandler();
 
+        DISPLAYLEVEL(6, "FIO_compressFilename_dstFile: closing dst: %s \n", dstFileName);
         if (fclose(dstFile)) { /* error closing dstFile */
             DISPLAYLEVEL(1, "zstd: %s: %s \n", dstFileName, strerror(errno));
             result=1;
@@ -1427,7 +1505,10 @@ static int FIO_compressFilename_dstFile(FIO_prefs_t* const prefs,
         } else if ( strcmp(dstFileName, stdoutmark)
                  && strcmp(dstFileName, nulmark)
                  && transfer_permissions) {
+            DISPLAYLEVEL(6, "FIO_compressFilename_dstFile: transfering permissions into dst: %s \n", dstFileName);
             UTIL_setFileStat(dstFileName, &statbuf);
+        } else {
+            DISPLAYLEVEL(6, "FIO_compressFilename_dstFile: do not transfer permissions into dst: %s \n", dstFileName);
         }
     }
 
@@ -1462,6 +1543,7 @@ FIO_compressFilename_srcFile(FIO_prefs_t* const prefs,
                              int compressionLevel)
 {
     int result;
+    DISPLAYLEVEL(6, "FIO_compressFilename_srcFile: %s \n", srcFileName);
 
     /* ensure src is not a directory */
     if (UTIL_isDirectory(srcFileName)) {
@@ -1507,9 +1589,9 @@ FIO_compressFilename_srcFile(FIO_prefs_t* const prefs,
 
 int FIO_compressFilename(FIO_prefs_t* const prefs, const char* dstFileName,
                          const char* srcFileName, const char* dictFileName,
-                         int compressionLevel,  ZSTD_compressionParameters comprParams)
+                         int compressionLevel, ZSTD_compressionParameters comprParams)
 {
-    cRess_t const ress = FIO_createCResources(prefs, dictFileName, compressionLevel, comprParams);
+    cRess_t const ress = FIO_createCResources(prefs, dictFileName, UTIL_getFileSize(srcFileName), compressionLevel, comprParams);
     int const result = FIO_compressFilename_srcFile(prefs, ress, dstFileName, srcFileName, compressionLevel);
 
 
@@ -1557,6 +1639,16 @@ FIO_determineCompressedName(const char* srcFileName, const char* outDirName, con
     return dstFileNameBuffer;
 }
 
+static unsigned long long FIO_getLargestFileSize(const char** inFileNames, unsigned nbFiles)
+{
+    size_t i;
+    unsigned long long fileSize, maxFileSize = 0;
+    for (i = 0; i < nbFiles; i++) {
+        fileSize = UTIL_getFileSize(inFileNames[i]);
+        maxFileSize = fileSize > maxFileSize ? fileSize : maxFileSize;
+    }
+    return maxFileSize;
+}
 
 /* FIO_compressMultipleFilenames() :
  * compress nbFiles files
@@ -1572,7 +1664,9 @@ int FIO_compressMultipleFilenames(FIO_prefs_t* const prefs,
                                   ZSTD_compressionParameters comprParams)
 {
     int error = 0;
-    cRess_t ress = FIO_createCResources(prefs, dictFileName, compressionLevel, comprParams);
+    cRess_t ress = FIO_createCResources(prefs, dictFileName,
+        FIO_getLargestFileSize(inFileNamesTable, nbFiles),
+        compressionLevel, comprParams);
 
     /* init */
     assert(outFileName != NULL || suffix != NULL);
@@ -1628,6 +1722,9 @@ static dRess_t FIO_createDResources(FIO_prefs_t* const prefs, const char* dictFi
     dRess_t ress;
     memset(&ress, 0, sizeof(ress));
 
+    if (prefs->patchFromMode)
+        FIO_adjustMemLimitForPatchFromMode(prefs, UTIL_getFileSize(dictFileName), 0 /* just use the dict size */);
+
     /* Allocation */
     ress.dctx = ZSTD_createDStream();
     if (ress.dctx==NULL)
@@ -1642,7 +1739,7 @@ static dRess_t FIO_createDResources(FIO_prefs_t* const prefs, const char* dictFi
 
     /* dictionary */
     {   void* dictBuffer;
-        size_t const dictBufferSize = FIO_createDictBuffer(&dictBuffer, dictFileName);
+        size_t const dictBufferSize = FIO_createDictBuffer(&dictBuffer, dictFileName, prefs);
         CHECK( ZSTD_initDStream_usingDict(ress.dctx, dictBuffer, dictBufferSize) );
         free(dictBuffer);
     }
@@ -1659,18 +1756,19 @@ static void FIO_freeDResources(dRess_t ress)
 
 
 /** FIO_fwriteSparse() :
-*   @return : storedSkips, to be provided to next call to FIO_fwriteSparse() of LZ4IO_fwriteSparseEnd() */
+*  @return : storedSkips,
+*            argument for next call to FIO_fwriteSparse() or FIO_fwriteSparseEnd() */
 static unsigned
-FIO_fwriteSparse(const FIO_prefs_t* const prefs,
-                 FILE* file,
+FIO_fwriteSparse(FILE* file,
                  const void* buffer, size_t bufferSize,
+                 const FIO_prefs_t* const prefs,
                  unsigned storedSkips)
 {
     const size_t* const bufferT = (const size_t*)buffer;   /* Buffer is supposed malloc'ed, hence aligned on size_t */
     size_t bufferSizeT = bufferSize / sizeof(size_t);
     const size_t* const bufferTEnd = bufferT + bufferSizeT;
     const size_t* ptrT = bufferT;
-    static const size_t segmentSizeT = (32 KB) / sizeof(size_t);   /* 0-test re-attempted every 32 KB */
+    static const size_t segmentSizeT = (32 KB) / sizeof(size_t);   /* check every 32 KB */
 
     if (prefs->testMode) return 0;  /* do not output anything in test mode */
 
@@ -1684,33 +1782,34 @@ FIO_fwriteSparse(const FIO_prefs_t* const prefs,
 
     /* avoid int overflow */
     if (storedSkips > 1 GB) {
-        int const seekResult = LONG_SEEK(file, 1 GB, SEEK_CUR);
-        if (seekResult != 0)
+        if (LONG_SEEK(file, 1 GB, SEEK_CUR) != 0)
             EXM_THROW(91, "1 GB skip error (sparse file support)");
         storedSkips -= 1 GB;
     }
 
     while (ptrT < bufferTEnd) {
-        size_t seg0SizeT = segmentSizeT;
         size_t nb0T;
 
-        /* count leading zeros */
+        /* adjust last segment if < 32 KB */
+        size_t seg0SizeT = segmentSizeT;
         if (seg0SizeT > bufferSizeT) seg0SizeT = bufferSizeT;
         bufferSizeT -= seg0SizeT;
+
+        /* count leading zeroes */
         for (nb0T=0; (nb0T < seg0SizeT) && (ptrT[nb0T] == 0); nb0T++) ;
         storedSkips += (unsigned)(nb0T * sizeof(size_t));
 
         if (nb0T != seg0SizeT) {   /* not all 0s */
-            int const seekResult = LONG_SEEK(file, storedSkips, SEEK_CUR);
-            if (seekResult) EXM_THROW(92, "Sparse skip error ; try --no-sparse");
+            size_t const nbNon0ST = seg0SizeT - nb0T;
+            /* skip leading zeros */
+            if (LONG_SEEK(file, storedSkips, SEEK_CUR) != 0)
+                EXM_THROW(92, "Sparse skip error ; try --no-sparse");
             storedSkips = 0;
-            seg0SizeT -= nb0T;
-            ptrT += nb0T;
-            {   size_t const sizeCheck = fwrite(ptrT, sizeof(size_t), seg0SizeT, file);
-                if (sizeCheck != seg0SizeT)
-                    EXM_THROW(93, "Write error : cannot write decoded block : %s",
+            /* write the rest */
+            if (fwrite(ptrT + nb0T, sizeof(size_t), nbNon0ST, file) != nbNon0ST)
+                EXM_THROW(93, "Write error : cannot write decoded block : %s",
                             strerror(errno));
-        }   }
+        }
         ptrT += seg0SizeT;
     }
 
@@ -1719,20 +1818,20 @@ FIO_fwriteSparse(const FIO_prefs_t* const prefs,
             /* size not multiple of sizeof(size_t) : implies end of block */
             const char* const restStart = (const char*)bufferTEnd;
             const char* restPtr = restStart;
-            size_t restSize =  bufferSize & maskT;
-            const char* const restEnd = restStart + restSize;
+            const char* const restEnd = (const char*)buffer + bufferSize;
+            assert(restEnd > restStart && restEnd < restStart + sizeof(size_t));
             for ( ; (restPtr < restEnd) && (*restPtr == 0); restPtr++) ;
             storedSkips += (unsigned) (restPtr - restStart);
             if (restPtr != restEnd) {
-                int seekResult = LONG_SEEK(file, storedSkips, SEEK_CUR);
-                if (seekResult)
-                    EXM_THROW(94, "Sparse skip error ; try --no-sparse");
+                /* not all remaining bytes are 0 */
+                size_t const restSize = (size_t)(restEnd - restPtr);
+                if (LONG_SEEK(file, storedSkips, SEEK_CUR) != 0)
+                    EXM_THROW(92, "Sparse skip error ; try --no-sparse");
+                if (fwrite(restPtr, 1, restSize, file) != restSize)
+                    EXM_THROW(95, "Write error : cannot write end of decoded block : %s",
+                        strerror(errno));
                 storedSkips = 0;
-                {   size_t const sizeCheck = fwrite(restPtr, 1, (size_t)(restEnd - restPtr), file);
-                    if (sizeCheck != (size_t)(restEnd - restPtr))
-                        EXM_THROW(95, "Write error : cannot write decoded end of block : %s",
-                            strerror(errno));
-    }   }   }   }
+    }   }   }
 
     return storedSkips;
 }
@@ -1763,7 +1862,7 @@ static int FIO_passThrough(const FIO_prefs_t* const prefs,
                            size_t alreadyLoaded)
 {
     size_t const blockSize = MIN(64 KB, bufferSize);
-    size_t readFromInput = 1;
+    size_t readFromInput;
     unsigned storedSkips = 0;
 
     /* assumption : ress->srcBufferLoaded bytes already loaded and stored within buffer */
@@ -1773,28 +1872,20 @@ static int FIO_passThrough(const FIO_prefs_t* const prefs,
             return 1;
     }   }
 
-    while (readFromInput) {
+    do {
         readFromInput = fread(buffer, 1, blockSize, finput);
-        storedSkips = FIO_fwriteSparse(prefs, foutput, buffer, readFromInput, storedSkips);
+        storedSkips = FIO_fwriteSparse(foutput, buffer, readFromInput, prefs, storedSkips);
+    } while (readFromInput == blockSize);
+    if (ferror(finput)) {
+        DISPLAYLEVEL(1, "Pass-through read error : %s\n", strerror(errno));
+        return 1;
     }
+    assert(feof(finput));
 
     FIO_fwriteSparseEnd(prefs, foutput, storedSkips);
     return 0;
 }
 
-/* FIO_highbit64() :
- * gives position of highest bit.
- * note : only works for v > 0 !
- */
-static unsigned FIO_highbit64(unsigned long long v)
-{
-    unsigned count = 0;
-    assert(v != 0);
-    v >>= 1;
-    while (v) { v >>= 1; count++; }
-    return count;
-}
-
 /* FIO_zstdErrorHelp() :
  * detailed error message when requested window size is too large */
 static void
@@ -1814,34 +1905,36 @@ FIO_zstdErrorHelp(const FIO_prefs_t* const prefs,
         unsigned long long const windowSize = header.windowSize;
         unsigned const windowLog = FIO_highbit64(windowSize) + ((windowSize & (windowSize - 1)) != 0);
         assert(prefs->memLimit > 0);
-        DISPLAYLEVEL(1, "%s : Window size larger than maximum : %llu > %u\n",
+        DISPLAYLEVEL(1, "%s : Window size larger than maximum : %llu > %u \n",
                         srcFileName, windowSize, prefs->memLimit);
         if (windowLog <= ZSTD_WINDOWLOG_MAX) {
             unsigned const windowMB = (unsigned)((windowSize >> 20) + ((windowSize & ((1 MB) - 1)) != 0));
             assert(windowSize < (U64)(1ULL << 52));   /* ensure now overflow for windowMB */
-            DISPLAYLEVEL(1, "%s : Use --long=%u or --memory=%uMB\n",
+            DISPLAYLEVEL(1, "%s : Use --long=%u or --memory=%uMB \n",
                             srcFileName, windowLog, windowMB);
             return;
-        }
-    }
-    DISPLAYLEVEL(1, "%s : Window log larger than ZSTD_WINDOWLOG_MAX=%u; not supported\n",
+    }   }
+    DISPLAYLEVEL(1, "%s : Window log larger than ZSTD_WINDOWLOG_MAX=%u; not supported \n",
                     srcFileName, ZSTD_WINDOWLOG_MAX);
 }
 
 /** FIO_decompressFrame() :
  *  @return : size of decoded zstd frame, or an error code
-*/
+ */
 #define FIO_ERROR_FRAME_DECODING   ((unsigned long long)(-2))
 static unsigned long long
-FIO_decompressZstdFrame(const FIO_prefs_t* const prefs,
-                        dRess_t* ress, FILE* finput,
-                        const char* srcFileName, U64 alreadyDecoded)
+FIO_decompressZstdFrame(dRess_t* ress, FILE* finput,
+                        const FIO_prefs_t* const prefs,
+                        const char* srcFileName,
+                        U64 alreadyDecoded)  /* for multi-frames streams */
 {
     U64 frameSize = 0;
     U32 storedSkips = 0;
 
-    size_t const srcFileLength = strlen(srcFileName);
-    if (srcFileLength>20) srcFileName += srcFileLength-20;  /* display last 20 characters only */
+    /* display last 20 characters only */
+    {   size_t const srcFileLength = strlen(srcFileName);
+        if (srcFileLength>20) srcFileName += srcFileLength-20;
+    }
 
     ZSTD_resetDStream(ress->dctx);
 
@@ -1866,7 +1959,7 @@ FIO_decompressZstdFrame(const FIO_prefs_t* const prefs,
         }
 
         /* Write block */
-        storedSkips = FIO_fwriteSparse(prefs, ress->dstFile, ress->dstBuffer, outBuff.pos, storedSkips);
+        storedSkips = FIO_fwriteSparse(ress->dstFile, ress->dstBuffer, outBuff.pos, prefs, storedSkips);
         frameSize += outBuff.pos;
         DISPLAYUPDATE(2, "\r%-20.20s : %u MB...     ",
                          srcFileName, (unsigned)((alreadyDecoded+frameSize)>>20) );
@@ -1900,8 +1993,8 @@ FIO_decompressZstdFrame(const FIO_prefs_t* const prefs,
 
 #ifdef ZSTD_GZDECOMPRESS
 static unsigned long long
-FIO_decompressGzFrame(const FIO_prefs_t* const prefs,
-                      dRess_t* ress, FILE* srcFile,
+FIO_decompressGzFrame(dRess_t* ress, FILE* srcFile,
+                      const FIO_prefs_t* const prefs,
                       const char* srcFileName)
 {
     unsigned long long outFileSize = 0;
@@ -1943,7 +2036,7 @@ FIO_decompressGzFrame(const FIO_prefs_t* const prefs,
         }
         {   size_t const decompBytes = ress->dstBufferSize - strm.avail_out;
             if (decompBytes) {
-                storedSkips = FIO_fwriteSparse(prefs, ress->dstFile, ress->dstBuffer, decompBytes, storedSkips);
+                storedSkips = FIO_fwriteSparse(ress->dstFile, ress->dstBuffer, decompBytes, prefs, storedSkips);
                 outFileSize += decompBytes;
                 strm.next_out = (Bytef*)ress->dstBuffer;
                 strm.avail_out = (uInt)ress->dstBufferSize;
@@ -1968,8 +2061,8 @@ FIO_decompressGzFrame(const FIO_prefs_t* const prefs,
 
 #ifdef ZSTD_LZMADECOMPRESS
 static unsigned long long
-FIO_decompressLzmaFrame(const FIO_prefs_t* const prefs,
-                        dRess_t* ress, FILE* srcFile,
+FIO_decompressLzmaFrame(dRess_t* ress, FILE* srcFile,
+                        const FIO_prefs_t* const prefs,
                         const char* srcFileName, int plain_lzma)
 {
     unsigned long long outFileSize = 0;
@@ -2020,7 +2113,7 @@ FIO_decompressLzmaFrame(const FIO_prefs_t* const prefs,
         }
         {   size_t const decompBytes = ress->dstBufferSize - strm.avail_out;
             if (decompBytes) {
-                storedSkips = FIO_fwriteSparse(prefs, ress->dstFile, ress->dstBuffer, decompBytes, storedSkips);
+                storedSkips = FIO_fwriteSparse(ress->dstFile, ress->dstBuffer, decompBytes, prefs, storedSkips);
                 outFileSize += decompBytes;
                 strm.next_out = (BYTE*)ress->dstBuffer;
                 strm.avail_out = ress->dstBufferSize;
@@ -2039,8 +2132,8 @@ FIO_decompressLzmaFrame(const FIO_prefs_t* const prefs,
 
 #ifdef ZSTD_LZ4DECOMPRESS
 static unsigned long long
-FIO_decompressLz4Frame(const FIO_prefs_t* const prefs,
-                       dRess_t* ress, FILE* srcFile,
+FIO_decompressLz4Frame(dRess_t* ress, FILE* srcFile,
+                       const FIO_prefs_t* const prefs,
                        const char* srcFileName)
 {
     unsigned long long filesize = 0;
@@ -2092,7 +2185,7 @@ FIO_decompressLz4Frame(const FIO_prefs_t* const prefs,
 
             /* Write Block */
             if (decodedBytes) {
-                storedSkips = FIO_fwriteSparse(prefs, ress->dstFile, ress->dstBuffer, decodedBytes, storedSkips);
+                storedSkips = FIO_fwriteSparse(ress->dstFile, ress->dstBuffer, decodedBytes, prefs, storedSkips);
                 filesize += decodedBytes;
                 DISPLAYUPDATE(2, "\rDecompressed : %u MB  ", (unsigned)(filesize>>20));
             }
@@ -2127,8 +2220,8 @@ FIO_decompressLz4Frame(const FIO_prefs_t* const prefs,
  * @return : 0 : OK
  *           1 : error
  */
-static int FIO_decompressFrames(const FIO_prefs_t* const prefs,
-                                dRess_t ress, FILE* srcFile,
+static int FIO_decompressFrames(dRess_t ress, FILE* srcFile,
+                          const FIO_prefs_t* const prefs,
                           const char* dstFileName, const char* srcFileName)
 {
     unsigned readSomething = 0;
@@ -2156,12 +2249,12 @@ static int FIO_decompressFrames(const FIO_prefs_t* const prefs,
             return 1;
         }
         if (ZSTD_isFrame(buf, ress.srcBufferLoaded)) {
-            unsigned long long const frameSize = FIO_decompressZstdFrame(prefs, &ress, srcFile, srcFileName, filesize);
+            unsigned long long const frameSize = FIO_decompressZstdFrame(&ress, srcFile, prefs, srcFileName, filesize);
             if (frameSize == FIO_ERROR_FRAME_DECODING) return 1;
             filesize += frameSize;
         } else if (buf[0] == 31 && buf[1] == 139) { /* gz magic number */
 #ifdef ZSTD_GZDECOMPRESS
-            unsigned long long const frameSize = FIO_decompressGzFrame(prefs, &ress, srcFile, srcFileName);
+            unsigned long long const frameSize = FIO_decompressGzFrame(&ress, srcFile, prefs, srcFileName);
             if (frameSize == FIO_ERROR_FRAME_DECODING) return 1;
             filesize += frameSize;
 #else
@@ -2171,7 +2264,7 @@ static int FIO_decompressFrames(const FIO_prefs_t* const prefs,
         } else if ((buf[0] == 0xFD && buf[1] == 0x37)  /* xz magic number */
                 || (buf[0] == 0x5D && buf[1] == 0x00)) { /* lzma header (no magic number) */
 #ifdef ZSTD_LZMADECOMPRESS
-            unsigned long long const frameSize = FIO_decompressLzmaFrame(prefs, &ress, srcFile, srcFileName, buf[0] != 0xFD);
+            unsigned long long const frameSize = FIO_decompressLzmaFrame(&ress, srcFile, prefs, srcFileName, buf[0] != 0xFD);
             if (frameSize == FIO_ERROR_FRAME_DECODING) return 1;
             filesize += frameSize;
 #else
@@ -2180,7 +2273,7 @@ static int FIO_decompressFrames(const FIO_prefs_t* const prefs,
 #endif
         } else if (MEM_readLE32(buf) == LZ4_MAGICNUMBER) {
 #ifdef ZSTD_LZ4DECOMPRESS
-            unsigned long long const frameSize = FIO_decompressLz4Frame(prefs, &ress, srcFile, srcFileName);
+            unsigned long long const frameSize = FIO_decompressLz4Frame(&ress, srcFile, prefs, srcFileName);
             if (frameSize == FIO_ERROR_FRAME_DECODING) return 1;
             filesize += frameSize;
 #else
@@ -2237,7 +2330,7 @@ static int FIO_decompressDstFile(FIO_prefs_t* const prefs,
             transfer_permissions = 1;
     }
 
-    result = FIO_decompressFrames(prefs, ress, srcFile, dstFileName, srcFileName);
+    result = FIO_decompressFrames(ress, srcFile, prefs, dstFileName, srcFileName);
 
     if (releaseDstFile) {
         FILE* const dstFile = ress.dstFile;
@@ -2718,7 +2811,7 @@ FIO_listFile(fileInfo_t* total, const char* inFileName, int displayLevel)
         displayInfo(inFileName, &info, displayLevel);
         *total = FIO_addFInfo(*total, info);
         assert(error == info_success || error == info_frame_error);
-        return error;
+        return (int)error;
     }
 }
 
diff --git a/programs/fileio.h b/programs/fileio.h
index a7da089f67da..2fbf01f82824 100644
--- a/programs/fileio.h
+++ b/programs/fileio.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -13,7 +13,7 @@
 #define FILEIO_H_23981798732
 
 #define ZSTD_STATIC_LINKING_ONLY   /* ZSTD_compressionParameters */
-#include "zstd.h"                  /* ZSTD_* */
+#include "../lib/zstd.h"           /* ZSTD_* */
 
 #if defined (__cplusplus)
 extern "C" {
@@ -94,6 +94,8 @@ void FIO_setLiteralCompressionMode(
 void FIO_setNoProgress(unsigned noProgress);
 void FIO_setNotificationLevel(int level);
 void FIO_setExcludeCompressedFile(FIO_prefs_t* const prefs, int excludeCompressedFiles);
+void FIO_setPatchFromMode(FIO_prefs_t* const prefs, int value);
+void FIO_setContentSize(FIO_prefs_t* const prefs, int value);
 
 /*-*************************************
 *  Single File functions
diff --git a/programs/platform.h b/programs/platform.h
index 5934e59cf12d..2b4b9f2d8677 100644
--- a/programs/platform.h
+++ b/programs/platform.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -90,7 +90,7 @@ extern "C" {
      && ( defined(__unix__) || defined(__unix) \
        || defined(__midipix__) || defined(__VMS) || defined(__HAIKU__) )
 
-#    if defined(__linux__) || defined(__linux)
+#    if defined(__linux__) || defined(__linux) || defined(__CYGWIN__)
 #      ifndef _POSIX_C_SOURCE
 #        define _POSIX_C_SOURCE 200809L  /* feature test macro : https://www.gnu.org/software/libc/manual/html_node/Feature-Test-Macros.html */
 #      endif
@@ -109,16 +109,25 @@ extern "C" {
 #endif   /* PLATFORM_POSIX_VERSION */
 
 
+#if PLATFORM_POSIX_VERSION > 1
+   /* glibc < 2.26 may not expose struct timespec def without this.
+    * See issue #1920. */
+#  ifndef _ATFILE_SOURCE
+#    define _ATFILE_SOURCE
+#  endif
+#endif
+
+
 /*-*********************************************
 *  Detect if isatty() and fileno() are available
 ************************************************/
 #if (defined(__linux__) && (PLATFORM_POSIX_VERSION > 1)) \
  || (PLATFORM_POSIX_VERSION >= 200112L) \
- || defined(__DJGPP__) \
- || defined(__MSYS__)
+ || defined(__DJGPP__)
 #  include <unistd.h>   /* isatty */
+#  include <stdio.h>    /* fileno */
 #  define IS_CONSOLE(stdStream) isatty(fileno(stdStream))
-#elif defined(MSDOS) || defined(OS2) || defined(__CYGWIN__)
+#elif defined(MSDOS) || defined(OS2)
 #  include <io.h>       /* _isatty */
 #  define IS_CONSOLE(stdStream) _isatty(_fileno(stdStream))
 #elif defined(WIN32) || defined(_WIN32)
diff --git a/programs/timefn.c b/programs/timefn.c
index 096e1910bf4b..95460d0d971d 100644
--- a/programs/timefn.c
+++ b/programs/timefn.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2019-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2019-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -82,9 +82,10 @@ PTime UTIL_getSpanTimeNano(UTIL_time_t clockStart, UTIL_time_t clockEnd)
 }
 
 
-
+/* C11 requires timespec_get, but FreeBSD 11 lacks it, while still claiming C11 compliance.
+   Android also lacks it but does define TIME_UTC. */
 #elif (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 201112L) /* C11 */) \
-    && defined(TIME_UTC) /* C11 requires timespec_get, but FreeBSD 11 lacks it, while still claiming C11 compliance */
+    && defined(TIME_UTC) && !defined(__ANDROID__)
 
 #include <stdlib.h>   /* abort */
 #include <stdio.h>    /* perror */
diff --git a/programs/timefn.h b/programs/timefn.h
index 2db3765b9308..eb3c130934eb 100644
--- a/programs/timefn.h
+++ b/programs/timefn.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -41,7 +41,7 @@ extern "C" {
 ******************************************/
 #if defined(_WIN32)   /* Windows */
 
-    #include <Windows.h>   /* LARGE_INTEGER */
+    #include <windows.h>   /* LARGE_INTEGER */
     typedef LARGE_INTEGER UTIL_time_t;
     #define UTIL_TIME_INITIALIZER { { 0, 0 } }
 
@@ -51,8 +51,10 @@ extern "C" {
     typedef PTime UTIL_time_t;
     #define UTIL_TIME_INITIALIZER 0
 
+/* C11 requires timespec_get, but FreeBSD 11 lacks it, while still claiming C11 compliance.
+   Android also lacks it but does define TIME_UTC. */
 #elif (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 201112L) /* C11 */) \
-    && defined(TIME_UTC) /* C11 requires timespec_get, but FreeBSD 11 lacks it, while still claiming C11 compliance */
+    && defined(TIME_UTC) && !defined(__ANDROID__)
 
     typedef struct timespec UTIL_time_t;
     #define UTIL_TIME_INITIALIZER { 0, 0 }
diff --git a/programs/util.c b/programs/util.c
index 5d15450d2e13..ab1abd3b1862 100644
--- a/programs/util.c
+++ b/programs/util.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -17,13 +17,89 @@ extern "C" {
 *  Dependencies
 ******************************************/
 #include "util.h"       /* note : ensure that platform.h is included first ! */
+#include <stdlib.h>     /* malloc, realloc, free */
+#include <stdio.h>      /* fprintf */
+#include <time.h>       /* clock_t, clock, CLOCKS_PER_SEC, nanosleep */
 #include <errno.h>
 #include <assert.h>
 
+#if defined(_WIN32)
+#  include <sys/utime.h>  /* utime */
+#  include <io.h>         /* _chmod */
+#else
+#  include <unistd.h>     /* chown, stat */
+#  if PLATFORM_POSIX_VERSION < 200809L
+#    include <utime.h>    /* utime */
+#  else
+#    include <fcntl.h>    /* AT_FDCWD */
+#    include <sys/stat.h> /* utimensat */
+#  endif
+#endif
+
 #if defined(_MSC_VER) || defined(__MINGW32__) || defined (__MSVCRT__)
 #include <direct.h>     /* needed for _mkdir in windows */
 #endif
 
+#if defined(__linux__) || (PLATFORM_POSIX_VERSION >= 200112L)  /* opendir, readdir require POSIX.1-2001 */
+#  include <dirent.h>       /* opendir, readdir */
+#  include <string.h>       /* strerror, memcpy */
+#endif /* #ifdef _WIN32 */
+
+
+/*-****************************************
+*  Internal Macros
+******************************************/
+
+/* CONTROL is almost like an assert(), but is never disabled.
+ * It's designed for failures that may happen rarely,
+ * but we don't want to maintain a specific error code path for them,
+ * such as a malloc() returning NULL for example.
+ * Since it's always active, this macro can trigger side effects.
+ */
+#define CONTROL(c)  {         \
+    if (!(c)) {               \
+        UTIL_DISPLAYLEVEL(1, "Error : %s, %i : %s",  \
+                          __FILE__, __LINE__, #c);   \
+        exit(1);              \
+}   }
+
+/* console log */
+#define UTIL_DISPLAY(...)         fprintf(stderr, __VA_ARGS__)
+#define UTIL_DISPLAYLEVEL(l, ...) { if (g_utilDisplayLevel>=l) { UTIL_DISPLAY(__VA_ARGS__); } }
+
+/* A modified version of realloc().
+ * If UTIL_realloc() fails the original block is freed.
+ */
+UTIL_STATIC void* UTIL_realloc(void *ptr, size_t size)
+{
+    void *newptr = realloc(ptr, size);
+    if (newptr) return newptr;
+    free(ptr);
+    return NULL;
+}
+
+#if defined(_MSC_VER)
+    #define chmod _chmod
+#endif
+
+
+/*-****************************************
+*  Console log
+******************************************/
+int g_utilDisplayLevel;
+
+
+/*-*************************************
+*  Constants
+***************************************/
+#define LIST_SIZE_INCREASE   (8*1024)
+#define MAX_FILE_OF_FILE_NAMES_SIZE (1<<20)*50
+
+
+/*-*************************************
+*  Functions
+***************************************/
+
 int UTIL_fileExist(const char* filename)
 {
     stat_t statbuf;
@@ -54,6 +130,13 @@ int UTIL_getFileStat(const char* infilename, stat_t *statbuf)
     return 1;
 }
 
+/* like chmod, but avoid changing permission of /dev/null */
+int UTIL_chmod(char const* filename, mode_t permissions)
+{
+    if (!strcmp(filename, "/dev/null")) return 0;   /* pretend success, but don't change anything */
+    return chmod(filename, permissions);
+}
+
 int UTIL_setFileStat(const char *filename, stat_t *statbuf)
 {
     int res = 0;
@@ -62,41 +145,44 @@ int UTIL_setFileStat(const char *filename, stat_t *statbuf)
         return -1;
 
     /* set access and modification times */
-#if defined(_WIN32) || (PLATFORM_POSIX_VERSION < 200809L)
-    {
-        struct utimbuf timebuf;
-        timebuf.actime = time(NULL);
-        timebuf.modtime = statbuf->st_mtime;
-        res += utime(filename, &timebuf);
-    }
-#else
+    /* We check that st_mtime is a macro here in order to give us confidence
+     * that struct stat has a struct timespec st_mtim member. We need this
+     * check because there are some platforms that claim to be POSIX 2008
+     * compliant but which do not have st_mtim... */
+#if (PLATFORM_POSIX_VERSION >= 200809L) && defined(st_mtime)
     {
         /* (atime, mtime) */
         struct timespec timebuf[2] = { {0, UTIME_NOW} };
         timebuf[1] = statbuf->st_mtim;
         res += utimensat(AT_FDCWD, filename, timebuf, 0);
     }
+#else
+    {
+        struct utimbuf timebuf;
+        timebuf.actime = time(NULL);
+        timebuf.modtime = statbuf->st_mtime;
+        res += utime(filename, &timebuf);
+    }
 #endif
 
 #if !defined(_WIN32)
     res += chown(filename, statbuf->st_uid, statbuf->st_gid);  /* Copy ownership */
 #endif
 
-    res += chmod(filename, statbuf->st_mode & 07777);  /* Copy file permissions */
+    res += UTIL_chmod(filename, statbuf->st_mode & 07777);  /* Copy file permissions */
 
     errno = 0;
     return -res; /* number of errors is returned */
 }
 
-U32 UTIL_isDirectory(const char* infilename)
+int UTIL_isDirectory(const char* infilename)
 {
-    int r;
     stat_t statbuf;
 #if defined(_MSC_VER)
-    r = _stat64(infilename, &statbuf);
+    int const r = _stat64(infilename, &statbuf);
     if (!r && (statbuf.st_mode & _S_IFDIR)) return 1;
 #else
-    r = stat(infilename, &statbuf);
+    int const r = stat(infilename, &statbuf);
     if (!r && S_ISDIR(statbuf.st_mode)) return 1;
 #endif
     return 0;
@@ -126,28 +212,25 @@ int UTIL_isSameFile(const char* fName1, const char* fName2)
 #endif
 }
 
-#ifndef _MSC_VER
-/* Using this to distinguish named pipes */
-U32 UTIL_isFIFO(const char* infilename)
+/* UTIL_isFIFO : distinguish named pipes */
+int UTIL_isFIFO(const char* infilename)
 {
 /* macro guards, as defined in : https://linux.die.net/man/2/lstat */
 #if PLATFORM_POSIX_VERSION >= 200112L
     stat_t statbuf;
-    int r = UTIL_getFileStat(infilename, &statbuf);
+    int const r = UTIL_getFileStat(infilename, &statbuf);
     if (!r && S_ISFIFO(statbuf.st_mode)) return 1;
 #endif
     (void)infilename;
     return 0;
 }
-#endif
 
-U32 UTIL_isLink(const char* infilename)
+int UTIL_isLink(const char* infilename)
 {
 /* macro guards, as defined in : https://linux.die.net/man/2/lstat */
 #if PLATFORM_POSIX_VERSION >= 200112L
-    int r;
     stat_t statbuf;
-    r = lstat(infilename, &statbuf);
+    int const r = lstat(infilename, &statbuf);
     if (!r && S_ISLNK(statbuf.st_mode)) return 1;
 #endif
     (void)infilename;
@@ -176,28 +259,226 @@ U64 UTIL_getFileSize(const char* infilename)
 }
 
 
-U64 UTIL_getTotalFileSize(const char* const * const fileNamesTable, unsigned nbFiles)
+U64 UTIL_getTotalFileSize(const char* const * fileNamesTable, unsigned nbFiles)
 {
     U64 total = 0;
-    int error = 0;
     unsigned n;
     for (n=0; n<nbFiles; n++) {
         U64 const size = UTIL_getFileSize(fileNamesTable[n]);
-        error |= (size == UTIL_FILESIZE_UNKNOWN);
+        if (size == UTIL_FILESIZE_UNKNOWN) return UTIL_FILESIZE_UNKNOWN;
         total += size;
     }
-    return error ? UTIL_FILESIZE_UNKNOWN : total;
+    return total;
+}
+
+
+/* condition : @file must be valid, and not have reached its end.
+ * @return : length of line written into @buf, ended with `\0` instead of '\n',
+ *           or 0, if there is no new line */
+static size_t readLineFromFile(char* buf, size_t len, FILE* file)
+{
+    assert(!feof(file));
+    /* Work around Cygwin problem when len == 1 it returns NULL. */
+    if (len <= 1) return 0;
+    CONTROL( fgets(buf, (int) len, file) );
+    {   size_t linelen = strlen(buf);
+        if (strlen(buf)==0) return 0;
+        if (buf[linelen-1] == '\n') linelen--;
+        buf[linelen] = '\0';
+        return linelen+1;
+    }
+}
+
+/* Conditions :
+ *   size of @inputFileName file must be < @dstCapacity
+ *   @dst must be initialized
+ * @return : nb of lines
+ *       or -1 if there's an error
+ */
+static int
+readLinesFromFile(void* dst, size_t dstCapacity,
+            const char* inputFileName)
+{
+    int nbFiles = 0;
+    size_t pos = 0;
+    char* const buf = (char*)dst;
+    FILE* const inputFile = fopen(inputFileName, "r");
+
+    assert(dst != NULL);
+
+    if(!inputFile) {
+        if (g_utilDisplayLevel >= 1) perror("zstd:util:readLinesFromFile");
+        return -1;
+    }
+
+    while ( !feof(inputFile) ) {
+        size_t const lineLength = readLineFromFile(buf+pos, dstCapacity-pos, inputFile);
+        if (lineLength == 0) break;
+        assert(pos + lineLength < dstCapacity);
+        pos += lineLength;
+        ++nbFiles;
+    }
+
+    CONTROL( fclose(inputFile) == 0 );
+
+    return nbFiles;
+}
+
+/*Note: buf is not freed in case function successfully created table because filesTable->fileNames[0] = buf*/
+FileNamesTable*
+UTIL_createFileNamesTable_fromFileName(const char* inputFileName)
+{
+    size_t nbFiles = 0;
+    char* buf;
+    size_t bufSize;
+    size_t pos = 0;
+
+    if (!UTIL_fileExist(inputFileName) || !UTIL_isRegularFile(inputFileName))
+        return NULL;
+
+    {   U64 const inputFileSize = UTIL_getFileSize(inputFileName);
+        if(inputFileSize > MAX_FILE_OF_FILE_NAMES_SIZE)
+            return NULL;
+        bufSize = (size_t)(inputFileSize + 1); /* (+1) to add '\0' at the end of last filename */
+    }
+
+    buf = (char*) malloc(bufSize);
+    CONTROL( buf != NULL );
+
+    {   int const ret_nbFiles = readLinesFromFile(buf, bufSize, inputFileName);
+
+        if (ret_nbFiles <= 0) {
+          free(buf);
+          return NULL;
+        }
+        nbFiles = (size_t)ret_nbFiles;
+    }
+
+    {   const char** filenamesTable = (const char**) malloc(nbFiles * sizeof(*filenamesTable));
+        CONTROL(filenamesTable != NULL);
+
+        {   size_t fnb;
+            for (fnb = 0, pos = 0; fnb < nbFiles; fnb++) {
+                filenamesTable[fnb] = buf+pos;
+                pos += strlen(buf+pos)+1;  /* +1 for the finishing `\0` */
+        }   }
+        assert(pos <= bufSize);
+
+        return UTIL_assembleFileNamesTable(filenamesTable, nbFiles, buf);
+    }
+}
+
+static FileNamesTable*
+UTIL_assembleFileNamesTable2(const char** filenames, size_t tableSize, size_t tableCapacity, char* buf)
+{
+    FileNamesTable* const table = (FileNamesTable*) malloc(sizeof(*table));
+    CONTROL(table != NULL);
+    table->fileNames = filenames;
+    table->buf = buf;
+    table->tableSize = tableSize;
+    table->tableCapacity = tableCapacity;
+    return table;
+}
+
+FileNamesTable*
+UTIL_assembleFileNamesTable(const char** filenames, size_t tableSize, char* buf)
+{
+    return UTIL_assembleFileNamesTable2(filenames, tableSize, tableSize, buf);
+}
+
+void UTIL_freeFileNamesTable(FileNamesTable* table)
+{
+    if (table==NULL) return;
+    free((void*)table->fileNames);
+    free(table->buf);
+    free(table);
+}
+
+FileNamesTable* UTIL_allocateFileNamesTable(size_t tableSize)
+{
+    const char** const fnTable = (const char**)malloc(tableSize * sizeof(*fnTable));
+    FileNamesTable* fnt;
+    if (fnTable==NULL) return NULL;
+    fnt = UTIL_assembleFileNamesTable(fnTable, tableSize, NULL);
+    fnt->tableSize = 0;   /* the table is empty */
+    return fnt;
+}
+
+void UTIL_refFilename(FileNamesTable* fnt, const char* filename)
+{
+    assert(fnt->tableSize < fnt->tableCapacity);
+    fnt->fileNames[fnt->tableSize] = filename;
+    fnt->tableSize++;
+}
+
+static size_t getTotalTableSize(FileNamesTable* table)
+{
+    size_t fnb = 0, totalSize = 0;
+    for(fnb = 0 ; fnb < table->tableSize && table->fileNames[fnb] ; ++fnb) {
+        totalSize += strlen(table->fileNames[fnb]) + 1; /* +1 to add '\0' at the end of each fileName */
+    }
+    return totalSize;
+}
+
+FileNamesTable*
+UTIL_mergeFileNamesTable(FileNamesTable* table1, FileNamesTable* table2)
+{
+    unsigned newTableIdx = 0;
+    size_t pos = 0;
+    size_t newTotalTableSize;
+    char* buf;
+
+    FileNamesTable* const newTable = UTIL_assembleFileNamesTable(NULL, 0, NULL);
+    CONTROL( newTable != NULL );
+
+    newTotalTableSize = getTotalTableSize(table1) + getTotalTableSize(table2);
+
+    buf = (char*) calloc(newTotalTableSize, sizeof(*buf));
+    CONTROL ( buf != NULL );
+
+    newTable->buf = buf;
+    newTable->tableSize = table1->tableSize + table2->tableSize;
+    newTable->fileNames = (const char **) calloc(newTable->tableSize, sizeof(*(newTable->fileNames)));
+    CONTROL ( newTable->fileNames != NULL );
+
+    {   unsigned idx1;
+        for( idx1=0 ; (idx1 < table1->tableSize) && table1->fileNames[idx1] && (pos < newTotalTableSize); ++idx1, ++newTableIdx) {
+            size_t const curLen = strlen(table1->fileNames[idx1]);
+            memcpy(buf+pos, table1->fileNames[idx1], curLen);
+            assert(newTableIdx <= newTable->tableSize);
+            newTable->fileNames[newTableIdx] = buf+pos;
+            pos += curLen+1;
+    }   }
+
+    {   unsigned idx2;
+        for( idx2=0 ; (idx2 < table2->tableSize) && table2->fileNames[idx2] && (pos < newTotalTableSize) ; ++idx2, ++newTableIdx) {
+            size_t const curLen = strlen(table2->fileNames[idx2]);
+            memcpy(buf+pos, table2->fileNames[idx2], curLen);
+            assert(newTableIdx <= newTable->tableSize);
+            newTable->fileNames[newTableIdx] = buf+pos;
+            pos += curLen+1;
+    }   }
+    assert(pos <= newTotalTableSize);
+    newTable->tableSize = newTableIdx;
+
+    UTIL_freeFileNamesTable(table1);
+    UTIL_freeFileNamesTable(table2);
+
+    return newTable;
 }
 
 #ifdef _WIN32
-int UTIL_prepareFileList(const char *dirName, char** bufStart, size_t* pos, char** bufEnd, int followLinks)
+static int UTIL_prepareFileList(const char* dirName,
+                                char** bufStart, size_t* pos,
+                                char** bufEnd, int followLinks)
 {
     char* path;
-    int dirLength, fnameLength, pathLength, nbFiles = 0;
+    size_t dirLength, pathLength;
+    int nbFiles = 0;
     WIN32_FIND_DATAA cFile;
     HANDLE hFile;
 
-    dirLength = (int)strlen(dirName);
+    dirLength = strlen(dirName);
     path = (char*) malloc(dirLength + 3);
     if (!path) return 0;
 
@@ -214,7 +495,7 @@ int UTIL_prepareFileList(const char *dirName, char** bufStart, size_t* pos, char
     free(path);
 
     do {
-        fnameLength = (int)strlen(cFile.cFileName);
+        size_t const fnameLength = strlen(cFile.cFileName);
         path = (char*) malloc(dirLength + fnameLength + 2);
         if (!path) { FindClose(hFile); return 0; }
         memcpy(path, dirName, dirLength);
@@ -242,8 +523,7 @@ int UTIL_prepareFileList(const char *dirName, char** bufStart, size_t* pos, char
                 memcpy(*bufStart + *pos, path, pathLength+1 /* include final \0 */);
                 *pos += pathLength + 1;
                 nbFiles++;
-            }
-        }
+        }   }
         free(path);
     } while (FindNextFileA(hFile, &cFile));
 
@@ -253,12 +533,13 @@ int UTIL_prepareFileList(const char *dirName, char** bufStart, size_t* pos, char
 
 #elif defined(__linux__) || (PLATFORM_POSIX_VERSION >= 200112L)  /* opendir, readdir require POSIX.1-2001 */
 
-int UTIL_prepareFileList(const char *dirName, char** bufStart, size_t* pos, char** bufEnd, int followLinks)
+static int UTIL_prepareFileList(const char *dirName,
+                                char** bufStart, size_t* pos,
+                                char** bufEnd, int followLinks)
 {
-    DIR *dir;
-    struct dirent *entry;
-    char* path;
-    size_t dirLength, fnameLength, pathLength;
+    DIR* dir;
+    struct dirent * entry;
+    size_t dirLength;
     int nbFiles = 0;
 
     if (!(dir = opendir(dirName))) {
@@ -269,6 +550,8 @@ int UTIL_prepareFileList(const char *dirName, char** bufStart, size_t* pos, char
     dirLength = strlen(dirName);
     errno = 0;
     while ((entry = readdir(dir)) != NULL) {
+        char* path;
+        size_t fnameLength, pathLength;
         if (strcmp (entry->d_name, "..") == 0 ||
             strcmp (entry->d_name, ".") == 0) continue;
         fnameLength = strlen(entry->d_name);
@@ -302,14 +585,13 @@ int UTIL_prepareFileList(const char *dirName, char** bufStart, size_t* pos, char
                 memcpy(*bufStart + *pos, path, pathLength + 1);  /* with final \0 */
                 *pos += pathLength + 1;
                 nbFiles++;
-            }
-        }
+        }   }
         free(path);
         errno = 0; /* clear errno after UTIL_isDirectory, UTIL_prepareFileList */
     }
 
     if (errno != 0) {
-        UTIL_DISPLAYLEVEL(1, "readdir(%s) error: %s\n", dirName, strerror(errno));
+        UTIL_DISPLAYLEVEL(1, "readdir(%s) error: %s \n", dirName, strerror(errno));
         free(*bufStart);
         *bufStart = NULL;
     }
@@ -319,10 +601,12 @@ int UTIL_prepareFileList(const char *dirName, char** bufStart, size_t* pos, char
 
 #else
 
-int UTIL_prepareFileList(const char *dirName, char** bufStart, size_t* pos, char** bufEnd, int followLinks)
+static int UTIL_prepareFileList(const char *dirName,
+                                char** bufStart, size_t* pos,
+                                char** bufEnd, int followLinks)
 {
     (void)bufStart; (void)bufEnd; (void)pos; (void)followLinks;
-    UTIL_DISPLAYLEVEL(1, "Directory %s ignored (compiled without _WIN32 or _POSIX_C_SOURCE)\n", dirName);
+    UTIL_DISPLAYLEVEL(1, "Directory %s ignored (compiled without _WIN32 or _POSIX_C_SOURCE) \n", dirName);
     return 0;
 }
 
@@ -349,68 +633,70 @@ const char* UTIL_getFileExtension(const char* infilename)
    return extension;
 }
 
-/*
- * UTIL_createFileList - takes a list of files and directories (params: inputNames, inputNamesNb), scans directories,
- *                       and returns a new list of files (params: return value, allocatedBuffer, allocatedNamesNb).
- * After finishing usage of the list the structures should be freed with UTIL_freeFileList(params: return value, allocatedBuffer)
- * In case of error UTIL_createFileList returns NULL and UTIL_freeFileList should not be called.
- */
-const char**
-UTIL_createFileList(const char **inputNames, unsigned inputNamesNb,
-                    char** allocatedBuffer, unsigned* allocatedNamesNb,
-                    int followLinks)
+
+FileNamesTable*
+UTIL_createExpandedFNT(const char** inputNames, size_t nbIfns, int followLinks)
 {
-    size_t pos;
-    unsigned i, nbFiles;
+    unsigned nbFiles;
     char* buf = (char*)malloc(LIST_SIZE_INCREASE);
     char* bufend = buf + LIST_SIZE_INCREASE;
 
     if (!buf) return NULL;
 
-    for (i=0, pos=0, nbFiles=0; i<inputNamesNb; i++) {
-        if (!UTIL_isDirectory(inputNames[i])) {
-            size_t const len = strlen(inputNames[i]);
-            if (buf + pos + len >= bufend) {
-                ptrdiff_t newListSize = (bufend - buf) + LIST_SIZE_INCREASE;
-                assert(newListSize >= 0);
-                buf = (char*)UTIL_realloc(buf, (size_t)newListSize);
-                bufend = buf + newListSize;
-                if (!buf) return NULL;
-            }
-            if (buf + pos + len < bufend) {
-                memcpy(buf+pos, inputNames[i], len+1);  /* including final \0 */
-                pos += len + 1;
-                nbFiles++;
-            }
-        } else {
-            nbFiles += (unsigned)UTIL_prepareFileList(inputNames[i], &buf, &pos, &bufend, followLinks);
-            if (buf == NULL) return NULL;
-    }   }
+    {   size_t ifnNb, pos;
+        for (ifnNb=0, pos=0, nbFiles=0; ifnNb<nbIfns; ifnNb++) {
+            if (!UTIL_isDirectory(inputNames[ifnNb])) {
+                size_t const len = strlen(inputNames[ifnNb]);
+                if (buf + pos + len >= bufend) {
+                    ptrdiff_t newListSize = (bufend - buf) + LIST_SIZE_INCREASE;
+                    assert(newListSize >= 0);
+                    buf = (char*)UTIL_realloc(buf, (size_t)newListSize);
+                    if (!buf) return NULL;
+                    bufend = buf + newListSize;
+                }
+                if (buf + pos + len < bufend) {
+                    memcpy(buf+pos, inputNames[ifnNb], len+1);  /* including final \0 */
+                    pos += len + 1;
+                    nbFiles++;
+                }
+            } else {
+                nbFiles += (unsigned)UTIL_prepareFileList(inputNames[ifnNb], &buf, &pos, &bufend, followLinks);
+                if (buf == NULL) return NULL;
+    }   }   }
 
-    if (nbFiles == 0) { free(buf); return NULL; }
+    /* note : even if nbFiles==0, function returns a valid, though empty, FileNamesTable* object */
 
-    {   const char** const fileTable = (const char**)malloc((nbFiles + 1) * sizeof(*fileTable));
-        if (!fileTable) { free(buf); return NULL; }
+    {   size_t ifnNb, pos;
+        size_t const fntCapacity = nbFiles + 1;  /* minimum 1, allows adding one reference, typically stdin */
+        const char** const fileNamesTable = (const char**)malloc(fntCapacity * sizeof(*fileNamesTable));
+        if (!fileNamesTable) { free(buf); return NULL; }
 
-        for (i = 0, pos = 0; i < nbFiles; i++) {
-            fileTable[i] = buf + pos;
-            if (buf + pos > bufend) { free(buf); free((void*)fileTable); return NULL; }
-            pos += strlen(fileTable[i]) + 1;
+        for (ifnNb = 0, pos = 0; ifnNb < nbFiles; ifnNb++) {
+            fileNamesTable[ifnNb] = buf + pos;
+            if (buf + pos > bufend) { free(buf); free((void*)fileNamesTable); return NULL; }
+            pos += strlen(fileNamesTable[ifnNb]) + 1;
         }
-
-        *allocatedBuffer = buf;
-        *allocatedNamesNb = nbFiles;
-
-        return fileTable;
+        return UTIL_assembleFileNamesTable2(fileNamesTable, nbFiles, fntCapacity, buf);
     }
 }
 
 
-/*-****************************************
-*  Console log
-******************************************/
-int g_utilDisplayLevel;
+void UTIL_expandFNT(FileNamesTable** fnt, int followLinks)
+{
+    FileNamesTable* const newFNT = UTIL_createExpandedFNT((*fnt)->fileNames, (*fnt)->tableSize, followLinks);
+    CONTROL(newFNT != NULL);
+    UTIL_freeFileNamesTable(*fnt);
+    *fnt = newFNT;
+}
 
+FileNamesTable* UTIL_createFNT_fromROTable(const char** filenames, size_t nbFilenames)
+{
+    size_t const sizeof_FNTable = nbFilenames * sizeof(*filenames);
+    const char** const newFNTable = (const char**)malloc(sizeof_FNTable);
+    if (newFNTable==NULL) return NULL;
+    memcpy((void*)newFNTable, filenames, sizeof_FNTable);  /* void* : mitigate a Visual compiler bug or limitation */
+    return UTIL_assembleFileNamesTable(newFNTable, nbFilenames, NULL);
+}
 
 
 /*-****************************************
@@ -465,8 +751,7 @@ int UTIL_countPhysicalCores(void)
                 }
             } else {
                 done = TRUE;
-            }
-        }
+        }   }
 
         ptr = buffer;
 
@@ -578,8 +863,7 @@ int UTIL_countPhysicalCores(void)
             } else if (ferror(cpuinfo)) {
                 /* fall back on the sysconf value */
                 goto failed;
-            }
-        }
+        }   }
         if (siblings && cpu_cores) {
             ratio = siblings / cpu_cores;
         }
@@ -621,7 +905,7 @@ int UTIL_countPhysicalCores(void)
     return numPhysicalCores;
 }
 
-#elif defined(__NetBSD__) || defined(__OpenBSD__) || defined(__DragonFly__)
+#elif defined(__NetBSD__) || defined(__OpenBSD__) || defined(__DragonFly__) || defined(__CYGWIN__)
 
 /* Use POSIX sysconf
  * see: man 3 sysconf */
diff --git a/programs/util.h b/programs/util.h
index 1f524f2934ad..8e187e4f2999 100644
--- a/programs/util.h
+++ b/programs/util.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -20,37 +20,23 @@ extern "C" {
 *  Dependencies
 ******************************************/
 #include "platform.h"     /* PLATFORM_POSIX_VERSION, ZSTD_NANOSLEEP_SUPPORT, ZSTD_SETPRIORITY_SUPPORT */
-#include <stdlib.h>       /* malloc, realloc, free */
 #include <stddef.h>       /* size_t, ptrdiff_t */
-#include <stdio.h>        /* fprintf */
 #include <sys/types.h>    /* stat, utime */
 #include <sys/stat.h>     /* stat, chmod */
-#if defined(_WIN32)
-#  include <sys/utime.h>  /* utime */
-#  include <io.h>         /* _chmod */
-#else
-#  include <unistd.h>     /* chown, stat */
-#if PLATFORM_POSIX_VERSION < 200809L
-#  include <utime.h>      /* utime */
-#else
-#  include <fcntl.h>      /* AT_FDCWD */
-#  include <sys/stat.h>   /* utimensat */
-#endif
-#endif
-#include <time.h>         /* clock_t, clock, CLOCKS_PER_SEC, nanosleep */
-#include "mem.h"          /* U32, U64 */
+#include "../lib/common/mem.h"          /* U64 */
+
 
 /*-************************************************************
 * Avoid fseek()'s 2GiB barrier with MSVC, macOS, *BSD, MinGW
 ***************************************************************/
 #if defined(_MSC_VER) && (_MSC_VER >= 1400)
-#   define UTIL_fseek _fseeki64
+#  define UTIL_fseek _fseeki64
 #elif !defined(__64BIT__) && (PLATFORM_POSIX_VERSION >= 200112L) /* No point defining Large file for 64 bit */
 #  define UTIL_fseek fseeko
 #elif defined(__MINGW32__) && defined(__MSVCRT__) && !defined(__STRICT_ANSI__) && !defined(__NO_MINGW_LFS)
-#   define UTIL_fseek fseeko64
+#  define UTIL_fseek fseeko64
 #else
-#   define UTIL_fseek fseek
+#  define UTIL_fseek fseek
 #endif
 
 
@@ -85,12 +71,6 @@ extern "C" {
 #endif
 
 
-/*-*************************************
-*  Constants
-***************************************/
-#define LIST_SIZE_INCREASE   (8*1024)
-
-
 /*-****************************************
 *  Compiler specifics
 ******************************************/
@@ -112,16 +92,14 @@ extern "C" {
 *  Console log
 ******************************************/
 extern int g_utilDisplayLevel;
-#define UTIL_DISPLAY(...)         fprintf(stderr, __VA_ARGS__)
-#define UTIL_DISPLAYLEVEL(l, ...) { if (g_utilDisplayLevel>=l) { UTIL_DISPLAY(__VA_ARGS__); } }
 
 
 /*-****************************************
 *  File functions
 ******************************************/
 #if defined(_MSC_VER)
-    #define chmod _chmod
     typedef struct __stat64 stat_t;
+    typedef int mode_t;
 #else
     typedef struct stat stat_t;
 #endif
@@ -129,64 +107,127 @@ extern int g_utilDisplayLevel;
 
 int UTIL_fileExist(const char* filename);
 int UTIL_isRegularFile(const char* infilename);
-int UTIL_setFileStat(const char* filename, stat_t* statbuf);
-U32 UTIL_isDirectory(const char* infilename);
-int UTIL_getFileStat(const char* infilename, stat_t* statbuf);
+int UTIL_isDirectory(const char* infilename);
 int UTIL_isSameFile(const char* file1, const char* file2);
-int UTIL_compareStr(const void *p1, const void *p2);
 int UTIL_isCompressedFile(const char* infilename, const char *extensionList[]);
-const char* UTIL_getFileExtension(const char* infilename);
+int UTIL_isLink(const char* infilename);
+int UTIL_isFIFO(const char* infilename);
 
-#ifndef _MSC_VER
-U32 UTIL_isFIFO(const char* infilename);
-#endif
-U32 UTIL_isLink(const char* infilename);
 #define UTIL_FILESIZE_UNKNOWN  ((U64)(-1))
 U64 UTIL_getFileSize(const char* infilename);
+U64 UTIL_getTotalFileSize(const char* const * fileNamesTable, unsigned nbFiles);
+int UTIL_getFileStat(const char* infilename, stat_t* statbuf);
+int UTIL_setFileStat(const char* filename, stat_t* statbuf);
+int UTIL_chmod(char const* filename, mode_t permissions);   /*< like chmod, but avoid changing permission of /dev/null */
+int UTIL_compareStr(const void *p1, const void *p2);
+const char* UTIL_getFileExtension(const char* infilename);
 
-U64 UTIL_getTotalFileSize(const char* const * const fileNamesTable, unsigned nbFiles);
 
-/*
- * A modified version of realloc().
- * If UTIL_realloc() fails the original block is freed.
-*/
-UTIL_STATIC void* UTIL_realloc(void *ptr, size_t size)
-{
-    void *newptr = realloc(ptr, size);
-    if (newptr) return newptr;
-    free(ptr);
-    return NULL;
-}
+/*-****************************************
+ *  Lists of Filenames
+ ******************************************/
+
+typedef struct
+{   const char** fileNames;
+    char* buf;            /* fileNames are stored in this buffer (or are read-only) */
+    size_t tableSize;     /* nb of fileNames */
+    size_t tableCapacity;
+} FileNamesTable;
+
+/*! UTIL_createFileNamesTable_fromFileName() :
+ *  read filenames from @inputFileName, and store them into returned object.
+ * @return : a FileNamesTable*, or NULL in case of error (ex: @inputFileName doesn't exist).
+ *  Note: inputFileSize must be less than 50MB
+ */
+FileNamesTable*
+UTIL_createFileNamesTable_fromFileName(const char* inputFileName);
+
+/*! UTIL_assembleFileNamesTable() :
+ *  This function takes ownership of its arguments, @filenames and @buf,
+ *  and store them inside the created object.
+ *  note : this function never fails,
+ *         it will rather exit() the program if internal allocation fails.
+ * @return : resulting FileNamesTable* object.
+ */
+FileNamesTable*
+UTIL_assembleFileNamesTable(const char** filenames, size_t tableSize, char* buf);
 
-int UTIL_prepareFileList(const char* dirName, char** bufStart, size_t* pos, char** bufEnd, int followLinks);
+/*! UTIL_freeFileNamesTable() :
+ *  This function is compatible with NULL argument and never fails.
+ */
+void UTIL_freeFileNamesTable(FileNamesTable* table);
+
+/*! UTIL_mergeFileNamesTable():
+ * @return : FileNamesTable*, concatenation of @table1 and @table2
+ *  note: @table1 and @table2 are consumed (freed) by this operation
+ */
+FileNamesTable*
+UTIL_mergeFileNamesTable(FileNamesTable* table1, FileNamesTable* table2);
+
+
+/*! UTIL_expandFNT() :
+ *  read names from @fnt, and expand those corresponding to directories
+ *  update @fnt, now containing only file names,
+ * @return : 0 in case of success, 1 if error
+ *  note : in case of error, @fnt[0] is NULL
+ */
+void UTIL_expandFNT(FileNamesTable** fnt, int followLinks);
+
+/*! UTIL_createFNT_fromROTable() :
+ *  copy the @filenames pointer table inside the returned object.
+ *  The names themselves are still stored in their original buffer, which must outlive the object.
+ * @return : a FileNamesTable* object,
+ *        or NULL in case of error
+ */
+FileNamesTable*
+UTIL_createFNT_fromROTable(const char** filenames, size_t nbFilenames);
+
+/*! UTIL_allocateFileNamesTable() :
+ *  Allocates a table of const char*, to insert read-only names later on.
+ *  The created FileNamesTable* doesn't hold a buffer.
+ * @return : FileNamesTable*, or NULL, if allocation fails.
+ */
+FileNamesTable* UTIL_allocateFileNamesTable(size_t tableSize);
+
+
+/*! UTIL_refFilename() :
+ *  Add a reference to read-only name into @fnt table.
+ *  As @filename is only referenced, its lifetime must outlive @fnt.
+ *  Internal table must be large enough to reference a new member,
+ *  otherwise its UB (protected by an `assert()`).
+ */
+void UTIL_refFilename(FileNamesTable* fnt, const char* filename);
+
+
+/* UTIL_createExpandedFNT() is only active if UTIL_HAS_CREATEFILELIST is defined.
+ * Otherwise, UTIL_createExpandedFNT() is a shell function which does nothing
+ * apart from displaying a warning message.
+ */
 #ifdef _WIN32
 #  define UTIL_HAS_CREATEFILELIST
 #elif defined(__linux__) || (PLATFORM_POSIX_VERSION >= 200112L)  /* opendir, readdir require POSIX.1-2001 */
 #  define UTIL_HAS_CREATEFILELIST
-#  include <dirent.h>       /* opendir, readdir */
-#  include <string.h>       /* strerror, memcpy */
 #else
-#endif /* #ifdef _WIN32 */
+   /* do not define UTIL_HAS_CREATEFILELIST */
+#endif
 
-/*
- * UTIL_createFileList - takes a list of files and directories (params: inputNames, inputNamesNb), scans directories,
- *                       and returns a new list of files (params: return value, allocatedBuffer, allocatedNamesNb).
- * After finishing usage of the list the structures should be freed with UTIL_freeFileList(params: return value, allocatedBuffer)
- * In case of error UTIL_createFileList returns NULL and UTIL_freeFileList should not be called.
+/*! UTIL_createExpandedFNT() :
+ *  read names from @filenames, and expand those corresponding to directories.
+ *  links are followed or not depending on @followLinks directive.
+ * @return : an expanded FileNamesTable*, where each name is a file
+ *        or NULL in case of error
  */
-const char**
-UTIL_createFileList(const char **inputNames, unsigned inputNamesNb,
-                    char** allocatedBuffer, unsigned* allocatedNamesNb,
-                    int followLinks);
-
-UTIL_STATIC void UTIL_freeFileList(const char** filenameTable, char* allocatedBuffer)
-{
-    if (allocatedBuffer) free(allocatedBuffer);
-    if (filenameTable) free((void*)filenameTable);
-}
+FileNamesTable*
+UTIL_createExpandedFNT(const char** filenames, size_t nbFilenames, int followLinks);
+
+
+/*-****************************************
+ *  System
+ ******************************************/
 
 int UTIL_countPhysicalCores(void);
 
+
 #if defined (__cplusplus)
 }
 #endif
diff --git a/programs/zstd.1 b/programs/zstd.1
index fef0e76e081a..9ba0b4fa4162 100644
--- a/programs/zstd.1
+++ b/programs/zstd.1
@@ -1,5 +1,5 @@
 .
-.TH "ZSTD" "1" "October 2019" "zstd 1.4.4" "User Commands"
+.TH "ZSTD" "1" "May 2020" "zstd 1.4.5" "User Commands"
 .
 .SH "NAME"
 \fBzstd\fR \- zstd, zstdmt, unzstd, zstdcat \- Compress or decompress \.zst files
@@ -95,120 +95,115 @@ Display information related to a zstd compressed file, such as size, ratio, and
 .
 .SS "Operation modifiers"
 .
-.TP
-\fB\-#\fR
-\fB#\fR compression level [1\-19] (default: 3)
+.IP "\(bu" 4
+\fB\-#\fR: \fB#\fR compression level [1\-19] (default: 3)
 .
-.TP
-\fB\-\-fast[=#]\fR
-switch to ultra\-fast compression levels\. If \fB=#\fR is not present, it defaults to \fB1\fR\. The higher the value, the faster the compression speed, at the cost of some compression ratio\. This setting overwrites compression level if one was set previously\. Similarly, if a compression level is set after \fB\-\-fast\fR, it overrides it\.
+.IP "\(bu" 4
+\fB\-\-fast[=#]\fR: switch to ultra\-fast compression levels\. If \fB=#\fR is not present, it defaults to \fB1\fR\. The higher the value, the faster the compression speed, at the cost of some compression ratio\. This setting overwrites compression level if one was set previously\. Similarly, if a compression level is set after \fB\-\-fast\fR, it overrides it\.
 .
-.TP
-\fB\-\-ultra\fR
-unlocks high compression levels 20+ (maximum 22), using a lot more memory\. Note that decompression will also require more memory when using these levels\.
+.IP "\(bu" 4
+\fB\-\-ultra\fR: unlocks high compression levels 20+ (maximum 22), using a lot more memory\. Note that decompression will also require more memory when using these levels\.
 .
-.TP
-\fB\-\-long[=#]\fR
-enables long distance matching with \fB#\fR \fBwindowLog\fR, if not \fB#\fR is not present it defaults to \fB27\fR\. This increases the window size (\fBwindowLog\fR) and memory usage for both the compressor and decompressor\. This setting is designed to improve the compression ratio for files with long matches at a large distance\.
+.IP "\(bu" 4
+\fB\-\-long[=#]\fR: enables long distance matching with \fB#\fR \fBwindowLog\fR, if not \fB#\fR is not present it defaults to \fB27\fR\. This increases the window size (\fBwindowLog\fR) and memory usage for both the compressor and decompressor\. This setting is designed to improve the compression ratio for files with long matches at a large distance\.
 .
 .IP
 Note: If \fBwindowLog\fR is set to larger than 27, \fB\-\-long=windowLog\fR or \fB\-\-memory=windowSize\fR needs to be passed to the decompressor\.
 .
-.TP
-\fB\-T#\fR, \fB\-\-threads=#\fR
-Compress using \fB#\fR working threads (default: 1)\. If \fB#\fR is 0, attempt to detect and use the number of physical CPU cores\. In all cases, the nb of threads is capped to ZSTDMT_NBTHREADS_MAX==200\. This modifier does nothing if \fBzstd\fR is compiled without multithread support\.
+.IP "\(bu" 4
+\fB\-\-patch\-from=FILE\fR: Specify the file to be used as a reference point for zstd\'s diff engine\. This is effectively dictionary compression with some convenient parameter selection, namely that windowSize > srcSize\.
 .
-.TP
-\fB\-\-single\-thread\fR
-Does not spawn a thread for compression, use a single thread for both I/O and compression\. In this mode, compression is serialized with I/O, which is slightly slower\. (This is different from \fB\-T1\fR, which spawns 1 compression thread in parallel of I/O)\. This mode is the only one available when multithread support is disabled\. Single\-thread mode features lower memory usage\. Final compressed result is slightly different from \fB\-T1\fR\.
+.IP
+Note: cannot use both this and \-D together Note: \fB\-\-long\fR mode will be automatically activated if chainLog < fileLog (fileLog being the windowLog requried to cover the whole file)\. You can also manually force it\. Node: for all levels, you can use \-\-patch\-from in \-\-single\-thread mode to improve compression ratio at the cost of speed Note: for level 19, you can get increased compression ratio at the cost of speed by specifying \fB\-\-zstd=targetLength=\fR to be something large (i\.e 4096), and by setting a large \fB\-\-zstd=chainLog=\fR
 .
-.TP
-\fB\-\-adapt[=min=#,max=#]\fR
-\fBzstd\fR will dynamically adapt compression level to perceived I/O conditions\. Compression level adaptation can be observed live by using command \fB\-v\fR\. Adaptation can be constrained between supplied \fBmin\fR and \fBmax\fR levels\. The feature works when combined with multi\-threading and \fB\-\-long\fR mode\. It does not work with \fB\-\-single\-thread\fR\. It sets window size to 8 MB by default (can be changed manually, see \fBwlog\fR)\. Due to the chaotic nature of dynamic adaptation, compressed result is not reproducible\. \fInote\fR : at the time of this writing, \fB\-\-adapt\fR can remain stuck at low speed when combined with multiple worker threads (>=2)\.
+.IP "\(bu" 4
+\fB\-M#\fR, \fB\-\-memory=#\fR: Set a memory usage limit\. By default, Zstandard uses 128 MB for decompression as the maximum amount of memory the decompressor is allowed to use, but you can override this manually if need be in either direction (ie\. you can increase or decrease it)\.
 .
-.TP
-\fB\-\-stream\-size=#\fR
-Sets the pledged source size of input coming from a stream\. This value must be exact, as it will be included in the produced frame header\. Incorrect stream sizes will cause an error\. This information will be used to better optimize compression parameters, resulting in better and potentially faster compression, especially for smaller source sizes\.
+.IP
+This is also used during compression when using with \-\-patch\-from=\. In this case, this parameter overrides that maximum size allowed for a dictionary\. (128 MB)\.
 .
-.TP
-\fB\-\-size\-hint=#\fR
-When handling input from a stream, \fBzstd\fR must guess how large the source size will be when optimizing compression parameters\. If the stream size is relatively small, this guess may be a poor one, resulting in a higher compression ratio than expected\. This feature allows for controlling the guess when needed\. Exact guesses result in better compression ratios\. Overestimates result in slightly degraded compression ratios, while underestimates may result in significant degradation\.
+.IP "\(bu" 4
+\fB\-T#\fR, \fB\-\-threads=#\fR: Compress using \fB#\fR working threads (default: 1)\. If \fB#\fR is 0, attempt to detect and use the number of physical CPU cores\. In all cases, the nb of threads is capped to ZSTDMT_NBTHREADS_MAX==200\. This modifier does nothing if \fBzstd\fR is compiled without multithread support\.
 .
-.TP
-\fB\-\-rsyncable\fR
-\fBzstd\fR will periodically synchronize the compression state to make the compressed file more rsync\-friendly\. There is a negligible impact to compression ratio, and the faster compression levels will see a small compression speed hit\. This feature does not work with \fB\-\-single\-thread\fR\. You probably don\'t want to use it with long range mode, since it will decrease the effectiveness of the synchronization points, but your milage may vary\.
+.IP "\(bu" 4
+\fB\-\-single\-thread\fR: Does not spawn a thread for compression, use a single thread for both I/O and compression\. In this mode, compression is serialized with I/O, which is slightly slower\. (This is different from \fB\-T1\fR, which spawns 1 compression thread in parallel of I/O)\. This mode is the only one available when multithread support is disabled\. Single\-thread mode features lower memory usage\. Final compressed result is slightly different from \fB\-T1\fR\.
 .
-.TP
-\fB\-D file\fR
-use \fBfile\fR as Dictionary to compress or decompress FILE(s)
+.IP "\(bu" 4
+\fB\-\-adapt[=min=#,max=#]\fR : \fBzstd\fR will dynamically adapt compression level to perceived I/O conditions\. Compression level adaptation can be observed live by using command \fB\-v\fR\. Adaptation can be constrained between supplied \fBmin\fR and \fBmax\fR levels\. The feature works when combined with multi\-threading and \fB\-\-long\fR mode\. It does not work with \fB\-\-single\-thread\fR\. It sets window size to 8 MB by default (can be changed manually, see \fBwlog\fR)\. Due to the chaotic nature of dynamic adaptation, compressed result is not reproducible\. \fInote\fR : at the time of this writing, \fB\-\-adapt\fR can remain stuck at low speed when combined with multiple worker threads (>=2)\.
 .
-.TP
-\fB\-\-no\-dictID\fR
-do not store dictionary ID within frame header (dictionary compression)\. The decoder will have to rely on implicit knowledge about which dictionary to use, it won\'t be able to check if it\'s correct\.
+.IP "\(bu" 4
+\fB\-\-stream\-size=#\fR : Sets the pledged source size of input coming from a stream\. This value must be exact, as it will be included in the produced frame header\. Incorrect stream sizes will cause an error\. This information will be used to better optimize compression parameters, resulting in better and potentially faster compression, especially for smaller source sizes\.
 .
-.TP
-\fB\-o file\fR
-save result into \fBfile\fR (only possible with a single \fIINPUT\-FILE\fR)
+.IP "\(bu" 4
+\fB\-\-size\-hint=#\fR: When handling input from a stream, \fBzstd\fR must guess how large the source size will be when optimizing compression parameters\. If the stream size is relatively small, this guess may be a poor one, resulting in a higher compression ratio than expected\. This feature allows for controlling the guess when needed\. Exact guesses result in better compression ratios\. Overestimates result in slightly degraded compression ratios, while underestimates may result in significant degradation\.
 .
-.TP
-\fB\-f\fR, \fB\-\-force\fR
-overwrite output without prompting, and (de)compress symbolic links
+.IP "\(bu" 4
+\fB\-\-rsyncable\fR : \fBzstd\fR will periodically synchronize the compression state to make the compressed file more rsync\-friendly\. There is a negligible impact to compression ratio, and the faster compression levels will see a small compression speed hit\. This feature does not work with \fB\-\-single\-thread\fR\. You probably don\'t want to use it with long range mode, since it will decrease the effectiveness of the synchronization points, but your milage may vary\.
 .
-.TP
-\fB\-c\fR, \fB\-\-stdout\fR
-force write to standard output, even if it is the console
+.IP "\(bu" 4
+\fB\-D file\fR: use \fBfile\fR as Dictionary to compress or decompress FILE(s)
 .
-.TP
-\fB\-\-[no\-]sparse\fR
-enable / disable sparse FS support, to make files with many zeroes smaller on disk\. Creating sparse files may save disk space and speed up decompression by reducing the amount of disk I/O\. default: enabled when output is into a file, and disabled when output is stdout\. This setting overrides default and can force sparse mode over stdout\.
+.IP "\(bu" 4
+\fB\-\-no\-dictID\fR: do not store dictionary ID within frame header (dictionary compression)\. The decoder will have to rely on implicit knowledge about which dictionary to use, it won\'t be able to check if it\'s correct\.
 .
-.TP
-\fB\-\-rm\fR
-remove source file(s) after successful compression or decompression
+.IP "\(bu" 4
+\fB\-o file\fR: save result into \fBfile\fR (only possible with a single \fIINPUT\-FILE\fR)
 .
-.TP
-\fB\-k\fR, \fB\-\-keep\fR
-keep source file(s) after successful compression or decompression\. This is the default behavior\.
+.IP "\(bu" 4
+\fB\-f\fR, \fB\-\-force\fR: overwrite output without prompting, and (de)compress symbolic links
 .
-.TP
-\fB\-r\fR
-operate recursively on directories
+.IP "\(bu" 4
+\fB\-c\fR, \fB\-\-stdout\fR: force write to standard output, even if it is the console
 .
-.TP
-\fB\-\-output\-dir\-flat[=dir]\fR
-resulting files are stored into target \fBdir\fR directory, instead of same directory as origin file\. Be aware that this command can introduce name collision issues, if multiple files, from different directories, end up having the same name\. Collision resolution ensures first file with a given name will be present in \fBdir\fR, while in combination with \fB\-f\fR, the last file will be present instead\.
+.IP "\(bu" 4
+\fB\-\-[no\-]sparse\fR: enable / disable sparse FS support, to make files with many zeroes smaller on disk\. Creating sparse files may save disk space and speed up decompression by reducing the amount of disk I/O\. default: enabled when output is into a file, and disabled when output is stdout\. This setting overrides default and can force sparse mode over stdout\.
 .
-.TP
-\fB\-\-format=FORMAT\fR
-compress and decompress in other formats\. If compiled with support, zstd can compress to or decompress from other compression algorithm formats\. Possibly available options are \fBzstd\fR, \fBgzip\fR, \fBxz\fR, \fBlzma\fR, and \fBlz4\fR\. If no such format is provided, \fBzstd\fR is the default\.
+.IP "\(bu" 4
+\fB\-\-[no\-]content\-size\fR: enable / disable whether or not the original size of the file is placed in the header of the compressed file\. The default option is \-\-content\-size (meaning that the original size will be placed in the header)\.
 .
-.TP
-\fB\-h\fR/\fB\-H\fR, \fB\-\-help\fR
-display help/long help and exit
+.IP "\(bu" 4
+\fB\-\-rm\fR: remove source file(s) after successful compression or decompression
 .
-.TP
-\fB\-V\fR, \fB\-\-version\fR
-display version number and exit\. Advanced : \fB\-vV\fR also displays supported formats\. \fB\-vvV\fR also displays POSIX support\.
+.IP "\(bu" 4
+\fB\-k\fR, \fB\-\-keep\fR: keep source file(s) after successful compression or decompression\. This is the default behavior\.
 .
-.TP
-\fB\-v\fR
-verbose mode
+.IP "\(bu" 4
+\fB\-r\fR: operate recursively on directories
 .
-.TP
-\fB\-q\fR, \fB\-\-quiet\fR
-suppress warnings, interactivity, and notifications\. specify twice to suppress errors too\.
+.IP "\(bu" 4
+\fB\-\-filelist=FILE\fR read a list of files to process as content from \fBFILE\fR\. Format is compatible with \fBls\fR output, with one file per line\.
 .
-.TP
-\fB\-\-no\-progress\fR
-do not display the progress bar, but keep all other messages\.
+.IP "\(bu" 4
+\fB\-\-output\-dir\-flat[=dir]\fR: resulting files are stored into target \fBdir\fR directory, instead of same directory as origin file\. Be aware that this command can introduce name collision issues, if multiple files, from different directories, end up having the same name\. Collision resolution ensures first file with a given name will be present in \fBdir\fR, while in combination with \fB\-f\fR, the last file will be present instead\.
 .
-.TP
-\fB\-C\fR, \fB\-\-[no\-]check\fR
-add integrity check computed from uncompressed data (default: enabled)
+.IP "\(bu" 4
+\fB\-\-format=FORMAT\fR: compress and decompress in other formats\. If compiled with support, zstd can compress to or decompress from other compression algorithm formats\. Possibly available options are \fBzstd\fR, \fBgzip\fR, \fBxz\fR, \fBlzma\fR, and \fBlz4\fR\. If no such format is provided, \fBzstd\fR is the default\.
 .
-.TP
-\fB\-\-\fR
-All arguments after \fB\-\-\fR are treated as files
+.IP "\(bu" 4
+\fB\-h\fR/\fB\-H\fR, \fB\-\-help\fR: display help/long help and exit
+.
+.IP "\(bu" 4
+\fB\-V\fR, \fB\-\-version\fR: display version number and exit\. Advanced : \fB\-vV\fR also displays supported formats\. \fB\-vvV\fR also displays POSIX support\.
+.
+.IP "\(bu" 4
+\fB\-v\fR, \fB\-\-verbose\fR: verbose mode
+.
+.IP "\(bu" 4
+\fB\-\-show\-default\-cparams\fR: Shows the default compresssion parameters that will be used for a particular src file\. If the provided src file is not a regular file (eg\. named pipe), the cli will just output the default paramters\. That is, the parameters that are used when the src size is unknown\.
+.
+.IP "\(bu" 4
+\fB\-q\fR, \fB\-\-quiet\fR: suppress warnings, interactivity, and notifications\. specify twice to suppress errors too\.
+.
+.IP "\(bu" 4
+\fB\-\-no\-progress\fR: do not display the progress bar, but keep all other messages\.
+.
+.IP "\(bu" 4
+\fB\-C\fR, \fB\-\-[no\-]check\fR: add integrity check computed from uncompressed data (default: enabled)
+.
+.IP "\(bu" 4
+\fB\-\-\fR: All arguments after \fB\-\-\fR are treated as files
+.
+.IP "" 0
 .
 .SS "Restricted usage of Environment Variables"
 Using environment variables to set parameters has security implications\. Therefore, this avenue is intentionally restricted\. Only \fBZSTD_CLEVEL\fR is supported currently, for setting compression level\. \fBZSTD_CLEVEL\fR can be used to set the level between 1 and 19 (the "normal" range)\. If the value of \fBZSTD_CLEVEL\fR is not a valid integer, it will be ignored with a warning message\. \fBZSTD_CLEVEL\fR just replaces the default compression level (\fB3\fR)\. It can be overridden by corresponding command line arguments\.
@@ -361,7 +356,7 @@ Specify the maximum number of bits for a hash table\.
 Bigger hash tables cause less collisions which usually makes compression faster, but requires more memory during compression\.
 .
 .IP
-The minimum \fIhlog\fR is 6 (64 B) and the maximum is 26 (128 MiB)\.
+The minimum \fIhlog\fR is 6 (64 B) and the maximum is 30 (1 GiB)\.
 .
 .TP
 \fBchainLog\fR=\fIclog\fR, \fBclog\fR=\fIclog\fR
@@ -371,7 +366,7 @@ Specify the maximum number of bits for a hash chain or a binary tree\.
 Higher numbers of bits increases the chance to find a match which usually improves compression ratio\. It also slows down compression speed and increases memory requirements for compression\. This option is ignored for the ZSTD_fast strategy\.
 .
 .IP
-The minimum \fIclog\fR is 6 (64 B) and the maximum is 28 (256 MiB)\.
+The minimum \fIclog\fR is 6 (64 B) and the maximum is 29 (524 Mib) on 32\-bit platforms and 30 (1 Gib) on 64\-bit platforms\.
 .
 .TP
 \fBsearchLog\fR=\fIslog\fR, \fBslog\fR=\fIslog\fR
@@ -381,7 +376,7 @@ Specify the maximum number of searches in a hash chain or a binary tree using lo
 More searches increases the chance to find a match which usually increases compression ratio but decreases compression speed\.
 .
 .IP
-The minimum \fIslog\fR is 1 and the maximum is 26\.
+The minimum \fIslog\fR is 1 and the maximum is \'windowLog\' \- 1\.
 .
 .TP
 \fBminMatch\fR=\fImml\fR, \fBmml\fR=\fImml\fR
@@ -394,20 +389,17 @@ Larger search lengths usually decrease compression ratio but improve decompressi
 The minimum \fImml\fR is 3 and the maximum is 7\.
 .
 .TP
-\fBtargetLen\fR=\fItlen\fR, \fBtlen\fR=\fItlen\fR
+\fBtargetLength\fR=\fItlen\fR, \fBtlen\fR=\fItlen\fR
 The impact of this field vary depending on selected strategy\.
 .
 .IP
-For ZSTD_btopt, ZSTD_btultra and ZSTD_btultra2, it specifies the minimum match length that causes match finder to stop searching\. A larger \fBtargetLen\fR usually improves compression ratio but decreases compression speed\.
-.
-.IP
-For ZSTD_fast, it triggers ultra\-fast mode when > 0\. The value represents the amount of data skipped between match sampling\. Impact is reversed : a larger \fBtargetLen\fR increases compression speed but decreases compression ratio\.
+For ZSTD_btopt, ZSTD_btultra and ZSTD_btultra2, it specifies the minimum match length that causes match finder to stop searching\. A larger \fBtargetLength\fR usually improves compression ratio but decreases compression speed\. t For ZSTD_fast, it triggers ultra\-fast mode when > 0\. The value represents the amount of data skipped between match sampling\. Impact is reversed : a larger \fBtargetLength\fR increases compression speed but decreases compression ratio\.
 .
 .IP
 For all other strategies, this field has no impact\.
 .
 .IP
-The minimum \fItlen\fR is 0 and the maximum is 999\.
+The minimum \fItlen\fR is 0 and the maximum is 128 Kib\.
 .
 .TP
 \fBoverlapLog\fR=\fIovlog\fR, \fBovlog\fR=\fIovlog\fR
@@ -427,7 +419,7 @@ This option is ignored unless long distance matching is enabled\.
 Bigger hash tables usually improve compression ratio at the expense of more memory during compression and a decrease in compression speed\.
 .
 .IP
-The minimum \fIlhlog\fR is 6 and the maximum is 26 (default: 20)\.
+The minimum \fIlhlog\fR is 6 and the maximum is 30 (default: 20)\.
 .
 .TP
 \fBldmMinMatch\fR=\fIlmml\fR, \fBlmml\fR=\fIlmml\fR
@@ -453,7 +445,7 @@ This option is ignored unless long distance matching is enabled\.
 Larger bucket sizes improve collision resolution but decrease compression speed\.
 .
 .IP
-The minimum \fIlblog\fR is 0 and the maximum is 8 (default: 3)\.
+The minimum \fIlblog\fR is 1 and the maximum is 8 (default: 3)\.
 .
 .TP
 \fBldmHashRateLog\fR=\fIlhrlog\fR, \fBlhrlog\fR=\fIlhrlog\fR
diff --git a/programs/zstd.1.md b/programs/zstd.1.md
index e3daa4c87ac7..550e2e53fe5f 100644
--- a/programs/zstd.1.md
+++ b/programs/zstd.1.md
@@ -122,6 +122,28 @@ the last one takes effect.
 
     Note: If `windowLog` is set to larger than 27, `--long=windowLog` or
     `--memory=windowSize` needs to be passed to the decompressor.
+* `--patch-from=FILE`:
+    Specify the file to be used as a reference point for zstd's diff engine.
+    This is effectively dictionary compression with some convenient parameter
+    selection, namely that windowSize > srcSize.
+
+    Note: cannot use both this and -D together
+    Note: `--long` mode will be automatically activated if chainLog < fileLog
+        (fileLog being the windowLog requried to cover the whole file). You 
+        can also manually force it.
+	Node: for all levels, you can use --patch-from in --single-thread mode
+		to improve compression ratio at the cost of speed
+    Note: for level 19, you can get increased compression ratio at the cost 
+        of speed by specifying `--zstd=targetLength=` to be something large 
+        (i.e 4096), and by setting a large `--zstd=chainLog=`
+* `-M#`, `--memory=#`:
+    Set a memory usage limit. By default, Zstandard uses 128 MB for decompression
+    as the maximum amount of memory the decompressor is allowed to use, but you can
+    override this manually if need be in either direction (ie. you can increase or
+    decrease it).
+
+    This is also used during compression when using with --patch-from=. In this case,
+    this parameter overrides that maximum size allowed for a dictionary. (128 MB).
 * `-T#`, `--threads=#`:
     Compress using `#` working threads (default: 1).
     If `#` is 0, attempt to detect and use the number of physical CPU cores.
@@ -184,6 +206,10 @@ the last one takes effect.
     default: enabled when output is into a file,
     and disabled when output is stdout.
     This setting overrides default and can force sparse mode over stdout.
+* `--[no-]content-size`:
+    enable / disable whether or not the original size of the file is placed in
+    the header of the compressed file. The default option is 
+    --content-size (meaning that the original size will be placed in the header).
 * `--rm`:
     remove source file(s) after successful compression or decompression
 * `-k`, `--keep`:
@@ -191,6 +217,9 @@ the last one takes effect.
     This is the default behavior.
 * `-r`:
     operate recursively on directories
+* `--filelist=FILE`
+    read a list of files to process as content from `FILE`.
+    Format is compatible with `ls` output, with one file per line.
 * `--output-dir-flat[=dir]`:
     resulting files are stored into target `dir` directory,
     instead of same directory as origin file.
@@ -209,8 +238,14 @@ the last one takes effect.
     display version number and exit.
     Advanced : `-vV` also displays supported formats.
     `-vvV` also displays POSIX support.
-* `-v`:
+* `-v`, `--verbose`:
     verbose mode
+* `--show-default-cparams`:
+    Shows the default compresssion parameters that will be used for a
+    particular src file. If the provided src file is not a regular file
+    (eg. named pipe), the cli will just output the default paramters.
+    That is, the parameters that are used when the src size is
+    unknown.
 * `-q`, `--quiet`:
     suppress warnings, interactivity, and notifications.
     specify twice to suppress errors too.
@@ -402,7 +437,7 @@ The list of available _options_:
     Bigger hash tables cause less collisions which usually makes compression
     faster, but requires more memory during compression.
 
-    The minimum _hlog_ is 6 (64 B) and the maximum is 26 (128 MiB).
+    The minimum _hlog_ is 6 (64 B) and the maximum is 30 (1 GiB).
 
 - `chainLog`=_clog_, `clog`=_clog_:
     Specify the maximum number of bits for a hash chain or a binary tree.
@@ -413,7 +448,8 @@ The list of available _options_:
     compression.
     This option is ignored for the ZSTD_fast strategy.
 
-    The minimum _clog_ is 6 (64 B) and the maximum is 28 (256 MiB).
+    The minimum _clog_ is 6 (64 B) and the maximum is 29 (524 Mib) on 32-bit platforms
+    and 30 (1 Gib) on 64-bit platforms.
 
 - `searchLog`=_slog_, `slog`=_slog_:
     Specify the maximum number of searches in a hash chain or a binary tree
@@ -422,7 +458,7 @@ The list of available _options_:
     More searches increases the chance to find a match which usually increases
     compression ratio but decreases compression speed.
 
-    The minimum _slog_ is 1 and the maximum is 26.
+    The minimum _slog_ is 1 and the maximum is 'windowLog' - 1.
 
 - `minMatch`=_mml_, `mml`=_mml_:
     Specify the minimum searched length of a match in a hash table.
@@ -432,22 +468,22 @@ The list of available _options_:
 
     The minimum _mml_ is 3 and the maximum is 7.
 
-- `targetLen`=_tlen_, `tlen`=_tlen_:
+- `targetLength`=_tlen_, `tlen`=_tlen_:
     The impact of this field vary depending on selected strategy.
 
     For ZSTD\_btopt, ZSTD\_btultra and ZSTD\_btultra2, it specifies
     the minimum match length that causes match finder to stop searching.
-    A larger `targetLen` usually improves compression ratio
+    A larger `targetLength` usually improves compression ratio
     but decreases compression speed.
-
+t
     For ZSTD\_fast, it triggers ultra-fast mode when > 0.
     The value represents the amount of data skipped between match sampling.
-    Impact is reversed : a larger `targetLen` increases compression speed
+    Impact is reversed : a larger `targetLength` increases compression speed
     but decreases compression ratio.
 
     For all other strategies, this field has no impact.
 
-    The minimum _tlen_ is 0 and the maximum is 999.
+    The minimum _tlen_ is 0 and the maximum is 128 Kib.
 
 - `overlapLog`=_ovlog_,  `ovlog`=_ovlog_:
     Determine `overlapSize`, amount of data reloaded from previous job.
@@ -470,7 +506,7 @@ The list of available _options_:
     Bigger hash tables usually improve compression ratio at the expense of more
     memory during compression and a decrease in compression speed.
 
-    The minimum _lhlog_ is 6 and the maximum is 26 (default: 20).
+    The minimum _lhlog_ is 6 and the maximum is 30 (default: 20).
 
 - `ldmMinMatch`=_lmml_, `lmml`=_lmml_:
     Specify the minimum searched length of a match for long distance matching.
@@ -490,7 +526,7 @@ The list of available _options_:
     Larger bucket sizes improve collision resolution but decrease compression
     speed.
 
-    The minimum _lblog_ is 0 and the maximum is 8 (default: 3).
+    The minimum _lblog_ is 1 and the maximum is 8 (default: 3).
 
 - `ldmHashRateLog`=_lhrlog_, `lhrlog`=_lhrlog_:
     Specify the frequency of inserting entries into the long distance matching
diff --git a/programs/zstdcli.c b/programs/zstdcli.c
index cd6b40bc089f..70e2b70666bf 100644
--- a/programs/zstdcli.c
+++ b/programs/zstdcli.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
@@ -27,10 +27,12 @@
 **************************************/
 #include "platform.h" /* IS_CONSOLE, PLATFORM_POSIX_VERSION */
 #include "util.h"     /* UTIL_HAS_CREATEFILELIST, UTIL_createFileList */
-#include <stdio.h>    /* fprintf(), stdin, stdout, stderr */
 #include <stdlib.h>   /* getenv */
 #include <string.h>   /* strcmp, strlen */
+#include <stdio.h>    /* fprintf(), stdin, stdout, stderr */
 #include <errno.h>    /* errno */
+#include <assert.h>   /* assert */
+
 #include "fileio.h"   /* stdinmark, stdoutmark, ZSTD_EXTENSION */
 #ifndef ZSTD_NOBENCH
 #  include "benchzstd.h"  /* BMK_benchFiles */
@@ -38,8 +40,7 @@
 #ifndef ZSTD_NODICT
 #  include "dibio.h"  /* ZDICT_cover_params_t, DiB_trainFromFiles() */
 #endif
-#define ZSTD_STATIC_LINKING_ONLY   /* ZSTD_minCLevel */
-#include "zstd.h"     /* ZSTD_VERSION_STRING, ZSTD_maxCLevel */
+#include "../lib/zstd.h"  /* ZSTD_VERSION_STRING, ZSTD_minCLevel, ZSTD_maxCLevel */
 
 
 /*-************************************
@@ -93,127 +94,146 @@ typedef enum { cover, fastCover, legacy } dictType;
 /*-************************************
 *  Display Macros
 **************************************/
-#define DISPLAY(...)         fprintf(g_displayOut, __VA_ARGS__)
+#define DISPLAY_F(f, ...)    fprintf((f), __VA_ARGS__)
+#define DISPLAYOUT(...)      DISPLAY_F(stdout, __VA_ARGS__)
+#define DISPLAY(...)         DISPLAY_F(stderr, __VA_ARGS__)
 #define DISPLAYLEVEL(l, ...) { if (g_displayLevel>=l) { DISPLAY(__VA_ARGS__); } }
 static int g_displayLevel = DISPLAY_LEVEL_DEFAULT;   /* 0 : no display,  1: errors,  2 : + result + interaction + warnings,  3 : + progression,  4 : + information */
-static FILE* g_displayOut;
 
 
 /*-************************************
 *  Command Line
 **************************************/
-static int usage(const char* programName)
+/* print help either in `stderr` or `stdout` depending on originating request
+ * error (badusage) => stderr
+ * help (usage_advanced) => stdout
+ */
+static void usage(FILE* f, const char* programName)
 {
-    DISPLAY( "Usage : \n");
-    DISPLAY( "      %s [args] [FILE(s)] [-o file] \n", programName);
-    DISPLAY( "\n");
-    DISPLAY( "FILE    : a filename \n");
-    DISPLAY( "          with no FILE, or when FILE is - , read standard input\n");
-    DISPLAY( "Arguments : \n");
+    DISPLAY_F(f, "Usage : \n");
+    DISPLAY_F(f, "      %s [args] [FILE(s)] [-o file] \n", programName);
+    DISPLAY_F(f, "\n");
+    DISPLAY_F(f, "FILE    : a filename \n");
+    DISPLAY_F(f, "          with no FILE, or when FILE is - , read standard input\n");
+    DISPLAY_F(f, "Arguments : \n");
 #ifndef ZSTD_NOCOMPRESS
-    DISPLAY( " -#     : # compression level (1-%d, default: %d) \n", ZSTDCLI_CLEVEL_MAX, ZSTDCLI_CLEVEL_DEFAULT);
+    DISPLAY_F(f, " -#     : # compression level (1-%d, default: %d) \n", ZSTDCLI_CLEVEL_MAX, ZSTDCLI_CLEVEL_DEFAULT);
 #endif
 #ifndef ZSTD_NODECOMPRESS
-    DISPLAY( " -d     : decompression \n");
+    DISPLAY_F(f, " -d     : decompression \n");
 #endif
-    DISPLAY( " -D file: use `file` as Dictionary \n");
-    DISPLAY( " -o file: result stored into `file` (only if 1 input file) \n");
-    DISPLAY( " -f     : overwrite output without prompting and (de)compress links \n");
-    DISPLAY( "--rm    : remove source file(s) after successful de/compression \n");
-    DISPLAY( " -k     : preserve source file(s) (default) \n");
-    DISPLAY( " -h/-H  : display help/long help and exit \n");
-    return 0;
+    DISPLAY_F(f, " -D DICT: use DICT as Dictionary for compression or decompression \n");
+    DISPLAY_F(f, " -o file: result stored into `file` (only 1 output file) \n");
+    DISPLAY_F(f, " -f     : overwrite output without prompting, also (de)compress links \n");
+    DISPLAY_F(f, "--rm    : remove source file(s) after successful de/compression \n");
+    DISPLAY_F(f, " -k     : preserve source file(s) (default) \n");
+    DISPLAY_F(f, " -h/-H  : display help/long help and exit \n");
 }
 
-static int usage_advanced(const char* programName)
+static void usage_advanced(const char* programName)
 {
-    DISPLAY(WELCOME_MESSAGE);
-    usage(programName);
-    DISPLAY( "\n");
-    DISPLAY( "Advanced arguments : \n");
-    DISPLAY( " -V     : display Version number and exit \n");
-    DISPLAY( " -v     : verbose mode; specify multiple times to increase verbosity\n");
-    DISPLAY( " -q     : suppress warnings; specify twice to suppress errors too\n");
-    DISPLAY( " -c     : force write to standard output, even if it is the console\n");
-    DISPLAY( " -l     : print information about zstd compressed files \n");
-    DISPLAY( "--exclude-compressed:  only compress files that are not previously compressed \n");
-#ifndef ZSTD_NOCOMPRESS
-    DISPLAY( "--ultra : enable levels beyond %i, up to %i (requires more memory)\n", ZSTDCLI_CLEVEL_MAX, ZSTD_maxCLevel());
-    DISPLAY( "--long[=#]: enable long distance matching with given window log (default: %u)\n", g_defaultMaxWindowLog);
-    DISPLAY( "--fast[=#]: switch to very fast compression levels (default: %u)\n", 1);
-    DISPLAY( "--adapt : dynamically adapt compression level to I/O conditions \n");
-    DISPLAY( "--stream-size=# : optimize compression parameters for streaming input of given number of bytes \n");
-    DISPLAY( "--size-hint=# optimize compression parameters for streaming input of approximately this size\n");
-    DISPLAY( "--target-compressed-block-size=# : make compressed block near targeted size \n");
-#ifdef ZSTD_MULTITHREAD
-    DISPLAY( " -T#    : spawns # compression threads (default: 1, 0==# cores) \n");
-    DISPLAY( " -B#    : select size of each job (default: 0==automatic) \n");
-    DISPLAY( "--rsyncable : compress using a rsync-friendly method (-B sets block size) \n");
-#endif
-    DISPLAY( "--no-dictID : don't write dictID into header (dictionary compression)\n");
-    DISPLAY( "--[no-]check : integrity check (default: enabled) \n");
-    DISPLAY( "--[no-]compress-literals : force (un)compressed literals \n");
-#endif
+    DISPLAYOUT(WELCOME_MESSAGE);
+    usage(stdout, programName);
+    DISPLAYOUT( "\n");
+    DISPLAYOUT( "Advanced arguments : \n");
+    DISPLAYOUT( " -V     : display Version number and exit \n");
+
+    DISPLAYOUT( " -c     : force write to standard output, even if it is the console \n");
+
+    DISPLAYOUT( " -v     : verbose mode; specify multiple times to increase verbosity \n");
+    DISPLAYOUT( " -q     : suppress warnings; specify twice to suppress errors too \n");
+    DISPLAYOUT( "--no-progress : do not display the progress counter \n");
+
 #ifdef UTIL_HAS_CREATEFILELIST
-    DISPLAY( " -r     : operate recursively on directories \n");
-    DISPLAY( "--output-dir-flat[=directory]: all resulting files stored into `directory`. \n");
+    DISPLAYOUT( " -r     : operate recursively on directories \n");
+    DISPLAYOUT( "--filelist=FILE : read list of files to operate upon from FILE \n");
+    DISPLAYOUT( "--output-dir-flat=DIR : all resulting files are stored into DIR \n");
 #endif
-    DISPLAY( "--format=zstd : compress files to the .zst format (default) \n");
+
+    DISPLAYOUT( "--      : All arguments after \"--\" are treated as files \n");
+
+#ifndef ZSTD_NOCOMPRESS
+    DISPLAYOUT( "\n");
+    DISPLAYOUT( "Advanced compression arguments : \n");
+    DISPLAYOUT( "--ultra : enable levels beyond %i, up to %i (requires more memory) \n", ZSTDCLI_CLEVEL_MAX, ZSTD_maxCLevel());
+    DISPLAYOUT( "--long[=#]: enable long distance matching with given window log (default: %u) \n", g_defaultMaxWindowLog);
+    DISPLAYOUT( "--fast[=#]: switch to very fast compression levels (default: %u) \n", 1);
+    DISPLAYOUT( "--adapt : dynamically adapt compression level to I/O conditions \n");
+# ifdef ZSTD_MULTITHREAD
+    DISPLAYOUT( " -T#    : spawns # compression threads (default: 1, 0==# cores) \n");
+    DISPLAYOUT( " -B#    : select size of each job (default: 0==automatic) \n");
+    DISPLAYOUT( "--single-thread : use a single thread for both I/O and compression (result slightly different than -T1) \n");
+    DISPLAYOUT( "--rsyncable : compress using a rsync-friendly method (-B sets block size) \n");
+# endif
+    DISPLAYOUT( "--exclude-compressed: only compress files that are not already compressed \n");
+    DISPLAYOUT( "--stream-size=# : specify size of streaming input from `stdin` \n");
+    DISPLAYOUT( "--size-hint=# optimize compression parameters for streaming input of approximately this size \n");
+    DISPLAYOUT( "--target-compressed-block-size=# : generate compressed block of approximately targeted size \n");
+    DISPLAYOUT( "--no-dictID : don't write dictID into header (dictionary compression only) \n");
+    DISPLAYOUT( "--[no-]check : add XXH64 integrity checksum to frame (default: enabled) \n");
+    DISPLAYOUT( "--[no-]compress-literals : force (un)compressed literals \n");
+
+    DISPLAYOUT( "--format=zstd : compress files to the .zst format (default) \n");
 #ifdef ZSTD_GZCOMPRESS
-    DISPLAY( "--format=gzip : compress files to the .gz format \n");
+    DISPLAYOUT( "--format=gzip : compress files to the .gz format \n");
 #endif
 #ifdef ZSTD_LZMACOMPRESS
-    DISPLAY( "--format=xz : compress files to the .xz format \n");
-    DISPLAY( "--format=lzma : compress files to the .lzma format \n");
+    DISPLAYOUT( "--format=xz : compress files to the .xz format \n");
+    DISPLAYOUT( "--format=lzma : compress files to the .lzma format \n");
 #endif
 #ifdef ZSTD_LZ4COMPRESS
-    DISPLAY( "--format=lz4 : compress files to the .lz4 format \n");
+    DISPLAYOUT( "--format=lz4 : compress files to the .lz4 format \n");
 #endif
+#endif  /* !ZSTD_NOCOMPRESS */
+
 #ifndef ZSTD_NODECOMPRESS
-    DISPLAY( "--test  : test compressed file integrity \n");
-#if ZSTD_SPARSE_DEFAULT
-    DISPLAY( "--[no-]sparse : sparse mode (default: enabled on file, disabled on stdout)\n");
-#else
-    DISPLAY( "--[no-]sparse : sparse mode (default: disabled)\n");
-#endif
-#endif
-    DISPLAY( " -M#    : Set a memory usage limit for decompression \n");
-    DISPLAY( "--no-progress : do not display the progress bar \n");
-    DISPLAY( "--      : All arguments after \"--\" are treated as files \n");
+    DISPLAYOUT( "\n");
+    DISPLAYOUT( "Advanced decompression arguments : \n");
+    DISPLAYOUT( " -l     : print information about zstd compressed files \n");
+    DISPLAYOUT( "--test  : test compressed file integrity \n");
+    DISPLAYOUT( " -M#    : Set a memory usage limit for decompression \n");
+# if ZSTD_SPARSE_DEFAULT
+    DISPLAYOUT( "--[no-]sparse : sparse mode (default: enabled on file, disabled on stdout) \n");
+# else
+    DISPLAYOUT( "--[no-]sparse : sparse mode (default: disabled) \n");
+# endif
+#endif  /* ZSTD_NODECOMPRESS */
+
 #ifndef ZSTD_NODICT
-    DISPLAY( "\n");
-    DISPLAY( "Dictionary builder : \n");
-    DISPLAY( "--train ## : create a dictionary from a training set of files \n");
-    DISPLAY( "--train-cover[=k=#,d=#,steps=#,split=#,shrink[=#]] : use the cover algorithm with optional args\n");
-    DISPLAY( "--train-fastcover[=k=#,d=#,f=#,steps=#,split=#,accel=#,shrink[=#]] : use the fast cover algorithm with optional args\n");
-    DISPLAY( "--train-legacy[=s=#] : use the legacy algorithm with selectivity (default: %u)\n", g_defaultSelectivityLevel);
-    DISPLAY( " -o file : `file` is dictionary name (default: %s) \n", g_defaultDictName);
-    DISPLAY( "--maxdict=# : limit dictionary to specified size (default: %u) \n", g_defaultMaxDictSize);
-    DISPLAY( "--dictID=# : force dictionary ID to specified value (default: random)\n");
+    DISPLAYOUT( "\n");
+    DISPLAYOUT( "Dictionary builder : \n");
+    DISPLAYOUT( "--train ## : create a dictionary from a training set of files \n");
+    DISPLAYOUT( "--train-cover[=k=#,d=#,steps=#,split=#,shrink[=#]] : use the cover algorithm with optional args \n");
+    DISPLAYOUT( "--train-fastcover[=k=#,d=#,f=#,steps=#,split=#,accel=#,shrink[=#]] : use the fast cover algorithm with optional args \n");
+    DISPLAYOUT( "--train-legacy[=s=#] : use the legacy algorithm with selectivity (default: %u) \n", g_defaultSelectivityLevel);
+    DISPLAYOUT( " -o DICT : DICT is dictionary name (default: %s) \n", g_defaultDictName);
+    DISPLAYOUT( "--maxdict=# : limit dictionary to specified size (default: %u) \n", g_defaultMaxDictSize);
+    DISPLAYOUT( "--dictID=# : force dictionary ID to specified value (default: random) \n");
 #endif
+
 #ifndef ZSTD_NOBENCH
-    DISPLAY( "\n");
-    DISPLAY( "Benchmark arguments : \n");
-    DISPLAY( " -b#    : benchmark file(s), using # compression level (default: %d) \n", ZSTDCLI_CLEVEL_DEFAULT);
-    DISPLAY( " -e#    : test all compression levels from -bX to # (default: 1)\n");
-    DISPLAY( " -i#    : minimum evaluation time in seconds (default: 3s) \n");
-    DISPLAY( " -B#    : cut file into independent blocks of size # (default: no block)\n");
-    DISPLAY( "--priority=rt : set process priority to real-time \n");
+    DISPLAYOUT( "\n");
+    DISPLAYOUT( "Benchmark arguments : \n");
+    DISPLAYOUT( " -b#    : benchmark file(s), using # compression level (default: %d) \n", ZSTDCLI_CLEVEL_DEFAULT);
+    DISPLAYOUT( " -e#    : test all compression levels successively from -b# to -e# (default: 1) \n");
+    DISPLAYOUT( " -i#    : minimum evaluation time in seconds (default: 3s) \n");
+    DISPLAYOUT( " -B#    : cut file into independent blocks of size # (default: no block) \n");
+    DISPLAYOUT( " -S     : output one benchmark result per input file (default: consolidated result) \n");
+    DISPLAYOUT( "--priority=rt : set process priority to real-time \n");
 #endif
-    return 0;
+
 }
 
-static int badusage(const char* programName)
+static void badusage(const char* programName)
 {
-    DISPLAYLEVEL(1, "Incorrect parameters\n");
-    if (g_displayLevel >= 2) usage(programName);
-    return 1;
+    DISPLAYLEVEL(1, "Incorrect parameters \n");
+    if (g_displayLevel >= 2) usage(stderr, programName);
 }
 
 static void waitEnter(void)
 {
     int unused;
-    DISPLAY("Press enter to continue...\n");
+    DISPLAY("Press enter to continue... \n");
     unused = getchar();
     (void)unused;
 }
@@ -249,10 +269,12 @@ static int readU32FromCharChecked(const char** stringPtr, unsigned* value)
 {
     unsigned result = 0;
     while ((**stringPtr >='0') && (**stringPtr <='9')) {
-        unsigned const max = (((unsigned)(-1)) / 10) - 1;
+        unsigned const max = ((unsigned)(-1)) / 10;
+        unsigned last = result;
         if (result > max) return 1; /* overflow error */
         result *= 10;
         result += (unsigned)(**stringPtr - '0');
+        if (result < last) return 1; /* overflow error */
         (*stringPtr)++ ;
     }
     if ((**stringPtr=='K') || (**stringPtr=='M')) {
@@ -277,12 +299,57 @@ static int readU32FromCharChecked(const char** stringPtr, unsigned* value)
  *  Will also modify `*stringPtr`, advancing it to position where it stopped reading.
  *  Note : function will exit() program if digit sequence overflows */
 static unsigned readU32FromChar(const char** stringPtr) {
-    static const char errorMsg[] = "error: numeric value too large";
+    static const char errorMsg[] = "error: numeric value overflows 32-bit unsigned int";
     unsigned result;
     if (readU32FromCharChecked(stringPtr, &result)) { errorOut(errorMsg); }
     return result;
 }
 
+/*! readSizeTFromCharChecked() :
+ * @return 0 if success, and store the result in *value.
+ *  allows and interprets K, KB, KiB, M, MB and MiB suffix.
+ *  Will also modify `*stringPtr`, advancing it to position where it stopped reading.
+ * @return 1 if an overflow error occurs */
+static int readSizeTFromCharChecked(const char** stringPtr, size_t* value)
+{
+    size_t result = 0;
+    while ((**stringPtr >='0') && (**stringPtr <='9')) {
+        size_t const max = ((size_t)(-1)) / 10;
+        size_t last = result;
+        if (result > max) return 1; /* overflow error */
+        result *= 10;
+        result += (size_t)(**stringPtr - '0');
+        if (result < last) return 1; /* overflow error */
+        (*stringPtr)++ ;
+    }
+    if ((**stringPtr=='K') || (**stringPtr=='M')) {
+        size_t const maxK = ((size_t)(-1)) >> 10;
+        if (result > maxK) return 1; /* overflow error */
+        result <<= 10;
+        if (**stringPtr=='M') {
+            if (result > maxK) return 1; /* overflow error */
+            result <<= 10;
+        }
+        (*stringPtr)++;  /* skip `K` or `M` */
+        if (**stringPtr=='i') (*stringPtr)++;
+        if (**stringPtr=='B') (*stringPtr)++;
+    }
+    *value = result;
+    return 0;
+}
+
+/*! readSizeTFromChar() :
+ * @return : size_t value read from input in `char` format.
+ *  allows and interprets K, KB, KiB, M, MB and MiB suffix.
+ *  Will also modify `*stringPtr`, advancing it to position where it stopped reading.
+ *  Note : function will exit() program if digit sequence overflows */
+static size_t readSizeTFromChar(const char** stringPtr) {
+    static const char errorMsg[] = "error: numeric value overflows size_t";
+    size_t result;
+    if (readSizeTFromCharChecked(stringPtr, &result)) { errorOut(errorMsg); }
+    return result;
+}
+
 /** longCommandWArg() :
  *  check if *stringPtr is the same as longCommand.
  *  If yes, @return 1 and advances *stringPtr to the position which immediately follows longCommand.
@@ -477,42 +544,45 @@ static unsigned parseCompressionParameters(const char* stringPtr, ZSTD_compressi
 
 static void printVersion(void)
 {
-    DISPLAY(WELCOME_MESSAGE);
+    DISPLAYOUT(WELCOME_MESSAGE);
+    if (g_displayLevel >= 3) {
     /* format support */
-    DISPLAYLEVEL(3, "*** supports: zstd");
-#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT>0) && (ZSTD_LEGACY_SUPPORT<8)
-    DISPLAYLEVEL(3, ", zstd legacy v0.%d+", ZSTD_LEGACY_SUPPORT);
-#endif
-#ifdef ZSTD_GZCOMPRESS
-    DISPLAYLEVEL(3, ", gzip");
-#endif
-#ifdef ZSTD_LZ4COMPRESS
-    DISPLAYLEVEL(3, ", lz4");
-#endif
-#ifdef ZSTD_LZMACOMPRESS
-    DISPLAYLEVEL(3, ", lzma, xz ");
-#endif
-    DISPLAYLEVEL(3, "\n");
-    /* posix support */
-#ifdef _POSIX_C_SOURCE
-    DISPLAYLEVEL(4, "_POSIX_C_SOURCE defined: %ldL\n", (long) _POSIX_C_SOURCE);
-#endif
-#ifdef _POSIX_VERSION
-    DISPLAYLEVEL(4, "_POSIX_VERSION defined: %ldL \n", (long) _POSIX_VERSION);
-#endif
-#ifdef PLATFORM_POSIX_VERSION
-    DISPLAYLEVEL(4, "PLATFORM_POSIX_VERSION defined: %ldL\n", (long) PLATFORM_POSIX_VERSION);
-#endif
+        DISPLAYOUT("*** supports: zstd");
+    #if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT>0) && (ZSTD_LEGACY_SUPPORT<8)
+        DISPLAYOUT(", zstd legacy v0.%d+", ZSTD_LEGACY_SUPPORT);
+    #endif
+    #ifdef ZSTD_GZCOMPRESS
+        DISPLAYOUT(", gzip");
+    #endif
+    #ifdef ZSTD_LZ4COMPRESS
+        DISPLAYOUT(", lz4");
+    #endif
+    #ifdef ZSTD_LZMACOMPRESS
+        DISPLAYOUT(", lzma, xz ");
+    #endif
+        DISPLAYOUT("\n");
+        if (g_displayLevel >= 4) {
+            /* posix support */
+        #ifdef _POSIX_C_SOURCE
+            DISPLAYOUT("_POSIX_C_SOURCE defined: %ldL\n", (long) _POSIX_C_SOURCE);
+        #endif
+        #ifdef _POSIX_VERSION
+            DISPLAYOUT("_POSIX_VERSION defined: %ldL \n", (long) _POSIX_VERSION);
+        #endif
+        #ifdef PLATFORM_POSIX_VERSION
+            DISPLAYOUT("PLATFORM_POSIX_VERSION defined: %ldL\n", (long) PLATFORM_POSIX_VERSION);
+        #endif
+    }   }
 }
 
 /* Environment variables for parameter setting */
 #define ENV_CLEVEL "ZSTD_CLEVEL"
 
-/* functions that pick up environment variables */
+/* pick up environment variable */
 static int init_cLevel(void) {
     const char* const env = getenv(ENV_CLEVEL);
-    if (env) {
-        const char *ptr = env;
+    if (env != NULL) {
+        const char* ptr = env;
         int sign = 1;
         if (*ptr == '-') {
             sign = -1;
@@ -524,33 +594,38 @@ static int init_cLevel(void) {
         if ((*ptr>='0') && (*ptr<='9')) {
             unsigned absLevel;
             if (readU32FromCharChecked(&ptr, &absLevel)) {
-                DISPLAYLEVEL(2, "Ignore environment variable setting %s=%s: numeric value too large\n", ENV_CLEVEL, env);
+                DISPLAYLEVEL(2, "Ignore environment variable setting %s=%s: numeric value too large \n", ENV_CLEVEL, env);
                 return ZSTDCLI_CLEVEL_DEFAULT;
             } else if (*ptr == 0) {
                 return sign * (int)absLevel;
-            }
-        }
+        }   }
 
-        DISPLAYLEVEL(2, "Ignore environment variable setting %s=%s: not a valid integer value\n", ENV_CLEVEL, env);
+        DISPLAYLEVEL(2, "Ignore environment variable setting %s=%s: not a valid integer value \n", ENV_CLEVEL, env);
     }
 
     return ZSTDCLI_CLEVEL_DEFAULT;
 }
 
+#define ZSTD_NB_STRATEGIES 9
+
+static const char* ZSTD_strategyMap[ZSTD_NB_STRATEGIES + 1] = { "", "ZSTD_fast",
+                "ZSTD_dfast", "ZSTD_greedy", "ZSTD_lazy", "ZSTD_lazy2", "ZSTD_btlazy2",
+                "ZSTD_btopt", "ZSTD_btultra", "ZSTD_btultra2"};
+
 typedef enum { zom_compress, zom_decompress, zom_test, zom_bench, zom_train, zom_list } zstd_operation_mode;
 
 #define CLEAN_RETURN(i) { operationResult = (i); goto _end; }
 
 #ifdef ZSTD_NOCOMPRESS
 /* symbols from compression library are not defined and should not be invoked */
-# define MINCLEVEL  -50
+# define MINCLEVEL  -99
 # define MAXCLEVEL   22
 #else
 # define MINCLEVEL  ZSTD_minCLevel()
 # define MAXCLEVEL  ZSTD_maxCLevel()
 #endif
 
-int main(int argCount, const char* argv[])
+int main(int const argCount, const char* argv[])
 {
     int argNb,
         followLinks = 0,
@@ -573,7 +648,9 @@ int main(int argCount, const char* argv[])
         separateFiles = 0,
         setRealTimePrio = 0,
         singleThread = 0,
-        ultra=0;
+        showDefaultCParams = 0,
+        ultra=0,
+        contentSize=1;
     double compressibility = 0.5;
     unsigned bench_nbSeconds = 3;   /* would be better if this value was synchronized from bench */
     size_t blockSize = 0;
@@ -581,16 +658,17 @@ int main(int argCount, const char* argv[])
     FIO_prefs_t* const prefs = FIO_createPreferences();
     zstd_operation_mode operation = zom_compress;
     ZSTD_compressionParameters compressionParams;
-    int cLevel;
-    int cLevelLast = -1000000000;
+    int cLevel = init_cLevel();
+    int cLevelLast = MINCLEVEL - 1;  /* lower than minimum */
     unsigned recursive = 0;
     unsigned memLimit = 0;
-    const char** filenameTable = (const char**)malloc((size_t)argCount * sizeof(const char*));   /* argCount >= 1 */
-    unsigned filenameIdx = 0;
+    FileNamesTable* filenames = UTIL_allocateFileNamesTable((size_t)argCount);  /* argCount >= 1 */
+    FileNamesTable* file_of_names = UTIL_allocateFileNamesTable((size_t)argCount);  /* argCount >= 1 */
     const char* programName = argv[0];
     const char* outFileName = NULL;
     const char* outDirName = NULL;
     const char* dictFileName = NULL;
+    const char* patchFromDictFileName = NULL;
     const char* suffix = ZSTD_EXTENSION;
     unsigned maxDictSize = g_defaultMaxDictSize;
     unsigned dictID = 0;
@@ -599,11 +677,6 @@ int main(int argCount, const char* argv[])
     size_t srcSizeHint = 0;
     int dictCLevel = g_defaultDictCLevel;
     unsigned dictSelect = g_defaultSelectivityLevel;
-#ifdef UTIL_HAS_CREATEFILELIST
-    const char** extendedFileList = NULL;
-    char* fileNamesBuf = NULL;
-    unsigned fileNamesNb;
-#endif
 #ifndef ZSTD_NODICT
     ZDICT_cover_params_t coverParams = defaultCoverParams();
     ZDICT_fastCover_params_t fastCoverParams = defaultFastCoverParams();
@@ -617,11 +690,9 @@ int main(int argCount, const char* argv[])
 
     /* init */
     (void)recursive; (void)cLevelLast;    /* not used when ZSTD_NOBENCH set */
-    (void)memLimit;   /* not used when ZSTD_NODECOMPRESS set */
-    if (filenameTable==NULL) { DISPLAY("zstd: %s \n", strerror(errno)); exit(1); }
-    filenameTable[0] = stdinmark;
-    g_displayOut = stderr;
-    cLevel = init_cLevel();
+    (void)memLimit;
+    assert(argCount >= 1);
+    if ((filenames==NULL) || (file_of_names==NULL)) { DISPLAY("zstd: allocation error \n"); exit(1); }
     programName = lastNameFromPath(programName);
 #ifdef ZSTD_MULTITHREAD
     nbWorkers = 1;
@@ -649,312 +720,324 @@ int main(int argCount, const char* argv[])
     /* command switches */
     for (argNb=1; argNb<argCount; argNb++) {
         const char* argument = argv[argNb];
-        if(!argument) continue;   /* Protection if argument empty */
-
-        if (nextArgumentsAreFiles==0) {
-            /* "-" means stdin/stdout */
-            if (!strcmp(argument, "-")){
-                if (!filenameIdx) {
-                    filenameIdx=1, filenameTable[0]=stdinmark;
-                    outFileName=stdoutmark;
-                    g_displayLevel-=(g_displayLevel==2);
-                    continue;
-            }   }
+        if (!argument) continue;   /* Protection if argument empty */
 
-            /* Decode commands (note : aggregated commands are allowed) */
-            if (argument[0]=='-') {
-
-                if (argument[1]=='-') {
-                    /* long commands (--long-word) */
-                    if (!strcmp(argument, "--")) { nextArgumentsAreFiles=1; continue; }   /* only file names allowed from now on */
-                    if (!strcmp(argument, "--list")) { operation=zom_list; continue; }
-                    if (!strcmp(argument, "--compress")) { operation=zom_compress; continue; }
-                    if (!strcmp(argument, "--decompress")) { operation=zom_decompress; continue; }
-                    if (!strcmp(argument, "--uncompress")) { operation=zom_decompress; continue; }
-                    if (!strcmp(argument, "--force")) { FIO_overwriteMode(prefs); forceStdout=1; followLinks=1; continue; }
-                    if (!strcmp(argument, "--version")) { g_displayOut=stdout; DISPLAY(WELCOME_MESSAGE); CLEAN_RETURN(0); }
-                    if (!strcmp(argument, "--help")) { g_displayOut=stdout; CLEAN_RETURN(usage_advanced(programName)); }
-                    if (!strcmp(argument, "--verbose")) { g_displayLevel++; continue; }
-                    if (!strcmp(argument, "--quiet")) { g_displayLevel--; continue; }
-                    if (!strcmp(argument, "--stdout")) { forceStdout=1; outFileName=stdoutmark; g_displayLevel-=(g_displayLevel==2); continue; }
-                    if (!strcmp(argument, "--ultra")) { ultra=1; continue; }
-                    if (!strcmp(argument, "--check")) { FIO_setChecksumFlag(prefs, 2); continue; }
-                    if (!strcmp(argument, "--no-check")) { FIO_setChecksumFlag(prefs, 0); continue; }
-                    if (!strcmp(argument, "--sparse")) { FIO_setSparseWrite(prefs, 2); continue; }
-                    if (!strcmp(argument, "--no-sparse")) { FIO_setSparseWrite(prefs, 0); continue; }
-                    if (!strcmp(argument, "--test")) { operation=zom_test; continue; }
-                    if (!strcmp(argument, "--train")) { operation=zom_train; if (outFileName==NULL) outFileName=g_defaultDictName; continue; }
-                    if (!strcmp(argument, "--maxdict")) { nextArgumentIsMaxDict=1; lastCommand=1; continue; }  /* kept available for compatibility with old syntax ; will be removed one day */
-                    if (!strcmp(argument, "--dictID")) { nextArgumentIsDictID=1; lastCommand=1; continue; }  /* kept available for compatibility with old syntax ; will be removed one day */
-                    if (!strcmp(argument, "--no-dictID")) { FIO_setDictIDFlag(prefs, 0); continue; }
-                    if (!strcmp(argument, "--keep")) { FIO_setRemoveSrcFile(prefs, 0); continue; }
-                    if (!strcmp(argument, "--rm")) { FIO_setRemoveSrcFile(prefs, 1); continue; }
-                    if (!strcmp(argument, "--priority=rt")) { setRealTimePrio = 1; continue; }
-                    if (!strcmp(argument, "--output-dir-flat")) {nextArgumentIsOutDirName=1; lastCommand=1; continue; }
-                    if (!strcmp(argument, "--adapt")) { adapt = 1; continue; }
-                    if (longCommandWArg(&argument, "--adapt=")) { adapt = 1; if (!parseAdaptParameters(argument, &adaptMin, &adaptMax)) CLEAN_RETURN(badusage(programName)); continue; }
-                    if (!strcmp(argument, "--single-thread")) { nbWorkers = 0; singleThread = 1; continue; }
-                    if (!strcmp(argument, "--format=zstd")) { suffix = ZSTD_EXTENSION; FIO_setCompressionType(prefs, FIO_zstdCompression); continue; }
+        if (nextArgumentsAreFiles) {
+            UTIL_refFilename(filenames, argument);
+            continue;
+        }
+
+        /* "-" means stdin/stdout */
+        if (!strcmp(argument, "-")){
+            UTIL_refFilename(filenames, stdinmark);
+            continue;
+        }
+
+        /* Decode commands (note : aggregated commands are allowed) */
+        if (argument[0]=='-') {
+
+            if (argument[1]=='-') {
+                /* long commands (--long-word) */
+                if (!strcmp(argument, "--")) { nextArgumentsAreFiles=1; continue; }   /* only file names allowed from now on */
+                if (!strcmp(argument, "--list")) { operation=zom_list; continue; }
+                if (!strcmp(argument, "--compress")) { operation=zom_compress; continue; }
+                if (!strcmp(argument, "--decompress")) { operation=zom_decompress; continue; }
+                if (!strcmp(argument, "--uncompress")) { operation=zom_decompress; continue; }
+                if (!strcmp(argument, "--force")) { FIO_overwriteMode(prefs); forceStdout=1; followLinks=1; continue; }
+                if (!strcmp(argument, "--version")) { printVersion(); CLEAN_RETURN(0); }
+                if (!strcmp(argument, "--help")) { usage_advanced(programName); CLEAN_RETURN(0); }
+                if (!strcmp(argument, "--verbose")) { g_displayLevel++; continue; }
+                if (!strcmp(argument, "--quiet")) { g_displayLevel--; continue; }
+                if (!strcmp(argument, "--stdout")) { forceStdout=1; outFileName=stdoutmark; g_displayLevel-=(g_displayLevel==2); continue; }
+                if (!strcmp(argument, "--ultra")) { ultra=1; continue; }
+                if (!strcmp(argument, "--check")) { FIO_setChecksumFlag(prefs, 2); continue; }
+                if (!strcmp(argument, "--no-check")) { FIO_setChecksumFlag(prefs, 0); continue; }
+                if (!strcmp(argument, "--sparse")) { FIO_setSparseWrite(prefs, 2); continue; }
+                if (!strcmp(argument, "--no-sparse")) { FIO_setSparseWrite(prefs, 0); continue; }
+                if (!strcmp(argument, "--test")) { operation=zom_test; continue; }
+                if (!strcmp(argument, "--train")) { operation=zom_train; if (outFileName==NULL) outFileName=g_defaultDictName; continue; }
+                if (!strcmp(argument, "--maxdict")) { nextArgumentIsMaxDict=1; lastCommand=1; continue; }  /* kept available for compatibility with old syntax ; will be removed one day */
+                if (!strcmp(argument, "--dictID")) { nextArgumentIsDictID=1; lastCommand=1; continue; }  /* kept available for compatibility with old syntax ; will be removed one day */
+                if (!strcmp(argument, "--no-dictID")) { FIO_setDictIDFlag(prefs, 0); continue; }
+                if (!strcmp(argument, "--keep")) { FIO_setRemoveSrcFile(prefs, 0); continue; }
+                if (!strcmp(argument, "--rm")) { FIO_setRemoveSrcFile(prefs, 1); continue; }
+                if (!strcmp(argument, "--priority=rt")) { setRealTimePrio = 1; continue; }
+                if (!strcmp(argument, "--output-dir-flat")) {nextArgumentIsOutDirName=1; lastCommand=1; continue; }
+                if (!strcmp(argument, "--show-default-cparams")) { showDefaultCParams = 1; continue; }
+                if (!strcmp(argument, "--content-size")) { contentSize = 1; continue; }
+                if (!strcmp(argument, "--no-content-size")) { contentSize = 0; continue; }
+                if (!strcmp(argument, "--adapt")) { adapt = 1; continue; }
+                if (longCommandWArg(&argument, "--adapt=")) { adapt = 1; if (!parseAdaptParameters(argument, &adaptMin, &adaptMax)) { badusage(programName); CLEAN_RETURN(1); } continue; }
+                if (!strcmp(argument, "--single-thread")) { nbWorkers = 0; singleThread = 1; continue; }
+                if (!strcmp(argument, "--format=zstd")) { suffix = ZSTD_EXTENSION; FIO_setCompressionType(prefs, FIO_zstdCompression); continue; }
 #ifdef ZSTD_GZCOMPRESS
-                    if (!strcmp(argument, "--format=gzip")) { suffix = GZ_EXTENSION; FIO_setCompressionType(prefs, FIO_gzipCompression); continue; }
+                if (!strcmp(argument, "--format=gzip")) { suffix = GZ_EXTENSION; FIO_setCompressionType(prefs, FIO_gzipCompression); continue; }
 #endif
 #ifdef ZSTD_LZMACOMPRESS
-                    if (!strcmp(argument, "--format=lzma")) { suffix = LZMA_EXTENSION; FIO_setCompressionType(prefs, FIO_lzmaCompression);  continue; }
-                    if (!strcmp(argument, "--format=xz")) { suffix = XZ_EXTENSION; FIO_setCompressionType(prefs, FIO_xzCompression);  continue; }
+                if (!strcmp(argument, "--format=lzma")) { suffix = LZMA_EXTENSION; FIO_setCompressionType(prefs, FIO_lzmaCompression);  continue; }
+                if (!strcmp(argument, "--format=xz")) { suffix = XZ_EXTENSION; FIO_setCompressionType(prefs, FIO_xzCompression);  continue; }
 #endif
 #ifdef ZSTD_LZ4COMPRESS
-                    if (!strcmp(argument, "--format=lz4")) { suffix = LZ4_EXTENSION; FIO_setCompressionType(prefs, FIO_lz4Compression);  continue; }
+                if (!strcmp(argument, "--format=lz4")) { suffix = LZ4_EXTENSION; FIO_setCompressionType(prefs, FIO_lz4Compression);  continue; }
 #endif
-                    if (!strcmp(argument, "--rsyncable")) { rsyncable = 1; continue; }
-                    if (!strcmp(argument, "--compress-literals")) { literalCompressionMode = ZSTD_lcm_huffman; continue; }
-                    if (!strcmp(argument, "--no-compress-literals")) { literalCompressionMode = ZSTD_lcm_uncompressed; continue; }
-                    if (!strcmp(argument, "--no-progress")) { FIO_setNoProgress(1); continue; }
-                    if (!strcmp(argument, "--exclude-compressed")) { FIO_setExcludeCompressedFile(prefs, 1); continue; }
-                    /* long commands with arguments */
+                if (!strcmp(argument, "--rsyncable")) { rsyncable = 1; continue; }
+                if (!strcmp(argument, "--compress-literals")) { literalCompressionMode = ZSTD_lcm_huffman; continue; }
+                if (!strcmp(argument, "--no-compress-literals")) { literalCompressionMode = ZSTD_lcm_uncompressed; continue; }
+                if (!strcmp(argument, "--no-progress")) { FIO_setNoProgress(1); continue; }
+                if (!strcmp(argument, "--exclude-compressed")) { FIO_setExcludeCompressedFile(prefs, 1); continue; }
+                /* long commands with arguments */
 #ifndef ZSTD_NODICT
-                    if (longCommandWArg(&argument, "--train-cover")) {
-                      operation = zom_train;
-                      if (outFileName == NULL)
-                          outFileName = g_defaultDictName;
-                      dict = cover;
-                      /* Allow optional arguments following an = */
-                      if (*argument == 0) { memset(&coverParams, 0, sizeof(coverParams)); }
-                      else if (*argument++ != '=') { CLEAN_RETURN(badusage(programName)); }
-                      else if (!parseCoverParameters(argument, &coverParams)) { CLEAN_RETURN(badusage(programName)); }
-                      continue;
-                    }
-                    if (longCommandWArg(&argument, "--train-fastcover")) {
-                      operation = zom_train;
-                      if (outFileName == NULL)
-                          outFileName = g_defaultDictName;
-                      dict = fastCover;
-                      /* Allow optional arguments following an = */
-                      if (*argument == 0) { memset(&fastCoverParams, 0, sizeof(fastCoverParams)); }
-                      else if (*argument++ != '=') { CLEAN_RETURN(badusage(programName)); }
-                      else if (!parseFastCoverParameters(argument, &fastCoverParams)) { CLEAN_RETURN(badusage(programName)); }
-                      continue;
-                    }
-                    if (longCommandWArg(&argument, "--train-legacy")) {
-                      operation = zom_train;
-                      if (outFileName == NULL)
-                          outFileName = g_defaultDictName;
-                      dict = legacy;
-                      /* Allow optional arguments following an = */
-                      if (*argument == 0) { continue; }
-                      else if (*argument++ != '=') { CLEAN_RETURN(badusage(programName)); }
-                      else if (!parseLegacyParameters(argument, &dictSelect)) { CLEAN_RETURN(badusage(programName)); }
-                      continue;
-                    }
+                if (longCommandWArg(&argument, "--train-cover")) {
+                  operation = zom_train;
+                  if (outFileName == NULL)
+                      outFileName = g_defaultDictName;
+                  dict = cover;
+                  /* Allow optional arguments following an = */
+                  if (*argument == 0) { memset(&coverParams, 0, sizeof(coverParams)); }
+                  else if (*argument++ != '=') { badusage(programName); CLEAN_RETURN(1); }
+                  else if (!parseCoverParameters(argument, &coverParams)) { badusage(programName); CLEAN_RETURN(1); }
+                  continue;
+                }
+                if (longCommandWArg(&argument, "--train-fastcover")) {
+                  operation = zom_train;
+                  if (outFileName == NULL)
+                      outFileName = g_defaultDictName;
+                  dict = fastCover;
+                  /* Allow optional arguments following an = */
+                  if (*argument == 0) { memset(&fastCoverParams, 0, sizeof(fastCoverParams)); }
+                  else if (*argument++ != '=') { badusage(programName); CLEAN_RETURN(1); }
+                  else if (!parseFastCoverParameters(argument, &fastCoverParams)) { badusage(programName); CLEAN_RETURN(1); }
+                  continue;
+                }
+                if (longCommandWArg(&argument, "--train-legacy")) {
+                  operation = zom_train;
+                  if (outFileName == NULL)
+                      outFileName = g_defaultDictName;
+                  dict = legacy;
+                  /* Allow optional arguments following an = */
+                  if (*argument == 0) { continue; }
+                  else if (*argument++ != '=') { badusage(programName); CLEAN_RETURN(1); }
+                  else if (!parseLegacyParameters(argument, &dictSelect)) { badusage(programName); CLEAN_RETURN(1); }
+                  continue;
+                }
 #endif
-                    if (longCommandWArg(&argument, "--threads=")) { nbWorkers = (int)readU32FromChar(&argument); continue; }
-                    if (longCommandWArg(&argument, "--memlimit=")) { memLimit = readU32FromChar(&argument); continue; }
-                    if (longCommandWArg(&argument, "--memory=")) { memLimit = readU32FromChar(&argument); continue; }
-                    if (longCommandWArg(&argument, "--memlimit-decompress=")) { memLimit = readU32FromChar(&argument); continue; }
-                    if (longCommandWArg(&argument, "--block-size=")) { blockSize = readU32FromChar(&argument); continue; }
-                    if (longCommandWArg(&argument, "--maxdict=")) { maxDictSize = readU32FromChar(&argument); continue; }
-                    if (longCommandWArg(&argument, "--dictID=")) { dictID = readU32FromChar(&argument); continue; }
-                    if (longCommandWArg(&argument, "--zstd=")) { if (!parseCompressionParameters(argument, &compressionParams)) CLEAN_RETURN(badusage(programName)); continue; }
-                    if (longCommandWArg(&argument, "--stream-size=")) { streamSrcSize = readU32FromChar(&argument); continue; }
-                    if (longCommandWArg(&argument, "--target-compressed-block-size=")) { targetCBlockSize = readU32FromChar(&argument); continue; }
-                    if (longCommandWArg(&argument, "--size-hint=")) { srcSizeHint = readU32FromChar(&argument); continue; }
-                    if (longCommandWArg(&argument, "--output-dir-flat=")) { outDirName = argument; continue; }
-                    if (longCommandWArg(&argument, "--long")) {
-                        unsigned ldmWindowLog = 0;
-                        ldmFlag = 1;
-                        /* Parse optional window log */
-                        if (*argument == '=') {
-                            ++argument;
-                            ldmWindowLog = readU32FromChar(&argument);
-                        } else if (*argument != 0) {
-                            /* Invalid character following --long */
-                            CLEAN_RETURN(badusage(programName));
-                        }
-                        /* Only set windowLog if not already set by --zstd */
-                        if (compressionParams.windowLog == 0)
-                            compressionParams.windowLog = ldmWindowLog;
-                        continue;
+                if (longCommandWArg(&argument, "--threads=")) { nbWorkers = (int)readU32FromChar(&argument); continue; }
+                if (longCommandWArg(&argument, "--memlimit=")) { memLimit = readU32FromChar(&argument); continue; }
+                if (longCommandWArg(&argument, "--memory=")) { memLimit = readU32FromChar(&argument); continue; }
+                if (longCommandWArg(&argument, "--memlimit-decompress=")) { memLimit = readU32FromChar(&argument); continue; }
+                if (longCommandWArg(&argument, "--block-size=")) { blockSize = readSizeTFromChar(&argument); continue; }
+                if (longCommandWArg(&argument, "--maxdict=")) { maxDictSize = readU32FromChar(&argument); continue; }
+                if (longCommandWArg(&argument, "--dictID=")) { dictID = readU32FromChar(&argument); continue; }
+                if (longCommandWArg(&argument, "--zstd=")) { if (!parseCompressionParameters(argument, &compressionParams)) { badusage(programName); CLEAN_RETURN(1); } continue; }
+                if (longCommandWArg(&argument, "--stream-size=")) { streamSrcSize = readSizeTFromChar(&argument); continue; }
+                if (longCommandWArg(&argument, "--target-compressed-block-size=")) { targetCBlockSize = readSizeTFromChar(&argument); continue; }
+                if (longCommandWArg(&argument, "--size-hint=")) { srcSizeHint = readSizeTFromChar(&argument); continue; }
+                if (longCommandWArg(&argument, "--output-dir-flat=")) { outDirName = argument; continue; }
+                if (longCommandWArg(&argument, "--patch-from=")) { patchFromDictFileName = argument; continue; }
+                if (longCommandWArg(&argument, "--long")) {
+                    unsigned ldmWindowLog = 0;
+                    ldmFlag = 1;
+                    /* Parse optional window log */
+                    if (*argument == '=') {
+                        ++argument;
+                        ldmWindowLog = readU32FromChar(&argument);
+                    } else if (*argument != 0) {
+                        /* Invalid character following --long */
+                        badusage(programName);
+                        CLEAN_RETURN(1);
                     }
+                    /* Only set windowLog if not already set by --zstd */
+                    if (compressionParams.windowLog == 0)
+                        compressionParams.windowLog = ldmWindowLog;
+                    continue;
+                }
 #ifndef ZSTD_NOCOMPRESS   /* linking ZSTD_minCLevel() requires compression support */
-                    if (longCommandWArg(&argument, "--fast")) {
-                        /* Parse optional acceleration factor */
-                        if (*argument == '=') {
-                            U32 const maxFast = (U32)-ZSTD_minCLevel();
-                            U32 fastLevel;
-                            ++argument;
-                            fastLevel = readU32FromChar(&argument);
-                            if (fastLevel > maxFast) fastLevel = maxFast;
-                            if (fastLevel) {
-                              dictCLevel = cLevel = -(int)fastLevel;
-                            } else {
-                              CLEAN_RETURN(badusage(programName));
-                            }
-                        } else if (*argument != 0) {
-                            /* Invalid character following --fast */
-                            CLEAN_RETURN(badusage(programName));
+                if (longCommandWArg(&argument, "--fast")) {
+                    /* Parse optional acceleration factor */
+                    if (*argument == '=') {
+                        U32 const maxFast = (U32)-ZSTD_minCLevel();
+                        U32 fastLevel;
+                        ++argument;
+                        fastLevel = readU32FromChar(&argument);
+                        if (fastLevel > maxFast) fastLevel = maxFast;
+                        if (fastLevel) {
+                            dictCLevel = cLevel = -(int)fastLevel;
                         } else {
-                            cLevel = -1;  /* default for --fast */
+                            badusage(programName);
+                            CLEAN_RETURN(1);
                         }
-                        continue;
+                    } else if (*argument != 0) {
+                        /* Invalid character following --fast */
+                        badusage(programName);
+                        CLEAN_RETURN(1);
+                    } else {
+                        cLevel = -1;  /* default for --fast */
                     }
+                    continue;
+                }
 #endif
-                    /* fall-through, will trigger bad_usage() later on */
+
+                if (longCommandWArg(&argument, "--filelist=")) {
+                    UTIL_refFilename(file_of_names, argument);
+                    continue;
                 }
 
-                argument++;
-                while (argument[0]!=0) {
-                    if (lastCommand) {
-                        DISPLAY("error : command must be followed by argument \n");
-                        CLEAN_RETURN(1);
-                    }
+                /* fall-through, will trigger bad_usage() later on */
+            }
+
+            argument++;
+            while (argument[0]!=0) {
+                if (lastCommand) {
+                    DISPLAY("error : command must be followed by argument \n");
+                    CLEAN_RETURN(1);
+                }
 #ifndef ZSTD_NOCOMPRESS
-                    /* compression Level */
-                    if ((*argument>='0') && (*argument<='9')) {
-                        dictCLevel = cLevel = (int)readU32FromChar(&argument);
-                        continue;
-                    }
+                /* compression Level */
+                if ((*argument>='0') && (*argument<='9')) {
+                    dictCLevel = cLevel = (int)readU32FromChar(&argument);
+                    continue;
+                }
 #endif
 
-                    switch(argument[0])
-                    {
-                        /* Display help */
-                    case 'V': g_displayOut=stdout; printVersion(); CLEAN_RETURN(0);   /* Version Only */
-                    case 'H':
-                    case 'h': g_displayOut=stdout; CLEAN_RETURN(usage_advanced(programName));
+                switch(argument[0])
+                {
+                    /* Display help */
+                case 'V': printVersion(); CLEAN_RETURN(0);   /* Version Only */
+                case 'H':
+                case 'h': usage_advanced(programName); CLEAN_RETURN(0);
 
-                         /* Compress */
-                    case 'z': operation=zom_compress; argument++; break;
+                     /* Compress */
+                case 'z': operation=zom_compress; argument++; break;
 
-                         /* Decoding */
-                    case 'd':
+                     /* Decoding */
+                case 'd':
 #ifndef ZSTD_NOBENCH
-                            benchParams.mode = BMK_decodeOnly;
-                            if (operation==zom_bench) { argument++; break; }  /* benchmark decode (hidden option) */
+                        benchParams.mode = BMK_decodeOnly;
+                        if (operation==zom_bench) { argument++; break; }  /* benchmark decode (hidden option) */
 #endif
-                            operation=zom_decompress; argument++; break;
+                        operation=zom_decompress; argument++; break;
 
-                        /* Force stdout, even if stdout==console */
-                    case 'c': forceStdout=1; outFileName=stdoutmark; argument++; break;
+                    /* Force stdout, even if stdout==console */
+                case 'c': forceStdout=1; outFileName=stdoutmark; argument++; break;
 
-                        /* Use file content as dictionary */
-                    case 'D': nextEntryIsDictionary = 1; lastCommand = 1; argument++; break;
+                    /* Use file content as dictionary */
+                case 'D': nextEntryIsDictionary = 1; lastCommand = 1; argument++; break;
 
-                        /* Overwrite */
-                    case 'f': FIO_overwriteMode(prefs); forceStdout=1; followLinks=1; argument++; break;
+                    /* Overwrite */
+                case 'f': FIO_overwriteMode(prefs); forceStdout=1; followLinks=1; argument++; break;
 
-                        /* Verbose mode */
-                    case 'v': g_displayLevel++; argument++; break;
+                    /* Verbose mode */
+                case 'v': g_displayLevel++; argument++; break;
 
-                        /* Quiet mode */
-                    case 'q': g_displayLevel--; argument++; break;
+                    /* Quiet mode */
+                case 'q': g_displayLevel--; argument++; break;
 
-                        /* keep source file (default) */
-                    case 'k': FIO_setRemoveSrcFile(prefs, 0); argument++; break;
+                    /* keep source file (default) */
+                case 'k': FIO_setRemoveSrcFile(prefs, 0); argument++; break;
 
-                        /* Checksum */
-                    case 'C': FIO_setChecksumFlag(prefs, 2); argument++; break;
+                    /* Checksum */
+                case 'C': FIO_setChecksumFlag(prefs, 2); argument++; break;
 
-                        /* test compressed file */
-                    case 't': operation=zom_test; argument++; break;
+                    /* test compressed file */
+                case 't': operation=zom_test; argument++; break;
 
-                        /* destination file name */
-                    case 'o': nextArgumentIsOutFileName=1; lastCommand=1; argument++; break;
+                    /* destination file name */
+                case 'o': nextArgumentIsOutFileName=1; lastCommand=1; argument++; break;
 
-                        /* limit decompression memory */
-                    case 'M':
-                        argument++;
-                        memLimit = readU32FromChar(&argument);
-                        break;
-                    case 'l': operation=zom_list; argument++; break;
+                    /* limit memory */
+                case 'M':
+                    argument++;
+                    memLimit = readU32FromChar(&argument);
+                    break;
+                case 'l': operation=zom_list; argument++; break;
 #ifdef UTIL_HAS_CREATEFILELIST
-                        /* recursive */
-                    case 'r': recursive=1; argument++; break;
+                    /* recursive */
+                case 'r': recursive=1; argument++; break;
 #endif
 
 #ifndef ZSTD_NOBENCH
-                        /* Benchmark */
-                    case 'b':
-                        operation=zom_bench;
-                        argument++;
-                        break;
-
-                        /* range bench (benchmark only) */
-                    case 'e':
-                        /* compression Level */
-                        argument++;
-                        cLevelLast = (int)readU32FromChar(&argument);
-                        break;
-
-                        /* Modify Nb Iterations (benchmark only) */
-                    case 'i':
-                        argument++;
-                        bench_nbSeconds = readU32FromChar(&argument);
-                        break;
-
-                        /* cut input into blocks (benchmark only) */
-                    case 'B':
-                        argument++;
-                        blockSize = readU32FromChar(&argument);
-                        break;
-
-                        /* benchmark files separately (hidden option) */
-                    case 'S':
-                        argument++;
-                        separateFiles = 1;
-                        break;
+                    /* Benchmark */
+                case 'b':
+                    operation=zom_bench;
+                    argument++;
+                    break;
+
+                    /* range bench (benchmark only) */
+                case 'e':
+                    /* compression Level */
+                    argument++;
+                    cLevelLast = (int)readU32FromChar(&argument);
+                    break;
+
+                    /* Modify Nb Iterations (benchmark only) */
+                case 'i':
+                    argument++;
+                    bench_nbSeconds = readU32FromChar(&argument);
+                    break;
+
+                    /* cut input into blocks (benchmark only) */
+                case 'B':
+                    argument++;
+                    blockSize = readU32FromChar(&argument);
+                    break;
+
+                    /* benchmark files separately (hidden option) */
+                case 'S':
+                    argument++;
+                    separateFiles = 1;
+                    break;
 
 #endif   /* ZSTD_NOBENCH */
 
-                        /* nb of threads (hidden option) */
-                    case 'T':
-                        argument++;
-                        nbWorkers = (int)readU32FromChar(&argument);
-                        break;
+                    /* nb of threads (hidden option) */
+                case 'T':
+                    argument++;
+                    nbWorkers = (int)readU32FromChar(&argument);
+                    break;
 
-                        /* Dictionary Selection level */
-                    case 's':
-                        argument++;
-                        dictSelect = readU32FromChar(&argument);
-                        break;
+                    /* Dictionary Selection level */
+                case 's':
+                    argument++;
+                    dictSelect = readU32FromChar(&argument);
+                    break;
 
-                        /* Pause at the end (-p) or set an additional param (-p#) (hidden option) */
-                    case 'p': argument++;
+                    /* Pause at the end (-p) or set an additional param (-p#) (hidden option) */
+                case 'p': argument++;
 #ifndef ZSTD_NOBENCH
-                        if ((*argument>='0') && (*argument<='9')) {
-                            benchParams.additionalParam = (int)readU32FromChar(&argument);
-                        } else
+                    if ((*argument>='0') && (*argument<='9')) {
+                        benchParams.additionalParam = (int)readU32FromChar(&argument);
+                    } else
 #endif
-                            main_pause=1;
-                        break;
-
-                        /* Select compressibility of synthetic sample */
-                    case 'P':
-                    {   argument++;
-                        compressibility = (double)readU32FromChar(&argument) / 100;
-                    }
+                        main_pause=1;
                     break;
 
-                        /* unknown command */
-                    default : CLEAN_RETURN(badusage(programName));
-                    }
+                    /* Select compressibility of synthetic sample */
+                case 'P':
+                {   argument++;
+                    compressibility = (double)readU32FromChar(&argument) / 100;
                 }
-                continue;
-            }   /* if (argument[0]=='-') */
-
-            if (nextArgumentIsMaxDict) {  /* kept available for compatibility with old syntax ; will be removed one day */
-                nextArgumentIsMaxDict = 0;
-                lastCommand = 0;
-                maxDictSize = readU32FromChar(&argument);
-                continue;
-            }
+                break;
 
-            if (nextArgumentIsDictID) {  /* kept available for compatibility with old syntax ; will be removed one day */
-                nextArgumentIsDictID = 0;
-                lastCommand = 0;
-                dictID = readU32FromChar(&argument);
-                continue;
+                    /* unknown command */
+                default : badusage(programName); CLEAN_RETURN(1);
+                }
             }
+            continue;
+        }   /* if (argument[0]=='-') */
+
+        if (nextArgumentIsMaxDict) {  /* kept available for compatibility with old syntax ; will be removed one day */
+            nextArgumentIsMaxDict = 0;
+            lastCommand = 0;
+            maxDictSize = readU32FromChar(&argument);
+            continue;
+        }
 
-        }   /* if (nextArgumentIsAFile==0) */
+        if (nextArgumentIsDictID) {  /* kept available for compatibility with old syntax ; will be removed one day */
+            nextArgumentIsDictID = 0;
+            lastCommand = 0;
+            dictID = readU32FromChar(&argument);
+            continue;
+        }
 
         if (nextEntryIsDictionary) {
             nextEntryIsDictionary = 0;
@@ -978,8 +1061,8 @@ int main(int argCount, const char* argv[])
             continue;
         }
 
-        /* add filename to list */
-        filenameTable[filenameIdx++] = argument;
+        /* none of the above : add filename to list */
+        UTIL_refFilename(filenames, argument);
     }
 
     if (lastCommand) { /* forgotten argument */
@@ -1003,31 +1086,37 @@ int main(int argCount, const char* argv[])
 #ifdef UTIL_HAS_CREATEFILELIST
     g_utilDisplayLevel = g_displayLevel;
     if (!followLinks) {
-        unsigned u;
-        for (u=0, fileNamesNb=0; u<filenameIdx; u++) {
-            if (UTIL_isLink(filenameTable[u])
-#ifndef _MSC_VER
-                && !UTIL_isFIFO(filenameTable[u])
-#endif /* _MSC_VER */
+        unsigned u, fileNamesNb;
+        unsigned const nbFilenames = (unsigned)filenames->tableSize;
+        for (u=0, fileNamesNb=0; u<nbFilenames; u++) {
+            if ( UTIL_isLink(filenames->fileNames[u])
+             && !UTIL_isFIFO(filenames->fileNames[u])
             ) {
-                DISPLAYLEVEL(2, "Warning : %s is a symbolic link, ignoring\n", filenameTable[u]);
+                DISPLAYLEVEL(2, "Warning : %s is a symbolic link, ignoring \n", filenames->fileNames[u]);
             } else {
-                filenameTable[fileNamesNb++] = filenameTable[u];
+                filenames->fileNames[fileNamesNb++] = filenames->fileNames[u];
+        }   }
+        if (fileNamesNb == 0 && nbFilenames > 0)  /* all names are eliminated */
+            CLEAN_RETURN(1);
+        filenames->tableSize = fileNamesNb;
+    }   /* if (!followLinks) */
+
+    /* read names from a file */
+    if (file_of_names->tableSize) {
+        size_t const nbFileLists = file_of_names->tableSize;
+        size_t flNb;
+        for (flNb=0; flNb < nbFileLists; flNb++) {
+            FileNamesTable* const fnt = UTIL_createFileNamesTable_fromFileName(file_of_names->fileNames[flNb]);
+            if (fnt==NULL) {
+                DISPLAYLEVEL(1, "zstd: error reading %s \n", file_of_names->fileNames[flNb]);
+                CLEAN_RETURN(1);
             }
+            filenames = UTIL_mergeFileNamesTable(filenames, fnt);
         }
-        if (fileNamesNb == 0 && filenameIdx > 0)
-            CLEAN_RETURN(1);
-        filenameIdx = fileNamesNb;
     }
+
     if (recursive) {  /* at this stage, filenameTable is a list of paths, which can contain both files and directories */
-        extendedFileList = UTIL_createFileList(filenameTable, filenameIdx, &fileNamesBuf, &fileNamesNb, followLinks);
-        if (extendedFileList) {
-            unsigned u;
-            for (u=0; u<fileNamesNb; u++) DISPLAYLEVEL(4, "%u %s\n", u, extendedFileList[u]);
-            free((void*)filenameTable);
-            filenameTable = extendedFileList;
-            filenameIdx = fileNamesNb;
-        }
+        UTIL_expandFNT(&filenames, followLinks);
     }
 #else
     (void)followLinks;
@@ -1035,7 +1124,7 @@ int main(int argCount, const char* argv[])
 
     if (operation == zom_list) {
 #ifndef ZSTD_NODECOMPRESS
-        int const ret = FIO_listMultipleFiles(filenameIdx, filenameTable, g_displayLevel);
+        int const ret = FIO_listMultipleFiles((unsigned)filenames->tableSize, filenames->fileNames, g_displayLevel);
         CLEAN_RETURN(ret);
 #else
         DISPLAY("file information is not supported \n");
@@ -1066,26 +1155,23 @@ int main(int argCount, const char* argv[])
         if (cLevelLast < cLevel) cLevelLast = cLevel;
         if (cLevelLast > cLevel)
             DISPLAYLEVEL(3, "Benchmarking levels from %d to %d\n", cLevel, cLevelLast);
-        if(filenameIdx) {
+        if (filenames->tableSize > 0) {
             if(separateFiles) {
                 unsigned i;
-                for(i = 0; i < filenameIdx; i++) {
+                for(i = 0; i < filenames->tableSize; i++) {
                     int c;
-                    DISPLAYLEVEL(3, "Benchmarking %s \n", filenameTable[i]);
+                    DISPLAYLEVEL(3, "Benchmarking %s \n", filenames->fileNames[i]);
                     for(c = cLevel; c <= cLevelLast; c++) {
-                        BMK_benchFilesAdvanced(&filenameTable[i], 1, dictFileName, c, &compressionParams, g_displayLevel, &benchParams);
-                    }
-                }
+                        BMK_benchFilesAdvanced(&filenames->fileNames[i], 1, dictFileName, c, &compressionParams, g_displayLevel, &benchParams);
+                }   }
             } else {
                 for(; cLevel <= cLevelLast; cLevel++) {
-                    BMK_benchFilesAdvanced(filenameTable, filenameIdx, dictFileName, cLevel, &compressionParams, g_displayLevel, &benchParams);
-                }
-            }
+                    BMK_benchFilesAdvanced(filenames->fileNames, (unsigned)filenames->tableSize, dictFileName, cLevel, &compressionParams, g_displayLevel, &benchParams);
+            }   }
         } else {
             for(; cLevel <= cLevelLast; cLevel++) {
                 BMK_syntheticTest(cLevel, compressibility, &compressionParams, g_displayLevel, &benchParams);
-            }
-        }
+        }   }
 
 #else
         (void)bench_nbSeconds; (void)blockSize; (void)setRealTimePrio; (void)separateFiles; (void)compressibility;
@@ -1104,18 +1190,18 @@ int main(int argCount, const char* argv[])
             int const optimize = !coverParams.k || !coverParams.d;
             coverParams.nbThreads = (unsigned)nbWorkers;
             coverParams.zParams = zParams;
-            operationResult = DiB_trainFromFiles(outFileName, maxDictSize, filenameTable, filenameIdx, blockSize, NULL, &coverParams, NULL, optimize);
+            operationResult = DiB_trainFromFiles(outFileName, maxDictSize, filenames->fileNames, (unsigned)filenames->tableSize, blockSize, NULL, &coverParams, NULL, optimize);
         } else if (dict == fastCover) {
             int const optimize = !fastCoverParams.k || !fastCoverParams.d;
             fastCoverParams.nbThreads = (unsigned)nbWorkers;
             fastCoverParams.zParams = zParams;
-            operationResult = DiB_trainFromFiles(outFileName, maxDictSize, filenameTable, filenameIdx, blockSize, NULL, NULL, &fastCoverParams, optimize);
+            operationResult = DiB_trainFromFiles(outFileName, maxDictSize, filenames->fileNames, (unsigned)filenames->tableSize, blockSize, NULL, NULL, &fastCoverParams, optimize);
         } else {
             ZDICT_legacy_params_t dictParams;
             memset(&dictParams, 0, sizeof(dictParams));
             dictParams.selectivityLevel = dictSelect;
             dictParams.zParams = zParams;
-            operationResult = DiB_trainFromFiles(outFileName, maxDictSize, filenameTable, filenameIdx, blockSize, &dictParams, NULL, NULL, 0);
+            operationResult = DiB_trainFromFiles(outFileName, maxDictSize, filenames->fileNames, (unsigned)filenames->tableSize, blockSize, &dictParams, NULL, NULL, 0);
         }
 #else
         (void)dictCLevel; (void)dictSelect; (void)dictID;  (void)maxDictSize; /* not used when ZSTD_NODICT set */
@@ -1130,19 +1216,23 @@ int main(int argCount, const char* argv[])
 #endif
 
     /* No input filename ==> use stdin and stdout */
-    filenameIdx += !filenameIdx;   /* filenameTable[0] is stdin by default */
-    if (!strcmp(filenameTable[0], stdinmark) && !outFileName)
+    if (filenames->tableSize == 0) UTIL_refFilename(filenames, stdinmark);
+    if (!strcmp(filenames->fileNames[0], stdinmark) && !outFileName)
         outFileName = stdoutmark;  /* when input is stdin, default output is stdout */
 
     /* Check if input/output defined as console; trigger an error in this case */
-    if (!strcmp(filenameTable[0], stdinmark) && IS_CONSOLE(stdin) )
-        CLEAN_RETURN(badusage(programName));
+    if (!strcmp(filenames->fileNames[0], stdinmark) && IS_CONSOLE(stdin) ) {
+        badusage(programName);
+        CLEAN_RETURN(1);
+    }
     if ( outFileName && !strcmp(outFileName, stdoutmark)
       && IS_CONSOLE(stdout)
-      && !strcmp(filenameTable[0], stdinmark)
+      && !strcmp(filenames->fileNames[0], stdinmark)
       && !forceStdout
-      && operation!=zom_decompress )
-        CLEAN_RETURN(badusage(programName));
+      && operation!=zom_decompress ) {
+        badusage(programName);
+        CLEAN_RETURN(1);
+    }
 
 #ifndef ZSTD_NOCOMPRESS
     /* check compression level limits */
@@ -1153,14 +1243,42 @@ int main(int argCount, const char* argv[])
     }   }
 #endif
 
+    if (showDefaultCParams) {
+        if (operation == zom_decompress) {
+            DISPLAY("error : can't use --show-default-cparams in decomrpession mode \n");
+            CLEAN_RETURN(1);
+        }
+    }
+
+    if (dictFileName != NULL && patchFromDictFileName != NULL) {
+        DISPLAY("error : can't use -D and --patch-from=# at the same time \n");
+        CLEAN_RETURN(1);
+    }
+
+    if (patchFromDictFileName != NULL && filenames->tableSize > 1) {
+        DISPLAY("error : can't use --patch-from=# on multiple files \n");
+        CLEAN_RETURN(1);
+    }
+
     /* No status message in pipe mode (stdin - stdout) or multi-files mode */
-    if (!strcmp(filenameTable[0], stdinmark) && outFileName && !strcmp(outFileName,stdoutmark) && (g_displayLevel==2)) g_displayLevel=1;
-    if ((filenameIdx>1) & (g_displayLevel==2)) g_displayLevel=1;
+    if (!strcmp(filenames->fileNames[0], stdinmark) && outFileName && !strcmp(outFileName,stdoutmark) && (g_displayLevel==2)) g_displayLevel=1;
+    if ((filenames->tableSize > 1) & (g_displayLevel==2)) g_displayLevel=1;
 
     /* IO Stream/File */
     FIO_setNotificationLevel(g_displayLevel);
+    FIO_setPatchFromMode(prefs, patchFromDictFileName != NULL);
+    if (memLimit == 0) {
+        if (compressionParams.windowLog == 0) {
+            memLimit = (U32)1 << g_defaultMaxWindowLog;
+        } else {
+            memLimit = (U32)1 << (compressionParams.windowLog & 31);
+    }   }
+    if (patchFromDictFileName != NULL)
+        dictFileName = patchFromDictFileName;
+    FIO_setMemLimit(prefs, memLimit);
     if (operation==zom_compress) {
 #ifndef ZSTD_NOCOMPRESS
+        FIO_setContentSize(prefs, contentSize);
         FIO_setNbWorkers(prefs, nbWorkers);
         FIO_setBlockSize(prefs, (int)blockSize);
         if (g_overlapLog!=OVERLAP_LOG_DEFAULT) FIO_setOverlapLog(prefs, (int)g_overlapLog);
@@ -1180,28 +1298,45 @@ int main(int argCount, const char* argv[])
         if (adaptMin > cLevel) cLevel = adaptMin;
         if (adaptMax < cLevel) cLevel = adaptMax;
 
-        if ((filenameIdx==1) && outFileName)
-          operationResult = FIO_compressFilename(prefs, outFileName, filenameTable[0], dictFileName, cLevel, compressionParams);
+        /* Compare strategies constant with the ground truth */
+        { ZSTD_bounds strategyBounds = ZSTD_cParam_getBounds(ZSTD_c_strategy);
+          assert(ZSTD_NB_STRATEGIES == strategyBounds.upperBound);
+          (void)strategyBounds; }
+
+        if (showDefaultCParams) {
+            size_t fileNb;
+            for (fileNb = 0; fileNb < (size_t)filenames->tableSize; fileNb++) {
+                unsigned long long fileSize = UTIL_getFileSize(filenames->fileNames[fileNb]);
+                const size_t dictSize = dictFileName != NULL ? (size_t)UTIL_getFileSize(dictFileName) : 0;
+                const ZSTD_compressionParameters cParams = ZSTD_getCParams(cLevel, fileSize, dictSize);
+                if (fileSize != UTIL_FILESIZE_UNKNOWN) DISPLAY("%s (%u bytes)\n", filenames->fileNames[fileNb], (unsigned)fileSize);
+                else DISPLAY("%s (src size unknown)\n", filenames->fileNames[fileNb]);
+                DISPLAY(" - windowLog     : %u\n", cParams.windowLog);
+                DISPLAY(" - chainLog      : %u\n", cParams.chainLog);
+                DISPLAY(" - hashLog       : %u\n", cParams.hashLog);
+                DISPLAY(" - searchLog     : %u\n", cParams.searchLog);
+                DISPLAY(" - minMatch      : %u\n", cParams.minMatch);
+                DISPLAY(" - targetLength  : %u\n", cParams.targetLength);
+                assert(cParams.strategy < ZSTD_NB_STRATEGIES + 1);
+                DISPLAY(" - strategy      : %s (%u)\n", ZSTD_strategyMap[(int)cParams.strategy], (unsigned)cParams.strategy);
+            }
+        }
+
+        if ((filenames->tableSize==1) && outFileName)
+          operationResult = FIO_compressFilename(prefs, outFileName, filenames->fileNames[0], dictFileName, cLevel, compressionParams);
         else
-          operationResult = FIO_compressMultipleFilenames(prefs, filenameTable, filenameIdx, outDirName, outFileName, suffix, dictFileName, cLevel, compressionParams);
+          operationResult = FIO_compressMultipleFilenames(prefs, filenames->fileNames, (unsigned)filenames->tableSize, outDirName, outFileName, suffix, dictFileName, cLevel, compressionParams);
 #else
-        (void)suffix; (void)adapt; (void)rsyncable; (void)ultra; (void)cLevel; (void)ldmFlag; (void)literalCompressionMode; (void)targetCBlockSize; (void)streamSrcSize; (void)srcSizeHint; /* not used when ZSTD_NOCOMPRESS set */
+        (void)contentSize; (void)suffix; (void)adapt; (void)rsyncable; (void)ultra; (void)cLevel; (void)ldmFlag; (void)literalCompressionMode; (void)targetCBlockSize; (void)streamSrcSize; (void)srcSizeHint; (void)ZSTD_strategyMap; /* not used when ZSTD_NOCOMPRESS set */
         DISPLAY("Compression not supported \n");
 #endif
     } else {  /* decompression or test */
 #ifndef ZSTD_NODECOMPRESS
-        if (memLimit == 0) {
-            if (compressionParams.windowLog == 0)
-                memLimit = (U32)1 << g_defaultMaxWindowLog;
-            else {
-                memLimit = (U32)1 << (compressionParams.windowLog & 31);
-            }
+        if (filenames->tableSize == 1 && outFileName) {
+            operationResult = FIO_decompressFilename(prefs, outFileName, filenames->fileNames[0], dictFileName);
+        } else {
+            operationResult = FIO_decompressMultipleFilenames(prefs, filenames->fileNames, (unsigned)filenames->tableSize, outDirName, outFileName, dictFileName);
         }
-        FIO_setMemLimit(prefs, memLimit);
-        if (filenameIdx==1 && outFileName)
-            operationResult = FIO_decompressFilename(prefs, outFileName, filenameTable[0], dictFileName);
-        else
-            operationResult = FIO_decompressMultipleFilenames(prefs, filenameTable, filenameIdx, outDirName, outFileName, dictFileName);
 #else
         DISPLAY("Decompression not supported \n");
 #endif
@@ -1209,13 +1344,9 @@ int main(int argCount, const char* argv[])
 
 _end:
     FIO_freePreferences(prefs);
-
     if (main_pause) waitEnter();
-#ifdef UTIL_HAS_CREATEFILELIST
-    if (extendedFileList)
-        UTIL_freeFileList(extendedFileList, fileNamesBuf);
-    else
-#endif
-        free((void*)filenameTable);
+    UTIL_freeFileNamesTable(filenames);
+    UTIL_freeFileNamesTable(file_of_names);
+
     return operationResult;
 }
diff --git a/programs/zstdgrep b/programs/zstdgrep
index 4879fb0dae49..61efaa9474a0 100755
--- a/programs/zstdgrep
+++ b/programs/zstdgrep
@@ -109,7 +109,7 @@ if [ "$#" -lt 1 ]; then
     # ... on stdin
     set -f # Disable file name generation (globbing).
     # shellcheck disable=SC2086
-    "${zcat}" -fq - | "${grep}" ${grep_args} -- "${pattern}" -
+    "${zcat}" - | "${grep}" ${grep_args} -- "${pattern}" -
     EXIT_CODE=$?
     set +f
 else
@@ -121,9 +121,9 @@ else
     while [ "$#" -gt 0 ]; do
         # shellcheck disable=SC2086
         if [ $pattern_found -eq 2 ]; then
-            "${zcat}" -fq -- "$1" | "${grep}" --label="${1}" ${grep_args} -- -
+            "${zcat}" -- "$1" | "${grep}" --label="${1}" ${grep_args} -- -
         else
-            "${zcat}" -fq -- "$1" | "${grep}" --label="${1}" ${grep_args} -- "${pattern}" -
+            "${zcat}" -- "$1" | "${grep}" --label="${1}" ${grep_args} -- "${pattern}" -
         fi
         [ "$?" -ne 0 ] && EXIT_CODE=1
         shift
diff --git a/programs/zstdgrep.1 b/programs/zstdgrep.1
index b97f8cabcc00..fe1d29bbb062 100644
--- a/programs/zstdgrep.1
+++ b/programs/zstdgrep.1
@@ -1,5 +1,5 @@
 .
-.TH "ZSTDGREP" "1" "October 2019" "zstd 1.4.4" "User Commands"
+.TH "ZSTDGREP" "1" "May 2020" "zstd 1.4.5" "User Commands"
 .
 .SH "NAME"
 \fBzstdgrep\fR \- print lines matching a pattern in zstandard\-compressed files
diff --git a/programs/zstdless.1 b/programs/zstdless.1
index 1ecc8bdc531c..d54c6400cee2 100644
--- a/programs/zstdless.1
+++ b/programs/zstdless.1
@@ -1,5 +1,5 @@
 .
-.TH "ZSTDLESS" "1" "October 2019" "zstd 1.4.4" "User Commands"
+.TH "ZSTDLESS" "1" "May 2020" "zstd 1.4.5" "User Commands"
 .
 .SH "NAME"
 \fBzstdless\fR \- view zstandard\-compressed files
diff --git a/tests/Makefile b/tests/Makefile
deleted file mode 100644
index 3917a7cf8ac5..000000000000
--- a/tests/Makefile
+++ /dev/null
@@ -1,484 +0,0 @@
-# ################################################################
-# Copyright (c) 2015-present, Yann Collet, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under both the BSD-style license (found in the
-# LICENSE file in the root directory of this source tree) and the GPLv2 (found
-# in the COPYING file in the root directory of this source tree).
-# ################################################################
-# datagen : Synthetic and parametrable data generator, for tests
-# fullbench  : Precisely measure speed for each zstd inner functions
-# fullbench32: Same as fullbench, but forced to compile in 32-bits mode
-# fuzzer  : Test tool, to check zstd integrity on target platform
-# fuzzer32: Same as fuzzer, but forced to compile in 32-bits mode
-# paramgrill : parameter tester for zstd
-# test-zstd-speed.py : script for testing zstd speed difference between commits
-# versionsTest : compatibility test between zstd versions stored on Github (v0.1+)
-# zstreamtest : Fuzzer test tool for zstd streaming API
-# zstreamtest32: Same as zstreamtest, but forced to compile in 32-bits mode
-# ##########################################################################
-
-ZSTDDIR = ../lib
-PRGDIR  = ../programs
-PYTHON ?= python3
-TESTARTEFACT := versionsTest
-
-DEBUGLEVEL ?= 1
-DEBUGFLAGS  = -g -DDEBUGLEVEL=$(DEBUGLEVEL)
-CPPFLAGS   += -I$(ZSTDDIR) -I$(ZSTDDIR)/common -I$(ZSTDDIR)/compress \
-              -I$(ZSTDDIR)/dictBuilder -I$(ZSTDDIR)/deprecated -I$(PRGDIR)
-ifeq ($(OS),Windows_NT)   # MinGW assumed
-CPPFLAGS   += -D__USE_MINGW_ANSI_STDIO   # compatibility with %zu formatting
-endif
-CFLAGS     ?= -O3
-CFLAGS     += -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow                 \
-              -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement \
-              -Wstrict-prototypes -Wundef                                     \
-              -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings      \
-              -Wredundant-decls -Wmissing-prototypes
-CFLAGS     += $(DEBUGFLAGS) $(MOREFLAGS)
-FLAGS       = $(CPPFLAGS) $(CFLAGS) $(LDFLAGS)
-
-
-ZSTDCOMMON_FILES := $(ZSTDDIR)/common/*.c
-ZSTDCOMP_FILES   := $(ZSTDDIR)/compress/*.c
-ZSTDDECOMP_FILES := $(ZSTDDIR)/decompress/*.c
-ZSTD_FILES  := $(ZSTDDECOMP_FILES) $(ZSTDCOMMON_FILES) $(ZSTDCOMP_FILES)
-ZBUFF_FILES := $(ZSTDDIR)/deprecated/*.c
-ZDICT_FILES := $(ZSTDDIR)/dictBuilder/*.c
-
-ZSTD_F1 := $(wildcard $(ZSTD_FILES))
-ZSTD_OBJ1 := $(subst $(ZSTDDIR)/common/,zstdm_,$(ZSTD_F1))
-ZSTD_OBJ2 := $(subst $(ZSTDDIR)/compress/,zstdc_,$(ZSTD_OBJ1))
-ZSTD_OBJ3 := $(subst $(ZSTDDIR)/decompress/,zstdd_,$(ZSTD_OBJ2))
-ZSTD_OBJECTS := $(ZSTD_OBJ3:.c=.o)
-
-ZSTDMT_OBJ1 := $(subst $(ZSTDDIR)/common/,zstdmt_m_,$(ZSTD_F1))
-ZSTDMT_OBJ2 := $(subst $(ZSTDDIR)/compress/,zstdmt_c_,$(ZSTDMT_OBJ1))
-ZSTDMT_OBJ3 := $(subst $(ZSTDDIR)/decompress/,zstdmt_d_,$(ZSTDMT_OBJ2))
-ZSTDMT_OBJECTS := $(ZSTDMT_OBJ3:.c=.o)
-
-# Define *.exe as extension for Windows systems
-ifneq (,$(filter Windows%,$(OS)))
-EXT =.exe
-MULTITHREAD_CPP = -DZSTD_MULTITHREAD
-MULTITHREAD_LD  =
-else
-EXT =
-MULTITHREAD_CPP = -DZSTD_MULTITHREAD
-MULTITHREAD_LD  = -pthread
-endif
-MULTITHREAD = $(MULTITHREAD_CPP) $(MULTITHREAD_LD)
-
-VOID = /dev/null
-ZSTREAM_TESTTIME ?= -T90s
-FUZZERTEST ?= -T200s
-ZSTDRTTEST = --test-large-data
-DECODECORPUS_TESTTIME ?= -T30
-
-.PHONY: default all all32 allnothread dll clean test test32 test-all versionsTest
-
-default: fullbench
-	@echo $(ZSTDMT_OBJECTS)
-
-all: fullbench fuzzer zstreamtest paramgrill datagen decodecorpus roundTripCrash \
-     fullbench-lib poolTests
-
-all32: fullbench32 fuzzer32 zstreamtest32
-
-allnothread: MULTITHREAD_CPP=
-allnothread: MULTITHREAD_LD=
-allnothread: fullbench fuzzer paramgrill datagen decodecorpus
-
-dll: fuzzer-dll zstreamtest-dll
-
-PHONY: zstd zstd32 zstd-nolegacy  # must be phony, only external makefile knows how to build them, or if they need an update
-zstd zstd32 zstd-nolegacy:
-	$(MAKE) -C $(PRGDIR) $@ MOREFLAGS+="$(DEBUGFLAGS)"
-
-gzstd:
-	$(MAKE) -C $(PRGDIR) zstd HAVE_ZLIB=1 MOREFLAGS+="$(DEBUGFLAGS)"
-
-.PHONY: zstd-dll
-zstd-dll :
-	$(MAKE) -C $(ZSTDDIR) libzstd
-
-.PHONY: zstd-staticLib
-zstd-staticLib :
-	$(MAKE) -C $(ZSTDDIR) libzstd.a
-
-zstdm_%.o : $(ZSTDDIR)/common/%.c
-	$(CC) -c $(CPPFLAGS) $(CFLAGS) $< -o $@
-
-zstdc_%.o : $(ZSTDDIR)/compress/%.c
-	$(CC) -c $(CPPFLAGS) $(CFLAGS) $< -o $@
-
-zstdd_%.o : $(ZSTDDIR)/decompress/%.c
-	$(CC) -c $(CPPFLAGS) $(CFLAGS) $< -o $@
-
-zstdmt%.o : CPPFLAGS += $(MULTITHREAD_CPP)
-
-zstdmt_m_%.o : $(ZSTDDIR)/common/%.c
-	$(CC) -c $(CPPFLAGS) $(CFLAGS) $< -o $@
-
-zstdmt_c_%.o : $(ZSTDDIR)/compress/%.c
-	$(CC) -c $(CPPFLAGS) $(CFLAGS) $< -o $@
-
-zstdmt_d_%.o : $(ZSTDDIR)/decompress/%.c
-	$(CC) -c $(CPPFLAGS) $(CFLAGS) $< -o $@
-
-fullbench32: CPPFLAGS += -m32
-fullbench fullbench32 : CPPFLAGS += $(MULTITHREAD_CPP)
-fullbench fullbench32 : LDFLAGS += $(MULTITHREAD_LD)
-fullbench fullbench32 : DEBUGFLAGS = -DNDEBUG  # turn off assert() for speed measurements
-fullbench fullbench32 : $(ZSTD_FILES)
-fullbench fullbench32 : $(PRGDIR)/datagen.c $(PRGDIR)/util.c $(PRGDIR)/timefn.c $(PRGDIR)/benchfn.c fullbench.c
-	$(CC) $(FLAGS) $^ -o $@$(EXT)
-
-fullbench-lib : CPPFLAGS += -DXXH_NAMESPACE=ZSTD_
-fullbench-lib : zstd-staticLib
-fullbench-lib : $(PRGDIR)/datagen.c $(PRGDIR)/util.c $(PRGDIR)/timefn.c $(PRGDIR)/benchfn.c fullbench.c
-	$(CC) $(FLAGS) $(filter %.c,$^) -o $@$(EXT) $(ZSTDDIR)/libzstd.a
-
-# note : broken : requires unavailable symbols
-fullbench-dll : zstd-dll
-fullbench-dll : LDFLAGS+= -L$(ZSTDDIR) -lzstd
-fullbench-dll: $(PRGDIR)/datagen.c $(PRGDIR)/util.c $(PRGDIR)/benchfn.c $(PRGDIR)/timefn.c fullbench.c
-#	$(CC) $(FLAGS) $(filter %.c,$^) -o $@$(EXT) -DZSTD_DLL_IMPORT=1 $(ZSTDDIR)/dll/libzstd.dll
-	$(CC) $(FLAGS) $(filter %.c,$^) -o $@$(EXT)
-
-fuzzer  : CPPFLAGS += $(MULTITHREAD_CPP)
-fuzzer  : LDFLAGS += $(MULTITHREAD_LD)
-fuzzer32: CFLAGS += -m32
-fuzzer  : $(ZSTDMT_OBJECTS)
-fuzzer32: $(ZSTD_FILES)
-fuzzer fuzzer32 : $(ZDICT_FILES) $(PRGDIR)/util.c $(PRGDIR)/timefn.c $(PRGDIR)/datagen.c fuzzer.c
-	$(CC) $(FLAGS) $^ -o $@$(EXT)
-
-fuzzer-dll : zstd-dll
-fuzzer-dll : LDFLAGS+= -L$(ZSTDDIR) -lzstd
-fuzzer-dll : $(ZSTDDIR)/common/xxhash.c $(PRGDIR)/util.c $(PRGDIR)/timefn.c $(PRGDIR)/datagen.c fuzzer.c
-	$(CC) $(CPPFLAGS) $(CFLAGS) $(filter %.c,$^) $(LDFLAGS) -o $@$(EXT)
-
-zbufftest zbufftest32 zbufftest-dll : CPPFLAGS += -I$(ZSTDDIR)/deprecated
-zbufftest zbufftest32 zbufftest-dll : CFLAGS += -Wno-deprecated-declarations   # required to silence deprecation warnings
-zbufftest32 : CFLAGS +=  -m32
-zbufftest zbufftest32 : $(ZSTD_OBJECTS) $(ZBUFF_FILES) $(PRGDIR)/util.c $(PRGDIR)/timefn.c $(PRGDIR)/datagen.c zbufftest.c
-	$(CC) $(FLAGS) $^ -o $@$(EXT)
-
-zbufftest-dll : zstd-dll
-zbufftest-dll : LDFLAGS+= -L$(ZSTDDIR) -lzstd
-zbufftest-dll : $(ZSTDDIR)/common/xxhash.c $(PRGDIR)/util.c $(PRGDIR)/timefn.c $(PRGDIR)/datagen.c zbufftest.c
-	$(CC) $(CPPFLAGS) $(CFLAGS) $(filter %.c,$^) $(LDFLAGS) -o $@$(EXT)
-
-ZSTREAM_LOCAL_FILES := $(PRGDIR)/datagen.c $(PRGDIR)/util.c $(PRGDIR)/timefn.c seqgen.c zstreamtest.c
-ZSTREAM_PROPER_FILES := $(ZDICT_FILES) $(ZSTREAM_LOCAL_FILES)
-ZSTREAMFILES := $(ZSTD_FILES) $(ZSTREAM_PROPER_FILES)
-zstreamtest32 : CFLAGS += -m32
-zstreamtest zstreamtest32 : CPPFLAGS += $(MULTITHREAD_CPP)
-zstreamtest zstreamtest32 : LDFLAGS += $(MULTITHREAD_LD)
-zstreamtest : $(ZSTDMT_OBJECTS) $(ZSTREAM_PROPER_FILES)
-zstreamtest32 : $(ZSTREAMFILES)
-zstreamtest zstreamtest32 :
-	$(CC) $(FLAGS) $^ -o $@$(EXT)
-
-zstreamtest_asan : CFLAGS += -fsanitize=address
-zstreamtest_asan : $(ZSTREAMFILES)
-	$(CC) $(FLAGS) $(MULTITHREAD) $^ -o $@$(EXT)
-
-zstreamtest_tsan : CFLAGS += -fsanitize=thread
-zstreamtest_tsan : $(ZSTREAMFILES)
-	$(CC) $(FLAGS) $(MULTITHREAD) $^ -o $@$(EXT)
-
-zstreamtest-dll : zstd-dll
-zstreamtest-dll : LDFLAGS+= -L$(ZSTDDIR) -lzstd
-zstreamtest-dll : $(ZSTDDIR)/common/xxhash.c  # xxh symbols not exposed from dll
-zstreamtest-dll : $(ZSTREAM_LOCAL_FILES)
-	$(CC) $(CPPFLAGS) $(CFLAGS) $(filter %.c,$^) $(LDFLAGS) -o $@$(EXT)
-
-paramgrill : DEBUGFLAGS =  # turn off assert() by default for speed measurements
-paramgrill : $(ZSTD_FILES) $(PRGDIR)/util.c $(PRGDIR)/timefn.c $(PRGDIR)/benchfn.c $(PRGDIR)/benchzstd.c $(PRGDIR)/datagen.c paramgrill.c
-	$(CC) $(FLAGS) $^ -lm -o $@$(EXT)
-
-datagen : $(PRGDIR)/datagen.c datagencli.c
-	$(CC) $(FLAGS) $^ -o $@$(EXT)
-
-roundTripCrash : $(ZSTD_OBJECTS) roundTripCrash.c
-	$(CC) $(FLAGS) $(MULTITHREAD) $^ -o $@$(EXT)
-
-longmatch  : $(ZSTD_OBJECTS) longmatch.c
-	$(CC) $(FLAGS) $^ -o $@$(EXT)
-
-bigdict: $(ZSTDMT_OBJECTS) $(PRGDIR)/datagen.c bigdict.c
-	$(CC) $(FLAGS) $(MULTITHREAD) $^ -o $@$(EXT)
-
-invalidDictionaries : $(ZSTD_OBJECTS) invalidDictionaries.c
-	$(CC) $(FLAGS) $^ -o $@$(EXT)
-
-legacy : CPPFLAGS += -I$(ZSTDDIR)/legacy -DZSTD_LEGACY_SUPPORT=4
-legacy : $(ZSTD_FILES) $(wildcard $(ZSTDDIR)/legacy/*.c) legacy.c
-	$(CC) $(FLAGS) $^ -o $@$(EXT)
-
-decodecorpus : $(filter-out zstdc_zstd_compress.o, $(ZSTD_OBJECTS)) $(ZDICT_FILES) $(PRGDIR)/util.c $(PRGDIR)/timefn.c decodecorpus.c
-	$(CC) $(FLAGS) $^ -o $@$(EXT) -lm
-
-symbols  : symbols.c zstd-dll
-ifneq (,$(filter Windows%,$(OS)))
-	cp $(ZSTDDIR)/dll/libzstd.dll .
-	$(CC) $(FLAGS) $< -o $@$(EXT) -DZSTD_DLL_IMPORT=1 libzstd.dll
-else
-	$(CC) $(FLAGS) $< -o $@$(EXT) -Wl,-rpath=$(ZSTDDIR) $(ZSTDDIR)/libzstd.so   # broken on Mac
-endif
-
-poolTests : $(PRGDIR)/util.c $(PRGDIR)/timefn.c poolTests.c $(ZSTDDIR)/common/pool.c $(ZSTDDIR)/common/threading.c $(ZSTDDIR)/common/zstd_common.c $(ZSTDDIR)/common/error_private.c
-	$(CC) $(FLAGS) $(MULTITHREAD) $^ -o $@$(EXT)
-
-.PHONY: versionsTest
-versionsTest: clean
-	$(PYTHON) test-zstd-versions.py
-
-checkTag: checkTag.c $(ZSTDDIR)/zstd.h
-	$(CC) $(FLAGS) $< -o $@$(EXT)
-
-clean:
-	$(MAKE) -C $(ZSTDDIR) clean
-	$(MAKE) -C $(PRGDIR) clean
-	@$(RM) -fR $(TESTARTEFACT)
-	@$(RM) -rf tmp*  # some test directories are named tmp*
-	@$(RM) core *.o *.tmp result* *.gcda dictionary *.zst \
-        $(PRGDIR)/zstd$(EXT) $(PRGDIR)/zstd32$(EXT) \
-        fullbench$(EXT) fullbench32$(EXT) \
-        fullbench-lib$(EXT) fullbench-dll$(EXT) \
-        fuzzer$(EXT) fuzzer32$(EXT) zbufftest$(EXT) zbufftest32$(EXT) \
-        fuzzer-dll$(EXT) zstreamtest-dll$(EXT) zbufftest-dll$(EXT) \
-        zstreamtest$(EXT) zstreamtest32$(EXT) \
-        datagen$(EXT) paramgrill$(EXT) roundTripCrash$(EXT) longmatch$(EXT) \
-        symbols$(EXT) invalidDictionaries$(EXT) legacy$(EXT) poolTests$(EXT) \
-        decodecorpus$(EXT) checkTag$(EXT) bigdict$(EXT)
-	@echo Cleaning completed
-
-
-#----------------------------------------------------------------------------------
-# valgrind tests are validated only for some posix platforms
-#----------------------------------------------------------------------------------
-ifneq (,$(filter $(shell uname),Linux Darwin GNU/kFreeBSD GNU OpenBSD FreeBSD NetBSD DragonFly SunOS))
-HOST_OS = POSIX
-
-valgrindTest: VALGRIND = valgrind --leak-check=full --show-leak-kinds=all --error-exitcode=1
-valgrindTest: zstd datagen fuzzer fullbench
-	@echo "\n ---- valgrind tests : memory analyzer ----"
-	$(VALGRIND) ./datagen -g50M > $(VOID)
-	$(VALGRIND) $(PRGDIR)/zstd ; if [ $$? -eq 0 ] ; then echo "zstd without argument should have failed"; false; fi
-	./datagen -g80 | $(VALGRIND) $(PRGDIR)/zstd - -c > $(VOID)
-	./datagen -g16KB | $(VALGRIND) $(PRGDIR)/zstd -vf - -c > $(VOID)
-	./datagen -g2930KB | $(VALGRIND) $(PRGDIR)/zstd -5 -vf - -o tmp
-	$(VALGRIND) $(PRGDIR)/zstd -vdf tmp -c > $(VOID)
-	./datagen -g64MB | $(VALGRIND) $(PRGDIR)/zstd -vf - -c > $(VOID)
-	@rm tmp
-	$(VALGRIND) ./fuzzer -T1mn -t1
-	$(VALGRIND) ./fullbench -i1
-
-endif
-
-
-ifneq (,$(filter MINGW% MSYS%,$(shell uname)))
-HOST_OS = MSYS
-endif
-
-
-#-----------------------------------------------------------------------------
-# make tests validated only for below targets
-#-----------------------------------------------------------------------------
-ifneq (,$(filter $(HOST_OS),MSYS POSIX))
-
-DIFF:=diff
-ifneq (,$(filter $(shell uname),SunOS))
-DIFF:=gdiff
-endif
-
-.PHONY: list
-list:
-	@$(MAKE) -pRrq -f $(lastword $(MAKEFILE_LIST)) : 2>/dev/null | awk -v RS= -F: '/^# File/,/^# Finished Make data base/ {if ($$1 !~ "^[#.]") {print $$1}}' | sort | egrep -v -e '^[^[:alnum:]]' -e '^$@$$' | xargs
-
-.PHONY: shortest
-shortest: ZSTDRTTEST=
-shortest: test-zstd
-
-.PHONY: fuzztest
-fuzztest: test-fuzzer test-zstream test-decodecorpus
-
-.PHONY: test
-test: test-zstd test-fullbench test-fuzzer test-zstream test-invalidDictionaries test-legacy test-decodecorpus
-ifeq ($(QEMU_SYS),)
-test: test-pool
-endif
-
-test32: test-zstd32 test-fullbench32 test-fuzzer32 test-zstream32
-
-test-all: test test32 valgrindTest test-decodecorpus-cli
-
-
-.PHONY: test-zstd test-zstd32 test-zstd-nolegacy test-zstdgrep
-test-zstd: ZSTD = $(PRGDIR)/zstd
-test-zstd: zstd
-
-test-zstd32: ZSTD = $(PRGDIR)/zstd32
-test-zstd32: zstd32
-
-test-zstd-nolegacy: ZSTD = $(PRGDIR)/zstd-nolegacy
-test-zstd-nolegacy: zstd-nolegacy
-
-test-zstd test-zstd32 test-zstd-nolegacy: datagen
-	file $(ZSTD)
-	ZSTD="$(QEMU_SYS) $(ZSTD)" ./playTests.sh $(ZSTDRTTEST)
-
-
-test-gzstd: gzstd
-	$(PRGDIR)/zstd -f README.md test-zstd-speed.py
-	gzip -f README.md test-zstd-speed.py
-	cat README.md.zst test-zstd-speed.py.gz >zstd_gz.zst
-	cat README.md.gz test-zstd-speed.py.zst >gz_zstd.gz
-	$(PRGDIR)/zstd -df README.md.gz -o README2.md
-	$(PRGDIR)/zstd -df README.md.gz test-zstd-speed.py.gz
-	$(PRGDIR)/zstd -df zstd_gz.zst gz_zstd.gz
-	$(DIFF) -q zstd_gz gz_zstd
-	echo Hello World ZSTD | $(PRGDIR)/zstd -c - >hello.zst
-	echo Hello World GZIP | gzip -c - >hello.gz
-	echo Hello World TEXT >hello.txt
-	cat hello.zst hello.gz hello.txt >hello_zst_gz_txt.gz
-	$(PRGDIR)/zstd -dcf hello.*
-	$(PRGDIR)/zstd -dcf - <hello_zst_gz_txt.gz
-	$(RM) *.gz *.zst README2.md gz_zstd zstd_gz hello.txt
-
-test-zstdgrep: gzstd
-	-[ -f /tmp/zstdcat ] || ln -s $(PWD)/$(PRGDIR)/zstd /tmp/zstdcat
-	echo a | $(PRGDIR)/zstd | env ZCAT=/tmp/zstdcat $(PRGDIR)/zstdgrep a
-	echo a | $(PRGDIR)/zstd | env ZCAT=/tmp/zstdcat $(PRGDIR)/zstdgrep b && return 1 || return 0
-	-echo 'hello world' > test.txt && $(PRGDIR)/zstd test.txt
-	env ZCAT=/tmp/zstdcat $(PRGDIR)/zstdgrep hello test.txt.zst
-	env ZCAT=/tmp/zstdcat $(PRGDIR)/zstdgrep weird test.txt.zst && return 1 || return 0
-	-echo 'hello' > pattern.txt
-	env ZCAT=/tmp/zstdcat $(PRGDIR)/zstdgrep -f pattern.txt test.txt.zst
-	$(RM) test.txt test.txt.zst pattern.txt
-
-test-fullbench: fullbench datagen
-	$(QEMU_SYS) ./fullbench -i1
-	$(QEMU_SYS) ./fullbench -i1 -P0
-
-test-fullbench32: fullbench32 datagen
-	$(QEMU_SYS) ./fullbench32 -i1
-	$(QEMU_SYS) ./fullbench32 -i1 -P0
-
-test-fuzzer: fuzzer
-	$(QEMU_SYS) ./fuzzer -v $(FUZZERTEST) $(FUZZER_FLAGS)
-
-test-fuzzer-stackmode: MOREFLAGS += -DZSTD_HEAPMODE=0
-test-fuzzer-stackmode: test-fuzzer
-
-test-fuzzer32: fuzzer32
-	$(QEMU_SYS) ./fuzzer32 -v $(FUZZERTEST) $(FUZZER_FLAGS)
-
-test-zbuff: zbufftest
-	$(QEMU_SYS) ./zbufftest $(ZSTREAM_TESTTIME)
-
-test-zbuff32: zbufftest32
-	$(QEMU_SYS) ./zbufftest32 $(ZSTREAM_TESTTIME)
-
-test-zstream: zstreamtest
-	$(QEMU_SYS) ./zstreamtest -v $(ZSTREAM_TESTTIME) $(FUZZER_FLAGS)
-	$(QEMU_SYS) ./zstreamtest --mt -t1 $(ZSTREAM_TESTTIME) $(FUZZER_FLAGS)
-	$(QEMU_SYS) ./zstreamtest --newapi -t1 $(ZSTREAM_TESTTIME) $(FUZZER_FLAGS)
-
-test-zstream32: zstreamtest32
-	$(QEMU_SYS) ./zstreamtest32 $(ZSTREAM_TESTTIME) $(FUZZER_FLAGS)
-
-test-longmatch: longmatch
-	$(QEMU_SYS) ./longmatch
-
-test-bigdict: bigdict
-	$(QEMU_SYS) ./bigdict
-
-test-invalidDictionaries: invalidDictionaries
-	$(QEMU_SYS) ./invalidDictionaries
-
-test-symbols: symbols
-	$(QEMU_SYS) ./symbols
-
-test-legacy: legacy
-	$(QEMU_SYS) ./legacy
-
-test-decodecorpus: decodecorpus
-	$(QEMU_SYS) ./decodecorpus -t $(DECODECORPUS_TESTTIME)
-
-test-decodecorpus-cli: decodecorpus
-	@echo "\n ---- decodecorpus basic cli tests ----"
-	@mkdir testdir
-	./decodecorpus -n5 -otestdir -ptestdir
-	@cd testdir && \
-	$(ZSTD) -d z000000.zst -o tmp0 && \
-	$(ZSTD) -d z000001.zst -o tmp1 && \
-	$(ZSTD) -d z000002.zst -o tmp2 && \
-	$(ZSTD) -d z000003.zst -o tmp3 && \
-	$(ZSTD) -d z000004.zst -o tmp4 && \
-	diff z000000 tmp0 && \
-	diff z000001 tmp1 && \
-	diff z000002 tmp2 && \
-	diff z000003 tmp3 && \
-	diff z000004 tmp4 && \
-	rm ./* && \
-	cd ..
-	@echo "\n ---- decodecorpus dictionary cli tests ----"
-	./decodecorpus -n5 -otestdir -ptestdir --use-dict=1MB
-	@cd testdir && \
-	$(ZSTD) -d z000000.zst -D dictionary -o tmp0 && \
-	$(ZSTD) -d z000001.zst -D dictionary -o tmp1 && \
-	$(ZSTD) -d z000002.zst -D dictionary -o tmp2 && \
-	$(ZSTD) -d z000003.zst -D dictionary -o tmp3 && \
-	$(ZSTD) -d z000004.zst -D dictionary -o tmp4 && \
-	diff z000000 tmp0 && \
-	diff z000001 tmp1 && \
-	diff z000002 tmp2 && \
-	diff z000003 tmp3 && \
-	diff z000004 tmp4 && \
-	cd ..
-	@rm -rf testdir
-
-test-pool: poolTests
-	$(QEMU_SYS) ./poolTests
-
-test-lz4: ZSTD = LD_LIBRARY_PATH=/usr/local/lib $(PRGDIR)/zstd
-test-lz4: ZSTD_LZ4 = LD_LIBRARY_PATH=/usr/local/lib ./lz4
-test-lz4: ZSTD_UNLZ4 = LD_LIBRARY_PATH=/usr/local/lib ./unlz4
-test-lz4: zstd decodecorpus datagen
-	[ -f lz4 ] || ln -s $(PRGDIR)/zstd lz4
-	[ -f unlz4 ] || ln -s $(PRGDIR)/zstd unlz4
-
-	./decodecorpus -ptmp
-	# lz4 -> zstd
-	lz4 < tmp | \
-	$(ZSTD) -d | \
-	cmp - tmp
-	lz4 < tmp | \
-	$(ZSTD_UNLZ4) | \
-	cmp - tmp
-	# zstd -> lz4
-	$(ZSTD) --format=lz4 < tmp | \
-	lz4 -d | \
-	cmp - tmp
-	$(ZSTD_LZ4) < tmp | \
-	lz4 -d | \
-	cmp - tmp
-	# zstd -> zstd
-	$(ZSTD) --format=lz4 < tmp | \
-	$(ZSTD) -d | \
-	cmp - tmp
-	# zstd -> zstd
-	$(ZSTD) < tmp | \
-	$(ZSTD) -d | \
-	cmp - tmp
-
-	./datagen -g384KB | $(ZSTD) --format=lz4 | $(ZSTD) -d > /dev/null
-
-	rm tmp lz4 unlz4
-
-endif
diff --git a/tests/README.md b/tests/README.md
deleted file mode 100644
index f34501197478..000000000000
--- a/tests/README.md
+++ /dev/null
@@ -1,143 +0,0 @@
-Programs and scripts for automated testing of Zstandard
-=======================================================
-
-This directory contains the following programs and scripts:
-- `datagen` : Synthetic and parametrable data generator, for tests
-- `fullbench`  : Precisely measure speed for each zstd inner functions
-- `fuzzer`  : Test tool, to check zstd integrity on target platform
-- `paramgrill` : parameter tester for zstd
-- `test-zstd-speed.py` : script for testing zstd speed difference between commits
-- `test-zstd-versions.py` : compatibility test between zstd versions stored on Github (v0.1+)
-- `zbufftest`  : Test tool to check ZBUFF (a buffered streaming API) integrity
-- `zstreamtest` : Fuzzer test tool for zstd streaming API
-- `legacy` : Test tool to test decoding of legacy zstd frames
-- `decodecorpus` : Tool to generate valid Zstandard frames, for verifying decoder implementations
-
-
-#### `test-zstd-versions.py` - script for testing zstd interoperability between versions
-
-This script creates `versionsTest` directory to which zstd repository is cloned.
-Then all tagged (released) versions of zstd are compiled.
-In the following step interoperability between zstd versions is checked.
-
-
-#### `test-zstd-speed.py` - script for testing zstd speed difference between commits
-
-This script creates `speedTest` directory to which zstd repository is cloned.
-Then it compiles all branches of zstd and performs a speed benchmark for a given list of files (the `testFileNames` parameter).
-After `sleepTime` (an optional parameter, default 300 seconds) seconds the script checks repository for new commits.
-If a new commit is found it is compiled and a speed benchmark for this commit is performed.
-The results of the speed benchmark are compared to the previous results.
-If compression or decompression speed for one of zstd levels is lower than `lowerLimit` (an optional parameter, default 0.98) the speed benchmark is restarted.
-If second results are also lower than `lowerLimit` the warning e-mail is send to recipients from the list (the `emails` parameter).
-
-Additional remarks:
-- To be sure that speed results are accurate the script should be run on a "stable" target system with no other jobs running in parallel
-- Using the script with virtual machines can lead to large variations of speed results
-- The speed benchmark is not performed until computers' load average is lower than `maxLoadAvg` (an optional parameter, default 0.75)
-- The script sends e-mails using `mutt`; if `mutt` is not available it sends e-mails without attachments using `mail`; if both are not available it only prints a warning
-
-
-The example usage with two test files, one e-mail address, and with an additional message:
-```
-./test-zstd-speed.py "silesia.tar calgary.tar" "email@gmail.com" --message "tested on my laptop" --sleepTime 60
-```
-
-To run the script in background please use:
-```
-nohup ./test-zstd-speed.py testFileNames emails &
-```
-
-The full list of parameters:
-```
-positional arguments:
-  testFileNames         file names list for speed benchmark
-  emails                list of e-mail addresses to send warnings
-
-optional arguments:
-  -h, --help            show this help message and exit
-  --message MESSAGE     attach an additional message to e-mail
-  --lowerLimit LOWERLIMIT
-                        send email if speed is lower than given limit
-  --maxLoadAvg MAXLOADAVG
-                        maximum load average to start testing
-  --lastCLevel LASTCLEVEL
-                        last compression level for testing
-  --sleepTime SLEEPTIME
-                        frequency of repository checking in seconds
-```
-
-#### `decodecorpus` - tool to generate Zstandard frames for decoder testing
-Command line tool to generate test .zst files.
-
-This tool will generate .zst files with checksums,
-as well as optionally output the corresponding correct uncompressed data for
-extra verification.
-
-Example:
-```
-./decodecorpus -ptestfiles -otestfiles -n10000 -s5
-```
-will generate 10,000 sample .zst files using a seed of 5 in the `testfiles` directory,
-with the zstd checksum field set,
-as well as the 10,000 original files for more detailed comparison of decompression results.
-
-```
-./decodecorpus -t -T1mn
-```
-will choose a random seed, and for 1 minute,
-generate random test frames and ensure that the
-zstd library correctly decompresses them in both simple and streaming modes.
-
-#### `paramgrill` - tool for generating compression table parameters and optimizing parameters on file given constraints
-
-Full list of arguments
-```
- -T#          : set level 1 speed objective
- -B#          : cut input into blocks of size # (default : single block)
- -S           : benchmarks a single run (example command: -Sl3w10h12)
-    w# - windowLog
-    h# - hashLog
-    c# - chainLog
-    s# - searchLog
-    l# - minMatch
-    t# - targetLength
-    S# - strategy
-    L# - level
- --zstd=      : Single run, parameter selection syntax same as zstdcli with more parameters
-                    (Added forceAttachDictionary / fadt)
-                    When invoked with --optimize, this represents the sample to exceed.
- --optimize=  : find parameters to maximize compression ratio given parameters
-                    Can use all --zstd= commands to constrain the type of solution found in addition to the following constraints
-    cSpeed=   : Minimum compression speed
-    dSpeed=   : Minimum decompression speed
-    cMem=     : Maximum compression memory
-    lvl=      : Searches for solutions which are strictly better than that compression lvl in ratio and cSpeed,
-    stc=      : When invoked with lvl=, represents percentage slack in ratio/cSpeed allowed for a solution to be considered (Default 100%)
-              : In normal operation, represents percentage slack in choosing viable starting strategy selection in choosing the default parameters
-                    (Lower value will begin with stronger strategies) (Default 90%)
-    speedRatio=   (accepts decimals)
-              : determines value of gains in speed vs gains in ratio
-                    when determining overall winner (default 5 (1% ratio = 5% speed)).
-    tries=    : Maximum number of random restarts on a single strategy before switching (Default 5)
-                    Higher values will make optimizer run longer, more chances to find better solution.
-    memLog    : Limits the log of the size of each memotable (1 per strategy). Will use hash tables when state space is larger than max size.
-                    Setting memLog = 0 turns off memoization
- --display=   : specify which parameters are included in the output
-                    can use all --zstd parameter names and 'cParams' as a shorthand for all parameters used in ZSTD_compressionParameters
-                    (Default: display all params available)
- -P#          : generated sample compressibility (when no file is provided)
- -t#          : Caps runtime of operation in seconds (default : 99999 seconds (about 27 hours ))
- -v           : Prints Benchmarking output
- -D           : Next argument dictionary file
- -s           : Benchmark all files separately
- -q           : Quiet, repeat for more quiet
-                  -q Prints parameters + results whenever a new best is found
-                  -qq Only prints parameters whenever a new best is found, prints final parameters + results
-                  -qqq Only print final parameters + results
-                  -qqqq Only prints final parameter set in the form --zstd=
- -v           : Verbose, cancels quiet, repeat for more volume
-                  -v Prints all candidate parameters and results
-
-```
- Any inputs afterwards are treated as files to benchmark.
diff --git a/tests/bigdict.c b/tests/bigdict.c
deleted file mode 100644
index 11501f6df0fa..000000000000
--- a/tests/bigdict.c
+++ /dev/null
@@ -1,128 +0,0 @@
-/*
- * Copyright (c) 2017-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#include <assert.h>
-#include <stdio.h>
-#include <stddef.h>
-#include <stdlib.h>
-#include <stdint.h>
-#include "datagen.h"
-#include "mem.h"
-#define ZSTD_STATIC_LINKING_ONLY
-#include "zstd.h"
-
-static int
-compress(ZSTD_CCtx* cctx, ZSTD_DCtx* dctx,
-         void* dst, size_t dstCapacity,
-         void const* src, size_t srcSize,
-         void* roundtrip, ZSTD_EndDirective end)
-{
-    ZSTD_inBuffer in = {src, srcSize, 0};
-    ZSTD_outBuffer out = {dst, dstCapacity, 0};
-    int ended = 0;
-
-    while (!ended && (in.pos < in.size || out.pos > 0)) {
-        size_t rc;
-        out.pos = 0;
-        rc = ZSTD_compressStream2(cctx, &out, &in, end);
-        if (ZSTD_isError(rc))
-            return 1;
-        if (end == ZSTD_e_end && rc == 0)
-            ended = 1;
-        {
-            ZSTD_inBuffer rtIn = {dst, out.pos, 0};
-            ZSTD_outBuffer rtOut = {roundtrip, srcSize, 0};
-            rc = 1;
-            while (rtIn.pos < rtIn.size || rtOut.pos > 0) {
-                rtOut.pos = 0;
-                rc = ZSTD_decompressStream(dctx, &rtOut, &rtIn);
-                if (ZSTD_isError(rc)) {
-                    fprintf(stderr, "Decompression error: %s\n", ZSTD_getErrorName(rc));
-                    return 1;
-                }
-                if (rc == 0)
-                    break;
-            }
-            if (ended && rc != 0) {
-                fprintf(stderr, "Frame not finished!\n");
-                return 1;
-            }
-        }
-    }
-
-    return 0;
-}
-
-int main(int argc, const char** argv)
-{
-    ZSTD_CCtx* cctx = ZSTD_createCCtx();
-    ZSTD_DCtx* dctx = ZSTD_createDCtx();
-    const size_t dataSize = (size_t)1 << 30;
-    const size_t outSize = ZSTD_compressBound(dataSize);
-    const size_t bufferSize = (size_t)1 << 31;
-    char* buffer = (char*)malloc(bufferSize);
-    void* out = malloc(outSize);
-    void* roundtrip = malloc(dataSize);
-    (void)argc;
-    (void)argv;
-
-    if (!buffer || !out || !roundtrip || !cctx || !dctx) {
-        fprintf(stderr, "Allocation failure\n");
-        return 1;
-    }
-
-    if (ZSTD_isError(ZSTD_CCtx_setParameter(cctx, ZSTD_c_windowLog, 31)))
-        return 1;
-    if (ZSTD_isError(ZSTD_CCtx_setParameter(cctx, ZSTD_c_nbWorkers, 1)))
-        return 1;
-    if (ZSTD_isError(ZSTD_CCtx_setParameter(cctx, ZSTD_c_overlapLog, 9)))
-        return 1;
-    if (ZSTD_isError(ZSTD_CCtx_setParameter(cctx, ZSTD_c_checksumFlag, 1)))
-        return 1;
-    if (ZSTD_isError(ZSTD_CCtx_setParameter(cctx, ZSTD_c_strategy, ZSTD_btopt)))
-        return 1;
-    if (ZSTD_isError(ZSTD_CCtx_setParameter(cctx, ZSTD_c_targetLength, 7)))
-        return 1;
-    if (ZSTD_isError(ZSTD_CCtx_setParameter(cctx, ZSTD_c_minMatch, 7)))
-        return 1;
-    if (ZSTD_isError(ZSTD_CCtx_setParameter(cctx, ZSTD_c_searchLog, 1)))
-        return 1;
-    if (ZSTD_isError(ZSTD_CCtx_setParameter(cctx, ZSTD_c_hashLog, 10)))
-        return 1;
-    if (ZSTD_isError(ZSTD_CCtx_setParameter(cctx, ZSTD_c_chainLog, 10)))
-        return 1;
-
-    if (ZSTD_isError(ZSTD_DCtx_setParameter(dctx, ZSTD_d_windowLogMax, 31)))
-        return 1;
-
-    RDG_genBuffer(buffer, bufferSize, 1.0, 0.0, 0xbeefcafe);
-
-    /* Compress 30 GB */
-    {
-        int i;
-        for (i = 0; i < 10; ++i) {
-            fprintf(stderr, "Compressing 1 GB\n");
-            if (compress(cctx, dctx, out, outSize, buffer, dataSize, roundtrip, ZSTD_e_continue))
-                return 1;
-        }
-    }
-    fprintf(stderr, "Compressing 1 GB\n");
-    if (compress(cctx, dctx, out, outSize, buffer, dataSize, roundtrip, ZSTD_e_end))
-        return 1;
-
-    fprintf(stderr, "Success!\n");
-
-    free(roundtrip);
-    free(out);
-    free(buffer);
-    ZSTD_freeDCtx(dctx);
-    ZSTD_freeCCtx(cctx);
-    return 0;
-}
diff --git a/tests/checkTag.c b/tests/checkTag.c
deleted file mode 100644
index fda3fd1f9df5..000000000000
--- a/tests/checkTag.c
+++ /dev/null
@@ -1,65 +0,0 @@
-/*
- * Copyright (c) 2018-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-/* checkTag : validation tool for libzstd
- * command :
- * $ ./checkTag tag
- * checkTag validates tags of following format : v[0-9].[0-9].[0-9]{any}
- * The tag is then compared to zstd version number.
- * They are compatible if first 3 digits are identical.
- * Anything beyond that is free, and doesn't impact validation.
- * Example : tag v1.8.1.2 is compatible with version 1.8.1
- * When tag and version are not compatible, program exits with error code 1.
- * When they are compatible, it exists with a code 0.
- * checkTag is intended to be used in automated testing environment.
- */
-
-#include <stdio.h>   /* printf */
-#include <string.h>  /* strlen, strncmp */
-#include "zstd.h"    /* ZSTD_VERSION_STRING */
-
-
-/*  validate() :
- * @return 1 if tag is compatible, 0 if not.
- */
-static int validate(const char* const tag)
-{
-    size_t const tagLength = strlen(tag);
-    size_t const verLength = strlen(ZSTD_VERSION_STRING);
-
-    if (tagLength < 2) return 0;
-    if (tag[0] != 'v') return 0;
-    if (tagLength <= verLength) return 0;
-
-    if (strncmp(ZSTD_VERSION_STRING, tag+1, verLength)) return 0;
-
-    return 1;
-}
-
-int main(int argc, const char** argv)
-{
-    const char* const exeName = argv[0];
-    const char* const tag = argv[1];
-    if (argc!=2) {
-        printf("incorrect usage : %s tag \n", exeName);
-        return 2;
-    }
-
-    printf("Version : %s \n", ZSTD_VERSION_STRING);
-    printf("Tag     : %s \n", tag);
-
-    if (validate(tag)) {
-        printf("OK : tag is compatible with zstd version \n");
-        return 0;
-    }
-
-    printf("!! error : tag and versions are not compatible !! \n");
-    return 1;
-}
diff --git a/tests/datagencli.c b/tests/datagencli.c
deleted file mode 100644
index 6c1dd4719edc..000000000000
--- a/tests/datagencli.c
+++ /dev/null
@@ -1,130 +0,0 @@
-/*
- * Copyright (c) 2015-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-
-/*-************************************
-*  Dependencies
-**************************************/
-#include "util.h"      /* Compiler options */
-#include <stdio.h>     /* fprintf, stderr */
-#include "datagen.h"   /* RDG_generate */
-
-
-/*-************************************
-*  Constants
-**************************************/
-#define KB *(1 <<10)
-#define MB *(1 <<20)
-#define GB *(1U<<30)
-
-#define SIZE_DEFAULT ((64 KB) + 1)
-#define SEED_DEFAULT 0
-#define COMPRESSIBILITY_DEFAULT 50
-
-
-/*-************************************
-*  Macros
-**************************************/
-#define DISPLAY(...)         fprintf(stderr, __VA_ARGS__)
-#define DISPLAYLEVEL(l, ...) if (displayLevel>=l) { DISPLAY(__VA_ARGS__); }
-static unsigned displayLevel = 2;
-
-
-/*-*******************************************************
-*  Command line
-*********************************************************/
-static int usage(const char* programName)
-{
-    DISPLAY( "Compressible data generator\n");
-    DISPLAY( "Usage :\n");
-    DISPLAY( "      %s [args]\n", programName);
-    DISPLAY( "\n");
-    DISPLAY( "Arguments :\n");
-    DISPLAY( " -g#    : generate # data (default:%i)\n", SIZE_DEFAULT);
-    DISPLAY( " -s#    : Select seed (default:%i)\n", SEED_DEFAULT);
-    DISPLAY( " -P#    : Select compressibility in %% (default:%i%%)\n",
-                        COMPRESSIBILITY_DEFAULT);
-    DISPLAY( " -h     : display help and exit\n");
-    return 0;
-}
-
-
-int main(int argc, const char** argv)
-{
-    unsigned probaU32 = COMPRESSIBILITY_DEFAULT;
-    double litProba = 0.0;
-    U64 size = SIZE_DEFAULT;
-    U32 seed = SEED_DEFAULT;
-    const char* const programName = argv[0];
-
-    int argNb;
-    for(argNb=1; argNb<argc; argNb++) {
-        const char* argument = argv[argNb];
-
-        if(!argument) continue;   /* Protection if argument empty */
-
-        /* Handle commands. Aggregated commands are allowed */
-        if (*argument=='-') {
-            argument++;
-            while (*argument!=0) {
-                switch(*argument)
-                {
-                case 'h':
-                    return usage(programName);
-                case 'g':
-                    argument++;
-                    size=0;
-                    while ((*argument>='0') && (*argument<='9'))
-                        size *= 10, size += *argument++ - '0';
-                    if (*argument=='K') { size <<= 10; argument++; }
-                    if (*argument=='M') { size <<= 20; argument++; }
-                    if (*argument=='G') { size <<= 30; argument++; }
-                    if (*argument=='B') { argument++; }
-                    break;
-                case 's':
-                    argument++;
-                    seed=0;
-                    while ((*argument>='0') && (*argument<='9'))
-                        seed *= 10, seed += *argument++ - '0';
-                    break;
-                case 'P':
-                    argument++;
-                    probaU32 = 0;
-                    while ((*argument>='0') && (*argument<='9'))
-                        probaU32 *= 10, probaU32 += *argument++ - '0';
-                    if (probaU32>100) probaU32 = 100;
-                    break;
-                case 'L':   /* hidden argument : Literal distribution probability */
-                    argument++;
-                    litProba=0.;
-                    while ((*argument>='0') && (*argument<='9'))
-                        litProba *= 10, litProba += *argument++ - '0';
-                    if (litProba>100.) litProba=100.;
-                    litProba /= 100.;
-                    break;
-                case 'v':
-                    displayLevel = 4;
-                    argument++;
-                    break;
-                default:
-                    return usage(programName);
-                }
-    }   }   }   /* for(argNb=1; argNb<argc; argNb++) */
-
-    DISPLAYLEVEL(4, "Compressible data Generator \n");
-    if (probaU32!=COMPRESSIBILITY_DEFAULT)
-        DISPLAYLEVEL(3, "Compressibility : %i%%\n", probaU32);
-    DISPLAYLEVEL(3, "Seed = %u \n", (unsigned)seed);
-
-    RDG_genStdout(size, (double)probaU32/100, litProba, seed);
-    DISPLAYLEVEL(1, "\n");
-
-    return 0;
-}
diff --git a/tests/decodecorpus.c b/tests/decodecorpus.c
deleted file mode 100644
index 91873ba46031..000000000000
--- a/tests/decodecorpus.c
+++ /dev/null
@@ -1,1932 +0,0 @@
-/*
- * Copyright (c) 2017-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#include <limits.h>
-#include <math.h>
-#include <stddef.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-
-#include "util.h"
-#include "timefn.h"   /* UTIL_clockSpanMicro, SEC_TO_MICRO, UTIL_TIME_INITIALIZER */
-#include "zstd.h"
-#include "zstd_internal.h"
-#include "mem.h"
-#define ZDICT_STATIC_LINKING_ONLY
-#include "zdict.h"
-
-/* Direct access to internal compression functions is required */
-#include "zstd_compress.c"
-
-#define XXH_STATIC_LINKING_ONLY
-#include "xxhash.h"     /* XXH64 */
-
-#ifndef MIN
-    #define MIN(a, b) ((a) < (b) ? (a) : (b))
-#endif
-
-#ifndef MAX_PATH
-    #ifdef PATH_MAX
-        #define MAX_PATH PATH_MAX
-    #else
-        #define MAX_PATH 256
-    #endif
-#endif
-
-/*-************************************
-*  DISPLAY Macros
-**************************************/
-#define DISPLAY(...)          fprintf(stderr, __VA_ARGS__)
-#define DISPLAYLEVEL(l, ...)  if (g_displayLevel>=l) { DISPLAY(__VA_ARGS__); }
-static U32 g_displayLevel = 2;
-
-#define DISPLAYUPDATE(...)                                                     \
-    do {                                                                       \
-        if ((UTIL_clockSpanMicro(g_displayClock) > g_refreshRate) ||           \
-            (g_displayLevel >= 4)) {                                           \
-            g_displayClock = UTIL_getTime();                                   \
-            DISPLAY(__VA_ARGS__);                                              \
-            if (g_displayLevel >= 4) fflush(stderr);                           \
-        }                                                                      \
-    } while (0)
-
-static const U64 g_refreshRate = SEC_TO_MICRO / 6;
-static UTIL_time_t g_displayClock = UTIL_TIME_INITIALIZER;
-
-#define CHECKERR(code)                                                         \
-    do {                                                                       \
-        if (ZSTD_isError(code)) {                                              \
-            DISPLAY("Error occurred while generating data: %s\n",              \
-                    ZSTD_getErrorName(code));                                  \
-            exit(1);                                                           \
-        }                                                                      \
-    } while (0)
-
-/*-*******************************************************
-*  Random function
-*********************************************************/
-static U32 RAND(U32* src)
-{
-#define RAND_rotl32(x,r) ((x << r) | (x >> (32 - r)))
-    static const U32 prime1 = 2654435761U;
-    static const U32 prime2 = 2246822519U;
-    U32 rand32 = *src;
-    rand32 *= prime1;
-    rand32 += prime2;
-    rand32  = RAND_rotl32(rand32, 13);
-    *src = rand32;
-    return RAND_rotl32(rand32, 27);
-#undef RAND_rotl32
-}
-
-#define DISTSIZE (8192)
-
-/* Write `size` bytes into `ptr`, all of which are less than or equal to `maxSymb` */
-static void RAND_bufferMaxSymb(U32* seed, void* ptr, size_t size, int maxSymb)
-{
-    size_t i;
-    BYTE* op = ptr;
-
-    for (i = 0; i < size; i++) {
-        op[i] = (BYTE) (RAND(seed) % (maxSymb + 1));
-    }
-}
-
-/* Write `size` random bytes into `ptr` */
-static void RAND_buffer(U32* seed, void* ptr, size_t size)
-{
-    size_t i;
-    BYTE* op = ptr;
-
-    for (i = 0; i + 4 <= size; i += 4) {
-        MEM_writeLE32(op + i, RAND(seed));
-    }
-    for (; i < size; i++) {
-        op[i] = RAND(seed) & 0xff;
-    }
-}
-
-/* Write `size` bytes into `ptr` following the distribution `dist` */
-static void RAND_bufferDist(U32* seed, BYTE* dist, void* ptr, size_t size)
-{
-    size_t i;
-    BYTE* op = ptr;
-
-    for (i = 0; i < size; i++) {
-        op[i] = dist[RAND(seed) % DISTSIZE];
-    }
-}
-
-/* Generate a random distribution where the frequency of each symbol follows a
- * geometric distribution defined by `weight`
- * `dist` should have size at least `DISTSIZE` */
-static void RAND_genDist(U32* seed, BYTE* dist, double weight)
-{
-    size_t i = 0;
-    size_t statesLeft = DISTSIZE;
-    BYTE symb = (BYTE) (RAND(seed) % 256);
-    BYTE step = (BYTE) ((RAND(seed) % 256) | 1); /* force it to be odd so it's relatively prime to 256 */
-
-    while (i < DISTSIZE) {
-        size_t states = ((size_t)(weight * statesLeft)) + 1;
-        size_t j;
-        for (j = 0; j < states && i < DISTSIZE; j++, i++) {
-            dist[i] = symb;
-        }
-
-        symb += step;
-        statesLeft -= states;
-    }
-}
-
-/* Generates a random number in the range [min, max) */
-static inline U32 RAND_range(U32* seed, U32 min, U32 max)
-{
-    return (RAND(seed) % (max-min)) + min;
-}
-
-#define ROUND(x) ((U32)(x + 0.5))
-
-/* Generates a random number in an exponential distribution with mean `mean` */
-static double RAND_exp(U32* seed, double mean)
-{
-    double const u = RAND(seed) / (double) UINT_MAX;
-    return log(1-u) * (-mean);
-}
-
-/*-*******************************************************
-*  Constants and Structs
-*********************************************************/
-const char *BLOCK_TYPES[] = {"raw", "rle", "compressed"};
-
-#define MAX_DECOMPRESSED_SIZE_LOG 20
-#define MAX_DECOMPRESSED_SIZE (1ULL << MAX_DECOMPRESSED_SIZE_LOG)
-
-#define MAX_WINDOW_LOG 22 /* Recommended support is 8MB, so limit to 4MB + mantissa */
-
-#define MIN_SEQ_LEN (3)
-#define MAX_NB_SEQ ((ZSTD_BLOCKSIZE_MAX + MIN_SEQ_LEN - 1) / MIN_SEQ_LEN)
-
-BYTE CONTENT_BUFFER[MAX_DECOMPRESSED_SIZE];
-BYTE FRAME_BUFFER[MAX_DECOMPRESSED_SIZE * 2];
-BYTE LITERAL_BUFFER[ZSTD_BLOCKSIZE_MAX];
-
-seqDef SEQUENCE_BUFFER[MAX_NB_SEQ];
-BYTE SEQUENCE_LITERAL_BUFFER[ZSTD_BLOCKSIZE_MAX]; /* storeSeq expects a place to copy literals to */
-BYTE SEQUENCE_LLCODE[ZSTD_BLOCKSIZE_MAX];
-BYTE SEQUENCE_MLCODE[ZSTD_BLOCKSIZE_MAX];
-BYTE SEQUENCE_OFCODE[ZSTD_BLOCKSIZE_MAX];
-
-unsigned WKSP[1024];
-
-typedef struct {
-    size_t contentSize; /* 0 means unknown (unless contentSize == windowSize == 0) */
-    unsigned windowSize; /* contentSize >= windowSize means single segment */
-} frameHeader_t;
-
-/* For repeat modes */
-typedef struct {
-    U32 rep[ZSTD_REP_NUM];
-
-    int hufInit;
-    /* the distribution used in the previous block for repeat mode */
-    BYTE hufDist[DISTSIZE];
-    U32 hufTable [256]; /* HUF_CElt is an incomplete type */
-
-    int fseInit;
-    FSE_CTable offcodeCTable  [FSE_CTABLE_SIZE_U32(OffFSELog, MaxOff)];
-    FSE_CTable matchlengthCTable[FSE_CTABLE_SIZE_U32(MLFSELog, MaxML)];
-    FSE_CTable litlengthCTable  [FSE_CTABLE_SIZE_U32(LLFSELog, MaxLL)];
-
-    /* Symbols that were present in the previous distribution, for use with
-     * set_repeat */
-    BYTE litlengthSymbolSet[36];
-    BYTE offsetSymbolSet[29];
-    BYTE matchlengthSymbolSet[53];
-} cblockStats_t;
-
-typedef struct {
-    void* data;
-    void* dataStart;
-    void* dataEnd;
-
-    void* src;
-    void* srcStart;
-    void* srcEnd;
-
-    frameHeader_t header;
-
-    cblockStats_t stats;
-    cblockStats_t oldStats; /* so they can be rolled back if uncompressible */
-} frame_t;
-
-typedef struct {
-    int useDict;
-    U32 dictID;
-    size_t dictContentSize;
-    BYTE* dictContent;
-} dictInfo;
-
-typedef enum {
-  gt_frame = 0,  /* generate frames */
-  gt_block,      /* generate compressed blocks without block/frame headers */
-} genType_e;
-
-/*-*******************************************************
-*  Global variables (set from command line)
-*********************************************************/
-U32 g_maxDecompressedSizeLog = MAX_DECOMPRESSED_SIZE_LOG;  /* <= 20 */
-U32 g_maxBlockSize = ZSTD_BLOCKSIZE_MAX;                       /* <= 128 KB */
-
-/*-*******************************************************
-*  Generator Functions
-*********************************************************/
-
-struct {
-    int contentSize; /* force the content size to be present */
-} opts; /* advanced options on generation */
-
-/* Generate and write a random frame header */
-static void writeFrameHeader(U32* seed, frame_t* frame, dictInfo info)
-{
-    BYTE* const op = frame->data;
-    size_t pos = 0;
-    frameHeader_t fh;
-
-    BYTE windowByte = 0;
-
-    int singleSegment = 0;
-    int contentSizeFlag = 0;
-    int fcsCode = 0;
-
-    memset(&fh, 0, sizeof(fh));
-
-    /* generate window size */
-    {
-        /* Follow window algorithm from specification */
-        int const exponent = RAND(seed) % (MAX_WINDOW_LOG - 10);
-        int const mantissa = RAND(seed) % 8;
-        windowByte = (BYTE) ((exponent << 3) | mantissa);
-        fh.windowSize = (1U << (exponent + 10));
-        fh.windowSize += fh.windowSize / 8 * mantissa;
-    }
-
-    {
-        /* Generate random content size */
-        size_t highBit;
-        if (RAND(seed) & 7 && g_maxDecompressedSizeLog > 7) {
-            /* do content of at least 128 bytes */
-            highBit = 1ULL << RAND_range(seed, 7, g_maxDecompressedSizeLog);
-        } else if (RAND(seed) & 3) {
-            /* do small content */
-            highBit = 1ULL << RAND_range(seed, 0, MIN(7, 1U << g_maxDecompressedSizeLog));
-        } else {
-            /* 0 size frame */
-            highBit = 0;
-        }
-        fh.contentSize = highBit ? highBit + (RAND(seed) % highBit) : 0;
-
-        /* provide size sometimes */
-        contentSizeFlag = opts.contentSize | (RAND(seed) & 1);
-
-        if (contentSizeFlag && (fh.contentSize == 0 || !(RAND(seed) & 7))) {
-            /* do single segment sometimes */
-            fh.windowSize = (U32) fh.contentSize;
-            singleSegment = 1;
-        }
-    }
-
-    if (contentSizeFlag) {
-        /* Determine how large fcs field has to be */
-        int minFcsCode = (fh.contentSize >= 256) +
-                               (fh.contentSize >= 65536 + 256) +
-                               (fh.contentSize > 0xFFFFFFFFU);
-        if (!singleSegment && !minFcsCode) {
-            minFcsCode = 1;
-        }
-        fcsCode = minFcsCode + (RAND(seed) % (4 - minFcsCode));
-        if (fcsCode == 1 && fh.contentSize < 256) fcsCode++;
-    }
-
-    /* write out the header */
-    MEM_writeLE32(op + pos, ZSTD_MAGICNUMBER);
-    pos += 4;
-
-    {
-        /*
-         * fcsCode: 2-bit flag specifying how many bytes used to represent Frame_Content_Size (bits 7-6)
-         * singleSegment: 1-bit flag describing if data must be regenerated within a single continuous memory segment. (bit 5)
-         * contentChecksumFlag: 1-bit flag that is set if frame includes checksum at the end -- set to 1 below (bit 2)
-         * dictBits: 2-bit flag describing how many bytes Dictionary_ID uses -- set to 3 (bits 1-0)
-         * For more information: https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#frame_header
-         */
-        int const dictBits = info.useDict ? 3 : 0;
-        BYTE const frameHeaderDescriptor =
-                (BYTE) ((fcsCode << 6) | (singleSegment << 5) | (1 << 2) | dictBits);
-        op[pos++] = frameHeaderDescriptor;
-    }
-
-    if (!singleSegment) {
-        op[pos++] = windowByte;
-    }
-    if (info.useDict) {
-        MEM_writeLE32(op + pos, (U32) info.dictID);
-        pos += 4;
-    }
-    if (contentSizeFlag) {
-        switch (fcsCode) {
-        default: /* Impossible */
-        case 0: op[pos++] = (BYTE) fh.contentSize; break;
-        case 1: MEM_writeLE16(op + pos, (U16) (fh.contentSize - 256)); pos += 2; break;
-        case 2: MEM_writeLE32(op + pos, (U32) fh.contentSize); pos += 4; break;
-        case 3: MEM_writeLE64(op + pos, (U64) fh.contentSize); pos += 8; break;
-        }
-    }
-
-    DISPLAYLEVEL(3, " frame content size:\t%u\n", (unsigned)fh.contentSize);
-    DISPLAYLEVEL(3, " frame window size:\t%u\n", fh.windowSize);
-    DISPLAYLEVEL(3, " content size flag:\t%d\n", contentSizeFlag);
-    DISPLAYLEVEL(3, " single segment flag:\t%d\n", singleSegment);
-
-    frame->data = op + pos;
-    frame->header = fh;
-}
-
-/* Write a literal block in either raw or RLE form, return the literals size */
-static size_t writeLiteralsBlockSimple(U32* seed, frame_t* frame, size_t contentSize)
-{
-    BYTE* op = (BYTE*)frame->data;
-    int const type = RAND(seed) % 2;
-    int const sizeFormatDesc = RAND(seed) % 8;
-    size_t litSize;
-    size_t maxLitSize = MIN(contentSize, g_maxBlockSize);
-
-    if (sizeFormatDesc == 0) {
-        /* Size_FormatDesc = ?0 */
-        maxLitSize = MIN(maxLitSize, 31);
-    } else if (sizeFormatDesc <= 4) {
-        /* Size_FormatDesc = 01 */
-        maxLitSize = MIN(maxLitSize, 4095);
-    } else {
-        /* Size_Format = 11 */
-        maxLitSize = MIN(maxLitSize, 1048575);
-    }
-
-    litSize = RAND(seed) % (maxLitSize + 1);
-    if (frame->src == frame->srcStart && litSize == 0) {
-        litSize = 1; /* no empty literals if there's nothing preceding this block */
-    }
-    if (litSize + 3 > contentSize) {
-        litSize = contentSize; /* no matches shorter than 3 are allowed */
-    }
-    /* use smallest size format that fits */
-    if (litSize < 32) {
-        op[0] = (type | (0 << 2) | (litSize << 3)) & 0xff;
-        op += 1;
-    } else if (litSize < 4096) {
-        op[0] = (type | (1 << 2) | (litSize << 4)) & 0xff;
-        op[1] = (litSize >> 4) & 0xff;
-        op += 2;
-    } else {
-        op[0] = (type | (3 << 2) | (litSize << 4)) & 0xff;
-        op[1] = (litSize >> 4) & 0xff;
-        op[2] = (litSize >> 12) & 0xff;
-        op += 3;
-    }
-
-    if (type == 0) {
-        /* Raw literals */
-        DISPLAYLEVEL(4, "   raw literals\n");
-
-        RAND_buffer(seed, LITERAL_BUFFER, litSize);
-        memcpy(op, LITERAL_BUFFER, litSize);
-        op += litSize;
-    } else {
-        /* RLE literals */
-        BYTE const symb = (BYTE) (RAND(seed) % 256);
-
-        DISPLAYLEVEL(4, "   rle literals: 0x%02x\n", (unsigned)symb);
-
-        memset(LITERAL_BUFFER, symb, litSize);
-        op[0] = symb;
-        op++;
-    }
-
-    frame->data = op;
-
-    return litSize;
-}
-
-/* Generate a Huffman header for the given source */
-static size_t writeHufHeader(U32* seed, HUF_CElt* hufTable, void* dst, size_t dstSize,
-                                 const void* src, size_t srcSize)
-{
-    BYTE* const ostart = (BYTE*)dst;
-    BYTE* op = ostart;
-
-    unsigned huffLog = 11;
-    unsigned maxSymbolValue = 255;
-
-    unsigned count[HUF_SYMBOLVALUE_MAX+1];
-
-    /* Scan input and build symbol stats */
-    {   size_t const largest = HIST_count_wksp (count, &maxSymbolValue, (const BYTE*)src, srcSize, WKSP, sizeof(WKSP));
-        assert(!HIST_isError(largest));
-        if (largest == srcSize) { *ostart = ((const BYTE*)src)[0]; return 0; }   /* single symbol, rle */
-        if (largest <= (srcSize >> 7)+1) return 0;   /* Fast heuristic : not compressible enough */
-    }
-
-    /* Build Huffman Tree */
-    /* Max Huffman log is 11, min is highbit(maxSymbolValue)+1 */
-    huffLog = RAND_range(seed, ZSTD_highbit32(maxSymbolValue)+1, huffLog+1);
-    DISPLAYLEVEL(6, "     huffman log: %u\n", huffLog);
-    {   size_t const maxBits = HUF_buildCTable_wksp (hufTable, count, maxSymbolValue, huffLog, WKSP, sizeof(WKSP));
-        CHECKERR(maxBits);
-        huffLog = (U32)maxBits;
-    }
-
-    /* Write table description header */
-    {   size_t const hSize = HUF_writeCTable (op, dstSize, hufTable, maxSymbolValue, huffLog);
-        if (hSize + 12 >= srcSize) return 0;   /* not useful to try compression */
-        op += hSize;
-    }
-
-    return op - ostart;
-}
-
-/* Write a Huffman coded literals block and return the literals size */
-static size_t writeLiteralsBlockCompressed(U32* seed, frame_t* frame, size_t contentSize)
-{
-    BYTE* origop = (BYTE*)frame->data;
-    BYTE* opend = (BYTE*)frame->dataEnd;
-    BYTE* op;
-    BYTE* const ostart = origop;
-    int const sizeFormat = RAND(seed) % 4;
-    size_t litSize;
-    size_t hufHeaderSize = 0;
-    size_t compressedSize = 0;
-    size_t maxLitSize = MIN(contentSize-3, g_maxBlockSize);
-
-    symbolEncodingType_e hType;
-
-    if (contentSize < 64) {
-        /* make sure we get reasonably-sized literals for compression */
-        return ERROR(GENERIC);
-    }
-
-    DISPLAYLEVEL(4, "   compressed literals\n");
-
-    switch (sizeFormat) {
-    case 0: /* fall through, size is the same as case 1 */
-    case 1:
-        maxLitSize = MIN(maxLitSize, 1023);
-        origop += 3;
-        break;
-    case 2:
-        maxLitSize = MIN(maxLitSize, 16383);
-        origop += 4;
-        break;
-    case 3:
-        maxLitSize = MIN(maxLitSize, 262143);
-        origop += 5;
-        break;
-    default:; /* impossible */
-    }
-
-    do {
-        op = origop;
-        do {
-            litSize = RAND(seed) % (maxLitSize + 1);
-        } while (litSize < 32); /* avoid small literal sizes */
-        if (litSize + 3 > contentSize) {
-            litSize = contentSize; /* no matches shorter than 3 are allowed */
-        }
-
-        /* most of the time generate a new distribution */
-        if ((RAND(seed) & 3) || !frame->stats.hufInit) {
-            do {
-                if (RAND(seed) & 3) {
-                    /* add 10 to ensure some compressibility */
-                    double const weight = ((RAND(seed) % 90) + 10) / 100.0;
-
-                    DISPLAYLEVEL(5, "    distribution weight: %d%%\n",
-                                 (int)(weight * 100));
-
-                    RAND_genDist(seed, frame->stats.hufDist, weight);
-                } else {
-                    /* sometimes do restricted range literals to force
-                     * non-huffman headers */
-                    DISPLAYLEVEL(5, "    small range literals\n");
-                    RAND_bufferMaxSymb(seed, frame->stats.hufDist, DISTSIZE,
-                                       15);
-                }
-                RAND_bufferDist(seed, frame->stats.hufDist, LITERAL_BUFFER,
-                                litSize);
-
-                /* generate the header from the distribution instead of the
-                 * actual data to avoid bugs with symbols that were in the
-                 * distribution but never showed up in the output */
-                hufHeaderSize = writeHufHeader(
-                        seed, (HUF_CElt*)frame->stats.hufTable, op, opend - op,
-                        frame->stats.hufDist, DISTSIZE);
-                CHECKERR(hufHeaderSize);
-                /* repeat until a valid header is written */
-            } while (hufHeaderSize == 0);
-            op += hufHeaderSize;
-            hType = set_compressed;
-
-            frame->stats.hufInit = 1;
-        } else {
-            /* repeat the distribution/table from last time */
-            DISPLAYLEVEL(5, "    huffman repeat stats\n");
-            RAND_bufferDist(seed, frame->stats.hufDist, LITERAL_BUFFER,
-                            litSize);
-            hufHeaderSize = 0;
-            hType = set_repeat;
-        }
-
-        do {
-            compressedSize =
-                    sizeFormat == 0
-                            ? HUF_compress1X_usingCTable(
-                                      op, opend - op, LITERAL_BUFFER, litSize,
-                                      (HUF_CElt*)frame->stats.hufTable)
-                            : HUF_compress4X_usingCTable(
-                                      op, opend - op, LITERAL_BUFFER, litSize,
-                                      (HUF_CElt*)frame->stats.hufTable);
-            CHECKERR(compressedSize);
-            /* this only occurs when it could not compress or similar */
-        } while (compressedSize <= 0);
-
-        op += compressedSize;
-
-        compressedSize += hufHeaderSize;
-        DISPLAYLEVEL(5, "    regenerated size: %u\n", (unsigned)litSize);
-        DISPLAYLEVEL(5, "    compressed size: %u\n", (unsigned)compressedSize);
-        if (compressedSize >= litSize) {
-            DISPLAYLEVEL(5, "     trying again\n");
-            /* if we have to try again, reset the stats so we don't accidentally
-             * try to repeat a distribution we just made */
-            frame->stats = frame->oldStats;
-        } else {
-            break;
-        }
-    } while (1);
-
-    /* write header */
-    switch (sizeFormat) {
-    case 0: /* fall through, size is the same as case 1 */
-    case 1: {
-        U32 const header = hType | (sizeFormat << 2) | ((U32)litSize << 4) |
-                           ((U32)compressedSize << 14);
-        MEM_writeLE24(ostart, header);
-        break;
-    }
-    case 2: {
-        U32 const header = hType | (sizeFormat << 2) | ((U32)litSize << 4) |
-                           ((U32)compressedSize << 18);
-        MEM_writeLE32(ostart, header);
-        break;
-    }
-    case 3: {
-        U32 const header = hType | (sizeFormat << 2) | ((U32)litSize << 4) |
-                           ((U32)compressedSize << 22);
-        MEM_writeLE32(ostart, header);
-        ostart[4] = (BYTE)(compressedSize >> 10);
-        break;
-    }
-    default:; /* impossible */
-    }
-
-    frame->data = op;
-    return litSize;
-}
-
-static size_t writeLiteralsBlock(U32* seed, frame_t* frame, size_t contentSize)
-{
-    /* only do compressed for larger segments to avoid compressibility issues */
-    if (RAND(seed) & 7 && contentSize >= 64) {
-        return writeLiteralsBlockCompressed(seed, frame, contentSize);
-    } else {
-        return writeLiteralsBlockSimple(seed, frame, contentSize);
-    }
-}
-
-static inline void initSeqStore(seqStore_t *seqStore) {
-    seqStore->maxNbSeq = MAX_NB_SEQ;
-    seqStore->maxNbLit = ZSTD_BLOCKSIZE_MAX;
-    seqStore->sequencesStart = SEQUENCE_BUFFER;
-    seqStore->litStart = SEQUENCE_LITERAL_BUFFER;
-    seqStore->llCode = SEQUENCE_LLCODE;
-    seqStore->mlCode = SEQUENCE_MLCODE;
-    seqStore->ofCode = SEQUENCE_OFCODE;
-
-    ZSTD_resetSeqStore(seqStore);
-}
-
-/* Randomly generate sequence commands */
-static U32 generateSequences(U32* seed, frame_t* frame, seqStore_t* seqStore,
-                                size_t contentSize, size_t literalsSize, dictInfo info)
-{
-    /* The total length of all the matches */
-    size_t const remainingMatch = contentSize - literalsSize;
-    size_t excessMatch = 0;
-    U32 numSequences = 0;
-
-    U32 i;
-
-
-    const BYTE* literals = LITERAL_BUFFER;
-    BYTE* srcPtr = frame->src;
-
-    if (literalsSize != contentSize) {
-        /* each match must be at least MIN_SEQ_LEN, so this is the maximum
-         * number of sequences we can have */
-        U32 const maxSequences = (U32)remainingMatch / MIN_SEQ_LEN;
-        numSequences = (RAND(seed) % maxSequences) + 1;
-
-        /* the extra match lengths we have to allocate to each sequence */
-        excessMatch = remainingMatch - numSequences * MIN_SEQ_LEN;
-    }
-
-    DISPLAYLEVEL(5, "    total match lengths: %u\n", (unsigned)remainingMatch);
-    for (i = 0; i < numSequences; i++) {
-        /* Generate match and literal lengths by exponential distribution to
-         * ensure nice numbers */
-        U32 matchLen =
-                MIN_SEQ_LEN +
-                ROUND(RAND_exp(seed, excessMatch / (double)(numSequences - i)));
-        U32 literalLen =
-                (RAND(seed) & 7)
-                        ? ROUND(RAND_exp(seed,
-                                         literalsSize /
-                                                 (double)(numSequences - i)))
-                        : 0;
-        /* actual offset, code to send, and point to copy up to when shifting
-         * codes in the repeat offsets history */
-        U32 offset, offsetCode, repIndex;
-
-        /* bounds checks */
-        matchLen = (U32) MIN(matchLen, excessMatch + MIN_SEQ_LEN);
-        literalLen = MIN(literalLen, (U32) literalsSize);
-        if (i == 0 && srcPtr == frame->srcStart && literalLen == 0) literalLen = 1;
-        if (i + 1 == numSequences) matchLen = MIN_SEQ_LEN + (U32) excessMatch;
-
-        memcpy(srcPtr, literals, literalLen);
-        srcPtr += literalLen;
-        do {
-            if (RAND(seed) & 7) {
-                /* do a normal offset */
-                U32 const dataDecompressed = (U32)((BYTE*)srcPtr-(BYTE*)frame->srcStart);
-                offset = (RAND(seed) %
-                          MIN(frame->header.windowSize,
-                              (size_t)((BYTE*)srcPtr - (BYTE*)frame->srcStart))) +
-                         1;
-                if (info.useDict && (RAND(seed) & 1) && i + 1 != numSequences && dataDecompressed < frame->header.windowSize) {
-                    /* need to occasionally generate offsets that go past the start */
-                    /* including i+1 != numSequences because the last sequences has to adhere to predetermined contentSize */
-                    U32 lenPastStart = (RAND(seed) % info.dictContentSize) + 1;
-                    offset = (U32)((BYTE*)srcPtr - (BYTE*)frame->srcStart)+lenPastStart;
-                    if (offset > frame->header.windowSize) {
-                        if (lenPastStart < MIN_SEQ_LEN) {
-                            /* when offset > windowSize, matchLen bound by end of dictionary (lenPastStart) */
-                            /* this also means that lenPastStart must be greater than MIN_SEQ_LEN */
-                            /* make sure lenPastStart does not go past dictionary start though */
-                            lenPastStart = MIN(lenPastStart+MIN_SEQ_LEN, (U32)info.dictContentSize);
-                            offset = (U32)((BYTE*)srcPtr - (BYTE*)frame->srcStart) + lenPastStart;
-                        }
-                        {
-                            U32 const matchLenBound = MIN(frame->header.windowSize, lenPastStart);
-                            matchLen = MIN(matchLen, matchLenBound);
-                        }
-                    }
-                }
-                offsetCode = offset + ZSTD_REP_MOVE;
-                repIndex = 2;
-            } else {
-                /* do a repeat offset */
-                offsetCode = RAND(seed) % 3;
-                if (literalLen > 0) {
-                    offset = frame->stats.rep[offsetCode];
-                    repIndex = offsetCode;
-                } else {
-                    /* special case */
-                    offset = offsetCode == 2 ? frame->stats.rep[0] - 1
-                                           : frame->stats.rep[offsetCode + 1];
-                    repIndex = MIN(2, offsetCode + 1);
-                }
-            }
-        } while (((!info.useDict) && (offset > (size_t)((BYTE*)srcPtr - (BYTE*)frame->srcStart))) || offset == 0);
-
-        {
-            size_t j;
-            BYTE* const dictEnd = info.dictContent + info.dictContentSize;
-            for (j = 0; j < matchLen; j++) {
-                if ((U32)((BYTE*)srcPtr - (BYTE*)frame->srcStart) < offset) {
-                    /* copy from dictionary instead of literals */
-                    size_t const dictOffset = offset - (srcPtr - (BYTE*)frame->srcStart);
-                    *srcPtr = *(dictEnd - dictOffset);
-                }
-                else {
-                    *srcPtr = *(srcPtr-offset);
-                }
-                srcPtr++;
-            }
-        }
-
-        {   int r;
-            for (r = repIndex; r > 0; r--) {
-                frame->stats.rep[r] = frame->stats.rep[r - 1];
-            }
-            frame->stats.rep[0] = offset;
-        }
-
-        DISPLAYLEVEL(6, "      LL: %5u OF: %5u ML: %5u",
-                    (unsigned)literalLen, (unsigned)offset, (unsigned)matchLen);
-        DISPLAYLEVEL(7, " srcPos: %8u seqNb: %3u",
-                     (unsigned)((BYTE*)srcPtr - (BYTE*)frame->srcStart), (unsigned)i);
-        DISPLAYLEVEL(6, "\n");
-        if (offsetCode < 3) {
-            DISPLAYLEVEL(7, "        repeat offset: %d\n", (int)repIndex);
-        }
-        /* use libzstd sequence handling */
-        ZSTD_storeSeq(seqStore, literalLen, literals, literals + literalLen,
-                      offsetCode, matchLen - MINMATCH);
-
-        literalsSize -= literalLen;
-        excessMatch -= (matchLen - MIN_SEQ_LEN);
-        literals += literalLen;
-    }
-
-    memcpy(srcPtr, literals, literalsSize);
-    srcPtr += literalsSize;
-    DISPLAYLEVEL(6, "      excess literals: %5u", (unsigned)literalsSize);
-    DISPLAYLEVEL(7, " srcPos: %8u", (unsigned)((BYTE*)srcPtr - (BYTE*)frame->srcStart));
-    DISPLAYLEVEL(6, "\n");
-
-    return numSequences;
-}
-
-static void initSymbolSet(const BYTE* symbols, size_t len, BYTE* set, BYTE maxSymbolValue)
-{
-    size_t i;
-
-    memset(set, 0, (size_t)maxSymbolValue+1);
-
-    for (i = 0; i < len; i++) {
-        set[symbols[i]] = 1;
-    }
-}
-
-static int isSymbolSubset(const BYTE* symbols, size_t len, const BYTE* set, BYTE maxSymbolValue)
-{
-    size_t i;
-
-    for (i = 0; i < len; i++) {
-        if (symbols[i] > maxSymbolValue || !set[symbols[i]]) {
-            return 0;
-        }
-    }
-    return 1;
-}
-
-static size_t writeSequences(U32* seed, frame_t* frame, seqStore_t* seqStorePtr,
-                             size_t nbSeq)
-{
-    /* This code is mostly copied from ZSTD_compressSequences in zstd_compress.c */
-    unsigned count[MaxSeq+1];
-    S16 norm[MaxSeq+1];
-    FSE_CTable* CTable_LitLength = frame->stats.litlengthCTable;
-    FSE_CTable* CTable_OffsetBits = frame->stats.offcodeCTable;
-    FSE_CTable* CTable_MatchLength = frame->stats.matchlengthCTable;
-    U32 LLtype, Offtype, MLtype;   /* compressed, raw or rle */
-    const seqDef* const sequences = seqStorePtr->sequencesStart;
-    const BYTE* const ofCodeTable = seqStorePtr->ofCode;
-    const BYTE* const llCodeTable = seqStorePtr->llCode;
-    const BYTE* const mlCodeTable = seqStorePtr->mlCode;
-    BYTE* const oend = (BYTE*)frame->dataEnd;
-    BYTE* op = (BYTE*)frame->data;
-    BYTE* seqHead;
-    BYTE scratchBuffer[1<<MAX(MLFSELog,LLFSELog)];
-
-    /* literals compressing block removed so that can be done separately */
-
-    /* Sequences Header */
-    if ((oend-op) < 3 /*max nbSeq Size*/ + 1 /*seqHead */) return ERROR(dstSize_tooSmall);
-    if (nbSeq < 0x7F) *op++ = (BYTE)nbSeq;
-    else if (nbSeq < LONGNBSEQ) op[0] = (BYTE)((nbSeq>>8) + 0x80), op[1] = (BYTE)nbSeq, op+=2;
-    else op[0]=0xFF, MEM_writeLE16(op+1, (U16)(nbSeq - LONGNBSEQ)), op+=3;
-
-    if (nbSeq==0) {
-        frame->data = op;
-        return 0;
-    }
-
-    /* seqHead : flags for FSE encoding type */
-    seqHead = op++;
-
-    /* convert length/distances into codes */
-    ZSTD_seqToCodes(seqStorePtr);
-
-    /* CTable for Literal Lengths */
-    {   unsigned max = MaxLL;
-        size_t const mostFrequent = HIST_countFast_wksp(count, &max, llCodeTable, nbSeq, WKSP, sizeof(WKSP));   /* cannot fail */
-        assert(!HIST_isError(mostFrequent));
-        if (frame->stats.fseInit && !(RAND(seed) & 3) &&
-                   isSymbolSubset(llCodeTable, nbSeq,
-                                  frame->stats.litlengthSymbolSet, 35)) {
-            /* maybe do repeat mode if we're allowed to */
-            LLtype = set_repeat;
-        } else if (mostFrequent == nbSeq) {
-            /* do RLE if we have the chance */
-            *op++ = llCodeTable[0];
-            FSE_buildCTable_rle(CTable_LitLength, (BYTE)max);
-            LLtype = set_rle;
-        } else if (!(RAND(seed) & 3)) {
-            /* maybe use the default distribution */
-            FSE_buildCTable_wksp(CTable_LitLength, LL_defaultNorm, MaxLL, LL_defaultNormLog, scratchBuffer, sizeof(scratchBuffer));
-            LLtype = set_basic;
-        } else {
-            /* fall back on a full table */
-            size_t nbSeq_1 = nbSeq;
-            const U32 tableLog = FSE_optimalTableLog(LLFSELog, nbSeq, max);
-            if (count[llCodeTable[nbSeq-1]]>1) { count[llCodeTable[nbSeq-1]]--; nbSeq_1--; }
-            FSE_normalizeCount(norm, tableLog, count, nbSeq_1, max);
-            { size_t const NCountSize = FSE_writeNCount(op, oend-op, norm, max, tableLog);   /* overflow protected */
-              if (FSE_isError(NCountSize)) return ERROR(GENERIC);
-              op += NCountSize; }
-            FSE_buildCTable_wksp(CTable_LitLength, norm, max, tableLog, scratchBuffer, sizeof(scratchBuffer));
-            LLtype = set_compressed;
-    }   }
-
-    /* CTable for Offsets */
-    /* see Literal Lengths for descriptions of mode choices */
-    {   unsigned max = MaxOff;
-        size_t const mostFrequent = HIST_countFast_wksp(count, &max, ofCodeTable, nbSeq, WKSP, sizeof(WKSP));   /* cannot fail */
-        assert(!HIST_isError(mostFrequent));
-        if (frame->stats.fseInit && !(RAND(seed) & 3) &&
-                   isSymbolSubset(ofCodeTable, nbSeq,
-                                  frame->stats.offsetSymbolSet, 28)) {
-            Offtype = set_repeat;
-        } else if (mostFrequent == nbSeq) {
-            *op++ = ofCodeTable[0];
-            FSE_buildCTable_rle(CTable_OffsetBits, (BYTE)max);
-            Offtype = set_rle;
-        } else if (!(RAND(seed) & 3)) {
-            FSE_buildCTable_wksp(CTable_OffsetBits, OF_defaultNorm, DefaultMaxOff, OF_defaultNormLog, scratchBuffer, sizeof(scratchBuffer));
-            Offtype = set_basic;
-        } else {
-            size_t nbSeq_1 = nbSeq;
-            const U32 tableLog = FSE_optimalTableLog(OffFSELog, nbSeq, max);
-            if (count[ofCodeTable[nbSeq-1]]>1) { count[ofCodeTable[nbSeq-1]]--; nbSeq_1--; }
-            FSE_normalizeCount(norm, tableLog, count, nbSeq_1, max);
-            { size_t const NCountSize = FSE_writeNCount(op, oend-op, norm, max, tableLog);   /* overflow protected */
-              if (FSE_isError(NCountSize)) return ERROR(GENERIC);
-              op += NCountSize; }
-            FSE_buildCTable_wksp(CTable_OffsetBits, norm, max, tableLog, scratchBuffer, sizeof(scratchBuffer));
-            Offtype = set_compressed;
-    }   }
-
-    /* CTable for MatchLengths */
-    /* see Literal Lengths for descriptions of mode choices */
-    {   unsigned max = MaxML;
-        size_t const mostFrequent = HIST_countFast_wksp(count, &max, mlCodeTable, nbSeq, WKSP, sizeof(WKSP));   /* cannot fail */
-        assert(!HIST_isError(mostFrequent));
-        if (frame->stats.fseInit && !(RAND(seed) & 3) &&
-                   isSymbolSubset(mlCodeTable, nbSeq,
-                                  frame->stats.matchlengthSymbolSet, 52)) {
-            MLtype = set_repeat;
-        } else if (mostFrequent == nbSeq) {
-            *op++ = *mlCodeTable;
-            FSE_buildCTable_rle(CTable_MatchLength, (BYTE)max);
-            MLtype = set_rle;
-        } else if (!(RAND(seed) & 3)) {
-            /* sometimes do default distribution */
-            FSE_buildCTable_wksp(CTable_MatchLength, ML_defaultNorm, MaxML, ML_defaultNormLog, scratchBuffer, sizeof(scratchBuffer));
-            MLtype = set_basic;
-        } else {
-            /* fall back on table */
-            size_t nbSeq_1 = nbSeq;
-            const U32 tableLog = FSE_optimalTableLog(MLFSELog, nbSeq, max);
-            if (count[mlCodeTable[nbSeq-1]]>1) { count[mlCodeTable[nbSeq-1]]--; nbSeq_1--; }
-            FSE_normalizeCount(norm, tableLog, count, nbSeq_1, max);
-            { size_t const NCountSize = FSE_writeNCount(op, oend-op, norm, max, tableLog);   /* overflow protected */
-              if (FSE_isError(NCountSize)) return ERROR(GENERIC);
-              op += NCountSize; }
-            FSE_buildCTable_wksp(CTable_MatchLength, norm, max, tableLog, scratchBuffer, sizeof(scratchBuffer));
-            MLtype = set_compressed;
-    }   }
-    frame->stats.fseInit = 1;
-    initSymbolSet(llCodeTable, nbSeq, frame->stats.litlengthSymbolSet, 35);
-    initSymbolSet(ofCodeTable, nbSeq, frame->stats.offsetSymbolSet, 28);
-    initSymbolSet(mlCodeTable, nbSeq, frame->stats.matchlengthSymbolSet, 52);
-
-    DISPLAYLEVEL(5, "    LL type: %d OF type: %d ML type: %d\n", (unsigned)LLtype, (unsigned)Offtype, (unsigned)MLtype);
-
-    *seqHead = (BYTE)((LLtype<<6) + (Offtype<<4) + (MLtype<<2));
-
-    /* Encoding Sequences */
-    {   BIT_CStream_t blockStream;
-        FSE_CState_t  stateMatchLength;
-        FSE_CState_t  stateOffsetBits;
-        FSE_CState_t  stateLitLength;
-
-        RETURN_ERROR_IF(
-            ERR_isError(BIT_initCStream(&blockStream, op, oend-op)),
-            dstSize_tooSmall, "not enough space remaining");
-
-        /* first symbols */
-        FSE_initCState2(&stateMatchLength, CTable_MatchLength, mlCodeTable[nbSeq-1]);
-        FSE_initCState2(&stateOffsetBits,  CTable_OffsetBits,  ofCodeTable[nbSeq-1]);
-        FSE_initCState2(&stateLitLength,   CTable_LitLength,   llCodeTable[nbSeq-1]);
-        BIT_addBits(&blockStream, sequences[nbSeq-1].litLength, LL_bits[llCodeTable[nbSeq-1]]);
-        if (MEM_32bits()) BIT_flushBits(&blockStream);
-        BIT_addBits(&blockStream, sequences[nbSeq-1].matchLength, ML_bits[mlCodeTable[nbSeq-1]]);
-        if (MEM_32bits()) BIT_flushBits(&blockStream);
-        BIT_addBits(&blockStream, sequences[nbSeq-1].offset, ofCodeTable[nbSeq-1]);
-        BIT_flushBits(&blockStream);
-
-        {   size_t n;
-            for (n=nbSeq-2 ; n<nbSeq ; n--) {      /* intentional underflow */
-                BYTE const llCode = llCodeTable[n];
-                BYTE const ofCode = ofCodeTable[n];
-                BYTE const mlCode = mlCodeTable[n];
-                U32  const llBits = LL_bits[llCode];
-                U32  const ofBits = ofCode;                                     /* 32b*/  /* 64b*/
-                U32  const mlBits = ML_bits[mlCode];
-                                                                                /* (7)*/  /* (7)*/
-                FSE_encodeSymbol(&blockStream, &stateOffsetBits, ofCode);       /* 15 */  /* 15 */
-                FSE_encodeSymbol(&blockStream, &stateMatchLength, mlCode);      /* 24 */  /* 24 */
-                if (MEM_32bits()) BIT_flushBits(&blockStream);                  /* (7)*/
-                FSE_encodeSymbol(&blockStream, &stateLitLength, llCode);        /* 16 */  /* 33 */
-                if (MEM_32bits() || (ofBits+mlBits+llBits >= 64-7-(LLFSELog+MLFSELog+OffFSELog)))
-                    BIT_flushBits(&blockStream);                                /* (7)*/
-                BIT_addBits(&blockStream, sequences[n].litLength, llBits);
-                if (MEM_32bits() && ((llBits+mlBits)>24)) BIT_flushBits(&blockStream);
-                BIT_addBits(&blockStream, sequences[n].matchLength, mlBits);
-                if (MEM_32bits()) BIT_flushBits(&blockStream);                  /* (7)*/
-                BIT_addBits(&blockStream, sequences[n].offset, ofBits);         /* 31 */
-                BIT_flushBits(&blockStream);                                    /* (7)*/
-        }   }
-
-        FSE_flushCState(&blockStream, &stateMatchLength);
-        FSE_flushCState(&blockStream, &stateOffsetBits);
-        FSE_flushCState(&blockStream, &stateLitLength);
-
-        {   size_t const streamSize = BIT_closeCStream(&blockStream);
-            if (streamSize==0) return ERROR(dstSize_tooSmall);   /* not enough space */
-            op += streamSize;
-    }   }
-
-    frame->data = op;
-
-    return 0;
-}
-
-static size_t writeSequencesBlock(U32* seed, frame_t* frame, size_t contentSize,
-                                  size_t literalsSize, dictInfo info)
-{
-    seqStore_t seqStore;
-    size_t numSequences;
-
-
-    initSeqStore(&seqStore);
-
-    /* randomly generate sequences */
-    numSequences = generateSequences(seed, frame, &seqStore, contentSize, literalsSize, info);
-    /* write them out to the frame data */
-    CHECKERR(writeSequences(seed, frame, &seqStore, numSequences));
-
-    return numSequences;
-}
-
-static size_t writeCompressedBlock(U32* seed, frame_t* frame, size_t contentSize, dictInfo info)
-{
-    BYTE* const blockStart = (BYTE*)frame->data;
-    size_t literalsSize;
-    size_t nbSeq;
-
-    DISPLAYLEVEL(4, "  compressed block:\n");
-
-    literalsSize = writeLiteralsBlock(seed, frame, contentSize);
-
-    DISPLAYLEVEL(4, "   literals size: %u\n", (unsigned)literalsSize);
-
-    nbSeq = writeSequencesBlock(seed, frame, contentSize, literalsSize, info);
-
-    DISPLAYLEVEL(4, "   number of sequences: %u\n", (unsigned)nbSeq);
-
-    return (BYTE*)frame->data - blockStart;
-}
-
-static void writeBlock(U32* seed, frame_t* frame, size_t contentSize,
-                       int lastBlock, dictInfo info)
-{
-    int const blockTypeDesc = RAND(seed) % 8;
-    size_t blockSize;
-    int blockType;
-
-    BYTE *const header = (BYTE*)frame->data;
-    BYTE *op = header + 3;
-
-    DISPLAYLEVEL(4, " block:\n");
-    DISPLAYLEVEL(4, "  block content size: %u\n", (unsigned)contentSize);
-    DISPLAYLEVEL(4, "  last block: %s\n", lastBlock ? "yes" : "no");
-
-    if (blockTypeDesc == 0) {
-        /* Raw data frame */
-
-        RAND_buffer(seed, frame->src, contentSize);
-        memcpy(op, frame->src, contentSize);
-
-        op += contentSize;
-        blockType = 0;
-        blockSize = contentSize;
-    } else if (blockTypeDesc == 1 && frame->header.contentSize > 0) {
-        /* RLE (Don't create RLE block if frame content is 0 since block size of 1 may exceed max block size)*/
-        BYTE const symbol = RAND(seed) & 0xff;
-
-        op[0] = symbol;
-        memset(frame->src, symbol, contentSize);
-
-        op++;
-        blockType = 1;
-        blockSize = contentSize;
-    } else {
-        /* compressed, most common */
-        size_t compressedSize;
-        blockType = 2;
-
-        frame->oldStats = frame->stats;
-
-        frame->data = op;
-        compressedSize = writeCompressedBlock(seed, frame, contentSize, info);
-        if (compressedSize >= contentSize) {   /* compressed block must be strictly smaller than uncompressed one */
-            blockType = 0;
-            memcpy(op, frame->src, contentSize);
-
-            op += contentSize;
-            blockSize = contentSize; /* fall back on raw block if data doesn't
-                                        compress */
-
-            frame->stats = frame->oldStats; /* don't update the stats */
-        } else {
-            op += compressedSize;
-            blockSize = compressedSize;
-        }
-    }
-    frame->src = (BYTE*)frame->src + contentSize;
-
-    DISPLAYLEVEL(4, "  block type: %s\n", BLOCK_TYPES[blockType]);
-    DISPLAYLEVEL(4, "  block size field: %u\n", (unsigned)blockSize);
-
-    header[0] = (BYTE) ((lastBlock | (blockType << 1) | (blockSize << 3)) & 0xff);
-    MEM_writeLE16(header + 1, (U16) (blockSize >> 5));
-
-    frame->data = op;
-}
-
-static void writeBlocks(U32* seed, frame_t* frame, dictInfo info)
-{
-    size_t contentLeft = frame->header.contentSize;
-    size_t const maxBlockSize = MIN(g_maxBlockSize, frame->header.windowSize);
-    while (1) {
-        /* 1 in 4 chance of ending frame */
-        int const lastBlock = contentLeft > maxBlockSize ? 0 : !(RAND(seed) & 3);
-        size_t blockContentSize;
-        if (lastBlock) {
-            blockContentSize = contentLeft;
-        } else {
-            if (contentLeft > 0 && (RAND(seed) & 7)) {
-                /* some variable size block */
-                blockContentSize = RAND(seed) % (MIN(maxBlockSize, contentLeft)+1);
-            } else if (contentLeft > maxBlockSize && (RAND(seed) & 1)) {
-                /* some full size block */
-                blockContentSize = maxBlockSize;
-            } else {
-                /* some empty block */
-                blockContentSize = 0;
-            }
-        }
-
-        writeBlock(seed, frame, blockContentSize, lastBlock, info);
-
-        contentLeft -= blockContentSize;
-        if (lastBlock) break;
-    }
-}
-
-static void writeChecksum(frame_t* frame)
-{
-    /* write checksum so implementations can verify their output */
-    U64 digest = XXH64(frame->srcStart, (BYTE*)frame->src-(BYTE*)frame->srcStart, 0);
-    DISPLAYLEVEL(3, "  checksum: %08x\n", (unsigned)digest);
-    MEM_writeLE32(frame->data, (U32)digest);
-    frame->data = (BYTE*)frame->data + 4;
-}
-
-static void outputBuffer(const void* buf, size_t size, const char* const path)
-{
-    /* write data out to file */
-    const BYTE* ip = (const BYTE*)buf;
-    FILE* out;
-    if (path) {
-        out = fopen(path, "wb");
-    } else {
-        out = stdout;
-    }
-    if (!out) {
-        fprintf(stderr, "Failed to open file at %s: ", path);
-        perror(NULL);
-        exit(1);
-    }
-
-    {   size_t fsize = size;
-        size_t written = 0;
-        while (written < fsize) {
-            written += fwrite(ip + written, 1, fsize - written, out);
-            if (ferror(out)) {
-                fprintf(stderr, "Failed to write to file at %s: ", path);
-                perror(NULL);
-                exit(1);
-            }
-        }
-    }
-
-    if (path) {
-        fclose(out);
-    }
-}
-
-static void initFrame(frame_t* fr)
-{
-    memset(fr, 0, sizeof(*fr));
-    fr->data = fr->dataStart = FRAME_BUFFER;
-    fr->dataEnd = FRAME_BUFFER + sizeof(FRAME_BUFFER);
-    fr->src = fr->srcStart = CONTENT_BUFFER;
-    fr->srcEnd = CONTENT_BUFFER + sizeof(CONTENT_BUFFER);
-
-    /* init repeat codes */
-    fr->stats.rep[0] = 1;
-    fr->stats.rep[1] = 4;
-    fr->stats.rep[2] = 8;
-}
-
-/**
- * Generated a single zstd compressed block with no block/frame header.
- * Returns the final seed.
- */
-static U32 generateCompressedBlock(U32 seed, frame_t* frame, dictInfo info)
-{
-    size_t blockContentSize;
-    int blockWritten = 0;
-    BYTE* op;
-    DISPLAYLEVEL(4, "block seed: %u\n", (unsigned)seed);
-    initFrame(frame);
-    op = (BYTE*)frame->data;
-
-    while (!blockWritten) {
-        size_t cSize;
-        /* generate window size */
-        {   int const exponent = RAND(&seed) % (MAX_WINDOW_LOG - 10);
-            int const mantissa = RAND(&seed) % 8;
-            frame->header.windowSize = (1U << (exponent + 10));
-            frame->header.windowSize += (frame->header.windowSize / 8) * mantissa;
-        }
-
-        /* generate content size */
-        {   size_t const maxBlockSize = MIN(g_maxBlockSize, frame->header.windowSize);
-            if (RAND(&seed) & 15) {
-                /* some full size blocks */
-                blockContentSize = maxBlockSize;
-            } else if (RAND(&seed) & 7 && g_maxBlockSize >= (1U << 7)) {
-                /* some small blocks <= 128 bytes*/
-                blockContentSize = RAND(&seed) % (1U << 7);
-            } else {
-                /* some variable size blocks */
-                blockContentSize = RAND(&seed) % maxBlockSize;
-            }
-        }
-
-        /* try generating a compressed block */
-        frame->oldStats = frame->stats;
-        frame->data = op;
-        cSize = writeCompressedBlock(&seed, frame, blockContentSize, info);
-        if (cSize >= blockContentSize) {  /* compressed size must be strictly smaller than decompressed size : https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#blocks */
-            /* data doesn't compress -- try again */
-            frame->stats = frame->oldStats; /* don't update the stats */
-            DISPLAYLEVEL(5, "   can't compress block : try again \n");
-        } else {
-            blockWritten = 1;
-            DISPLAYLEVEL(4, "   block size: %u \n", (unsigned)cSize);
-            frame->src = (BYTE*)frame->src + blockContentSize;
-        }
-    }
-    return seed;
-}
-
-/* Return the final seed */
-static U32 generateFrame(U32 seed, frame_t* fr, dictInfo info)
-{
-    /* generate a complete frame */
-    DISPLAYLEVEL(3, "frame seed: %u\n", (unsigned)seed);
-    initFrame(fr);
-
-    writeFrameHeader(&seed, fr, info);
-    writeBlocks(&seed, fr, info);
-    writeChecksum(fr);
-
-    return seed;
-}
-
-/*_*******************************************************
-*  Dictionary Helper Functions
-*********************************************************/
-/* returns 0 if successful, otherwise returns 1 upon error */
-static int genRandomDict(U32 dictID, U32 seed, size_t dictSize, BYTE* fullDict)
-{
-    /* allocate space for samples */
-    int ret = 0;
-    unsigned const numSamples = 4;
-    size_t sampleSizes[4];
-    BYTE* const samples = malloc(5000*sizeof(BYTE));
-    if (samples == NULL) {
-        DISPLAY("Error: could not allocate space for samples\n");
-        return 1;
-    }
-
-    /* generate samples */
-    {   unsigned literalValue = 1;
-        unsigned samplesPos = 0;
-        size_t currSize = 1;
-        while (literalValue <= 4) {
-            sampleSizes[literalValue - 1] = currSize;
-            {   size_t k;
-                for (k = 0; k < currSize; k++) {
-                    *(samples + (samplesPos++)) = (BYTE)literalValue;
-            }   }
-            literalValue++;
-            currSize *= 16;
-    }   }
-
-    {   size_t dictWriteSize = 0;
-        ZDICT_params_t zdictParams;
-        size_t const headerSize = MAX(dictSize/4, 256);
-        size_t const dictContentSize = dictSize - headerSize;
-        BYTE* const dictContent = fullDict + headerSize;
-        if (dictContentSize < ZDICT_CONTENTSIZE_MIN || dictSize < ZDICT_DICTSIZE_MIN) {
-            DISPLAY("Error: dictionary size is too small\n");
-            ret = 1;
-            goto exitGenRandomDict;
-        }
-
-        /* init dictionary params */
-        memset(&zdictParams, 0, sizeof(zdictParams));
-        zdictParams.dictID = dictID;
-        zdictParams.notificationLevel = 1;
-
-        /* fill in dictionary content */
-        RAND_buffer(&seed, (void*)dictContent, dictContentSize);
-
-        /* finalize dictionary with random samples */
-        dictWriteSize = ZDICT_finalizeDictionary(fullDict, dictSize,
-                                    dictContent, dictContentSize,
-                                    samples, sampleSizes, numSamples,
-                                    zdictParams);
-
-        if (ZDICT_isError(dictWriteSize)) {
-            DISPLAY("Could not finalize dictionary: %s\n", ZDICT_getErrorName(dictWriteSize));
-            ret = 1;
-        }
-    }
-
-exitGenRandomDict:
-    free(samples);
-    return ret;
-}
-
-static dictInfo initDictInfo(int useDict, size_t dictContentSize, BYTE* dictContent, U32 dictID){
-    /* allocate space statically */
-    dictInfo dictOp;
-    memset(&dictOp, 0, sizeof(dictOp));
-    dictOp.useDict = useDict;
-    dictOp.dictContentSize = dictContentSize;
-    dictOp.dictContent = dictContent;
-    dictOp.dictID = dictID;
-    return dictOp;
-}
-
-/*-*******************************************************
-*  Test Mode
-*********************************************************/
-
-BYTE DECOMPRESSED_BUFFER[MAX_DECOMPRESSED_SIZE];
-
-static size_t testDecodeSimple(frame_t* fr)
-{
-    /* test decoding the generated data with the simple API */
-    size_t const ret = ZSTD_decompress(DECOMPRESSED_BUFFER, MAX_DECOMPRESSED_SIZE,
-                           fr->dataStart, (BYTE*)fr->data - (BYTE*)fr->dataStart);
-
-    if (ZSTD_isError(ret)) return ret;
-
-    if (memcmp(DECOMPRESSED_BUFFER, fr->srcStart,
-               (BYTE*)fr->src - (BYTE*)fr->srcStart) != 0) {
-        return ERROR(corruption_detected);
-    }
-
-    return ret;
-}
-
-static size_t testDecodeStreaming(frame_t* fr)
-{
-    /* test decoding the generated data with the streaming API */
-    ZSTD_DStream* zd = ZSTD_createDStream();
-    ZSTD_inBuffer in;
-    ZSTD_outBuffer out;
-    size_t ret;
-
-    if (!zd) return ERROR(memory_allocation);
-
-    in.src = fr->dataStart;
-    in.pos = 0;
-    in.size = (BYTE*)fr->data - (BYTE*)fr->dataStart;
-
-    out.dst = DECOMPRESSED_BUFFER;
-    out.pos = 0;
-    out.size = ZSTD_DStreamOutSize();
-
-    ZSTD_initDStream(zd);
-    while (1) {
-        ret = ZSTD_decompressStream(zd, &out, &in);
-        if (ZSTD_isError(ret)) goto cleanup; /* error */
-        if (ret == 0) break; /* frame is done */
-
-        /* force decoding to be done in chunks */
-        out.size += MIN(ZSTD_DStreamOutSize(), MAX_DECOMPRESSED_SIZE - out.size);
-    }
-
-    ret = out.pos;
-
-    if (memcmp(out.dst, fr->srcStart, out.pos) != 0) {
-        return ERROR(corruption_detected);
-    }
-
-cleanup:
-    ZSTD_freeDStream(zd);
-    return ret;
-}
-
-static size_t testDecodeWithDict(U32 seed, genType_e genType)
-{
-    /* create variables */
-    size_t const dictSize = RAND(&seed) % (10 << 20) + ZDICT_DICTSIZE_MIN + ZDICT_CONTENTSIZE_MIN;
-    U32 const dictID = RAND(&seed);
-    size_t errorDetected = 0;
-    BYTE* const fullDict = malloc(dictSize);
-    if (fullDict == NULL) {
-        return ERROR(GENERIC);
-    }
-
-    /* generate random dictionary */
-    if (genRandomDict(dictID, seed, dictSize, fullDict)) {  /* return 0 on success */
-        errorDetected = ERROR(GENERIC);
-        goto dictTestCleanup;
-    }
-
-
-    {   frame_t fr;
-        dictInfo info;
-        ZSTD_DCtx* const dctx = ZSTD_createDCtx();
-        size_t ret;
-
-        /* get dict info */
-        {   size_t const headerSize = MAX(dictSize/4, 256);
-            size_t const dictContentSize = dictSize-headerSize;
-            BYTE* const dictContent = fullDict+headerSize;
-            info = initDictInfo(1, dictContentSize, dictContent, dictID);
-        }
-
-        /* manually decompress and check difference */
-        if (genType == gt_frame) {
-            /* Test frame */
-            generateFrame(seed, &fr, info);
-            ret = ZSTD_decompress_usingDict(dctx, DECOMPRESSED_BUFFER, MAX_DECOMPRESSED_SIZE,
-                                            fr.dataStart, (BYTE*)fr.data - (BYTE*)fr.dataStart,
-                                            fullDict, dictSize);
-        } else {
-            /* Test block */
-            generateCompressedBlock(seed, &fr, info);
-            ret = ZSTD_decompressBegin_usingDict(dctx, fullDict, dictSize);
-            if (ZSTD_isError(ret)) {
-                errorDetected = ret;
-                ZSTD_freeDCtx(dctx);
-                goto dictTestCleanup;
-            }
-            ret = ZSTD_decompressBlock(dctx, DECOMPRESSED_BUFFER, MAX_DECOMPRESSED_SIZE,
-                                       fr.dataStart, (BYTE*)fr.data - (BYTE*)fr.dataStart);
-        }
-        ZSTD_freeDCtx(dctx);
-
-        if (ZSTD_isError(ret)) {
-            errorDetected = ret;
-            goto dictTestCleanup;
-        }
-
-        if (memcmp(DECOMPRESSED_BUFFER, fr.srcStart, (BYTE*)fr.src - (BYTE*)fr.srcStart) != 0) {
-            errorDetected = ERROR(corruption_detected);
-            goto dictTestCleanup;
-        }
-    }
-
-dictTestCleanup:
-    free(fullDict);
-    return errorDetected;
-}
-
-static size_t testDecodeRawBlock(frame_t* fr)
-{
-    ZSTD_DCtx* dctx = ZSTD_createDCtx();
-    size_t ret = ZSTD_decompressBegin(dctx);
-    if (ZSTD_isError(ret)) return ret;
-
-    ret = ZSTD_decompressBlock(
-            dctx,
-            DECOMPRESSED_BUFFER, MAX_DECOMPRESSED_SIZE,
-            fr->dataStart, (BYTE*)fr->data - (BYTE*)fr->dataStart);
-    ZSTD_freeDCtx(dctx);
-    if (ZSTD_isError(ret)) return ret;
-
-    if (memcmp(DECOMPRESSED_BUFFER, fr->srcStart,
-               (BYTE*)fr->src - (BYTE*)fr->srcStart) != 0) {
-        return ERROR(corruption_detected);
-    }
-
-    return ret;
-}
-
-static int runBlockTest(U32* seed)
-{
-    frame_t fr;
-    U32 const seedCopy = *seed;
-    {   dictInfo const info = initDictInfo(0, 0, NULL, 0);
-        *seed = generateCompressedBlock(*seed, &fr, info);
-    }
-
-    {   size_t const r = testDecodeRawBlock(&fr);
-        if (ZSTD_isError(r)) {
-            DISPLAY("Error in block mode on test seed %u: %s\n",
-                    (unsigned)seedCopy, ZSTD_getErrorName(r));
-            return 1;
-        }
-    }
-
-    {   size_t const r = testDecodeWithDict(*seed, gt_block);
-        if (ZSTD_isError(r)) {
-            DISPLAY("Error in block mode with dictionary on test seed %u: %s\n",
-                    (unsigned)seedCopy, ZSTD_getErrorName(r));
-            return 1;
-        }
-    }
-    return 0;
-}
-
-static int runFrameTest(U32* seed)
-{
-    frame_t fr;
-    U32 const seedCopy = *seed;
-    {   dictInfo const info = initDictInfo(0, 0, NULL, 0);
-        *seed = generateFrame(*seed, &fr, info);
-    }
-
-    {   size_t const r = testDecodeSimple(&fr);
-        if (ZSTD_isError(r)) {
-            DISPLAY("Error in simple mode on test seed %u: %s\n",
-                    (unsigned)seedCopy, ZSTD_getErrorName(r));
-            return 1;
-        }
-    }
-    {   size_t const r = testDecodeStreaming(&fr);
-        if (ZSTD_isError(r)) {
-            DISPLAY("Error in streaming mode on test seed %u: %s\n",
-                    (unsigned)seedCopy, ZSTD_getErrorName(r));
-            return 1;
-        }
-    }
-    {   size_t const r = testDecodeWithDict(*seed, gt_frame);  /* avoid big dictionaries */
-        if (ZSTD_isError(r)) {
-            DISPLAY("Error in dictionary mode on test seed %u: %s\n",
-                    (unsigned)seedCopy, ZSTD_getErrorName(r));
-            return 1;
-        }
-    }
-    return 0;
-}
-
-static int runTestMode(U32 seed, unsigned numFiles, unsigned const testDurationS,
-                       genType_e genType)
-{
-    unsigned fnum;
-
-    UTIL_time_t const startClock = UTIL_getTime();
-    U64 const maxClockSpan = testDurationS * SEC_TO_MICRO;
-
-    if (numFiles == 0 && !testDurationS) numFiles = 1;
-
-    DISPLAY("seed: %u\n", (unsigned)seed);
-
-    for (fnum = 0; fnum < numFiles || UTIL_clockSpanMicro(startClock) < maxClockSpan; fnum++) {
-        if (fnum < numFiles)
-            DISPLAYUPDATE("\r%u/%u        ", fnum, numFiles);
-        else
-            DISPLAYUPDATE("\r%u           ", fnum);
-
-        {   int const ret = (genType == gt_frame) ?
-                            runFrameTest(&seed) :
-                            runBlockTest(&seed);
-            if (ret) return ret;
-        }
-    }
-
-    DISPLAY("\r%u tests completed: ", fnum);
-    DISPLAY("OK\n");
-
-    return 0;
-}
-
-/*-*******************************************************
-*  File I/O
-*********************************************************/
-
-static int generateFile(U32 seed, const char* const path,
-                        const char* const origPath, genType_e genType)
-{
-    frame_t fr;
-
-    DISPLAY("seed: %u\n", (unsigned)seed);
-
-    {   dictInfo const info = initDictInfo(0, 0, NULL, 0);
-        if (genType == gt_frame) {
-            generateFrame(seed, &fr, info);
-        } else {
-            generateCompressedBlock(seed, &fr, info);
-        }
-    }
-    outputBuffer(fr.dataStart, (BYTE*)fr.data - (BYTE*)fr.dataStart, path);
-    if (origPath) {
-        outputBuffer(fr.srcStart, (BYTE*)fr.src - (BYTE*)fr.srcStart, origPath);
-    }
-    return 0;
-}
-
-static int generateCorpus(U32 seed, unsigned numFiles, const char* const path,
-                          const char* const origPath, genType_e genType)
-{
-    char outPath[MAX_PATH];
-    unsigned fnum;
-
-    DISPLAY("seed: %u\n", (unsigned)seed);
-
-    for (fnum = 0; fnum < numFiles; fnum++) {
-        frame_t fr;
-
-        DISPLAYUPDATE("\r%u/%u        ", fnum, numFiles);
-
-        {   dictInfo const info = initDictInfo(0, 0, NULL, 0);
-            if (genType == gt_frame) {
-                seed = generateFrame(seed, &fr, info);
-            } else {
-                seed = generateCompressedBlock(seed, &fr, info);
-            }
-        }
-
-        if (snprintf(outPath, MAX_PATH, "%s/z%06u.zst", path, fnum) + 1 > MAX_PATH) {
-            DISPLAY("Error: path too long\n");
-            return 1;
-        }
-        outputBuffer(fr.dataStart, (BYTE*)fr.data - (BYTE*)fr.dataStart, outPath);
-
-        if (origPath) {
-            if (snprintf(outPath, MAX_PATH, "%s/z%06u", origPath, fnum) + 1 > MAX_PATH) {
-                DISPLAY("Error: path too long\n");
-                return 1;
-            }
-            outputBuffer(fr.srcStart, (BYTE*)fr.src - (BYTE*)fr.srcStart, outPath);
-        }
-    }
-
-    DISPLAY("\r%u/%u      \n", fnum, numFiles);
-
-    return 0;
-}
-
-static int generateCorpusWithDict(U32 seed, unsigned numFiles, const char* const path,
-                                  const char* const origPath, const size_t dictSize,
-                                  genType_e genType)
-{
-    char outPath[MAX_PATH];
-    BYTE* fullDict;
-    U32 const dictID = RAND(&seed);
-    int errorDetected = 0;
-
-    if (snprintf(outPath, MAX_PATH, "%s/dictionary", path) + 1 > MAX_PATH) {
-        DISPLAY("Error: path too long\n");
-        return 1;
-    }
-
-    /* allocate space for the dictionary */
-    fullDict = malloc(dictSize);
-    if (fullDict == NULL) {
-        DISPLAY("Error: could not allocate space for full dictionary.\n");
-        return 1;
-    }
-
-    /* randomly generate the dictionary */
-    {   int const ret = genRandomDict(dictID, seed, dictSize, fullDict);
-        if (ret != 0) {
-            errorDetected = ret;
-            goto dictCleanup;
-        }
-    }
-
-    /* write out dictionary */
-    if (numFiles != 0) {
-        if (snprintf(outPath, MAX_PATH, "%s/dictionary", path) + 1 > MAX_PATH) {
-            DISPLAY("Error: dictionary path too long\n");
-            errorDetected = 1;
-            goto dictCleanup;
-        }
-        outputBuffer(fullDict, dictSize, outPath);
-    }
-    else {
-        outputBuffer(fullDict, dictSize, "dictionary");
-    }
-
-    /* generate random compressed/decompressed files */
-    {   unsigned fnum;
-        for (fnum = 0; fnum < MAX(numFiles, 1); fnum++) {
-            frame_t fr;
-            DISPLAYUPDATE("\r%u/%u        ", fnum, numFiles);
-            {
-                size_t const headerSize = MAX(dictSize/4, 256);
-                size_t const dictContentSize = dictSize-headerSize;
-                BYTE* const dictContent = fullDict+headerSize;
-                dictInfo const info = initDictInfo(1, dictContentSize, dictContent, dictID);
-                if (genType == gt_frame) {
-                    seed = generateFrame(seed, &fr, info);
-                } else {
-                    seed = generateCompressedBlock(seed, &fr, info);
-                }
-            }
-
-            if (numFiles != 0) {
-                if (snprintf(outPath, MAX_PATH, "%s/z%06u.zst", path, fnum) + 1 > MAX_PATH) {
-                    DISPLAY("Error: path too long\n");
-                    errorDetected = 1;
-                    goto dictCleanup;
-                }
-                outputBuffer(fr.dataStart, (BYTE*)fr.data - (BYTE*)fr.dataStart, outPath);
-
-                if (origPath) {
-                    if (snprintf(outPath, MAX_PATH, "%s/z%06u", origPath, fnum) + 1 > MAX_PATH) {
-                        DISPLAY("Error: path too long\n");
-                        errorDetected = 1;
-                        goto dictCleanup;
-                    }
-                    outputBuffer(fr.srcStart, (BYTE*)fr.src - (BYTE*)fr.srcStart, outPath);
-                }
-            }
-            else {
-                outputBuffer(fr.dataStart, (BYTE*)fr.data - (BYTE*)fr.dataStart, path);
-                if (origPath) {
-                    outputBuffer(fr.srcStart, (BYTE*)fr.src - (BYTE*)fr.srcStart, origPath);
-                }
-            }
-        }
-    }
-
-dictCleanup:
-    free(fullDict);
-    return errorDetected;
-}
-
-
-/*_*******************************************************
-*  Command line
-*********************************************************/
-static U32 makeSeed(void)
-{
-    U32 t = (U32) time(NULL);
-    return XXH32(&t, sizeof(t), 0) % 65536;
-}
-
-static unsigned readInt(const char** argument)
-{
-    unsigned val = 0;
-    while ((**argument>='0') && (**argument<='9')) {
-        val *= 10;
-        val += **argument - '0';
-        (*argument)++;
-    }
-    return val;
-}
-
-static void usage(const char* programName)
-{
-    DISPLAY( "Usage :\n");
-    DISPLAY( "      %s [args]\n", programName);
-    DISPLAY( "\n");
-    DISPLAY( "Arguments :\n");
-    DISPLAY( " -p<path> : select output path (default:stdout)\n");
-    DISPLAY( "                in multiple files mode this should be a directory\n");
-    DISPLAY( " -o<path> : select path to output original file (default:no output)\n");
-    DISPLAY( "                in multiple files mode this should be a directory\n");
-    DISPLAY( " -s#      : select seed (default:random based on time)\n");
-    DISPLAY( " -n#      : number of files to generate (default:1)\n");
-    DISPLAY( " -t       : activate test mode (test files against libzstd instead of outputting them)\n");
-    DISPLAY( " -T#      : length of time to run tests for\n");
-    DISPLAY( " -v       : increase verbosity level (default:0, max:7)\n");
-    DISPLAY( " -h/H     : display help/long help and exit\n");
-}
-
-static void advancedUsage(const char* programName)
-{
-    usage(programName);
-    DISPLAY( "\n");
-    DISPLAY( "Advanced arguments        :\n");
-    DISPLAY( " --content-size           : always include the content size in the frame header\n");
-    DISPLAY( " --use-dict=#             : include a dictionary used to decompress the corpus\n");
-    DISPLAY( " --gen-blocks             : generate raw compressed blocks without block/frame headers\n");
-    DISPLAY( " --max-block-size-log=#   : max block size log, must be in range [2, 17]\n");
-    DISPLAY( " --max-content-size-log=# : max content size log, must be <= 20\n");
-    DISPLAY( "                            (this is ignored with gen-blocks)\n");
-}
-
-/*! readU32FromChar() :
-    @return : unsigned integer value read from input in `char` format
-    allows and interprets K, KB, KiB, M, MB and MiB suffix.
-    Will also modify `*stringPtr`, advancing it to position where it stopped reading.
-    Note : function result can overflow if digit string > MAX_UINT */
-static unsigned readU32FromChar(const char** stringPtr)
-{
-    unsigned result = 0;
-    while ((**stringPtr >='0') && (**stringPtr <='9'))
-        result *= 10, result += **stringPtr - '0', (*stringPtr)++ ;
-    if ((**stringPtr=='K') || (**stringPtr=='M')) {
-        result <<= 10;
-        if (**stringPtr=='M') result <<= 10;
-        (*stringPtr)++ ;
-        if (**stringPtr=='i') (*stringPtr)++;
-        if (**stringPtr=='B') (*stringPtr)++;
-    }
-    return result;
-}
-
-/** longCommandWArg() :
- *  check if *stringPtr is the same as longCommand.
- *  If yes, @return 1 and advances *stringPtr to the position which immediately follows longCommand.
- *  @return 0 and doesn't modify *stringPtr otherwise.
- */
-static unsigned longCommandWArg(const char** stringPtr, const char* longCommand)
-{
-    size_t const comSize = strlen(longCommand);
-    int const result = !strncmp(*stringPtr, longCommand, comSize);
-    if (result) *stringPtr += comSize;
-    return result;
-}
-
-int main(int argc, char** argv)
-{
-    U32 seed = 0;
-    int seedset = 0;
-    unsigned numFiles = 0;
-    unsigned testDuration = 0;
-    int testMode = 0;
-    const char* path = NULL;
-    const char* origPath = NULL;
-    int useDict = 0;
-    unsigned dictSize = (10 << 10); /* 10 kB default */
-    genType_e genType = gt_frame;
-
-    int argNb;
-
-    /* Check command line */
-    for (argNb=1; argNb<argc; argNb++) {
-        const char* argument = argv[argNb];
-        if(!argument) continue;   /* Protection if argument empty */
-
-        /* Handle commands. Aggregated commands are allowed */
-        if (argument[0]=='-') {
-            argument++;
-            while (*argument!=0) {
-                switch(*argument)
-                {
-                case 'h':
-                    usage(argv[0]);
-                    return 0;
-                case 'H':
-                    advancedUsage(argv[0]);
-                    return 0;
-                case 'v':
-                    argument++;
-                    g_displayLevel++;
-                    break;
-                case 's':
-                    argument++;
-                    seedset=1;
-                    seed = readInt(&argument);
-                    break;
-                case 'n':
-                    argument++;
-                    numFiles = readInt(&argument);
-                    break;
-                case 'T':
-                    argument++;
-                    testDuration = readInt(&argument);
-                    if (*argument == 'm') {
-                        testDuration *= 60;
-                        argument++;
-                        if (*argument == 'n') argument++;
-                    }
-                    break;
-                case 'o':
-                    argument++;
-                    origPath = argument;
-                    argument += strlen(argument);
-                    break;
-                case 'p':
-                    argument++;
-                    path = argument;
-                    argument += strlen(argument);
-                    break;
-                case 't':
-                    argument++;
-                    testMode = 1;
-                    break;
-                case '-':
-                    argument++;
-                    if (strcmp(argument, "content-size") == 0) {
-                        opts.contentSize = 1;
-                    } else if (longCommandWArg(&argument, "use-dict=")) {
-                        dictSize = readU32FromChar(&argument);
-                        useDict = 1;
-                    } else if (strcmp(argument, "gen-blocks") == 0) {
-                        genType = gt_block;
-                    } else if (longCommandWArg(&argument, "max-block-size-log=")) {
-                        U32 value = readU32FromChar(&argument);
-                        if (value >= 2 && value <= ZSTD_BLOCKSIZE_MAX) {
-                            g_maxBlockSize = 1U << value;
-                        }
-                    } else if (longCommandWArg(&argument, "max-content-size-log=")) {
-                        U32 value = readU32FromChar(&argument);
-                        g_maxDecompressedSizeLog =
-                                MIN(MAX_DECOMPRESSED_SIZE_LOG, value);
-                    } else {
-                        advancedUsage(argv[0]);
-                        return 1;
-                    }
-                    argument += strlen(argument);
-                    break;
-                default:
-                    usage(argv[0]);
-                    return 1;
-    }   }   }   }   /* for (argNb=1; argNb<argc; argNb++) */
-
-    if (!seedset) {
-        seed = makeSeed();
-    }
-
-    if (testMode) {
-        return runTestMode(seed, numFiles, testDuration, genType);
-    } else {
-        if (testDuration) {
-            DISPLAY("Error: -T requires test mode (-t)\n\n");
-            usage(argv[0]);
-            return 1;
-        }
-    }
-
-    if (!path) {
-        DISPLAY("Error: path is required in file generation mode\n");
-        usage(argv[0]);
-        return 1;
-    }
-
-    if (numFiles == 0 && useDict == 0) {
-        return generateFile(seed, path, origPath, genType);
-    } else if (useDict == 0){
-        return generateCorpus(seed, numFiles, path, origPath, genType);
-    } else {
-        /* should generate files with a dictionary */
-        return generateCorpusWithDict(seed, numFiles, path, origPath, dictSize, genType);
-    }
-
-}
diff --git a/tests/fullbench.c b/tests/fullbench.c
deleted file mode 100644
index be49b1428ff5..000000000000
--- a/tests/fullbench.c
+++ /dev/null
@@ -1,843 +0,0 @@
-/*
- * Copyright (c) 2015-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-
-/*_************************************
-*  Includes
-**************************************/
-#include "util.h"        /* Compiler options, UTIL_GetFileSize */
-#include <stdlib.h>      /* malloc */
-#include <stdio.h>       /* fprintf, fopen, ftello64 */
-#include <assert.h>
-
-#include "timefn.h"      /* UTIL_clockSpanNano, UTIL_getTime */
-#include "mem.h"         /* U32 */
-#ifndef ZSTD_DLL_IMPORT
-    #include "zstd_internal.h"   /* ZSTD_decodeSeqHeaders, ZSTD_blockHeaderSize, blockType_e, KB, MB */
-#else
-    #define KB *(1 <<10)
-    #define MB *(1 <<20)
-    #define GB *(1U<<30)
-    typedef enum { bt_raw, bt_rle, bt_compressed, bt_reserved } blockType_e;
-#endif
-#define ZSTD_STATIC_LINKING_ONLY  /* ZSTD_compressBegin, ZSTD_compressContinue, etc. */
-#include "zstd.h"        /* ZSTD_versionString */
-#include "util.h"        /* time functions */
-#include "datagen.h"
-#include "benchfn.h"     /* CustomBench */
-#include "benchzstd.h"   /* MB_UNIT */
-
-
-/*_************************************
-*  Constants
-**************************************/
-#define PROGRAM_DESCRIPTION "Zstandard speed analyzer"
-#define AUTHOR "Yann Collet"
-#define WELCOME_MESSAGE "*** %s %s %i-bits, by %s (%s) ***\n", PROGRAM_DESCRIPTION, ZSTD_versionString(), (int)(sizeof(void*)*8), AUTHOR, __DATE__
-
-#define NBLOOPS    6
-#define TIMELOOP_S 2
-
-#define MAX_MEM    (1984 MB)
-
-#define DEFAULT_CLEVEL 1
-
-#define COMPRESSIBILITY_DEFAULT 0.50
-static const size_t kSampleSizeDefault = 10000000;
-
-#define TIMELOOP_NANOSEC      (1*1000000000ULL) /* 1 second */
-
-
-/*_************************************
-*  Macros
-**************************************/
-#define DISPLAY(...)  fprintf(stderr, __VA_ARGS__)
-
-#define CONTROL(c)  { if (!(c)) { abort(); } }   /* like assert(), but cannot be disabled */
-
-/*_************************************
-*  Benchmark Parameters
-**************************************/
-static unsigned g_nbIterations = NBLOOPS;
-
-
-/*_*******************************************************
-*  Private functions
-*********************************************************/
-static size_t BMK_findMaxMem(U64 requiredMem)
-{
-    size_t const step = 64 MB;
-    void* testmem = NULL;
-
-    requiredMem = (((requiredMem >> 26) + 1) << 26);
-    if (requiredMem > MAX_MEM) requiredMem = MAX_MEM;
-
-    requiredMem += step;
-    do {
-        testmem = malloc ((size_t)requiredMem);
-        requiredMem -= step;
-    } while (!testmem);
-
-    free (testmem);
-    return (size_t) requiredMem;
-}
-
-
-/*_*******************************************************
-*  Benchmark wrappers
-*********************************************************/
-
-static ZSTD_CCtx* g_zcc = NULL;
-
-static size_t
-local_ZSTD_compress(const void* src, size_t srcSize,
-                    void* dst, size_t dstSize,
-                    void* payload)
-{
-    ZSTD_parameters p;
-    ZSTD_frameParameters f = { 1 /* contentSizeHeader*/, 0, 0 };
-    p.fParams = f;
-    p.cParams = *(ZSTD_compressionParameters*)payload;
-    return ZSTD_compress_advanced (g_zcc, dst, dstSize, src, srcSize, NULL ,0, p);
-    //return ZSTD_compress(dst, dstSize, src, srcSize, cLevel);
-}
-
-static size_t g_cSize = 0;
-static size_t local_ZSTD_decompress(const void* src, size_t srcSize,
-                                    void* dst, size_t dstSize,
-                                    void* buff2)
-{
-    (void)src; (void)srcSize;
-    return ZSTD_decompress(dst, dstSize, buff2, g_cSize);
-}
-
-static ZSTD_DCtx* g_zdc = NULL;
-
-#ifndef ZSTD_DLL_IMPORT
-extern size_t ZSTD_decodeLiteralsBlock(ZSTD_DCtx* ctx, const void* src, size_t srcSize);
-static size_t local_ZSTD_decodeLiteralsBlock(const void* src, size_t srcSize, void* dst, size_t dstSize, void* buff2)
-{
-    (void)src; (void)srcSize; (void)dst; (void)dstSize;
-    return ZSTD_decodeLiteralsBlock(g_zdc, buff2, g_cSize);
-}
-
-static size_t local_ZSTD_decodeSeqHeaders(const void* src, size_t srcSize, void* dst, size_t dstSize, void* buff2)
-{
-    int nbSeq;
-    (void)src; (void)srcSize; (void)dst; (void)dstSize;
-    return ZSTD_decodeSeqHeaders(g_zdc, &nbSeq, buff2, g_cSize);
-}
-#endif
-
-static ZSTD_CStream* g_cstream= NULL;
-static size_t
-local_ZSTD_compressStream(const void* src, size_t srcSize,
-                          void* dst, size_t dstCapacity,
-                          void* payload)
-{
-    ZSTD_outBuffer buffOut;
-    ZSTD_inBuffer buffIn;
-    ZSTD_parameters p;
-    ZSTD_frameParameters f = {1 /* contentSizeHeader*/, 0, 0};
-    p.fParams = f;
-    p.cParams = *(ZSTD_compressionParameters*)payload;
-    ZSTD_initCStream_advanced(g_cstream, NULL, 0, p, ZSTD_CONTENTSIZE_UNKNOWN);
-    buffOut.dst = dst;
-    buffOut.size = dstCapacity;
-    buffOut.pos = 0;
-    buffIn.src = src;
-    buffIn.size = srcSize;
-    buffIn.pos = 0;
-    ZSTD_compressStream(g_cstream, &buffOut, &buffIn);
-    ZSTD_endStream(g_cstream, &buffOut);
-    return buffOut.pos;
-}
-
-static size_t
-local_ZSTD_compressStream_freshCCtx(const void* src, size_t srcSize,
-                          void* dst, size_t dstCapacity,
-                          void* payload)
-{
-    ZSTD_CCtx* const cctx = ZSTD_createCCtx();
-    size_t r;
-    assert(cctx != NULL);
-
-    r = local_ZSTD_compressStream(src, srcSize, dst, dstCapacity, payload);
-
-    ZSTD_freeCCtx(cctx);
-
-    return r;
-}
-
-static size_t
-local_ZSTD_compress_generic_end(const void* src, size_t srcSize,
-                                void* dst, size_t dstCapacity,
-                                void* payload)
-{
-    (void)payload;
-    return ZSTD_compress2(g_cstream, dst, dstCapacity, src, srcSize);
-}
-
-static size_t
-local_ZSTD_compress_generic_continue(const void* src, size_t srcSize,
-                                     void* dst, size_t dstCapacity,
-                                     void* payload)
-{
-    ZSTD_outBuffer buffOut;
-    ZSTD_inBuffer buffIn;
-    (void)payload;
-    buffOut.dst = dst;
-    buffOut.size = dstCapacity;
-    buffOut.pos = 0;
-    buffIn.src = src;
-    buffIn.size = srcSize;
-    buffIn.pos = 0;
-    ZSTD_compressStream2(g_cstream, &buffOut, &buffIn, ZSTD_e_continue);
-    ZSTD_compressStream2(g_cstream, &buffOut, &buffIn, ZSTD_e_end);
-    return buffOut.pos;
-}
-
-static size_t
-local_ZSTD_compress_generic_T2_end(const void* src, size_t srcSize,
-                                   void* dst, size_t dstCapacity,
-                                   void* payload)
-{
-    (void)payload;
-    ZSTD_CCtx_setParameter(g_cstream, ZSTD_c_nbWorkers, 2);
-    return ZSTD_compress2(g_cstream, dst, dstCapacity, src, srcSize);
-}
-
-static size_t
-local_ZSTD_compress_generic_T2_continue(const void* src, size_t srcSize,
-                                        void* dst, size_t dstCapacity,
-                                        void* payload)
-{
-    ZSTD_outBuffer buffOut;
-    ZSTD_inBuffer buffIn;
-    (void)payload;
-    ZSTD_CCtx_setParameter(g_cstream, ZSTD_c_nbWorkers, 2);
-    buffOut.dst = dst;
-    buffOut.size = dstCapacity;
-    buffOut.pos = 0;
-    buffIn.src = src;
-    buffIn.size = srcSize;
-    buffIn.pos = 0;
-    ZSTD_compressStream2(g_cstream, &buffOut, &buffIn, ZSTD_e_continue);
-    while(ZSTD_compressStream2(g_cstream, &buffOut, &buffIn, ZSTD_e_end)) {}
-    return buffOut.pos;
-}
-
-static ZSTD_DStream* g_dstream= NULL;
-static size_t
-local_ZSTD_decompressStream(const void* src, size_t srcSize,
-                            void* dst, size_t dstCapacity,
-                            void* buff2)
-{
-    ZSTD_outBuffer buffOut;
-    ZSTD_inBuffer buffIn;
-    (void)src; (void)srcSize;
-    ZSTD_initDStream(g_dstream);
-    buffOut.dst = dst;
-    buffOut.size = dstCapacity;
-    buffOut.pos = 0;
-    buffIn.src = buff2;
-    buffIn.size = g_cSize;
-    buffIn.pos = 0;
-    ZSTD_decompressStream(g_dstream, &buffOut, &buffIn);
-    return buffOut.pos;
-}
-
-#ifndef ZSTD_DLL_IMPORT
-static size_t local_ZSTD_compressContinue(const void* src, size_t srcSize,
-                                          void* dst, size_t dstCapacity,
-                                          void* payload)
-{
-    ZSTD_parameters p;
-    ZSTD_frameParameters f = { 1 /* contentSizeHeader*/, 0, 0 };
-    p.fParams = f;
-    p.cParams = *(ZSTD_compressionParameters*)payload;
-    ZSTD_compressBegin_advanced(g_zcc, NULL, 0, p, srcSize);
-    return ZSTD_compressEnd(g_zcc, dst, dstCapacity, src, srcSize);
-}
-
-#define FIRST_BLOCK_SIZE 8
-static size_t
-local_ZSTD_compressContinue_extDict(const void* src, size_t srcSize,
-                                    void* dst, size_t dstCapacity,
-                                    void* payload)
-{
-    BYTE firstBlockBuf[FIRST_BLOCK_SIZE];
-
-    ZSTD_parameters p;
-    ZSTD_frameParameters const f = { 1, 0, 0 };
-    p.fParams = f;
-    p.cParams = *(ZSTD_compressionParameters*)payload;
-    ZSTD_compressBegin_advanced(g_zcc, NULL, 0, p, srcSize);
-    memcpy(firstBlockBuf, src, FIRST_BLOCK_SIZE);
-
-    {   size_t const compressResult = ZSTD_compressContinue(g_zcc,
-                                            dst, dstCapacity,
-                                            firstBlockBuf, FIRST_BLOCK_SIZE);
-        if (ZSTD_isError(compressResult)) {
-            DISPLAY("local_ZSTD_compressContinue_extDict error : %s\n",
-                    ZSTD_getErrorName(compressResult));
-            return compressResult;
-        }
-        dst = (BYTE*)dst + compressResult;
-        dstCapacity -= compressResult;
-    }
-    return ZSTD_compressEnd(g_zcc, dst, dstCapacity,
-                            (const BYTE*)src + FIRST_BLOCK_SIZE,
-                            srcSize - FIRST_BLOCK_SIZE);
-}
-
-static size_t local_ZSTD_decompressContinue(const void* src, size_t srcSize,
-                                            void* dst, size_t dstCapacity,
-                                            void* buff2)
-{
-    size_t regeneratedSize = 0;
-    const BYTE* ip = (const BYTE*)buff2;
-    const BYTE* const iend = ip + g_cSize;
-    BYTE* op = (BYTE*)dst;
-    size_t remainingCapacity = dstCapacity;
-
-    (void)src; (void)srcSize;  /* unused */
-    ZSTD_decompressBegin(g_zdc);
-    while (ip < iend) {
-        size_t const iSize = ZSTD_nextSrcSizeToDecompress(g_zdc);
-        size_t const decodedSize = ZSTD_decompressContinue(g_zdc, op, remainingCapacity, ip, iSize);
-        ip += iSize;
-        regeneratedSize += decodedSize;
-        op += decodedSize;
-        remainingCapacity -= decodedSize;
-    }
-
-    return regeneratedSize;
-}
-#endif
-
-
-/*_*******************************************************
-*  Bench functions
-*********************************************************/
-static int benchMem(unsigned benchNb,
-                    const void* src, size_t srcSize,
-                    int cLevel, ZSTD_compressionParameters cparams)
-{
-    size_t dstBuffSize = ZSTD_compressBound(srcSize);
-    BYTE*  dstBuff;
-    void*  dstBuff2;
-    void*  payload;
-    const char* benchName;
-    BMK_benchFn_t benchFunction;
-    int errorcode = 0;
-
-    /* Selection */
-    switch(benchNb)
-    {
-    case 1:
-        benchFunction = local_ZSTD_compress; benchName = "compress";
-        break;
-    case 2:
-        benchFunction = local_ZSTD_decompress; benchName = "decompress";
-        break;
-#ifndef ZSTD_DLL_IMPORT
-    case 11:
-        benchFunction = local_ZSTD_compressContinue; benchName = "compressContinue";
-        break;
-    case 12:
-        benchFunction = local_ZSTD_compressContinue_extDict; benchName = "compressContinue_extDict";
-        break;
-    case 13:
-        benchFunction = local_ZSTD_decompressContinue; benchName = "decompressContinue";
-        break;
-    case 31:
-        benchFunction = local_ZSTD_decodeLiteralsBlock; benchName = "decodeLiteralsBlock";
-        break;
-    case 32:
-        benchFunction = local_ZSTD_decodeSeqHeaders; benchName = "decodeSeqHeaders";
-        break;
-#endif
-    case 41:
-        benchFunction = local_ZSTD_compressStream; benchName = "compressStream";
-        break;
-    case 42:
-        benchFunction = local_ZSTD_decompressStream; benchName = "decompressStream";
-        break;
-    case 43:
-        benchFunction = local_ZSTD_compressStream_freshCCtx; benchName = "compressStream_freshCCtx";
-        break;
-    case 51:
-        benchFunction = local_ZSTD_compress_generic_continue; benchName = "compress_generic, continue";
-        break;
-    case 52:
-        benchFunction = local_ZSTD_compress_generic_end; benchName = "compress_generic, end";
-        break;
-    case 61:
-        benchFunction = local_ZSTD_compress_generic_T2_continue; benchName = "compress_generic, -T2, continue";
-        break;
-    case 62:
-        benchFunction = local_ZSTD_compress_generic_T2_end; benchName = "compress_generic, -T2, end";
-        break;
-    default :
-        return 0;
-    }
-
-    /* Allocation */
-    dstBuff = (BYTE*)malloc(dstBuffSize);
-    dstBuff2 = malloc(dstBuffSize);
-    if ((!dstBuff) || (!dstBuff2)) {
-        DISPLAY("\nError: not enough memory!\n");
-        free(dstBuff); free(dstBuff2);
-        return 12;
-    }
-    payload = dstBuff2;
-    if (g_zcc==NULL) g_zcc = ZSTD_createCCtx();
-    if (g_zdc==NULL) g_zdc = ZSTD_createDCtx();
-    if (g_cstream==NULL) g_cstream = ZSTD_createCStream();
-    if (g_dstream==NULL) g_dstream = ZSTD_createDStream();
-
-    /* DISPLAY("params: cLevel %d, wlog %d hlog %d clog %d slog %d mml %d tlen %d strat %d \n",
-          cLevel, cparams->windowLog, cparams->hashLog, cparams->chainLog, cparams->searchLog,
-          cparams->minMatch, cparams->targetLength, cparams->strategy); */
-
-    ZSTD_CCtx_setParameter(g_zcc, ZSTD_c_compressionLevel, cLevel);
-    ZSTD_CCtx_setParameter(g_zcc, ZSTD_c_windowLog, (int)cparams.windowLog);
-    ZSTD_CCtx_setParameter(g_zcc, ZSTD_c_hashLog, (int)cparams.hashLog);
-    ZSTD_CCtx_setParameter(g_zcc, ZSTD_c_chainLog, (int)cparams.chainLog);
-    ZSTD_CCtx_setParameter(g_zcc, ZSTD_c_searchLog, (int)cparams.searchLog);
-    ZSTD_CCtx_setParameter(g_zcc, ZSTD_c_minMatch, (int)cparams.minMatch);
-    ZSTD_CCtx_setParameter(g_zcc, ZSTD_c_targetLength, (int)cparams.targetLength);
-    ZSTD_CCtx_setParameter(g_zcc, ZSTD_c_strategy, cparams.strategy);
-
-
-    ZSTD_CCtx_setParameter(g_cstream, ZSTD_c_compressionLevel, cLevel);
-    ZSTD_CCtx_setParameter(g_cstream, ZSTD_c_windowLog, (int)cparams.windowLog);
-    ZSTD_CCtx_setParameter(g_cstream, ZSTD_c_hashLog, (int)cparams.hashLog);
-    ZSTD_CCtx_setParameter(g_cstream, ZSTD_c_chainLog, (int)cparams.chainLog);
-    ZSTD_CCtx_setParameter(g_cstream, ZSTD_c_searchLog, (int)cparams.searchLog);
-    ZSTD_CCtx_setParameter(g_cstream, ZSTD_c_minMatch, (int)cparams.minMatch);
-    ZSTD_CCtx_setParameter(g_cstream, ZSTD_c_targetLength, (int)cparams.targetLength);
-    ZSTD_CCtx_setParameter(g_cstream, ZSTD_c_strategy, cparams.strategy);
-
-    /* Preparation */
-    switch(benchNb)
-    {
-    case 1:
-        payload = &cparams;
-        break;
-    case 2:
-        g_cSize = ZSTD_compress(dstBuff2, dstBuffSize, src, srcSize, cLevel);
-        break;
-#ifndef ZSTD_DLL_IMPORT
-    case 11:
-        payload = &cparams;
-        break;
-    case 12:
-        payload = &cparams;
-        break;
-    case 13 :
-        g_cSize = ZSTD_compress(dstBuff2, dstBuffSize, src, srcSize, cLevel);
-        break;
-    case 31:  /* ZSTD_decodeLiteralsBlock : starts literals block in dstBuff2 */
-        {   size_t frameHeaderSize;
-            g_cSize = ZSTD_compress(dstBuff, dstBuffSize, src, srcSize, cLevel);
-            frameHeaderSize = ZSTD_frameHeaderSize(dstBuff, ZSTD_FRAMEHEADERSIZE_PREFIX(ZSTD_f_zstd1));
-            CONTROL(!ZSTD_isError(frameHeaderSize));
-            /* check block is compressible, hence contains a literals section */
-            {   blockProperties_t bp;
-                ZSTD_getcBlockSize(dstBuff+frameHeaderSize, dstBuffSize, &bp);  /* Get 1st block type */
-                if (bp.blockType != bt_compressed) {
-                    DISPLAY("ZSTD_decodeLiteralsBlock : impossible to test on this sample (not compressible)\n");
-                    goto _cleanOut;
-            }   }
-            {   size_t const skippedSize = frameHeaderSize + ZSTD_blockHeaderSize;
-                memcpy(dstBuff2, dstBuff+skippedSize, g_cSize-skippedSize);
-            }
-            srcSize = srcSize > 128 KB ? 128 KB : srcSize;    /* speed relative to block */
-            ZSTD_decompressBegin(g_zdc);
-            break;
-        }
-    case 32:   /* ZSTD_decodeSeqHeaders */
-        {   blockProperties_t bp;
-            const BYTE* ip = dstBuff;
-            const BYTE* iend;
-            {   size_t const cSize = ZSTD_compress(dstBuff, dstBuffSize, src, srcSize, cLevel);
-                CONTROL(cSize > ZSTD_FRAMEHEADERSIZE_PREFIX(ZSTD_f_zstd1));
-            }
-            /* Skip frame Header */
-            {   size_t const frameHeaderSize = ZSTD_frameHeaderSize(dstBuff, ZSTD_FRAMEHEADERSIZE_PREFIX(ZSTD_f_zstd1));
-                CONTROL(!ZSTD_isError(frameHeaderSize));
-                ip += frameHeaderSize;
-            }
-            /* Find end of block */
-            {   size_t const cBlockSize = ZSTD_getcBlockSize(ip, dstBuffSize, &bp);   /* Get 1st block type */
-                if (bp.blockType != bt_compressed) {
-                    DISPLAY("ZSTD_decodeSeqHeaders : impossible to test on this sample (not compressible)\n");
-                    goto _cleanOut;
-                }
-                iend = ip + ZSTD_blockHeaderSize + cBlockSize;   /* End of first block */
-            }
-            ip += ZSTD_blockHeaderSize;    /* skip block header */
-            ZSTD_decompressBegin(g_zdc);
-            CONTROL(iend > ip);
-            ip += ZSTD_decodeLiteralsBlock(g_zdc, ip, (size_t)(iend-ip));   /* skip literal segment */
-            g_cSize = (size_t)(iend-ip);
-            memcpy(dstBuff2, ip, g_cSize);   /* copy rest of block (it starts by SeqHeader) */
-            srcSize = srcSize > 128 KB ? 128 KB : srcSize;   /* speed relative to block */
-            break;
-        }
-#else
-    case 31:
-        goto _cleanOut;
-#endif
-    case 41 :
-        payload = &cparams;
-        break;
-    case 42 :
-        g_cSize = ZSTD_compress(payload, dstBuffSize, src, srcSize, cLevel);
-        break;
-    case 43 :
-        payload = &cparams;
-        break;
-
-    /* test functions */
-    /* convention: test functions have ID > 100 */
-
-    default : ;
-    }
-
-     /* warming up dstBuff */
-    { size_t i; for (i=0; i<dstBuffSize; i++) dstBuff[i]=(BYTE)i; }
-
-    /* benchmark loop */
-    {   BMK_timedFnState_t* const tfs = BMK_createTimedFnState(g_nbIterations * 1000, 1000);
-        void* const avoidStrictAliasingPtr = &dstBuff;
-        BMK_benchParams_t bp;
-        BMK_runTime_t bestResult;
-        bestResult.sumOfReturn = 0;
-        bestResult.nanoSecPerRun = (double)TIMELOOP_NANOSEC * 2000000000;  /* hopefully large enough : must be larger than any potential measurement */
-        CONTROL(tfs != NULL);
-
-        bp.benchFn = benchFunction;
-        bp.benchPayload = payload;
-        bp.initFn = NULL;
-        bp.initPayload = NULL;
-        bp.errorFn = ZSTD_isError;
-        bp.blockCount = 1;
-        bp.srcBuffers = &src;
-        bp.srcSizes = &srcSize;
-        bp.dstBuffers = (void* const*) avoidStrictAliasingPtr;  /* circumvent strict aliasing warning on gcc-8,
-                                                                 * because gcc considers that `void* const *`  and `void**` are 2 different types */
-        bp.dstCapacities = &dstBuffSize;
-        bp.blockResults = NULL;
-
-        for (;;) {
-            BMK_runOutcome_t const bOutcome = BMK_benchTimedFn(tfs, bp);
-
-            if (!BMK_isSuccessful_runOutcome(bOutcome)) {
-                DISPLAY("ERROR benchmarking function ! ! \n");
-                errorcode = 1;
-                goto _cleanOut;
-            }
-
-            {   BMK_runTime_t const newResult = BMK_extract_runTime(bOutcome);
-                if (newResult.nanoSecPerRun < bestResult.nanoSecPerRun )
-                    bestResult.nanoSecPerRun = newResult.nanoSecPerRun;
-                DISPLAY("\r%2u#%-29.29s:%8.1f MB/s  (%8u) ",
-                        benchNb, benchName,
-                        (double)srcSize * TIMELOOP_NANOSEC / bestResult.nanoSecPerRun / MB_UNIT,
-                        (unsigned)newResult.sumOfReturn );
-            }
-
-            if ( BMK_isCompleted_TimedFn(tfs) ) break;
-        }
-        BMK_freeTimedFnState(tfs);
-    }
-    DISPLAY("\n");
-
-_cleanOut:
-    free(dstBuff);
-    free(dstBuff2);
-    ZSTD_freeCCtx(g_zcc); g_zcc=NULL;
-    ZSTD_freeDCtx(g_zdc); g_zdc=NULL;
-    ZSTD_freeCStream(g_cstream); g_cstream=NULL;
-    ZSTD_freeDStream(g_dstream); g_dstream=NULL;
-    return errorcode;
-}
-
-
-static int benchSample(U32 benchNb,
-                       size_t benchedSize, double compressibility,
-                       int cLevel, ZSTD_compressionParameters cparams)
-{
-    /* Allocation */
-    void* const origBuff = malloc(benchedSize);
-    if (!origBuff) { DISPLAY("\nError: not enough memory!\n"); return 12; }
-
-    /* Fill buffer */
-    RDG_genBuffer(origBuff, benchedSize, compressibility, 0.0, 0);
-
-    /* bench */
-    DISPLAY("\r%70s\r", "");
-    DISPLAY(" Sample %u bytes : \n", (unsigned)benchedSize);
-    if (benchNb) {
-        benchMem(benchNb, origBuff, benchedSize, cLevel, cparams);
-    } else {  /* 0 == run all tests */
-        for (benchNb=0; benchNb<100; benchNb++) {
-            benchMem(benchNb, origBuff, benchedSize, cLevel, cparams);
-    }   }
-
-    free(origBuff);
-    return 0;
-}
-
-
-static int benchFiles(U32 benchNb,
-                      const char** fileNamesTable, const int nbFiles,
-                      int cLevel, ZSTD_compressionParameters cparams)
-{
-    /* Loop for each file */
-    int fileIdx;
-    for (fileIdx=0; fileIdx<nbFiles; fileIdx++) {
-        const char* const inFileName = fileNamesTable[fileIdx];
-        FILE* const inFile = fopen( inFileName, "rb" );
-        size_t benchedSize;
-
-        /* Check file existence */
-        if (inFile==NULL) { DISPLAY( "Pb opening %s\n", inFileName); return 11; }
-
-        /* Memory allocation & restrictions */
-        {   U64 const inFileSize = UTIL_getFileSize(inFileName);
-            if (inFileSize == UTIL_FILESIZE_UNKNOWN) {
-                DISPLAY( "Cannot measure size of %s\n", inFileName);
-                fclose(inFile);
-                return 11;
-            }
-            benchedSize = BMK_findMaxMem(inFileSize*3) / 3;
-            if ((U64)benchedSize > inFileSize)
-                benchedSize = (size_t)inFileSize;
-            if ((U64)benchedSize < inFileSize) {
-                DISPLAY("Not enough memory for '%s' full size; testing %u MB only... \n",
-                        inFileName, (unsigned)(benchedSize>>20));
-        }   }
-
-        /* Alloc */
-        {   void* const origBuff = malloc(benchedSize);
-            if (!origBuff) { DISPLAY("\nError: not enough memory!\n"); fclose(inFile); return 12; }
-
-            /* Fill input buffer */
-            DISPLAY("Loading %s...       \r", inFileName);
-            {   size_t const readSize = fread(origBuff, 1, benchedSize, inFile);
-                fclose(inFile);
-                if (readSize != benchedSize) {
-                    DISPLAY("\nError: problem reading file '%s' !!    \n", inFileName);
-                    free(origBuff);
-                    return 13;
-            }   }
-
-            /* bench */
-            DISPLAY("\r%70s\r", "");   /* blank line */
-            DISPLAY(" %s : \n", inFileName);
-            if (benchNb) {
-                benchMem(benchNb, origBuff, benchedSize, cLevel, cparams);
-            } else {
-                for (benchNb=0; benchNb<100; benchNb++) {
-                    benchMem(benchNb, origBuff, benchedSize, cLevel, cparams);
-            }   }
-
-            free(origBuff);
-    }   }
-
-    return 0;
-}
-
-
-
-/*_*******************************************************
-*  Argument Parsing
-*********************************************************/
-
-#define ERROR_OUT(msg) { DISPLAY("%s \n", msg); exit(1); }
-
-static unsigned readU32FromChar(const char** stringPtr)
-{
-    const char errorMsg[] = "error: numeric value too large";
-    unsigned result = 0;
-    while ((**stringPtr >='0') && (**stringPtr <='9')) {
-        unsigned const max = (((unsigned)(-1)) / 10) - 1;
-        if (result > max) ERROR_OUT(errorMsg);
-        result *= 10;
-        result += (unsigned)(**stringPtr - '0');
-        (*stringPtr)++ ;
-    }
-    if ((**stringPtr=='K') || (**stringPtr=='M')) {
-        unsigned const maxK = ((unsigned)(-1)) >> 10;
-        if (result > maxK) ERROR_OUT(errorMsg);
-        result <<= 10;
-        if (**stringPtr=='M') {
-            if (result > maxK) ERROR_OUT(errorMsg);
-            result <<= 10;
-        }
-        (*stringPtr)++;  /* skip `K` or `M` */
-        if (**stringPtr=='i') (*stringPtr)++;
-        if (**stringPtr=='B') (*stringPtr)++;
-    }
-    return result;
-}
-
-static int longCommandWArg(const char** stringPtr, const char* longCommand)
-{
-    size_t const comSize = strlen(longCommand);
-    int const result = !strncmp(*stringPtr, longCommand, comSize);
-    if (result) *stringPtr += comSize;
-    return result;
-}
-
-
-/*_*******************************************************
-*  Command line
-*********************************************************/
-
-static int usage(const char* exename)
-{
-    DISPLAY( "Usage :\n");
-    DISPLAY( "      %s [arg] file1 file2 ... fileX\n", exename);
-    DISPLAY( "Arguments :\n");
-    DISPLAY( " -H/-h  : Help (this text + advanced options)\n");
-    return 0;
-}
-
-static int usage_advanced(const char* exename)
-{
-    usage(exename);
-    DISPLAY( "\nAdvanced options :\n");
-    DISPLAY( " -b#    : test only function # \n");
-    DISPLAY( " -l#    : benchmark functions at that compression level (default : %i)\n", DEFAULT_CLEVEL);
-    DISPLAY( "--zstd= : custom parameter selection. Format same as zstdcli \n");
-    DISPLAY( " -P#    : sample compressibility (default : %.1f%%)\n", COMPRESSIBILITY_DEFAULT * 100);
-    DISPLAY( " -B#    : sample size (default : %u)\n", (unsigned)kSampleSizeDefault);
-    DISPLAY( " -i#    : iteration loops [1-9](default : %i)\n", NBLOOPS);
-    return 0;
-}
-
-static int badusage(const char* exename)
-{
-    DISPLAY("Wrong parameters\n");
-    usage(exename);
-    return 1;
-}
-
-int main(int argc, const char** argv)
-{
-    int argNb, filenamesStart=0, result;
-    const char* const exename = argv[0];
-    const char* input_filename = NULL;
-    U32 benchNb = 0, main_pause = 0;
-    int cLevel = DEFAULT_CLEVEL;
-    ZSTD_compressionParameters cparams = ZSTD_getCParams(cLevel, 0, 0);
-    size_t sampleSize = kSampleSizeDefault;
-    double compressibility = COMPRESSIBILITY_DEFAULT;
-
-    DISPLAY(WELCOME_MESSAGE);
-    if (argc<1) return badusage(exename);
-
-    for (argNb=1; argNb<argc; argNb++) {
-        const char* argument = argv[argNb];
-        CONTROL(argument != NULL);
-
-        if (longCommandWArg(&argument, "--zstd=")) {
-            for ( ; ;) {
-                if (longCommandWArg(&argument, "windowLog=") || longCommandWArg(&argument, "wlog=")) { cparams.windowLog = readU32FromChar(&argument); if (argument[0]==',') { argument++; continue; } else break; }
-                if (longCommandWArg(&argument, "chainLog=") || longCommandWArg(&argument, "clog=")) { cparams.chainLog = readU32FromChar(&argument); if (argument[0]==',') { argument++; continue; } else break; }
-                if (longCommandWArg(&argument, "hashLog=") || longCommandWArg(&argument, "hlog=")) { cparams.hashLog = readU32FromChar(&argument); if (argument[0]==',') { argument++; continue; } else break; }
-                if (longCommandWArg(&argument, "searchLog=") || longCommandWArg(&argument, "slog=")) { cparams.searchLog = readU32FromChar(&argument); if (argument[0]==',') { argument++; continue; } else break; }
-                if (longCommandWArg(&argument, "minMatch=") || longCommandWArg(&argument, "mml=")) { cparams.minMatch = readU32FromChar(&argument); if (argument[0]==',') { argument++; continue; } else break; }
-                if (longCommandWArg(&argument, "targetLength=") || longCommandWArg(&argument, "tlen=")) { cparams.targetLength = readU32FromChar(&argument); if (argument[0]==',') { argument++; continue; } else break; }
-                if (longCommandWArg(&argument, "strategy=") || longCommandWArg(&argument, "strat=")) { cparams.strategy = (ZSTD_strategy)(readU32FromChar(&argument)); if (argument[0]==',') { argument++; continue; } else break; }
-                if (longCommandWArg(&argument, "level=") || longCommandWArg(&argument, "lvl=")) { cLevel = (int)readU32FromChar(&argument); cparams = ZSTD_getCParams(cLevel, 0, 0); if (argument[0]==',') { argument++; continue; } else break; }
-                DISPLAY("invalid compression parameter \n");
-                return 1;
-            }
-
-            /* check end of string */
-            if (argument[0] != 0) {
-                DISPLAY("invalid --zstd= format \n");
-                return 1;
-            } else {
-                continue;
-            }
-
-        } else if (argument[0]=='-') { /* Commands (note : aggregated commands are allowed) */
-            argument++;
-            while (argument[0]!=0) {
-
-                switch(argument[0])
-                {
-                    /* Display help on usage */
-                case 'h':
-                case 'H': return usage_advanced(exename);
-
-                    /* Pause at the end (hidden option) */
-                case 'p': main_pause = 1; break;
-
-                    /* Select specific algorithm to bench */
-                case 'b':
-                    argument++;
-                    benchNb = readU32FromChar(&argument);
-                    break;
-
-                    /* Select compression level to use */
-                case 'l':
-                    argument++;
-                    cLevel = (int)readU32FromChar(&argument);
-                    cparams = ZSTD_getCParams(cLevel, 0, 0);
-                    break;
-
-                    /* Select compressibility of synthetic sample */
-                case 'P':
-                    argument++;
-                    compressibility = (double)readU32FromChar(&argument) / 100.;
-                    break;
-
-                    /* Select size of synthetic sample */
-                case 'B':
-                    argument++;
-                    sampleSize = (size_t)readU32FromChar(&argument);
-                    break;
-
-                    /* Modify Nb Iterations */
-                case 'i':
-                    argument++;
-                    g_nbIterations = readU32FromChar(&argument);
-                    break;
-
-                    /* Unknown command */
-                default : return badusage(exename);
-                }
-            }
-            continue;
-        }
-
-        /* first provided filename is input */
-        if (!input_filename) { input_filename=argument; filenamesStart=argNb; continue; }
-    }
-
-
-
-    if (filenamesStart==0)   /* no input file */
-        result = benchSample(benchNb, sampleSize, compressibility, cLevel, cparams);
-    else
-        result = benchFiles(benchNb, argv+filenamesStart, argc-filenamesStart, cLevel, cparams);
-
-    if (main_pause) { int unused; printf("press enter...\n"); unused = getchar(); (void)unused; }
-
-    return result;
-}
diff --git a/tests/fuzz/Makefile b/tests/fuzz/Makefile
deleted file mode 100644
index f66dadef041c..000000000000
--- a/tests/fuzz/Makefile
+++ /dev/null
@@ -1,147 +0,0 @@
-# ################################################################
-# Copyright (c) 2016-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under both the BSD-style license (found in the
-# LICENSE file in the root directory of this source tree) and the GPLv2 (found
-# in the COPYING file in the root directory of this source tree).
-# ################################################################
-
-# Optionally user defined flags
-CFLAGS ?= -O3
-CXXFLAGS ?= -O3
-CPPFLAGS ?=
-LDFLAGS ?=
-ARFLAGS ?=
-LIB_FUZZING_ENGINE ?= libregression.a
-PYTHON ?= python
-ifeq ($(shell uname), Darwin)
-	DOWNLOAD?=curl -L -o
-else
-	DOWNLOAD?=wget -O
-endif
-CORPORA_URL_PREFIX:=https://github.com/facebook/zstd/releases/download/fuzz-corpora/
-
-ZSTDDIR = ../../lib
-PRGDIR = ../../programs
-
-FUZZ_CPPFLAGS := -I$(ZSTDDIR) -I$(ZSTDDIR)/common -I$(ZSTDDIR)/compress \
-	-I$(ZSTDDIR)/dictBuilder -I$(ZSTDDIR)/deprecated -I$(ZSTDDIR)/legacy \
-	-I$(PRGDIR) -DZSTD_MULTITHREAD -DZSTD_LEGACY_SUPPORT=1 $(CPPFLAGS)
-FUZZ_EXTRA_FLAGS := -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow \
-	-Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement \
-	-Wstrict-prototypes -Wundef \
-	-Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings \
-	-Wredundant-decls \
-	-g -fno-omit-frame-pointer
-FUZZ_CFLAGS := $(FUZZ_EXTRA_FLAGS) $(CFLAGS)
-FUZZ_CXXFLAGS := $(FUZZ_EXTRA_FLAGS) -std=c++11 $(CXXFLAGS)
-FUZZ_LDFLAGS := -pthread $(LDFLAGS)
-FUZZ_ARFLAGS := $(ARFLAGS)
-FUZZ_TARGET_FLAGS = $(FUZZ_CPPFLAGS) $(FUZZ_CXXFLAGS) $(FUZZ_LDFLAGS)
-
-FUZZ_HEADERS := fuzz_helpers.h fuzz.h zstd_helpers.h fuzz_data_producer.h
-FUZZ_SRC := $(PRGDIR)/util.c zstd_helpers.c fuzz_data_producer.c
-
-ZSTDCOMMON_SRC := $(ZSTDDIR)/common/*.c
-ZSTDCOMP_SRC   := $(ZSTDDIR)/compress/*.c
-ZSTDDECOMP_SRC := $(ZSTDDIR)/decompress/*.c
-ZSTDDICT_SRC := $(ZSTDDIR)/dictBuilder/*.c
-ZSTDLEGACY_SRC := $(ZSTDDIR)/legacy/*.c
-FUZZ_SRC       := \
-	$(FUZZ_SRC) \
-	$(ZSTDDECOMP_SRC) \
-	$(ZSTDCOMMON_SRC) \
-	$(ZSTDCOMP_SRC) \
-	$(ZSTDDICT_SRC) \
-	$(ZSTDLEGACY_SRC)
-
-FUZZ_OBJ := $(patsubst %.c,%.o, $(wildcard $(FUZZ_SRC)))
-
-
-.PHONY: default all clean cleanall
-
-default: all
-
-FUZZ_TARGETS :=       \
-	simple_round_trip \
-	stream_round_trip \
-	block_round_trip  \
-	simple_decompress \
-	stream_decompress \
-	block_decompress  \
-	dictionary_round_trip \
-	dictionary_decompress \
-	zstd_frame_info \
-	simple_compress \
-	dictionary_loader
-
-all: $(FUZZ_TARGETS)
-
-%.o: %.c
-	$(CC) $(FUZZ_CPPFLAGS) $(FUZZ_CFLAGS) $^ -c -o $@
-
-simple_round_trip: $(FUZZ_HEADERS) $(FUZZ_OBJ) simple_round_trip.o
-	$(CXX) $(FUZZ_TARGET_FLAGS) $(FUZZ_OBJ) simple_round_trip.o $(LIB_FUZZING_ENGINE) -o $@
-
-stream_round_trip: $(FUZZ_HEADERS) $(FUZZ_OBJ) stream_round_trip.o
-	$(CXX) $(FUZZ_TARGET_FLAGS) $(FUZZ_OBJ) stream_round_trip.o $(LIB_FUZZING_ENGINE) -o $@
-
-block_round_trip: $(FUZZ_HEADERS) $(FUZZ_OBJ) block_round_trip.o
-	$(CXX) $(FUZZ_TARGET_FLAGS) $(FUZZ_OBJ) block_round_trip.o $(LIB_FUZZING_ENGINE) -o $@
-
-simple_decompress: $(FUZZ_HEADERS) $(FUZZ_OBJ) simple_decompress.o
-	$(CXX) $(FUZZ_TARGET_FLAGS) $(FUZZ_OBJ) simple_decompress.o $(LIB_FUZZING_ENGINE) -o $@
-
-stream_decompress: $(FUZZ_HEADERS) $(FUZZ_OBJ) stream_decompress.o
-	$(CXX) $(FUZZ_TARGET_FLAGS) $(FUZZ_OBJ) stream_decompress.o $(LIB_FUZZING_ENGINE) -o $@
-
-block_decompress: $(FUZZ_HEADERS) $(FUZZ_OBJ) block_decompress.o
-	$(CXX) $(FUZZ_TARGET_FLAGS) $(FUZZ_OBJ) block_decompress.o $(LIB_FUZZING_ENGINE) -o $@
-
-dictionary_round_trip: $(FUZZ_HEADERS) $(FUZZ_OBJ) dictionary_round_trip.o
-	$(CXX) $(FUZZ_TARGET_FLAGS) $(FUZZ_OBJ) dictionary_round_trip.o $(LIB_FUZZING_ENGINE) -o $@
-
-dictionary_decompress: $(FUZZ_HEADERS) $(FUZZ_OBJ) dictionary_decompress.o
-	$(CXX) $(FUZZ_TARGET_FLAGS) $(FUZZ_OBJ) dictionary_decompress.o $(LIB_FUZZING_ENGINE) -o $@
-
-simple_compress: $(FUZZ_HEADERS) $(FUZZ_OBJ) simple_compress.o
-	$(CXX) $(FUZZ_TARGET_FLAGS) $(FUZZ_OBJ) simple_compress.o $(LIB_FUZZING_ENGINE) -o $@
-
-zstd_frame_info: $(FUZZ_HEADERS) $(FUZZ_OBJ) zstd_frame_info.o
-	$(CXX) $(FUZZ_TARGET_FLAGS) $(FUZZ_OBJ) zstd_frame_info.o $(LIB_FUZZING_ENGINE) -o $@
-
-dictionary_loader: $(FUZZ_HEADERS) $(FUZZ_OBJ) dictionary_loader.o
-	$(CXX) $(FUZZ_TARGET_FLAGS) $(FUZZ_OBJ) dictionary_loader.o $(LIB_FUZZING_ENGINE) -o $@
-
-libregression.a: $(FUZZ_HEADERS) $(PRGDIR)/util.h $(PRGDIR)/util.c regression_driver.o
-	$(AR) $(FUZZ_ARFLAGS) $@ regression_driver.o
-
-corpora/%_seed_corpus.zip:
-	@mkdir -p corpora
-	$(DOWNLOAD) $@ $(CORPORA_URL_PREFIX)$*_seed_corpus.zip
-
-corpora/%: corpora/%_seed_corpus.zip
-	unzip -q $^ -d $@
-
-.PHONY: corpora
-corpora: $(patsubst %,corpora/%,$(FUZZ_TARGETS))
-
-.PHONY: seedcorpora
-seedcorpora: $(patsubst %,corpora/%_seed_corpus.zip,$(FUZZ_TARGETS))
-
-regressiontest: corpora
-	CC="$(CC)" CXX="$(CXX)" CFLAGS="$(CFLAGS)" CXXFLAGS="$(CXXFLAGS)" LDFLAGS="$(LDFLAGS)" $(PYTHON) ./fuzz.py build all
-	$(PYTHON) ./fuzz.py regression all
-
-clean:
-	@$(MAKE) -C $(ZSTDDIR) clean
-	@$(RM) *.a *.o
-	@$(RM) simple_round_trip stream_round_trip simple_decompress \
-           stream_decompress block_decompress block_round_trip \
-           simple_compress dictionary_round_trip dictionary_decompress \
-           zstd_frame_info
-
-cleanall:
-	@$(RM) -r Fuzzer
-	@$(RM) -r corpora
diff --git a/tests/fuzz/README.md b/tests/fuzz/README.md
deleted file mode 100644
index 71afa40631a1..000000000000
--- a/tests/fuzz/README.md
+++ /dev/null
@@ -1,101 +0,0 @@
-# Fuzzing
-
-Each fuzzing target can be built with multiple engines.
-Zstd provides a fuzz corpus for each target that can be downloaded with
-the command:
-
-```
-make corpora
-```
-
-It will download each corpus into `./corpora/TARGET`.
-
-## fuzz.py
-
-`fuzz.py` is a helper script for building and running fuzzers.
-Run `./fuzz.py -h` for the commands and run `./fuzz.py COMMAND -h` for
-command specific help.
-
-### Generating Data
-
-`fuzz.py` provides a utility to generate seed data for each fuzzer.
-
-```
-make -C ../tests decodecorpus
-./fuzz.py gen TARGET
-```
-
-By default it outputs 100 samples, each at most 8KB into `corpora/TARGET-seed`,
-but that can be configured with the `--number`, `--max-size-log` and `--seed`
-flags.
-
-### Build
-It respects the usual build environment variables `CC`, `CFLAGS`, etc.
-The environment variables can be overridden with the corresponding flags
-`--cc`, `--cflags`, etc.
-The specific fuzzing engine is selected with `LIB_FUZZING_ENGINE` or
-`--lib-fuzzing-engine`, the default is `libregression.a`.
-Alternatively, you can use Clang's built in fuzzing engine with
-`--enable-fuzzer`.
-It has flags that can easily set up sanitizers `--enable-{a,ub,m}san`, and
-coverage instrumentation `--enable-coverage`.
-It sets sane defaults which can be overridden with flags `--debug`,
-`--enable-ubsan-pointer-overflow`, etc.
-Run `./fuzz.py build -h` for help.
-
-### Running Fuzzers
-
-`./fuzz.py` can run `libfuzzer`, `afl`, and `regression` tests.
-See the help of the relevant command for options.
-Flags not parsed by `fuzz.py` are passed to the fuzzing engine.
-The command used to run the fuzzer is printed for debugging.
-
-## LibFuzzer
-
-```
-# Build the fuzz targets
-./fuzz.py build all --enable-fuzzer --enable-asan --enable-ubsan --cc clang --cxx clang++
-# OR equivalently
-CC=clang CXX=clang++ ./fuzz.py build all --enable-fuzzer --enable-asan --enable-ubsan
-# Run the fuzzer
-./fuzz.py libfuzzer TARGET <libfuzzer args like -jobs=4>
-```
-
-where `TARGET` could be `simple_decompress`, `stream_round_trip`, etc.
-
-### MSAN
-
-Fuzzing with `libFuzzer` and `MSAN` is as easy as:
-
-```
-CC=clang CXX=clang++ ./fuzz.py build all --enable-fuzzer --enable-msan
-./fuzz.py libfuzzer TARGET <libfuzzer args>
-```
-
-`fuzz.py` respects the environment variables / flags `MSAN_EXTRA_CPPFLAGS`,
-`MSAN_EXTRA_CFLAGS`, `MSAN_EXTRA_CXXFLAGS`, `MSAN_EXTRA_LDFLAGS` to easily pass
-the extra parameters only for MSAN.
-
-## AFL
-
-The default `LIB_FUZZING_ENGINE` is `libregression.a`, which produces a binary
-that AFL can use.
-
-```
-# Build the fuzz targets
-CC=afl-clang CXX=afl-clang++ ./fuzz.py build all --enable-asan --enable-ubsan
-# Run the fuzzer without a memory limit because of ASAN
-./fuzz.py afl TARGET -m none
-```
-
-## Regression Testing
-
-The regression test supports the `all` target to run all the fuzzers in one
-command.
-
-```
-CC=clang CXX=clang++ ./fuzz.py build all --enable-asan --enable-ubsan
-./fuzz.py regression all
-CC=clang CXX=clang++ ./fuzz.py build all --enable-msan
-./fuzz.py regression all
-```
diff --git a/tests/fuzz/block_decompress.c b/tests/fuzz/block_decompress.c
deleted file mode 100644
index a904b44624d9..000000000000
--- a/tests/fuzz/block_decompress.c
+++ /dev/null
@@ -1,49 +0,0 @@
-/**
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-
-/**
- * This fuzz target attempts to decompress the fuzzed data with the simple
- * decompression function to ensure the decompressor never crashes.
- */
-
-#define ZSTD_STATIC_LINKING_ONLY
-
-#include <stddef.h>
-#include <stdlib.h>
-#include <stdio.h>
-#include "fuzz_helpers.h"
-#include "zstd.h"
-
-static ZSTD_DCtx *dctx = NULL;
-static void* rBuf = NULL;
-static size_t bufSize = 0;
-
-int LLVMFuzzerTestOneInput(const uint8_t *src, size_t size)
-{
-    size_t const neededBufSize = ZSTD_BLOCKSIZE_MAX;
-
-    /* Allocate all buffers and contexts if not already allocated */
-    if (neededBufSize > bufSize) {
-        free(rBuf);
-        rBuf = malloc(neededBufSize);
-        bufSize = neededBufSize;
-        FUZZ_ASSERT(rBuf);
-    }
-    if (!dctx) {
-        dctx = ZSTD_createDCtx();
-        FUZZ_ASSERT(dctx);
-    }
-    ZSTD_decompressBegin(dctx);
-    ZSTD_decompressBlock(dctx, rBuf, neededBufSize, src, size);
-
-#ifndef STATEFUL_FUZZING
-    ZSTD_freeDCtx(dctx); dctx = NULL;
-#endif
-    return 0;
-}
diff --git a/tests/fuzz/block_round_trip.c b/tests/fuzz/block_round_trip.c
deleted file mode 100644
index 89f060a6eb38..000000000000
--- a/tests/fuzz/block_round_trip.c
+++ /dev/null
@@ -1,97 +0,0 @@
-/**
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-
-/**
- * This fuzz target performs a zstd round-trip test (compress & decompress),
- * compares the result with the original, and calls abort() on corruption.
- */
-
-#define ZSTD_STATIC_LINKING_ONLY
-
-#include <stddef.h>
-#include <stdlib.h>
-#include <stdio.h>
-#include <string.h>
-#include "fuzz_helpers.h"
-#include "zstd.h"
-#include "zstd_helpers.h"
-#include "fuzz_data_producer.h"
-
-static ZSTD_CCtx *cctx = NULL;
-static ZSTD_DCtx *dctx = NULL;
-static void* cBuf = NULL;
-static void* rBuf = NULL;
-static size_t bufSize = 0;
-
-static size_t roundTripTest(void *result, size_t resultCapacity,
-                            void *compressed, size_t compressedCapacity,
-                            const void *src, size_t srcSize,
-                            int cLevel)
-{
-    ZSTD_parameters const params = ZSTD_getParams(cLevel, srcSize, 0);
-    size_t ret = ZSTD_compressBegin_advanced(cctx, NULL, 0, params, srcSize);
-    FUZZ_ZASSERT(ret);
-
-    ret = ZSTD_compressBlock(cctx, compressed, compressedCapacity, src, srcSize);
-    FUZZ_ZASSERT(ret);
-    if (ret == 0) {
-        FUZZ_ASSERT(resultCapacity >= srcSize);
-        memcpy(result, src, srcSize);
-        return srcSize;
-    }
-    ZSTD_decompressBegin(dctx);
-    return ZSTD_decompressBlock(dctx, result, resultCapacity, compressed, ret);
-}
-
-int LLVMFuzzerTestOneInput(const uint8_t *src, size_t size)
-{
-    /* Give a random portion of src data to the producer, to use for
-    parameter generation. The rest will be used for (de)compression */
-    FUZZ_dataProducer_t *producer = FUZZ_dataProducer_create(src, size);
-    size = FUZZ_dataProducer_reserveDataPrefix(producer);
-
-    int const cLevel = FUZZ_dataProducer_int32Range(producer, kMinClevel, kMaxClevel);
-
-    size_t neededBufSize = size;
-    if (size > ZSTD_BLOCKSIZE_MAX)
-        size = ZSTD_BLOCKSIZE_MAX;
-
-    /* Allocate all buffers and contexts if not already allocated */
-    if (neededBufSize > bufSize || !cBuf || !rBuf) {
-        free(cBuf);
-        free(rBuf);
-        cBuf = malloc(neededBufSize);
-        rBuf = malloc(neededBufSize);
-        bufSize = neededBufSize;
-        FUZZ_ASSERT(cBuf && rBuf);
-    }
-    if (!cctx) {
-        cctx = ZSTD_createCCtx();
-        FUZZ_ASSERT(cctx);
-    }
-    if (!dctx) {
-        dctx = ZSTD_createDCtx();
-        FUZZ_ASSERT(dctx);
-    }
-
-    {
-        size_t const result =
-            roundTripTest(rBuf, neededBufSize, cBuf, neededBufSize, src, size,
-              cLevel);
-        FUZZ_ZASSERT(result);
-        FUZZ_ASSERT_MSG(result == size, "Incorrect regenerated size");
-        FUZZ_ASSERT_MSG(!memcmp(src, rBuf, size), "Corruption!");
-    }
-    FUZZ_dataProducer_free(producer);
-#ifndef STATEFUL_FUZZING
-    ZSTD_freeCCtx(cctx); cctx = NULL;
-    ZSTD_freeDCtx(dctx); dctx = NULL;
-#endif
-    return 0;
-}
diff --git a/tests/fuzz/dictionary_decompress.c b/tests/fuzz/dictionary_decompress.c
deleted file mode 100644
index 9cc69fa37866..000000000000
--- a/tests/fuzz/dictionary_decompress.c
+++ /dev/null
@@ -1,68 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-
-/**
- * This fuzz target attempts to decompress the fuzzed data with the dictionary
- * decompression function to ensure the decompressor never crashes. It does not
- * fuzz the dictionary.
- */
-
-#include <stddef.h>
-#include <stdlib.h>
-#include <stdio.h>
-#include "fuzz_helpers.h"
-#include "zstd_helpers.h"
-#include "fuzz_data_producer.h"
-
-static ZSTD_DCtx *dctx = NULL;
-
-int LLVMFuzzerTestOneInput(const uint8_t *src, size_t size)
-{
-    /* Give a random portion of src data to the producer, to use for
-    parameter generation. The rest will be used for (de)compression */
-    FUZZ_dataProducer_t *producer = FUZZ_dataProducer_create(src, size);
-    size = FUZZ_dataProducer_reserveDataPrefix(producer);
-
-    FUZZ_dict_t dict;
-    ZSTD_DDict* ddict = NULL;
-
-    if (!dctx) {
-        dctx = ZSTD_createDCtx();
-        FUZZ_ASSERT(dctx);
-    }
-    dict = FUZZ_train(src, size, producer);
-    if (FUZZ_dataProducer_uint32Range(producer, 0, 1) == 0) {
-        ddict = ZSTD_createDDict(dict.buff, dict.size);
-        FUZZ_ASSERT(ddict);
-    } else {
-        FUZZ_ZASSERT(ZSTD_DCtx_loadDictionary_advanced(
-                dctx, dict.buff, dict.size,
-                (ZSTD_dictLoadMethod_e)FUZZ_dataProducer_uint32Range(producer, 0, 1),
-                (ZSTD_dictContentType_e)FUZZ_dataProducer_uint32Range(producer, 0, 2)));
-    }
-
-    {
-        size_t const bufSize = FUZZ_dataProducer_uint32Range(producer, 0, 10 * size);
-        void* rBuf = malloc(bufSize);
-        FUZZ_ASSERT(rBuf);
-        if (ddict) {
-            ZSTD_decompress_usingDDict(dctx, rBuf, bufSize, src, size, ddict);
-        } else {
-            ZSTD_decompressDCtx(dctx, rBuf, bufSize, src, size);
-        }
-        free(rBuf);
-    }
-    free(dict.buff);
-    FUZZ_dataProducer_free(producer);
-    ZSTD_freeDDict(ddict);
-#ifndef STATEFUL_FUZZING
-    ZSTD_freeDCtx(dctx); dctx = NULL;
-#endif
-    return 0;
-}
diff --git a/tests/fuzz/dictionary_loader.c b/tests/fuzz/dictionary_loader.c
deleted file mode 100644
index cb34f5d22d9a..000000000000
--- a/tests/fuzz/dictionary_loader.c
+++ /dev/null
@@ -1,93 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-
-/**
- * This fuzz target makes sure that whenever a compression dictionary can be
- * loaded, the data can be round tripped.
- */
-
-#include <stddef.h>
-#include <stdlib.h>
-#include <stdio.h>
-#include <string.h>
-#include "fuzz_helpers.h"
-#include "zstd_helpers.h"
-#include "fuzz_data_producer.h"
-
-/**
- * Compresses the data and returns the compressed size or an error.
- */
-static size_t compress(void* compressed, size_t compressedCapacity,
-                       void const* source, size_t sourceSize,
-                       void const* dict, size_t dictSize,
-                       ZSTD_dictLoadMethod_e dictLoadMethod,
-                       ZSTD_dictContentType_e dictContentType)
-{
-    ZSTD_CCtx* cctx = ZSTD_createCCtx();
-    FUZZ_ZASSERT(ZSTD_CCtx_loadDictionary_advanced(
-            cctx, dict, dictSize, dictLoadMethod, dictContentType));
-    size_t const compressedSize = ZSTD_compress2(
-            cctx, compressed, compressedCapacity, source, sourceSize);
-    ZSTD_freeCCtx(cctx);
-    return compressedSize;
-}
-
-static size_t decompress(void* result, size_t resultCapacity,
-                         void const* compressed, size_t compressedSize,
-                         void const* dict, size_t dictSize,
-                       ZSTD_dictLoadMethod_e dictLoadMethod,
-                         ZSTD_dictContentType_e dictContentType)
-{
-    ZSTD_DCtx* dctx = ZSTD_createDCtx();
-    FUZZ_ZASSERT(ZSTD_DCtx_loadDictionary_advanced(
-            dctx, dict, dictSize, dictLoadMethod, dictContentType));
-    size_t const resultSize = ZSTD_decompressDCtx(
-            dctx, result, resultCapacity, compressed, compressedSize);
-    FUZZ_ZASSERT(resultSize);
-    ZSTD_freeDCtx(dctx);
-    return resultSize;
-}
-
-int LLVMFuzzerTestOneInput(const uint8_t *src, size_t size)
-{
-    FUZZ_dataProducer_t *producer = FUZZ_dataProducer_create(src, size);
-    ZSTD_dictLoadMethod_e const dlm =
-    size = FUZZ_dataProducer_uint32Range(producer, 0, 1);
-    ZSTD_dictContentType_e const dct =
-            FUZZ_dataProducer_uint32Range(producer, 0, 2);
-    size = FUZZ_dataProducer_remainingBytes(producer);
-
-    DEBUGLOG(2, "Dict load method %d", dlm);
-    DEBUGLOG(2, "Dict content type %d", dct);
-    DEBUGLOG(2, "Dict size %u", (unsigned)size);
-
-    void* const rBuf = malloc(size);
-    FUZZ_ASSERT(rBuf);
-    size_t const cBufSize = ZSTD_compressBound(size);
-    void* const cBuf = malloc(cBufSize);
-    FUZZ_ASSERT(cBuf);
-
-    size_t const cSize =
-            compress(cBuf, cBufSize, src, size, src, size, dlm, dct);
-    /* compression failing is okay */
-    if (ZSTD_isError(cSize)) {
-      FUZZ_ASSERT_MSG(dct != ZSTD_dct_rawContent, "Raw must always succeed!");
-      goto out;
-    }
-    size_t const rSize =
-            decompress(rBuf, size, cBuf, cSize, src, size, dlm, dct);
-    FUZZ_ASSERT_MSG(rSize == size, "Incorrect regenerated size");
-    FUZZ_ASSERT_MSG(!memcmp(src, rBuf, size), "Corruption!");
-
-out:
-    free(cBuf);
-    free(rBuf);
-    FUZZ_dataProducer_free(producer);
-    return 0;
-}
diff --git a/tests/fuzz/dictionary_round_trip.c b/tests/fuzz/dictionary_round_trip.c
deleted file mode 100644
index 9411b50a74eb..000000000000
--- a/tests/fuzz/dictionary_round_trip.c
+++ /dev/null
@@ -1,109 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-
-/**
- * This fuzz target performs a zstd round-trip test (compress & decompress) with
- * a dictionary, compares the result with the original, and calls abort() on
- * corruption.
- */
-
-#include <stddef.h>
-#include <stdlib.h>
-#include <stdio.h>
-#include <string.h>
-#include "fuzz_helpers.h"
-#include "zstd_helpers.h"
-#include "fuzz_data_producer.h"
-
-static ZSTD_CCtx *cctx = NULL;
-static ZSTD_DCtx *dctx = NULL;
-
-static size_t roundTripTest(void *result, size_t resultCapacity,
-                            void *compressed, size_t compressedCapacity,
-                            const void *src, size_t srcSize,
-                            FUZZ_dataProducer_t *producer)
-{
-    ZSTD_dictContentType_e dictContentType = ZSTD_dct_auto;
-    FUZZ_dict_t dict = FUZZ_train(src, srcSize, producer);
-    size_t cSize;
-    if (FUZZ_dataProducer_uint32Range(producer, 0, 15) == 0) {
-        int const cLevel = FUZZ_dataProducer_int32Range(producer, kMinClevel, kMaxClevel);
-
-        cSize = ZSTD_compress_usingDict(cctx,
-                compressed, compressedCapacity,
-                src, srcSize,
-                dict.buff, dict.size,
-                cLevel);
-    } else {
-        dictContentType = FUZZ_dataProducer_uint32Range(producer, 0, 2);
-        FUZZ_setRandomParameters(cctx, srcSize, producer);
-        /* Disable checksum so we can use sizes smaller than compress bound. */
-        FUZZ_ZASSERT(ZSTD_CCtx_setParameter(cctx, ZSTD_c_checksumFlag, 0));
-        FUZZ_ZASSERT(ZSTD_CCtx_loadDictionary_advanced(
-                cctx, dict.buff, dict.size,
-                (ZSTD_dictLoadMethod_e)FUZZ_dataProducer_uint32Range(producer, 0, 1),
-                dictContentType));
-        cSize = ZSTD_compress2(cctx, compressed, compressedCapacity, src, srcSize);
-    }
-    FUZZ_ZASSERT(cSize);
-    FUZZ_ZASSERT(ZSTD_DCtx_loadDictionary_advanced(
-        dctx, dict.buff, dict.size,
-        (ZSTD_dictLoadMethod_e)FUZZ_dataProducer_uint32Range(producer, 0, 1),
-        dictContentType));
-    {
-        size_t const ret = ZSTD_decompressDCtx(
-                dctx, result, resultCapacity, compressed, cSize);
-        free(dict.buff);
-        return ret;
-    }
-}
-
-int LLVMFuzzerTestOneInput(const uint8_t *src, size_t size)
-{
-    /* Give a random portion of src data to the producer, to use for
-    parameter generation. The rest will be used for (de)compression */
-    FUZZ_dataProducer_t *producer = FUZZ_dataProducer_create(src, size);
-    size = FUZZ_dataProducer_reserveDataPrefix(producer);
-
-    size_t const rBufSize = size;
-    void* rBuf = malloc(rBufSize);
-    size_t cBufSize = ZSTD_compressBound(size);
-    void *cBuf;
-    /* Half of the time fuzz with a 1 byte smaller output size.
-     * This will still succeed because we force the checksum to be disabled,
-     * giving us 4 bytes of overhead.
-     */
-    cBufSize -= FUZZ_dataProducer_uint32Range(producer, 0, 1);
-    cBuf = malloc(cBufSize);
-
-    if (!cctx) {
-        cctx = ZSTD_createCCtx();
-        FUZZ_ASSERT(cctx);
-    }
-    if (!dctx) {
-        dctx = ZSTD_createDCtx();
-        FUZZ_ASSERT(dctx);
-    }
-
-    {
-        size_t const result =
-            roundTripTest(rBuf, rBufSize, cBuf, cBufSize, src, size, producer);
-        FUZZ_ZASSERT(result);
-        FUZZ_ASSERT_MSG(result == size, "Incorrect regenerated size");
-        FUZZ_ASSERT_MSG(!memcmp(src, rBuf, size), "Corruption!");
-    }
-    free(rBuf);
-    free(cBuf);
-    FUZZ_dataProducer_free(producer);
-#ifndef STATEFUL_FUZZING
-    ZSTD_freeCCtx(cctx); cctx = NULL;
-    ZSTD_freeDCtx(dctx); dctx = NULL;
-#endif
-    return 0;
-}
diff --git a/tests/fuzz/fuzz.h b/tests/fuzz/fuzz.h
deleted file mode 100644
index 6d53aa6d5c7f..000000000000
--- a/tests/fuzz/fuzz.h
+++ /dev/null
@@ -1,52 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-
-/**
- * Fuzz target interface.
- * Fuzz targets have some common parameters passed as macros during compilation.
- * Check the documentation for each individual fuzzer for more parameters.
- *
- * @param STATEFUL_FUZZING:
- *        Define this to reuse state between fuzzer runs. This can be useful to
- *        test code paths which are only executed when contexts are reused.
- *        WARNING: Makes reproducing crashes much harder.
- *        Default: Not defined.
- * @param DEBUGLEVEL:
- *        This is a parameter for the zstd library. Defining `DEBUGLEVEL=1`
- *        enables assert() statements in the zstd library. Higher levels enable
- *        logging, so aren't recommended. Defining `DEBUGLEVEL=1` is
- *        recommended.
- * @param MEM_FORCE_MEMORY_ACCESS:
- *        This flag controls how the zstd library accesses unaligned memory.
- *        It can be undefined, or 0 through 2. If it is undefined, it selects
- *        the method to use based on the compiler. If testing with UBSAN set
- *        MEM_FORCE_MEMORY_ACCESS=0 to use the standard compliant method.
- * @param FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
- *        This is the canonical flag to enable deterministic builds for fuzzing.
- *        Changes to zstd for fuzzing are gated behind this define.
- *        It is recommended to define this when building zstd for fuzzing.
- */
-
-#ifndef FUZZ_H
-#define FUZZ_H
-
-#include <stddef.h>
-#include <stdint.h>
-
-#ifdef __cplusplus
-extern "C" {
-#endif
-
-int LLVMFuzzerTestOneInput(const uint8_t *src, size_t size);
-
-#ifdef __cplusplus
-}
-#endif
-
-#endif
diff --git a/tests/fuzz/fuzz.py b/tests/fuzz/fuzz.py
deleted file mode 100755
index 87f115afde99..000000000000
--- a/tests/fuzz/fuzz.py
+++ /dev/null
@@ -1,884 +0,0 @@
-#!/usr/bin/env python
-
-# ################################################################
-# Copyright (c) 2016-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under both the BSD-style license (found in the
-# LICENSE file in the root directory of this source tree) and the GPLv2 (found
-# in the COPYING file in the root directory of this source tree).
-# ##########################################################################
-
-import argparse
-import contextlib
-import os
-import re
-import shlex
-import shutil
-import subprocess
-import sys
-import tempfile
-
-
-def abs_join(a, *p):
-    return os.path.abspath(os.path.join(a, *p))
-
-
-class InputType(object):
-    RAW_DATA = 1
-    COMPRESSED_DATA = 2
-    DICTIONARY_DATA = 3
-
-
-class FrameType(object):
-    ZSTD = 1
-    BLOCK = 2
-
-
-class TargetInfo(object):
-    def __init__(self, input_type, frame_type=FrameType.ZSTD):
-        self.input_type = input_type
-        self.frame_type = frame_type
-
-
-# Constants
-FUZZ_DIR = os.path.abspath(os.path.dirname(__file__))
-CORPORA_DIR = abs_join(FUZZ_DIR, 'corpora')
-TARGET_INFO = {
-    'simple_round_trip': TargetInfo(InputType.RAW_DATA),
-    'stream_round_trip': TargetInfo(InputType.RAW_DATA),
-    'block_round_trip': TargetInfo(InputType.RAW_DATA, FrameType.BLOCK),
-    'simple_decompress': TargetInfo(InputType.COMPRESSED_DATA),
-    'stream_decompress': TargetInfo(InputType.COMPRESSED_DATA),
-    'block_decompress': TargetInfo(InputType.COMPRESSED_DATA, FrameType.BLOCK),
-    'dictionary_round_trip': TargetInfo(InputType.RAW_DATA),
-    'dictionary_decompress': TargetInfo(InputType.COMPRESSED_DATA),
-    'zstd_frame_info': TargetInfo(InputType.COMPRESSED_DATA),
-    'simple_compress': TargetInfo(InputType.RAW_DATA),
-    'dictionary_loader': TargetInfo(InputType.DICTIONARY_DATA),
-}
-TARGETS = list(TARGET_INFO.keys())
-ALL_TARGETS = TARGETS + ['all']
-FUZZ_RNG_SEED_SIZE = 4
-
-# Standard environment variables
-CC = os.environ.get('CC', 'cc')
-CXX = os.environ.get('CXX', 'c++')
-CPPFLAGS = os.environ.get('CPPFLAGS', '')
-CFLAGS = os.environ.get('CFLAGS', '-O3')
-CXXFLAGS = os.environ.get('CXXFLAGS', CFLAGS)
-LDFLAGS = os.environ.get('LDFLAGS', '')
-MFLAGS = os.environ.get('MFLAGS', '-j')
-
-# Fuzzing environment variables
-LIB_FUZZING_ENGINE = os.environ.get('LIB_FUZZING_ENGINE', 'libregression.a')
-AFL_FUZZ = os.environ.get('AFL_FUZZ', 'afl-fuzz')
-DECODECORPUS = os.environ.get('DECODECORPUS',
-                              abs_join(FUZZ_DIR, '..', 'decodecorpus'))
-ZSTD = os.environ.get('ZSTD', abs_join(FUZZ_DIR, '..', '..', 'zstd'))
-
-# Sanitizer environment variables
-MSAN_EXTRA_CPPFLAGS = os.environ.get('MSAN_EXTRA_CPPFLAGS', '')
-MSAN_EXTRA_CFLAGS = os.environ.get('MSAN_EXTRA_CFLAGS', '')
-MSAN_EXTRA_CXXFLAGS = os.environ.get('MSAN_EXTRA_CXXFLAGS', '')
-MSAN_EXTRA_LDFLAGS = os.environ.get('MSAN_EXTRA_LDFLAGS', '')
-
-
-def create(r):
-    d = os.path.abspath(r)
-    if not os.path.isdir(d):
-        os.makedirs(d)
-    return d
-
-
-def check(r):
-    d = os.path.abspath(r)
-    if not os.path.isdir(d):
-        return None
-    return d
-
-
-@contextlib.contextmanager
-def tmpdir():
-    dirpath = tempfile.mkdtemp()
-    try:
-        yield dirpath
-    finally:
-        shutil.rmtree(dirpath, ignore_errors=True)
-
-
-def parse_targets(in_targets):
-    targets = set()
-    for target in in_targets:
-        if not target:
-            continue
-        if target == 'all':
-            targets = targets.union(TARGETS)
-        elif target in TARGETS:
-            targets.add(target)
-        else:
-            raise RuntimeError('{} is not a valid target'.format(target))
-    return list(targets)
-
-
-def targets_parser(args, description):
-    parser = argparse.ArgumentParser(prog=args.pop(0), description=description)
-    parser.add_argument(
-        'TARGET',
-        nargs='*',
-        type=str,
-        help='Fuzz target(s) to build {{{}}}'.format(', '.join(ALL_TARGETS)))
-    args, extra = parser.parse_known_args(args)
-    args.extra = extra
-
-    args.TARGET = parse_targets(args.TARGET)
-
-    return args
-
-
-def parse_env_flags(args, flags):
-    """
-    Look for flags set by environment variables.
-    """
-    san_flags = ','.join(re.findall('-fsanitize=((?:[a-z]+,?)+)', flags))
-    nosan_flags = ','.join(re.findall('-fno-sanitize=((?:[a-z]+,?)+)', flags))
-
-    def set_sanitizer(sanitizer, default, san, nosan):
-        if sanitizer in san and sanitizer in nosan:
-            raise RuntimeError('-fno-sanitize={s} and -fsanitize={s} passed'.
-                               format(s=sanitizer))
-        if sanitizer in san:
-            return True
-        if sanitizer in nosan:
-            return False
-        return default
-
-    san = set(san_flags.split(','))
-    nosan = set(nosan_flags.split(','))
-
-    args.asan = set_sanitizer('address', args.asan, san, nosan)
-    args.msan = set_sanitizer('memory', args.msan, san, nosan)
-    args.ubsan = set_sanitizer('undefined', args.ubsan, san, nosan)
-
-    args.sanitize = args.asan or args.msan or args.ubsan
-
-    return args
-
-
-def compiler_version(cc, cxx):
-    """
-    Determines the compiler and version.
-    Only works for clang and gcc.
-    """
-    cc_version_bytes = subprocess.check_output([cc, "--version"])
-    cxx_version_bytes = subprocess.check_output([cxx, "--version"])
-    compiler = None
-    version = None
-    if b'clang' in cc_version_bytes:
-        assert(b'clang' in cxx_version_bytes)
-        compiler = 'clang'
-    elif b'gcc' in cc_version_bytes:
-        assert(b'gcc' in cxx_version_bytes or b'g++' in cxx_version_bytes)
-        compiler = 'gcc'
-    if compiler is not None:
-        version_regex = b'([0-9])+\.([0-9])+\.([0-9])+'
-        version_match = re.search(version_regex, cc_version_bytes)
-        version = tuple(int(version_match.group(i)) for i in range(1, 4))
-    return compiler, version
-
-
-def overflow_ubsan_flags(cc, cxx):
-    compiler, version = compiler_version(cc, cxx)
-    if compiler == 'gcc':
-        return ['-fno-sanitize=signed-integer-overflow']
-    if compiler == 'clang' and version >= (5, 0, 0):
-        return ['-fno-sanitize=pointer-overflow']
-    return []
-
-
-def build_parser(args):
-    description = """
-    Cleans the repository and builds a fuzz target (or all).
-    Many flags default to environment variables (default says $X='y').
-    Options that aren't enabling features default to the correct values for
-    zstd.
-    Enable sanitizers with --enable-*san.
-    For regression testing just build.
-    For libFuzzer set LIB_FUZZING_ENGINE and pass --enable-coverage.
-    For AFL set CC and CXX to AFL's compilers and set
-    LIB_FUZZING_ENGINE='libregression.a'.
-    """
-    parser = argparse.ArgumentParser(prog=args.pop(0), description=description)
-    parser.add_argument(
-        '--lib-fuzzing-engine',
-        dest='lib_fuzzing_engine',
-        type=str,
-        default=LIB_FUZZING_ENGINE,
-        help=('The fuzzing engine to use e.g. /path/to/libFuzzer.a '
-              "(default: $LIB_FUZZING_ENGINE='{})".format(LIB_FUZZING_ENGINE)))
-
-    fuzz_group = parser.add_mutually_exclusive_group()
-    fuzz_group.add_argument(
-        '--enable-coverage',
-        dest='coverage',
-        action='store_true',
-        help='Enable coverage instrumentation (-fsanitize-coverage)')
-    fuzz_group.add_argument(
-        '--enable-fuzzer',
-        dest='fuzzer',
-        action='store_true',
-        help=('Enable clang fuzzer (-fsanitize=fuzzer). When enabled '
-              'LIB_FUZZING_ENGINE is ignored')
-    )
-
-    parser.add_argument(
-        '--enable-asan', dest='asan', action='store_true', help='Enable UBSAN')
-    parser.add_argument(
-        '--enable-ubsan',
-        dest='ubsan',
-        action='store_true',
-        help='Enable UBSAN')
-    parser.add_argument(
-        '--enable-ubsan-pointer-overflow',
-        dest='ubsan_pointer_overflow',
-        action='store_true',
-        help='Enable UBSAN pointer overflow check (known failure)')
-    parser.add_argument(
-        '--enable-msan', dest='msan', action='store_true', help='Enable MSAN')
-    parser.add_argument(
-        '--enable-msan-track-origins', dest='msan_track_origins',
-        action='store_true', help='Enable MSAN origin tracking')
-    parser.add_argument(
-        '--msan-extra-cppflags',
-        dest='msan_extra_cppflags',
-        type=str,
-        default=MSAN_EXTRA_CPPFLAGS,
-        help="Extra CPPFLAGS for MSAN (default: $MSAN_EXTRA_CPPFLAGS='{}')".
-        format(MSAN_EXTRA_CPPFLAGS))
-    parser.add_argument(
-        '--msan-extra-cflags',
-        dest='msan_extra_cflags',
-        type=str,
-        default=MSAN_EXTRA_CFLAGS,
-        help="Extra CFLAGS for MSAN (default: $MSAN_EXTRA_CFLAGS='{}')".format(
-            MSAN_EXTRA_CFLAGS))
-    parser.add_argument(
-        '--msan-extra-cxxflags',
-        dest='msan_extra_cxxflags',
-        type=str,
-        default=MSAN_EXTRA_CXXFLAGS,
-        help="Extra CXXFLAGS for MSAN (default: $MSAN_EXTRA_CXXFLAGS='{}')".
-        format(MSAN_EXTRA_CXXFLAGS))
-    parser.add_argument(
-        '--msan-extra-ldflags',
-        dest='msan_extra_ldflags',
-        type=str,
-        default=MSAN_EXTRA_LDFLAGS,
-        help="Extra LDFLAGS for MSAN (default: $MSAN_EXTRA_LDFLAGS='{}')".
-        format(MSAN_EXTRA_LDFLAGS))
-    parser.add_argument(
-        '--enable-sanitize-recover',
-        dest='sanitize_recover',
-        action='store_true',
-        help='Non-fatal sanitizer errors where possible')
-    parser.add_argument(
-        '--debug',
-        dest='debug',
-        type=int,
-        default=1,
-        help='Set DEBUGLEVEL (default: 1)')
-    parser.add_argument(
-        '--force-memory-access',
-        dest='memory_access',
-        type=int,
-        default=0,
-        help='Set MEM_FORCE_MEMORY_ACCESS (default: 0)')
-    parser.add_argument(
-        '--fuzz-rng-seed-size',
-        dest='fuzz_rng_seed_size',
-        type=int,
-        default=4,
-        help='Set FUZZ_RNG_SEED_SIZE (default: 4)')
-    parser.add_argument(
-        '--disable-fuzzing-mode',
-        dest='fuzzing_mode',
-        action='store_false',
-        help='Do not define FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION')
-    parser.add_argument(
-        '--enable-stateful-fuzzing',
-        dest='stateful_fuzzing',
-        action='store_true',
-        help='Reuse contexts between runs (makes reproduction impossible)')
-    parser.add_argument(
-        '--cc',
-        dest='cc',
-        type=str,
-        default=CC,
-        help="CC (default: $CC='{}')".format(CC))
-    parser.add_argument(
-        '--cxx',
-        dest='cxx',
-        type=str,
-        default=CXX,
-        help="CXX (default: $CXX='{}')".format(CXX))
-    parser.add_argument(
-        '--cppflags',
-        dest='cppflags',
-        type=str,
-        default=CPPFLAGS,
-        help="CPPFLAGS (default: $CPPFLAGS='{}')".format(CPPFLAGS))
-    parser.add_argument(
-        '--cflags',
-        dest='cflags',
-        type=str,
-        default=CFLAGS,
-        help="CFLAGS (default: $CFLAGS='{}')".format(CFLAGS))
-    parser.add_argument(
-        '--cxxflags',
-        dest='cxxflags',
-        type=str,
-        default=CXXFLAGS,
-        help="CXXFLAGS (default: $CXXFLAGS='{}')".format(CXXFLAGS))
-    parser.add_argument(
-        '--ldflags',
-        dest='ldflags',
-        type=str,
-        default=LDFLAGS,
-        help="LDFLAGS (default: $LDFLAGS='{}')".format(LDFLAGS))
-    parser.add_argument(
-        '--mflags',
-        dest='mflags',
-        type=str,
-        default=MFLAGS,
-        help="Extra Make flags (default: $MFLAGS='{}')".format(MFLAGS))
-    parser.add_argument(
-        'TARGET',
-        nargs='*',
-        type=str,
-        help='Fuzz target(s) to build {{{}}}'.format(', '.join(ALL_TARGETS))
-    )
-    args = parser.parse_args(args)
-    args = parse_env_flags(args, ' '.join(
-        [args.cppflags, args.cflags, args.cxxflags, args.ldflags]))
-
-    # Check option sanity
-    if args.msan and (args.asan or args.ubsan):
-        raise RuntimeError('MSAN may not be used with any other sanitizers')
-    if args.msan_track_origins and not args.msan:
-        raise RuntimeError('--enable-msan-track-origins requires MSAN')
-    if args.ubsan_pointer_overflow and not args.ubsan:
-        raise RuntimeError('--enable-ubsan-pointer-overflow requires UBSAN')
-    if args.sanitize_recover and not args.sanitize:
-        raise RuntimeError('--enable-sanitize-recover but no sanitizers used')
-
-    return args
-
-
-def build(args):
-    try:
-        args = build_parser(args)
-    except Exception as e:
-        print(e)
-        return 1
-    # The compilation flags we are setting
-    targets = args.TARGET
-    cc = args.cc
-    cxx = args.cxx
-    cppflags = shlex.split(args.cppflags)
-    cflags = shlex.split(args.cflags)
-    ldflags = shlex.split(args.ldflags)
-    cxxflags = shlex.split(args.cxxflags)
-    mflags = shlex.split(args.mflags)
-    # Flags to be added to both cflags and cxxflags
-    common_flags = []
-
-    cppflags += [
-        '-DDEBUGLEVEL={}'.format(args.debug),
-        '-DMEM_FORCE_MEMORY_ACCESS={}'.format(args.memory_access),
-        '-DFUZZ_RNG_SEED_SIZE={}'.format(args.fuzz_rng_seed_size),
-    ]
-
-    # Set flags for options
-    assert not (args.fuzzer and args.coverage)
-    if args.coverage:
-        common_flags += [
-            '-fsanitize-coverage=trace-pc-guard,indirect-calls,trace-cmp'
-        ]
-    if args.fuzzer:
-        common_flags += ['-fsanitize=fuzzer']
-        args.lib_fuzzing_engine = ''
-
-    mflags += ['LIB_FUZZING_ENGINE={}'.format(args.lib_fuzzing_engine)]
-
-    if args.sanitize_recover:
-        recover_flags = ['-fsanitize-recover=all']
-    else:
-        recover_flags = ['-fno-sanitize-recover=all']
-    if args.sanitize:
-        common_flags += recover_flags
-
-    if args.msan:
-        msan_flags = ['-fsanitize=memory']
-        if args.msan_track_origins:
-            msan_flags += ['-fsanitize-memory-track-origins']
-        common_flags += msan_flags
-        # Append extra MSAN flags (it might require special setup)
-        cppflags += [args.msan_extra_cppflags]
-        cflags += [args.msan_extra_cflags]
-        cxxflags += [args.msan_extra_cxxflags]
-        ldflags += [args.msan_extra_ldflags]
-
-    if args.asan:
-        common_flags += ['-fsanitize=address']
-
-    if args.ubsan:
-        ubsan_flags = ['-fsanitize=undefined']
-        if not args.ubsan_pointer_overflow:
-            ubsan_flags += overflow_ubsan_flags(cc, cxx)
-        common_flags += ubsan_flags
-
-    if args.stateful_fuzzing:
-        cppflags += ['-DSTATEFUL_FUZZING']
-
-    if args.fuzzing_mode:
-        cppflags += ['-DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION']
-
-    if args.lib_fuzzing_engine == 'libregression.a':
-        targets = ['libregression.a'] + targets
-
-    # Append the common flags
-    cflags += common_flags
-    cxxflags += common_flags
-
-    # Prepare the flags for Make
-    cc_str = "CC={}".format(cc)
-    cxx_str = "CXX={}".format(cxx)
-    cppflags_str = "CPPFLAGS={}".format(' '.join(cppflags))
-    cflags_str = "CFLAGS={}".format(' '.join(cflags))
-    cxxflags_str = "CXXFLAGS={}".format(' '.join(cxxflags))
-    ldflags_str = "LDFLAGS={}".format(' '.join(ldflags))
-
-    # Print the flags
-    print('MFLAGS={}'.format(' '.join(mflags)))
-    print(cc_str)
-    print(cxx_str)
-    print(cppflags_str)
-    print(cflags_str)
-    print(cxxflags_str)
-    print(ldflags_str)
-
-    # Clean and build
-    clean_cmd = ['make', 'clean'] + mflags
-    print(' '.join(clean_cmd))
-    subprocess.check_call(clean_cmd)
-    build_cmd = [
-        'make',
-        cc_str,
-        cxx_str,
-        cppflags_str,
-        cflags_str,
-        cxxflags_str,
-        ldflags_str,
-    ] + mflags + targets
-    print(' '.join(build_cmd))
-    subprocess.check_call(build_cmd)
-    return 0
-
-
-def libfuzzer_parser(args):
-    description = """
-    Runs a libfuzzer binary.
-    Passes all extra arguments to libfuzzer.
-    The fuzzer should have been build with LIB_FUZZING_ENGINE pointing to
-    libFuzzer.a.
-    Generates output in the CORPORA directory, puts crashes in the ARTIFACT
-    directory, and takes extra input from the SEED directory.
-    To merge AFL's output pass the SEED as AFL's output directory and pass
-    '-merge=1'.
-    """
-    parser = argparse.ArgumentParser(prog=args.pop(0), description=description)
-    parser.add_argument(
-        '--corpora',
-        type=str,
-        help='Override the default corpora dir (default: {})'.format(
-            abs_join(CORPORA_DIR, 'TARGET')))
-    parser.add_argument(
-        '--artifact',
-        type=str,
-        help='Override the default artifact dir (default: {})'.format(
-            abs_join(CORPORA_DIR, 'TARGET-crash')))
-    parser.add_argument(
-        '--seed',
-        type=str,
-        help='Override the default seed dir (default: {})'.format(
-            abs_join(CORPORA_DIR, 'TARGET-seed')))
-    parser.add_argument(
-        'TARGET',
-        type=str,
-        help='Fuzz target(s) to build {{{}}}'.format(', '.join(TARGETS)))
-    args, extra = parser.parse_known_args(args)
-    args.extra = extra
-
-    if args.TARGET and args.TARGET not in TARGETS:
-        raise RuntimeError('{} is not a valid target'.format(args.TARGET))
-
-    return args
-
-
-def libfuzzer(target, corpora=None, artifact=None, seed=None, extra_args=None):
-    if corpora is None:
-        corpora = abs_join(CORPORA_DIR, target)
-    if artifact is None:
-        artifact = abs_join(CORPORA_DIR, '{}-crash'.format(target))
-    if seed is None:
-        seed = abs_join(CORPORA_DIR, '{}-seed'.format(target))
-    if extra_args is None:
-        extra_args = []
-
-    target = abs_join(FUZZ_DIR, target)
-
-    corpora = [create(corpora)]
-    artifact = create(artifact)
-    seed = check(seed)
-
-    corpora += [artifact]
-    if seed is not None:
-        corpora += [seed]
-
-    cmd = [target, '-artifact_prefix={}/'.format(artifact)]
-    cmd += corpora + extra_args
-    print(' '.join(cmd))
-    subprocess.check_call(cmd)
-
-
-def libfuzzer_cmd(args):
-    try:
-        args = libfuzzer_parser(args)
-    except Exception as e:
-        print(e)
-        return 1
-    libfuzzer(args.TARGET, args.corpora, args.artifact, args.seed, args.extra)
-    return 0
-
-
-def afl_parser(args):
-    description = """
-    Runs an afl-fuzz job.
-    Passes all extra arguments to afl-fuzz.
-    The fuzzer should have been built with CC/CXX set to the AFL compilers,
-    and with LIB_FUZZING_ENGINE='libregression.a'.
-    Takes input from CORPORA and writes output to OUTPUT.
-    Uses AFL_FUZZ as the binary (set from flag or environment variable).
-    """
-    parser = argparse.ArgumentParser(prog=args.pop(0), description=description)
-    parser.add_argument(
-        '--corpora',
-        type=str,
-        help='Override the default corpora dir (default: {})'.format(
-            abs_join(CORPORA_DIR, 'TARGET')))
-    parser.add_argument(
-        '--output',
-        type=str,
-        help='Override the default AFL output dir (default: {})'.format(
-            abs_join(CORPORA_DIR, 'TARGET-afl')))
-    parser.add_argument(
-        '--afl-fuzz',
-        type=str,
-        default=AFL_FUZZ,
-        help='AFL_FUZZ (default: $AFL_FUZZ={})'.format(AFL_FUZZ))
-    parser.add_argument(
-        'TARGET',
-        type=str,
-        help='Fuzz target(s) to build {{{}}}'.format(', '.join(TARGETS)))
-    args, extra = parser.parse_known_args(args)
-    args.extra = extra
-
-    if args.TARGET and args.TARGET not in TARGETS:
-        raise RuntimeError('{} is not a valid target'.format(args.TARGET))
-
-    if not args.corpora:
-        args.corpora = abs_join(CORPORA_DIR, args.TARGET)
-    if not args.output:
-        args.output = abs_join(CORPORA_DIR, '{}-afl'.format(args.TARGET))
-
-    return args
-
-
-def afl(args):
-    try:
-        args = afl_parser(args)
-    except Exception as e:
-        print(e)
-        return 1
-    target = abs_join(FUZZ_DIR, args.TARGET)
-
-    corpora = create(args.corpora)
-    output = create(args.output)
-
-    cmd = [args.afl_fuzz, '-i', corpora, '-o', output] + args.extra
-    cmd += [target, '@@']
-    print(' '.join(cmd))
-    subprocess.call(cmd)
-    return 0
-
-
-def regression(args):
-    try:
-        description = """
-        Runs one or more regression tests.
-        The fuzzer should have been built with with
-        LIB_FUZZING_ENGINE='libregression.a'.
-        Takes input from CORPORA.
-        """
-        args = targets_parser(args, description)
-    except Exception as e:
-        print(e)
-        return 1
-    for target in args.TARGET:
-        corpora = create(abs_join(CORPORA_DIR, target))
-        target = abs_join(FUZZ_DIR, target)
-        cmd = [target, corpora]
-        print(' '.join(cmd))
-        subprocess.check_call(cmd)
-    return 0
-
-
-def gen_parser(args):
-    description = """
-    Generate a seed corpus appropriate for TARGET with data generated with
-    decodecorpus.
-    The fuzz inputs are prepended with a seed before the zstd data, so the
-    output of decodecorpus shouldn't be used directly.
-    Generates NUMBER samples prepended with FUZZ_RNG_SEED_SIZE random bytes and
-    puts the output in SEED.
-    DECODECORPUS is the decodecorpus binary, and must already be built.
-    """
-    parser = argparse.ArgumentParser(prog=args.pop(0), description=description)
-    parser.add_argument(
-        '--number',
-        '-n',
-        type=int,
-        default=100,
-        help='Number of samples to generate')
-    parser.add_argument(
-        '--max-size-log',
-        type=int,
-        default=18,
-        help='Maximum sample size to generate')
-    parser.add_argument(
-        '--seed',
-        type=str,
-        help='Override the default seed dir (default: {})'.format(
-            abs_join(CORPORA_DIR, 'TARGET-seed')))
-    parser.add_argument(
-        '--decodecorpus',
-        type=str,
-        default=DECODECORPUS,
-        help="decodecorpus binary (default: $DECODECORPUS='{}')".format(
-            DECODECORPUS))
-    parser.add_argument(
-        '--zstd',
-        type=str,
-        default=ZSTD,
-        help="zstd binary (default: $ZSTD='{}')".format(ZSTD))
-    parser.add_argument(
-        '--fuzz-rng-seed-size',
-        type=int,
-        default=4,
-        help="FUZZ_RNG_SEED_SIZE used for generate the samples (must match)"
-    )
-    parser.add_argument(
-        'TARGET',
-        type=str,
-        help='Fuzz target(s) to build {{{}}}'.format(', '.join(TARGETS)))
-    args, extra = parser.parse_known_args(args)
-    args.extra = extra
-
-    if args.TARGET and args.TARGET not in TARGETS:
-        raise RuntimeError('{} is not a valid target'.format(args.TARGET))
-
-    if not args.seed:
-        args.seed = abs_join(CORPORA_DIR, '{}-seed'.format(args.TARGET))
-
-    if not os.path.isfile(args.decodecorpus):
-        raise RuntimeError("{} is not a file run 'make -C {} decodecorpus'".
-                           format(args.decodecorpus, abs_join(FUZZ_DIR, '..')))
-
-    return args
-
-
-def gen(args):
-    try:
-        args = gen_parser(args)
-    except Exception as e:
-        print(e)
-        return 1
-
-    seed = create(args.seed)
-    with tmpdir() as compressed, tmpdir() as decompressed, tmpdir() as dict:
-        info = TARGET_INFO[args.TARGET]
-
-        if info.input_type == InputType.DICTIONARY_DATA:
-            number = max(args.number, 1000)
-        else:
-            number = args.number
-        cmd = [
-            args.decodecorpus,
-            '-n{}'.format(args.number),
-            '-p{}/'.format(compressed),
-            '-o{}'.format(decompressed),
-        ]
-
-        if info.frame_type == FrameType.BLOCK:
-            cmd += [
-                '--gen-blocks',
-                '--max-block-size-log={}'.format(min(args.max_size_log, 17))
-            ]
-        else:
-            cmd += ['--max-content-size-log={}'.format(args.max_size_log)]
-
-        print(' '.join(cmd))
-        subprocess.check_call(cmd)
-
-        if info.input_type == InputType.RAW_DATA:
-            print('using decompressed data in {}'.format(decompressed))
-            samples = decompressed
-        elif info.input_type == InputType.COMPRESSED_DATA:
-            print('using compressed data in {}'.format(compressed))
-            samples = compressed
-        else:
-            assert info.input_type == InputType.DICTIONARY_DATA
-            print('making dictionary data from {}'.format(decompressed))
-            samples = dict
-            min_dict_size_log = 9
-            max_dict_size_log = max(min_dict_size_log + 1, args.max_size_log)
-            for dict_size_log in range(min_dict_size_log, max_dict_size_log):
-                dict_size = 1 << dict_size_log
-                cmd = [
-                    args.zstd,
-                    '--train',
-                    '-r', decompressed,
-                    '--maxdict={}'.format(dict_size),
-                    '-o', abs_join(dict, '{}.zstd-dict'.format(dict_size))
-                ]
-                print(' '.join(cmd))
-                subprocess.check_call(cmd)
-
-        # Copy the samples over and prepend the RNG seeds
-        for name in os.listdir(samples):
-            samplename = abs_join(samples, name)
-            outname = abs_join(seed, name)
-            with open(samplename, 'rb') as sample:
-                with open(outname, 'wb') as out:
-                    CHUNK_SIZE = 131072
-                    chunk = sample.read(CHUNK_SIZE)
-                    while len(chunk) > 0:
-                        out.write(chunk)
-                        chunk = sample.read(CHUNK_SIZE)
-    return 0
-
-
-def minimize(args):
-    try:
-        description = """
-        Runs a libfuzzer fuzzer with -merge=1 to build a minimal corpus in
-        TARGET_seed_corpus. All extra args are passed to libfuzzer.
-        """
-        args = targets_parser(args, description)
-    except Exception as e:
-        print(e)
-        return 1
-
-    for target in args.TARGET:
-        # Merge the corpus + anything else into the seed_corpus
-        corpus = abs_join(CORPORA_DIR, target)
-        seed_corpus = abs_join(CORPORA_DIR, "{}_seed_corpus".format(target))
-        extra_args = [corpus, "-merge=1"] + args.extra
-        libfuzzer(target, corpora=seed_corpus, extra_args=extra_args)
-        seeds = set(os.listdir(seed_corpus))
-        # Copy all crashes directly into the seed_corpus if not already present
-        crashes = abs_join(CORPORA_DIR, '{}-crash'.format(target))
-        for crash in os.listdir(crashes):
-            if crash not in seeds:
-                shutil.copy(abs_join(crashes, crash), seed_corpus)
-                seeds.add(crash)
-
-
-def zip_cmd(args):
-    try:
-        description = """
-        Zips up the seed corpus.
-        """
-        args = targets_parser(args, description)
-    except Exception as e:
-        print(e)
-        return 1
-
-    for target in args.TARGET:
-        # Zip the seed_corpus
-        seed_corpus = abs_join(CORPORA_DIR, "{}_seed_corpus".format(target))
-        zip_file = "{}.zip".format(seed_corpus)
-        cmd = ["zip", "-r", "-q", "-j", "-9", zip_file, "."]
-        print(' '.join(cmd))
-        subprocess.check_call(cmd, cwd=seed_corpus)
-
-
-def list_cmd(args):
-    print("\n".join(TARGETS))
-
-
-def short_help(args):
-    name = args[0]
-    print("Usage: {} [OPTIONS] COMMAND [ARGS]...\n".format(name))
-
-
-def help(args):
-    short_help(args)
-    print("\tfuzzing helpers (select a command and pass -h for help)\n")
-    print("Options:")
-    print("\t-h, --help\tPrint this message")
-    print("")
-    print("Commands:")
-    print("\tbuild\t\tBuild a fuzzer")
-    print("\tlibfuzzer\tRun a libFuzzer fuzzer")
-    print("\tafl\t\tRun an AFL fuzzer")
-    print("\tregression\tRun a regression test")
-    print("\tgen\t\tGenerate a seed corpus for a fuzzer")
-    print("\tminimize\tMinimize the test corpora")
-    print("\tzip\t\tZip the minimized corpora up")
-    print("\tlist\t\tList the available targets")
-
-
-def main():
-    args = sys.argv
-    if len(args) < 2:
-        help(args)
-        return 1
-    if args[1] == '-h' or args[1] == '--help' or args[1] == '-H':
-        help(args)
-        return 1
-    command = args.pop(1)
-    args[0] = "{} {}".format(args[0], command)
-    if command == "build":
-        return build(args)
-    if command == "libfuzzer":
-        return libfuzzer_cmd(args)
-    if command == "regression":
-        return regression(args)
-    if command == "afl":
-        return afl(args)
-    if command == "gen":
-        return gen(args)
-    if command == "minimize":
-        return minimize(args)
-    if command == "zip":
-        return zip_cmd(args)
-    if command == "list":
-        return list_cmd(args)
-    short_help(args)
-    print("Error: No such command {} (pass -h for help)".format(command))
-    return 1
-
-
-if __name__ == "__main__":
-    sys.exit(main())
diff --git a/tests/fuzz/fuzz_data_producer.c b/tests/fuzz/fuzz_data_producer.c
deleted file mode 100644
index b465337eac3b..000000000000
--- a/tests/fuzz/fuzz_data_producer.c
+++ /dev/null
@@ -1,85 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-
-#include "fuzz_data_producer.h"
-
-struct FUZZ_dataProducer_s{
-  const uint8_t *data;
-  size_t size;
-};
-
-FUZZ_dataProducer_t *FUZZ_dataProducer_create(const uint8_t *data, size_t size) {
-    FUZZ_dataProducer_t *producer = malloc(sizeof(FUZZ_dataProducer_t));
-
-    FUZZ_ASSERT(producer != NULL);
-
-    producer->data = data;
-    producer->size = size;
-    return producer;
-}
-
-void FUZZ_dataProducer_free(FUZZ_dataProducer_t *producer) { free(producer); }
-
-uint32_t FUZZ_dataProducer_uint32Range(FUZZ_dataProducer_t *producer, uint32_t min,
-                                  uint32_t max) {
-    FUZZ_ASSERT(min <= max);
-
-    uint32_t range = max - min;
-    uint32_t rolling = range;
-    uint32_t result = 0;
-
-    while (rolling > 0 && producer->size > 0) {
-      uint8_t next = *(producer->data + producer->size - 1);
-      producer->size -= 1;
-      result = (result << 8) | next;
-      rolling >>= 8;
-    }
-
-    if (range == 0xffffffff) {
-      return result;
-    }
-
-    return min + result % (range + 1);
-}
-
-uint32_t FUZZ_dataProducer_uint32(FUZZ_dataProducer_t *producer) {
-    return FUZZ_dataProducer_uint32Range(producer, 0, 0xffffffff);
-}
-
-int32_t FUZZ_dataProducer_int32Range(FUZZ_dataProducer_t *producer,
-                                    int32_t min, int32_t max)
-{
-    FUZZ_ASSERT(min <= max);
-
-    if (min < 0)
-      return (int)FUZZ_dataProducer_uint32Range(producer, 0, max - min) + min;
-
-    return FUZZ_dataProducer_uint32Range(producer, min, max);
-}
-
-size_t FUZZ_dataProducer_remainingBytes(FUZZ_dataProducer_t *producer){
-    return producer->size;
-}
-
-size_t FUZZ_dataProducer_contract(FUZZ_dataProducer_t *producer, size_t newSize)
-{
-    newSize = newSize > producer->size ? producer->size : newSize;
-
-    size_t remaining = producer->size - newSize;
-    producer->data = producer->data + remaining;
-    producer->size = newSize;
-    return remaining;
-}
-
-size_t FUZZ_dataProducer_reserveDataPrefix(FUZZ_dataProducer_t *producer)
-{
-    size_t producerSliceSize = FUZZ_dataProducer_uint32Range(
-                                  producer, 0, producer->size);
-    return FUZZ_dataProducer_contract(producer, producerSliceSize);
-}
diff --git a/tests/fuzz/fuzz_data_producer.h b/tests/fuzz/fuzz_data_producer.h
deleted file mode 100644
index f2b609677d7c..000000000000
--- a/tests/fuzz/fuzz_data_producer.h
+++ /dev/null
@@ -1,60 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-
-/**
- * Helper APIs for generating random data from input data stream.
- The producer reads bytes from the end of the input and appends them together
- to generate  a random number in the requested range. If it runs out of input
- data, it will keep returning the same value (min) over and over again.
-
- */
-
-#ifndef FUZZ_DATA_PRODUCER_H
-#define FUZZ_DATA_PRODUCER_H
-
-#include <stddef.h>
-#include <stdint.h>
-#include <stdio.h>
-#include <stdlib.h>
-
-#include "fuzz_helpers.h"
-
-/* Struct used for maintaining the state of the data */
-typedef struct FUZZ_dataProducer_s FUZZ_dataProducer_t;
-
-/* Returns a data producer state struct. Use for producer initialization. */
-FUZZ_dataProducer_t *FUZZ_dataProducer_create(const uint8_t *data, size_t size);
-
-/* Frees the data producer */
-void FUZZ_dataProducer_free(FUZZ_dataProducer_t *producer);
-
-/* Returns value between [min, max] */
-uint32_t FUZZ_dataProducer_uint32Range(FUZZ_dataProducer_t *producer, uint32_t min,
-                                  uint32_t max);
-
-/* Returns a uint32 value */
-uint32_t FUZZ_dataProducer_uint32(FUZZ_dataProducer_t *producer);
-
-/* Returns a signed value between [min, max] */
-int32_t FUZZ_dataProducer_int32Range(FUZZ_dataProducer_t *producer,
-                                    int32_t min, int32_t max);
-
-/* Returns the size of the remaining bytes of data in the producer */
-size_t FUZZ_dataProducer_remainingBytes(FUZZ_dataProducer_t *producer);
-
-/* Restricts the producer to only the last newSize bytes of data.
-If newSize > current data size, nothing happens. Returns the number of bytes
-the producer won't use anymore, after contracting. */
-size_t FUZZ_dataProducer_contract(FUZZ_dataProducer_t *producer, size_t newSize);
-
-/* Restricts the producer to use only the last X bytes of data, where X is
- a random number in the interval [0, data_size]. Returns the size of the
- remaining data the producer won't use anymore (the prefix). */
-size_t FUZZ_dataProducer_reserveDataPrefix(FUZZ_dataProducer_t *producer);
-#endif // FUZZ_DATA_PRODUCER_H
diff --git a/tests/fuzz/fuzz_helpers.h b/tests/fuzz/fuzz_helpers.h
deleted file mode 100644
index 3de917fd1263..000000000000
--- a/tests/fuzz/fuzz_helpers.h
+++ /dev/null
@@ -1,62 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-
-/**
- * Helper functions for fuzzing.
- */
-
-#ifndef FUZZ_HELPERS_H
-#define FUZZ_HELPERS_H
-
-#include "debug.h"
-#include "fuzz.h"
-#include "xxhash.h"
-#include "zstd.h"
-#include <stdint.h>
-#include <stdio.h>
-#include <stdlib.h>
-
-#ifdef __cplusplus
-extern "C" {
-#endif
-
-#define MIN(a, b) ((a) < (b) ? (a) : (b))
-#define MAX(a, b) ((a) > (b) ? (a) : (b))
-
-#define FUZZ_QUOTE_IMPL(str) #str
-#define FUZZ_QUOTE(str) FUZZ_QUOTE_IMPL(str)
-
-/**
- * Asserts for fuzzing that are always enabled.
- */
-#define FUZZ_ASSERT_MSG(cond, msg)                                             \
-  ((cond) ? (void)0                                                            \
-          : (fprintf(stderr, "%s: %u: Assertion: `%s' failed. %s\n", __FILE__, \
-                     __LINE__, FUZZ_QUOTE(cond), (msg)),                       \
-             abort()))
-#define FUZZ_ASSERT(cond) FUZZ_ASSERT_MSG((cond), "");
-#define FUZZ_ZASSERT(code)                                                     \
-  FUZZ_ASSERT_MSG(!ZSTD_isError(code), ZSTD_getErrorName(code))
-
-#if defined(__GNUC__)
-#define FUZZ_STATIC static __inline __attribute__((unused))
-#elif defined(__cplusplus) ||                                                  \
-    (defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */)
-#define FUZZ_STATIC static inline
-#elif defined(_MSC_VER)
-#define FUZZ_STATIC static __inline
-#else
-#define FUZZ_STATIC static
-#endif
-
-#ifdef __cplusplus
-}
-#endif
-
-#endif
diff --git a/tests/fuzz/regression_driver.c b/tests/fuzz/regression_driver.c
deleted file mode 100644
index e3ebcd5ced74..000000000000
--- a/tests/fuzz/regression_driver.c
+++ /dev/null
@@ -1,77 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-
-#include "fuzz.h"
-#include "fuzz_helpers.h"
-#include "util.h"
-#include <stddef.h>
-#include <stdint.h>
-#include <stdio.h>
-#include <stdlib.h>
-
-int main(int argc, char const **argv) {
-  size_t const kMaxFileSize = (size_t)1 << 27;
-  int const kFollowLinks = 1;
-  char *fileNamesBuf = NULL;
-  char const **files = argv + 1;
-  unsigned numFiles = argc - 1;
-  uint8_t *buffer = NULL;
-  size_t bufferSize = 0;
-  unsigned i;
-  int ret;
-
-#ifdef UTIL_HAS_CREATEFILELIST
-  files = UTIL_createFileList(files, numFiles, &fileNamesBuf, &numFiles,
-                              kFollowLinks);
-  if (!files)
-    numFiles = 0;
-#endif
-  if (numFiles == 0)
-    fprintf(stderr, "WARNING: No files passed to %s\n", argv[0]);
-  for (i = 0; i < numFiles; ++i) {
-    char const *fileName = files[i];
-    DEBUGLOG(3, "Running %s", fileName);
-    size_t const fileSize = UTIL_getFileSize(fileName);
-    size_t readSize;
-    FILE *file;
-
-    /* Check that it is a regular file, and that the fileSize is valid.
-     * If it is not a regular file, then it may have been deleted since we
-     * constructed the list, so just skip it.
-     */
-    if (!UTIL_isRegularFile(fileName)) {
-      continue;
-    }
-    FUZZ_ASSERT_MSG(fileSize <= kMaxFileSize, fileName);
-    /* Ensure we have a large enough buffer allocated */
-    if (fileSize > bufferSize) {
-      free(buffer);
-      buffer = (uint8_t *)malloc(fileSize);
-      FUZZ_ASSERT_MSG(buffer, fileName);
-      bufferSize = fileSize;
-    }
-    /* Open the file */
-    file = fopen(fileName, "rb");
-    FUZZ_ASSERT_MSG(file, fileName);
-    /* Read the file */
-    readSize = fread(buffer, 1, fileSize, file);
-    FUZZ_ASSERT_MSG(readSize == fileSize, fileName);
-    /* Close the file */
-    fclose(file);
-    /* Run the fuzz target */
-    LLVMFuzzerTestOneInput(buffer, fileSize);
-  }
-
-  ret = 0;
-  free(buffer);
-#ifdef UTIL_HAS_CREATEFILELIST
-  UTIL_freeFileList(files, fileNamesBuf);
-#endif
-  return ret;
-}
diff --git a/tests/fuzz/simple_compress.c b/tests/fuzz/simple_compress.c
deleted file mode 100644
index 487be3a354ce..000000000000
--- a/tests/fuzz/simple_compress.c
+++ /dev/null
@@ -1,52 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-
-/**
- * This fuzz target attempts to comprss the fuzzed data with the simple
- * compression function with an output buffer that may be too small to
- * ensure that the compressor never crashes.
- */
-
-#include <stddef.h>
-#include <stdlib.h>
-#include <stdio.h>
-#include "fuzz_helpers.h"
-#include "zstd.h"
-#include "zstd_helpers.h"
-#include "fuzz_data_producer.h"
-
-static ZSTD_CCtx *cctx = NULL;
-
-int LLVMFuzzerTestOneInput(const uint8_t *src, size_t size)
-{
-    /* Give a random portion of src data to the producer, to use for
-    parameter generation. The rest will be used for (de)compression */
-    FUZZ_dataProducer_t *producer = FUZZ_dataProducer_create(src, size);
-    size = FUZZ_dataProducer_reserveDataPrefix(producer);
-
-    size_t const maxSize = ZSTD_compressBound(size);
-    size_t const bufSize = FUZZ_dataProducer_uint32Range(producer, 0, maxSize);
-
-    int const cLevel = FUZZ_dataProducer_int32Range(producer, kMinClevel, kMaxClevel);
-
-    if (!cctx) {
-        cctx = ZSTD_createCCtx();
-        FUZZ_ASSERT(cctx);
-    }
-
-    void *rBuf = malloc(bufSize);
-    FUZZ_ASSERT(rBuf);
-    ZSTD_compressCCtx(cctx, rBuf, bufSize, src, size, cLevel);
-    free(rBuf);
-    FUZZ_dataProducer_free(producer);
-#ifndef STATEFUL_FUZZING
-    ZSTD_freeCCtx(cctx); cctx = NULL;
-#endif
-    return 0;
-}
diff --git a/tests/fuzz/simple_decompress.c b/tests/fuzz/simple_decompress.c
deleted file mode 100644
index 6182746a1d16..000000000000
--- a/tests/fuzz/simple_decompress.c
+++ /dev/null
@@ -1,49 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-
-/**
- * This fuzz target attempts to decompress the fuzzed data with the simple
- * decompression function to ensure the decompressor never crashes.
- */
-
-#include <stddef.h>
-#include <stdlib.h>
-#include <stdio.h>
-#include "fuzz_helpers.h"
-#include "zstd.h"
-#include "fuzz_data_producer.h"
-
-static ZSTD_DCtx *dctx = NULL;
-
-int LLVMFuzzerTestOneInput(const uint8_t *src, size_t size)
-{
-    /* Give a random portion of src data to the producer, to use for
-    parameter generation. The rest will be used for (de)compression */
-    FUZZ_dataProducer_t *producer = FUZZ_dataProducer_create(src, size);
-    size = FUZZ_dataProducer_reserveDataPrefix(producer);
-
-    if (!dctx) {
-        dctx = ZSTD_createDCtx();
-        FUZZ_ASSERT(dctx);
-    }
-
-    size_t const bufSize = FUZZ_dataProducer_uint32Range(producer, 0, 10 * size);
-    void *rBuf = malloc(bufSize);
-    FUZZ_ASSERT(rBuf);
-
-    ZSTD_decompressDCtx(dctx, rBuf, bufSize, src, size);
-    free(rBuf);
-
-    FUZZ_dataProducer_free(producer);
-
-#ifndef STATEFUL_FUZZING
-    ZSTD_freeDCtx(dctx); dctx = NULL;
-#endif
-    return 0;
-}
diff --git a/tests/fuzz/simple_round_trip.c b/tests/fuzz/simple_round_trip.c
deleted file mode 100644
index 2d1d0598a14b..000000000000
--- a/tests/fuzz/simple_round_trip.c
+++ /dev/null
@@ -1,93 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-
-/**
- * This fuzz target performs a zstd round-trip test (compress & decompress),
- * compares the result with the original, and calls abort() on corruption.
- */
-
-#define ZSTD_STATIC_LINKING_ONLY
-
-#include <stddef.h>
-#include <stdlib.h>
-#include <stdio.h>
-#include <string.h>
-#include "fuzz_helpers.h"
-#include "zstd_helpers.h"
-#include "fuzz_data_producer.h"
-
-static ZSTD_CCtx *cctx = NULL;
-static ZSTD_DCtx *dctx = NULL;
-
-static size_t roundTripTest(void *result, size_t resultCapacity,
-                            void *compressed, size_t compressedCapacity,
-                            const void *src, size_t srcSize,
-                            FUZZ_dataProducer_t *producer)
-{
-    size_t cSize;
-    if (FUZZ_dataProducer_uint32Range(producer, 0, 1)) {
-        FUZZ_setRandomParameters(cctx, srcSize, producer);
-        cSize = ZSTD_compress2(cctx, compressed, compressedCapacity, src, srcSize);
-    } else {
-      int const cLevel = FUZZ_dataProducer_int32Range(producer, kMinClevel, kMaxClevel);
-
-        cSize = ZSTD_compressCCtx(
-            cctx, compressed, compressedCapacity, src, srcSize, cLevel);
-    }
-    FUZZ_ZASSERT(cSize);
-    return ZSTD_decompressDCtx(dctx, result, resultCapacity, compressed, cSize);
-}
-
-int LLVMFuzzerTestOneInput(const uint8_t *src, size_t size)
-{
-    size_t const rBufSize = size;
-    void* rBuf = malloc(rBufSize);
-    size_t cBufSize = ZSTD_compressBound(size);
-    void* cBuf;
-
-    /* Give a random portion of src data to the producer, to use for
-    parameter generation. The rest will be used for (de)compression */
-    FUZZ_dataProducer_t *producer = FUZZ_dataProducer_create(src, size);
-    size = FUZZ_dataProducer_reserveDataPrefix(producer);
-
-    /* Half of the time fuzz with a 1 byte smaller output size.
-     * This will still succeed because we don't use a dictionary, so the dictID
-     * field is empty, giving us 4 bytes of overhead.
-     */
-    cBufSize -= FUZZ_dataProducer_uint32Range(producer, 0, 1);
-
-    cBuf = malloc(cBufSize);
-
-    FUZZ_ASSERT(cBuf && rBuf);
-
-    if (!cctx) {
-        cctx = ZSTD_createCCtx();
-        FUZZ_ASSERT(cctx);
-    }
-    if (!dctx) {
-        dctx = ZSTD_createDCtx();
-        FUZZ_ASSERT(dctx);
-    }
-
-    {
-        size_t const result =
-            roundTripTest(rBuf, rBufSize, cBuf, cBufSize, src, size, producer);
-        FUZZ_ZASSERT(result);
-        FUZZ_ASSERT_MSG(result == size, "Incorrect regenerated size");
-        FUZZ_ASSERT_MSG(!memcmp(src, rBuf, size), "Corruption!");
-    }
-    free(rBuf);
-    free(cBuf);
-    FUZZ_dataProducer_free(producer);
-#ifndef STATEFUL_FUZZING
-    ZSTD_freeCCtx(cctx); cctx = NULL;
-    ZSTD_freeDCtx(dctx); dctx = NULL;
-#endif
-    return 0;
-}
diff --git a/tests/fuzz/stream_decompress.c b/tests/fuzz/stream_decompress.c
deleted file mode 100644
index c71cc9d3ec6f..000000000000
--- a/tests/fuzz/stream_decompress.c
+++ /dev/null
@@ -1,89 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-
-/**
- * This fuzz target attempts to decompress the fuzzed data with the simple
- * decompression function to ensure the decompressor never crashes.
- */
-
-#define ZSTD_STATIC_LINKING_ONLY
-
-#include <stddef.h>
-#include <stdlib.h>
-#include <stdio.h>
-#include "fuzz_helpers.h"
-#include "zstd.h"
-#include "fuzz_data_producer.h"
-
-static size_t const kBufSize = ZSTD_BLOCKSIZE_MAX;
-
-static ZSTD_DStream *dstream = NULL;
-static void* buf = NULL;
-uint32_t seed;
-
-static ZSTD_outBuffer makeOutBuffer(FUZZ_dataProducer_t *producer)
-{
-  ZSTD_outBuffer buffer = { buf, 0, 0 };
-
-  buffer.size = (FUZZ_dataProducer_uint32Range(producer, 1, kBufSize));
-  FUZZ_ASSERT(buffer.size <= kBufSize);
-
-  return buffer;
-}
-
-static ZSTD_inBuffer makeInBuffer(const uint8_t **src, size_t *size,
-                                  FUZZ_dataProducer_t *producer)
-{
-  ZSTD_inBuffer buffer = { *src, 0, 0 };
-
-  FUZZ_ASSERT(*size > 0);
-  buffer.size = (FUZZ_dataProducer_uint32Range(producer, 1, *size));
-  FUZZ_ASSERT(buffer.size <= *size);
-  *src += buffer.size;
-  *size -= buffer.size;
-
-  return buffer;
-}
-
-int LLVMFuzzerTestOneInput(const uint8_t *src, size_t size)
-{
-    /* Give a random portion of src data to the producer, to use for
-    parameter generation. The rest will be used for (de)compression */
-    FUZZ_dataProducer_t *producer = FUZZ_dataProducer_create(src, size);
-    size = FUZZ_dataProducer_reserveDataPrefix(producer);
-
-    /* Allocate all buffers and contexts if not already allocated */
-    if (!buf) {
-      buf = malloc(kBufSize);
-        FUZZ_ASSERT(buf);
-      }
-
-    if (!dstream) {
-        dstream = ZSTD_createDStream();
-        FUZZ_ASSERT(dstream);
-    } else {
-        FUZZ_ZASSERT(ZSTD_DCtx_reset(dstream, ZSTD_reset_session_only));
-    }
-
-    while (size > 0) {
-        ZSTD_inBuffer in = makeInBuffer(&src, &size, producer);
-        while (in.pos != in.size) {
-            ZSTD_outBuffer out = makeOutBuffer(producer);
-            size_t const rc = ZSTD_decompressStream(dstream, &out, &in);
-            if (ZSTD_isError(rc)) goto error;
-        }
-    }
-
-error:
-#ifndef STATEFUL_FUZZING
-    ZSTD_freeDStream(dstream); dstream = NULL;
-#endif
-    FUZZ_dataProducer_free(producer);
-    return 0;
-}
diff --git a/tests/fuzz/stream_round_trip.c b/tests/fuzz/stream_round_trip.c
deleted file mode 100644
index 703b11713364..000000000000
--- a/tests/fuzz/stream_round_trip.c
+++ /dev/null
@@ -1,170 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-
-/**
- * This fuzz target performs a zstd round-trip test (compress & decompress),
- * compares the result with the original, and calls abort() on corruption.
- */
-
-#define ZSTD_STATIC_LINKING_ONLY
-
-#include <stddef.h>
-#include <stdlib.h>
-#include <stdio.h>
-#include <string.h>
-#include "fuzz_helpers.h"
-#include "zstd_helpers.h"
-#include "fuzz_data_producer.h"
-
-ZSTD_CCtx *cctx = NULL;
-static ZSTD_DCtx *dctx = NULL;
-static uint8_t* cBuf = NULL;
-static uint8_t* rBuf = NULL;
-static size_t bufSize = 0;
-
-static ZSTD_outBuffer makeOutBuffer(uint8_t *dst, size_t capacity,
-                                    FUZZ_dataProducer_t *producer)
-{
-    ZSTD_outBuffer buffer = { dst, 0, 0 };
-
-    FUZZ_ASSERT(capacity > 0);
-    buffer.size = (FUZZ_dataProducer_uint32Range(producer, 1, capacity));
-    FUZZ_ASSERT(buffer.size <= capacity);
-
-    return buffer;
-}
-
-static ZSTD_inBuffer makeInBuffer(const uint8_t **src, size_t *size,
-                                  FUZZ_dataProducer_t *producer)
-{
-    ZSTD_inBuffer buffer = { *src, 0, 0 };
-
-    FUZZ_ASSERT(*size > 0);
-    buffer.size = (FUZZ_dataProducer_uint32Range(producer, 1, *size));
-    FUZZ_ASSERT(buffer.size <= *size);
-    *src += buffer.size;
-    *size -= buffer.size;
-
-    return buffer;
-}
-
-static size_t compress(uint8_t *dst, size_t capacity,
-                       const uint8_t *src, size_t srcSize,
-                     FUZZ_dataProducer_t *producer)
-{
-    size_t dstSize = 0;
-    ZSTD_CCtx_reset(cctx, ZSTD_reset_session_only);
-    FUZZ_setRandomParameters(cctx, srcSize, producer);
-
-    while (srcSize > 0) {
-        ZSTD_inBuffer in = makeInBuffer(&src, &srcSize, producer);
-        /* Mode controls the action. If mode == -1 we pick a new mode */
-        int mode = -1;
-        while (in.pos < in.size || mode != -1) {
-            ZSTD_outBuffer out = makeOutBuffer(dst, capacity, producer);
-            /* Previous action finished, pick a new mode. */
-            if (mode == -1) mode = FUZZ_dataProducer_uint32Range(producer, 0, 9);
-            switch (mode) {
-                case 0: /* fall-through */
-                case 1: /* fall-through */
-                case 2: {
-                    size_t const ret =
-                        ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_flush);
-                    FUZZ_ZASSERT(ret);
-                    if (ret == 0)
-                        mode = -1;
-                    break;
-                }
-                case 3: {
-                    size_t ret =
-                        ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_end);
-                    FUZZ_ZASSERT(ret);
-                    /* Reset the compressor when the frame is finished */
-                    if (ret == 0) {
-                        ZSTD_CCtx_reset(cctx, ZSTD_reset_session_only);
-                        if (FUZZ_dataProducer_uint32Range(producer, 0, 7) == 0) {
-                            size_t const remaining = in.size - in.pos;
-                            FUZZ_setRandomParameters(cctx, remaining, producer);
-                        }
-                        mode = -1;
-                    }
-                    break;
-                }
-                default: {
-                    size_t const ret =
-                        ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_continue);
-                    FUZZ_ZASSERT(ret);
-                    mode = -1;
-                }
-            }
-            dst += out.pos;
-            dstSize += out.pos;
-            capacity -= out.pos;
-        }
-    }
-    for (;;) {
-        ZSTD_inBuffer in = {NULL, 0, 0};
-        ZSTD_outBuffer out = makeOutBuffer(dst, capacity, producer);
-        size_t const ret = ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_end);
-        FUZZ_ZASSERT(ret);
-
-        dst += out.pos;
-        dstSize += out.pos;
-        capacity -= out.pos;
-        if (ret == 0)
-            break;
-    }
-    return dstSize;
-}
-
-int LLVMFuzzerTestOneInput(const uint8_t *src, size_t size)
-{
-    size_t neededBufSize;
-
-    /* Give a random portion of src data to the producer, to use for
-    parameter generation. The rest will be used for (de)compression */
-    FUZZ_dataProducer_t *producer = FUZZ_dataProducer_create(src, size);
-    size = FUZZ_dataProducer_reserveDataPrefix(producer);
-
-    neededBufSize = ZSTD_compressBound(size) * 15;
-
-    /* Allocate all buffers and contexts if not already allocated */
-    if (neededBufSize > bufSize) {
-        free(cBuf);
-        free(rBuf);
-        cBuf = (uint8_t*)malloc(neededBufSize);
-        rBuf = (uint8_t*)malloc(neededBufSize);
-        bufSize = neededBufSize;
-        FUZZ_ASSERT(cBuf && rBuf);
-    }
-    if (!cctx) {
-        cctx = ZSTD_createCCtx();
-        FUZZ_ASSERT(cctx);
-    }
-    if (!dctx) {
-        dctx = ZSTD_createDCtx();
-        FUZZ_ASSERT(dctx);
-    }
-
-    {
-        size_t const cSize = compress(cBuf, neededBufSize, src, size, producer);
-        size_t const rSize =
-            ZSTD_decompressDCtx(dctx, rBuf, neededBufSize, cBuf, cSize);
-        FUZZ_ZASSERT(rSize);
-        FUZZ_ASSERT_MSG(rSize == size, "Incorrect regenerated size");
-        FUZZ_ASSERT_MSG(!memcmp(src, rBuf, size), "Corruption!");
-    }
-
-    FUZZ_dataProducer_free(producer);
-#ifndef STATEFUL_FUZZING
-    ZSTD_freeCCtx(cctx); cctx = NULL;
-    ZSTD_freeDCtx(dctx); dctx = NULL;
-#endif
-    return 0;
-}
diff --git a/tests/fuzz/zstd_frame_info.c b/tests/fuzz/zstd_frame_info.c
deleted file mode 100644
index 359cf128f5b8..000000000000
--- a/tests/fuzz/zstd_frame_info.c
+++ /dev/null
@@ -1,39 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-
-/**
- * This fuzz target fuzzes all of the helper functions that consume compressed
- * input.
- */
-
-#include <stddef.h>
-#include <stdlib.h>
-#include <stdio.h>
-#include "fuzz_helpers.h"
-#include "zstd_helpers.h"
-
-int LLVMFuzzerTestOneInput(const uint8_t *src, size_t size)
-{
-    ZSTD_frameHeader zfh;
-    /* You can fuzz any helper functions here that are fast, and take zstd
-     * compressed data as input. E.g. don't expect the input to be a dictionary,
-     * so don't fuzz ZSTD_getDictID_fromDict().
-     */
-    ZSTD_getFrameContentSize(src, size);
-    ZSTD_getDecompressedSize(src, size);
-    ZSTD_findFrameCompressedSize(src, size);
-    ZSTD_getDictID_fromFrame(src, size);
-    ZSTD_findDecompressedSize(src, size);
-    ZSTD_decompressBound(src, size);
-    ZSTD_frameHeaderSize(src, size);
-    ZSTD_isFrame(src, size);
-    ZSTD_getFrameHeader(&zfh, src, size);
-    ZSTD_getFrameHeader_advanced(&zfh, src, size, ZSTD_f_zstd1);
-    return 0;
-}
diff --git a/tests/fuzz/zstd_helpers.c b/tests/fuzz/zstd_helpers.c
deleted file mode 100644
index 90bf1a156787..000000000000
--- a/tests/fuzz/zstd_helpers.c
+++ /dev/null
@@ -1,143 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-
-#define ZSTD_STATIC_LINKING_ONLY
-#define ZDICT_STATIC_LINKING_ONLY
-
-#include <string.h>
-
-#include "zstd_helpers.h"
-#include "fuzz_helpers.h"
-#include "zstd.h"
-#include "zdict.h"
-
-const int kMinClevel = -3;
-const int kMaxClevel = 19;
-
-static void set(ZSTD_CCtx *cctx, ZSTD_cParameter param, int value)
-{
-    FUZZ_ZASSERT(ZSTD_CCtx_setParameter(cctx, param, value));
-}
-
-static void setRand(ZSTD_CCtx *cctx, ZSTD_cParameter param, unsigned min,
-                    unsigned max, FUZZ_dataProducer_t *producer) {
-    unsigned const value = FUZZ_dataProducer_uint32Range(producer, min, max);
-    set(cctx, param, value);
-}
-
-ZSTD_compressionParameters FUZZ_randomCParams(size_t srcSize, FUZZ_dataProducer_t *producer)
-{
-    /* Select compression parameters */
-    ZSTD_compressionParameters cParams;
-    cParams.windowLog = FUZZ_dataProducer_uint32Range(producer, ZSTD_WINDOWLOG_MIN, 15);
-    cParams.hashLog = FUZZ_dataProducer_uint32Range(producer, ZSTD_HASHLOG_MIN, 15);
-    cParams.chainLog = FUZZ_dataProducer_uint32Range(producer, ZSTD_CHAINLOG_MIN, 16);
-    cParams.searchLog = FUZZ_dataProducer_uint32Range(producer, ZSTD_SEARCHLOG_MIN, 9);
-    cParams.minMatch = FUZZ_dataProducer_uint32Range(producer, ZSTD_MINMATCH_MIN,
-                                          ZSTD_MINMATCH_MAX);
-    cParams.targetLength = FUZZ_dataProducer_uint32Range(producer, 0, 512);
-    cParams.strategy = FUZZ_dataProducer_uint32Range(producer, ZSTD_STRATEGY_MIN, ZSTD_STRATEGY_MAX);
-    return ZSTD_adjustCParams(cParams, srcSize, 0);
-}
-
-ZSTD_frameParameters FUZZ_randomFParams(FUZZ_dataProducer_t *producer)
-{
-    /* Select frame parameters */
-    ZSTD_frameParameters fParams;
-    fParams.contentSizeFlag = FUZZ_dataProducer_uint32Range(producer, 0, 1);
-    fParams.checksumFlag = FUZZ_dataProducer_uint32Range(producer, 0, 1);
-    fParams.noDictIDFlag = FUZZ_dataProducer_uint32Range(producer, 0, 1);
-    return fParams;
-}
-
-ZSTD_parameters FUZZ_randomParams(size_t srcSize, FUZZ_dataProducer_t *producer)
-{
-    ZSTD_parameters params;
-    params.cParams = FUZZ_randomCParams(srcSize, producer);
-    params.fParams = FUZZ_randomFParams(producer);
-    return params;
-}
-
-void FUZZ_setRandomParameters(ZSTD_CCtx *cctx, size_t srcSize, FUZZ_dataProducer_t *producer)
-{
-    ZSTD_compressionParameters cParams = FUZZ_randomCParams(srcSize, producer);
-    set(cctx, ZSTD_c_windowLog, cParams.windowLog);
-    set(cctx, ZSTD_c_hashLog, cParams.hashLog);
-    set(cctx, ZSTD_c_chainLog, cParams.chainLog);
-    set(cctx, ZSTD_c_searchLog, cParams.searchLog);
-    set(cctx, ZSTD_c_minMatch, cParams.minMatch);
-    set(cctx, ZSTD_c_targetLength, cParams.targetLength);
-    set(cctx, ZSTD_c_strategy, cParams.strategy);
-    /* Select frame parameters */
-    setRand(cctx, ZSTD_c_contentSizeFlag, 0, 1, producer);
-    setRand(cctx, ZSTD_c_checksumFlag, 0, 1, producer);
-    setRand(cctx, ZSTD_c_dictIDFlag, 0, 1, producer);
-    /* Select long distance matching parameters */
-    setRand(cctx, ZSTD_c_enableLongDistanceMatching, 0, 1, producer);
-    setRand(cctx, ZSTD_c_ldmHashLog, ZSTD_HASHLOG_MIN, 16, producer);
-    setRand(cctx, ZSTD_c_ldmMinMatch, ZSTD_LDM_MINMATCH_MIN,
-            ZSTD_LDM_MINMATCH_MAX, producer);
-    setRand(cctx, ZSTD_c_ldmBucketSizeLog, 0, ZSTD_LDM_BUCKETSIZELOG_MAX,
-            producer);
-    setRand(cctx, ZSTD_c_ldmHashRateLog, ZSTD_LDM_HASHRATELOG_MIN,
-            ZSTD_LDM_HASHRATELOG_MAX, producer);
-    /* Set misc parameters */
-    setRand(cctx, ZSTD_c_nbWorkers, 0, 2, producer);
-    setRand(cctx, ZSTD_c_rsyncable, 0, 1, producer);
-    setRand(cctx, ZSTD_c_forceMaxWindow, 0, 1, producer);
-    setRand(cctx, ZSTD_c_literalCompressionMode, 0, 2, producer);
-    setRand(cctx, ZSTD_c_forceAttachDict, 0, 2, producer);
-    if (FUZZ_dataProducer_uint32Range(producer, 0, 1) == 0) {
-      setRand(cctx, ZSTD_c_srcSizeHint, ZSTD_SRCSIZEHINT_MIN, 2 * srcSize, producer);
-    }
-}
-
-FUZZ_dict_t FUZZ_train(void const* src, size_t srcSize, FUZZ_dataProducer_t *producer)
-{
-    size_t const dictSize = MAX(srcSize / 8, 1024);
-    size_t const totalSampleSize = dictSize * 11;
-    FUZZ_dict_t dict = { malloc(dictSize), dictSize };
-    char* const samples = (char*)malloc(totalSampleSize);
-    unsigned nbSamples = 100;
-    size_t* const samplesSizes = (size_t*)malloc(sizeof(size_t) * nbSamples);
-    size_t pos = 0;
-    size_t sample = 0;
-    ZDICT_fastCover_params_t params;
-    FUZZ_ASSERT(dict.buff && samples && samplesSizes);
-
-    for (sample = 0; sample < nbSamples; ++sample) {
-      size_t const remaining = totalSampleSize - pos;
-      size_t const offset = FUZZ_dataProducer_uint32Range(producer, 0, MAX(srcSize, 1) - 1);
-      size_t const limit = MIN(srcSize - offset, remaining);
-      size_t const toCopy = MIN(limit, remaining / (nbSamples - sample));
-      memcpy(samples + pos, src + offset, toCopy);
-      pos += toCopy;
-      samplesSizes[sample] = toCopy;
-
-    }
-    memset(samples + pos, 0, totalSampleSize - pos);
-
-    memset(&params, 0, sizeof(params));
-    params.accel = 5;
-    params.k = 40;
-    params.d = 8;
-    params.f = 14;
-    params.zParams.compressionLevel = 1;
-    dict.size = ZDICT_trainFromBuffer_fastCover(dict.buff, dictSize,
-        samples, samplesSizes, nbSamples, params);
-    if (ZSTD_isError(dict.size)) {
-        free(dict.buff);
-        memset(&dict, 0, sizeof(dict));
-    }
-
-    free(samplesSizes);
-    free(samples);
-
-    return dict;
-}
diff --git a/tests/fuzz/zstd_helpers.h b/tests/fuzz/zstd_helpers.h
deleted file mode 100644
index 2210bcaf8f54..000000000000
--- a/tests/fuzz/zstd_helpers.h
+++ /dev/null
@@ -1,51 +0,0 @@
-/*
- * Copyright (c) 2016-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- */
-
-/**
- * Helper functions for fuzzing.
- */
-
-#ifndef ZSTD_HELPERS_H
-#define ZSTD_HELPERS_H
-
-#define ZSTD_STATIC_LINKING_ONLY
-
-#include "zstd.h"
-#include "fuzz_data_producer.h"
-#include <stdint.h>
-
-#ifdef __cplusplus
-extern "C" {
-#endif
-
-extern const int kMinClevel;
-extern const int kMaxClevel;
-
-void FUZZ_setRandomParameters(ZSTD_CCtx *cctx, size_t srcSize, FUZZ_dataProducer_t *producer);
-
-ZSTD_compressionParameters FUZZ_randomCParams(size_t srcSize, FUZZ_dataProducer_t *producer);
-ZSTD_frameParameters FUZZ_randomFParams(FUZZ_dataProducer_t *producer);
-ZSTD_parameters FUZZ_randomParams(size_t srcSize, FUZZ_dataProducer_t *producer);
-
-typedef struct {
-  void* buff;
-  size_t size;
-} FUZZ_dict_t;
-
-/* Quickly train a dictionary from a source for fuzzing.
- * NOTE: Don't use this to train production dictionaries, it is only optimized
- * for speed, and doesn't care about dictionary quality.
- */
-FUZZ_dict_t FUZZ_train(void const* src, size_t srcSize, FUZZ_dataProducer_t *producer);
-
-#ifdef __cplusplus
-}
-#endif
-
-#endif /* ZSTD_HELPERS_H */
diff --git a/tests/fuzzer.c b/tests/fuzzer.c
deleted file mode 100644
index 88f3b83f834c..000000000000
--- a/tests/fuzzer.c
+++ /dev/null
@@ -1,2839 +0,0 @@
-/*
- * Copyright (c) 2015-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-
-/*-************************************
-*  Compiler specific
-**************************************/
-#ifdef _MSC_VER    /* Visual Studio */
-#  define _CRT_SECURE_NO_WARNINGS   /* fgets */
-#  pragma warning(disable : 4127)   /* disable: C4127: conditional expression is constant */
-#  pragma warning(disable : 4204)   /* disable: C4204: non-constant aggregate initializer */
-#endif
-
-
-/*-************************************
-*  Includes
-**************************************/
-#include <stdlib.h>       /* free */
-#include <stdio.h>        /* fgets, sscanf */
-#include <string.h>       /* strcmp */
-#include <assert.h>
-#define ZSTD_STATIC_LINKING_ONLY  /* ZSTD_compressContinue, ZSTD_compressBlock */
-#include "fse.h"
-#include "zstd.h"         /* ZSTD_VERSION_STRING */
-#include "zstd_errors.h"  /* ZSTD_getErrorCode */
-#include "zstdmt_compress.h"
-#define ZDICT_STATIC_LINKING_ONLY
-#include "zdict.h"        /* ZDICT_trainFromBuffer */
-#include "datagen.h"      /* RDG_genBuffer */
-#include "mem.h"
-#define XXH_STATIC_LINKING_ONLY   /* XXH64_state_t */
-#include "xxhash.h"       /* XXH64 */
-#include "util.h"
-#include "timefn.h"       /* SEC_TO_MICRO, UTIL_time_t, UTIL_TIME_INITIALIZER, UTIL_clockSpanMicro, UTIL_getTime */
-
-
-/*-************************************
-*  Constants
-**************************************/
-#define KB *(1U<<10)
-#define MB *(1U<<20)
-#define GB *(1U<<30)
-
-static const int FUZ_compressibility_default = 50;
-static const int nbTestsDefault = 30000;
-
-
-/*-************************************
-*  Display Macros
-**************************************/
-#define DISPLAY(...)          fprintf(stderr, __VA_ARGS__)
-#define DISPLAYLEVEL(l, ...)  if (g_displayLevel>=l) { DISPLAY(__VA_ARGS__); }
-static U32 g_displayLevel = 2;
-
-static const U64 g_refreshRate = SEC_TO_MICRO / 6;
-static UTIL_time_t g_displayClock = UTIL_TIME_INITIALIZER;
-
-#define DISPLAYUPDATE(l, ...) \
-    if (g_displayLevel>=l) { \
-        if ((UTIL_clockSpanMicro(g_displayClock) > g_refreshRate) || (g_displayLevel>=4)) \
-        { g_displayClock = UTIL_getTime(); DISPLAY(__VA_ARGS__); \
-        if (g_displayLevel>=4) fflush(stderr); } \
-    }
-
-
-/*-*******************************************************
-*  Compile time test
-*********************************************************/
-#undef MIN
-#undef MAX
-/* Declaring the function, to avoid -Wmissing-prototype */
-void FUZ_bug976(void);
-void FUZ_bug976(void)
-{   /* these constants shall not depend on MIN() macro */
-    assert(ZSTD_HASHLOG_MAX < 31);
-    assert(ZSTD_CHAINLOG_MAX < 31);
-}
-
-
-/*-*******************************************************
-*  Internal functions
-*********************************************************/
-#define MIN(a,b) ((a)<(b)?(a):(b))
-#define MAX(a,b) ((a)>(b)?(a):(b))
-
-#define FUZ_rotl32(x,r) ((x << r) | (x >> (32 - r)))
-static U32 FUZ_rand(U32* src)
-{
-    static const U32 prime1 = 2654435761U;
-    static const U32 prime2 = 2246822519U;
-    U32 rand32 = *src;
-    rand32 *= prime1;
-    rand32 += prime2;
-    rand32  = FUZ_rotl32(rand32, 13);
-    *src = rand32;
-    return rand32 >> 5;
-}
-
-static U32 FUZ_highbit32(U32 v32)
-{
-    unsigned nbBits = 0;
-    if (v32==0) return 0;
-    while (v32) v32 >>= 1, nbBits++;
-    return nbBits;
-}
-
-
-/*=============================================
-*   Test macros
-=============================================*/
-#define CHECK_Z(f) {                               \
-    size_t const err = f;                          \
-    if (ZSTD_isError(err)) {                       \
-        DISPLAY("Error => %s : %s ",               \
-                #f, ZSTD_getErrorName(err));       \
-        exit(1);                                   \
-}   }
-
-#define CHECK_VAR(var, fn)  var = fn; if (ZSTD_isError(var)) { DISPLAYLEVEL(1, "%s : fails : %s \n", #fn, ZSTD_getErrorName(var)); goto _output_error; }
-#define CHECK_NEWV(var, fn)  size_t const CHECK_VAR(var, fn)
-#define CHECK(fn)  { CHECK_NEWV(err, fn); }
-#define CHECKPLUS(var, fn, more)  { CHECK_NEWV(var, fn); more; }
-
-#define CHECK_OP(op, lhs, rhs) {                                  \
-    if (!((lhs) op (rhs))) {                                      \
-        DISPLAY("Error L%u => FAILED %s %s %s ", __LINE__, #lhs, #op, #rhs);  \
-        goto _output_error;                                       \
-    }                                                             \
-}
-#define CHECK_EQ(lhs, rhs) CHECK_OP(==, lhs, rhs)
-#define CHECK_LT(lhs, rhs) CHECK_OP(<, lhs, rhs)
-
-
-/*=============================================
-*   Memory Tests
-=============================================*/
-#if defined(__APPLE__) && defined(__MACH__)
-
-#include <malloc/malloc.h>    /* malloc_size */
-
-typedef struct {
-    unsigned long long totalMalloc;
-    size_t currentMalloc;
-    size_t peakMalloc;
-    unsigned nbMalloc;
-    unsigned nbFree;
-} mallocCounter_t;
-
-static const mallocCounter_t INIT_MALLOC_COUNTER = { 0, 0, 0, 0, 0 };
-
-static void* FUZ_mallocDebug(void* counter, size_t size)
-{
-    mallocCounter_t* const mcPtr = (mallocCounter_t*)counter;
-    void* const ptr = malloc(size);
-    if (ptr==NULL) return NULL;
-    DISPLAYLEVEL(4, "allocating %u KB => effectively %u KB \n",
-        (unsigned)(size >> 10), (unsigned)(malloc_size(ptr) >> 10));  /* OS-X specific */
-    mcPtr->totalMalloc += size;
-    mcPtr->currentMalloc += size;
-    if (mcPtr->currentMalloc > mcPtr->peakMalloc)
-        mcPtr->peakMalloc = mcPtr->currentMalloc;
-    mcPtr->nbMalloc += 1;
-    return ptr;
-}
-
-static void FUZ_freeDebug(void* counter, void* address)
-{
-    mallocCounter_t* const mcPtr = (mallocCounter_t*)counter;
-    DISPLAYLEVEL(4, "freeing %u KB \n", (unsigned)(malloc_size(address) >> 10));
-    mcPtr->nbFree += 1;
-    mcPtr->currentMalloc -= malloc_size(address);  /* OS-X specific */
-    free(address);
-}
-
-static void FUZ_displayMallocStats(mallocCounter_t count)
-{
-    DISPLAYLEVEL(3, "peak:%6u KB,  nbMallocs:%2u, total:%6u KB \n",
-        (unsigned)(count.peakMalloc >> 10),
-        count.nbMalloc,
-        (unsigned)(count.totalMalloc >> 10));
-}
-
-static int FUZ_mallocTests_internal(unsigned seed, double compressibility, unsigned part,
-                void* inBuffer, size_t inSize, void* outBuffer, size_t outSize)
-{
-    /* test only played in verbose mode, as they are long */
-    if (g_displayLevel<3) return 0;
-
-    /* Create compressible noise */
-    if (!inBuffer || !outBuffer) {
-        DISPLAY("Not enough memory, aborting\n");
-        exit(1);
-    }
-    RDG_genBuffer(inBuffer, inSize, compressibility, 0. /*auto*/, seed);
-
-    /* simple compression tests */
-    if (part <= 1)
-    {   int compressionLevel;
-        for (compressionLevel=1; compressionLevel<=6; compressionLevel++) {
-            mallocCounter_t malcount = INIT_MALLOC_COUNTER;
-            ZSTD_customMem const cMem = { FUZ_mallocDebug, FUZ_freeDebug, &malcount };
-            ZSTD_CCtx* const cctx = ZSTD_createCCtx_advanced(cMem);
-            CHECK_Z( ZSTD_compressCCtx(cctx, outBuffer, outSize, inBuffer, inSize, compressionLevel) );
-            ZSTD_freeCCtx(cctx);
-            DISPLAYLEVEL(3, "compressCCtx level %i : ", compressionLevel);
-            FUZ_displayMallocStats(malcount);
-    }   }
-
-    /* streaming compression tests */
-    if (part <= 2)
-    {   int compressionLevel;
-        for (compressionLevel=1; compressionLevel<=6; compressionLevel++) {
-            mallocCounter_t malcount = INIT_MALLOC_COUNTER;
-            ZSTD_customMem const cMem = { FUZ_mallocDebug, FUZ_freeDebug, &malcount };
-            ZSTD_CCtx* const cstream = ZSTD_createCStream_advanced(cMem);
-            ZSTD_outBuffer out = { outBuffer, outSize, 0 };
-            ZSTD_inBuffer in = { inBuffer, inSize, 0 };
-            CHECK_Z( ZSTD_initCStream(cstream, compressionLevel) );
-            CHECK_Z( ZSTD_compressStream(cstream, &out, &in) );
-            CHECK_Z( ZSTD_endStream(cstream, &out) );
-            ZSTD_freeCStream(cstream);
-            DISPLAYLEVEL(3, "compressStream level %i : ", compressionLevel);
-            FUZ_displayMallocStats(malcount);
-    }   }
-
-    /* advanced MT API test */
-    if (part <= 3)
-    {   int nbThreads;
-        for (nbThreads=1; nbThreads<=4; nbThreads++) {
-            int compressionLevel;
-            for (compressionLevel=1; compressionLevel<=6; compressionLevel++) {
-                mallocCounter_t malcount = INIT_MALLOC_COUNTER;
-                ZSTD_customMem const cMem = { FUZ_mallocDebug, FUZ_freeDebug, &malcount };
-                ZSTD_CCtx* const cctx = ZSTD_createCCtx_advanced(cMem);
-                CHECK_Z( ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, compressionLevel) );
-                CHECK_Z( ZSTD_CCtx_setParameter(cctx, ZSTD_c_nbWorkers, nbThreads) );
-                CHECK_Z( ZSTD_compress2(cctx, outBuffer, outSize, inBuffer, inSize) );
-                ZSTD_freeCCtx(cctx);
-                DISPLAYLEVEL(3, "compress_generic,-T%i,end level %i : ",
-                                nbThreads, compressionLevel);
-                FUZ_displayMallocStats(malcount);
-    }   }   }
-
-    /* advanced MT streaming API test */
-    if (part <= 4)
-    {   int nbThreads;
-        for (nbThreads=1; nbThreads<=4; nbThreads++) {
-            int compressionLevel;
-            for (compressionLevel=1; compressionLevel<=6; compressionLevel++) {
-                mallocCounter_t malcount = INIT_MALLOC_COUNTER;
-                ZSTD_customMem const cMem = { FUZ_mallocDebug, FUZ_freeDebug, &malcount };
-                ZSTD_CCtx* const cctx = ZSTD_createCCtx_advanced(cMem);
-                ZSTD_outBuffer out = { outBuffer, outSize, 0 };
-                ZSTD_inBuffer in = { inBuffer, inSize, 0 };
-                CHECK_Z( ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, compressionLevel) );
-                CHECK_Z( ZSTD_CCtx_setParameter(cctx, ZSTD_c_nbWorkers, nbThreads) );
-                CHECK_Z( ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_continue) );
-                while ( ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_end) ) {}
-                ZSTD_freeCCtx(cctx);
-                DISPLAYLEVEL(3, "compress_generic,-T%i,continue level %i : ",
-                                nbThreads, compressionLevel);
-                FUZ_displayMallocStats(malcount);
-    }   }   }
-
-    return 0;
-}
-
-static int FUZ_mallocTests(unsigned seed, double compressibility, unsigned part)
-{
-    size_t const inSize = 64 MB + 16 MB + 4 MB + 1 MB + 256 KB + 64 KB; /* 85.3 MB */
-    size_t const outSize = ZSTD_compressBound(inSize);
-    void* const inBuffer = malloc(inSize);
-    void* const outBuffer = malloc(outSize);
-    int result;
-
-    /* Create compressible noise */
-    if (!inBuffer || !outBuffer) {
-        DISPLAY("Not enough memory, aborting \n");
-        exit(1);
-    }
-
-    result = FUZ_mallocTests_internal(seed, compressibility, part,
-                    inBuffer, inSize, outBuffer, outSize);
-
-    free(inBuffer);
-    free(outBuffer);
-    return result;
-}
-
-#else
-
-static int FUZ_mallocTests(unsigned seed, double compressibility, unsigned part)
-{
-    (void)seed; (void)compressibility; (void)part;
-    return 0;
-}
-
-#endif
-
-static void FUZ_decodeSequences(BYTE* dst, ZSTD_Sequence* seqs, size_t seqsSize, BYTE* src, size_t size)
-{
-    size_t i;
-    size_t j;
-    for(i = 0; i < seqsSize - 1; ++i) {
-        assert(dst + seqs[i].litLength + seqs[i].matchLength < dst + size);
-        assert(src + seqs[i].litLength + seqs[i].matchLength < src + size);
-
-        memcpy(dst, src, seqs[i].litLength);
-        dst += seqs[i].litLength;
-        src += seqs[i].litLength;
-        size -= seqs[i].litLength;
-
-        for (j = 0; j < seqs[i].matchLength; ++j)
-            dst[j] = dst[j - seqs[i].offset];
-        dst += seqs[i].matchLength;
-        src += seqs[i].matchLength;
-        size -= seqs[i].matchLength;
-    }
-    memcpy(dst, src, size);
-}
-
-/*=============================================
-*   Unit tests
-=============================================*/
-
-static int basicUnitTests(U32 const seed, double compressibility)
-{
-    size_t const CNBuffSize = 5 MB;
-    void* const CNBuffer = malloc(CNBuffSize);
-    size_t const compressedBufferSize = ZSTD_compressBound(CNBuffSize);
-    void* const compressedBuffer = malloc(compressedBufferSize);
-    void* const decodedBuffer = malloc(CNBuffSize);
-    int testResult = 0;
-    unsigned testNb=0;
-    size_t cSize;
-
-    /* Create compressible noise */
-    if (!CNBuffer || !compressedBuffer || !decodedBuffer) {
-        DISPLAY("Not enough memory, aborting\n");
-        testResult = 1;
-        goto _end;
-    }
-    RDG_genBuffer(CNBuffer, CNBuffSize, compressibility, 0., seed);
-
-    /* Basic tests */
-    DISPLAYLEVEL(3, "test%3u : ZSTD_getErrorName : ", testNb++);
-    {   const char* errorString = ZSTD_getErrorName(0);
-        DISPLAYLEVEL(3, "OK : %s \n", errorString);
-    }
-
-    DISPLAYLEVEL(3, "test%3u : ZSTD_getErrorName with wrong value : ", testNb++);
-    {   const char* errorString = ZSTD_getErrorName(499);
-        DISPLAYLEVEL(3, "OK : %s \n", errorString);
-    }
-
-    DISPLAYLEVEL(3, "test%3u : min compression level : ", testNb++);
-    {   int const mcl = ZSTD_minCLevel();
-        DISPLAYLEVEL(3, "%i (OK) \n", mcl);
-    }
-
-    DISPLAYLEVEL(3, "test%3u : compress %u bytes : ", testNb++, (unsigned)CNBuffSize);
-    {   ZSTD_CCtx* const cctx = ZSTD_createCCtx();
-        if (cctx==NULL) goto _output_error;
-        CHECK_VAR(cSize, ZSTD_compressCCtx(cctx,
-                            compressedBuffer, compressedBufferSize,
-                            CNBuffer, CNBuffSize, 1) );
-        DISPLAYLEVEL(3, "OK (%u bytes : %.2f%%)\n", (unsigned)cSize, (double)cSize/CNBuffSize*100);
-
-        DISPLAYLEVEL(3, "test%3i : size of cctx for level 1 : ", testNb++);
-        {   size_t const cctxSize = ZSTD_sizeof_CCtx(cctx);
-            DISPLAYLEVEL(3, "%u bytes \n", (unsigned)cctxSize);
-        }
-        ZSTD_freeCCtx(cctx);
-    }
-
-    DISPLAYLEVEL(3, "test%3i : decompress skippable frame -8 size : ", testNb++);
-    {
-       char const skippable8[] = "\x50\x2a\x4d\x18\xf8\xff\xff\xff";
-       size_t const size = ZSTD_decompress(NULL, 0, skippable8, 8);
-       if (!ZSTD_isError(size)) goto _output_error;
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-
-    DISPLAYLEVEL(3, "test%3i : ZSTD_getFrameContentSize test : ", testNb++);
-    {   unsigned long long const rSize = ZSTD_getFrameContentSize(compressedBuffer, cSize);
-        if (rSize != CNBuffSize) goto _output_error;
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : ZSTD_findDecompressedSize test : ", testNb++);
-    {   unsigned long long const rSize = ZSTD_findDecompressedSize(compressedBuffer, cSize);
-        if (rSize != CNBuffSize) goto _output_error;
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : tight ZSTD_decompressBound test : ", testNb++);
-    {
-        unsigned long long bound = ZSTD_decompressBound(compressedBuffer, cSize);
-        if (bound != CNBuffSize) goto _output_error;
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : ZSTD_decompressBound test with invalid srcSize : ", testNb++);
-    {
-        unsigned long long bound = ZSTD_decompressBound(compressedBuffer, cSize - 1);
-        if (bound != ZSTD_CONTENTSIZE_ERROR) goto _output_error;
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : decompress %u bytes : ", testNb++, (unsigned)CNBuffSize);
-    { size_t const r = ZSTD_decompress(decodedBuffer, CNBuffSize, compressedBuffer, cSize);
-      if (r != CNBuffSize) goto _output_error; }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : check decompressed result : ", testNb++);
-    {   size_t u;
-        for (u=0; u<CNBuffSize; u++) {
-            if (((BYTE*)decodedBuffer)[u] != ((BYTE*)CNBuffer)[u]) goto _output_error;
-    }   }
-    DISPLAYLEVEL(3, "OK \n");
-
-
-    DISPLAYLEVEL(3, "test%3i : decompress with null dict : ", testNb++);
-    {   ZSTD_DCtx* const dctx = ZSTD_createDCtx(); assert(dctx != NULL);
-        {   size_t const r = ZSTD_decompress_usingDict(dctx,
-                                                    decodedBuffer, CNBuffSize,
-                                                    compressedBuffer, cSize,
-                                                    NULL, 0);
-            if (r != CNBuffSize) goto _output_error;
-        }
-        ZSTD_freeDCtx(dctx);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : decompress with null DDict : ", testNb++);
-    {   ZSTD_DCtx* const dctx = ZSTD_createDCtx(); assert(dctx != NULL);
-        {   size_t const r = ZSTD_decompress_usingDDict(dctx,
-                                                    decodedBuffer, CNBuffSize,
-                                                    compressedBuffer, cSize,
-                                                    NULL);
-            if (r != CNBuffSize) goto _output_error;
-        }
-        ZSTD_freeDCtx(dctx);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : decompress with 1 missing byte : ", testNb++);
-    { size_t const r = ZSTD_decompress(decodedBuffer, CNBuffSize, compressedBuffer, cSize-1);
-      if (!ZSTD_isError(r)) goto _output_error;
-      if (ZSTD_getErrorCode((size_t)r) != ZSTD_error_srcSize_wrong) goto _output_error; }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : decompress with 1 too much byte : ", testNb++);
-    { size_t const r = ZSTD_decompress(decodedBuffer, CNBuffSize, compressedBuffer, cSize+1);
-      if (!ZSTD_isError(r)) goto _output_error;
-      if (ZSTD_getErrorCode(r) != ZSTD_error_srcSize_wrong) goto _output_error; }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : decompress too large input : ", testNb++);
-    { size_t const r = ZSTD_decompress(decodedBuffer, CNBuffSize, compressedBuffer, compressedBufferSize);
-      if (!ZSTD_isError(r)) goto _output_error;
-      if (ZSTD_getErrorCode(r) != ZSTD_error_srcSize_wrong) goto _output_error; }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : ZSTD_decompressBound test with content size missing : ", testNb++);
-    {   /* create compressed buffer with content size missing */
-        ZSTD_CCtx* const cctx = ZSTD_createCCtx();
-        CHECK_Z( ZSTD_CCtx_setParameter(cctx, ZSTD_c_contentSizeFlag, 0) );
-        CHECK_VAR(cSize, ZSTD_compress2(cctx,
-                            compressedBuffer, compressedBufferSize,
-                            CNBuffer, CNBuffSize) );
-        ZSTD_freeCCtx(cctx);
-    }
-    {   /* ensure frame content size is missing */
-        ZSTD_frameHeader zfh;
-        size_t const ret = ZSTD_getFrameHeader(&zfh, compressedBuffer, compressedBufferSize);
-        if (ret != 0 || zfh.frameContentSize !=  ZSTD_CONTENTSIZE_UNKNOWN) goto _output_error;
-    }
-    {   /* ensure CNBuffSize <= decompressBound */
-        unsigned long long const bound = ZSTD_decompressBound(compressedBuffer, compressedBufferSize);
-        if (CNBuffSize > bound) goto _output_error;
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3d : check CCtx size after compressing empty input : ", testNb++);
-    {   ZSTD_CCtx* const cctx = ZSTD_createCCtx();
-        size_t const r = ZSTD_compressCCtx(cctx, compressedBuffer, compressedBufferSize, NULL, 0, 19);
-        if (ZSTD_isError(r)) goto _output_error;
-        if (ZSTD_sizeof_CCtx(cctx) > (1U << 20)) goto _output_error;
-        ZSTD_freeCCtx(cctx);
-        cSize = r;
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3d : decompress empty frame into NULL : ", testNb++);
-    {   size_t const r = ZSTD_decompress(NULL, 0, compressedBuffer, cSize);
-        if (ZSTD_isError(r)) goto _output_error;
-        if (r != 0) goto _output_error;
-    }
-    {   ZSTD_CCtx* const cctx = ZSTD_createCCtx();
-        ZSTD_outBuffer output;
-        if (cctx==NULL) goto _output_error;
-        output.dst = compressedBuffer;
-        output.size = compressedBufferSize;
-        output.pos = 0;
-        CHECK_Z( ZSTD_initCStream(cctx, 1) );    /* content size unknown */
-        CHECK_Z( ZSTD_flushStream(cctx, &output) );   /* ensure no possibility to "concatenate" and determine the content size */
-        CHECK_Z( ZSTD_endStream(cctx, &output) );
-        ZSTD_freeCCtx(cctx);
-        /* single scan decompression */
-        {   size_t const r = ZSTD_decompress(NULL, 0, compressedBuffer, output.pos);
-            if (ZSTD_isError(r)) goto _output_error;
-            if (r != 0) goto _output_error;
-        }
-        /* streaming decompression */
-        {   ZSTD_DCtx* const dstream = ZSTD_createDStream();
-            ZSTD_inBuffer dinput;
-            ZSTD_outBuffer doutput;
-            size_t ipos;
-            if (dstream==NULL) goto _output_error;
-            dinput.src = compressedBuffer;
-            dinput.size = 0;
-            dinput.pos = 0;
-            doutput.dst = NULL;
-            doutput.size = 0;
-            doutput.pos = 0;
-            CHECK_Z ( ZSTD_initDStream(dstream) );
-            for (ipos=1; ipos<=output.pos; ipos++) {
-                dinput.size = ipos;
-                CHECK_Z ( ZSTD_decompressStream(dstream, &doutput, &dinput) );
-            }
-            if (doutput.pos != 0) goto _output_error;
-            ZSTD_freeDStream(dstream);
-        }
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3d : re-use CCtx with expanding block size : ", testNb++);
-    {   ZSTD_CCtx* const cctx = ZSTD_createCCtx();
-        ZSTD_parameters const params = ZSTD_getParams(1, ZSTD_CONTENTSIZE_UNKNOWN, 0);
-        assert(params.fParams.contentSizeFlag == 1);  /* block size will be adapted if pledgedSrcSize is enabled */
-        CHECK_Z( ZSTD_compressBegin_advanced(cctx, NULL, 0, params, 1 /*pledgedSrcSize*/) );
-        CHECK_Z( ZSTD_compressEnd(cctx, compressedBuffer, compressedBufferSize, CNBuffer, 1) ); /* creates a block size of 1 */
-
-        CHECK_Z( ZSTD_compressBegin_advanced(cctx, NULL, 0, params, ZSTD_CONTENTSIZE_UNKNOWN) );  /* re-use same parameters */
-        {   size_t const inSize = 2* 128 KB;
-            size_t const outSize = ZSTD_compressBound(inSize);
-            CHECK_Z( ZSTD_compressEnd(cctx, compressedBuffer, outSize, CNBuffer, inSize) );
-            /* will fail if blockSize is not resized */
-        }
-        ZSTD_freeCCtx(cctx);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3d : re-using a CCtx should compress the same : ", testNb++);
-    {   size_t const sampleSize = 30;
-        int i;
-        for (i=0; i<20; i++)
-            ((char*)CNBuffer)[i] = (char)i;   /* ensure no match during initial section */
-        memcpy((char*)CNBuffer + 20, CNBuffer, 10);   /* create one match, starting from beginning of sample, which is the difficult case (see #1241) */
-        for (i=1; i<=19; i++) {
-            ZSTD_CCtx* const cctx = ZSTD_createCCtx();
-            size_t size1, size2;
-            DISPLAYLEVEL(5, "l%i ", i);
-            size1 = ZSTD_compressCCtx(cctx, compressedBuffer, compressedBufferSize, CNBuffer, sampleSize, i);
-            CHECK_Z(size1);
-
-            size2 = ZSTD_compressCCtx(cctx, compressedBuffer, compressedBufferSize, CNBuffer, sampleSize, i);
-            CHECK_Z(size2);
-            CHECK_EQ(size1, size2);
-
-            CHECK_Z( ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, i) );
-            size2 = ZSTD_compress2(cctx, compressedBuffer, compressedBufferSize, CNBuffer, sampleSize);
-            CHECK_Z(size2);
-            CHECK_EQ(size1, size2);
-
-            size2 = ZSTD_compress2(cctx, compressedBuffer, ZSTD_compressBound(sampleSize) - 1, CNBuffer, sampleSize);  /* force streaming, as output buffer is not large enough to guarantee success */
-            CHECK_Z(size2);
-            CHECK_EQ(size1, size2);
-
-            {   ZSTD_inBuffer inb;
-                ZSTD_outBuffer outb;
-                inb.src = CNBuffer;
-                inb.pos = 0;
-                inb.size = sampleSize;
-                outb.dst = compressedBuffer;
-                outb.pos = 0;
-                outb.size = ZSTD_compressBound(sampleSize) - 1;  /* force streaming, as output buffer is not large enough to guarantee success */
-                CHECK_Z( ZSTD_compressStream2(cctx, &outb, &inb, ZSTD_e_end) );
-                assert(inb.pos == inb.size);
-                CHECK_EQ(size1, outb.pos);
-            }
-
-            ZSTD_freeCCtx(cctx);
-        }
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3d : btultra2 & 1st block : ", testNb++);
-    {   size_t const sampleSize = 1024;
-        ZSTD_CCtx* const cctx = ZSTD_createCCtx();
-        ZSTD_inBuffer inb;
-        ZSTD_outBuffer outb;
-        inb.src = CNBuffer;
-        inb.pos = 0;
-        inb.size = 0;
-        outb.dst = compressedBuffer;
-        outb.pos = 0;
-        outb.size = compressedBufferSize;
-        CHECK_Z( ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, ZSTD_maxCLevel()) );
-
-        inb.size = sampleSize;   /* start with something, so that context is already used */
-        CHECK_Z( ZSTD_compressStream2(cctx, &outb, &inb, ZSTD_e_end) );   /* will break internal assert if stats_init is not disabled */
-        assert(inb.pos == inb.size);
-        outb.pos = 0;     /* cancel output */
-
-        CHECK_Z( ZSTD_CCtx_setPledgedSrcSize(cctx, sampleSize) );
-        inb.size = 4;   /* too small size : compression will be skipped */
-        inb.pos = 0;
-        CHECK_Z( ZSTD_compressStream2(cctx, &outb, &inb, ZSTD_e_flush) );
-        assert(inb.pos == inb.size);
-
-        inb.size += 5;   /* too small size : compression will be skipped */
-        CHECK_Z( ZSTD_compressStream2(cctx, &outb, &inb, ZSTD_e_flush) );
-        assert(inb.pos == inb.size);
-
-        inb.size += 11;   /* small enough to attempt compression */
-        CHECK_Z( ZSTD_compressStream2(cctx, &outb, &inb, ZSTD_e_flush) );
-        assert(inb.pos == inb.size);
-
-        assert(inb.pos < sampleSize);
-        inb.size = sampleSize;   /* large enough to trigger stats_init, but no longer at beginning */
-        CHECK_Z( ZSTD_compressStream2(cctx, &outb, &inb, ZSTD_e_end) );   /* will break internal assert if stats_init is not disabled */
-        assert(inb.pos == inb.size);
-        ZSTD_freeCCtx(cctx);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3d : ZSTD_CCtx_getParameter() : ", testNb++);
-    {   ZSTD_CCtx* const cctx = ZSTD_createCCtx();
-        ZSTD_outBuffer out = {NULL, 0, 0};
-        ZSTD_inBuffer in = {NULL, 0, 0};
-        int value;
-
-        CHECK_Z(ZSTD_CCtx_getParameter(cctx, ZSTD_c_compressionLevel, &value));
-        CHECK_EQ(value, 3);
-        CHECK_Z(ZSTD_CCtx_getParameter(cctx, ZSTD_c_hashLog, &value));
-        CHECK_EQ(value, 0);
-        CHECK_Z(ZSTD_CCtx_setParameter(cctx, ZSTD_c_hashLog, ZSTD_HASHLOG_MIN));
-        CHECK_Z(ZSTD_CCtx_getParameter(cctx, ZSTD_c_compressionLevel, &value));
-        CHECK_EQ(value, 3);
-        CHECK_Z(ZSTD_CCtx_getParameter(cctx, ZSTD_c_hashLog, &value));
-        CHECK_EQ(value, ZSTD_HASHLOG_MIN);
-        CHECK_Z(ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, 7));
-        CHECK_Z(ZSTD_CCtx_getParameter(cctx, ZSTD_c_compressionLevel, &value));
-        CHECK_EQ(value, 7);
-        CHECK_Z(ZSTD_CCtx_getParameter(cctx, ZSTD_c_hashLog, &value));
-        CHECK_EQ(value, ZSTD_HASHLOG_MIN);
-        /* Start a compression job */
-        ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_continue);
-        CHECK_Z(ZSTD_CCtx_getParameter(cctx, ZSTD_c_compressionLevel, &value));
-        CHECK_EQ(value, 7);
-        CHECK_Z(ZSTD_CCtx_getParameter(cctx, ZSTD_c_hashLog, &value));
-        CHECK_EQ(value, ZSTD_HASHLOG_MIN);
-        /* Reset the CCtx */
-        ZSTD_CCtx_reset(cctx, ZSTD_reset_session_only);
-        CHECK_Z(ZSTD_CCtx_getParameter(cctx, ZSTD_c_compressionLevel, &value));
-        CHECK_EQ(value, 7);
-        CHECK_Z(ZSTD_CCtx_getParameter(cctx, ZSTD_c_hashLog, &value));
-        CHECK_EQ(value, ZSTD_HASHLOG_MIN);
-        /* Reset the parameters */
-        ZSTD_CCtx_reset(cctx, ZSTD_reset_parameters);
-        CHECK_Z(ZSTD_CCtx_getParameter(cctx, ZSTD_c_compressionLevel, &value));
-        CHECK_EQ(value, 3);
-        CHECK_Z(ZSTD_CCtx_getParameter(cctx, ZSTD_c_hashLog, &value));
-        CHECK_EQ(value, 0);
-
-        ZSTD_freeCCtx(cctx);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    /* this test is really too long, and should be made faster */
-    DISPLAYLEVEL(3, "test%3d : overflow protection with large windowLog : ", testNb++);
-    {   ZSTD_CCtx* const cctx = ZSTD_createCCtx();
-        ZSTD_parameters params = ZSTD_getParams(-999, ZSTD_CONTENTSIZE_UNKNOWN, 0);
-        size_t const nbCompressions = ((1U << 31) / CNBuffSize) + 2;   /* ensure U32 overflow protection is triggered */
-        size_t cnb;
-        assert(cctx != NULL);
-        params.fParams.contentSizeFlag = 0;
-        params.cParams.windowLog = ZSTD_WINDOWLOG_MAX;
-        for (cnb = 0; cnb < nbCompressions; ++cnb) {
-            DISPLAYLEVEL(6, "run %zu / %zu \n", cnb, nbCompressions);
-            CHECK_Z( ZSTD_compressBegin_advanced(cctx, NULL, 0, params, ZSTD_CONTENTSIZE_UNKNOWN) );  /* re-use same parameters */
-            CHECK_Z( ZSTD_compressEnd(cctx, compressedBuffer, compressedBufferSize, CNBuffer, CNBuffSize) );
-        }
-        ZSTD_freeCCtx(cctx);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3d : size down context : ", testNb++);
-    {   ZSTD_CCtx* const largeCCtx = ZSTD_createCCtx();
-        assert(largeCCtx != NULL);
-        CHECK_Z( ZSTD_compressBegin(largeCCtx, 19) );   /* streaming implies ZSTD_CONTENTSIZE_UNKNOWN, which maximizes memory usage */
-        CHECK_Z( ZSTD_compressEnd(largeCCtx, compressedBuffer, compressedBufferSize, CNBuffer, 1) );
-        {   size_t const largeCCtxSize = ZSTD_sizeof_CCtx(largeCCtx);   /* size of context must be measured after compression */
-            {   ZSTD_CCtx* const smallCCtx = ZSTD_createCCtx();
-                assert(smallCCtx != NULL);
-                CHECK_Z(ZSTD_compressCCtx(smallCCtx, compressedBuffer, compressedBufferSize, CNBuffer, 1, 1));
-                {   size_t const smallCCtxSize = ZSTD_sizeof_CCtx(smallCCtx);
-                    DISPLAYLEVEL(5, "(large) %zuKB > 32*%zuKB (small) : ",
-                                largeCCtxSize>>10, smallCCtxSize>>10);
-                    assert(largeCCtxSize > 32* smallCCtxSize);  /* note : "too large" definition is handled within zstd_compress.c .
-                                                                 * make this test case extreme, so that it doesn't depend on a possibly fluctuating definition */
-                }
-                ZSTD_freeCCtx(smallCCtx);
-            }
-            {   U32 const maxNbAttempts = 1100;   /* nb of usages before triggering size down is handled within zstd_compress.c.
-                                                   * currently defined as 128x, but could be adjusted in the future.
-                                                   * make this test long enough so that it's not too much tied to the current definition within zstd_compress.c */
-                unsigned u;
-                for (u=0; u<maxNbAttempts; u++) {
-                    CHECK_Z(ZSTD_compressCCtx(largeCCtx, compressedBuffer, compressedBufferSize, CNBuffer, 1, 1));
-                    if (ZSTD_sizeof_CCtx(largeCCtx) < largeCCtxSize) break;   /* sized down */
-                }
-                DISPLAYLEVEL(5, "size down after %u attempts : ", u);
-                if (u==maxNbAttempts) goto _output_error;   /* no sizedown happened */
-            }
-        }
-        ZSTD_freeCCtx(largeCCtx);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    /* Static CCtx tests */
-#define STATIC_CCTX_LEVEL 3
-    DISPLAYLEVEL(3, "test%3i : create static CCtx for level %u :", testNb++, STATIC_CCTX_LEVEL);
-    {   size_t const staticCCtxSize = ZSTD_estimateCStreamSize(STATIC_CCTX_LEVEL);
-        void* const staticCCtxBuffer = malloc(staticCCtxSize);
-        size_t const staticDCtxSize = ZSTD_estimateDCtxSize();
-        void* const staticDCtxBuffer = malloc(staticDCtxSize);
-        if (staticCCtxBuffer==NULL || staticDCtxBuffer==NULL) {
-            free(staticCCtxBuffer);
-            free(staticDCtxBuffer);
-            DISPLAY("Not enough memory, aborting\n");
-            testResult = 1;
-            goto _end;
-        }
-        {   ZSTD_CCtx* staticCCtx = ZSTD_initStaticCCtx(staticCCtxBuffer, staticCCtxSize);
-            ZSTD_DCtx* staticDCtx = ZSTD_initStaticDCtx(staticDCtxBuffer, staticDCtxSize);
-            if ((staticCCtx==NULL) || (staticDCtx==NULL)) goto _output_error;
-            DISPLAYLEVEL(3, "OK \n");
-
-            DISPLAYLEVEL(3, "test%3i : init CCtx for level %u : ", testNb++, STATIC_CCTX_LEVEL);
-            { size_t const r = ZSTD_compressBegin(staticCCtx, STATIC_CCTX_LEVEL);
-              if (ZSTD_isError(r)) goto _output_error; }
-            DISPLAYLEVEL(3, "OK \n");
-
-            DISPLAYLEVEL(3, "test%3i : simple compression test with static CCtx : ", testNb++);
-            CHECK_VAR(cSize, ZSTD_compressCCtx(staticCCtx,
-                                  compressedBuffer, compressedBufferSize,
-                                  CNBuffer, CNBuffSize, STATIC_CCTX_LEVEL) );
-            DISPLAYLEVEL(3, "OK (%u bytes : %.2f%%)\n",
-                            (unsigned)cSize, (double)cSize/CNBuffSize*100);
-
-            DISPLAYLEVEL(3, "test%3i : simple decompression test with static DCtx : ", testNb++);
-            { size_t const r = ZSTD_decompressDCtx(staticDCtx,
-                                                decodedBuffer, CNBuffSize,
-                                                compressedBuffer, cSize);
-              if (r != CNBuffSize) goto _output_error; }
-            DISPLAYLEVEL(3, "OK \n");
-
-            DISPLAYLEVEL(3, "test%3i : check decompressed result : ", testNb++);
-            {   size_t u;
-                for (u=0; u<CNBuffSize; u++) {
-                    if (((BYTE*)decodedBuffer)[u] != ((BYTE*)CNBuffer)[u])
-                        goto _output_error;
-            }   }
-            DISPLAYLEVEL(3, "OK \n");
-
-            DISPLAYLEVEL(3, "test%3i : init CCtx for too large level (must fail) : ", testNb++);
-            { size_t const r = ZSTD_compressBegin(staticCCtx, ZSTD_maxCLevel());
-              if (!ZSTD_isError(r)) goto _output_error; }
-            DISPLAYLEVEL(3, "OK \n");
-
-            DISPLAYLEVEL(3, "test%3i : init CCtx for small level %u (should work again) : ", testNb++, 1);
-            CHECK( ZSTD_compressBegin(staticCCtx, 1) );
-            DISPLAYLEVEL(3, "OK \n");
-
-            DISPLAYLEVEL(3, "test%3i : init CStream for small level %u : ", testNb++, 1);
-            CHECK( ZSTD_initCStream(staticCCtx, 1) );
-            DISPLAYLEVEL(3, "OK \n");
-
-            DISPLAYLEVEL(3, "test%3i : init static CStream with dictionary (should fail) : ", testNb++);
-            { size_t const r = ZSTD_initCStream_usingDict(staticCCtx, CNBuffer, 64 KB, 1);
-              if (!ZSTD_isError(r)) goto _output_error; }
-            DISPLAYLEVEL(3, "OK \n");
-
-            DISPLAYLEVEL(3, "test%3i : init DStream (should fail) : ", testNb++);
-            { size_t const r = ZSTD_initDStream(staticDCtx);
-              if (ZSTD_isError(r)) goto _output_error; }
-            {   ZSTD_outBuffer output = { decodedBuffer, CNBuffSize, 0 };
-                ZSTD_inBuffer input = { compressedBuffer, ZSTD_FRAMEHEADERSIZE_MAX+1, 0 };
-                size_t const r = ZSTD_decompressStream(staticDCtx, &output, &input);
-                if (!ZSTD_isError(r)) goto _output_error;
-            }
-            DISPLAYLEVEL(3, "OK \n");
-        }
-        free(staticCCtxBuffer);
-        free(staticDCtxBuffer);
-    }
-
-    DISPLAYLEVEL(3, "test%3i : Static negative levels : ", testNb++);
-    {   size_t const cctxSizeN1 = ZSTD_estimateCCtxSize(-1);
-        size_t const cctxSizeP1 = ZSTD_estimateCCtxSize(1);
-        size_t const cstreamSizeN1 = ZSTD_estimateCStreamSize(-1);
-        size_t const cstreamSizeP1 = ZSTD_estimateCStreamSize(1);
-
-        if (!(0 < cctxSizeN1 && cctxSizeN1 <= cctxSizeP1)) goto _output_error;
-        if (!(0 < cstreamSizeN1 && cstreamSizeN1 <= cstreamSizeP1)) goto _output_error;
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-
-    /* ZSTDMT simple MT compression test */
-    DISPLAYLEVEL(3, "test%3i : create ZSTDMT CCtx : ", testNb++);
-    {   ZSTDMT_CCtx* const mtctx = ZSTDMT_createCCtx(2);
-        if (mtctx==NULL) {
-            DISPLAY("mtctx : not enough memory, aborting \n");
-            testResult = 1;
-            goto _end;
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3u : compress %u bytes with 2 threads : ", testNb++, (unsigned)CNBuffSize);
-        CHECK_VAR(cSize, ZSTDMT_compressCCtx(mtctx,
-                                compressedBuffer, compressedBufferSize,
-                                CNBuffer, CNBuffSize,
-                                1) );
-        DISPLAYLEVEL(3, "OK (%u bytes : %.2f%%)\n", (unsigned)cSize, (double)cSize/CNBuffSize*100);
-
-        DISPLAYLEVEL(3, "test%3i : decompressed size test : ", testNb++);
-        {   unsigned long long const rSize = ZSTD_getFrameContentSize(compressedBuffer, cSize);
-            if (rSize != CNBuffSize)  {
-                DISPLAY("ZSTD_getFrameContentSize incorrect : %u != %u \n", (unsigned)rSize, (unsigned)CNBuffSize);
-                goto _output_error;
-        }   }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : decompress %u bytes : ", testNb++, (unsigned)CNBuffSize);
-        { size_t const r = ZSTD_decompress(decodedBuffer, CNBuffSize, compressedBuffer, cSize);
-          if (r != CNBuffSize) goto _output_error; }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : check decompressed result : ", testNb++);
-        {   size_t u;
-            for (u=0; u<CNBuffSize; u++) {
-                if (((BYTE*)decodedBuffer)[u] != ((BYTE*)CNBuffer)[u]) goto _output_error;
-        }   }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : compress -T2 with checksum : ", testNb++);
-        {   ZSTD_parameters params = ZSTD_getParams(1, CNBuffSize, 0);
-            params.fParams.checksumFlag = 1;
-            params.fParams.contentSizeFlag = 1;
-            CHECK_VAR(cSize, ZSTDMT_compress_advanced(mtctx,
-                                    compressedBuffer, compressedBufferSize,
-                                    CNBuffer, CNBuffSize,
-                                    NULL, params, 3 /*overlapRLog*/) );
-        }
-        DISPLAYLEVEL(3, "OK (%u bytes : %.2f%%)\n", (unsigned)cSize, (double)cSize/CNBuffSize*100);
-
-        DISPLAYLEVEL(3, "test%3i : decompress %u bytes : ", testNb++, (unsigned)CNBuffSize);
-        { size_t const r = ZSTD_decompress(decodedBuffer, CNBuffSize, compressedBuffer, cSize);
-          if (r != CNBuffSize) goto _output_error; }
-        DISPLAYLEVEL(3, "OK \n");
-
-        ZSTDMT_freeCCtx(mtctx);
-    }
-
-    DISPLAYLEVEL(3, "test%3i : compress -T2 with/without literals compression : ", testNb++)
-    {   ZSTD_CCtx* cctx = ZSTD_createCCtx();
-        size_t cSize1, cSize2;
-        CHECK( ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, 1) );
-        CHECK( ZSTD_CCtx_setParameter(cctx, ZSTD_c_nbWorkers, 2) );
-        cSize1 = ZSTD_compress2(cctx, compressedBuffer, compressedBufferSize, CNBuffer, CNBuffSize);
-        CHECK(cSize1);
-        CHECK( ZSTD_CCtx_setParameter(cctx, ZSTD_c_literalCompressionMode, ZSTD_lcm_uncompressed) );
-        cSize2 = ZSTD_compress2(cctx, compressedBuffer, compressedBufferSize, CNBuffer, CNBuffSize);
-        CHECK(cSize2);
-        CHECK_LT(cSize1, cSize2);
-        ZSTD_freeCCtx(cctx);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : Multithreaded ZSTD_compress2() with rsyncable : ", testNb++)
-    {   ZSTD_CCtx* cctx = ZSTD_createCCtx();
-        /* Set rsyncable and don't give the ZSTD_compressBound(CNBuffSize) so
-         * ZSTDMT is forced to not take the shortcut.
-         */
-        CHECK( ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, 1) );
-        CHECK( ZSTD_CCtx_setParameter(cctx, ZSTD_c_nbWorkers, 1) );
-        CHECK( ZSTD_CCtx_setParameter(cctx, ZSTD_c_rsyncable, 1) );
-        CHECK( ZSTD_compress2(cctx, compressedBuffer, compressedBufferSize - 1, CNBuffer, CNBuffSize) );
-        ZSTD_freeCCtx(cctx);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : setting multithreaded parameters : ", testNb++)
-    {   ZSTD_CCtx_params* params = ZSTD_createCCtxParams();
-        int value;
-        /* Check that the overlap log and job size are unset. */
-        CHECK( ZSTD_CCtxParams_getParameter(params, ZSTD_c_overlapLog, &value) );
-        CHECK_EQ(value, 0);
-        CHECK( ZSTD_CCtxParams_getParameter(params, ZSTD_c_jobSize, &value) );
-        CHECK_EQ(value, 0);
-        /* Set and check the overlap log and job size. */
-        CHECK( ZSTD_CCtxParams_setParameter(params, ZSTD_c_overlapLog, 5) );
-        CHECK( ZSTD_CCtxParams_setParameter(params, ZSTD_c_jobSize, 2 MB) );
-        CHECK( ZSTD_CCtxParams_getParameter(params, ZSTD_c_overlapLog, &value) );
-        CHECK_EQ(value, 5);
-        CHECK( ZSTD_CCtxParams_getParameter(params, ZSTD_c_jobSize, &value) );
-        CHECK_EQ(value, 2 MB);
-        /* Set the number of workers and check the overlap log and job size. */
-        CHECK( ZSTD_CCtxParams_setParameter(params, ZSTD_c_nbWorkers, 2) );
-        CHECK( ZSTD_CCtxParams_getParameter(params, ZSTD_c_overlapLog, &value) );
-        CHECK_EQ(value, 5);
-        CHECK( ZSTD_CCtxParams_getParameter(params, ZSTD_c_jobSize, &value) );
-        CHECK_EQ(value, 2 MB);
-        ZSTD_freeCCtxParams(params);
-
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    /* Simple API multiframe test */
-    DISPLAYLEVEL(3, "test%3i : compress multiple frames : ", testNb++);
-    {   size_t off = 0;
-        int i;
-        int const segs = 4;
-        /* only use the first half so we don't push against size limit of compressedBuffer */
-        size_t const segSize = (CNBuffSize / 2) / segs;
-        for (i = 0; i < segs; i++) {
-            CHECK_NEWV(r, ZSTD_compress(
-                            (BYTE*)compressedBuffer + off, CNBuffSize - off,
-                            (BYTE*)CNBuffer + segSize * (size_t)i, segSize,
-                            5) );
-            off += r;
-            if (i == segs/2) {
-                /* insert skippable frame */
-                const U32 skipLen = 129 KB;
-                MEM_writeLE32((BYTE*)compressedBuffer + off, ZSTD_MAGIC_SKIPPABLE_START);
-                MEM_writeLE32((BYTE*)compressedBuffer + off + 4, skipLen);
-                off += skipLen + ZSTD_SKIPPABLEHEADERSIZE;
-            }
-        }
-        cSize = off;
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : get decompressed size of multiple frames : ", testNb++);
-    {   unsigned long long const r = ZSTD_findDecompressedSize(compressedBuffer, cSize);
-        if (r != CNBuffSize / 2) goto _output_error; }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : get tight decompressed bound of multiple frames : ", testNb++);
-    {   unsigned long long const bound = ZSTD_decompressBound(compressedBuffer, cSize);
-        if (bound != CNBuffSize / 2) goto _output_error; }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : decompress multiple frames : ", testNb++);
-    {   CHECK_NEWV(r, ZSTD_decompress(decodedBuffer, CNBuffSize, compressedBuffer, cSize));
-        if (r != CNBuffSize / 2) goto _output_error; }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : check decompressed result : ", testNb++);
-    if (memcmp(decodedBuffer, CNBuffer, CNBuffSize / 2) != 0) goto _output_error;
-    DISPLAYLEVEL(3, "OK \n");
-
-    /* Dictionary and CCtx Duplication tests */
-    {   ZSTD_CCtx* const ctxOrig = ZSTD_createCCtx();
-        ZSTD_CCtx* const ctxDuplicated = ZSTD_createCCtx();
-        ZSTD_DCtx* const dctx = ZSTD_createDCtx();
-        static const size_t dictSize = 551;
-        assert(dctx != NULL); assert(ctxOrig != NULL); assert(ctxDuplicated != NULL);
-
-        DISPLAYLEVEL(3, "test%3i : copy context too soon : ", testNb++);
-        { size_t const copyResult = ZSTD_copyCCtx(ctxDuplicated, ctxOrig, 0);
-          if (!ZSTD_isError(copyResult)) goto _output_error; }   /* error must be detected */
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : load dictionary into context : ", testNb++);
-        CHECK( ZSTD_compressBegin_usingDict(ctxOrig, CNBuffer, dictSize, 2) );
-        CHECK( ZSTD_copyCCtx(ctxDuplicated, ctxOrig, 0) ); /* Begin_usingDict implies unknown srcSize, so match that */
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : compress with flat dictionary : ", testNb++);
-        cSize = 0;
-        CHECKPLUS(r, ZSTD_compressEnd(ctxOrig,
-                                      compressedBuffer, compressedBufferSize,
-                         (const char*)CNBuffer + dictSize, CNBuffSize - dictSize),
-                  cSize += r);
-        DISPLAYLEVEL(3, "OK (%u bytes : %.2f%%)\n", (unsigned)cSize, (double)cSize/CNBuffSize*100);
-
-        DISPLAYLEVEL(3, "test%3i : frame built with flat dictionary should be decompressible : ", testNb++);
-        CHECKPLUS(r, ZSTD_decompress_usingDict(dctx,
-                                       decodedBuffer, CNBuffSize,
-                                       compressedBuffer, cSize,
-                                       CNBuffer, dictSize),
-                  if (r != CNBuffSize - dictSize) goto _output_error);
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : compress with duplicated context : ", testNb++);
-        {   size_t const cSizeOrig = cSize;
-            cSize = 0;
-            CHECKPLUS(r, ZSTD_compressEnd(ctxDuplicated,
-                                    compressedBuffer, compressedBufferSize,
-                       (const char*)CNBuffer + dictSize, CNBuffSize - dictSize),
-                      cSize += r);
-            if (cSize != cSizeOrig) goto _output_error;   /* should be identical ==> same size */
-        }
-        DISPLAYLEVEL(3, "OK (%u bytes : %.2f%%)\n", (unsigned)cSize, (double)cSize/CNBuffSize*100);
-
-        DISPLAYLEVEL(3, "test%3i : frame built with duplicated context should be decompressible : ", testNb++);
-        CHECKPLUS(r, ZSTD_decompress_usingDict(dctx,
-                                           decodedBuffer, CNBuffSize,
-                                           compressedBuffer, cSize,
-                                           CNBuffer, dictSize),
-                  if (r != CNBuffSize - dictSize) goto _output_error);
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : decompress with DDict : ", testNb++);
-        {   ZSTD_DDict* const ddict = ZSTD_createDDict(CNBuffer, dictSize);
-            size_t const r = ZSTD_decompress_usingDDict(dctx, decodedBuffer, CNBuffSize, compressedBuffer, cSize, ddict);
-            if (r != CNBuffSize - dictSize) goto _output_error;
-            DISPLAYLEVEL(3, "OK (size of DDict : %u) \n", (unsigned)ZSTD_sizeof_DDict(ddict));
-            ZSTD_freeDDict(ddict);
-        }
-
-        DISPLAYLEVEL(3, "test%3i : decompress with static DDict : ", testNb++);
-        {   size_t const ddictBufferSize = ZSTD_estimateDDictSize(dictSize, ZSTD_dlm_byCopy);
-            void* const ddictBuffer = malloc(ddictBufferSize);
-            if (ddictBuffer == NULL) goto _output_error;
-            {   const ZSTD_DDict* const ddict = ZSTD_initStaticDDict(ddictBuffer, ddictBufferSize, CNBuffer, dictSize, ZSTD_dlm_byCopy, ZSTD_dct_auto);
-                size_t const r = ZSTD_decompress_usingDDict(dctx, decodedBuffer, CNBuffSize, compressedBuffer, cSize, ddict);
-                if (r != CNBuffSize - dictSize) goto _output_error;
-            }
-            free(ddictBuffer);
-            DISPLAYLEVEL(3, "OK (size of static DDict : %u) \n", (unsigned)ddictBufferSize);
-        }
-
-        DISPLAYLEVEL(3, "test%3i : check content size on duplicated context : ", testNb++);
-        {   size_t const testSize = CNBuffSize / 3;
-            {   ZSTD_parameters p = ZSTD_getParams(2, testSize, dictSize);
-                p.fParams.contentSizeFlag = 1;
-                CHECK( ZSTD_compressBegin_advanced(ctxOrig, CNBuffer, dictSize, p, testSize-1) );
-            }
-            CHECK( ZSTD_copyCCtx(ctxDuplicated, ctxOrig, testSize) );
-
-            CHECK_VAR(cSize, ZSTD_compressEnd(ctxDuplicated, compressedBuffer, ZSTD_compressBound(testSize),
-                                          (const char*)CNBuffer + dictSize, testSize) );
-            {   ZSTD_frameHeader zfh;
-                if (ZSTD_getFrameHeader(&zfh, compressedBuffer, cSize)) goto _output_error;
-                if ((zfh.frameContentSize != testSize) && (zfh.frameContentSize != 0)) goto _output_error;
-        }   }
-        DISPLAYLEVEL(3, "OK \n");
-
-        if ((int)(compressibility * 100 + 0.1) == FUZ_compressibility_default) { /* test only valid with known input */
-            size_t const flatdictSize = 22 KB;
-            size_t const contentSize = 9 KB;
-            const void* const dict = (const char*)CNBuffer;
-            const void* const contentStart = (const char*)dict + flatdictSize;
-            size_t const target_nodict_cSize[22+1] = { 3840, 3770, 3870, 3830, 3770,
-                                                       3770, 3770, 3770, 3750, 3750,
-                                                       3740, 3670, 3670, 3660, 3660,
-                                                       3660, 3660, 3660, 3660, 3660,
-                                                       3660, 3660, 3660 };
-            size_t const target_wdict_cSize[22+1] =  { 2830, 2890, 2890, 2820, 2940,
-                                                       2950, 2950, 2920, 2900, 2890,
-                                                       2910, 2910, 2910, 2770, 2760,
-                                                       2750, 2750, 2750, 2750, 2750,
-                                                       2750, 2750, 2750 };
-            int l = 1;
-            int const maxLevel = ZSTD_maxCLevel();
-
-            DISPLAYLEVEL(3, "test%3i : flat-dictionary efficiency test : \n", testNb++);
-            assert(maxLevel == 22);
-            RDG_genBuffer(CNBuffer, flatdictSize + contentSize, compressibility, 0., seed);
-            DISPLAYLEVEL(4, "content hash : %016llx;  dict hash : %016llx \n", XXH64(contentStart, contentSize, 0), XXH64(dict, flatdictSize, 0));
-
-            for ( ; l <= maxLevel; l++) {
-                size_t const nodict_cSize = ZSTD_compress(compressedBuffer, compressedBufferSize,
-                                                          contentStart, contentSize, l);
-                if (nodict_cSize > target_nodict_cSize[l]) {
-                    DISPLAYLEVEL(1, "error : compression at level %i worse than expected (%u > %u) \n",
-                                    l, (unsigned)nodict_cSize, (unsigned)target_nodict_cSize[l]);
-                    goto _output_error;
-                }
-                DISPLAYLEVEL(4, "level %i : max expected %u >= reached %u \n",
-                                l, (unsigned)target_nodict_cSize[l], (unsigned)nodict_cSize);
-            }
-            for ( l=1 ; l <= maxLevel; l++) {
-                size_t const wdict_cSize = ZSTD_compress_usingDict(ctxOrig,
-                                                          compressedBuffer, compressedBufferSize,
-                                                          contentStart, contentSize,
-                                                          dict, flatdictSize,
-                                                          l);
-                if (wdict_cSize > target_wdict_cSize[l]) {
-                    DISPLAYLEVEL(1, "error : compression with dictionary at level %i worse than expected (%u > %u) \n",
-                                    l, (unsigned)wdict_cSize, (unsigned)target_wdict_cSize[l]);
-                    goto _output_error;
-                }
-                DISPLAYLEVEL(4, "level %i with dictionary : max expected %u >= reached %u \n",
-                                l, (unsigned)target_wdict_cSize[l], (unsigned)wdict_cSize);
-            }
-
-            DISPLAYLEVEL(4, "compression efficiency tests OK \n");
-        }
-
-        ZSTD_freeCCtx(ctxOrig);
-        ZSTD_freeCCtx(ctxDuplicated);
-        ZSTD_freeDCtx(dctx);
-    }
-
-    /* Dictionary and dictBuilder tests */
-    {   ZSTD_CCtx* const cctx = ZSTD_createCCtx();
-        size_t const dictBufferCapacity = 16 KB;
-        void* const dictBuffer = malloc(dictBufferCapacity);
-        size_t const totalSampleSize = 1 MB;
-        size_t const sampleUnitSize = 8 KB;
-        U32 const nbSamples = (U32)(totalSampleSize / sampleUnitSize);
-        size_t* const samplesSizes = (size_t*) malloc(nbSamples * sizeof(size_t));
-        size_t dictSize;
-        U32 dictID;
-
-        if (dictBuffer==NULL || samplesSizes==NULL) {
-            free(dictBuffer);
-            free(samplesSizes);
-            goto _output_error;
-        }
-
-        DISPLAYLEVEL(3, "test%3i : dictBuilder on cyclic data : ", testNb++);
-        assert(compressedBufferSize >= totalSampleSize);
-        { U32 u; for (u=0; u<totalSampleSize; u++) ((BYTE*)decodedBuffer)[u] = (BYTE)u; }
-        { U32 u; for (u=0; u<nbSamples; u++) samplesSizes[u] = sampleUnitSize; }
-        {   size_t const sDictSize = ZDICT_trainFromBuffer(dictBuffer, dictBufferCapacity,
-                                         decodedBuffer, samplesSizes, nbSamples);
-            if (ZDICT_isError(sDictSize)) goto _output_error;
-            DISPLAYLEVEL(3, "OK, created dictionary of size %u \n", (unsigned)sDictSize);
-        }
-
-        DISPLAYLEVEL(3, "test%3i : dictBuilder : ", testNb++);
-        { U32 u; for (u=0; u<nbSamples; u++) samplesSizes[u] = sampleUnitSize; }
-        dictSize = ZDICT_trainFromBuffer(dictBuffer, dictBufferCapacity,
-                                         CNBuffer, samplesSizes, nbSamples);
-        if (ZDICT_isError(dictSize)) goto _output_error;
-        DISPLAYLEVEL(3, "OK, created dictionary of size %u \n", (unsigned)dictSize);
-
-        DISPLAYLEVEL(3, "test%3i : Multithreaded COVER dictBuilder : ", testNb++);
-        { U32 u; for (u=0; u<nbSamples; u++) samplesSizes[u] = sampleUnitSize; }
-        {   ZDICT_cover_params_t coverParams;
-            memset(&coverParams, 0, sizeof(coverParams));
-            coverParams.steps = 8;
-            coverParams.nbThreads = 4;
-            dictSize = ZDICT_optimizeTrainFromBuffer_cover(
-                dictBuffer, dictBufferCapacity,
-                CNBuffer, samplesSizes, nbSamples/8,  /* less samples for faster tests */
-                &coverParams);
-            if (ZDICT_isError(dictSize)) goto _output_error;
-        }
-        DISPLAYLEVEL(3, "OK, created dictionary of size %u \n", (unsigned)dictSize);
-
-        DISPLAYLEVEL(3, "test%3i : COVER dictBuilder with shrinkDict: ", testNb++);
-        { U32 u; for (u=0; u<nbSamples; u++) samplesSizes[u] = sampleUnitSize; }
-        {   ZDICT_cover_params_t coverParams;
-            memset(&coverParams, 0, sizeof(coverParams));
-            coverParams.steps = 8;
-            coverParams.nbThreads = 4;
-            coverParams.shrinkDict = 1;
-            coverParams.shrinkDictMaxRegression = 1;
-            dictSize = ZDICT_optimizeTrainFromBuffer_cover(
-                dictBuffer, dictBufferCapacity,
-                CNBuffer, samplesSizes, nbSamples/8,  /* less samples for faster tests */
-                &coverParams);
-            if (ZDICT_isError(dictSize)) goto _output_error;
-        }
-        DISPLAYLEVEL(3, "OK, created dictionary of size %u \n", (unsigned)dictSize);
-
-        DISPLAYLEVEL(3, "test%3i : Multithreaded FASTCOVER dictBuilder : ", testNb++);
-        { U32 u; for (u=0; u<nbSamples; u++) samplesSizes[u] = sampleUnitSize; }
-        {   ZDICT_fastCover_params_t fastCoverParams;
-            memset(&fastCoverParams, 0, sizeof(fastCoverParams));
-            fastCoverParams.steps = 8;
-            fastCoverParams.nbThreads = 4;
-            dictSize = ZDICT_optimizeTrainFromBuffer_fastCover(
-                dictBuffer, dictBufferCapacity,
-                CNBuffer, samplesSizes, nbSamples,
-                &fastCoverParams);
-            if (ZDICT_isError(dictSize)) goto _output_error;
-        }
-        DISPLAYLEVEL(3, "OK, created dictionary of size %u \n", (unsigned)dictSize);
-
-        DISPLAYLEVEL(3, "test%3i : FASTCOVER dictBuilder with shrinkDict: ", testNb++);
-        { U32 u; for (u=0; u<nbSamples; u++) samplesSizes[u] = sampleUnitSize; }
-        {   ZDICT_fastCover_params_t fastCoverParams;
-            memset(&fastCoverParams, 0, sizeof(fastCoverParams));
-            fastCoverParams.steps = 8;
-            fastCoverParams.nbThreads = 4;
-            fastCoverParams.shrinkDict = 1;
-            fastCoverParams.shrinkDictMaxRegression = 1;
-            dictSize = ZDICT_optimizeTrainFromBuffer_fastCover(
-                dictBuffer, dictBufferCapacity,
-                CNBuffer, samplesSizes, nbSamples,
-                &fastCoverParams);
-            if (ZDICT_isError(dictSize)) goto _output_error;
-        }
-        DISPLAYLEVEL(3, "OK, created dictionary of size %u \n", (unsigned)dictSize);
-
-        DISPLAYLEVEL(3, "test%3i : check dictID : ", testNb++);
-        dictID = ZDICT_getDictID(dictBuffer, dictSize);
-        if (dictID==0) goto _output_error;
-        DISPLAYLEVEL(3, "OK : %u \n", (unsigned)dictID);
-
-        DISPLAYLEVEL(3, "test%3i : compress with dictionary : ", testNb++);
-        cSize = ZSTD_compress_usingDict(cctx, compressedBuffer, compressedBufferSize,
-                                        CNBuffer, CNBuffSize,
-                                        dictBuffer, dictSize, 4);
-        if (ZSTD_isError(cSize)) goto _output_error;
-        DISPLAYLEVEL(3, "OK (%u bytes : %.2f%%)\n", (unsigned)cSize, (double)cSize/CNBuffSize*100);
-
-        DISPLAYLEVEL(3, "test%3i : retrieve dictID from dictionary : ", testNb++);
-        {   U32 const did = ZSTD_getDictID_fromDict(dictBuffer, dictSize);
-            if (did != dictID) goto _output_error;   /* non-conformant (content-only) dictionary */
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : retrieve dictID from frame : ", testNb++);
-        {   U32 const did = ZSTD_getDictID_fromFrame(compressedBuffer, cSize);
-            if (did != dictID) goto _output_error;   /* non-conformant (content-only) dictionary */
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : frame built with dictionary should be decompressible : ", testNb++);
-        {   ZSTD_DCtx* const dctx = ZSTD_createDCtx(); assert(dctx != NULL);
-            CHECKPLUS(r, ZSTD_decompress_usingDict(dctx,
-                                           decodedBuffer, CNBuffSize,
-                                           compressedBuffer, cSize,
-                                           dictBuffer, dictSize),
-                      if (r != CNBuffSize) goto _output_error);
-            ZSTD_freeDCtx(dctx);
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : estimate CDict size : ", testNb++);
-        {   ZSTD_compressionParameters const cParams = ZSTD_getCParams(1, CNBuffSize, dictSize);
-            size_t const estimatedSize = ZSTD_estimateCDictSize_advanced(dictSize, cParams, ZSTD_dlm_byRef);
-            DISPLAYLEVEL(3, "OK : %u \n", (unsigned)estimatedSize);
-        }
-
-        DISPLAYLEVEL(3, "test%3i : compress with CDict ", testNb++);
-        {   ZSTD_compressionParameters const cParams = ZSTD_getCParams(1, CNBuffSize, dictSize);
-            ZSTD_CDict* const cdict = ZSTD_createCDict_advanced(dictBuffer, dictSize,
-                                            ZSTD_dlm_byRef, ZSTD_dct_auto,
-                                            cParams, ZSTD_defaultCMem);
-            assert(cdict != NULL);
-            DISPLAYLEVEL(3, "(size : %u) : ", (unsigned)ZSTD_sizeof_CDict(cdict));
-            cSize = ZSTD_compress_usingCDict(cctx, compressedBuffer, compressedBufferSize,
-                                                 CNBuffer, CNBuffSize, cdict);
-            ZSTD_freeCDict(cdict);
-            if (ZSTD_isError(cSize)) goto _output_error;
-        }
-        DISPLAYLEVEL(3, "OK (%u bytes : %.2f%%)\n", (unsigned)cSize, (double)cSize/CNBuffSize*100);
-
-        DISPLAYLEVEL(3, "test%3i : retrieve dictID from frame : ", testNb++);
-        {   U32 const did = ZSTD_getDictID_fromFrame(compressedBuffer, cSize);
-            if (did != dictID) goto _output_error;   /* non-conformant (content-only) dictionary */
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : frame built with dictionary should be decompressible : ", testNb++);
-        {   ZSTD_DCtx* const dctx = ZSTD_createDCtx(); assert(dctx != NULL);
-            CHECKPLUS(r, ZSTD_decompress_usingDict(dctx,
-                                           decodedBuffer, CNBuffSize,
-                                           compressedBuffer, cSize,
-                                           dictBuffer, dictSize),
-                      if (r != CNBuffSize) goto _output_error);
-            ZSTD_freeDCtx(dctx);
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : compress with static CDict : ", testNb++);
-        {   int const maxLevel = ZSTD_maxCLevel();
-            int level;
-            for (level = 1; level <= maxLevel; ++level) {
-                ZSTD_compressionParameters const cParams = ZSTD_getCParams(level, CNBuffSize, dictSize);
-                size_t const cdictSize = ZSTD_estimateCDictSize_advanced(dictSize, cParams, ZSTD_dlm_byCopy);
-                void* const cdictBuffer = malloc(cdictSize);
-                if (cdictBuffer==NULL) goto _output_error;
-                {   const ZSTD_CDict* const cdict = ZSTD_initStaticCDict(
-                                                cdictBuffer, cdictSize,
-                                                dictBuffer, dictSize,
-                                                ZSTD_dlm_byCopy, ZSTD_dct_auto,
-                                                cParams);
-                    if (cdict == NULL) {
-                        DISPLAY("ZSTD_initStaticCDict failed ");
-                        goto _output_error;
-                    }
-                    cSize = ZSTD_compress_usingCDict(cctx,
-                                    compressedBuffer, compressedBufferSize,
-                                    CNBuffer, MIN(10 KB, CNBuffSize), cdict);
-                    if (ZSTD_isError(cSize)) {
-                        DISPLAY("ZSTD_compress_usingCDict failed ");
-                        goto _output_error;
-                }   }
-                free(cdictBuffer);
-        }   }
-        DISPLAYLEVEL(3, "OK (%u bytes : %.2f%%)\n", (unsigned)cSize, (double)cSize/CNBuffSize*100);
-
-        DISPLAYLEVEL(3, "test%3i : ZSTD_compress_usingCDict_advanced, no contentSize, no dictID : ", testNb++);
-        {   ZSTD_frameParameters const fParams = { 0 /* frameSize */, 1 /* checksum */, 1 /* noDictID*/ };
-            ZSTD_compressionParameters const cParams = ZSTD_getCParams(1, CNBuffSize, dictSize);
-            ZSTD_CDict* const cdict = ZSTD_createCDict_advanced(dictBuffer, dictSize, ZSTD_dlm_byRef, ZSTD_dct_auto, cParams, ZSTD_defaultCMem);
-            assert(cdict != NULL);
-            cSize = ZSTD_compress_usingCDict_advanced(cctx,
-                                                      compressedBuffer, compressedBufferSize,
-                                                      CNBuffer, CNBuffSize,
-                                                      cdict, fParams);
-            ZSTD_freeCDict(cdict);
-            if (ZSTD_isError(cSize)) goto _output_error;
-        }
-        DISPLAYLEVEL(3, "OK (%u bytes : %.2f%%)\n", (unsigned)cSize, (double)cSize/CNBuffSize*100);
-
-        DISPLAYLEVEL(3, "test%3i : try retrieving contentSize from frame : ", testNb++);
-        {   U64 const contentSize = ZSTD_getFrameContentSize(compressedBuffer, cSize);
-            if (contentSize != ZSTD_CONTENTSIZE_UNKNOWN) goto _output_error;
-        }
-        DISPLAYLEVEL(3, "OK (unknown)\n");
-
-        DISPLAYLEVEL(3, "test%3i : frame built without dictID should be decompressible : ", testNb++);
-        {   ZSTD_DCtx* const dctx = ZSTD_createDCtx();
-            assert(dctx != NULL);
-            CHECKPLUS(r, ZSTD_decompress_usingDict(dctx,
-                                           decodedBuffer, CNBuffSize,
-                                           compressedBuffer, cSize,
-                                           dictBuffer, dictSize),
-                      if (r != CNBuffSize) goto _output_error);
-            ZSTD_freeDCtx(dctx);
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : ZSTD_compress_advanced, no dictID : ", testNb++);
-        {   ZSTD_parameters p = ZSTD_getParams(3, CNBuffSize, dictSize);
-            p.fParams.noDictIDFlag = 1;
-            cSize = ZSTD_compress_advanced(cctx, compressedBuffer, compressedBufferSize,
-                                           CNBuffer, CNBuffSize,
-                                           dictBuffer, dictSize, p);
-            if (ZSTD_isError(cSize)) goto _output_error;
-        }
-        DISPLAYLEVEL(3, "OK (%u bytes : %.2f%%)\n", (unsigned)cSize, (double)cSize/CNBuffSize*100);
-
-        DISPLAYLEVEL(3, "test%3i : frame built without dictID should be decompressible : ", testNb++);
-        {   ZSTD_DCtx* const dctx = ZSTD_createDCtx(); assert(dctx != NULL);
-            CHECKPLUS(r, ZSTD_decompress_usingDict(dctx,
-                                           decodedBuffer, CNBuffSize,
-                                           compressedBuffer, cSize,
-                                           dictBuffer, dictSize),
-                      if (r != CNBuffSize) goto _output_error);
-            ZSTD_freeDCtx(dctx);
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : dictionary containing only header should return error : ", testNb++);
-        {   ZSTD_DCtx* const dctx = ZSTD_createDCtx();
-            assert(dctx != NULL);
-            {   const size_t ret = ZSTD_decompress_usingDict(
-                    dctx, decodedBuffer, CNBuffSize, compressedBuffer, cSize,
-                    "\x37\xa4\x30\xec\x11\x22\x33\x44", 8);
-                if (ZSTD_getErrorCode(ret) != ZSTD_error_dictionary_corrupted)
-                    goto _output_error;
-            }
-            ZSTD_freeDCtx(dctx);
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : Building cdict w/ ZSTD_dct_fullDict on a good dictionary : ", testNb++);
-        {   ZSTD_compressionParameters const cParams = ZSTD_getCParams(1, CNBuffSize, dictSize);
-            ZSTD_CDict* const cdict = ZSTD_createCDict_advanced(dictBuffer, dictSize, ZSTD_dlm_byRef, ZSTD_dct_fullDict, cParams, ZSTD_defaultCMem);
-            if (cdict==NULL) goto _output_error;
-            ZSTD_freeCDict(cdict);
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : Building cdict w/ ZSTD_dct_fullDict on a rawContent (must fail) : ", testNb++);
-        {   ZSTD_compressionParameters const cParams = ZSTD_getCParams(1, CNBuffSize, dictSize);
-            ZSTD_CDict* const cdict = ZSTD_createCDict_advanced((const char*)dictBuffer+1, dictSize-1, ZSTD_dlm_byRef, ZSTD_dct_fullDict, cParams, ZSTD_defaultCMem);
-            if (cdict!=NULL) goto _output_error;
-            ZSTD_freeCDict(cdict);
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : Loading rawContent starting with dict header w/ ZSTD_dct_auto should fail : ", testNb++);
-        {
-            size_t ret;
-            MEM_writeLE32((char*)dictBuffer+2, ZSTD_MAGIC_DICTIONARY);
-            /* Either operation is allowed to fail, but one must fail. */
-            ret = ZSTD_CCtx_loadDictionary_advanced(
-                    cctx, (const char*)dictBuffer+2, dictSize-2, ZSTD_dlm_byRef, ZSTD_dct_auto);
-            if (!ZSTD_isError(ret)) {
-                ret = ZSTD_compress2(cctx, compressedBuffer, compressedBufferSize, CNBuffer, MIN(CNBuffSize, 100));
-                if (!ZSTD_isError(ret)) goto _output_error;
-            }
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : Loading rawContent starting with dict header w/ ZSTD_dct_rawContent should pass : ", testNb++);
-        {
-            size_t ret;
-            MEM_writeLE32((char*)dictBuffer+2, ZSTD_MAGIC_DICTIONARY);
-            ret = ZSTD_CCtx_loadDictionary_advanced(
-                    cctx, (const char*)dictBuffer+2, dictSize-2, ZSTD_dlm_byRef, ZSTD_dct_rawContent);
-            if (ZSTD_isError(ret)) goto _output_error;
-            ret = ZSTD_compress2(cctx, compressedBuffer, compressedBufferSize, CNBuffer, MIN(CNBuffSize, 100));
-            if (ZSTD_isError(ret)) goto _output_error;
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : ZSTD_CCtx_refCDict() then set parameters : ", testNb++);
-        {   ZSTD_CDict* const cdict = ZSTD_createCDict(CNBuffer, dictSize, 1);
-            CHECK_Z( ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, 1) );
-            CHECK_Z( ZSTD_CCtx_setParameter(cctx, ZSTD_c_hashLog, 12 ));
-            CHECK_Z( ZSTD_CCtx_refCDict(cctx, cdict) );
-            CHECK_Z( ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, 1) );
-            CHECK_Z( ZSTD_CCtx_setParameter(cctx, ZSTD_c_hashLog, 12 ));
-            ZSTD_freeCDict(cdict);
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : Loading dictionary before setting parameters is the same as loading after : ", testNb++);
-        {
-            size_t size1, size2;
-            ZSTD_CCtx_reset(cctx, ZSTD_reset_session_and_parameters);
-            CHECK_Z( ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, 7) );
-            CHECK_Z( ZSTD_CCtx_loadDictionary(cctx, CNBuffer, MIN(CNBuffSize, 10 KB)) );
-            size1 = ZSTD_compress2(cctx, compressedBuffer, compressedBufferSize, CNBuffer, MIN(CNBuffSize, 100 KB));
-            if (ZSTD_isError(size1)) goto _output_error;
-
-            ZSTD_CCtx_reset(cctx, ZSTD_reset_session_and_parameters);
-            CHECK_Z( ZSTD_CCtx_loadDictionary(cctx, CNBuffer, MIN(CNBuffSize, 10 KB)) );
-            CHECK_Z( ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, 7) );
-            size2 = ZSTD_compress2(cctx, compressedBuffer, compressedBufferSize, CNBuffer, MIN(CNBuffSize, 100 KB));
-            if (ZSTD_isError(size2)) goto _output_error;
-
-            if (size1 != size2) goto _output_error;
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : Loading a dictionary clears the prefix : ", testNb++);
-        {
-            CHECK_Z( ZSTD_CCtx_refPrefix(cctx, (const char*)dictBuffer, dictSize) );
-            CHECK_Z( ZSTD_CCtx_loadDictionary(cctx, (const char*)dictBuffer, dictSize) );
-            CHECK_Z( ZSTD_compress2(cctx, compressedBuffer, compressedBufferSize, CNBuffer, MIN(CNBuffSize, 100)) );
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : Loading a dictionary clears the cdict : ", testNb++);
-        {
-            ZSTD_CDict* const cdict = ZSTD_createCDict(dictBuffer, dictSize, 1);
-            CHECK_Z( ZSTD_CCtx_refCDict(cctx, cdict) );
-            CHECK_Z( ZSTD_CCtx_loadDictionary(cctx, (const char*)dictBuffer, dictSize) );
-            CHECK_Z( ZSTD_compress2(cctx, compressedBuffer, compressedBufferSize, CNBuffer, MIN(CNBuffSize, 100)) );
-            ZSTD_freeCDict(cdict);
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : Loading a cdict clears the prefix : ", testNb++);
-        {
-            ZSTD_CDict* const cdict = ZSTD_createCDict(dictBuffer, dictSize, 1);
-            CHECK_Z( ZSTD_CCtx_refPrefix(cctx, (const char*)dictBuffer, dictSize) );
-            CHECK_Z( ZSTD_CCtx_refCDict(cctx, cdict) );
-            CHECK_Z( ZSTD_compress2(cctx, compressedBuffer, compressedBufferSize, CNBuffer, MIN(CNBuffSize, 100)) );
-            ZSTD_freeCDict(cdict);
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : Loading a cdict clears the dictionary : ", testNb++);
-        {
-            ZSTD_CDict* const cdict = ZSTD_createCDict(dictBuffer, dictSize, 1);
-            CHECK_Z( ZSTD_CCtx_loadDictionary(cctx, (const char*)dictBuffer, dictSize) );
-            CHECK_Z( ZSTD_CCtx_refCDict(cctx, cdict) );
-            CHECK_Z( ZSTD_compress2(cctx, compressedBuffer, compressedBufferSize, CNBuffer, MIN(CNBuffSize, 100)) );
-            ZSTD_freeCDict(cdict);
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : Loading a prefix clears the dictionary : ", testNb++);
-        {
-            CHECK_Z( ZSTD_CCtx_loadDictionary(cctx, (const char*)dictBuffer, dictSize) );
-            CHECK_Z( ZSTD_CCtx_refPrefix(cctx, (const char*)dictBuffer, dictSize) );
-            CHECK_Z( ZSTD_compress2(cctx, compressedBuffer, compressedBufferSize, CNBuffer, MIN(CNBuffSize, 100)) );
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : Loading a prefix clears the cdict : ", testNb++);
-        {
-            ZSTD_CDict* const cdict = ZSTD_createCDict(dictBuffer, dictSize, 1);
-            CHECK_Z( ZSTD_CCtx_refCDict(cctx, cdict) );
-            CHECK_Z( ZSTD_CCtx_refPrefix(cctx, (const char*)dictBuffer, dictSize) );
-            CHECK_Z( ZSTD_compress2(cctx, compressedBuffer, compressedBufferSize, CNBuffer, MIN(CNBuffSize, 100)) );
-            ZSTD_freeCDict(cdict);
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : Loaded dictionary persists across reset session : ", testNb++);
-        {
-            size_t size1, size2;
-            ZSTD_CCtx_reset(cctx, ZSTD_reset_session_and_parameters);
-            CHECK_Z( ZSTD_CCtx_loadDictionary(cctx, CNBuffer, MIN(CNBuffSize, 10 KB)) );
-            size1 = ZSTD_compress2(cctx, compressedBuffer, compressedBufferSize, CNBuffer, MIN(CNBuffSize, 100 KB));
-            if (ZSTD_isError(size1)) goto _output_error;
-
-            ZSTD_CCtx_reset(cctx, ZSTD_reset_session_only);
-            size2 = ZSTD_compress2(cctx, compressedBuffer, compressedBufferSize, CNBuffer, MIN(CNBuffSize, 100 KB));
-            if (ZSTD_isError(size2)) goto _output_error;
-
-            if (size1 != size2) goto _output_error;
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : Loaded dictionary is cleared after resetting parameters : ", testNb++);
-        {
-            size_t size1, size2;
-            ZSTD_CCtx_reset(cctx, ZSTD_reset_session_and_parameters);
-            CHECK_Z( ZSTD_CCtx_loadDictionary(cctx, CNBuffer, MIN(CNBuffSize, 10 KB)) );
-            size1 = ZSTD_compress2(cctx, compressedBuffer, compressedBufferSize, CNBuffer, MIN(CNBuffSize, 100 KB));
-            if (ZSTD_isError(size1)) goto _output_error;
-
-            ZSTD_CCtx_reset(cctx, ZSTD_reset_session_and_parameters);
-            size2 = ZSTD_compress2(cctx, compressedBuffer, compressedBufferSize, CNBuffer, MIN(CNBuffSize, 100 KB));
-            if (ZSTD_isError(size2)) goto _output_error;
-
-            if (size1 == size2) goto _output_error;
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        ZSTD_CCtx_reset(cctx, ZSTD_reset_session_and_parameters);
-        CHECK_Z( ZSTD_CCtx_loadDictionary(cctx, dictBuffer, dictSize) );
-        cSize = ZSTD_compress2(cctx, compressedBuffer, compressedBufferSize, CNBuffer, MIN(CNBuffSize, 100 KB));
-        CHECK_Z(cSize);
-        DISPLAYLEVEL(3, "test%3i : ZSTD_decompressDCtx() with dictionary : ", testNb++);
-        {
-            ZSTD_DCtx* dctx = ZSTD_createDCtx();
-            size_t ret;
-            /* We should fail to decompress without a dictionary. */
-            ZSTD_DCtx_reset(dctx, ZSTD_reset_session_and_parameters);
-            ret = ZSTD_decompressDCtx(dctx, decodedBuffer, CNBuffSize, compressedBuffer, cSize);
-            if (!ZSTD_isError(ret)) goto _output_error;
-            /* We should succeed to decompress with the dictionary. */
-            ZSTD_DCtx_reset(dctx, ZSTD_reset_session_and_parameters);
-            CHECK_Z( ZSTD_DCtx_loadDictionary(dctx, dictBuffer, dictSize) );
-            CHECK_Z( ZSTD_decompressDCtx(dctx, decodedBuffer, CNBuffSize, compressedBuffer, cSize) );
-            /* The dictionary should presist across calls. */
-            CHECK_Z( ZSTD_decompressDCtx(dctx, decodedBuffer, CNBuffSize, compressedBuffer, cSize) );
-            /* When we reset the context the dictionary is cleared. */
-            ZSTD_DCtx_reset(dctx, ZSTD_reset_session_and_parameters);
-            ret = ZSTD_decompressDCtx(dctx, decodedBuffer, CNBuffSize, compressedBuffer, cSize);
-            if (!ZSTD_isError(ret)) goto _output_error;
-            ZSTD_freeDCtx(dctx);
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : ZSTD_decompressDCtx() with ddict : ", testNb++);
-        {
-            ZSTD_DCtx* dctx = ZSTD_createDCtx();
-            ZSTD_DDict* ddict = ZSTD_createDDict(dictBuffer, dictSize);
-            size_t ret;
-            /* We should succeed to decompress with the ddict. */
-            ZSTD_DCtx_reset(dctx, ZSTD_reset_session_and_parameters);
-            CHECK_Z( ZSTD_DCtx_refDDict(dctx, ddict) );
-            CHECK_Z( ZSTD_decompressDCtx(dctx, decodedBuffer, CNBuffSize, compressedBuffer, cSize) );
-            /* The ddict should presist across calls. */
-            CHECK_Z( ZSTD_decompressDCtx(dctx, decodedBuffer, CNBuffSize, compressedBuffer, cSize) );
-            /* When we reset the context the ddict is cleared. */
-            ZSTD_DCtx_reset(dctx, ZSTD_reset_session_and_parameters);
-            ret = ZSTD_decompressDCtx(dctx, decodedBuffer, CNBuffSize, compressedBuffer, cSize);
-            if (!ZSTD_isError(ret)) goto _output_error;
-            ZSTD_freeDCtx(dctx);
-            ZSTD_freeDDict(ddict);
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : ZSTD_decompressDCtx() with prefix : ", testNb++);
-        {
-            ZSTD_DCtx* dctx = ZSTD_createDCtx();
-            size_t ret;
-            /* We should succeed to decompress with the prefix. */
-            ZSTD_DCtx_reset(dctx, ZSTD_reset_session_and_parameters);
-            CHECK_Z( ZSTD_DCtx_refPrefix_advanced(dctx, dictBuffer, dictSize, ZSTD_dct_auto) );
-            CHECK_Z( ZSTD_decompressDCtx(dctx, decodedBuffer, CNBuffSize, compressedBuffer, cSize) );
-            /* The prefix should be cleared after the first compression. */
-            ret = ZSTD_decompressDCtx(dctx, decodedBuffer, CNBuffSize, compressedBuffer, cSize);
-            if (!ZSTD_isError(ret)) goto _output_error;
-            ZSTD_freeDCtx(dctx);
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : Dictionary with non-default repcodes : ", testNb++);
-        { U32 u; for (u=0; u<nbSamples; u++) samplesSizes[u] = sampleUnitSize; }
-        dictSize = ZDICT_trainFromBuffer(dictBuffer, dictSize,
-                                         CNBuffer, samplesSizes, nbSamples);
-        if (ZDICT_isError(dictSize)) goto _output_error;
-        /* Set all the repcodes to non-default */
-        {
-            BYTE* dictPtr = (BYTE*)dictBuffer;
-            BYTE* dictLimit = dictPtr + dictSize - 12;
-            /* Find the repcodes */
-            while (dictPtr < dictLimit &&
-                   (MEM_readLE32(dictPtr) != 1 || MEM_readLE32(dictPtr + 4) != 4 ||
-                    MEM_readLE32(dictPtr + 8) != 8)) {
-                ++dictPtr;
-            }
-            if (dictPtr >= dictLimit) goto _output_error;
-            MEM_writeLE32(dictPtr + 0, 10);
-            MEM_writeLE32(dictPtr + 4, 10);
-            MEM_writeLE32(dictPtr + 8, 10);
-            /* Set the last 8 bytes to 'x' */
-            memset((BYTE*)dictBuffer + dictSize - 8, 'x', 8);
-        }
-        /* The optimal parser checks all the repcodes.
-         * Make sure at least one is a match >= targetLength so that it is
-         * immediately chosen. This will make sure that the compressor and
-         * decompressor agree on at least one of the repcodes.
-         */
-        {   size_t dSize;
-            BYTE data[1024];
-            ZSTD_DCtx* const dctx = ZSTD_createDCtx();
-            ZSTD_compressionParameters const cParams = ZSTD_getCParams(19, CNBuffSize, dictSize);
-            ZSTD_CDict* const cdict = ZSTD_createCDict_advanced(dictBuffer, dictSize,
-                                            ZSTD_dlm_byRef, ZSTD_dct_auto,
-                                            cParams, ZSTD_defaultCMem);
-            assert(dctx != NULL); assert(cdict != NULL);
-            memset(data, 'x', sizeof(data));
-            cSize = ZSTD_compress_usingCDict(cctx, compressedBuffer, compressedBufferSize,
-                                             data, sizeof(data), cdict);
-            ZSTD_freeCDict(cdict);
-            if (ZSTD_isError(cSize)) { DISPLAYLEVEL(5, "Compression error %s : ", ZSTD_getErrorName(cSize)); goto _output_error; }
-            dSize = ZSTD_decompress_usingDict(dctx, decodedBuffer, sizeof(data), compressedBuffer, cSize, dictBuffer, dictSize);
-            if (ZSTD_isError(dSize)) { DISPLAYLEVEL(5, "Decompression error %s : ", ZSTD_getErrorName(dSize)); goto _output_error; }
-            if (memcmp(data, decodedBuffer, sizeof(data))) { DISPLAYLEVEL(5, "Data corruption : "); goto _output_error; }
-            ZSTD_freeDCtx(dctx);
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        ZSTD_freeCCtx(cctx);
-        free(dictBuffer);
-        free(samplesSizes);
-    }
-
-    /* COVER dictionary builder tests */
-    {   ZSTD_CCtx* const cctx = ZSTD_createCCtx();
-        size_t dictSize = 16 KB;
-        size_t optDictSize = dictSize;
-        void* dictBuffer = malloc(dictSize);
-        size_t const totalSampleSize = 1 MB;
-        size_t const sampleUnitSize = 8 KB;
-        U32 const nbSamples = (U32)(totalSampleSize / sampleUnitSize);
-        size_t* const samplesSizes = (size_t*) malloc(nbSamples * sizeof(size_t));
-        U32 seed32 = seed;
-        ZDICT_cover_params_t params;
-        U32 dictID;
-
-        if (dictBuffer==NULL || samplesSizes==NULL) {
-            free(dictBuffer);
-            free(samplesSizes);
-            goto _output_error;
-        }
-
-        DISPLAYLEVEL(3, "test%3i : ZDICT_trainFromBuffer_cover : ", testNb++);
-        { U32 u; for (u=0; u<nbSamples; u++) samplesSizes[u] = sampleUnitSize; }
-        memset(&params, 0, sizeof(params));
-        params.d = 1 + (FUZ_rand(&seed32) % 16);
-        params.k = params.d + (FUZ_rand(&seed32) % 256);
-        dictSize = ZDICT_trainFromBuffer_cover(dictBuffer, dictSize,
-                                               CNBuffer, samplesSizes, nbSamples,
-                                               params);
-        if (ZDICT_isError(dictSize)) goto _output_error;
-        DISPLAYLEVEL(3, "OK, created dictionary of size %u \n", (unsigned)dictSize);
-
-        DISPLAYLEVEL(3, "test%3i : check dictID : ", testNb++);
-        dictID = ZDICT_getDictID(dictBuffer, dictSize);
-        if (dictID==0) goto _output_error;
-        DISPLAYLEVEL(3, "OK : %u \n", (unsigned)dictID);
-
-        DISPLAYLEVEL(3, "test%3i : ZDICT_optimizeTrainFromBuffer_cover : ", testNb++);
-        memset(&params, 0, sizeof(params));
-        params.steps = 4;
-        optDictSize = ZDICT_optimizeTrainFromBuffer_cover(dictBuffer, optDictSize,
-                                                          CNBuffer, samplesSizes,
-                                                          nbSamples / 4, &params);
-        if (ZDICT_isError(optDictSize)) goto _output_error;
-        DISPLAYLEVEL(3, "OK, created dictionary of size %u \n", (unsigned)optDictSize);
-
-        DISPLAYLEVEL(3, "test%3i : check dictID : ", testNb++);
-        dictID = ZDICT_getDictID(dictBuffer, optDictSize);
-        if (dictID==0) goto _output_error;
-        DISPLAYLEVEL(3, "OK : %u \n", (unsigned)dictID);
-
-        ZSTD_freeCCtx(cctx);
-        free(dictBuffer);
-        free(samplesSizes);
-    }
-
-    /* Decompression defense tests */
-    DISPLAYLEVEL(3, "test%3i : Check input length for magic number : ", testNb++);
-    { size_t const r = ZSTD_decompress(decodedBuffer, CNBuffSize, CNBuffer, 3);   /* too small input */
-      if (!ZSTD_isError(r)) goto _output_error;
-      if (ZSTD_getErrorCode(r) != ZSTD_error_srcSize_wrong) goto _output_error; }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : Check magic Number : ", testNb++);
-    ((char*)(CNBuffer))[0] = 1;
-    { size_t const r = ZSTD_decompress(decodedBuffer, CNBuffSize, CNBuffer, 4);
-      if (!ZSTD_isError(r)) goto _output_error; }
-    DISPLAYLEVEL(3, "OK \n");
-
-    /* content size verification test */
-    DISPLAYLEVEL(3, "test%3i : Content size verification : ", testNb++);
-    {   ZSTD_CCtx* const cctx = ZSTD_createCCtx();
-        size_t const srcSize = 5000;
-        size_t const wrongSrcSize = (srcSize + 1000);
-        ZSTD_parameters params = ZSTD_getParams(1, wrongSrcSize, 0);
-        params.fParams.contentSizeFlag = 1;
-        CHECK( ZSTD_compressBegin_advanced(cctx, NULL, 0, params, wrongSrcSize) );
-        {   size_t const result = ZSTD_compressEnd(cctx, decodedBuffer, CNBuffSize, CNBuffer, srcSize);
-            if (!ZSTD_isError(result)) goto _output_error;
-            if (ZSTD_getErrorCode(result) != ZSTD_error_srcSize_wrong) goto _output_error;
-            DISPLAYLEVEL(3, "OK : %s \n", ZSTD_getErrorName(result));
-        }
-        ZSTD_freeCCtx(cctx);
-    }
-
-    /* negative compression level test : ensure simple API and advanced API produce same result */
-    DISPLAYLEVEL(3, "test%3i : negative compression level : ", testNb++);
-    {   ZSTD_CCtx* const cctx = ZSTD_createCCtx();
-        size_t const srcSize = CNBuffSize / 5;
-        int const compressionLevel = -1;
-
-        assert(cctx != NULL);
-        {   ZSTD_parameters const params = ZSTD_getParams(compressionLevel, srcSize, 0);
-            size_t const cSize_1pass = ZSTD_compress_advanced(cctx,
-                                        compressedBuffer, compressedBufferSize,
-                                        CNBuffer, srcSize,
-                                        NULL, 0,
-                                        params);
-            if (ZSTD_isError(cSize_1pass)) goto _output_error;
-
-            CHECK( ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, compressionLevel) );
-            {   size_t const compressionResult = ZSTD_compress2(cctx,
-                                    compressedBuffer, compressedBufferSize,
-                                    CNBuffer, srcSize);
-                DISPLAYLEVEL(5, "simple=%zu vs %zu=advanced : ", cSize_1pass, compressionResult);
-                if (ZSTD_isError(compressionResult)) goto _output_error;
-                if (compressionResult != cSize_1pass) goto _output_error;
-        }   }
-        ZSTD_freeCCtx(cctx);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    /* parameters order test */
-    {   size_t const inputSize = CNBuffSize / 2;
-        U64 xxh64;
-
-        {   ZSTD_CCtx* const cctx = ZSTD_createCCtx();
-            DISPLAYLEVEL(3, "test%3i : parameters in order : ", testNb++);
-            assert(cctx != NULL);
-            CHECK( ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, 2) );
-            CHECK( ZSTD_CCtx_setParameter(cctx, ZSTD_c_enableLongDistanceMatching, 1) );
-            CHECK( ZSTD_CCtx_setParameter(cctx, ZSTD_c_windowLog, 18) );
-            {   size_t const compressedSize = ZSTD_compress2(cctx,
-                                compressedBuffer, ZSTD_compressBound(inputSize),
-                                CNBuffer, inputSize);
-                CHECK(compressedSize);
-                cSize = compressedSize;
-                xxh64 = XXH64(compressedBuffer, compressedSize, 0);
-            }
-            DISPLAYLEVEL(3, "OK (compress : %u -> %u bytes)\n", (unsigned)inputSize, (unsigned)cSize);
-            ZSTD_freeCCtx(cctx);
-        }
-
-        {   ZSTD_CCtx* cctx = ZSTD_createCCtx();
-            DISPLAYLEVEL(3, "test%3i : parameters disordered : ", testNb++);
-            CHECK( ZSTD_CCtx_setParameter(cctx, ZSTD_c_windowLog, 18) );
-            CHECK( ZSTD_CCtx_setParameter(cctx, ZSTD_c_enableLongDistanceMatching, 1) );
-            CHECK( ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, 2) );
-            {   size_t const result = ZSTD_compress2(cctx,
-                                compressedBuffer, ZSTD_compressBound(inputSize),
-                                CNBuffer, inputSize);
-                CHECK(result);
-                if (result != cSize) goto _output_error;   /* must result in same compressed result, hence same size */
-                if (XXH64(compressedBuffer, result, 0) != xxh64) goto _output_error;  /* must result in exactly same content, hence same hash */
-                DISPLAYLEVEL(3, "OK (compress : %u -> %u bytes)\n", (unsigned)inputSize, (unsigned)result);
-            }
-            ZSTD_freeCCtx(cctx);
-        }
-    }
-
-    /* advanced parameters for decompression */
-    {   ZSTD_DCtx* const dctx = ZSTD_createDCtx();
-        assert(dctx != NULL);
-
-        DISPLAYLEVEL(3, "test%3i : get dParameter bounds ", testNb++);
-        {   ZSTD_bounds const bounds = ZSTD_dParam_getBounds(ZSTD_d_windowLogMax);
-            CHECK(bounds.error);
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : wrong dParameter : ", testNb++);
-        {   size_t const sr = ZSTD_DCtx_setParameter(dctx, (ZSTD_dParameter)999999, 0);
-            if (!ZSTD_isError(sr)) goto _output_error;
-        }
-        {   ZSTD_bounds const bounds = ZSTD_dParam_getBounds((ZSTD_dParameter)999998);
-            if (!ZSTD_isError(bounds.error)) goto _output_error;
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : out of bound dParameter : ", testNb++);
-        {   size_t const sr = ZSTD_DCtx_setParameter(dctx, ZSTD_d_windowLogMax, 9999);
-            if (!ZSTD_isError(sr)) goto _output_error;
-        }
-        {   size_t const sr = ZSTD_DCtx_setParameter(dctx, ZSTD_d_format, (ZSTD_format_e)888);
-            if (!ZSTD_isError(sr)) goto _output_error;
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        ZSTD_freeDCtx(dctx);
-    }
-
-
-    /* custom formats tests */
-    {   ZSTD_CCtx* const cctx = ZSTD_createCCtx();
-        ZSTD_DCtx* const dctx = ZSTD_createDCtx();
-        size_t const inputSize = CNBuffSize / 2;   /* won't cause pb with small dict size */
-        assert(dctx != NULL); assert(cctx != NULL);
-
-        /* basic block compression */
-        DISPLAYLEVEL(3, "test%3i : magic-less format test : ", testNb++);
-        CHECK( ZSTD_CCtx_setParameter(cctx, ZSTD_c_format, ZSTD_f_zstd1_magicless) );
-        {   ZSTD_inBuffer in = { CNBuffer, inputSize, 0 };
-            ZSTD_outBuffer out = { compressedBuffer, ZSTD_compressBound(inputSize), 0 };
-            size_t const result = ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_end);
-            if (result != 0) goto _output_error;
-            if (in.pos != in.size) goto _output_error;
-            cSize = out.pos;
-        }
-        DISPLAYLEVEL(3, "OK (compress : %u -> %u bytes)\n", (unsigned)inputSize, (unsigned)cSize);
-
-        DISPLAYLEVEL(3, "test%3i : decompress normally (should fail) : ", testNb++);
-        {   size_t const decodeResult = ZSTD_decompressDCtx(dctx, decodedBuffer, CNBuffSize, compressedBuffer, cSize);
-            if (ZSTD_getErrorCode(decodeResult) != ZSTD_error_prefix_unknown) goto _output_error;
-            DISPLAYLEVEL(3, "OK : %s \n", ZSTD_getErrorName(decodeResult));
-        }
-
-        DISPLAYLEVEL(3, "test%3i : decompress of magic-less frame : ", testNb++);
-        ZSTD_DCtx_reset(dctx, ZSTD_reset_session_and_parameters);
-        CHECK( ZSTD_DCtx_setParameter(dctx, ZSTD_d_format, ZSTD_f_zstd1_magicless) );
-        {   ZSTD_frameHeader zfh;
-            size_t const zfhrt = ZSTD_getFrameHeader_advanced(&zfh, compressedBuffer, cSize, ZSTD_f_zstd1_magicless);
-            if (zfhrt != 0) goto _output_error;
-        }
-        /* one shot */
-        {   size_t const result = ZSTD_decompressDCtx(dctx, decodedBuffer, CNBuffSize, compressedBuffer, cSize);
-            if (result != inputSize) goto _output_error;
-            DISPLAYLEVEL(3, "one-shot OK, ");
-        }
-        /* streaming */
-        {   ZSTD_inBuffer in = { compressedBuffer, cSize, 0 };
-            ZSTD_outBuffer out = { decodedBuffer, CNBuffSize, 0 };
-            size_t const result = ZSTD_decompressStream(dctx, &out, &in);
-            if (result != 0) goto _output_error;
-            if (in.pos != in.size) goto _output_error;
-            if (out.pos != inputSize) goto _output_error;
-            DISPLAYLEVEL(3, "streaming OK : regenerated %u bytes \n", (unsigned)out.pos);
-        }
-
-        /* basic block compression */
-        DISPLAYLEVEL(3, "test%3i : empty magic-less format test : ", testNb++);
-        CHECK( ZSTD_CCtx_setParameter(cctx, ZSTD_c_format, ZSTD_f_zstd1_magicless) );
-        {   ZSTD_inBuffer in = { CNBuffer, 0, 0 };
-            ZSTD_outBuffer out = { compressedBuffer, ZSTD_compressBound(0), 0 };
-            size_t const result = ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_end);
-            if (result != 0) goto _output_error;
-            if (in.pos != in.size) goto _output_error;
-            cSize = out.pos;
-        }
-        DISPLAYLEVEL(3, "OK (compress : %u -> %u bytes)\n", (unsigned)0, (unsigned)cSize);
-
-        DISPLAYLEVEL(3, "test%3i : decompress of empty magic-less frame : ", testNb++);
-        ZSTD_DCtx_reset(dctx, ZSTD_reset_session_and_parameters);
-        CHECK( ZSTD_DCtx_setParameter(dctx, ZSTD_d_format, ZSTD_f_zstd1_magicless) );
-        /* one shot */
-        {   size_t const result = ZSTD_decompressDCtx(dctx, decodedBuffer, CNBuffSize, compressedBuffer, cSize);
-            if (result != 0) goto _output_error;
-            DISPLAYLEVEL(3, "one-shot OK, ");
-        }
-        /* streaming */
-        {   ZSTD_inBuffer in = { compressedBuffer, cSize, 0 };
-            ZSTD_outBuffer out = { decodedBuffer, CNBuffSize, 0 };
-            size_t const result = ZSTD_decompressStream(dctx, &out, &in);
-            if (result != 0) goto _output_error;
-            if (in.pos != in.size) goto _output_error;
-            if (out.pos != 0) goto _output_error;
-            DISPLAYLEVEL(3, "streaming OK : regenerated %u bytes \n", (unsigned)out.pos);
-        }
-
-        ZSTD_freeCCtx(cctx);
-        ZSTD_freeDCtx(dctx);
-    }
-
-    /* block API tests */
-    {   ZSTD_CCtx* const cctx = ZSTD_createCCtx();
-        ZSTD_DCtx* const dctx = ZSTD_createDCtx();
-        static const size_t dictSize = 65 KB;
-        static const size_t blockSize = 100 KB;   /* won't cause pb with small dict size */
-        size_t cSize2;
-        assert(cctx != NULL); assert(dctx != NULL);
-
-        /* basic block compression */
-        DISPLAYLEVEL(3, "test%3i : Block compression test : ", testNb++);
-        CHECK( ZSTD_compressBegin(cctx, 5) );
-        CHECK( ZSTD_getBlockSize(cctx) >= blockSize);
-        CHECK_VAR(cSize, ZSTD_compressBlock(cctx, compressedBuffer, ZSTD_compressBound(blockSize), CNBuffer, blockSize) );
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : Block decompression test : ", testNb++);
-        CHECK( ZSTD_decompressBegin(dctx) );
-        { CHECK_NEWV(r, ZSTD_decompressBlock(dctx, decodedBuffer, CNBuffSize, compressedBuffer, cSize) );
-          if (r != blockSize) goto _output_error; }
-        DISPLAYLEVEL(3, "OK \n");
-
-        /* very long stream of block compression */
-        DISPLAYLEVEL(3, "test%3i : Huge block streaming compression test : ", testNb++);
-        CHECK( ZSTD_compressBegin(cctx, -199) );  /* we just want to quickly overflow internal U32 index */
-        CHECK( ZSTD_getBlockSize(cctx) >= blockSize);
-        {   U64 const toCompress = 5000000000ULL;   /* > 4 GB */
-            U64 compressed = 0;
-            while (compressed < toCompress) {
-                size_t const blockCSize = ZSTD_compressBlock(cctx, compressedBuffer, ZSTD_compressBound(blockSize), CNBuffer, blockSize);
-                assert(blockCSize != 0);
-                if (ZSTD_isError(blockCSize)) goto _output_error;
-                compressed += blockCSize;
-        }   }
-        DISPLAYLEVEL(3, "OK \n");
-
-        /* dictionary block compression */
-        DISPLAYLEVEL(3, "test%3i : Dictionary Block compression test : ", testNb++);
-        CHECK( ZSTD_compressBegin_usingDict(cctx, CNBuffer, dictSize, 5) );
-        CHECK_VAR(cSize,  ZSTD_compressBlock(cctx, compressedBuffer, ZSTD_compressBound(blockSize), (char*)CNBuffer+dictSize, blockSize));
-        RDG_genBuffer((char*)CNBuffer+dictSize+blockSize, blockSize, 0.0, 0.0, seed);  /* create a non-compressible second block */
-        { CHECK_NEWV(r, ZSTD_compressBlock(cctx, (char*)compressedBuffer+cSize, ZSTD_compressBound(blockSize), (char*)CNBuffer+dictSize+blockSize, blockSize) );  /* for cctx history consistency */
-          assert(r == 0); /* non-compressible block */ }
-        memcpy((char*)compressedBuffer+cSize, (char*)CNBuffer+dictSize+blockSize, blockSize);   /* send non-compressed block (without header) */
-        CHECK_VAR(cSize2, ZSTD_compressBlock(cctx, (char*)compressedBuffer+cSize+blockSize, ZSTD_compressBound(blockSize),
-                                                   (char*)CNBuffer+dictSize+2*blockSize, blockSize));
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : Dictionary Block decompression test : ", testNb++);
-        CHECK( ZSTD_decompressBegin_usingDict(dctx, CNBuffer, dictSize) );
-        {   CHECK_NEWV( r, ZSTD_decompressBlock(dctx, decodedBuffer, blockSize, compressedBuffer, cSize) );
-            if (r != blockSize) {
-                DISPLAYLEVEL(1, "ZSTD_decompressBlock() with _usingDict() fails : %u, instead of %u expected \n", (unsigned)r, (unsigned)blockSize);
-                goto _output_error;
-        }   }
-        memcpy((char*)decodedBuffer+blockSize, (char*)compressedBuffer+cSize, blockSize);
-        ZSTD_insertBlock(dctx, (char*)decodedBuffer+blockSize, blockSize);   /* insert non-compressed block into dctx history */
-        {   CHECK_NEWV( r, ZSTD_decompressBlock(dctx, (char*)decodedBuffer+2*blockSize, blockSize, (char*)compressedBuffer+cSize+blockSize, cSize2) );
-            if (r != blockSize) {
-                DISPLAYLEVEL(1, "ZSTD_decompressBlock() with _usingDict() and after insertBlock() fails : %u, instead of %u expected \n", (unsigned)r, (unsigned)blockSize);
-                goto _output_error;
-        }   }
-        assert(memcpy((char*)CNBuffer+dictSize, decodedBuffer, blockSize*3));  /* ensure regenerated content is identical to origin */
-        DISPLAYLEVEL(3, "OK \n");
-
-        DISPLAYLEVEL(3, "test%3i : Block compression with CDict : ", testNb++);
-        {   ZSTD_CDict* const cdict = ZSTD_createCDict(CNBuffer, dictSize, 3);
-            if (cdict==NULL) goto _output_error;
-            CHECK( ZSTD_compressBegin_usingCDict(cctx, cdict) );
-            CHECK( ZSTD_compressBlock(cctx, compressedBuffer, ZSTD_compressBound(blockSize), (char*)CNBuffer+dictSize, blockSize) );
-            ZSTD_freeCDict(cdict);
-        }
-        DISPLAYLEVEL(3, "OK \n");
-
-        ZSTD_freeCCtx(cctx);
-        ZSTD_freeDCtx(dctx);
-    }
-
-    /* long rle test */
-    {   size_t sampleSize = 0;
-        DISPLAYLEVEL(3, "test%3i : Long RLE test : ", testNb++);
-        RDG_genBuffer(CNBuffer, sampleSize, compressibility, 0., seed+1);
-        memset((char*)CNBuffer+sampleSize, 'B', 256 KB - 1);
-        sampleSize += 256 KB - 1;
-        RDG_genBuffer((char*)CNBuffer+sampleSize, 96 KB, compressibility, 0., seed+2);
-        sampleSize += 96 KB;
-        cSize = ZSTD_compress(compressedBuffer, ZSTD_compressBound(sampleSize), CNBuffer, sampleSize, 1);
-        if (ZSTD_isError(cSize)) goto _output_error;
-        { CHECK_NEWV(regenSize, ZSTD_decompress(decodedBuffer, sampleSize, compressedBuffer, cSize));
-          if (regenSize!=sampleSize) goto _output_error; }
-        DISPLAYLEVEL(3, "OK \n");
-    }
-
-    DISPLAYLEVEL(3, "test%3i : ZSTD_getSequences decode from sequences test : ", testNb++);
-    {
-        size_t srcSize = 100 KB;
-        BYTE* src = (BYTE*)CNBuffer;
-        BYTE* decoded = (BYTE*)compressedBuffer;
-
-        ZSTD_CCtx* cctx = ZSTD_createCCtx();
-        ZSTD_Sequence* seqs = (ZSTD_Sequence*)malloc(srcSize * sizeof(ZSTD_Sequence));
-        size_t seqsSize;
-
-        if (seqs == NULL) goto _output_error;
-        assert(cctx != NULL);
-
-        /* Populate src with random data */
-        RDG_genBuffer(CNBuffer, srcSize, compressibility, 0., seed);
-
-        /* get the sequences */
-        seqsSize = ZSTD_getSequences(cctx, seqs, srcSize, src, srcSize);
-
-        /* "decode" and compare the sequences */
-        FUZ_decodeSequences(decoded, seqs, seqsSize, src, srcSize);
-        assert(!memcmp(CNBuffer, compressedBuffer, srcSize));
-
-        ZSTD_freeCCtx(cctx);
-        free(seqs);
-    }
-
-    /* Multiple blocks of zeros test */
-    #define LONGZEROSLENGTH 1000000 /* 1MB of zeros */
-    DISPLAYLEVEL(3, "test%3i : compress %u zeroes : ", testNb++, LONGZEROSLENGTH);
-    memset(CNBuffer, 0, LONGZEROSLENGTH);
-    CHECK_VAR(cSize, ZSTD_compress(compressedBuffer, ZSTD_compressBound(LONGZEROSLENGTH), CNBuffer, LONGZEROSLENGTH, 1) );
-    DISPLAYLEVEL(3, "OK (%u bytes : %.2f%%)\n", (unsigned)cSize, (double)cSize/LONGZEROSLENGTH*100);
-
-    DISPLAYLEVEL(3, "test%3i : decompress %u zeroes : ", testNb++, LONGZEROSLENGTH);
-    { CHECK_NEWV(r, ZSTD_decompress(decodedBuffer, LONGZEROSLENGTH, compressedBuffer, cSize) );
-      if (r != LONGZEROSLENGTH) goto _output_error; }
-    DISPLAYLEVEL(3, "OK \n");
-
-    /* All zeroes test (test bug #137) */
-    #define ZEROESLENGTH 100
-    DISPLAYLEVEL(3, "test%3i : compress %u zeroes : ", testNb++, ZEROESLENGTH);
-    memset(CNBuffer, 0, ZEROESLENGTH);
-    CHECK_VAR(cSize, ZSTD_compress(compressedBuffer, ZSTD_compressBound(ZEROESLENGTH), CNBuffer, ZEROESLENGTH, 1) );
-    DISPLAYLEVEL(3, "OK (%u bytes : %.2f%%)\n", (unsigned)cSize, (double)cSize/ZEROESLENGTH*100);
-
-    DISPLAYLEVEL(3, "test%3i : decompress %u zeroes : ", testNb++, ZEROESLENGTH);
-    { CHECK_NEWV(r, ZSTD_decompress(decodedBuffer, ZEROESLENGTH, compressedBuffer, cSize) );
-      if (r != ZEROESLENGTH) goto _output_error; }
-    DISPLAYLEVEL(3, "OK \n");
-
-    /* nbSeq limit test */
-    #define _3BYTESTESTLENGTH 131000
-    #define NB3BYTESSEQLOG   9
-    #define NB3BYTESSEQ     (1 << NB3BYTESSEQLOG)
-    #define NB3BYTESSEQMASK (NB3BYTESSEQ-1)
-    /* creates a buffer full of 3-bytes sequences */
-    {   BYTE _3BytesSeqs[NB3BYTESSEQ][3];
-        U32 rSeed = 1;
-
-        /* create batch of 3-bytes sequences */
-        {   int i;
-            for (i=0; i < NB3BYTESSEQ; i++) {
-                _3BytesSeqs[i][0] = (BYTE)(FUZ_rand(&rSeed) & 255);
-                _3BytesSeqs[i][1] = (BYTE)(FUZ_rand(&rSeed) & 255);
-                _3BytesSeqs[i][2] = (BYTE)(FUZ_rand(&rSeed) & 255);
-        }   }
-
-        /* randomly fills CNBuffer with prepared 3-bytes sequences */
-        {   int i;
-            for (i=0; i < _3BYTESTESTLENGTH; i += 3) {   /* note : CNBuffer size > _3BYTESTESTLENGTH+3 */
-                U32 const id = FUZ_rand(&rSeed) & NB3BYTESSEQMASK;
-                ((BYTE*)CNBuffer)[i+0] = _3BytesSeqs[id][0];
-                ((BYTE*)CNBuffer)[i+1] = _3BytesSeqs[id][1];
-                ((BYTE*)CNBuffer)[i+2] = _3BytesSeqs[id][2];
-    }   }   }
-    DISPLAYLEVEL(3, "test%3i : growing nbSeq : ", testNb++);
-    {   ZSTD_CCtx* const cctx = ZSTD_createCCtx();
-        size_t const maxNbSeq = _3BYTESTESTLENGTH / 3;
-        size_t const bound = ZSTD_compressBound(_3BYTESTESTLENGTH);
-        size_t nbSeq = 1;
-        while (nbSeq <= maxNbSeq) {
-          CHECK(ZSTD_compressCCtx(cctx, compressedBuffer, bound, CNBuffer, nbSeq * 3, 19));
-          /* Check every sequence for the first 100, then skip more rapidly. */
-          if (nbSeq < 100) {
-            ++nbSeq;
-          } else {
-            nbSeq += (nbSeq >> 2);
-          }
-        }
-        ZSTD_freeCCtx(cctx);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : compress lots 3-bytes sequences : ", testNb++);
-    CHECK_VAR(cSize, ZSTD_compress(compressedBuffer, ZSTD_compressBound(_3BYTESTESTLENGTH),
-                                   CNBuffer, _3BYTESTESTLENGTH, 19) );
-    DISPLAYLEVEL(3, "OK (%u bytes : %.2f%%)\n", (unsigned)cSize, (double)cSize/_3BYTESTESTLENGTH*100);
-
-    DISPLAYLEVEL(3, "test%3i : decompress lots 3-bytes sequence : ", testNb++);
-    { CHECK_NEWV(r, ZSTD_decompress(decodedBuffer, _3BYTESTESTLENGTH, compressedBuffer, cSize) );
-      if (r != _3BYTESTESTLENGTH) goto _output_error; }
-    DISPLAYLEVEL(3, "OK \n");
-
-
-    DISPLAYLEVEL(3, "test%3i : growing literals buffer : ", testNb++);
-    RDG_genBuffer(CNBuffer, CNBuffSize, 0.0, 0.1, seed);
-    {   ZSTD_CCtx* const cctx = ZSTD_createCCtx();
-        size_t const bound = ZSTD_compressBound(CNBuffSize);
-        size_t size = 1;
-        while (size <= CNBuffSize) {
-          CHECK(ZSTD_compressCCtx(cctx, compressedBuffer, bound, CNBuffer, size, 3));
-          /* Check every size for the first 100, then skip more rapidly. */
-          if (size < 100) {
-            ++size;
-          } else {
-            size += (size >> 2);
-          }
-        }
-        ZSTD_freeCCtx(cctx);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : incompressible data and ill suited dictionary : ", testNb++);
-    {   /* Train a dictionary on low characters */
-        size_t dictSize = 16 KB;
-        void* const dictBuffer = malloc(dictSize);
-        size_t const totalSampleSize = 1 MB;
-        size_t const sampleUnitSize = 8 KB;
-        U32 const nbSamples = (U32)(totalSampleSize / sampleUnitSize);
-        size_t* const samplesSizes = (size_t*) malloc(nbSamples * sizeof(size_t));
-        if (!dictBuffer || !samplesSizes) goto _output_error;
-        { U32 u; for (u=0; u<nbSamples; u++) samplesSizes[u] = sampleUnitSize; }
-        dictSize = ZDICT_trainFromBuffer(dictBuffer, dictSize, CNBuffer, samplesSizes, nbSamples);
-        if (ZDICT_isError(dictSize)) goto _output_error;
-        /* Reverse the characters to make the dictionary ill suited */
-        {   U32 u;
-            for (u = 0; u < CNBuffSize; ++u) {
-              ((BYTE*)CNBuffer)[u] = 255 - ((BYTE*)CNBuffer)[u];
-            }
-        }
-        {   /* Compress the data */
-            size_t const inputSize = 500;
-            size_t const outputSize = ZSTD_compressBound(inputSize);
-            void* const outputBuffer = malloc(outputSize);
-            ZSTD_CCtx* const cctx = ZSTD_createCCtx();
-            if (!outputBuffer || !cctx) goto _output_error;
-            CHECK(ZSTD_compress_usingDict(cctx, outputBuffer, outputSize, CNBuffer, inputSize, dictBuffer, dictSize, 1));
-            free(outputBuffer);
-            ZSTD_freeCCtx(cctx);
-        }
-
-        free(dictBuffer);
-        free(samplesSizes);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-
-    /* findFrameCompressedSize on skippable frames */
-    DISPLAYLEVEL(3, "test%3i : frame compressed size of skippable frame : ", testNb++);
-    {   const char* frame = "\x50\x2a\x4d\x18\x05\x0\x0\0abcde";
-        size_t const frameSrcSize = 13;
-        if (ZSTD_findFrameCompressedSize(frame, frameSrcSize) != frameSrcSize) goto _output_error; }
-    DISPLAYLEVEL(3, "OK \n");
-
-    /* error string tests */
-    DISPLAYLEVEL(3, "test%3i : testing ZSTD error code strings : ", testNb++);
-    if (strcmp("No error detected", ZSTD_getErrorName((ZSTD_ErrorCode)(0-ZSTD_error_no_error))) != 0) goto _output_error;
-    if (strcmp("No error detected", ZSTD_getErrorString(ZSTD_error_no_error)) != 0) goto _output_error;
-    if (strcmp("Unspecified error code", ZSTD_getErrorString((ZSTD_ErrorCode)(0-ZSTD_error_GENERIC))) != 0) goto _output_error;
-    if (strcmp("Error (generic)", ZSTD_getErrorName((size_t)0-ZSTD_error_GENERIC)) != 0) goto _output_error;
-    if (strcmp("Error (generic)", ZSTD_getErrorString(ZSTD_error_GENERIC)) != 0) goto _output_error;
-    if (strcmp("No error detected", ZSTD_getErrorName(ZSTD_error_GENERIC)) != 0) goto _output_error;
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : testing ZSTD dictionary sizes : ", testNb++);
-    RDG_genBuffer(CNBuffer, CNBuffSize, compressibility, 0., seed);
-    {
-        size_t const size = MIN(128 KB, CNBuffSize);
-        ZSTD_CCtx* const cctx = ZSTD_createCCtx();
-        ZSTD_CDict* const lgCDict = ZSTD_createCDict(CNBuffer, size, 1);
-        ZSTD_CDict* const smCDict = ZSTD_createCDict(CNBuffer, 1 KB, 1);
-        ZSTD_frameHeader lgHeader;
-        ZSTD_frameHeader smHeader;
-
-        CHECK_Z(ZSTD_compress_usingCDict(cctx, compressedBuffer, compressedBufferSize, CNBuffer, size, lgCDict));
-        CHECK_Z(ZSTD_getFrameHeader(&lgHeader, compressedBuffer, compressedBufferSize));
-        CHECK_Z(ZSTD_compress_usingCDict(cctx, compressedBuffer, compressedBufferSize, CNBuffer, size, smCDict));
-        CHECK_Z(ZSTD_getFrameHeader(&smHeader, compressedBuffer, compressedBufferSize));
-
-        if (lgHeader.windowSize != smHeader.windowSize) goto _output_error;
-
-        ZSTD_freeCDict(smCDict);
-        ZSTD_freeCDict(lgCDict);
-        ZSTD_freeCCtx(cctx);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : testing FSE_normalizeCount() PR#1255: ", testNb++);
-    {
-        short norm[32];
-        unsigned count[32];
-        unsigned const tableLog = 5;
-        size_t const nbSeq = 32;
-        unsigned const maxSymbolValue = 31;
-        size_t i;
-
-        for (i = 0; i < 32; ++i)
-            count[i] = 1;
-        /* Calling FSE_normalizeCount() on a uniform distribution should not
-         * cause a division by zero.
-         */
-        FSE_normalizeCount(norm, tableLog, count, nbSeq, maxSymbolValue);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : table cleanliness through index reduction : ", testNb++);
-    {
-        int cLevel;
-        size_t approxIndex = 0;
-        size_t maxIndex = ((3U << 29) + (1U << ZSTD_WINDOWLOG_MAX)); /* ZSTD_CURRENT_MAX from zstd_compress_internal.h */
-
-        /* Provision enough space in a static context so that we can do all
-         * this without ever reallocating, which would reset the indices. */
-        size_t const staticCCtxSize = ZSTD_estimateCStreamSize(22);
-        void* const staticCCtxBuffer = malloc(staticCCtxSize);
-        ZSTD_CCtx* cctx = ZSTD_initStaticCCtx(staticCCtxBuffer, staticCCtxSize);
-
-        /* bump the indices so the following compressions happen at high
-         * indices. */
-        {
-            ZSTD_outBuffer out = { compressedBuffer, compressedBufferSize, 0 };
-            ZSTD_inBuffer in = { CNBuffer, CNBuffSize, 0 };
-            ZSTD_CCtx_reset(cctx, ZSTD_reset_session_and_parameters);
-            CHECK_Z(ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, -500));
-            while (approxIndex <= (maxIndex / 4) * 3) {
-                CHECK_Z(ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_flush));
-                approxIndex += in.pos;
-                CHECK(in.pos == in.size);
-                in.pos = 0;
-                out.pos = 0;
-            }
-            CHECK_Z(ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_end));
-        }
-
-        /* spew a bunch of stuff into the table area */
-        for (cLevel = 1; cLevel <= 22; cLevel++) {
-            ZSTD_outBuffer out = { compressedBuffer, compressedBufferSize / cLevel, 0 };
-            ZSTD_inBuffer in = { CNBuffer, CNBuffSize, 0 };
-            ZSTD_CCtx_reset(cctx, ZSTD_reset_session_and_parameters);
-            CHECK_Z(ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, cLevel));
-            CHECK_Z(ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_flush));
-            CHECK_Z(ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_end));
-            approxIndex += in.pos;
-        }
-
-        /* now crank the indices so we overflow */
-        {
-            ZSTD_outBuffer out = { compressedBuffer, compressedBufferSize, 0 };
-            ZSTD_inBuffer in = { CNBuffer, CNBuffSize, 0 };
-            ZSTD_CCtx_reset(cctx, ZSTD_reset_session_and_parameters);
-            CHECK_Z(ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, -500));
-            while (approxIndex <= maxIndex) {
-                CHECK_Z(ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_flush));
-                approxIndex += in.pos;
-                CHECK(in.pos == in.size);
-                in.pos = 0;
-                out.pos = 0;
-            }
-            CHECK_Z(ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_end));
-        }
-
-        /* do a bunch of compressions again in low indices and ensure we don't
-         * hit untracked invalid indices */
-        for (cLevel = 1; cLevel <= 22; cLevel++) {
-            ZSTD_outBuffer out = { compressedBuffer, compressedBufferSize / cLevel, 0 };
-            ZSTD_inBuffer in = { CNBuffer, CNBuffSize, 0 };
-            ZSTD_CCtx_reset(cctx, ZSTD_reset_session_and_parameters);
-            CHECK_Z(ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, cLevel));
-            CHECK_Z(ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_flush));
-            CHECK_Z(ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_end));
-            approxIndex += in.pos;
-        }
-
-        ZSTD_freeCCtx(cctx);
-        free(staticCCtxBuffer);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-_end:
-    free(CNBuffer);
-    free(compressedBuffer);
-    free(decodedBuffer);
-    return testResult;
-
-_output_error:
-    testResult = 1;
-    DISPLAY("Error detected in Unit tests ! \n");
-    goto _end;
-}
-
-
-static size_t findDiff(const void* buf1, const void* buf2, size_t max)
-{
-    const BYTE* b1 = (const BYTE*)buf1;
-    const BYTE* b2 = (const BYTE*)buf2;
-    size_t u;
-    for (u=0; u<max; u++) {
-        if (b1[u] != b2[u]) break;
-    }
-    return u;
-}
-
-
-static ZSTD_parameters FUZ_makeParams(ZSTD_compressionParameters cParams, ZSTD_frameParameters fParams)
-{
-    ZSTD_parameters params;
-    params.cParams = cParams;
-    params.fParams = fParams;
-    return params;
-}
-
-static size_t FUZ_rLogLength(U32* seed, U32 logLength)
-{
-    size_t const lengthMask = ((size_t)1 << logLength) - 1;
-    return (lengthMask+1) + (FUZ_rand(seed) & lengthMask);
-}
-
-static size_t FUZ_randomLength(U32* seed, U32 maxLog)
-{
-    U32 const logLength = FUZ_rand(seed) % maxLog;
-    return FUZ_rLogLength(seed, logLength);
-}
-
-#undef CHECK
-#define CHECK(cond, ...) {                                    \
-    if (cond) {                                               \
-        DISPLAY("Error => ");                                 \
-        DISPLAY(__VA_ARGS__);                                 \
-        DISPLAY(" (seed %u, test nb %u)  \n", (unsigned)seed, testNb);  \
-        goto _output_error;                                   \
-}   }
-
-#undef CHECK_Z
-#define CHECK_Z(f) {                                          \
-    size_t const err = f;                                     \
-    if (ZSTD_isError(err)) {                                  \
-        DISPLAY("Error => %s : %s ",                          \
-                #f, ZSTD_getErrorName(err));                  \
-        DISPLAY(" (seed %u, test nb %u)  \n", (unsigned)seed, testNb);  \
-        goto _output_error;                                   \
-}   }
-
-
-static int fuzzerTests(U32 seed, unsigned nbTests, unsigned startTest, U32 const maxDurationS, double compressibility, int bigTests)
-{
-    static const U32 maxSrcLog = 23;
-    static const U32 maxSampleLog = 22;
-    size_t const srcBufferSize = (size_t)1<<maxSrcLog;
-    size_t const dstBufferSize = (size_t)1<<maxSampleLog;
-    size_t const cBufferSize   = ZSTD_compressBound(dstBufferSize);
-    BYTE* cNoiseBuffer[5];
-    BYTE* const cBuffer = (BYTE*) malloc (cBufferSize);
-    BYTE* const dstBuffer = (BYTE*) malloc (dstBufferSize);
-    BYTE* const mirrorBuffer = (BYTE*) malloc (dstBufferSize);
-    ZSTD_CCtx* const refCtx = ZSTD_createCCtx();
-    ZSTD_CCtx* const ctx = ZSTD_createCCtx();
-    ZSTD_DCtx* const dctx = ZSTD_createDCtx();
-    U32 result = 0;
-    unsigned testNb = 0;
-    U32 coreSeed = seed;
-    UTIL_time_t const startClock = UTIL_getTime();
-    U64 const maxClockSpan = maxDurationS * SEC_TO_MICRO;
-    int const cLevelLimiter = bigTests ? 3 : 2;
-
-    /* allocation */
-    cNoiseBuffer[0] = (BYTE*)malloc (srcBufferSize);
-    cNoiseBuffer[1] = (BYTE*)malloc (srcBufferSize);
-    cNoiseBuffer[2] = (BYTE*)malloc (srcBufferSize);
-    cNoiseBuffer[3] = (BYTE*)malloc (srcBufferSize);
-    cNoiseBuffer[4] = (BYTE*)malloc (srcBufferSize);
-    CHECK (!cNoiseBuffer[0] || !cNoiseBuffer[1] || !cNoiseBuffer[2] || !cNoiseBuffer[3] || !cNoiseBuffer[4]
-           || !dstBuffer || !mirrorBuffer || !cBuffer || !refCtx || !ctx || !dctx,
-           "Not enough memory, fuzzer tests cancelled");
-
-    /* Create initial samples */
-    RDG_genBuffer(cNoiseBuffer[0], srcBufferSize, 0.00, 0., coreSeed);    /* pure noise */
-    RDG_genBuffer(cNoiseBuffer[1], srcBufferSize, 0.05, 0., coreSeed);    /* barely compressible */
-    RDG_genBuffer(cNoiseBuffer[2], srcBufferSize, compressibility, 0., coreSeed);
-    RDG_genBuffer(cNoiseBuffer[3], srcBufferSize, 0.95, 0., coreSeed);    /* highly compressible */
-    RDG_genBuffer(cNoiseBuffer[4], srcBufferSize, 1.00, 0., coreSeed);    /* sparse content */
-
-    /* catch up testNb */
-    for (testNb=1; testNb < startTest; testNb++) FUZ_rand(&coreSeed);
-
-    /* main test loop */
-    for ( ; (testNb <= nbTests) || (UTIL_clockSpanMicro(startClock) < maxClockSpan); testNb++ ) {
-        BYTE* srcBuffer;   /* jumping pointer */
-        U32 lseed;
-        size_t sampleSize, maxTestSize, totalTestSize;
-        size_t cSize, totalCSize, totalGenSize;
-        U64 crcOrig;
-        BYTE* sampleBuffer;
-        const BYTE* dict;
-        size_t dictSize;
-
-        /* notification */
-        if (nbTests >= testNb) { DISPLAYUPDATE(2, "\r%6u/%6u    ", testNb, nbTests); }
-        else { DISPLAYUPDATE(2, "\r%6u          ", testNb); }
-
-        FUZ_rand(&coreSeed);
-        { U32 const prime1 = 2654435761U; lseed = coreSeed ^ prime1; }
-
-        /* srcBuffer selection [0-4] */
-        {   U32 buffNb = FUZ_rand(&lseed) & 0x7F;
-            if (buffNb & 7) buffNb=2;   /* most common : compressible (P) */
-            else {
-                buffNb >>= 3;
-                if (buffNb & 7) {
-                    const U32 tnb[2] = { 1, 3 };   /* barely/highly compressible */
-                    buffNb = tnb[buffNb >> 3];
-                } else {
-                    const U32 tnb[2] = { 0, 4 };   /* not compressible / sparse */
-                    buffNb = tnb[buffNb >> 3];
-            }   }
-            srcBuffer = cNoiseBuffer[buffNb];
-        }
-
-        /* select src segment */
-        sampleSize = FUZ_randomLength(&lseed, maxSampleLog);
-
-        /* create sample buffer (to catch read error with valgrind & sanitizers)  */
-        sampleBuffer = (BYTE*)malloc(sampleSize);
-        CHECK(sampleBuffer==NULL, "not enough memory for sample buffer");
-        { size_t const sampleStart = FUZ_rand(&lseed) % (srcBufferSize - sampleSize);
-          memcpy(sampleBuffer, srcBuffer + sampleStart, sampleSize); }
-        crcOrig = XXH64(sampleBuffer, sampleSize, 0);
-
-        /* compression tests */
-        {   int const cLevelPositive =
-                    ( FUZ_rand(&lseed) %
-                     (ZSTD_maxCLevel() - (FUZ_highbit32((U32)sampleSize) / cLevelLimiter)) )
-                    + 1;
-            int const cLevel = ((FUZ_rand(&lseed) & 15) == 3) ?
-                             - (int)((FUZ_rand(&lseed) & 7) + 1) :   /* test negative cLevel */
-                             cLevelPositive;
-            DISPLAYLEVEL(5, "fuzzer t%u: Simple compression test (level %i) \n", testNb, cLevel);
-            cSize = ZSTD_compressCCtx(ctx, cBuffer, cBufferSize, sampleBuffer, sampleSize, cLevel);
-            CHECK(ZSTD_isError(cSize), "ZSTD_compressCCtx failed : %s", ZSTD_getErrorName(cSize));
-
-            /* compression failure test : too small dest buffer */
-            assert(cSize > 3);
-            {   const size_t missing = (FUZ_rand(&lseed) % (cSize-2)) + 1;
-                const size_t tooSmallSize = cSize - missing;
-                const unsigned endMark = 0x4DC2B1A9;
-                memcpy(dstBuffer+tooSmallSize, &endMark, sizeof(endMark));
-                DISPLAYLEVEL(5, "fuzzer t%u: compress into too small buffer of size %u (missing %u bytes) \n",
-                            testNb, (unsigned)tooSmallSize, (unsigned)missing);
-                { size_t const errorCode = ZSTD_compressCCtx(ctx, dstBuffer, tooSmallSize, sampleBuffer, sampleSize, cLevel);
-                  CHECK(!ZSTD_isError(errorCode), "ZSTD_compressCCtx should have failed ! (buffer too small : %u < %u)", (unsigned)tooSmallSize, (unsigned)cSize); }
-                { unsigned endCheck; memcpy(&endCheck, dstBuffer+tooSmallSize, sizeof(endCheck));
-                  CHECK(endCheck != endMark, "ZSTD_compressCCtx : dst buffer overflow  (check.%08X != %08X.mark)", endCheck, endMark); }
-        }   }
-
-        /* frame header decompression test */
-        {   ZSTD_frameHeader zfh;
-            CHECK_Z( ZSTD_getFrameHeader(&zfh, cBuffer, cSize) );
-            CHECK(zfh.frameContentSize != sampleSize, "Frame content size incorrect");
-        }
-
-        /* Decompressed size test */
-        {   unsigned long long const rSize = ZSTD_findDecompressedSize(cBuffer, cSize);
-            CHECK(rSize != sampleSize, "decompressed size incorrect");
-        }
-
-        /* successful decompression test */
-        DISPLAYLEVEL(5, "fuzzer t%u: simple decompression test \n", testNb);
-        {   size_t const margin = (FUZ_rand(&lseed) & 1) ? 0 : (FUZ_rand(&lseed) & 31) + 1;
-            size_t const dSize = ZSTD_decompress(dstBuffer, sampleSize + margin, cBuffer, cSize);
-            CHECK(dSize != sampleSize, "ZSTD_decompress failed (%s) (srcSize : %u ; cSize : %u)", ZSTD_getErrorName(dSize), (unsigned)sampleSize, (unsigned)cSize);
-            {   U64 const crcDest = XXH64(dstBuffer, sampleSize, 0);
-                CHECK(crcOrig != crcDest, "decompression result corrupted (pos %u / %u)", (unsigned)findDiff(sampleBuffer, dstBuffer, sampleSize), (unsigned)sampleSize);
-        }   }
-
-        free(sampleBuffer);   /* no longer useful after this point */
-
-        /* truncated src decompression test */
-        DISPLAYLEVEL(5, "fuzzer t%u: decompression of truncated source \n", testNb);
-        {   size_t const missing = (FUZ_rand(&lseed) % (cSize-2)) + 1;   /* no problem, as cSize > 4 (frameHeaderSizer) */
-            size_t const tooSmallSize = cSize - missing;
-            void* cBufferTooSmall = malloc(tooSmallSize);   /* valgrind will catch read overflows */
-            CHECK(cBufferTooSmall == NULL, "not enough memory !");
-            memcpy(cBufferTooSmall, cBuffer, tooSmallSize);
-            { size_t const errorCode = ZSTD_decompress(dstBuffer, dstBufferSize, cBufferTooSmall, tooSmallSize);
-              CHECK(!ZSTD_isError(errorCode), "ZSTD_decompress should have failed ! (truncated src buffer)"); }
-            free(cBufferTooSmall);
-        }
-
-        /* too small dst decompression test */
-        DISPLAYLEVEL(5, "fuzzer t%u: decompress into too small dst buffer \n", testNb);
-        if (sampleSize > 3) {
-            size_t const missing = (FUZ_rand(&lseed) % (sampleSize-2)) + 1;   /* no problem, as cSize > 4 (frameHeaderSizer) */
-            size_t const tooSmallSize = sampleSize - missing;
-            static const BYTE token = 0xA9;
-            dstBuffer[tooSmallSize] = token;
-            { size_t const errorCode = ZSTD_decompress(dstBuffer, tooSmallSize, cBuffer, cSize);
-              CHECK(!ZSTD_isError(errorCode), "ZSTD_decompress should have failed : %u > %u (dst buffer too small)", (unsigned)errorCode, (unsigned)tooSmallSize); }
-            CHECK(dstBuffer[tooSmallSize] != token, "ZSTD_decompress : dst buffer overflow");
-        }
-
-        /* noisy src decompression test */
-        if (cSize > 6) {
-            /* insert noise into src */
-            {   U32 const maxNbBits = FUZ_highbit32((U32)(cSize-4));
-                size_t pos = 4;   /* preserve magic number (too easy to detect) */
-                for (;;) {
-                    /* keep some original src */
-                    {   U32 const nbBits = FUZ_rand(&lseed) % maxNbBits;
-                        size_t const mask = (1<<nbBits) - 1;
-                        size_t const skipLength = FUZ_rand(&lseed) & mask;
-                        pos += skipLength;
-                    }
-                    if (pos >= cSize) break;
-                    /* add noise */
-                    {   U32 const nbBitsCodes = FUZ_rand(&lseed) % maxNbBits;
-                        U32 const nbBits = nbBitsCodes ? nbBitsCodes-1 : 0;
-                        size_t const mask = (1<<nbBits) - 1;
-                        size_t const rNoiseLength = (FUZ_rand(&lseed) & mask) + 1;
-                        size_t const noiseLength = MIN(rNoiseLength, cSize-pos);
-                        size_t const noiseStart = FUZ_rand(&lseed) % (srcBufferSize - noiseLength);
-                        memcpy(cBuffer + pos, srcBuffer + noiseStart, noiseLength);
-                        pos += noiseLength;
-            }   }   }
-
-            /* decompress noisy source */
-            DISPLAYLEVEL(5, "fuzzer t%u: decompress noisy source \n", testNb);
-            {   U32 const endMark = 0xA9B1C3D6;
-                memcpy(dstBuffer+sampleSize, &endMark, 4);
-                {   size_t const decompressResult = ZSTD_decompress(dstBuffer, sampleSize, cBuffer, cSize);
-                    /* result *may* be an unlikely success, but even then, it must strictly respect dst buffer boundaries */
-                    CHECK((!ZSTD_isError(decompressResult)) && (decompressResult>sampleSize),
-                          "ZSTD_decompress on noisy src : result is too large : %u > %u (dst buffer)", (unsigned)decompressResult, (unsigned)sampleSize);
-                }
-                {   U32 endCheck; memcpy(&endCheck, dstBuffer+sampleSize, 4);
-                    CHECK(endMark!=endCheck, "ZSTD_decompress on noisy src : dst buffer overflow");
-        }   }   }   /* noisy src decompression test */
-
-        /*=====   Bufferless streaming compression test, scattered segments and dictionary   =====*/
-        DISPLAYLEVEL(5, "fuzzer t%u: Bufferless streaming compression test \n", testNb);
-        {   U32 const testLog = FUZ_rand(&lseed) % maxSrcLog;
-            U32 const dictLog = FUZ_rand(&lseed) % maxSrcLog;
-            int const cLevel = (FUZ_rand(&lseed) %
-                                (ZSTD_maxCLevel() -
-                                 (MAX(testLog, dictLog) / cLevelLimiter))) +
-                               1;
-            maxTestSize = FUZ_rLogLength(&lseed, testLog);
-            if (maxTestSize >= dstBufferSize) maxTestSize = dstBufferSize-1;
-
-            dictSize = FUZ_rLogLength(&lseed, dictLog);   /* needed also for decompression */
-            dict = srcBuffer + (FUZ_rand(&lseed) % (srcBufferSize - dictSize));
-
-            DISPLAYLEVEL(6, "fuzzer t%u: Compressing up to <=%u bytes at level %i with dictionary size %u \n",
-                            testNb, (unsigned)maxTestSize, cLevel, (unsigned)dictSize);
-
-            if (FUZ_rand(&lseed) & 0xF) {
-                CHECK_Z ( ZSTD_compressBegin_usingDict(refCtx, dict, dictSize, cLevel) );
-            } else {
-                ZSTD_compressionParameters const cPar = ZSTD_getCParams(cLevel, ZSTD_CONTENTSIZE_UNKNOWN, dictSize);
-                ZSTD_frameParameters const fPar = { FUZ_rand(&lseed)&1 /* contentSizeFlag */,
-                                                    !(FUZ_rand(&lseed)&3) /* contentChecksumFlag*/,
-                                                    0 /*NodictID*/ };   /* note : since dictionary is fake, dictIDflag has no impact */
-                ZSTD_parameters const p = FUZ_makeParams(cPar, fPar);
-                CHECK_Z ( ZSTD_compressBegin_advanced(refCtx, dict, dictSize, p, 0) );
-            }
-            CHECK_Z( ZSTD_copyCCtx(ctx, refCtx, 0) );
-        }
-
-        {   U32 const nbChunks = (FUZ_rand(&lseed) & 127) + 2;
-            U32 n;
-            XXH64_state_t xxhState;
-            XXH64_reset(&xxhState, 0);
-            for (totalTestSize=0, cSize=0, n=0 ; n<nbChunks ; n++) {
-                size_t const segmentSize = FUZ_randomLength(&lseed, maxSampleLog);
-                size_t const segmentStart = FUZ_rand(&lseed) % (srcBufferSize - segmentSize);
-
-                if (cBufferSize-cSize < ZSTD_compressBound(segmentSize)) break;   /* avoid invalid dstBufferTooSmall */
-                if (totalTestSize+segmentSize > maxTestSize) break;
-
-                {   size_t const compressResult = ZSTD_compressContinue(ctx, cBuffer+cSize, cBufferSize-cSize, srcBuffer+segmentStart, segmentSize);
-                    CHECK (ZSTD_isError(compressResult), "multi-segments compression error : %s", ZSTD_getErrorName(compressResult));
-                    cSize += compressResult;
-                }
-                XXH64_update(&xxhState, srcBuffer+segmentStart, segmentSize);
-                memcpy(mirrorBuffer + totalTestSize, srcBuffer+segmentStart, segmentSize);
-                totalTestSize += segmentSize;
-            }
-
-            {   size_t const flushResult = ZSTD_compressEnd(ctx, cBuffer+cSize, cBufferSize-cSize, NULL, 0);
-                CHECK (ZSTD_isError(flushResult), "multi-segments epilogue error : %s", ZSTD_getErrorName(flushResult));
-                cSize += flushResult;
-            }
-            crcOrig = XXH64_digest(&xxhState);
-        }
-
-        /* streaming decompression test */
-        DISPLAYLEVEL(5, "fuzzer t%u: Bufferless streaming decompression test \n", testNb);
-        /* ensure memory requirement is good enough (should always be true) */
-        {   ZSTD_frameHeader zfh;
-            CHECK( ZSTD_getFrameHeader(&zfh, cBuffer, ZSTD_FRAMEHEADERSIZE_MAX),
-                  "ZSTD_getFrameHeader(): error retrieving frame information");
-            {   size_t const roundBuffSize = ZSTD_decodingBufferSize_min(zfh.windowSize, zfh.frameContentSize);
-                CHECK_Z(roundBuffSize);
-                CHECK((roundBuffSize > totalTestSize) && (zfh.frameContentSize!=ZSTD_CONTENTSIZE_UNKNOWN),
-                      "ZSTD_decodingBufferSize_min() requires more memory (%u) than necessary (%u)",
-                      (unsigned)roundBuffSize, (unsigned)totalTestSize );
-        }   }
-        if (dictSize<8) dictSize=0, dict=NULL;   /* disable dictionary */
-        CHECK_Z( ZSTD_decompressBegin_usingDict(dctx, dict, dictSize) );
-        totalCSize = 0;
-        totalGenSize = 0;
-        while (totalCSize < cSize) {
-            size_t const inSize = ZSTD_nextSrcSizeToDecompress(dctx);
-            size_t const genSize = ZSTD_decompressContinue(dctx, dstBuffer+totalGenSize, dstBufferSize-totalGenSize, cBuffer+totalCSize, inSize);
-            CHECK (ZSTD_isError(genSize), "ZSTD_decompressContinue error : %s", ZSTD_getErrorName(genSize));
-            totalGenSize += genSize;
-            totalCSize += inSize;
-        }
-        CHECK (ZSTD_nextSrcSizeToDecompress(dctx) != 0, "frame not fully decoded");
-        CHECK (totalGenSize != totalTestSize, "streaming decompressed data : wrong size")
-        CHECK (totalCSize != cSize, "compressed data should be fully read")
-        {   U64 const crcDest = XXH64(dstBuffer, totalTestSize, 0);
-            CHECK(crcOrig != crcDest, "streaming decompressed data corrupted (pos %u / %u)",
-                (unsigned)findDiff(mirrorBuffer, dstBuffer, totalTestSize), (unsigned)totalTestSize);
-        }
-    }   /* for ( ; (testNb <= nbTests) */
-    DISPLAY("\r%u fuzzer tests completed   \n", testNb-1);
-
-_cleanup:
-    ZSTD_freeCCtx(refCtx);
-    ZSTD_freeCCtx(ctx);
-    ZSTD_freeDCtx(dctx);
-    free(cNoiseBuffer[0]);
-    free(cNoiseBuffer[1]);
-    free(cNoiseBuffer[2]);
-    free(cNoiseBuffer[3]);
-    free(cNoiseBuffer[4]);
-    free(cBuffer);
-    free(dstBuffer);
-    free(mirrorBuffer);
-    return result;
-
-_output_error:
-    result = 1;
-    goto _cleanup;
-}
-
-
-/*_*******************************************************
-*  Command line
-*********************************************************/
-static int FUZ_usage(const char* programName)
-{
-    DISPLAY( "Usage :\n");
-    DISPLAY( "      %s [args]\n", programName);
-    DISPLAY( "\n");
-    DISPLAY( "Arguments :\n");
-    DISPLAY( " -i#    : Nb of tests (default:%i) \n", nbTestsDefault);
-    DISPLAY( " -s#    : Select seed (default:prompt user)\n");
-    DISPLAY( " -t#    : Select starting test number (default:0)\n");
-    DISPLAY( " -P#    : Select compressibility in %% (default:%i%%)\n", FUZ_compressibility_default);
-    DISPLAY( " -v     : verbose\n");
-    DISPLAY( " -p     : pause at the end\n");
-    DISPLAY( " -h     : display help and exit\n");
-    return 0;
-}
-
-/*! readU32FromChar() :
-    @return : unsigned integer value read from input in `char` format
-    allows and interprets K, KB, KiB, M, MB and MiB suffix.
-    Will also modify `*stringPtr`, advancing it to position where it stopped reading.
-    Note : function result can overflow if digit string > MAX_UINT */
-static unsigned readU32FromChar(const char** stringPtr)
-{
-    unsigned result = 0;
-    while ((**stringPtr >='0') && (**stringPtr <='9'))
-        result *= 10, result += **stringPtr - '0', (*stringPtr)++ ;
-    if ((**stringPtr=='K') || (**stringPtr=='M')) {
-        result <<= 10;
-        if (**stringPtr=='M') result <<= 10;
-        (*stringPtr)++ ;
-        if (**stringPtr=='i') (*stringPtr)++;
-        if (**stringPtr=='B') (*stringPtr)++;
-    }
-    return result;
-}
-
-/** longCommandWArg() :
- *  check if *stringPtr is the same as longCommand.
- *  If yes, @return 1 and advances *stringPtr to the position which immediately follows longCommand.
- *  @return 0 and doesn't modify *stringPtr otherwise.
- */
-static int longCommandWArg(const char** stringPtr, const char* longCommand)
-{
-    size_t const comSize = strlen(longCommand);
-    int const result = !strncmp(*stringPtr, longCommand, comSize);
-    if (result) *stringPtr += comSize;
-    return result;
-}
-
-int main(int argc, const char** argv)
-{
-    U32 seed = 0;
-    int seedset = 0;
-    int argNb;
-    int nbTests = nbTestsDefault;
-    int testNb = 0;
-    int proba = FUZ_compressibility_default;
-    int result = 0;
-    U32 mainPause = 0;
-    U32 maxDuration = 0;
-    int bigTests = 1;
-    U32 memTestsOnly = 0;
-    const char* const programName = argv[0];
-
-    /* Check command line */
-    for (argNb=1; argNb<argc; argNb++) {
-        const char* argument = argv[argNb];
-        if(!argument) continue;   /* Protection if argument empty */
-
-        /* Handle commands. Aggregated commands are allowed */
-        if (argument[0]=='-') {
-
-            if (longCommandWArg(&argument, "--memtest=")) { memTestsOnly = readU32FromChar(&argument); continue; }
-
-            if (!strcmp(argument, "--memtest")) { memTestsOnly=1; continue; }
-            if (!strcmp(argument, "--no-big-tests")) { bigTests=0; continue; }
-
-            argument++;
-            while (*argument!=0) {
-                switch(*argument)
-                {
-                case 'h':
-                    return FUZ_usage(programName);
-
-                case 'v':
-                    argument++;
-                    g_displayLevel++;
-                    break;
-
-                case 'q':
-                    argument++;
-                    g_displayLevel--;
-                    break;
-
-                case 'p': /* pause at the end */
-                    argument++;
-                    mainPause = 1;
-                    break;
-
-                case 'i':
-                    argument++; maxDuration = 0;
-                    nbTests = (int)readU32FromChar(&argument);
-                    break;
-
-                case 'T':
-                    argument++;
-                    nbTests = 0;
-                    maxDuration = readU32FromChar(&argument);
-                    if (*argument=='s') argument++;   /* seconds */
-                    if (*argument=='m') maxDuration *= 60, argument++;   /* minutes */
-                    if (*argument=='n') argument++;
-                    break;
-
-                case 's':
-                    argument++;
-                    seedset = 1;
-                    seed = readU32FromChar(&argument);
-                    break;
-
-                case 't':
-                    argument++;
-                    testNb = (int)readU32FromChar(&argument);
-                    break;
-
-                case 'P':   /* compressibility % */
-                    argument++;
-                    proba = (int)readU32FromChar(&argument);
-                    if (proba>100) proba = 100;
-                    break;
-
-                default:
-                    return (FUZ_usage(programName), 1);
-    }   }   }   }   /* for (argNb=1; argNb<argc; argNb++) */
-
-    /* Get Seed */
-    DISPLAY("Starting zstd tester (%i-bits, %s)\n", (int)(sizeof(size_t)*8), ZSTD_VERSION_STRING);
-
-    if (!seedset) {
-        time_t const t = time(NULL);
-        U32 const h = XXH32(&t, sizeof(t), 1);
-        seed = h % 10000;
-    }
-
-    DISPLAY("Seed = %u\n", (unsigned)seed);
-    if (proba!=FUZ_compressibility_default) DISPLAY("Compressibility : %i%%\n", proba);
-
-    if (memTestsOnly) {
-        g_displayLevel = MAX(3, g_displayLevel);
-        return FUZ_mallocTests(seed, ((double)proba) / 100, memTestsOnly);
-    }
-
-    if (nbTests < testNb) nbTests = testNb;
-
-    if (testNb==0)
-        result = basicUnitTests(0, ((double)proba) / 100);  /* constant seed for predictability */
-    if (!result)
-        result = fuzzerTests(seed, nbTests, testNb, maxDuration, ((double)proba) / 100, bigTests);
-    if (mainPause) {
-        int unused;
-        DISPLAY("Press Enter \n");
-        unused = getchar();
-        (void)unused;
-    }
-    return result;
-}
diff --git a/tests/golden-compression/huffman-compressed-larger b/tests/golden-compression/huffman-compressed-larger
deleted file mode 100644
index f594f1ae9816..000000000000
--- a/tests/golden-compression/huffman-compressed-larger
+++ /dev/null
diff --git a/tests/golden-decompression/rle-first-block.zst b/tests/golden-decompression/rle-first-block.zst
deleted file mode 100644
index fd067edd74ef..000000000000
--- a/tests/golden-decompression/rle-first-block.zst
+++ /dev/null
diff --git a/tests/gzip/Makefile b/tests/gzip/Makefile
deleted file mode 100644
index c5d67206b99d..000000000000
--- a/tests/gzip/Makefile
+++ /dev/null
@@ -1,44 +0,0 @@
-# ################################################################
-# Copyright (c) 2017-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under both the BSD-style license (found in the
-# LICENSE file in the root directory of this source tree) and the GPLv2 (found
-# in the COPYING file in the root directory of this source tree).
-# ################################################################
-
-PRGDIR = ../../programs
-VOID   = /dev/null
-export PATH := .:$(PATH)
-
-.PHONY: all
-#all: test-gzip-env
-all: test-helin-segv test-hufts test-keep test-list test-memcpy-abuse test-mixed
-all: test-null-suffix-clobber test-stdin test-trailing-nul test-unpack-invalid
-all: test-zdiff test-zgrep-context test-zgrep-f test-zgrep-signal test-znew-k test-z-suffix
-	@echo Testing completed
-
-.PHONY: zstd
-zstd:
-	$(MAKE) -C $(PRGDIR) zstd
-	ln -sf $(PRGDIR)/zstd gzip
-	@echo PATH=$(PATH)
-	gzip --version
-
-.PHONY: clean
-clean:
-	@$(MAKE) -C $(PRGDIR) $@ > $(VOID)
-	@$(RM) *.trs *.log
-	@echo Cleaning completed
-
-
-#------------------------------------------------------------------------------
-# validated only for Linux, macOS, Hurd and some BSD targets
-#------------------------------------------------------------------------------
-ifneq (,$(filter $(shell uname),Linux Darwin GNU/kFreeBSD GNU FreeBSD DragonFly NetBSD))
-
-test-%: zstd
-	@./test-driver.sh --test-name $* --log-file $*.log --trs-file $*.trs --expect-failure "no" --color-tests "yes" --enable-hard-errors "yes" ./$*.sh
-	# || echo ignoring error
-
-endif
diff --git a/tests/gzip/gzip-env.sh b/tests/gzip/gzip-env.sh
deleted file mode 100755
index 120e52d78d49..000000000000
--- a/tests/gzip/gzip-env.sh
+++ /dev/null
@@ -1,46 +0,0 @@
-#!/bin/sh
-# Test the obsolescent GZIP environment variable.
-
-# Copyright 2015-2016 Free Software Foundation, Inc.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-# limit so don't run it by default.
-
-. "${srcdir=.}/init.sh"; path_prepend_ .
-
-#echo PATH=$PATH
-#gzip --version
-
-echo a >exp || framework_failure_
-gzip <exp >in || framework_failure_
-
-fail=0
-GZIP=-qv gzip -d <in >out 2>err || fail=1
-compare exp out || fail=1
-
-for badopt in -- -c --stdout -d --decompress -f --force -h --help -k --keep \
-  -l --list -L --license -r --recursive -Sxxx --suffix=xxx '--suffix xxx' \
-  -t --test -V --version
-do
-  GZIP=$badopt gzip -d <in >out 2>err && fail=1
-done
-
-for goodopt in -n --no-name -N --name -q --quiet -v --verbose \
-  -1 --fast -2 -3 -4 -5 -6 -7 -8 -9 --best
-do
-  GZIP=$goodopt gzip -d <in >out 2>err || fail=1
-  compare exp out || fail=1
-done
-
-Exit $fail
diff --git a/tests/gzip/helin-segv.sh b/tests/gzip/helin-segv.sh
deleted file mode 100755
index f182c8066f34..000000000000
--- a/tests/gzip/helin-segv.sh
+++ /dev/null
@@ -1,31 +0,0 @@
-#!/bin/sh
-# Before gzip-1.4, gzip -d would segfault on some inputs.
-
-# Copyright (C) 2010-2016 Free Software Foundation, Inc.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-# limit so don't run it by default.
-
-. "${srcdir=.}/init.sh"; path_prepend_ .
-
-# This test case was provided by Aki Helin.
-printf '\037\235\220\0\0\0\304' > helin.gz || framework_failure_
-printf '\0\0' > exp || framework_failure_
-
-fail=0
-
-gzip -dc helin.gz > out || fail=1
-compare exp out || fail=1
-
-Exit $fail
diff --git a/tests/gzip/help-version.sh b/tests/gzip/help-version.sh
deleted file mode 100755
index ee0c19f7d1fc..000000000000
--- a/tests/gzip/help-version.sh
+++ /dev/null
@@ -1,270 +0,0 @@
-#! /bin/sh
-# Make sure all these programs work properly
-# when invoked with --help or --version.
-
-# Copyright (C) 2000-2016 Free Software Foundation, Inc.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-# Ensure that $SHELL is set to *some* value and exported.
-# This is required for dircolors, which would fail e.g., when
-# invoked via debuild (which removes SHELL from the environment).
-test "x$SHELL" = x && SHELL=/bin/sh
-export SHELL
-
-. "${srcdir=.}/init.sh"; path_prepend_ .
-
-expected_failure_status_chroot=125
-expected_failure_status_env=125
-expected_failure_status_nice=125
-expected_failure_status_nohup=125
-expected_failure_status_stdbuf=125
-expected_failure_status_su=125
-expected_failure_status_timeout=125
-expected_failure_status_printenv=2
-expected_failure_status_tty=3
-expected_failure_status_sort=2
-expected_failure_status_expr=3
-expected_failure_status_lbracket=2
-expected_failure_status_dir=2
-expected_failure_status_ls=2
-expected_failure_status_vdir=2
-
-expected_failure_status_cmp=2
-expected_failure_status_zcmp=2
-expected_failure_status_sdiff=2
-expected_failure_status_diff3=2
-expected_failure_status_diff=2
-expected_failure_status_zdiff=2
-expected_failure_status_zgrep=2
-expected_failure_status_zegrep=2
-expected_failure_status_zfgrep=2
-
-expected_failure_status_grep=2
-expected_failure_status_egrep=2
-expected_failure_status_fgrep=2
-
-test "$built_programs" \
-  || fail_ "built_programs not specified!?!"
-
-test "$VERSION" \
-  || fail_ "set envvar VERSION; it is required for a PATH sanity-check"
-
-# Extract version from --version output of the first program
-for i in $built_programs; do
-  v=$(env $i --version | sed -n '1s/.* //p;q')
-  break
-done
-
-# Ensure that it matches $VERSION.
-test "x$v" = "x$VERSION" \
-  || fail_ "--version-\$VERSION mismatch"
-
-for lang in C fr da; do
-  for i in $built_programs; do
-
-    # Skip `test'; it doesn't accept --help or --version.
-    test $i = test && continue;
-
-    # false fails even when invoked with --help or --version.
-    if test $i = false; then
-      env LC_MESSAGES=$lang $i --help >/dev/null && fail=1
-      env LC_MESSAGES=$lang $i --version >/dev/null && fail=1
-      continue
-    fi
-
-    args=
-
-    # The just-built install executable is always named `ginstall'.
-    test $i = install && i=ginstall
-
-    # Make sure they exit successfully, under normal conditions.
-    eval "env \$i $args --help    > h-\$i   " || fail=1
-    eval "env \$i $args --version >/dev/null" || fail=1
-
-    # Make sure they mention the bug-reporting address in --help output.
-    grep "$PACKAGE_BUGREPORT" h-$i > /dev/null || fail=1
-    rm -f h-$i
-
-    # Make sure they fail upon `disk full' error.
-    if test -w /dev/full && test -c /dev/full; then
-      eval "env \$i $args --help    >/dev/full 2>/dev/null" && fail=1
-      eval "env \$i $args --version >/dev/full 2>/dev/null" && fail=1
-      status=$?
-      test $i = [ && prog=lbracket || prog=$i
-      eval "expected=\$expected_failure_status_$prog"
-      test x$expected = x && expected=1
-      if test $status = $expected; then
-        : # ok
-      else
-        fail=1
-        echo "*** $i: bad exit status \`$status' (expected $expected)," 1>&2
-        echo "  with --help or --version output redirected to /dev/full" 1>&2
-      fi
-    fi
-  done
-done
-
-bigZ_in=bigZ-in.Z
-zin=zin.gz
-zin2=zin2.gz
-
-tmp=tmp-$$
-tmp_in=in-$$
-tmp_in2=in2-$$
-tmp_dir=dir-$$
-tmp_out=out-$$
-mkdir $tmp || fail=1
-cd $tmp || fail=1
-
-comm_setup () { args="$tmp_in $tmp_in"; }
-csplit_setup () { args="$tmp_in //"; }
-cut_setup () { args='-f 1'; }
-join_setup () { args="$tmp_in $tmp_in"; }
-tr_setup () { args='a a'; }
-
-chmod_setup () { args="a+x $tmp_in"; }
-# Punt on these.
-chgrp_setup () { args=--version; }
-chown_setup () { args=--version; }
-mkfifo_setup () { args=--version; }
-mknod_setup () { args=--version; }
-# Punt on uptime, since it fails (e.g., failing to get boot time)
-# on some systems, and we shouldn't let that stop `make check'.
-uptime_setup () { args=--version; }
-
-# Create a file in the current directory, not in $TMPDIR.
-mktemp_setup () { args=mktemp.XXXX; }
-
-cmp_setup () { args="$tmp_in $tmp_in2"; }
-
-# Tell dd not to print the line with transfer rate and total.
-# The transfer rate would vary between runs.
-dd_setup () { args=status=noxfer; }
-
-zdiff_setup () { args="$args $zin $zin2"; }
-zcmp_setup () { zdiff_setup; }
-zcat_setup () { args="$args $zin"; }
-gunzip_setup () { zcat_setup; }
-zmore_setup () { zcat_setup; }
-zless_setup () { zcat_setup; }
-znew_setup () { args="$args $bigZ_in"; }
-zforce_setup () { zcat_setup; }
-zgrep_setup () { args="$args z $zin"; }
-zegrep_setup () { zgrep_setup; }
-zfgrep_setup () { zgrep_setup; }
-gzexe_setup () { args="$args $tmp_in"; }
-
-# We know that $tmp_in contains a "0"
-grep_setup () { args="0 $tmp_in"; }
-egrep_setup () { args="0 $tmp_in"; }
-fgrep_setup () { args="0 $tmp_in"; }
-
-diff_setup () { args="$tmp_in $tmp_in2"; }
-sdiff_setup () { args="$tmp_in $tmp_in2"; }
-diff3_setup () { args="$tmp_in $tmp_in2 $tmp_in2"; }
-cp_setup () { args="$tmp_in $tmp_in2"; }
-ln_setup () { args="$tmp_in ln-target"; }
-ginstall_setup () { args="$tmp_in $tmp_in2"; }
-mv_setup () { args="$tmp_in $tmp_in2"; }
-mkdir_setup () { args=$tmp_dir/subdir; }
-rmdir_setup () { args=$tmp_dir; }
-rm_setup () { args=$tmp_in; }
-shred_setup () { args=$tmp_in; }
-touch_setup () { args=$tmp_in2; }
-truncate_setup () { args="--reference=$tmp_in $tmp_in2"; }
-
-basename_setup () { args=$tmp_in; }
-dirname_setup () { args=$tmp_in; }
-expr_setup () { args=foo; }
-
-# Punt, in case GNU `id' hasn't been installed yet.
-groups_setup () { args=--version; }
-
-pathchk_setup () { args=$tmp_in; }
-yes_setup () { args=--version; }
-logname_setup () { args=--version; }
-nohup_setup () { args=--version; }
-printf_setup () { args=foo; }
-seq_setup () { args=10; }
-sleep_setup () { args=0; }
-su_setup () { args=--version; }
-stdbuf_setup () { args="-oL true"; }
-timeout_setup () { args=--version; }
-
-# I'd rather not run sync, since it spins up disks that I've
-# deliberately caused to spin down (but not unmounted).
-sync_setup () { args=--version; }
-
-test_setup () { args=foo; }
-
-# This is necessary in the unusual event that there is
-# no valid entry in /etc/mtab.
-df_setup () { args=/; }
-
-# This is necessary in the unusual event that getpwuid (getuid ()) fails.
-id_setup () { args=-u; }
-
-# Use env to avoid invoking built-in sleep of Solaris 11's /bin/sh.
-kill_setup () {
-  env sleep 10m &
-  args=$!
-}
-
-link_setup () { args="$tmp_in link-target"; }
-unlink_setup () { args=$tmp_in; }
-
-readlink_setup () {
-  ln -s . slink
-  args=slink;
-}
-
-stat_setup () { args=$tmp_in; }
-unlink_setup () { args=$tmp_in; }
-lbracket_setup () { args=": ]"; }
-
-# Ensure that each program "works" (exits successfully) when doing
-# something more than --help or --version.
-for i in $built_programs; do
-  # Skip these.
-  case $i in chroot|stty|tty|false|chcon|runcon) continue;; esac
-
-  rm -rf $tmp_in $tmp_in2 $tmp_dir $tmp_out $bigZ_in $zin $zin2
-  echo z |gzip > $zin
-  cp $zin $zin2
-  cp $zin $bigZ_in
-
-  # This is sort of kludgey: use numbers so this is valid input for factor,
-  # and two tokens so it's valid input for tsort.
-  echo 2147483647 0 > $tmp_in
-  # Make $tmp_in2 identical. Then, using $tmp_in and $tmp_in2 as arguments
-  # to the likes of cmp and diff makes them exit successfully.
-  cp $tmp_in $tmp_in2
-  mkdir $tmp_dir
-  # echo ================== $i
-  test $i = [ && prog=lbracket || prog=$i
-  args=
-  if type ${prog}_setup > /dev/null 2>&1; then
-    ${prog}_setup
-  fi
-  if eval "env \$i $args < \$tmp_in > \$tmp_out"; then
-    : # ok
-  else
-    echo FAIL: $i
-    fail=1
-  fi
-  rm -rf $tmp_in $tmp_in2 $tmp_out $tmp_dir
-done
-
-Exit $fail
diff --git a/tests/gzip/hufts-segv.gz b/tests/gzip/hufts-segv.gz
deleted file mode 100644
index 32cb2a256844..000000000000
--- a/tests/gzip/hufts-segv.gz
+++ /dev/null
diff --git a/tests/gzip/hufts.sh b/tests/gzip/hufts.sh
deleted file mode 100755
index 9b9576ce34e5..000000000000
--- a/tests/gzip/hufts.sh
+++ /dev/null
@@ -1,34 +0,0 @@
-#!/bin/sh
-# Exercise a bug whereby an invalid input could make gzip -d misbehave.
-
-# Copyright (C) 2009-2016 Free Software Foundation, Inc.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-# limit so don't run it by default.
-
-. "${srcdir=.}/init.sh"; path_prepend_ .
-
-printf '\n...: invalid compressed data--format violated\n' > exp \
-  || framework_failure_
-
-fail=0
-gzip -dc "$abs_srcdir/hufts-segv.gz" > out 2> err
-test $? = 1 || fail=1
-
-compare /dev/null out || fail=1
-
-sed 's/.*hufts-segv.gz: /...: /' err > k; mv k err || fail=1
-compare exp err || fail=1
-
-Exit $fail
diff --git a/tests/gzip/init.cfg b/tests/gzip/init.cfg
deleted file mode 100644
index 901209ceae96..000000000000
--- a/tests/gzip/init.cfg
+++ /dev/null
@@ -1,5 +0,0 @@
-# This file is sourced by init.sh, *before* its initialization.
-
-# This goes hand in hand with the "exec 9>&2;" in Makefile.am's
-# TESTS_ENVIRONMENT definition.
-stderr_fileno_=9
diff --git a/tests/gzip/init.sh b/tests/gzip/init.sh
deleted file mode 100755
index 97e4e4ba5e69..000000000000
--- a/tests/gzip/init.sh
+++ /dev/null
@@ -1,616 +0,0 @@
-# source this file; set up for tests
-
-# Copyright (C) 2009-2016 Free Software Foundation, Inc.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-# Using this file in a test
-# =========================
-#
-# The typical skeleton of a test looks like this:
-#
-#   #!/bin/sh
-#   . "${srcdir=.}/init.sh"; path_prepend_ .
-#   Execute some commands.
-#   Note that these commands are executed in a subdirectory, therefore you
-#   need to prepend "../" to relative filenames in the build directory.
-#   Note that the "path_prepend_ ." is useful only if the body of your
-#   test invokes programs residing in the initial directory.
-#   For example, if the programs you want to test are in src/, and this test
-#   script is named tests/test-1, then you would use "path_prepend_ ../src",
-#   or perhaps export PATH='$(abs_top_builddir)/src$(PATH_SEPARATOR)'"$$PATH"
-#   to all tests via automake's TESTS_ENVIRONMENT.
-#   Set the exit code 0 for success, 77 for skipped, or 1 or other for failure.
-#   Use the skip_ and fail_ functions to print a diagnostic and then exit
-#   with the corresponding exit code.
-#   Exit $?
-
-# Executing a test that uses this file
-# ====================================
-#
-# Running a single test:
-#   $ make check TESTS=test-foo.sh
-#
-# Running a single test, with verbose output:
-#   $ make check TESTS=test-foo.sh VERBOSE=yes
-#
-# Running a single test, with single-stepping:
-#   1. Go into a sub-shell:
-#   $ bash
-#   2. Set relevant environment variables from TESTS_ENVIRONMENT in the
-#      Makefile:
-#   $ export srcdir=../../tests # this is an example
-#   3. Execute the commands from the test, copy&pasting them one by one:
-#   $ . "$srcdir/init.sh"; path_prepend_ .
-#   ...
-#   4. Finally
-#   $ exit
-
-ME_=`expr "./$0" : '.*/\(.*\)$'`
-
-# We use a trap below for cleanup.  This requires us to go through
-# hoops to get the right exit status transported through the handler.
-# So use 'Exit STATUS' instead of 'exit STATUS' inside of the tests.
-# Turn off errexit here so that we don't trip the bug with OSF1/Tru64
-# sh inside this function.
-Exit () { set +e; (exit $1); exit $1; }
-
-# Print warnings (e.g., about skipped and failed tests) to this file number.
-# Override by defining to say, 9, in init.cfg, and putting say,
-#   export ...ENVVAR_SETTINGS...; $(SHELL) 9>&2
-# in the definition of TESTS_ENVIRONMENT in your tests/Makefile.am file.
-# This is useful when using automake's parallel tests mode, to print
-# the reason for skip/failure to console, rather than to the .log files.
-: ${stderr_fileno_=2}
-
-# Note that correct expansion of "$*" depends on IFS starting with ' '.
-# Always write the full diagnostic to stderr.
-# When stderr_fileno_ is not 2, also emit the first line of the
-# diagnostic to that file descriptor.
-warn_ ()
-{
-  # If IFS does not start with ' ', set it and emit the warning in a subshell.
-  case $IFS in
-    ' '*) printf '%s\n' "$*" >&2
-          test $stderr_fileno_ = 2 \
-            || { printf '%s\n' "$*" | sed 1q >&$stderr_fileno_ ; } ;;
-    *) (IFS=' '; warn_ "$@");;
-  esac
-}
-fail_ () { warn_ "$ME_: failed test: $@"; Exit 1; }
-skip_ () { warn_ "$ME_: skipped test: $@"; Exit 77; }
-fatal_ () { warn_ "$ME_: hard error: $@"; Exit 99; }
-framework_failure_ () { warn_ "$ME_: set-up failure: $@"; Exit 99; }
-
-# This is used to simplify checking of the return value
-# which is useful when ensuring a command fails as desired.
-# I.e., just doing `command ... &&fail=1` will not catch
-# a segfault in command for example.  With this helper you
-# instead check an explicit exit code like
-#   returns_ 1 command ... || fail
-returns_ () {
-  # Disable tracing so it doesn't interfere with stderr of the wrapped command
-  { set +x; } 2>/dev/null
-
-  local exp_exit="$1"
-  shift
-  "$@"
-  test $? -eq $exp_exit && ret_=0 || ret_=1
-
-  if test "$VERBOSE" = yes && test "$gl_set_x_corrupts_stderr_" = false; then
-    set -x
-  fi
-  { return $ret_; } 2>/dev/null
-}
-
-# Sanitize this shell to POSIX mode, if possible.
-DUALCASE=1; export DUALCASE
-if test -n "${ZSH_VERSION+set}" && (emulate sh) >/dev/null 2>&1; then
-  emulate sh
-  NULLCMD=:
-  alias -g '${1+"$@"}'='"$@"'
-  setopt NO_GLOB_SUBST
-else
-  case `(set -o) 2>/dev/null` in
-    *posix*) set -o posix ;;
-  esac
-fi
-
-# We require $(...) support unconditionally.
-# We require a few additional shell features only when $EXEEXT is nonempty,
-# in order to support automatic $EXEEXT emulation:
-# - hyphen-containing alias names
-# - we prefer to use ${var#...} substitution, rather than having
-#   to work around lack of support for that feature.
-# The following code attempts to find a shell with support for these features.
-# If the current shell passes the test, we're done.  Otherwise, test other
-# shells until we find one that passes.  If one is found, re-exec it.
-# If no acceptable shell is found, skip the current test.
-#
-# The "...set -x; P=1 true 2>err..." test is to disqualify any shell that
-# emits "P=1" into err, as /bin/sh from SunOS 5.11 and OpenBSD 4.7 do.
-#
-# Use "9" to indicate success (rather than 0), in case some shell acts
-# like Solaris 10's /bin/sh but exits successfully instead of with status 2.
-
-# Eval this code in a subshell to determine a shell's suitability.
-# 10 - passes all tests; ok to use
-#  9 - ok, but enabling "set -x" corrupts app stderr; prefer higher score
-#  ? - not ok
-gl_shell_test_script_='
-test $(echo y) = y || exit 1
-f_local_() { local v=1; }; f_local_ || exit 1
-score_=10
-if test "$VERBOSE" = yes; then
-  test -n "$( (exec 3>&1; set -x; P=1 true 2>&3) 2> /dev/null)" && score_=9
-fi
-test -z "$EXEEXT" && exit $score_
-shopt -s expand_aliases
-alias a-b="echo zoo"
-v=abx
-     test ${v%x} = ab \
-  && test ${v#a} = bx \
-  && test $(a-b) = zoo \
-  && exit $score_
-'
-
-if test "x$1" = "x--no-reexec"; then
-  shift
-else
-  # Assume a working shell.  Export to subshells (setup_ needs this).
-  gl_set_x_corrupts_stderr_=false
-  export gl_set_x_corrupts_stderr_
-
-  # Record the first marginally acceptable shell.
-  marginal_=
-
-  # Search for a shell that meets our requirements.
-  for re_shell_ in __current__ "${CONFIG_SHELL:-no_shell}" \
-      /bin/sh bash dash zsh pdksh fail
-  do
-    test "$re_shell_" = no_shell && continue
-
-    # If we've made it all the way to the sentinel, "fail" without
-    # finding even a marginal shell, skip this test.
-    if test "$re_shell_" = fail; then
-      test -z "$marginal_" && skip_ failed to find an adequate shell
-      re_shell_=$marginal_
-      break
-    fi
-
-    # When testing the current shell, simply "eval" the test code.
-    # Otherwise, run it via $re_shell_ -c ...
-    if test "$re_shell_" = __current__; then
-      # 'eval'ing this code makes Solaris 10's /bin/sh exit with
-      # $? set to 2.  It does not evaluate any of the code after the
-      # "unexpected" first '('.  Thus, we must run it in a subshell.
-      ( eval "$gl_shell_test_script_" ) > /dev/null 2>&1
-    else
-      "$re_shell_" -c "$gl_shell_test_script_" 2>/dev/null
-    fi
-
-    st_=$?
-
-    # $re_shell_ works just fine.  Use it.
-    if test $st_ = 10; then
-      gl_set_x_corrupts_stderr_=false
-      break
-    fi
-
-    # If this is our first marginally acceptable shell, remember it.
-    if test "$st_:$marginal_" = 9: ; then
-      marginal_="$re_shell_"
-      gl_set_x_corrupts_stderr_=true
-    fi
-  done
-
-  if test "$re_shell_" != __current__; then
-    # Found a usable shell.  Preserve -v and -x.
-    case $- in
-      *v*x* | *x*v*) opts_=-vx ;;
-      *v*) opts_=-v ;;
-      *x*) opts_=-x ;;
-      *) opts_= ;;
-    esac
-    re_shell=$re_shell_
-    export re_shell
-    exec "$re_shell_" $opts_ "$0" --no-reexec "$@"
-    echo "$ME_: exec failed" 1>&2
-    exit 127
-  fi
-fi
-
-# If this is bash, turn off all aliases.
-test -n "$BASH_VERSION" && unalias -a
-
-# Note that when supporting $EXEEXT (transparently mapping from PROG_NAME to
-# PROG_NAME.exe), we want to support hyphen-containing names like test-acos.
-# That is part of the shell-selection test above.  Why use aliases rather
-# than functions?  Because support for hyphen-containing aliases is more
-# widespread than that for hyphen-containing function names.
-test -n "$EXEEXT" && shopt -s expand_aliases
-
-# Enable glibc's malloc-perturbing option.
-# This is useful for exposing code that depends on the fact that
-# malloc-related functions often return memory that is mostly zeroed.
-# If you have the time and cycles, use valgrind to do an even better job.
-: ${MALLOC_PERTURB_=87}
-export MALLOC_PERTURB_
-
-# This is a stub function that is run upon trap (upon regular exit and
-# interrupt).  Override it with a per-test function, e.g., to unmount
-# a partition, or to undo any other global state changes.
-cleanup_ () { :; }
-
-# Emit a header similar to that from diff -u;  Print the simulated "diff"
-# command so that the order of arguments is clear.  Don't bother with @@ lines.
-emit_diff_u_header_ ()
-{
-  printf '%s\n' "diff -u $*" \
-    "--- $1	1970-01-01" \
-    "+++ $2	1970-01-01"
-}
-
-# Arrange not to let diff or cmp operate on /dev/null,
-# since on some systems (at least OSF/1 5.1), that doesn't work.
-# When there are not two arguments, or no argument is /dev/null, return 2.
-# When one argument is /dev/null and the other is not empty,
-# cat the nonempty file to stderr and return 1.
-# Otherwise, return 0.
-compare_dev_null_ ()
-{
-  test $# = 2 || return 2
-
-  if test "x$1" = x/dev/null; then
-    test -s "$2" || return 0
-    emit_diff_u_header_ "$@"; sed 's/^/+/' "$2"
-    return 1
-  fi
-
-  if test "x$2" = x/dev/null; then
-    test -s "$1" || return 0
-    emit_diff_u_header_ "$@"; sed 's/^/-/' "$1"
-    return 1
-  fi
-
-  return 2
-}
-
-if diff_out_=`exec 2>/dev/null; diff -u "$0" "$0" < /dev/null` \
-   && diff -u Makefile "$0" 2>/dev/null | grep '^[+]#!' >/dev/null; then
-  # diff accepts the -u option and does not (like AIX 7 'diff') produce an
-  # extra space on column 1 of every content line.
-  if test -z "$diff_out_"; then
-    compare_ () { diff -u "$@"; }
-  else
-    compare_ ()
-    {
-      if diff -u "$@" > diff.out; then
-        # No differences were found, but Solaris 'diff' produces output
-        # "No differences encountered". Hide this output.
-        rm -f diff.out
-        true
-      else
-        cat diff.out
-        rm -f diff.out
-        false
-      fi
-    }
-  fi
-elif
-  for diff_opt_ in -U3 -c '' no; do
-    test "$diff_opt_" = no && break
-    diff_out_=`exec 2>/dev/null; diff $diff_opt_ "$0" "$0" </dev/null` && break
-  done
-  test "$diff_opt_" != no
-then
-  if test -z "$diff_out_"; then
-    compare_ () { diff $diff_opt_ "$@"; }
-  else
-    compare_ ()
-    {
-      if diff $diff_opt_ "$@" > diff.out; then
-        # No differences were found, but AIX and HP-UX 'diff' produce output
-        # "No differences encountered" or "There are no differences between the
-        # files.". Hide this output.
-        rm -f diff.out
-        true
-      else
-        cat diff.out
-        rm -f diff.out
-        false
-      fi
-    }
-  fi
-elif cmp -s /dev/null /dev/null 2>/dev/null; then
-  compare_ () { cmp -s "$@"; }
-else
-  compare_ () { cmp "$@"; }
-fi
-
-# Usage: compare EXPECTED ACTUAL
-#
-# Given compare_dev_null_'s preprocessing, defer to compare_ if 2 or more.
-# Otherwise, propagate $? to caller: any diffs have already been printed.
-compare ()
-{
-  # This looks like it can be factored to use a simple "case $?"
-  # after unchecked compare_dev_null_ invocation, but that would
-  # fail in a "set -e" environment.
-  if compare_dev_null_ "$@"; then
-    return 0
-  else
-    case $? in
-      1) return 1;;
-      *) compare_ "$@";;
-    esac
-  fi
-}
-
-# An arbitrary prefix to help distinguish test directories.
-testdir_prefix_ () { printf gt; }
-
-# Run the user-overridable cleanup_ function, remove the temporary
-# directory and exit with the incoming value of $?.
-remove_tmp_ ()
-{
-  __st=$?
-  cleanup_
-  # cd out of the directory we're about to remove
-  cd "$initial_cwd_" || cd / || cd /tmp
-  chmod -R u+rwx "$test_dir_"
-  # If removal fails and exit status was to be 0, then change it to 1.
-  rm -rf "$test_dir_" || { test $__st = 0 && __st=1; }
-  exit $__st
-}
-
-# Given a directory name, DIR, if every entry in it that matches *.exe
-# contains only the specified bytes (see the case stmt below), then print
-# a space-separated list of those names and return 0.  Otherwise, don't
-# print anything and return 1.  Naming constraints apply also to DIR.
-find_exe_basenames_ ()
-{
-  feb_dir_=$1
-  feb_fail_=0
-  feb_result_=
-  feb_sp_=
-  for feb_file_ in $feb_dir_/*.exe; do
-    # If there was no *.exe file, or there existed a file named "*.exe" that
-    # was deleted between the above glob expansion and the existence test
-    # below, just skip it.
-    test "x$feb_file_" = "x$feb_dir_/*.exe" && test ! -f "$feb_file_" \
-      && continue
-    # Exempt [.exe, since we can't create a function by that name, yet
-    # we can't invoke [ by PATH search anyways due to shell builtins.
-    test "x$feb_file_" = "x$feb_dir_/[.exe" && continue
-    case $feb_file_ in
-      *[!-a-zA-Z/0-9_.+]*) feb_fail_=1; break;;
-      *) # Remove leading file name components as well as the .exe suffix.
-         feb_file_=${feb_file_##*/}
-         feb_file_=${feb_file_%.exe}
-         feb_result_="$feb_result_$feb_sp_$feb_file_";;
-    esac
-    feb_sp_=' '
-  done
-  test $feb_fail_ = 0 && printf %s "$feb_result_"
-  return $feb_fail_
-}
-
-# Consider the files in directory, $1.
-# For each file name of the form PROG.exe, create an alias named
-# PROG that simply invokes PROG.exe, then return 0.  If any selected
-# file name or the directory name, $1, contains an unexpected character,
-# define no alias and return 1.
-create_exe_shims_ ()
-{
-  case $EXEEXT in
-    '') return 0 ;;
-    .exe) ;;
-    *) echo "$0: unexpected \$EXEEXT value: $EXEEXT" 1>&2; return 1 ;;
-  esac
-
-  base_names_=`find_exe_basenames_ $1` \
-    || { echo "$0 (exe_shim): skipping directory: $1" 1>&2; return 0; }
-
-  if test -n "$base_names_"; then
-    for base_ in $base_names_; do
-      alias "$base_"="$base_$EXEEXT"
-    done
-  fi
-
-  return 0
-}
-
-# Use this function to prepend to PATH an absolute name for each
-# specified, possibly-$initial_cwd_-relative, directory.
-path_prepend_ ()
-{
-  while test $# != 0; do
-    path_dir_=$1
-    case $path_dir_ in
-      '') fail_ "invalid path dir: '$1'";;
-      /*) abs_path_dir_=$path_dir_;;
-      *) abs_path_dir_=$initial_cwd_/$path_dir_;;
-    esac
-    case $abs_path_dir_ in
-      *:*) fail_ "invalid path dir: '$abs_path_dir_'";;
-    esac
-    PATH="$abs_path_dir_:$PATH"
-
-    # Create an alias, FOO, for each FOO.exe in this directory.
-    create_exe_shims_ "$abs_path_dir_" \
-      || fail_ "something failed (above): $abs_path_dir_"
-    shift
-  done
-  export PATH
-}
-
-setup_ ()
-{
-  if test "$VERBOSE" = yes; then
-    # Test whether set -x may cause the selected shell to corrupt an
-    # application's stderr.  Many do, including zsh-4.3.10 and the /bin/sh
-    # from SunOS 5.11, OpenBSD 4.7 and Irix 5.x and 6.5.
-    # If enabling verbose output this way would cause trouble, simply
-    # issue a warning and refrain.
-    if $gl_set_x_corrupts_stderr_; then
-      warn_ "using SHELL=$SHELL with 'set -x' corrupts stderr"
-    else
-      set -x
-    fi
-  fi
-
-  initial_cwd_=$PWD
-
-  pfx_=`testdir_prefix_`
-  test_dir_=`mktempd_ "$initial_cwd_" "$pfx_-$ME_.XXXX"` \
-    || fail_ "failed to create temporary directory in $initial_cwd_"
-  cd "$test_dir_" || fail_ "failed to cd to temporary directory"
-
-  # As autoconf-generated configure scripts do, ensure that IFS
-  # is defined initially, so that saving and restoring $IFS works.
-  gl_init_sh_nl_='
-'
-  IFS=" ""	$gl_init_sh_nl_"
-
-  # This trap statement, along with a trap on 0 below, ensure that the
-  # temporary directory, $test_dir_, is removed upon exit as well as
-  # upon receipt of any of the listed signals.
-  for sig_ in 1 2 3 13 15; do
-    eval "trap 'Exit $(expr $sig_ + 128)' $sig_"
-  done
-}
-
-# Create a temporary directory, much like mktemp -d does.
-# Written by Jim Meyering.
-#
-# Usage: mktempd_ /tmp phoey.XXXXXXXXXX
-#
-# First, try to use the mktemp program.
-# Failing that, we'll roll our own mktemp-like function:
-#  - try to get random bytes from /dev/urandom
-#  - failing that, generate output from a combination of quickly-varying
-#      sources and gzip.  Ignore non-varying gzip header, and extract
-#      "random" bits from there.
-#  - given those bits, map to file-name bytes using tr, and try to create
-#      the desired directory.
-#  - make only $MAX_TRIES_ attempts
-
-# Helper function.  Print $N pseudo-random bytes from a-zA-Z0-9.
-rand_bytes_ ()
-{
-  n_=$1
-
-  # Maybe try openssl rand -base64 $n_prime_|tr '+/=\012' abcd first?
-  # But if they have openssl, they probably have mktemp, too.
-
-  chars_=abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789
-  dev_rand_=/dev/urandom
-  if test -r "$dev_rand_"; then
-    # Note: 256-length($chars_) == 194; 3 copies of $chars_ is 186 + 8 = 194.
-    dd ibs=$n_ count=1 if=$dev_rand_ 2>/dev/null \
-      | LC_ALL=C tr -c $chars_ 01234567$chars_$chars_$chars_
-    return
-  fi
-
-  n_plus_50_=`expr $n_ + 50`
-  cmds_='date; date +%N; free; who -a; w; ps auxww; ps ef; netstat -n'
-  data_=` (eval "$cmds_") 2>&1 | gzip `
-
-  # Ensure that $data_ has length at least 50+$n_
-  while :; do
-    len_=`echo "$data_"|wc -c`
-    test $n_plus_50_ -le $len_ && break;
-    data_=` (echo "$data_"; eval "$cmds_") 2>&1 | gzip `
-  done
-
-  echo "$data_" \
-    | dd bs=1 skip=50 count=$n_ 2>/dev/null \
-    | LC_ALL=C tr -c $chars_ 01234567$chars_$chars_$chars_
-}
-
-mktempd_ ()
-{
-  case $# in
-  2);;
-  *) fail_ "Usage: mktempd_ DIR TEMPLATE";;
-  esac
-
-  destdir_=$1
-  template_=$2
-
-  MAX_TRIES_=4
-
-  # Disallow any trailing slash on specified destdir:
-  # it would subvert the post-mktemp "case"-based destdir test.
-  case $destdir_ in
-  / | //) destdir_slash_=$destdir;;
-  */) fail_ "invalid destination dir: remove trailing slash(es)";;
-  *) destdir_slash_=$destdir_/;;
-  esac
-
-  case $template_ in
-  *XXXX) ;;
-  *) fail_ \
-       "invalid template: $template_ (must have a suffix of at least 4 X's)";;
-  esac
-
-  # First, try to use mktemp.
-  d=`unset TMPDIR; { mktemp -d -t -p "$destdir_" "$template_"; } 2>/dev/null` &&
-
-  # The resulting name must be in the specified directory.
-  case $d in "$destdir_slash_"*) :;; *) false;; esac &&
-
-  # It must have created the directory.
-  test -d "$d" &&
-
-  # It must have 0700 permissions.  Handle sticky "S" bits.
-  perms=`ls -dgo "$d" 2>/dev/null` &&
-  case $perms in drwx--[-S]---*) :;; *) false;; esac && {
-    echo "$d"
-    return
-  }
-
-  # If we reach this point, we'll have to create a directory manually.
-
-  # Get a copy of the template without its suffix of X's.
-  base_template_=`echo "$template_"|sed 's/XX*$//'`
-
-  # Calculate how many X's we've just removed.
-  template_length_=`echo "$template_" | wc -c`
-  nx_=`echo "$base_template_" | wc -c`
-  nx_=`expr $template_length_ - $nx_`
-
-  err_=
-  i_=1
-  while :; do
-    X_=`rand_bytes_ $nx_`
-    candidate_dir_="$destdir_slash_$base_template_$X_"
-    err_=`mkdir -m 0700 "$candidate_dir_" 2>&1` \
-      && { echo "$candidate_dir_"; return; }
-    test $MAX_TRIES_ -le $i_ && break;
-    i_=`expr $i_ + 1`
-  done
-  fail_ "$err_"
-}
-
-# If you want to override the testdir_prefix_ function,
-# or to add more utility functions, use this file.
-test -f "$srcdir/init.cfg" \
-  && . "$srcdir/init.cfg"
-
-setup_ "$@"
-# This trap is here, rather than in the setup_ function, because some
-# shells run the exit trap at shell function exit, rather than script exit.
-trap remove_tmp_ 0
diff --git a/tests/gzip/keep.sh b/tests/gzip/keep.sh
deleted file mode 100755
index ab9a21811d31..000000000000
--- a/tests/gzip/keep.sh
+++ /dev/null
@@ -1,51 +0,0 @@
-#!/bin/sh
-# Exercise the --keep option.
-
-# Copyright (C) 2013-2016 Free Software Foundation, Inc.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-# limit so don't run it by default.
-
-. "${srcdir=.}/init.sh"; path_prepend_ .
-
-echo fooooooooo > in || framework_failure_
-cp in orig || framework_failure_
-
-fail=0
-
-# Compress and decompress both with and without --keep.
-for k in --keep ''; do
-  # With --keep, the source must be retained, otherwise, it must be removed.
-  case $k in --keep) op='||' ;; *) op='&&' ;; esac
-
-  gzip $k in || fail=1
-  eval "test -f in $op fail=1"
-  test -f in.gz || fail=1
-  rm -f in || fail=1
-
-  gzip -d $k in.gz || fail=1
-  eval "test -f in.gz $op fail=1"
-  test -f in || fail=1
-  compare in orig || fail=1
-  rm -f in.gz || fail=1
-done
-
-cp orig in || framework_failure_
-log=$(gzip -kv in 2>&1) || fail=1
-case $log in
-  *'created in.gz'*) ;;
-  *) fail=1;;
-esac
-
-Exit $fail
diff --git a/tests/gzip/list.sh b/tests/gzip/list.sh
deleted file mode 100755
index 75912e1e26d7..000000000000
--- a/tests/gzip/list.sh
+++ /dev/null
@@ -1,31 +0,0 @@
-#!/bin/sh
-# Exercise the --list option.
-
-# Copyright 2016 Free Software Foundation, Inc.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-# limit so don't run it by default.
-
-. "${srcdir=.}/init.sh"; path_prepend_ .
-
-echo zoology zucchini > in || framework_failure_
-cp in orig || framework_failure_
-
-gzip -l in && fail=1
-gzip -9 in || fail=1
-gzip -l in.gz >out1 || fail=1
-gzip -l in.gz | cat >out2 || fail=1
-compare out1 out2 || fail=1
-
-Exit $fail
diff --git a/tests/gzip/memcpy-abuse.sh b/tests/gzip/memcpy-abuse.sh
deleted file mode 100755
index 7d5c056debc6..000000000000
--- a/tests/gzip/memcpy-abuse.sh
+++ /dev/null
@@ -1,34 +0,0 @@
-#!/bin/sh
-# Before gzip-1.4, this the use of memcpy in inflate_codes could
-# mistakenly operate on overlapping regions.  Exercise that code.
-
-# Copyright (C) 2010-2016 Free Software Foundation, Inc.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-# limit so don't run it by default.
-
-. "${srcdir=.}/init.sh"; path_prepend_ .
-
-# The input must be larger than 32KiB and slightly
-# less uniform than e.g., all zeros.
-printf wxy%032767d 0 | tee in | gzip > in.gz || framework_failure_
-
-fail=0
-
-# Before the fix, this would call memcpy with overlapping regions.
-gzip -dc in.gz > out || fail=1
-
-compare in out || fail=1
-
-Exit $fail
diff --git a/tests/gzip/mixed.sh b/tests/gzip/mixed.sh
deleted file mode 100755
index 383a54f5e46e..000000000000
--- a/tests/gzip/mixed.sh
+++ /dev/null
@@ -1,68 +0,0 @@
-#!/bin/sh
-# Ensure that gzip -cdf handles mixed compressed/not-compressed data
-# Before gzip-1.5, it would produce invalid output.
-
-# Copyright (C) 2010-2016 Free Software Foundation, Inc.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-# limit so don't run it by default.
-
-. "${srcdir=.}/init.sh"; path_prepend_ .
-
-printf 'xxx\nyyy\n'      > exp2 || framework_failure_
-printf 'aaa\nbbb\nccc\n' > exp3 || framework_failure_
-
-fail=0
-
-(echo xxx; echo yyy) > in || fail=1
-gzip -cdf < in > out || fail=1
-compare exp2 out || fail=1
-
-# Uncompressed input, followed by compressed data.
-# Currently fails, so skip it.
-# (echo xxx; echo yyy|gzip) > in || fail=1
-# gzip -cdf < in > out || fail=1
-# compare exp2 out || fail=1
-
-# Compressed input, followed by regular (not-compressed) data.
-(echo xxx|gzip; echo yyy) > in || fail=1
-gzip -cdf < in > out || fail=1
-compare exp2 out || fail=1
-
-(echo xxx|gzip; echo yyy|gzip) > in || fail=1
-gzip -cdf < in > out || fail=1
-compare exp2 out || fail=1
-
-in_str=0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_-+=%
-for i in 0 1 2 3 4 5 6 7 8 9 a; do in_str="$in_str$in_str" ;done
-
-# Start with some small sizes.  $(seq 64)
-sizes=$(i=0; while :; do echo $i; test $i = 64 && break; i=$(expr $i + 1); done)
-
-# gzip's internal buffer size is 32KiB + 64 bytes:
-sizes="$sizes 32831 32832 32833"
-
-# 128KiB, +/- 1
-sizes="$sizes 131071 131072 131073"
-
-# Ensure that "gzip -cdf" acts like cat, for a range of small input files.
-i=0
-for i in $sizes; do
-  echo $i
-  printf %$i.${i}s $in_str > in
-  gzip -cdf < in > out
-  compare in out || fail=1
-done
-
-Exit $fail
diff --git a/tests/gzip/null-suffix-clobber.sh b/tests/gzip/null-suffix-clobber.sh
deleted file mode 100755
index 0efd0e344908..000000000000
--- a/tests/gzip/null-suffix-clobber.sh
+++ /dev/null
@@ -1,35 +0,0 @@
-#!/bin/sh
-# Before gzip-1.5, gzip -d -S '' k.gz would delete F.gz and not create "F"
-
-# Copyright (C) 2010-2016 Free Software Foundation, Inc.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-# limit so don't run it by default.
-
-. "${srcdir=.}/init.sh"; path_prepend_ .
-
-printf anything | gzip > F.gz || framework_failure_
-echo y > yes || framework_failure_
-echo "gzip: invalid suffix ''" > expected-err || framework_failure_
-
-fail=0
-
-gzip ---presume-input-tty -d -S '' F.gz < yes > out 2>err && fail=1
-
-compare /dev/null out || fail=1
-compare expected-err err || fail=1
-
-test -f F.gz || fail=1
-
-Exit $fail
diff --git a/tests/gzip/stdin.sh b/tests/gzip/stdin.sh
deleted file mode 100755
index eef4cd8b1078..000000000000
--- a/tests/gzip/stdin.sh
+++ /dev/null
@@ -1,31 +0,0 @@
-#!/bin/sh
-# Ensure that gzip interprets "-" as stdin.
-
-# Copyright (C) 2009-2016 Free Software Foundation, Inc.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-# limit so don't run it by default.
-
-. "${srcdir=.}/init.sh"; path_prepend_ .
-
-printf a | gzip > in || framework_failure_
-printf aaa > exp || framework_failure_
-
-fail=0
-gzip -dc in - in < in > out 2>err || fail=1
-
-compare exp out || fail=1
-compare /dev/null err || fail=1
-
-Exit $fail
diff --git a/tests/gzip/test-driver.sh b/tests/gzip/test-driver.sh
deleted file mode 100755
index 649c084e4b0f..000000000000
--- a/tests/gzip/test-driver.sh
+++ /dev/null
@@ -1,150 +0,0 @@
-#! /bin/sh
-# test-driver - basic testsuite driver script.
-
-scriptversion=2016-01-11.22; # UTC
-
-# Copyright (C) 2011-2015 Free Software Foundation, Inc.
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2, or (at your option)
-# any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-# As a special exception to the GNU General Public License, if you
-# distribute this file as part of a program that contains a
-# configuration script generated by Autoconf, you may include it under
-# the same distribution terms that you use for the rest of that program.
-
-# This file is maintained in Automake, please report
-# bugs to <bug-automake@gnu.org> or send patches to
-# <automake-patches@gnu.org>.
-
-# Make unconditional expansion of undefined variables an error.  This
-# helps a lot in preventing typo-related bugs.
-set -u
-
-usage_error ()
-{
-  echo "$0: $*" >&2
-  print_usage >&2
-  exit 2
-}
-
-print_usage ()
-{
-  cat <<END
-Usage:
-  test-driver --test-name=NAME --log-file=PATH --trs-file=PATH
-              [--expect-failure={yes|no}] [--color-tests={yes|no}]
-              [--enable-hard-errors={yes|no}] [--]
-              TEST-SCRIPT [TEST-SCRIPT-ARGUMENTS]
-The '--test-name', '--log-file' and '--trs-file' options are mandatory.
-END
-}
-
-test_name= # Used for reporting.
-log_file=  # Where to save the output of the test script.
-trs_file=  # Where to save the metadata of the test run.
-expect_failure=no
-color_tests=no
-enable_hard_errors=yes
-while test $# -gt 0; do
-  case $1 in
-  --help) print_usage; exit $?;;
-  --version) echo "test-driver $scriptversion"; exit $?;;
-  --test-name) test_name=$2; shift;;
-  --log-file) log_file=$2; shift;;
-  --trs-file) trs_file=$2; shift;;
-  --color-tests) color_tests=$2; shift;;
-  --expect-failure) expect_failure=$2; shift;;
-  --enable-hard-errors) enable_hard_errors=$2; shift;;
-  --) shift; break;;
-  -*) usage_error "invalid option: '$1'";;
-   *) break;;
-  esac
-  shift
-done
-
-missing_opts=
-test x"$test_name" = x && missing_opts="$missing_opts --test-name"
-test x"$log_file"  = x && missing_opts="$missing_opts --log-file"
-test x"$trs_file"  = x && missing_opts="$missing_opts --trs-file"
-if test x"$missing_opts" != x; then
-  usage_error "the following mandatory options are missing:$missing_opts"
-fi
-
-if test $# -eq 0; then
-  usage_error "missing argument"
-fi
-
-if test $color_tests = yes; then
-  # Keep this in sync with 'lib/am/check.am:$(am__tty_colors)'.
-  red='' # Red.
-  grn='' # Green.
-  lgn='' # Light green.
-  blu='' # Blue.
-  mgn='' # Magenta.
-  std=''     # No color.
-else
-  red= grn= lgn= blu= mgn= std=
-fi
-
-do_exit='rm -f $log_file $trs_file; (exit $st); exit $st'
-trap "st=129; $do_exit" 1
-trap "st=130; $do_exit" 2
-trap "st=141; $do_exit" 13
-trap "st=143; $do_exit" 15
-
-# Test script is run here.
-"$@" >$log_file 2>&1
-estatus=$?
-
-if test $enable_hard_errors = no && test $estatus -eq 99; then
-  tweaked_estatus=1
-else
-  tweaked_estatus=$estatus
-fi
-
-case $tweaked_estatus:$expect_failure in
-  0:yes) col=$red res=XPASS recheck=yes gcopy=yes;;
-  0:*)   col=$grn res=PASS  recheck=no  gcopy=no;;
-  77:*)  col=$blu res=SKIP  recheck=no  gcopy=yes;;
-  99:*)  col=$mgn res=ERROR recheck=yes gcopy=yes;;
-  *:yes) col=$lgn res=XFAIL recheck=no  gcopy=yes;;
-  *:*)   col=$red res=FAIL  recheck=yes gcopy=yes;;
-esac
-
-# Report the test outcome and exit status in the logs, so that one can
-# know whether the test passed or failed simply by looking at the '.log'
-# file, without the need of also peaking into the corresponding '.trs'
-# file (automake bug#11814).
-echo "$res $test_name (exit status: $estatus)" >>$log_file
-
-# Report outcome to console.
-echo "${col}${res}${std}: $test_name"
-
-# Register the test result, and other relevant metadata.
-echo ":test-result: $res" > $trs_file
-echo ":global-test-result: $res" >> $trs_file
-echo ":recheck: $recheck" >> $trs_file
-echo ":copy-in-global-log: $gcopy" >> $trs_file
-
-# Local Variables:
-# mode: shell-script
-# sh-indentation: 2
-# eval: (add-hook 'write-file-hooks 'time-stamp)
-# time-stamp-start: "scriptversion="
-# time-stamp-format: "%:y-%02m-%02d.%02H"
-# time-stamp-time-zone: "UTC0"
-# time-stamp-end: "; # UTC"
-# End:
-
-exit $tweaked_estatus
diff --git a/tests/gzip/trailing-nul.sh b/tests/gzip/trailing-nul.sh
deleted file mode 100755
index 7b15d5e55788..000000000000
--- a/tests/gzip/trailing-nul.sh
+++ /dev/null
@@ -1,37 +0,0 @@
-#!/bin/sh
-# gzip accepts trailing NUL bytes; don't fail if there is exactly one.
-# Before gzip-1.4, this would fail.
-
-# Copyright (C) 2009-2016 Free Software Foundation, Inc.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-# limit so don't run it by default.
-
-. "${srcdir=.}/init.sh"; path_prepend_ .
-
-(echo 0 | gzip; printf '\0') > 0.gz || framework_failure_
-(echo 00 | gzip; printf '\0\0') > 00.gz || framework_failure_
-(echo 1 | gzip; printf '\1') > 1.gz || framework_failure_
-
-fail=0
-
-for i in 0 00 1; do
-  gzip -d $i.gz; ret=$?
-  test $ret -eq $i || fail=1
-  test $ret = 1 && continue
-  echo $i > exp || fail=1
-  compare exp $i || fail=1
-done
-
-Exit $fail
diff --git a/tests/gzip/unpack-invalid.sh b/tests/gzip/unpack-invalid.sh
deleted file mode 100755
index fe8384d73cde..000000000000
--- a/tests/gzip/unpack-invalid.sh
+++ /dev/null
@@ -1,36 +0,0 @@
-#!/bin/sh
-# gzip should report invalid 'unpack' input when uncompressing.
-# With gzip-1.5, it would output invalid data instead.
-
-# Copyright (C) 2012-2016 Free Software Foundation, Inc.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-# limit so don't run it by default.
-
-. "${srcdir=.}/init.sh"; path_prepend_ .
-
-for input in \
-  '\037\036\000\000\037\213\010\000\000\000\000\000\002\003\036\000\000\000\002\003\037\213\010\000\000\000\000\000\002\003\355\301\001\015\000\000\000\302\240\037\000\302\240\037\213\010\000\000\000\000\000\002\003\355\301' \
-  '\037\213\010\000\000\000\000\000\002\003\355\301\001\015\000\000\000\302\240\076\366\017\370\036\016\030\000\000\000\000\000\000\000\000\000\034\010\105\140\104\025\020\047\000\000\037\036\016\030\000\000\000'; do
-
-  printf "$input" >in || framework_failure_
-
-  if gzip -d <in >out 2>err; then
-    fail=1
-  else
-    fail=0
-  fi
-done
-
-Exit $fail
diff --git a/tests/gzip/z-suffix.sh b/tests/gzip/z-suffix.sh
deleted file mode 100755
index a870a5408de0..000000000000
--- a/tests/gzip/z-suffix.sh
+++ /dev/null
@@ -1,30 +0,0 @@
-#!/bin/sh
-# Check that -Sz works.
-
-# Copyright 2014-2016 Free Software Foundation, Inc.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-# limit so don't run it by default.
-
-. "${srcdir=.}/init.sh"; path_prepend_ .
-
-printf anything > F && cp F G || framework_failure_
-gzip -Sz F || fail=1
-test ! -f F || fail=1
-test -f Fz || fail=1
-gzip -dSz F || fail=1
-test ! -f Fz || fail=1
-compare F G || fail\1
-
-Exit $fail
diff --git a/tests/gzip/zdiff.sh b/tests/gzip/zdiff.sh
deleted file mode 100755
index d62a84606ba4..000000000000
--- a/tests/gzip/zdiff.sh
+++ /dev/null
@@ -1,48 +0,0 @@
-#!/bin/sh
-# Exercise zdiff with two compressed inputs.
-# Before gzip-1.4, this would fail.
-
-# Copyright (C) 2009-2016 Free Software Foundation, Inc.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-# limit so don't run it by default.
-
-. "${srcdir=.}/init.sh"; path_prepend_ .
-
-echo a > a || framework_failure_
-echo b > b || framework_failure_
-gzip a b || framework_failure_
-
-cat <<EOF > exp
-1c1
-< a
----
-> b
-EOF
-
-fail=0
-zdiff a.gz b.gz > out 2>&1
-test $? = 1 || fail=1
-
-compare exp out || fail=1
-
-rm -f out
-# expect success, for equal files
-zdiff a.gz a.gz > out 2> err || fail=1
-# expect no output
-test -s out && fail=1
-# expect no stderr
-test -s err && fail=1
-
-Exit $fail
diff --git a/tests/gzip/zgrep-context.sh b/tests/gzip/zgrep-context.sh
deleted file mode 100755
index c8648b7e4f55..000000000000
--- a/tests/gzip/zgrep-context.sh
+++ /dev/null
@@ -1,47 +0,0 @@
-#!/bin/sh
-# Ensure that zgrep -15 works.  Before gzip-1.5, it would fail.
-
-# Copyright (C) 2012-2016 Free Software Foundation, Inc.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-# limit so don't run it by default.
-
-. "${srcdir=.}/init.sh"; path_prepend_ .
-
-# A limited replacement for seq: handle 1 or 2 args; increment must be 1
-seq()
-{
-  case $# in
-    1) start=1  final=$1;;
-    2) start=$1 final=$2;;
-    *) echo you lose 1>&2; exit 1;;
-  esac
-  awk 'BEGIN{for(i='$start';i<='$final';i++) print i}' < /dev/null
-}
-
-seq 40 > in || framework_failure_
-gzip < in > in.gz || framework_failure_
-seq 2 32 > exp || framework_failure_
-
-: ${GREP=grep}
-$GREP -15 17 - < in > out && compare exp out || {
-  echo >&2 "$0: $GREP does not support context options; skipping this test"
-  exit 77
-}
-
-fail=0
-zgrep -15 17 - < in.gz > out || fail=1
-compare exp out || fail=1
-
-Exit $fail
diff --git a/tests/gzip/zgrep-f.sh b/tests/gzip/zgrep-f.sh
deleted file mode 100755
index d0cf27f7e231..000000000000
--- a/tests/gzip/zgrep-f.sh
+++ /dev/null
@@ -1,43 +0,0 @@
-#!/bin/sh
-# Ensure that zgrep -f - works like grep -f -
-# Before gzip-1.4, it would fail.
-
-# Copyright (C) 2009-2016 Free Software Foundation, Inc.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-# limit so don't run it by default.
-
-. "${srcdir=.}/init.sh"; path_prepend_ .
-
-printf 'needle\nn2\n' > n || framework_failure_
-cp n haystack || framework_failure_
-gzip haystack || framework_failure_
-
-fail=0
-zgrep -f - haystack.gz < n > out 2>&1 || fail=1
-
-compare out n || fail=1
-
-if ${BASH_VERSION+:} false; then
-  set +o posix
-  # This failed with gzip 1.6.
-  cat n n >nn || framework_failure_
-  eval 'zgrep -h -f <(cat n) haystack.gz haystack.gz' >out || fail=1
-  compare out nn || fail=1
-fi
-
-# This failed with gzip 1.4.
-echo a-b | zgrep -e - > /dev/null || fail=1
-
-Exit $fail
diff --git a/tests/gzip/zgrep-signal.sh b/tests/gzip/zgrep-signal.sh
deleted file mode 100755
index a8c53881adde..000000000000
--- a/tests/gzip/zgrep-signal.sh
+++ /dev/null
@@ -1,64 +0,0 @@
-#!/bin/sh
-# Check that zgrep is terminated gracefully by signal when
-# its grep/sed pipeline is terminated by a signal.
-
-# Copyright (C) 2010-2016 Free Software Foundation, Inc.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-# limit so don't run it by default.
-
-. "${srcdir=.}/init.sh"; path_prepend_ .
-
-echo a | gzip -c > f.gz || framework_failure_
-
-test "x$PERL" = x && PERL=perl
-("$PERL" -e 'use POSIX qw(dup2)') >/dev/null 2>&1 ||
-   skip_ "no suitable perl found"
-
-# Run the arguments as a command, in a process where stdout is a
-# dangling pipe and SIGPIPE has the default signal-handling action.
-# This can't be done portably in the shell, because if SIGPIPE is
-# ignored when the shell is entered, the shell might refuse to trap
-# it.  Fall back on Perl+POSIX, if available.  Take care to close the
-# pipe's read end before running the program; the equivalent of the
-# shell's "command | :" has a race condition in that COMMAND could
-# write before ":" exits.
-write_to_dangling_pipe () {
-  program=${1?}
-  shift
-  args=
-  for arg; do
-    args="$args, '$arg'"
-  done
-  "$PERL" -e '
-     use POSIX qw(dup2);
-     $SIG{PIPE} = "DEFAULT";
-     pipe my ($read_end, $write_end) or die "pipe: $!\n";
-     dup2 fileno $write_end, 1 or die "dup2: $!\n";
-     close $read_end or die "close: $!\n";
-     exec '"'$program'$args"';
-  '
-}
-
-write_to_dangling_pipe cat f.gz f.gz
-signal_status=$?
-test 128 -lt $signal_status ||
-  framework_failure_ 'signal handling busted on this host'
-
-fail=0
-
-write_to_dangling_pipe zgrep a f.gz f.gz
-test $? -eq $signal_status || fail=1
-
-Exit $fail
diff --git a/tests/gzip/znew-k.sh b/tests/gzip/znew-k.sh
deleted file mode 100755
index 6c239e28ea8f..000000000000
--- a/tests/gzip/znew-k.sh
+++ /dev/null
@@ -1,40 +0,0 @@
-#!/bin/sh
-# Check that znew -K works without compress(1).
-
-# Copyright (C) 2010-2016 Free Software Foundation, Inc.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-# limit so don't run it by default.
-
-. "${srcdir=.}/init.sh"; path_prepend_ .
-
-cat <<'EOF' >compress || framework_failure_
-#!/bin/sh
-echo >&2 'compress has been invoked'
-exit 1
-EOF
-chmod +x compress || framework_failure_
-
-# Note that the basename must have a length of 6 or greater.
-# Otherwise, "test -f $name" below would fail.
-name=123456.Z
-
-printf '%1012977s' ' ' | gzip -c > $name || framework_failure_
-
-fail=0
-
-znew -K $name || fail=1
-test -f $name || fail=1
-
-Exit $fail
diff --git a/tests/invalidDictionaries.c b/tests/invalidDictionaries.c
deleted file mode 100644
index b23db036f489..000000000000
--- a/tests/invalidDictionaries.c
+++ /dev/null
@@ -1,61 +0,0 @@
-/*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#include <stddef.h>
-#include "zstd.h"
-
-static const char invalidRepCode[] = {
-  0x37, 0xa4, 0x30, 0xec, 0x2a, 0x00, 0x00, 0x00, 0x39, 0x10, 0xc0, 0xc2,
-  0xa6, 0x00, 0x0c, 0x30, 0xc0, 0x00, 0x03, 0x0c, 0x30, 0x20, 0x72, 0xf8,
-  0xb4, 0x6d, 0x4b, 0x9f, 0xfc, 0x97, 0x29, 0x49, 0xb2, 0xdf, 0x4b, 0x29,
-  0x7d, 0x4a, 0xfc, 0x83, 0x18, 0x22, 0x75, 0x23, 0x24, 0x44, 0x4d, 0x02,
-  0xb7, 0x97, 0x96, 0xf6, 0xcb, 0xd1, 0xcf, 0xe8, 0x22, 0xea, 0x27, 0x36,
-  0xb7, 0x2c, 0x40, 0x46, 0x01, 0x08, 0x23, 0x01, 0x00, 0x00, 0x06, 0x1e,
-  0x3c, 0x83, 0x81, 0xd6, 0x18, 0xd4, 0x12, 0x3a, 0x04, 0x00, 0x80, 0x03,
-  0x08, 0x0e, 0x12, 0x1c, 0x12, 0x11, 0x0d, 0x0e, 0x0a, 0x0b, 0x0a, 0x09,
-  0x10, 0x0c, 0x09, 0x05, 0x04, 0x03, 0x06, 0x06, 0x06, 0x02, 0x00, 0x03,
-  0x00, 0x00, 0x02, 0x02, 0x00, 0x04, 0x06, 0x03, 0x06, 0x08, 0x24, 0x6b,
-  0x0d, 0x01, 0x10, 0x04, 0x81, 0x07, 0x00, 0x00, 0x04, 0xb9, 0x58, 0x18,
-  0x06, 0x59, 0x92, 0x43, 0xce, 0x28, 0xa5, 0x08, 0x88, 0xc0, 0x80, 0x88,
-  0x8c, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00,
-  0x08, 0x00, 0x00, 0x00
-};
-
-typedef struct dictionary_s {
-  const char *data;
-  size_t size;
-} dictionary;
-
-static const dictionary dictionaries[] = {
-  {invalidRepCode, sizeof(invalidRepCode)},
-  {NULL, 0},
-};
-
-int main(int argc, const char** argv) {
-  const dictionary *dict;
-  for (dict = dictionaries; dict->data != NULL; ++dict) {
-    ZSTD_CDict *cdict;
-    ZSTD_DDict *ddict;
-    cdict = ZSTD_createCDict(dict->data, dict->size, 1);
-    if (cdict) {
-      ZSTD_freeCDict(cdict);
-      return 1;
-    }
-    ddict = ZSTD_createDDict(dict->data, dict->size);
-    if (ddict) {
-      ZSTD_freeDDict(ddict);
-      return 2;
-    }
-  }
-
-  (void)argc;
-  (void)argv;
-  return 0;
-}
diff --git a/tests/legacy.c b/tests/legacy.c
deleted file mode 100644
index eb329203833f..000000000000
--- a/tests/legacy.c
+++ /dev/null
@@ -1,260 +0,0 @@
-/*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-/*
-    This program uses hard-coded data compressed with Zstd legacy versions
-    and tests that the API decompresses them correctly
-*/
-
-/*===========================================
-*   Dependencies
-*==========================================*/
-#include <stddef.h>              /* size_t */
-#include <stdlib.h>              /* malloc, free */
-#include <stdio.h>               /* fprintf */
-#include <string.h>              /* strlen */
-#define ZSTD_STATIC_LINKING_ONLY /* ZSTD_decompressBound */
-#include "zstd.h"
-#include "zstd_errors.h"
-
-/*===========================================
-*   Macros
-*==========================================*/
-#define DISPLAY(...)          fprintf(stderr, __VA_ARGS__)
-
-/*===========================================
-*   Precompressed frames
-*==========================================*/
-const char* const COMPRESSED; /* content is at end of file */
-size_t const COMPRESSED_SIZE = 917;
-const char* const EXPECTED; /* content is at end of file */
-
-
-static int testSimpleAPI(void)
-{
-    size_t const size = strlen(EXPECTED);
-    char* const output = malloc(size);
-
-    if (!output) {
-        DISPLAY("ERROR: Not enough memory!\n");
-        return 1;
-    }
-
-    {
-        size_t const ret = ZSTD_decompress(output, size, COMPRESSED, COMPRESSED_SIZE);
-        if (ZSTD_isError(ret)) {
-            if (ret == ZSTD_error_prefix_unknown) {
-                DISPLAY("ERROR: Invalid frame magic number, was this compiled "
-                        "without legacy support?\n");
-            } else {
-                DISPLAY("ERROR: %s\n", ZSTD_getErrorName(ret));
-            }
-            return 1;
-        }
-        if (ret != size) {
-            DISPLAY("ERROR: Wrong decoded size\n");
-        }
-    }
-    if (memcmp(EXPECTED, output, size) != 0) {
-        DISPLAY("ERROR: Wrong decoded output produced\n");
-        return 1;
-    }
-
-    free(output);
-    DISPLAY("Simple API OK\n");
-    return 0;
-}
-
-
-static int testStreamingAPI(void)
-{
-    int error_code = 0;
-    size_t const outBuffSize = ZSTD_DStreamOutSize();
-    char* const outBuff = malloc(outBuffSize);
-    ZSTD_DStream* const stream = ZSTD_createDStream();
-    ZSTD_inBuffer input = { COMPRESSED, COMPRESSED_SIZE, 0 };
-    size_t outputPos = 0;
-    int needsInit = 1;
-
-    if (outBuff == NULL) {
-        DISPLAY("ERROR: Could not allocate memory\n");
-        return 1;
-    }
-    if (stream == NULL) {
-        DISPLAY("ERROR: Could not create dstream\n");
-        free(outBuff);
-        return 1;
-    }
-
-    while (1) {
-        ZSTD_outBuffer output = {outBuff, outBuffSize, 0};
-        if (needsInit) {
-            size_t const ret = ZSTD_initDStream(stream);
-            if (ZSTD_isError(ret)) {
-                DISPLAY("ERROR: ZSTD_initDStream: %s\n", ZSTD_getErrorName(ret));
-                error_code = 1;
-                break;
-        }   }
-
-        {   size_t const ret = ZSTD_decompressStream(stream, &output, &input);
-            if (ZSTD_isError(ret)) {
-                DISPLAY("ERROR: ZSTD_decompressStream: %s\n", ZSTD_getErrorName(ret));
-                error_code = 1;
-                break;
-            }
-
-            if (ret == 0) {
-                needsInit = 1;
-        }   }
-
-        if (memcmp(outBuff, EXPECTED + outputPos, output.pos) != 0) {
-            DISPLAY("ERROR: Wrong decoded output produced\n");
-            error_code = 1;
-            break;
-        }
-        outputPos += output.pos;
-        if (input.pos == input.size && output.pos < output.size) {
-            break;
-        }
-    }
-
-    free(outBuff);
-    ZSTD_freeDStream(stream);
-    if (error_code == 0) DISPLAY("Streaming API OK\n");
-    return error_code;
-}
-
-static int testFrameDecoding(void)
-{
-    if (strlen(EXPECTED) > ZSTD_decompressBound(COMPRESSED, COMPRESSED_SIZE)) {
-        DISPLAY("ERROR: ZSTD_decompressBound: decompressed bound too small\n");
-        return 1;
-    }
-    {   const char* ip = COMPRESSED;
-        size_t remainingSize = COMPRESSED_SIZE;
-        while (1) {
-            size_t frameSize = ZSTD_findFrameCompressedSize(ip, remainingSize);
-            if (ZSTD_isError(frameSize)) {
-                DISPLAY("ERROR: ZSTD_findFrameCompressedSize: %s\n", ZSTD_getErrorName(frameSize));
-                return 1;
-            }
-            if (frameSize > remainingSize) {
-                DISPLAY("ERROR: ZSTD_findFrameCompressedSize: expected frameSize to align with src buffer");
-                return 1;
-            }
-            ip += frameSize;
-            remainingSize -= frameSize;
-            if (remainingSize == 0) break;
-        }
-    }
-    DISPLAY("Frame Decoding OK\n");
-    return 0;
-}
-
-int main(void)
-{
-    {   int const ret = testSimpleAPI();
-        if (ret) return ret; }
-    {   int const ret = testStreamingAPI();
-        if (ret) return ret; }
-    {   int const ret = testFrameDecoding();
-        if (ret) return ret; }
-
-    DISPLAY("OK\n");
-    return 0;
-}
-
-/* Consists of the "EXPECTED" string compressed with default settings on
-    - v0.4.3
-    - v0.5.0
-    - v0.6.0
-    - v0.7.0
-    - v0.8.0
-*/
-const char* const COMPRESSED =
-    "\x24\xB5\x2F\xFD\x00\x00\x00\xBB\xB0\x02\xC0\x10\x00\x1E\xB0\x01"
-    "\x02\x00\x00\x80\x00\xE8\x92\x34\x12\x97\xC8\xDF\xE9\xF3\xEF\x53"
-    "\xEA\x1D\x27\x4F\x0C\x44\x90\x0C\x8D\xF1\xB4\x89\x17\x00\x18\x00"
-    "\x18\x00\x3F\xE6\xE2\xE3\x74\xD6\xEC\xC9\x4A\xE0\x71\x71\x42\x3E"
-    "\x64\x4F\x6A\x45\x4E\x78\xEC\x49\x03\x3F\xC6\x80\xAB\x8F\x75\x5E"
-    "\x6F\x2E\x3E\x7E\xC6\xDC\x45\x69\x6C\xC5\xFD\xC7\x40\xB8\x84\x8A"
-    "\x01\xEB\xA8\xD1\x40\x39\x90\x4C\x64\xF8\xEB\x53\xE6\x18\x0B\x67"
-    "\x12\xAD\xB8\x99\xB3\x5A\x6F\x8A\x19\x03\x01\x50\x67\x56\xF5\x9F"
-    "\x35\x84\x60\xA0\x60\x91\xC9\x0A\xDC\xAB\xAB\xE0\xE2\x81\xFA\xCF"
-    "\xC6\xBA\x01\x0E\x00\x54\x00\x00\x19\x00\x00\x54\x14\x00\x24\x24"
-    "\x04\xFE\x04\x84\x4E\x41\x00\x27\xE2\x02\xC4\xB1\x00\xD2\x51\x00"
-    "\x79\x58\x41\x28\x00\xE0\x0C\x01\x68\x65\x00\x04\x13\x0C\xDA\x0C"
-    "\x80\x22\x06\xC0\x00\x00\x25\xB5\x2F\xFD\x00\x00\x00\xAD\x12\xB0"
-    "\x7D\x1E\xB0\x01\x02\x00\x00\x80\x00\xE8\x92\x34\x12\x97\xC8\xDF"
-    "\xE9\xF3\xEF\x53\xEA\x1D\x27\x4F\x0C\x44\x90\x0C\x8D\xF1\xB4\x89"
-    "\x03\x01\x50\x67\x56\xF5\x9F\x35\x84\x60\xA0\x60\x91\xC9\x0A\xDC"
-    "\xAB\xAB\xE0\xE2\x81\xFA\xCF\xC6\xBA\xEB\xA8\xD1\x40\x39\x90\x4C"
-    "\x64\xF8\xEB\x53\xE6\x18\x0B\x67\x12\xAD\xB8\x99\xB3\x5A\x6F\x8A"
-    "\xF9\x63\x0C\xB8\xFA\x58\xE7\xF5\xE6\xE2\xE3\x67\xCC\x5D\x94\xC6"
-    "\x56\xDC\x7F\x0C\x84\x4B\xA8\xF8\x63\x2E\x3E\x4E\x67\xCD\x9E\xAC"
-    "\x04\x1E\x17\x27\xE4\x43\xF6\xA4\x56\xE4\x84\xC7\x9E\x34\x0E\x00"
-    "\x00\x32\x40\x80\xA8\x00\x01\x49\x81\xE0\x3C\x01\x29\x1D\x00\x87"
-    "\xCE\x80\x75\x08\x80\x72\x24\x00\x7B\x52\x00\x94\x00\x20\xCC\x01"
-    "\x86\xD2\x00\x81\x09\x83\xC1\x34\xA0\x88\x01\xC0\x00\x00\x26\xB5"
-    "\x2F\xFD\x42\xEF\x00\x00\xA6\x12\xB0\x7D\x1E\xB0\x01\x02\x00\x00"
-    "\x54\xA0\xBA\x24\x8D\xC4\x25\xF2\x77\xFA\xFC\xFB\x94\x7A\xC7\xC9"
-    "\x13\x03\x11\x24\x43\x63\x3C\x6D\x22\x03\x01\x50\x67\x56\xF5\x9F"
-    "\x35\x84\x60\xA0\x60\x91\xC9\x0A\xDC\xAB\xAB\xE0\xE2\x81\xFA\xCF"
-    "\xC6\xBA\xEB\xA8\xD1\x40\x39\x90\x4C\x64\xF8\xEB\x53\xE6\x18\x0B"
-    "\x67\x12\xAD\xB8\x99\xB3\x5A\x6F\x8A\xF9\x63\x0C\xB8\xFA\x58\xE7"
-    "\xF5\xE6\xE2\xE3\x67\xCC\x5D\x94\xC6\x56\xDC\x7F\x0C\x84\x4B\xA8"
-    "\xF8\x63\x2E\x3E\x4E\x67\xCD\x9E\xAC\x04\x1E\x17\x27\xE4\x43\xF6"
-    "\xA4\x56\xE4\x84\xC7\x9E\x34\x0E\x00\x35\x0B\x71\xB5\xC0\x2A\x5C"
-    "\x26\x94\x22\x20\x8B\x4C\x8D\x13\x47\x58\x67\x15\x6C\xF1\x1C\x4B"
-    "\x54\x10\x9D\x31\x50\x85\x4B\x54\x0E\x01\x4B\x3D\x01\xC0\x00\x00"
-    "\x27\xB5\x2F\xFD\x20\xEF\x00\x00\xA6\x12\xE4\x84\x1F\xB0\x01\x10"
-    "\x00\x00\x00\x35\x59\xA6\xE7\xA1\xEF\x7C\xFC\xBD\x3F\xFF\x9F\xEF"
-    "\xEE\xEF\x61\xC3\xAA\x31\x1D\x34\x38\x22\x22\x04\x44\x21\x80\x32"
-    "\xAD\x28\xF3\xD6\x28\x0C\x0A\x0E\xD6\x5C\xAC\x19\x8D\x20\x5F\x45"
-    "\x02\x2E\x17\x50\x66\x6D\xAC\x8B\x9C\x6E\x07\x73\x46\xBB\x44\x14"
-    "\xE7\x98\xC3\xB9\x17\x32\x6E\x33\x7C\x0E\x21\xB1\xDB\xCB\x89\x51"
-    "\x23\x34\xAB\x9D\xBC\x6D\x20\xF5\x03\xA9\x91\x4C\x2E\x1F\x59\xDB"
-    "\xD9\x35\x67\x4B\x0C\x95\x79\x10\x00\x85\xA6\x96\x95\x2E\xDF\x78"
-    "\x7B\x4A\x5C\x09\x76\x97\xD1\x5C\x96\x12\x75\x35\xA3\x55\x4A\xD4"
-    "\x0B\x00\x35\x0B\x71\xB5\xC0\x2A\x5C\xE6\x08\x45\xF1\x39\x43\xF1"
-    "\x1C\x4B\x54\x10\x9D\x31\x50\x85\x4B\x54\x0E\x01\x4B\x3D\x01\xC0"
-    "\x00\x00\x28\xB5\x2F\xFD\x24\xEF\x35\x05\x00\x92\x0B\x21\x1F\xB0"
-    "\x01\x10\x00\x00\x00\x35\x59\xA6\xE7\xA1\xEF\x7C\xFC\xBD\x3F\xFF"
-    "\x9F\xEF\xEE\xEF\x61\xC3\xAA\x31\x1D\x34\x38\x22\x22\x04\x44\x21"
-    "\x80\x32\xAD\x28\xF3\xD6\x28\x0C\x0A\x0E\xD6\x5C\xAC\x19\x8D\x20"
-    "\x5F\x45\x02\x2E\x17\x50\x66\x6D\xAC\x8B\x9C\x6E\x07\x73\x46\xBB"
-    "\x44\x14\xE7\x98\xC3\xB9\x17\x32\x6E\x33\x7C\x0E\x21\xB1\xDB\xCB"
-    "\x89\x51\x23\x34\xAB\x9D\xBC\x6D\x20\xF5\x03\xA9\x91\x4C\x2E\x1F"
-    "\x59\xDB\xD9\x35\x67\x4B\x0C\x95\x79\x10\x00\x85\xA6\x96\x95\x2E"
-    "\xDF\x78\x7B\x4A\x5C\x09\x76\x97\xD1\x5C\x96\x12\x75\x35\xA3\x55"
-    "\x4A\xD4\x0B\x00\x35\x0B\x71\xB5\xC0\x2A\x5C\xE6\x08\x45\xF1\x39"
-    "\x43\xF1\x1C\x4B\x54\x10\x9D\x31\x50\x85\x4B\x54\x0E\x01\x4B\x3D"
-    "\x01\xD2\x2F\x21\x80";
-
-const char* const EXPECTED =
-    "snowden is snowed in / he's now then in his snow den / when does the snow end?\n"
-    "goodbye little dog / you dug some holes in your day / they'll be hard to fill.\n"
-    "when life shuts a door, / just open it. it’s a door. / that is how doors work.\n"
-
-    "snowden is snowed in / he's now then in his snow den / when does the snow end?\n"
-    "goodbye little dog / you dug some holes in your day / they'll be hard to fill.\n"
-    "when life shuts a door, / just open it. it’s a door. / that is how doors work.\n"
-
-    "snowden is snowed in / he's now then in his snow den / when does the snow end?\n"
-    "goodbye little dog / you dug some holes in your day / they'll be hard to fill.\n"
-    "when life shuts a door, / just open it. it’s a door. / that is how doors work.\n"
-
-    "snowden is snowed in / he's now then in his snow den / when does the snow end?\n"
-    "goodbye little dog / you dug some holes in your day / they'll be hard to fill.\n"
-    "when life shuts a door, / just open it. it’s a door. / that is how doors work.\n"
-
-    "snowden is snowed in / he's now then in his snow den / when does the snow end?\n"
-    "goodbye little dog / you dug some holes in your day / they'll be hard to fill.\n"
-    "when life shuts a door, / just open it. it’s a door. / that is how doors work.\n";
diff --git a/tests/libzstd_partial_builds.sh b/tests/libzstd_partial_builds.sh
deleted file mode 100755
index b1c1e3b1a7e3..000000000000
--- a/tests/libzstd_partial_builds.sh
+++ /dev/null
@@ -1,89 +0,0 @@
-#!/bin/sh -e
-
-DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
-
-ECHO=echo
-RM="rm -f"
-GREP="grep"
-INTOVOID="/dev/null"
-
-die() {
-    $ECHO "$@" 1>&2
-    exit 1
-}
-
-isPresent() {
-    $GREP $@ tmplog || die "$@" "should be present"
-}
-
-mustBeAbsent() {
-    $GREP $@ tmplog && die "$@ should not be there !!"
-    $ECHO "$@ correctly not present"  # for some reason, this $ECHO must exist, otherwise mustBeAbsent() always fails (??)
-}
-
-# default compilation : all features enabled
-make clean > /dev/null
-$ECHO "testing default library compilation"
-CFLAGS= make -C $DIR/../lib libzstd.a > $INTOVOID
-nm $DIR/../lib/libzstd.a | $GREP "\.o" > tmplog
-isPresent "zstd_compress.o"
-isPresent "zstd_decompress.o"
-isPresent "zdict.o"
-isPresent "zstd_v07.o"
-isPresent "zbuff_compress.o"
-$RM $DIR/../lib/libzstd.a tmplog
-
-# compression disabled => also disable zdict and zbuff
-$ECHO "testing with compression disabled"
-ZSTD_LIB_COMPRESSION=0 CFLAGS= make -C $DIR/../lib libzstd.a > $INTOVOID
-nm $DIR/../lib/libzstd.a | $GREP "\.o" > tmplog
-mustBeAbsent "zstd_compress.o"
-isPresent "zstd_decompress.o"
-mustBeAbsent "zdict.o"
-isPresent "zstd_v07.o"
-mustBeAbsent "zbuff_compress.o"
-$RM $DIR/../lib/libzstd.a tmplog
-
-# decompression disabled => also disable legacy and zbuff
-$ECHO "testing with decompression disabled"
-ZSTD_LIB_DECOMPRESSION=0 CFLAGS= make -C $DIR/../lib libzstd.a > $INTOVOID
-nm $DIR/../lib/libzstd.a | $GREP "\.o" > tmplog
-isPresent "zstd_compress.o"
-mustBeAbsent "zstd_decompress.o"
-isPresent "zdict.o"
-mustBeAbsent "zstd_v07.o"
-mustBeAbsent "zbuff_compress.o"
-$RM $DIR/../lib/libzstd.a tmplog
-
-# deprecated function disabled => only remove zbuff
-$ECHO "testing with deprecated functions disabled"
-ZSTD_LIB_DEPRECATED=0 CFLAGS= make -C $DIR/../lib libzstd.a > $INTOVOID
-nm $DIR/../lib/libzstd.a | $GREP "\.o" > tmplog
-isPresent "zstd_compress.o"
-isPresent "zstd_decompress.o"
-isPresent "zdict.o"
-isPresent "zstd_v07.o"
-mustBeAbsent "zbuff_compress.o"
-$RM $DIR/../lib/libzstd.a tmplog
-
-# dictionary builder disabled => only remove zdict
-$ECHO "testing with dictionary builder disabled"
-ZSTD_LIB_DICTBUILDER=0 CFLAGS= make -C $DIR/../lib libzstd.a > $INTOVOID
-nm $DIR/../lib/libzstd.a | $GREP "\.o" > tmplog
-isPresent "zstd_compress.o"
-isPresent "zstd_decompress.o"
-mustBeAbsent "zdict.o"
-isPresent "zstd_v07.o"
-isPresent "zbuff_compress.o"
-$RM $DIR/../lib/libzstd.a tmplog
-
-# both decompression and dictionary builder disabled => only compression remains
-$ECHO "testing with both decompression and dictionary builder disabled (only compression remains)"
-ZSTD_LIB_DECOMPRESSION=0 ZSTD_LIB_DICTBUILDER=0 CFLAGS= make -C $DIR/../lib libzstd.a > $INTOVOID
-nm $DIR/../lib/libzstd.a | $GREP "\.o" > tmplog
-isPresent "zstd_compress.o"
-mustBeAbsent "zstd_decompress.o"
-mustBeAbsent "zdict.o"
-mustBeAbsent "zstd_v07.o"
-mustBeAbsent "zbuff_compress.o"
-$RM $DIR/../lib/libzstd.a tmplog
diff --git a/tests/longmatch.c b/tests/longmatch.c
deleted file mode 100644
index b673baa60140..000000000000
--- a/tests/longmatch.c
+++ /dev/null
@@ -1,101 +0,0 @@
-/*
- * Copyright (c) 2017-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-
-#include <stdio.h>
-#include <stddef.h>
-#include <stdlib.h>
-#include <stdint.h>
-#include "mem.h"
-#define ZSTD_STATIC_LINKING_ONLY
-#include "zstd.h"
-
-static int
-compress(ZSTD_CStream *ctx, ZSTD_outBuffer out, const void *data, size_t size)
-{
-  ZSTD_inBuffer in = { data, size, 0 };
-  while (in.pos < in.size) {
-    ZSTD_outBuffer tmp = out;
-    const size_t rc = ZSTD_compressStream(ctx, &tmp, &in);
-    if (ZSTD_isError(rc)) return 1;
-  }
-  { ZSTD_outBuffer tmp = out;
-    const size_t rc = ZSTD_flushStream(ctx, &tmp);
-    if (rc != 0) { return 1; }
-  }
-  return 0;
-}
-
-int main(int argc, const char** argv)
-{
-  ZSTD_CStream* ctx;
-  ZSTD_parameters params;
-  size_t rc;
-  unsigned windowLog;
-  (void)argc;
-  (void)argv;
-  /* Create stream */
-  ctx = ZSTD_createCStream();
-  if (!ctx) { return 1; }
-  /* Set parameters */
-  memset(&params, 0, sizeof(params));
-  params.cParams.windowLog = 18;
-  params.cParams.chainLog = 13;
-  params.cParams.hashLog = 14;
-  params.cParams.searchLog = 1;
-  params.cParams.minMatch = 7;
-  params.cParams.targetLength = 16;
-  params.cParams.strategy = ZSTD_fast;
-  windowLog = params.cParams.windowLog;
-  /* Initialize stream */
-  rc = ZSTD_initCStream_advanced(ctx, NULL, 0, params, 0);
-  if (ZSTD_isError(rc)) { return 2; }
-  {
-    U64 compressed = 0;
-    const U64 toCompress = ((U64)1) << 33;
-    const size_t size = 1 << windowLog;
-    size_t pos = 0;
-    char *srcBuffer = (char*) malloc(1 << windowLog);
-    char *dstBuffer = (char*) malloc(ZSTD_compressBound(1 << windowLog));
-    ZSTD_outBuffer out = { dstBuffer, ZSTD_compressBound(1 << windowLog), 0 };
-    const char match[] = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
-    const size_t randomData = (1 << windowLog) - 2*sizeof(match);
-    size_t i;
-    printf("\n ===   Long Match Test   === \n");
-    printf("Creating random data to produce long matches \n");
-    for (i = 0; i < sizeof(match); ++i) {
-      srcBuffer[i] = match[i];
-    }
-    for (i = 0; i < randomData; ++i) {
-      srcBuffer[sizeof(match) + i] = (char)(rand() & 0xFF);
-    }
-    for (i = 0; i < sizeof(match); ++i) {
-      srcBuffer[sizeof(match) + randomData + i] = match[i];
-    }
-    printf("Compressing, trying to generate a segfault \n");
-    if (compress(ctx, out, srcBuffer, size)) {
-      return 1;
-    }
-    compressed += size;
-    while (compressed < toCompress) {
-      const size_t block = rand() % (size - pos + 1);
-      if (pos == size) { pos = 0; }
-      if (compress(ctx, out, srcBuffer + pos, block)) {
-        return 1;
-      }
-      pos += block;
-      compressed += block;
-    }
-    printf("Compression completed successfully (no error triggered)\n");
-    free(srcBuffer);
-    free(dstBuffer);
-  }
-  return 0;
-}
diff --git a/tests/paramgrill.c b/tests/paramgrill.c
deleted file mode 100644
index 98fb313783a8..000000000000
--- a/tests/paramgrill.c
+++ /dev/null
@@ -1,2966 +0,0 @@
-/*
- * Copyright (c) 2015-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-
-/*-************************************
-*  Dependencies
-**************************************/
-#include "util.h"      /* Ensure platform.h is compiled first; also : compiler options, UTIL_GetFileSize */
-#include <stdlib.h>    /* malloc */
-#include <stdio.h>     /* fprintf, fopen, ftello64 */
-#include <string.h>    /* strcmp */
-#include <math.h>      /* log */
-#include <assert.h>
-
-#include "timefn.h"    /* SEC_TO_MICRO, UTIL_time_t, UTIL_clockSpanMicro, UTIL_clockSpanNano, UTIL_getTime */
-#include "mem.h"
-#define ZSTD_STATIC_LINKING_ONLY   /* ZSTD_parameters, ZSTD_estimateCCtxSize */
-#include "zstd.h"
-#include "datagen.h"
-#include "xxhash.h"
-#include "benchfn.h"
-#include "benchzstd.h"
-#include "zstd_errors.h"
-#include "zstd_internal.h"     /* should not be needed */
-
-
-/*-************************************
-*  Constants
-**************************************/
-#define PROGRAM_DESCRIPTION "ZSTD parameters tester"
-#define AUTHOR "Yann Collet"
-#define WELCOME_MESSAGE "*** %s %s %i-bits, by %s ***\n", PROGRAM_DESCRIPTION, ZSTD_VERSION_STRING, (int)(sizeof(void*)*8), AUTHOR
-
-#define TIMELOOP_NANOSEC      (1*1000000000ULL) /* 1 second */
-#define NB_LEVELS_TRACKED 22   /* ensured being >= ZSTD_maxCLevel() in BMK_init_level_constraints() */
-
-static const size_t maxMemory = (sizeof(size_t)==4)  ?  (2 GB - 64 MB) : (size_t)(1ULL << ((sizeof(size_t)*8)-31));
-
-#define COMPRESSIBILITY_DEFAULT 0.50
-
-static const U64 g_maxVariationTime = 60 * SEC_TO_MICRO;
-static const int g_maxNbVariations = 64;
-
-
-/*-************************************
-*  Macros
-**************************************/
-#define DISPLAY(...)  fprintf(stderr, __VA_ARGS__)
-#define DISPLAYLEVEL(n, ...) if(g_displayLevel >= n) { fprintf(stderr, __VA_ARGS__); }
-#define DEBUGOUTPUT(...) { if (DEBUG) DISPLAY(__VA_ARGS__); }
-
-#define TIMED 0
-#ifndef DEBUG
-#  define DEBUG 0
-#endif
-
-#undef MIN
-#undef MAX
-#define MIN(a,b)   ( (a) < (b) ? (a) : (b) )
-#define MAX(a,b)   ( (a) > (b) ? (a) : (b) )
-#define CUSTOM_LEVEL 99
-#define BASE_CLEVEL 1
-
-#define FADT_MIN 0
-#define FADT_MAX ((U32)-1)
-
-#define WLOG_RANGE (ZSTD_WINDOWLOG_MAX - ZSTD_WINDOWLOG_MIN + 1)
-#define CLOG_RANGE (ZSTD_CHAINLOG_MAX - ZSTD_CHAINLOG_MIN + 1)
-#define HLOG_RANGE (ZSTD_HASHLOG_MAX - ZSTD_HASHLOG_MIN + 1)
-#define SLOG_RANGE (ZSTD_SEARCHLOG_MAX - ZSTD_SEARCHLOG_MIN + 1)
-#define MML_RANGE  (ZSTD_MINMATCH_MAX - ZSTD_MINMATCH_MIN + 1)
-#define TLEN_RANGE  17
-#define STRT_RANGE (ZSTD_STRATEGY_MAX - ZSTD_STRATEGY_MIN + 1)
-#define FADT_RANGE   3
-
-#define CHECKTIME(r) { if(BMK_timeSpan_s(g_time) > g_timeLimit_s) { DEBUGOUTPUT("Time Limit Reached\n"); return r; } }
-#define CHECKTIMEGT(ret, val, _gototag) { if(BMK_timeSpan_s(g_time) > g_timeLimit_s) { DEBUGOUTPUT("Time Limit Reached\n"); ret = val; goto _gototag; } }
-
-#define PARAM_UNSET ((U32)-2) /* can't be -1 b/c fadt uses -1 */
-
-static const char* g_stratName[ZSTD_STRATEGY_MAX+1] = {
-                "(none)       ", "ZSTD_fast    ", "ZSTD_dfast   ",
-                "ZSTD_greedy  ", "ZSTD_lazy    ", "ZSTD_lazy2   ",
-                "ZSTD_btlazy2 ", "ZSTD_btopt   ", "ZSTD_btultra ",
-                "ZSTD_btultra2"};
-
-static const U32 tlen_table[TLEN_RANGE] = { 0, 1, 2, 4, 6, 8, 12, 16, 24, 32, 48, 64, 96, 128, 256, 512, 999 };
-
-
-/*-************************************
-*  Setup for Adding new params
-**************************************/
-
-/* indices for each of the variables */
-typedef enum {
-    wlog_ind = 0,
-    clog_ind = 1,
-    hlog_ind = 2,
-    slog_ind = 3,
-    mml_ind  = 4,
-    tlen_ind = 5,
-    strt_ind = 6,
-    fadt_ind = 7, /* forceAttachDict */
-    NUM_PARAMS = 8
-} varInds_t;
-
-typedef struct {
-    U32 vals[NUM_PARAMS];
-} paramValues_t;
-
-/* minimum value of parameters */
-static const U32 mintable[NUM_PARAMS] =
-        { ZSTD_WINDOWLOG_MIN, ZSTD_CHAINLOG_MIN, ZSTD_HASHLOG_MIN, ZSTD_SEARCHLOG_MIN, ZSTD_MINMATCH_MIN, ZSTD_TARGETLENGTH_MIN, ZSTD_STRATEGY_MIN, FADT_MIN };
-
-/* maximum value of parameters */
-static const U32 maxtable[NUM_PARAMS] =
-        { ZSTD_WINDOWLOG_MAX, ZSTD_CHAINLOG_MAX, ZSTD_HASHLOG_MAX, ZSTD_SEARCHLOG_MAX, ZSTD_MINMATCH_MAX, ZSTD_TARGETLENGTH_MAX, ZSTD_STRATEGY_MAX, FADT_MAX };
-
-/* # of values parameters can take on */
-static const U32 rangetable[NUM_PARAMS] =
-        { WLOG_RANGE, CLOG_RANGE, HLOG_RANGE, SLOG_RANGE, MML_RANGE, TLEN_RANGE, STRT_RANGE, FADT_RANGE };
-
-/* ZSTD_cctxSetParameter() index to set */
-static const ZSTD_cParameter cctxSetParamTable[NUM_PARAMS] =
-        { ZSTD_c_windowLog, ZSTD_c_chainLog, ZSTD_c_hashLog, ZSTD_c_searchLog, ZSTD_c_minMatch, ZSTD_c_targetLength, ZSTD_c_strategy, ZSTD_c_forceAttachDict };
-
-/* names of parameters */
-static const char* g_paramNames[NUM_PARAMS] =
-        { "windowLog", "chainLog", "hashLog","searchLog", "minMatch", "targetLength", "strategy", "forceAttachDict" };
-
-/* shortened names of parameters */
-static const char* g_shortParamNames[NUM_PARAMS] =
-        { "wlog", "clog", "hlog", "slog", "mml", "tlen", "strat", "fadt" };
-
-/* maps value from { 0 to rangetable[param] - 1 } to valid paramvalues */
-static U32 rangeMap(varInds_t param, int ind)
-{
-    U32 const uind = (U32)MAX(MIN(ind, (int)rangetable[param] - 1), 0);
-    switch(param) {
-        case wlog_ind: /* using default: triggers -Wswitch-enum */
-        case clog_ind:
-        case hlog_ind:
-        case slog_ind:
-        case mml_ind:
-        case strt_ind:
-            return mintable[param] + uind;
-        case tlen_ind:
-            return tlen_table[uind];
-        case fadt_ind: /* 0, 1, 2 -> -1, 0, 1 */
-            return uind - 1;
-        case NUM_PARAMS:
-        default:;
-    }
-    DISPLAY("Error, not a valid param\n ");
-    assert(0);
-    return (U32)-1;
-}
-
-/* inverse of rangeMap */
-static int invRangeMap(varInds_t param, U32 value)
-{
-    value = MIN(MAX(mintable[param], value), maxtable[param]);
-    switch(param) {
-        case wlog_ind:
-        case clog_ind:
-        case hlog_ind:
-        case slog_ind:
-        case mml_ind:
-        case strt_ind:
-            return (int)(value - mintable[param]);
-        case tlen_ind: /* bin search */
-        {
-            int lo = 0;
-            int hi = TLEN_RANGE;
-            while(lo < hi) {
-                int mid = (lo + hi) / 2;
-                if(tlen_table[mid] < value) {
-                    lo = mid + 1;
-                } if(tlen_table[mid] == value) {
-                    return mid;
-                } else {
-                    hi = mid;
-                }
-            }
-            return lo;
-        }
-        case fadt_ind:
-            return (int)value + 1;
-        case NUM_PARAMS:
-        default:;
-    }
-    DISPLAY("Error, not a valid param\n ");
-    assert(0);
-    return -2;
-}
-
-/* display of params */
-static void displayParamVal(FILE* f, varInds_t param, unsigned value, int width)
-{
-    switch(param) {
-        case wlog_ind:
-        case clog_ind:
-        case hlog_ind:
-        case slog_ind:
-        case mml_ind:
-        case tlen_ind:
-            if(width) {
-                fprintf(f, "%*u", width, value);
-            } else {
-                fprintf(f, "%u", value);
-            }
-            break;
-        case strt_ind:
-            if(width) {
-                fprintf(f, "%*s", width, g_stratName[value]);
-            } else {
-                fprintf(f, "%s", g_stratName[value]);
-            }
-            break;
-        case fadt_ind:   /* force attach dict */
-            if(width) {
-                fprintf(f, "%*d", width, (int)value);
-            } else {
-                fprintf(f, "%d", (int)value);
-            }
-            break;
-        case NUM_PARAMS:
-        default:
-            DISPLAY("Error, not a valid param\n ");
-            assert(0);
-            break;
-    }
-}
-
-
-/*-************************************
-*  Benchmark Parameters/Global Variables
-**************************************/
-
-/* General Utility */
-static U32 g_timeLimit_s = 99999;   /* about 27 hours */
-static UTIL_time_t g_time; /* to be used to compare solution finding speeds to compare to original */
-static U32 g_blockSize = 0;
-static U32 g_rand = 1;
-
-/* Display */
-static int g_displayLevel = 3;
-static BYTE g_silenceParams[NUM_PARAMS];   /* can selectively silence some params when displaying them */
-
-/* Mode Selection */
-static U32 g_singleRun = 0;
-static U32 g_optimizer = 0;
-static int g_optmode = 0;
-
-/* For cLevel Table generation */
-static U32 g_target = 0;
-static U32 g_noSeed = 0;
-
-/* For optimizer */
-static paramValues_t g_params; /* Initialized at the beginning of main w/ emptyParams() function */
-static double g_ratioMultiplier = 5.;
-static U32 g_strictness = PARAM_UNSET; /* range 1 - 100, measure of how strict  */
-static BMK_benchResult_t g_lvltarget;
-
-typedef enum {
-    directMap,
-    xxhashMap,
-    noMemo
-} memoTableType_t;
-
-typedef struct {
-    memoTableType_t tableType;
-    BYTE* table;
-    size_t tableLen;
-    varInds_t varArray[NUM_PARAMS];
-    size_t varLen;
-} memoTable_t;
-
-typedef struct {
-    BMK_benchResult_t result;
-    paramValues_t params;
-} winnerInfo_t;
-
-typedef struct {
-    U32 cSpeed;  /* bytes / sec */
-    U32 dSpeed;
-    U32 cMem;    /* bytes */
-} constraint_t;
-
-typedef struct winner_ll_node winner_ll_node;
-struct winner_ll_node {
-    winnerInfo_t res;
-    winner_ll_node* next;
-};
-
-static winner_ll_node* g_winners; /* linked list sorted ascending by cSize & cSpeed */
-
-/*
- * Additional Global Variables (Defined Above Use)
- * g_level_constraint
- * g_alreadyTested
- * g_maxTries
- * g_clockGranularity
- */
-
-
-/*-*******************************************************
-*  General Util Functions
-*********************************************************/
-
-/* nullified useless params, to ensure count stats */
-/* cleans up params for memoizing / display */
-static paramValues_t sanitizeParams(paramValues_t params)
-{
-    if (params.vals[strt_ind] == ZSTD_fast)
-        params.vals[clog_ind] = 0, params.vals[slog_ind] = 0;
-    if (params.vals[strt_ind] == ZSTD_dfast)
-        params.vals[slog_ind] = 0;
-    if ( (params.vals[strt_ind] < ZSTD_btopt) && (params.vals[strt_ind] != ZSTD_fast) )
-        params.vals[tlen_ind] = 0;
-
-    return params;
-}
-
-static ZSTD_compressionParameters pvalsToCParams(paramValues_t p)
-{
-    ZSTD_compressionParameters c;
-    memset(&c, 0, sizeof(ZSTD_compressionParameters));
-    c.windowLog = p.vals[wlog_ind];
-    c.chainLog = p.vals[clog_ind];
-    c.hashLog = p.vals[hlog_ind];
-    c.searchLog = p.vals[slog_ind];
-    c.minMatch = p.vals[mml_ind];
-    c.targetLength = p.vals[tlen_ind];
-    c.strategy = p.vals[strt_ind];
-    /* no forceAttachDict */
-    return c;
-}
-
-static paramValues_t cParamsToPVals(ZSTD_compressionParameters c)
-{
-    paramValues_t p;
-    varInds_t i;
-    p.vals[wlog_ind] = c.windowLog;
-    p.vals[clog_ind] = c.chainLog;
-    p.vals[hlog_ind] = c.hashLog;
-    p.vals[slog_ind] = c.searchLog;
-    p.vals[mml_ind]  = c.minMatch;
-    p.vals[tlen_ind] = c.targetLength;
-    p.vals[strt_ind] = c.strategy;
-
-    /* set all other params to their minimum value */
-    for (i = strt_ind + 1; i < NUM_PARAMS; i++) {
-        p.vals[i] = mintable[i];
-    }
-    return p;
-}
-
-/* equivalent of ZSTD_adjustCParams for paramValues_t */
-static paramValues_t
-adjustParams(paramValues_t p, const size_t maxBlockSize, const size_t dictSize)
-{
-    paramValues_t ot = p;
-    varInds_t i;
-    p = cParamsToPVals(ZSTD_adjustCParams(pvalsToCParams(p), maxBlockSize, dictSize));
-    if (!dictSize) { p.vals[fadt_ind] = 0; }
-    /* retain value of all other parameters */
-    for(i = strt_ind + 1; i < NUM_PARAMS; i++) {
-        p.vals[i] = ot.vals[i];
-    }
-    return p;
-}
-
-static size_t BMK_findMaxMem(U64 requiredMem)
-{
-    size_t const step = 64 MB;
-    void* testmem = NULL;
-
-    requiredMem = (((requiredMem >> 26) + 1) << 26);
-    if (requiredMem > maxMemory) requiredMem = maxMemory;
-
-    requiredMem += 2 * step;
-    while (!testmem && requiredMem > 0) {
-        testmem = malloc ((size_t)requiredMem);
-        requiredMem -= step;
-    }
-
-    free (testmem);
-    return (size_t) requiredMem;
-}
-
-/* accuracy in seconds only, span can be multiple years */
-static U32 BMK_timeSpan_s(const UTIL_time_t tStart)
-{
-    return (U32)(UTIL_clockSpanMicro(tStart) / 1000000ULL);
-}
-
-static U32 FUZ_rotl32(U32 x, U32 r)
-{
-    return ((x << r) | (x >> (32 - r)));
-}
-
-static U32 FUZ_rand(U32* src)
-{
-    const U32 prime1 = 2654435761U;
-    const U32 prime2 = 2246822519U;
-    U32 rand32 = *src;
-    rand32 *= prime1;
-    rand32 += prime2;
-    rand32  = FUZ_rotl32(rand32, 13);
-    *src = rand32;
-    return rand32 >> 5;
-}
-
-#define BOUNDCHECK(val,min,max) {                     \
-    if (((val)<(min)) | ((val)>(max))) {              \
-        DISPLAY("INVALID PARAMETER CONSTRAINTS\n");   \
-        return 0;                                     \
-}   }
-
-static int paramValid(const paramValues_t paramTarget)
-{
-    U32 i;
-    for(i = 0; i < NUM_PARAMS; i++) {
-        BOUNDCHECK(paramTarget.vals[i], mintable[i], maxtable[i]);
-    }
-    return 1;
-}
-
-/* cParamUnsetMin() :
- * if any parameter in paramTarget is not yet set,
- * it will receive its corresponding minimal value.
- * This function never fails */
-static paramValues_t cParamUnsetMin(paramValues_t paramTarget)
-{
-    varInds_t vi;
-    for (vi = 0; vi < NUM_PARAMS; vi++) {
-        if (paramTarget.vals[vi] == PARAM_UNSET) {
-            paramTarget.vals[vi] = mintable[vi];
-        }
-    }
-    return paramTarget;
-}
-
-static paramValues_t emptyParams(void)
-{
-    U32 i;
-    paramValues_t p;
-    for(i = 0; i < NUM_PARAMS; i++) {
-        p.vals[i] = PARAM_UNSET;
-    }
-    return p;
-}
-
-static winnerInfo_t initWinnerInfo(const paramValues_t p)
-{
-    winnerInfo_t w1;
-    w1.result.cSpeed = 0.;
-    w1.result.dSpeed = 0.;
-    w1.result.cMem = (size_t)-1;
-    w1.result.cSize = (size_t)-1;
-    w1.params = p;
-    return w1;
-}
-
-static paramValues_t
-overwriteParams(paramValues_t base, const paramValues_t mask)
-{
-    U32 i;
-    for(i = 0; i < NUM_PARAMS; i++) {
-        if(mask.vals[i] != PARAM_UNSET) {
-            base.vals[i] = mask.vals[i];
-        }
-    }
-    return base;
-}
-
-static void
-paramVaryOnce(const varInds_t paramIndex, const int amt, paramValues_t* ptr)
-{
-    ptr->vals[paramIndex] = rangeMap(paramIndex,
-                                     invRangeMap(paramIndex, ptr->vals[paramIndex]) + amt);
-}
-
-/* varies ptr by nbChanges respecting varyParams*/
-static void
-paramVariation(paramValues_t* ptr, memoTable_t* mtAll, const U32 nbChanges)
-{
-    paramValues_t p;
-    int validated = 0;
-    while (!validated) {
-        U32 i;
-        p = *ptr;
-        for (i = 0 ; i < nbChanges ; i++) {
-            const U32 changeID = (U32)FUZ_rand(&g_rand) % (mtAll[p.vals[strt_ind]].varLen << 1);
-            paramVaryOnce(mtAll[p.vals[strt_ind]].varArray[changeID >> 1],
-                          (int)((changeID & 1) << 1) - 1,
-                          &p);
-        }
-        validated = paramValid(p);
-    }
-    *ptr = p;
-}
-
-/* Completely random parameter selection */
-static paramValues_t randomParams(void)
-{
-    varInds_t v; paramValues_t p;
-    for(v = 0; v < NUM_PARAMS; v++) {
-        p.vals[v] = rangeMap(v, (int)(FUZ_rand(&g_rand) % rangetable[v]));
-    }
-    return p;
-}
-
-static U64 g_clockGranularity = 100000000ULL;
-
-static void init_clockGranularity(void)
-{
-    UTIL_time_t const clockStart = UTIL_getTime();
-    U64 el1 = 0, el2 = 0;
-    int i = 0;
-    do {
-        el1 = el2;
-        el2 = UTIL_clockSpanNano(clockStart);
-        if(el1 < el2) {
-            U64 iv = el2 - el1;
-            if(g_clockGranularity > iv) {
-                g_clockGranularity = iv;
-                i = 0;
-            } else {
-                i++;
-            }
-        }
-    } while(i < 10);
-    DEBUGOUTPUT("Granularity: %llu\n", (unsigned long long)g_clockGranularity);
-}
-
-/*-************************************
-*  Optimizer Util Functions
-**************************************/
-
-/* checks results are feasible */
-static int feasible(const BMK_benchResult_t results, const constraint_t target) {
-    return (results.cSpeed >= target.cSpeed)
-        && (results.dSpeed >= target.dSpeed)
-        && (results.cMem <= target.cMem)
-        && (!g_optmode || results.cSize <= g_lvltarget.cSize);
-}
-
-/* hill climbing value for part 1 */
-/* Scoring here is a linear reward for all set constraints normalized between 0 to 1
- * (with 0 at 0 and 1 being fully fulfilling the constraint), summed with a logarithmic
- * bonus to exceeding the constraint value. We also give linear ratio for compression ratio.
- * The constant factors are experimental.
- */
-static double
-resultScore(const BMK_benchResult_t res, const size_t srcSize, const constraint_t target)
-{
-    double cs = 0., ds = 0., rt, cm = 0.;
-    const double r1 = 1, r2 = 0.1, rtr = 0.5;
-    double ret;
-    if(target.cSpeed) { cs = res.cSpeed / (double)target.cSpeed; }
-    if(target.dSpeed) { ds = res.dSpeed / (double)target.dSpeed; }
-    if(target.cMem != (U32)-1) { cm = (double)target.cMem / res.cMem; }
-    rt = ((double)srcSize / res.cSize);
-
-    ret = (MIN(1, cs) + MIN(1, ds)  + MIN(1, cm))*r1 + rt * rtr +
-         (MAX(0, log(cs))+ MAX(0, log(ds))+ MAX(0, log(cm))) * r2;
-
-    return ret;
-}
-
-/* calculates normalized squared euclidean distance of result1 if it is in the first quadrant relative to lvlRes */
-static double
-resultDistLvl(const BMK_benchResult_t result1, const BMK_benchResult_t lvlRes)
-{
-    double normalizedCSpeedGain1 = (result1.cSpeed / lvlRes.cSpeed) - 1;
-    double normalizedRatioGain1 = ((double)lvlRes.cSize / result1.cSize) - 1;
-    if(normalizedRatioGain1 < 0 || normalizedCSpeedGain1 < 0) {
-        return 0.0;
-    }
-    return normalizedRatioGain1 * g_ratioMultiplier + normalizedCSpeedGain1;
-}
-
-/* return true if r2 strictly better than r1 */
-static int
-compareResultLT(const BMK_benchResult_t result1, const BMK_benchResult_t result2, const constraint_t target, size_t srcSize)
-{
-    if(feasible(result1, target) && feasible(result2, target)) {
-        if(g_optmode) {
-            return resultDistLvl(result1, g_lvltarget) < resultDistLvl(result2, g_lvltarget);
-        } else {
-            return (result1.cSize > result2.cSize)
-                || (result1.cSize == result2.cSize && result2.cSpeed > result1.cSpeed)
-                || (result1.cSize == result2.cSize && result2.cSpeed == result1.cSpeed && result2.dSpeed > result1.dSpeed);
-        }
-    }
-    return feasible(result2, target)
-        || (!feasible(result1, target)
-            && (resultScore(result1, srcSize, target) < resultScore(result2, srcSize, target)));
-}
-
-static constraint_t relaxTarget(constraint_t target) {
-    target.cMem = (U32)-1;
-    target.cSpeed = (target.cSpeed * g_strictness) / 100;
-    target.dSpeed = (target.dSpeed * g_strictness) / 100;
-    return target;
-}
-
-static void optimizerAdjustInput(paramValues_t* pc, const size_t maxBlockSize)
-{
-    varInds_t v;
-    for(v = 0; v < NUM_PARAMS; v++) {
-        if(pc->vals[v] != PARAM_UNSET) {
-            U32 newval = MIN(MAX(pc->vals[v], mintable[v]), maxtable[v]);
-            if(newval != pc->vals[v]) {
-                pc->vals[v] = newval;
-                DISPLAY("Warning: parameter %s not in valid range, adjusting to ",
-                        g_paramNames[v]);
-                displayParamVal(stderr, v, newval, 0); DISPLAY("\n");
-            }
-        }
-    }
-
-    if(pc->vals[wlog_ind] != PARAM_UNSET) {
-
-        U32 sshb = maxBlockSize > 1 ? ZSTD_highbit32((U32)(maxBlockSize-1)) + 1 : 1;
-        /* edge case of highBit not working for 0 */
-
-        if(maxBlockSize < (1ULL << 31) && sshb + 1 < pc->vals[wlog_ind]) {
-            U32 adjust = MAX(mintable[wlog_ind], sshb);
-            if(adjust != pc->vals[wlog_ind]) {
-                pc->vals[wlog_ind] = adjust;
-                DISPLAY("Warning: windowLog larger than src/block size, adjusted to %u\n",
-                        (unsigned)pc->vals[wlog_ind]);
-            }
-        }
-    }
-
-    if(pc->vals[wlog_ind] != PARAM_UNSET && pc->vals[clog_ind] != PARAM_UNSET) {
-        U32 maxclog;
-        if(pc->vals[strt_ind] == PARAM_UNSET || pc->vals[strt_ind] >= (U32)ZSTD_btlazy2) {
-            maxclog = pc->vals[wlog_ind] + 1;
-        } else {
-            maxclog = pc->vals[wlog_ind];
-        }
-
-        if(pc->vals[clog_ind] > maxclog) {
-            pc->vals[clog_ind] = maxclog;
-            DISPLAY("Warning: chainlog too much larger than windowLog size, adjusted to %u\n",
-                    (unsigned)pc->vals[clog_ind]);
-        }
-    }
-
-    if(pc->vals[wlog_ind] != PARAM_UNSET && pc->vals[hlog_ind] != PARAM_UNSET) {
-        if(pc->vals[wlog_ind] + 1 < pc->vals[hlog_ind]) {
-            pc->vals[hlog_ind] = pc->vals[wlog_ind] + 1;
-            DISPLAY("Warning: hashlog too much larger than windowLog size, adjusted to %u\n",
-                    (unsigned)pc->vals[hlog_ind]);
-        }
-    }
-
-    if(pc->vals[slog_ind] != PARAM_UNSET && pc->vals[clog_ind] != PARAM_UNSET) {
-        if(pc->vals[slog_ind] > pc->vals[clog_ind]) {
-            pc->vals[clog_ind] = pc->vals[slog_ind];
-            DISPLAY("Warning: searchLog larger than chainLog, adjusted to %u\n",
-                    (unsigned)pc->vals[slog_ind]);
-        }
-    }
-}
-
-static int
-redundantParams(const paramValues_t paramValues, const constraint_t target, const size_t maxBlockSize)
-{
-    return
-       (ZSTD_estimateCStreamSize_usingCParams(pvalsToCParams(paramValues)) > (size_t)target.cMem) /* Uses too much memory */
-    || ((1ULL << (paramValues.vals[wlog_ind] - 1)) >= maxBlockSize && paramValues.vals[wlog_ind] != mintable[wlog_ind]) /* wlog too much bigger than src size */
-    || (paramValues.vals[clog_ind] > (paramValues.vals[wlog_ind] + (paramValues.vals[strt_ind] > ZSTD_btlazy2))) /* chainLog larger than windowLog*/
-    || (paramValues.vals[slog_ind] > paramValues.vals[clog_ind]) /* searchLog larger than chainLog */
-    || (paramValues.vals[hlog_ind] > paramValues.vals[wlog_ind] + 1); /* hashLog larger than windowLog + 1 */
-}
-
-
-/*-************************************
-*  Display Functions
-**************************************/
-
-/* BMK_paramValues_into_commandLine() :
- * transform a set of parameters paramValues_t
- * into a command line compatible with `zstd` syntax
- * and writes it into FILE* f.
- * f must be already opened and writable */
-static void
-BMK_paramValues_into_commandLine(FILE* f, const paramValues_t params)
-{
-    varInds_t v;
-    int first = 1;
-    fprintf(f,"--zstd=");
-    for (v = 0; v < NUM_PARAMS; v++) {
-        if (g_silenceParams[v]) { continue; }
-        if (!first) { fprintf(f, ","); }
-        fprintf(f,"%s=", g_paramNames[v]);
-
-        if (v == strt_ind) { fprintf(f,"%u", (unsigned)params.vals[v]); }
-        else { displayParamVal(f, v, params.vals[v], 0); }
-        first = 0;
-    }
-    fprintf(f, "\n");
-}
-
-
-/* comparison function: */
-/* strictly better, strictly worse, equal, speed-side adv, size-side adv */
-#define WORSE_RESULT 0
-#define BETTER_RESULT 1
-#define ERROR_RESULT 2
-
-#define SPEED_RESULT 4
-#define SIZE_RESULT 5
-/* maybe have epsilon-eq to limit table size? */
-static int
-speedSizeCompare(const BMK_benchResult_t r1, const BMK_benchResult_t r2)
-{
-    if(r1.cSpeed < r2.cSpeed) {
-        if(r1.cSize >= r2.cSize) {
-            return BETTER_RESULT;
-        }
-        return SPEED_RESULT; /* r2 is smaller but not faster. */
-    } else {
-        if(r1.cSize <= r2.cSize) {
-            return WORSE_RESULT;
-        }
-        return SIZE_RESULT; /* r2 is faster but not smaller */
-    }
-}
-
-/* 0 for insertion, 1 for no insert */
-/* maintain invariant speedSizeCompare(n, n->next) = SPEED_RESULT */
-static int
-insertWinner(const winnerInfo_t w, const constraint_t targetConstraints)
-{
-    BMK_benchResult_t r = w.result;
-    winner_ll_node* cur_node = g_winners;
-    /* first node to insert */
-    if(!feasible(r, targetConstraints)) {
-        return 1;
-    }
-
-    if(g_winners == NULL) {
-        winner_ll_node* first_node = malloc(sizeof(winner_ll_node));
-        if(first_node == NULL) {
-            return 1;
-        }
-        first_node->next = NULL;
-        first_node->res = w;
-        g_winners = first_node;
-        return 0;
-    }
-
-    while(cur_node->next != NULL) {
-        switch(speedSizeCompare(cur_node->res.result, r)) {
-            case WORSE_RESULT:
-            {
-                return 1; /* never insert if better */
-            }
-            case BETTER_RESULT:
-            {
-                winner_ll_node* tmp;
-                cur_node->res = cur_node->next->res;
-                tmp = cur_node->next;
-                cur_node->next = cur_node->next->next;
-                free(tmp);
-                break;
-            }
-            case SIZE_RESULT:
-            {
-                cur_node = cur_node->next;
-                break;
-            }
-            case SPEED_RESULT: /* insert after first size result, then return */
-            {
-                winner_ll_node* newnode = malloc(sizeof(winner_ll_node));
-                if(newnode == NULL) {
-                    return 1;
-                }
-                newnode->res = cur_node->res;
-                cur_node->res = w;
-                newnode->next = cur_node->next;
-                cur_node->next = newnode;
-                return 0;
-            }
-        }
-
-    }
-
-    assert(cur_node->next == NULL);
-    switch(speedSizeCompare(cur_node->res.result, r)) {
-        case WORSE_RESULT:
-        {
-            return 1; /* never insert if better */
-        }
-        case BETTER_RESULT:
-        {
-            cur_node->res = w;
-            return 0;
-        }
-        case SIZE_RESULT:
-        {
-            winner_ll_node* newnode = malloc(sizeof(winner_ll_node));
-            if(newnode == NULL) {
-                return 1;
-            }
-            newnode->res = w;
-            newnode->next = NULL;
-            cur_node->next = newnode;
-            return 0;
-        }
-        case SPEED_RESULT: /* insert before first size result, then return */
-        {
-            winner_ll_node* newnode = malloc(sizeof(winner_ll_node));
-            if(newnode == NULL) {
-                return 1;
-            }
-            newnode->res = cur_node->res;
-            cur_node->res = w;
-            newnode->next = cur_node->next;
-            cur_node->next = newnode;
-            return 0;
-        }
-        default:
-            return 1;
-    }
-}
-
-static void
-BMK_displayOneResult(FILE* f, winnerInfo_t res, const size_t srcSize)
-{
-    varInds_t v;
-    int first = 1;
-    res.params = cParamUnsetMin(res.params);
-    fprintf(f, "    {");
-    for (v = 0; v < NUM_PARAMS; v++) {
-        if (g_silenceParams[v]) { continue; }
-        if (!first) { fprintf(f, ","); }
-        displayParamVal(f, v, res.params.vals[v], 3);
-        first = 0;
-    }
-
-    {   double const ratio = res.result.cSize ?
-                            (double)srcSize / res.result.cSize : 0;
-        double const cSpeedMBps = (double)res.result.cSpeed / MB_UNIT;
-        double const dSpeedMBps = (double)res.result.dSpeed / MB_UNIT;
-
-        fprintf(f, " },     /* R:%5.3f at %5.1f MB/s - %5.1f MB/s */\n",
-                            ratio, cSpeedMBps, dSpeedMBps);
-    }
-}
-
-/* Writes to f the results of a parameter benchmark */
-/* when used with --optimize, will only print results better than previously discovered */
-static void
-BMK_printWinner(FILE* f, const int cLevel, const BMK_benchResult_t result, const paramValues_t params, const size_t srcSize)
-{
-    char lvlstr[15] = "Custom Level";
-    winnerInfo_t w;
-    w.params = params;
-    w.result = result;
-
-    fprintf(f, "\r%79s\r", "");
-
-    if(cLevel != CUSTOM_LEVEL) {
-        snprintf(lvlstr, 15, "  Level %2d  ", cLevel);
-    }
-
-    if(TIMED) {
-        const U64 mn_in_ns = 60ULL * TIMELOOP_NANOSEC;
-        const U64 time_ns = UTIL_clockSpanNano(g_time);
-        const U64 minutes = time_ns / mn_in_ns;
-        fprintf(f, "%1lu:%2lu:%05.2f - ",
-                (unsigned long) minutes / 60,
-                (unsigned long) minutes % 60,
-                (double)(time_ns - (minutes * mn_in_ns)) / TIMELOOP_NANOSEC );
-    }
-
-    fprintf(f, "/* %s */   ", lvlstr);
-    BMK_displayOneResult(f, w, srcSize);
-}
-
-static void
-BMK_printWinnerOpt(FILE* f, const U32 cLevel, const BMK_benchResult_t result, const paramValues_t params, const constraint_t targetConstraints, const size_t srcSize)
-{
-    /* global winner used for constraints */
-                                    /* cSize, cSpeed, dSpeed, cMem */
-    static winnerInfo_t g_winner = { { (size_t)-1LL, 0, 0, (size_t)-1LL },
-                                     { { PARAM_UNSET, PARAM_UNSET, PARAM_UNSET, PARAM_UNSET, PARAM_UNSET, PARAM_UNSET, PARAM_UNSET, PARAM_UNSET } }
-                                   };
-    if ( DEBUG
-      || compareResultLT(g_winner.result, result, targetConstraints, srcSize)
-      || g_displayLevel >= 4) {
-        if ( DEBUG
-          && compareResultLT(g_winner.result, result, targetConstraints, srcSize)) {
-            DISPLAY("New Winner: \n");
-        }
-
-        if(g_displayLevel >= 2) {
-            BMK_printWinner(f, cLevel, result, params, srcSize);
-        }
-
-        if(compareResultLT(g_winner.result, result, targetConstraints, srcSize)) {
-            if(g_displayLevel >= 1) { BMK_paramValues_into_commandLine(f, params); }
-            g_winner.result = result;
-            g_winner.params = params;
-        }
-    }
-
-    if(g_optmode && g_optimizer && (DEBUG || g_displayLevel == 3)) {
-        winnerInfo_t w;
-        winner_ll_node* n;
-        w.result = result;
-        w.params = params;
-        insertWinner(w, targetConstraints);
-
-        if(!DEBUG) { fprintf(f, "\033c"); }
-        fprintf(f, "\n");
-
-        /* the table */
-        fprintf(f, "================================\n");
-        for(n = g_winners; n != NULL; n = n->next) {
-            BMK_displayOneResult(f, n->res, srcSize);
-        }
-        fprintf(f, "================================\n");
-        fprintf(f, "Level Bounds: R: > %.3f AND C: < %.1f MB/s \n\n",
-            (double)srcSize / g_lvltarget.cSize, (double)g_lvltarget.cSpeed / MB_UNIT);
-
-
-        fprintf(f, "Overall Winner: \n");
-        BMK_displayOneResult(f, g_winner, srcSize);
-        BMK_paramValues_into_commandLine(f, g_winner.params);
-
-        fprintf(f, "Latest BMK: \n");\
-        BMK_displayOneResult(f, w, srcSize);
-    }
-}
-
-
-/* BMK_print_cLevelEntry() :
- * Writes one cLevelTable entry, for one level.
- * f must exist, be already opened, and be seekable.
- * this function cannot error.
- */
-static void
-BMK_print_cLevelEntry(FILE* f, const int cLevel,
-                      paramValues_t params,
-                      const BMK_benchResult_t result, const size_t srcSize)
-{
-    varInds_t v;
-    int first = 1;
-
-    assert(cLevel >= 0);
-    assert(cLevel <= NB_LEVELS_TRACKED);
-    params = cParamUnsetMin(params);
-
-    fprintf(f, "   {");
-    /* print cParams.
-     * assumption : all cParams are present and in order in the following range */
-    for (v = 0; v <= strt_ind; v++) {
-        if (!first) { fprintf(f, ","); }
-        displayParamVal(f, v, params.vals[v], 3);
-        first = 0;
-    }
-    /* print comment */
-    {   double const ratio = result.cSize ?
-                            (double)srcSize / result.cSize : 0;
-        double const cSpeedMBps = (double)result.cSpeed / MB_UNIT;
-        double const dSpeedMBps = (double)result.dSpeed / MB_UNIT;
-
-        fprintf(f, " },   /* level %2i:  R=%5.3f at %5.1f MB/s - %5.1f MB/s */\n",
-                             cLevel, ratio, cSpeedMBps, dSpeedMBps);
-    }
-}
-
-
-/* BMK_print_cLevelTable() :
- * print candidate compression table into proposed FILE* f.
- * f must exist, be already opened, and be seekable.
- * winners must be a table of NB_LEVELS_TRACKED+1 elements winnerInfo_t, all entries presumed initialized
- * this function cannot error.
- */
-static void
-BMK_print_cLevelTable(FILE* f, const winnerInfo_t* winners, const size_t srcSize)
-{
-    int cLevel;
-
-    fprintf(f, "\n /* Proposed configurations : */ \n");
-    fprintf(f, "   /* W,  C,  H,  S,  L,  T, strat */ \n");
-
-    for (cLevel=0; cLevel <= NB_LEVELS_TRACKED; cLevel++)
-        BMK_print_cLevelEntry(f,
-                              cLevel, winners[cLevel].params,
-                              winners[cLevel].result, srcSize);
-}
-
-
-/* BMK_saveAndPrint_cLevelTable() :
- * save candidate compression table into FILE* f,
- * and then to stdout.
- * f must exist, be already opened, and be seekable.
- * winners must be a table of NB_LEVELS_TRACKED+1 elements winnerInfo_t, all entries presumed initialized
- * this function cannot error.
- */
-static void
-BMK_saveAndPrint_cLevelTable(FILE* const f,
-                       const winnerInfo_t* winners,
-                       const size_t srcSize)
-{
-    fseek(f, 0, SEEK_SET);
-    BMK_print_cLevelTable(f, winners, srcSize);
-    fflush(f);
-    BMK_print_cLevelTable(stdout, winners, srcSize);
-}
-
-
-/*-*******************************************************
-*  Functions to Benchmark
-*********************************************************/
-
-typedef struct {
-    ZSTD_CCtx* cctx;
-    const void* dictBuffer;
-    size_t dictBufferSize;
-    int cLevel;
-    const paramValues_t* comprParams;
-} BMK_initCCtxArgs;
-
-static size_t local_initCCtx(void* payload) {
-    const BMK_initCCtxArgs* ag = (const BMK_initCCtxArgs*)payload;
-    varInds_t i;
-    ZSTD_CCtx_reset(ag->cctx, ZSTD_reset_session_and_parameters);
-    ZSTD_CCtx_setParameter(ag->cctx, ZSTD_c_compressionLevel, ag->cLevel);
-
-    for(i = 0; i < NUM_PARAMS; i++) {
-        if(ag->comprParams->vals[i] != PARAM_UNSET)
-        ZSTD_CCtx_setParameter(ag->cctx, cctxSetParamTable[i], ag->comprParams->vals[i]);
-    }
-    ZSTD_CCtx_loadDictionary(ag->cctx, ag->dictBuffer, ag->dictBufferSize);
-
-    return 0;
-}
-
-typedef struct {
-    ZSTD_DCtx* dctx;
-    const void* dictBuffer;
-    size_t dictBufferSize;
-} BMK_initDCtxArgs;
-
-static size_t local_initDCtx(void* payload) {
-    const BMK_initDCtxArgs* ag = (const BMK_initDCtxArgs*)payload;
-    ZSTD_DCtx_reset(ag->dctx, ZSTD_reset_session_and_parameters);
-    ZSTD_DCtx_loadDictionary(ag->dctx, ag->dictBuffer, ag->dictBufferSize);
-    return 0;
-}
-
-/* additional argument is just the context */
-static size_t local_defaultCompress(
-                            const void* srcBuffer, size_t srcSize,
-                            void* dstBuffer, size_t dstSize,
-                            void* addArgs)
-{
-    ZSTD_CCtx* cctx = (ZSTD_CCtx*)addArgs;
-    assert(dstSize == ZSTD_compressBound(srcSize)); /* specific to this version, which is only used in paramgrill */
-    return ZSTD_compress2(cctx, dstBuffer, dstSize, srcBuffer, srcSize);
-}
-
-/* additional argument is just the context */
-static size_t local_defaultDecompress(
-    const void* srcBuffer, size_t srcSize,
-    void* dstBuffer, size_t dstSize,
-    void* addArgs) {
-    size_t moreToFlush = 1;
-    ZSTD_DCtx* dctx = (ZSTD_DCtx*)addArgs;
-    ZSTD_inBuffer in;
-    ZSTD_outBuffer out;
-    in.src = srcBuffer;
-    in.size = srcSize;
-    in.pos = 0;
-    out.dst = dstBuffer;
-    out.size = dstSize;
-    out.pos = 0;
-    while (moreToFlush) {
-        if(out.pos == out.size) {
-            return (size_t)-ZSTD_error_dstSize_tooSmall;
-        }
-        moreToFlush = ZSTD_decompressStream(dctx,
-                            &out, &in);
-        if (ZSTD_isError(moreToFlush)) {
-            return moreToFlush;
-        }
-    }
-    return out.pos;
-
-}
-
-/*-************************************
-*  Data Initialization Functions
-**************************************/
-
-typedef struct {
-    void* srcBuffer;
-    size_t srcSize;
-    const void** srcPtrs;
-    size_t* srcSizes;
-    void** dstPtrs;
-    size_t* dstCapacities;
-    size_t* dstSizes;
-    void** resPtrs;
-    size_t* resSizes;
-    size_t nbBlocks;
-    size_t maxBlockSize;
-} buffers_t;
-
-typedef struct {
-    size_t dictSize;
-    void* dictBuffer;
-    ZSTD_CCtx* cctx;
-    ZSTD_DCtx* dctx;
-} contexts_t;
-
-static void freeNonSrcBuffers(const buffers_t b) {
-    free(b.srcPtrs);
-    free(b.srcSizes);
-
-    if(b.dstPtrs != NULL) {
-        free(b.dstPtrs[0]);
-    }
-    free(b.dstPtrs);
-    free(b.dstCapacities);
-    free(b.dstSizes);
-
-    if(b.resPtrs != NULL) {
-        free(b.resPtrs[0]);
-    }
-    free(b.resPtrs);
-    free(b.resSizes);
-}
-
-static void freeBuffers(const buffers_t b) {
-    if(b.srcPtrs != NULL) {
-        free(b.srcBuffer);
-    }
-    freeNonSrcBuffers(b);
-}
-
-/* srcBuffer will be freed by freeBuffers now */
-static int createBuffersFromMemory(buffers_t* buff, void * srcBuffer, const size_t nbFiles,
-    const size_t* fileSizes)
-{
-    size_t pos = 0, n, blockSize;
-    U32 maxNbBlocks, blockNb = 0;
-    buff->srcSize = 0;
-    for(n = 0; n < nbFiles; n++) {
-        buff->srcSize += fileSizes[n];
-    }
-
-    if(buff->srcSize == 0) {
-        DISPLAY("No data to bench\n");
-        return 1;
-    }
-
-    blockSize = g_blockSize ? g_blockSize : buff->srcSize;
-    maxNbBlocks = (U32) ((buff->srcSize + (blockSize-1)) / blockSize) + (U32)nbFiles;
-
-    buff->srcPtrs = (const void**)calloc(maxNbBlocks, sizeof(void*));
-    buff->srcSizes = (size_t*)malloc(maxNbBlocks * sizeof(size_t));
-
-    buff->dstPtrs = (void**)calloc(maxNbBlocks, sizeof(void*));
-    buff->dstCapacities = (size_t*)malloc(maxNbBlocks * sizeof(size_t));
-    buff->dstSizes = (size_t*)malloc(maxNbBlocks * sizeof(size_t));
-
-    buff->resPtrs = (void**)calloc(maxNbBlocks, sizeof(void*));
-    buff->resSizes = (size_t*)malloc(maxNbBlocks * sizeof(size_t));
-
-    if(!buff->srcPtrs || !buff->srcSizes || !buff->dstPtrs || !buff->dstCapacities || !buff->dstSizes || !buff->resPtrs || !buff->resSizes) {
-        DISPLAY("alloc error\n");
-        freeNonSrcBuffers(*buff);
-        return 1;
-    }
-
-    buff->srcBuffer = srcBuffer;
-    buff->srcPtrs[0] = (const void*)buff->srcBuffer;
-    buff->dstPtrs[0] = malloc(ZSTD_compressBound(buff->srcSize) + (maxNbBlocks * 1024));
-    buff->resPtrs[0] = malloc(buff->srcSize);
-
-    if(!buff->dstPtrs[0] || !buff->resPtrs[0]) {
-        DISPLAY("alloc error\n");
-        freeNonSrcBuffers(*buff);
-        return 1;
-    }
-
-    for(n = 0; n < nbFiles; n++) {
-        size_t pos_end = pos + fileSizes[n];
-        for(; pos < pos_end; blockNb++) {
-            buff->srcPtrs[blockNb] = (const void*)((char*)srcBuffer + pos);
-            buff->srcSizes[blockNb] = blockSize;
-            pos += blockSize;
-        }
-
-        if(fileSizes[n] > 0) { buff->srcSizes[blockNb - 1] = ((fileSizes[n] - 1) % blockSize) + 1; }
-        pos = pos_end;
-    }
-
-    buff->dstCapacities[0] = ZSTD_compressBound(buff->srcSizes[0]);
-    buff->dstSizes[0] = buff->dstCapacities[0];
-    buff->resSizes[0] = buff->srcSizes[0];
-    buff->maxBlockSize = buff->srcSizes[0];
-
-    for(n = 1; n < blockNb; n++) {
-        buff->dstPtrs[n] = ((char*)buff->dstPtrs[n-1]) + buff->dstCapacities[n-1];
-        buff->resPtrs[n] = ((char*)buff->resPtrs[n-1]) + buff->resSizes[n-1];
-        buff->dstCapacities[n] = ZSTD_compressBound(buff->srcSizes[n]);
-        buff->dstSizes[n] = buff->dstCapacities[n];
-        buff->resSizes[n] = buff->srcSizes[n];
-
-        buff->maxBlockSize = MAX(buff->maxBlockSize, buff->srcSizes[n]);
-    }
-
-    buff->nbBlocks = blockNb;
-
-    return 0;
-}
-
-/* allocates buffer's arguments. returns success / failure */
-static int createBuffers(buffers_t* buff, const char* const * const fileNamesTable,
-                          size_t nbFiles) {
-    size_t pos = 0;
-    size_t n;
-    size_t totalSizeToLoad = UTIL_getTotalFileSize(fileNamesTable, (U32)nbFiles);
-    size_t benchedSize = MIN(BMK_findMaxMem(totalSizeToLoad * 3) / 3, totalSizeToLoad);
-    size_t* fileSizes = calloc(sizeof(size_t), nbFiles);
-    void* srcBuffer = NULL;
-    int ret = 0;
-
-    if(!totalSizeToLoad || !benchedSize) {
-        ret = 1;
-        DISPLAY("Nothing to Bench\n");
-        goto _cleanUp;
-    }
-
-    srcBuffer = malloc(benchedSize);
-
-    if(!fileSizes || !srcBuffer) {
-        ret = 1;
-        goto _cleanUp;
-    }
-
-    for(n = 0; n < nbFiles; n++) {
-        FILE* f;
-        U64 fileSize = UTIL_getFileSize(fileNamesTable[n]);
-        if (UTIL_isDirectory(fileNamesTable[n])) {
-            DISPLAY("Ignoring %s directory...       \n", fileNamesTable[n]);
-            continue;
-        }
-        if (fileSize == UTIL_FILESIZE_UNKNOWN) {
-            DISPLAY("Cannot evaluate size of %s, ignoring ... \n", fileNamesTable[n]);
-            continue;
-        }
-        f = fopen(fileNamesTable[n], "rb");
-        if (f==NULL) {
-            DISPLAY("impossible to open file %s\n", fileNamesTable[n]);
-            fclose(f);
-            ret = 10;
-            goto _cleanUp;
-        }
-
-        DISPLAYLEVEL(2, "Loading %s...       \r", fileNamesTable[n]);
-
-        if (fileSize + pos > benchedSize) fileSize = benchedSize - pos, nbFiles=n;   /* buffer too small - stop after this file */
-        {
-            char* buffer = (char*)(srcBuffer);
-            size_t const readSize = fread((buffer)+pos, 1, (size_t)fileSize, f);
-            fclose(f);
-            if (readSize != (size_t)fileSize) {
-                DISPLAY("could not read %s", fileNamesTable[n]);
-                ret = 1;
-                goto _cleanUp;
-            }
-
-            fileSizes[n] = readSize;
-            pos += readSize;
-        }
-    }
-
-    ret = createBuffersFromMemory(buff, srcBuffer, nbFiles, fileSizes);
-
-_cleanUp:
-    if(ret) { free(srcBuffer); }
-    free(fileSizes);
-    return ret;
-}
-
-static void freeContexts(const contexts_t ctx) {
-    free(ctx.dictBuffer);
-    ZSTD_freeCCtx(ctx.cctx);
-    ZSTD_freeDCtx(ctx.dctx);
-}
-
-static int createContexts(contexts_t* ctx, const char* dictFileName) {
-    FILE* f;
-    size_t readSize;
-    ctx->cctx = ZSTD_createCCtx();
-    ctx->dctx = ZSTD_createDCtx();
-    assert(ctx->cctx != NULL);
-    assert(ctx->dctx != NULL);
-
-    if(dictFileName == NULL) {
-        ctx->dictSize = 0;
-        ctx->dictBuffer = NULL;
-        return 0;
-    }
-    {   U64 const dictFileSize = UTIL_getFileSize(dictFileName);
-        assert(dictFileSize != UTIL_FILESIZE_UNKNOWN);
-        ctx->dictSize = dictFileSize;
-        assert((U64)ctx->dictSize == dictFileSize); /* check overflow */
-    }
-    ctx->dictBuffer = malloc(ctx->dictSize);
-
-    f = fopen(dictFileName, "rb");
-
-    if (f==NULL) {
-        DISPLAY("unable to open file\n");
-        freeContexts(*ctx);
-        return 1;
-    }
-
-    if (ctx->dictSize > 64 MB || !(ctx->dictBuffer)) {
-        DISPLAY("dictionary too large\n");
-        fclose(f);
-        freeContexts(*ctx);
-        return 1;
-    }
-    readSize = fread(ctx->dictBuffer, 1, ctx->dictSize, f);
-    fclose(f);
-    if (readSize != ctx->dictSize) {
-        DISPLAY("unable to read file\n");
-        freeContexts(*ctx);
-        return 1;
-    }
-    return 0;
-}
-
-/*-************************************
-*  Optimizer Memoization Functions
-**************************************/
-
-/* return: new length */
-/* keep old array, will need if iter over strategy. */
-/* prunes useless params */
-static size_t sanitizeVarArray(varInds_t* varNew, const size_t varLength, const varInds_t* varArray, const ZSTD_strategy strat) {
-    size_t i, j = 0;
-    for(i = 0; i < varLength; i++) {
-        if( !((varArray[i] == clog_ind && strat == ZSTD_fast)
-            || (varArray[i] == slog_ind && strat == ZSTD_fast)
-            || (varArray[i] == slog_ind && strat == ZSTD_dfast)
-            || (varArray[i] == tlen_ind && strat < ZSTD_btopt && strat != ZSTD_fast))) {
-            varNew[j] = varArray[i];
-            j++;
-        }
-    }
-    return j;
-}
-
-/* res should be NUM_PARAMS size */
-/* constructs varArray from paramValues_t style parameter */
-/* pass in using dict. */
-static size_t variableParams(const paramValues_t paramConstraints, varInds_t* res, const int usingDictionary) {
-    varInds_t i;
-    size_t j = 0;
-    for(i = 0; i < NUM_PARAMS; i++) {
-        if(paramConstraints.vals[i] == PARAM_UNSET) {
-            if(i == fadt_ind && !usingDictionary) continue; /* don't use fadt if no dictionary */
-            res[j] = i; j++;
-        }
-    }
-    return j;
-}
-
-/* length of memo table given free variables */
-static size_t memoTableLen(const varInds_t* varyParams, const size_t varyLen) {
-    size_t arrayLen = 1;
-    size_t i;
-    for(i = 0; i < varyLen; i++) {
-        if(varyParams[i] == strt_ind) continue; /* strategy separated by table */
-        arrayLen *= rangetable[varyParams[i]];
-    }
-    return arrayLen;
-}
-
-/* returns unique index in memotable of compression parameters */
-static unsigned memoTableIndDirect(const paramValues_t* ptr, const varInds_t* varyParams, const size_t varyLen) {
-    size_t i;
-    unsigned ind = 0;
-    for(i = 0; i < varyLen; i++) {
-        varInds_t v = varyParams[i];
-        if(v == strt_ind) continue; /* exclude strategy from memotable */
-        ind *= rangetable[v]; ind += (unsigned)invRangeMap(v, ptr->vals[v]);
-    }
-    return ind;
-}
-
-static size_t memoTableGet(const memoTable_t* memoTableArray, const paramValues_t p) {
-    const memoTable_t mt = memoTableArray[p.vals[strt_ind]];
-    switch(mt.tableType) {
-        case directMap:
-            return mt.table[memoTableIndDirect(&p, mt.varArray, mt.varLen)];
-        case xxhashMap:
-            return mt.table[(XXH64(&p.vals, sizeof(U32) * NUM_PARAMS, 0) >> 3) % mt.tableLen];
-        case noMemo:
-            return 0;
-    }
-    return 0; /* should never happen, stop compiler warnings */
-}
-
-static void memoTableSet(const memoTable_t* memoTableArray, const paramValues_t p, const BYTE value) {
-    const memoTable_t mt = memoTableArray[p.vals[strt_ind]];
-    switch(mt.tableType) {
-        case directMap:
-            mt.table[memoTableIndDirect(&p, mt.varArray, mt.varLen)] = value; break;
-        case xxhashMap:
-            mt.table[(XXH64(&p.vals, sizeof(U32) * NUM_PARAMS, 0) >> 3) % mt.tableLen] = value; break;
-        case noMemo:
-            break;
-    }
-}
-
-/* frees all allocated memotables */
-/* secret contract :
- * mtAll is a table of (ZSTD_STRATEGY_MAX+1) memoTable_t */
-static void freeMemoTableArray(memoTable_t* const mtAll) {
-    int i;
-    if(mtAll == NULL) { return; }
-    for(i = 1; i <= (int)ZSTD_STRATEGY_MAX; i++) {
-        free(mtAll[i].table);
-    }
-    free(mtAll);
-}
-
-/* inits memotables for all (including mallocs), all strategies */
-/* takes unsanitized varyParams */
-static memoTable_t*
-createMemoTableArray(const paramValues_t p,
-                     const varInds_t* const varyParams,
-                     const size_t varyLen,
-                     const U32 memoTableLog)
-{
-    memoTable_t* const mtAll = (memoTable_t*)calloc(sizeof(memoTable_t),(ZSTD_STRATEGY_MAX + 1));
-    ZSTD_strategy i, stratMin = ZSTD_STRATEGY_MIN, stratMax = ZSTD_STRATEGY_MAX;
-
-    if(mtAll == NULL) {
-        return NULL;
-    }
-
-    for(i = 1; i <= (int)ZSTD_STRATEGY_MAX; i++) {
-        mtAll[i].varLen = sanitizeVarArray(mtAll[i].varArray, varyLen, varyParams, i);
-    }
-
-    /* no memoization */
-    if(memoTableLog == 0) {
-        for(i = 1; i <= (int)ZSTD_STRATEGY_MAX; i++) {
-            mtAll[i].tableType = noMemo;
-            mtAll[i].table = NULL;
-            mtAll[i].tableLen = 0;
-        }
-        return mtAll;
-    }
-
-
-    if(p.vals[strt_ind] != PARAM_UNSET) {
-        stratMin = p.vals[strt_ind];
-        stratMax = p.vals[strt_ind];
-    }
-
-
-    for(i = stratMin; i <= stratMax; i++) {
-        size_t mtl = memoTableLen(mtAll[i].varArray, mtAll[i].varLen);
-        mtAll[i].tableType = directMap;
-
-        if(memoTableLog != PARAM_UNSET && mtl > (1ULL << memoTableLog)) { /* use hash table */ /* provide some option to only use hash tables? */
-            mtAll[i].tableType = xxhashMap;
-            mtl = (1ULL << memoTableLog);
-        }
-
-        mtAll[i].table = (BYTE*)calloc(sizeof(BYTE), mtl);
-        mtAll[i].tableLen = mtl;
-
-        if(mtAll[i].table == NULL) {
-            freeMemoTableArray(mtAll);
-            return NULL;
-        }
-    }
-
-    return mtAll;
-}
-
-/* Sets pc to random unmeasured set of parameters */
-/* specify strategy */
-static void randomConstrainedParams(paramValues_t* pc, const memoTable_t* memoTableArray, const ZSTD_strategy st)
-{
-    size_t j;
-    const memoTable_t mt = memoTableArray[st];
-    pc->vals[strt_ind] = st;
-    for(j = 0; j < mt.tableLen; j++) {
-        int i;
-        for(i = 0; i < NUM_PARAMS; i++) {
-            varInds_t v = mt.varArray[i];
-            if(v == strt_ind) continue;
-            pc->vals[v] = rangeMap(v, FUZ_rand(&g_rand) % rangetable[v]);
-        }
-
-        if(!(memoTableGet(memoTableArray, *pc))) break; /* only pick unpicked params. */
-    }
-}
-
-/*-************************************
-*  Benchmarking Functions
-**************************************/
-
-static void display_params_tested(paramValues_t cParams)
-{
-    varInds_t vi;
-    DISPLAYLEVEL(3, "\r testing :");
-    for (vi=0; vi < NUM_PARAMS; vi++) {
-        DISPLAYLEVEL(3, "%3u,", (unsigned)cParams.vals[vi]);
-    }
-    DISPLAYLEVEL(3, "\b    \r");
-}
-
-/* Replicate functionality of benchMemAdvanced, but with pre-split src / dst buffers */
-/* The purpose is so that sufficient information is returned so that a decompression call to benchMemInvertible is possible */
-/* BMK_benchMemAdvanced(srcBuffer,srcSize, dstBuffer, dstSize, fileSizes, nbFiles, 0, &cParams, dictBuffer, dictSize, ctx, dctx, 0, "File", &adv); */
-/* nbSeconds used in same way as in BMK_advancedParams_t */
-/* if in decodeOnly, then srcPtr's will be compressed blocks, and uncompressedBlocks will be written to dstPtrs */
-/* dictionary nullable, nothing else though. */
-/* note : it would be a lot better if this function was present in benchzstd.c,
- * sharing code with benchMemAdvanced(), since it's technically a part of it */
-static BMK_benchOutcome_t
-BMK_benchMemInvertible( buffers_t buf, contexts_t ctx,
-                        int cLevel, const paramValues_t* comprParams,
-                        BMK_mode_t mode, unsigned nbSeconds)
-{
-    U32 i;
-    BMK_benchResult_t bResult;
-    const void *const *const srcPtrs = (const void *const *const)buf.srcPtrs;
-    size_t const *const srcSizes = buf.srcSizes;
-    void** const dstPtrs = buf.dstPtrs;
-    size_t const *const dstCapacities = buf.dstCapacities;
-    size_t* const dstSizes = buf.dstSizes;
-    void** const resPtrs = buf.resPtrs;
-    size_t const *const resSizes = buf.resSizes;
-    const void* dictBuffer = ctx.dictBuffer;
-    const size_t dictBufferSize = ctx.dictSize;
-    const size_t nbBlocks = buf.nbBlocks;
-    const size_t srcSize = buf.srcSize;
-    ZSTD_CCtx* cctx = ctx.cctx;
-    ZSTD_DCtx* dctx = ctx.dctx;
-
-    /* init */
-    display_params_tested(*comprParams);
-    memset(&bResult, 0, sizeof(bResult));
-
-    /* warming up memory */
-    for (i = 0; i < buf.nbBlocks; i++) {
-        if (mode != BMK_decodeOnly) {
-            RDG_genBuffer(dstPtrs[i], dstCapacities[i], 0.10, 0.50, 1);
-        } else {
-            RDG_genBuffer(resPtrs[i], resSizes[i], 0.10, 0.50, 1);
-        }
-    }
-
-    /* Bench */
-    {
-        /* init args */
-        int compressionCompleted = (mode == BMK_decodeOnly);
-        int decompressionCompleted = (mode == BMK_compressOnly);
-        BMK_timedFnState_t* timeStateCompress = BMK_createTimedFnState(nbSeconds * 1000, 1000);
-        BMK_timedFnState_t* timeStateDecompress = BMK_createTimedFnState(nbSeconds * 1000, 1000);
-        BMK_benchParams_t cbp, dbp;
-        BMK_initCCtxArgs cctxprep;
-        BMK_initDCtxArgs dctxprep;
-
-        cbp.benchFn = local_defaultCompress;
-        cbp.benchPayload = cctx;
-        cbp.initFn = local_initCCtx;
-        cbp.initPayload = &cctxprep;
-        cbp.errorFn = ZSTD_isError;
-        cbp.blockCount = nbBlocks;
-        cbp.srcBuffers = srcPtrs;
-        cbp.srcSizes = srcSizes;
-        cbp.dstBuffers = dstPtrs;
-        cbp.dstCapacities = dstCapacities;
-        cbp.blockResults = dstSizes;
-
-        cctxprep.cctx = cctx;
-        cctxprep.dictBuffer = dictBuffer;
-        cctxprep.dictBufferSize = dictBufferSize;
-        cctxprep.cLevel = cLevel;
-        cctxprep.comprParams = comprParams;
-
-        dbp.benchFn = local_defaultDecompress;
-        dbp.benchPayload = dctx;
-        dbp.initFn = local_initDCtx;
-        dbp.initPayload = &dctxprep;
-        dbp.errorFn = ZSTD_isError;
-        dbp.blockCount = nbBlocks;
-        dbp.srcBuffers = (const void* const *) dstPtrs;
-        dbp.srcSizes = dstCapacities;
-        dbp.dstBuffers = resPtrs;
-        dbp.dstCapacities = resSizes;
-        dbp.blockResults = NULL;
-
-        dctxprep.dctx = dctx;
-        dctxprep.dictBuffer = dictBuffer;
-        dctxprep.dictBufferSize = dictBufferSize;
-
-        assert(timeStateCompress != NULL);
-        assert(timeStateDecompress != NULL);
-        while(!compressionCompleted) {
-            BMK_runOutcome_t const cOutcome = BMK_benchTimedFn(timeStateCompress, cbp);
-
-            if (!BMK_isSuccessful_runOutcome(cOutcome)) {
-                BMK_benchOutcome_t bOut;
-                memset(&bOut, 0, sizeof(bOut));
-                bOut.tag = 1;   /* should rather be a function or a constant */
-                BMK_freeTimedFnState(timeStateCompress);
-                BMK_freeTimedFnState(timeStateDecompress);
-                return bOut;
-            }
-            {   BMK_runTime_t const rResult = BMK_extract_runTime(cOutcome);
-                bResult.cSpeed = (unsigned long long)((double)srcSize * TIMELOOP_NANOSEC / rResult.nanoSecPerRun);
-                bResult.cSize = rResult.sumOfReturn;
-            }
-            compressionCompleted = BMK_isCompleted_TimedFn(timeStateCompress);
-        }
-
-        while (!decompressionCompleted) {
-            BMK_runOutcome_t const dOutcome = BMK_benchTimedFn(timeStateDecompress, dbp);
-
-            if (!BMK_isSuccessful_runOutcome(dOutcome)) {
-                BMK_benchOutcome_t bOut;
-                memset(&bOut, 0, sizeof(bOut));
-                bOut.tag = 1;   /* should rather be a function or a constant */
-                BMK_freeTimedFnState(timeStateCompress);
-                BMK_freeTimedFnState(timeStateDecompress);
-                return bOut;
-            }
-            {   BMK_runTime_t const rResult = BMK_extract_runTime(dOutcome);
-                bResult.dSpeed = (unsigned long long)((double)srcSize * TIMELOOP_NANOSEC / rResult.nanoSecPerRun);
-            }
-            decompressionCompleted = BMK_isCompleted_TimedFn(timeStateDecompress);
-        }
-
-        BMK_freeTimedFnState(timeStateCompress);
-        BMK_freeTimedFnState(timeStateDecompress);
-    }
-
-   /* Bench */
-    bResult.cMem = (1 << (comprParams->vals[wlog_ind])) + ZSTD_sizeof_CCtx(cctx);
-
-    {   BMK_benchOutcome_t bOut;
-        bOut.tag = 0;
-        bOut.internal_never_use_directly = bResult;  /* should be a function */
-        return bOut;
-    }
-}
-
-/* BMK_benchParam() :
- * benchmark a set of `cParams` over sample `buf`,
- * store the result in `resultPtr`.
- * @return : 0 if success, 1 if error */
-static int BMK_benchParam ( BMK_benchResult_t* resultPtr,
-                            buffers_t buf, contexts_t ctx,
-                            paramValues_t cParams)
-{
-    BMK_benchOutcome_t const outcome = BMK_benchMemInvertible(buf, ctx,
-                                                        BASE_CLEVEL, &cParams,
-                                                        BMK_both, 3);
-    if (!BMK_isSuccessful_benchOutcome(outcome)) return 1;
-    *resultPtr = BMK_extract_benchResult(outcome);
-    return 0;
-}
-
-
-/* Benchmarking which stops when we are sufficiently sure the solution is infeasible / worse than the winner */
-#define VARIANCE 1.2
-static int allBench(BMK_benchResult_t* resultPtr,
-                const buffers_t buf, const contexts_t ctx,
-                const paramValues_t cParams,
-                const constraint_t target,
-                BMK_benchResult_t* winnerResult, int feas)
-{
-    BMK_benchResult_t benchres;
-    double uncertaintyConstantC = 3., uncertaintyConstantD = 3.;
-    double winnerRS;
-
-    BMK_benchOutcome_t const outcome = BMK_benchMemInvertible(buf, ctx, BASE_CLEVEL, &cParams, BMK_both, 2);
-    if (!BMK_isSuccessful_benchOutcome(outcome)) {
-        DEBUGOUTPUT("Benchmarking failed \n");
-        return ERROR_RESULT;
-    }
-    benchres = BMK_extract_benchResult(outcome);
-
-    winnerRS = resultScore(*winnerResult, buf.srcSize, target);
-    DEBUGOUTPUT("WinnerScore: %f \n ", winnerRS);
-
-    *resultPtr = benchres;
-
-    /* anything with worse ratio in feas is definitely worse, discard */
-    if(feas && benchres.cSize < winnerResult->cSize && !g_optmode) {
-        return WORSE_RESULT;
-    }
-
-    /* calculate uncertainty in compression / decompression runs */
-    if (benchres.cSpeed) {
-        U64 const loopDurationC = (((U64)buf.srcSize * TIMELOOP_NANOSEC) / benchres.cSpeed);
-        uncertaintyConstantC = ((loopDurationC + (double)(2 * g_clockGranularity))/loopDurationC);
-    }
-
-    if (benchres.dSpeed) {
-        U64 const loopDurationD = (((U64)buf.srcSize * TIMELOOP_NANOSEC) / benchres.dSpeed);
-        uncertaintyConstantD = ((loopDurationD + (double)(2 * g_clockGranularity))/loopDurationD);
-    }
-
-    /* optimistic assumption of benchres */
-    {   BMK_benchResult_t resultMax = benchres;
-        resultMax.cSpeed = (unsigned long long)(resultMax.cSpeed * uncertaintyConstantC * VARIANCE);
-        resultMax.dSpeed = (unsigned long long)(resultMax.dSpeed * uncertaintyConstantD * VARIANCE);
-
-        /* disregard infeasible results in feas mode */
-        /* disregard if resultMax < winner in infeas mode */
-        if((feas && !feasible(resultMax, target)) ||
-          (!feas && (winnerRS > resultScore(resultMax, buf.srcSize, target)))) {
-            return WORSE_RESULT;
-        }
-    }
-
-    /* compare by resultScore when in infeas */
-    /* compare by compareResultLT when in feas */
-    if((!feas && (resultScore(benchres, buf.srcSize, target) > resultScore(*winnerResult, buf.srcSize, target))) ||
-       (feas && (compareResultLT(*winnerResult, benchres, target, buf.srcSize))) )  {
-        return BETTER_RESULT;
-    } else {
-        return WORSE_RESULT;
-    }
-}
-
-
-#define INFEASIBLE_THRESHOLD 200
-/* Memoized benchmarking, won't benchmark anything which has already been benchmarked before. */
-static int benchMemo(BMK_benchResult_t* resultPtr,
-                const buffers_t buf, const contexts_t ctx,
-                const paramValues_t cParams,
-                const constraint_t target,
-                BMK_benchResult_t* winnerResult, memoTable_t* const memoTableArray,
-                const int feas) {
-    static int bmcount = 0;
-    int res;
-
-    if ( memoTableGet(memoTableArray, cParams) >= INFEASIBLE_THRESHOLD
-      || redundantParams(cParams, target, buf.maxBlockSize) ) {
-        return WORSE_RESULT;
-    }
-
-    res = allBench(resultPtr, buf, ctx, cParams, target, winnerResult, feas);
-
-    if(DEBUG && !(bmcount % 250)) {
-        DISPLAY("Count: %d\n", bmcount);
-        bmcount++;
-    }
-    BMK_printWinnerOpt(stdout, CUSTOM_LEVEL, *resultPtr, cParams, target, buf.srcSize);
-
-    if(res == BETTER_RESULT || feas) {
-        memoTableSet(memoTableArray, cParams, 255); /* what happens if collisions are frequent */
-    }
-    return res;
-}
-
-
-typedef struct {
-    U64 cSpeed_min;
-    U64 dSpeed_min;
-    U32 windowLog_max;
-    ZSTD_strategy strategy_max;
-} level_constraints_t;
-
-static level_constraints_t g_level_constraint[NB_LEVELS_TRACKED+1];
-
-static void BMK_init_level_constraints(int bytePerSec_level1)
-{
-    assert(NB_LEVELS_TRACKED >= ZSTD_maxCLevel());
-    memset(g_level_constraint, 0, sizeof(g_level_constraint));
-    g_level_constraint[1].cSpeed_min = bytePerSec_level1;
-    g_level_constraint[1].dSpeed_min = 0.;
-    g_level_constraint[1].windowLog_max = 19;
-    g_level_constraint[1].strategy_max = ZSTD_fast;
-
-    /* establish speed objectives (relative to level 1) */
-    {   int l;
-        for (l=2; l<=NB_LEVELS_TRACKED; l++) {
-            g_level_constraint[l].cSpeed_min = (g_level_constraint[l-1].cSpeed_min * 49) / 64;
-            g_level_constraint[l].dSpeed_min = 0.;
-            g_level_constraint[l].windowLog_max = (l<20) ? 23 : l+5;   /* only --ultra levels >= 20 can use windowlog > 23 */
-            g_level_constraint[l].strategy_max = ZSTD_STRATEGY_MAX;
-    }   }
-}
-
-static int BMK_seed(winnerInfo_t* winners,
-                    const paramValues_t params,
-                    const buffers_t buf,
-                    const contexts_t ctx)
-{
-    BMK_benchResult_t testResult;
-    int better = 0;
-    int cLevel;
-
-    BMK_benchParam(&testResult, buf, ctx, params);
-
-    for (cLevel = 1; cLevel <= NB_LEVELS_TRACKED; cLevel++) {
-
-        if (testResult.cSpeed < g_level_constraint[cLevel].cSpeed_min)
-            continue;   /* not fast enough for this level */
-        if (testResult.dSpeed < g_level_constraint[cLevel].dSpeed_min)
-            continue;   /* not fast enough for this level */
-        if (params.vals[wlog_ind] > g_level_constraint[cLevel].windowLog_max)
-            continue;   /* too much memory for this level */
-        if (params.vals[strt_ind] > g_level_constraint[cLevel].strategy_max)
-            continue;   /* forbidden strategy for this level */
-        if (winners[cLevel].result.cSize==0) {
-            /* first solution for this cLevel */
-            winners[cLevel].result = testResult;
-            winners[cLevel].params = params;
-            BMK_print_cLevelEntry(stdout, cLevel, params, testResult, buf.srcSize);
-            better = 1;
-            continue;
-        }
-
-        if ((double)testResult.cSize <= ((double)winners[cLevel].result.cSize * (1. + (0.02 / cLevel))) ) {
-            /* Validate solution is "good enough" */
-            double W_ratio = (double)buf.srcSize / testResult.cSize;
-            double O_ratio = (double)buf.srcSize / winners[cLevel].result.cSize;
-            double W_ratioNote = log (W_ratio);
-            double O_ratioNote = log (O_ratio);
-            size_t W_DMemUsed = (1 << params.vals[wlog_ind]) + (16 KB);
-            size_t O_DMemUsed = (1 << winners[cLevel].params.vals[wlog_ind]) + (16 KB);
-            double W_DMemUsed_note = W_ratioNote * ( 40 + 9*cLevel) - log((double)W_DMemUsed);
-            double O_DMemUsed_note = O_ratioNote * ( 40 + 9*cLevel) - log((double)O_DMemUsed);
-
-            size_t W_CMemUsed = (1 << params.vals[wlog_ind]) + ZSTD_estimateCCtxSize_usingCParams(pvalsToCParams(params));
-            size_t O_CMemUsed = (1 << winners[cLevel].params.vals[wlog_ind]) + ZSTD_estimateCCtxSize_usingCParams(pvalsToCParams(winners[cLevel].params));
-            double W_CMemUsed_note = W_ratioNote * ( 50 + 13*cLevel) - log((double)W_CMemUsed);
-            double O_CMemUsed_note = O_ratioNote * ( 50 + 13*cLevel) - log((double)O_CMemUsed);
-
-            double W_CSpeed_note = W_ratioNote * ( 30 + 10*cLevel) + log(testResult.cSpeed);
-            double O_CSpeed_note = O_ratioNote * ( 30 + 10*cLevel) + log(winners[cLevel].result.cSpeed);
-
-            double W_DSpeed_note = W_ratioNote * ( 20 + 2*cLevel) + log(testResult.dSpeed);
-            double O_DSpeed_note = O_ratioNote * ( 20 + 2*cLevel) + log(winners[cLevel].result.dSpeed);
-
-            if (W_DMemUsed_note < O_DMemUsed_note) {
-                /* uses too much Decompression memory for too little benefit */
-                if (W_ratio > O_ratio)
-                DISPLAYLEVEL(3, "Decompression Memory : %5.3f @ %4.1f MB  vs  %5.3f @ %4.1f MB   : not enough for level %i\n",
-                         W_ratio, (double)(W_DMemUsed) / 1024 / 1024,
-                         O_ratio, (double)(O_DMemUsed) / 1024 / 1024,   cLevel);
-                continue;
-            }
-            if (W_CMemUsed_note < O_CMemUsed_note) {
-                /* uses too much memory for compression for too little benefit */
-                if (W_ratio > O_ratio)
-                DISPLAYLEVEL(3, "Compression Memory : %5.3f @ %4.1f MB  vs  %5.3f @ %4.1f MB   : not enough for level %i\n",
-                         W_ratio, (double)(W_CMemUsed) / 1024 / 1024,
-                         O_ratio, (double)(O_CMemUsed) / 1024 / 1024,
-                         cLevel);
-                continue;
-            }
-            if (W_CSpeed_note   < O_CSpeed_note  ) {
-                /* too large compression speed difference for the compression benefit */
-                if (W_ratio > O_ratio)
-                DISPLAYLEVEL(3, "Compression Speed : %5.3f @ %4.1f MB/s  vs  %5.3f @ %4.1f MB/s   : not enough for level %i\n",
-                         W_ratio, (double)testResult.cSpeed / MB_UNIT,
-                         O_ratio, (double)winners[cLevel].result.cSpeed / MB_UNIT,
-                         cLevel);
-                continue;
-            }
-            if (W_DSpeed_note   < O_DSpeed_note  ) {
-                /* too large decompression speed difference for the compression benefit */
-                if (W_ratio > O_ratio)
-                DISPLAYLEVEL(3, "Decompression Speed : %5.3f @ %4.1f MB/s  vs  %5.3f @ %4.1f MB/s   : not enough for level %i\n",
-                         W_ratio, (double)testResult.dSpeed / MB_UNIT,
-                         O_ratio, (double)winners[cLevel].result.dSpeed / MB_UNIT,
-                         cLevel);
-                continue;
-            }
-
-            if (W_ratio < O_ratio)
-                DISPLAYLEVEL(3, "Solution %4.3f selected over %4.3f at level %i, due to better secondary statistics \n",
-                                W_ratio, O_ratio, cLevel);
-
-            winners[cLevel].result = testResult;
-            winners[cLevel].params = params;
-            BMK_print_cLevelEntry(stdout, cLevel, params, testResult, buf.srcSize);
-
-            better = 1;
-    }   }
-
-    return better;
-}
-
-/*-************************************
-*  Compression Level Table Generation Functions
-**************************************/
-
-#define PARAMTABLELOG   25
-#define PARAMTABLESIZE (1<<PARAMTABLELOG)
-#define PARAMTABLEMASK (PARAMTABLESIZE-1)
-static BYTE g_alreadyTested[PARAMTABLESIZE] = {0};   /* init to zero */
-
-static BYTE* NB_TESTS_PLAYED(paramValues_t p)
-{
-    ZSTD_compressionParameters const cParams = pvalsToCParams(sanitizeParams(p));
-    unsigned long long const h64 = XXH64(&cParams, sizeof(cParams), 0);
-    return &g_alreadyTested[(h64 >> 3) & PARAMTABLEMASK];
-}
-
-static void playAround(FILE* f,
-                       winnerInfo_t* winners,
-                       paramValues_t p,
-                       const buffers_t buf, const contexts_t ctx)
-{
-    int nbVariations = 0;
-    UTIL_time_t const clockStart = UTIL_getTime();
-
-    while (UTIL_clockSpanMicro(clockStart) < g_maxVariationTime) {
-        if (nbVariations++ > g_maxNbVariations) break;
-
-        do {
-            int i;
-            for(i = 0; i < 4; i++) {
-                paramVaryOnce(FUZ_rand(&g_rand) % (strt_ind + 1),
-                              ((FUZ_rand(&g_rand) & 1) << 1) - 1,
-                              &p);
-            }
-        } while (!paramValid(p));
-
-        /* exclude faster if already played params */
-        if (FUZ_rand(&g_rand) & ((1 << *NB_TESTS_PLAYED(p))-1))
-            continue;
-
-        /* test */
-        {   BYTE* const b = NB_TESTS_PLAYED(p);
-            (*b)++;
-        }
-        if (!BMK_seed(winners, p, buf, ctx)) continue;
-
-        /* improvement found => search more */
-        BMK_saveAndPrint_cLevelTable(f, winners, buf.srcSize);
-        playAround(f, winners, p, buf, ctx);
-    }
-
-}
-
-static void
-BMK_selectRandomStart( FILE* f,
-                       winnerInfo_t* winners,
-                       const buffers_t buf, const contexts_t ctx)
-{
-    U32 const id = FUZ_rand(&g_rand) % (NB_LEVELS_TRACKED+1);
-    if ((id==0) || (winners[id].params.vals[wlog_ind]==0)) {
-        /* use some random entry */
-        paramValues_t const p = adjustParams(cParamsToPVals(pvalsToCParams(randomParams())), /* defaults nonCompression parameters */
-                                             buf.srcSize, 0);
-        playAround(f, winners, p, buf, ctx);
-    } else {
-        playAround(f, winners, winners[id].params, buf, ctx);
-    }
-}
-
-
-/* BMK_generate_cLevelTable() :
- * test a large number of configurations
- * and distribute them across compression levels according to speed conditions.
- * display and save all intermediate results into rfName = "grillResults.txt".
- * the function automatically stops after g_timeLimit_s.
- * this function cannot error, it directly exit() in case of problem.
- */
-static void BMK_generate_cLevelTable(const buffers_t buf, const contexts_t ctx)
-{
-    paramValues_t params;
-    winnerInfo_t winners[NB_LEVELS_TRACKED+1];
-    const char* const rfName = "grillResults.txt";
-    FILE* const f = fopen(rfName, "w");
-
-    /* init */
-    assert(g_singleRun==0);
-    memset(winners, 0, sizeof(winners));
-    if (f==NULL) { DISPLAY("error opening %s \n", rfName); exit(1); }
-
-    if (g_target) {
-        BMK_init_level_constraints(g_target * MB_UNIT);
-    } else {
-        /* baseline config for level 1 */
-        paramValues_t const l1params = cParamsToPVals(ZSTD_getCParams(1, buf.maxBlockSize, ctx.dictSize));
-        BMK_benchResult_t testResult;
-        BMK_benchParam(&testResult, buf, ctx, l1params);
-        BMK_init_level_constraints((int)((testResult.cSpeed * 31) / 32));
-    }
-
-    /* populate initial solution */
-    {   const int maxSeeds = g_noSeed ? 1 : ZSTD_maxCLevel();
-        int i;
-        for (i=0; i<=maxSeeds; i++) {
-            params = cParamsToPVals(ZSTD_getCParams(i, buf.maxBlockSize, 0));
-            BMK_seed(winners, params, buf, ctx);
-    }   }
-    BMK_saveAndPrint_cLevelTable(f, winners, buf.srcSize);
-
-    /* start tests */
-    {   const UTIL_time_t grillStart = UTIL_getTime();
-        do {
-            BMK_selectRandomStart(f, winners, buf, ctx);
-        } while (BMK_timeSpan_s(grillStart) < g_timeLimit_s);
-    }
-
-    /* end summary */
-    BMK_saveAndPrint_cLevelTable(f, winners, buf.srcSize);
-    DISPLAY("grillParams operations completed \n");
-
-    /* clean up*/
-    fclose(f);
-}
-
-
-/*-************************************
-*  Single Benchmark Functions
-**************************************/
-
-static int
-benchOnce(const buffers_t buf, const contexts_t ctx, const int cLevel)
-{
-    BMK_benchResult_t testResult;
-    g_params = adjustParams(overwriteParams(cParamsToPVals(ZSTD_getCParams(cLevel, buf.maxBlockSize, ctx.dictSize)), g_params), buf.maxBlockSize, ctx.dictSize);
-
-    if (BMK_benchParam(&testResult, buf, ctx, g_params)) {
-        DISPLAY("Error during benchmarking\n");
-        return 1;
-    }
-
-    BMK_printWinner(stdout, CUSTOM_LEVEL, testResult, g_params, buf.srcSize);
-
-    return 0;
-}
-
-static int benchSample(double compressibility, int cLevel)
-{
-    const char* const name = "Sample 10MB";
-    size_t const benchedSize = 10 MB;
-    void* const srcBuffer = malloc(benchedSize);
-    int ret = 0;
-
-    buffers_t buf;
-    contexts_t ctx;
-
-    if(srcBuffer == NULL) {
-        DISPLAY("Out of Memory\n");
-        return 2;
-    }
-
-    RDG_genBuffer(srcBuffer, benchedSize, compressibility, 0.0, 0);
-
-    if(createBuffersFromMemory(&buf, srcBuffer, 1, &benchedSize)) {
-        DISPLAY("Buffer Creation Error\n");
-        free(srcBuffer);
-        return 3;
-    }
-
-    if(createContexts(&ctx, NULL)) {
-        DISPLAY("Context Creation Error\n");
-        freeBuffers(buf);
-        return 1;
-    }
-
-    /* bench */
-    DISPLAY("\r%79s\r", "");
-    DISPLAY("using %s %i%%: \n", name, (int)(compressibility*100));
-
-    if(g_singleRun) {
-        ret = benchOnce(buf, ctx, cLevel);
-    } else {
-        BMK_generate_cLevelTable(buf, ctx);
-    }
-
-    freeBuffers(buf);
-    freeContexts(ctx);
-
-    return ret;
-}
-
-/* benchFiles() :
- * note: while this function takes a table of filenames,
- * in practice, only the first filename will be used */
-static int benchFiles(const char** fileNamesTable, int nbFiles,
-                      const char* dictFileName, int cLevel)
-{
-    buffers_t buf;
-    contexts_t ctx;
-    int ret = 0;
-
-    if (createBuffers(&buf, fileNamesTable, nbFiles)) {
-        DISPLAY("unable to load files\n");
-        return 1;
-    }
-
-    if (createContexts(&ctx, dictFileName)) {
-        DISPLAY("unable to load dictionary\n");
-        freeBuffers(buf);
-        return 2;
-    }
-
-    DISPLAY("\r%79s\r", "");
-    if (nbFiles == 1) {
-        DISPLAY("using %s : \n", fileNamesTable[0]);
-    } else {
-        DISPLAY("using %d Files : \n", nbFiles);
-    }
-
-    if (g_singleRun) {
-        ret = benchOnce(buf, ctx, cLevel);
-    } else {
-        BMK_generate_cLevelTable(buf, ctx);
-    }
-
-    freeBuffers(buf);
-    freeContexts(ctx);
-    return ret;
-}
-
-
-/*-************************************
-*  Local Optimization Functions
-**************************************/
-
-/* One iteration of hill climbing. Specifically, it first tries all
- * valid parameter configurations w/ manhattan distance 1 and picks the best one
- * failing that, it progressively tries candidates further and further away (up to #dim + 2)
- * if it finds a candidate exceeding winnerInfo, it will repeat. Otherwise, it will stop the
- * current stage of hill climbing.
- * Each iteration of hill climbing proceeds in 2 'phases'. Phase 1 climbs according to
- * the resultScore function, which is effectively a linear increase in reward until it reaches
- * the constraint-satisfying value, it which point any excess results in only logarithmic reward.
- * This aims to find some constraint-satisfying point.
- * Phase 2 optimizes in accordance with what the original function sets out to maximize, with
- * all feasible solutions valued over all infeasible solutions.
- */
-
-/* sanitize all params here.
- * all generation after random should be sanitized. (maybe sanitize random)
- */
-static winnerInfo_t climbOnce(const constraint_t target,
-                memoTable_t* mtAll,
-                const buffers_t buf, const contexts_t ctx,
-                const paramValues_t init)
-{
-    /*
-     * cparam - currently considered 'center'
-     * candidate - params to benchmark/results
-     * winner - best option found so far.
-     */
-    paramValues_t cparam = init;
-    winnerInfo_t candidateInfo, winnerInfo;
-    int better = 1;
-    int feas = 0;
-
-    winnerInfo = initWinnerInfo(init);
-    candidateInfo = winnerInfo;
-
-    {   winnerInfo_t bestFeasible1 = initWinnerInfo(cparam);
-        DEBUGOUTPUT("Climb Part 1\n");
-        while(better) {
-            int offset;
-            size_t i, dist;
-            const size_t varLen = mtAll[cparam.vals[strt_ind]].varLen;
-            better = 0;
-            DEBUGOUTPUT("Start\n");
-            cparam = winnerInfo.params;
-            candidateInfo.params = cparam;
-             /* all dist-1 candidates */
-            for (i = 0; i < varLen; i++) {
-                for (offset = -1; offset <= 1; offset += 2) {
-                    CHECKTIME(winnerInfo);
-                    candidateInfo.params = cparam;
-                    paramVaryOnce(mtAll[cparam.vals[strt_ind]].varArray[i],
-                                  offset,
-                                  &candidateInfo.params);
-
-                    if(paramValid(candidateInfo.params)) {
-                        int res;
-                        res = benchMemo(&candidateInfo.result, buf, ctx,
-                            sanitizeParams(candidateInfo.params), target, &winnerInfo.result, mtAll, feas);
-                        DEBUGOUTPUT("Res: %d\n", res);
-                        if(res == BETTER_RESULT) { /* synonymous with better when called w/ infeasibleBM */
-                            winnerInfo = candidateInfo;
-                            better = 1;
-                            if(compareResultLT(bestFeasible1.result, winnerInfo.result, target, buf.srcSize)) {
-                                bestFeasible1 = winnerInfo;
-                            }
-                        }
-                    }
-                }  /* for (offset = -1; offset <= 1; offset += 2) */
-            }   /* for (i = 0; i < varLen; i++) */
-
-            if(better) {
-                continue;
-            }
-
-            for (dist = 2; dist < varLen + 2; dist++) { /* varLen is # dimensions */
-                for (i = 0; i < (1 << varLen) / varLen + 2; i++) {
-                    int res;
-                    CHECKTIME(winnerInfo);
-                    candidateInfo.params = cparam;
-                    /* param error checking already done here */
-                    paramVariation(&candidateInfo.params, mtAll, (U32)dist);
-
-                    res = benchMemo(&candidateInfo.result,
-                                buf, ctx,
-                                sanitizeParams(candidateInfo.params), target,
-                                &winnerInfo.result, mtAll, feas);
-                    DEBUGOUTPUT("Res: %d\n", res);
-                    if (res == BETTER_RESULT) { /* synonymous with better in this case*/
-                        winnerInfo = candidateInfo;
-                        better = 1;
-                        if (compareResultLT(bestFeasible1.result, winnerInfo.result, target, buf.srcSize)) {
-                            bestFeasible1 = winnerInfo;
-                        }
-                        break;
-                    }
-                }
-
-                if (better) {
-                    break;
-                }
-            }   /* for(dist = 2; dist < varLen + 2; dist++) */
-
-            if (!better) { /* infeas -> feas -> stop */
-                if (feas) return winnerInfo;
-                feas = 1;
-                better = 1;
-                winnerInfo = bestFeasible1; /* note with change, bestFeasible may not necessarily be feasible, but if one has been benchmarked, it will be. */
-                DEBUGOUTPUT("Climb Part 2\n");
-            }
-        }
-        winnerInfo = bestFeasible1;
-    }
-
-    return winnerInfo;
-}
-
-/* Optimizes for a fixed strategy */
-
-/* flexible parameters: iterations of failed climbing (or if we do non-random, maybe this is when everything is close to visited)
-   weight more on visit for bad results, less on good results/more on later results / ones with more failures.
-   allocate memoTable here.
- */
-static winnerInfo_t
-optimizeFixedStrategy(const buffers_t buf, const contexts_t ctx,
-                      const constraint_t target, paramValues_t paramTarget,
-                      const ZSTD_strategy strat,
-                      memoTable_t* memoTableArray, const int tries)
-{
-    int i = 0;
-
-    paramValues_t init;
-    winnerInfo_t winnerInfo, candidateInfo;
-    winnerInfo = initWinnerInfo(emptyParams());
-    /* so climb is given the right fixed strategy */
-    paramTarget.vals[strt_ind] = strat;
-    /* to pass ZSTD_checkCParams */
-    paramTarget = cParamUnsetMin(paramTarget);
-
-    init = paramTarget;
-
-    for(i = 0; i < tries; i++) {
-        DEBUGOUTPUT("Restart\n");
-        do {
-            randomConstrainedParams(&init, memoTableArray, strat);
-        } while(redundantParams(init, target, buf.maxBlockSize));
-        candidateInfo = climbOnce(target, memoTableArray, buf, ctx, init);
-        if (compareResultLT(winnerInfo.result, candidateInfo.result, target, buf.srcSize)) {
-            winnerInfo = candidateInfo;
-            BMK_printWinnerOpt(stdout, CUSTOM_LEVEL, winnerInfo.result, winnerInfo.params, target, buf.srcSize);
-            i = 0;
-            continue;
-        }
-        CHECKTIME(winnerInfo);
-        i++;
-    }
-    return winnerInfo;
-}
-
-/* goes best, best-1, best+1, best-2, ... */
-/* return 0 if nothing remaining */
-static int nextStrategy(const int currentStrategy, const int bestStrategy)
-{
-    if(bestStrategy <= currentStrategy) {
-        int candidate = 2 * bestStrategy - currentStrategy - 1;
-        if(candidate < 1) {
-            candidate = currentStrategy + 1;
-            if(candidate > (int)ZSTD_STRATEGY_MAX) {
-                return 0;
-            } else {
-                return candidate;
-            }
-        } else {
-            return candidate;
-        }
-    } else { /* bestStrategy >= currentStrategy */
-        int candidate = 2 * bestStrategy - currentStrategy;
-        if(candidate > (int)ZSTD_STRATEGY_MAX) {
-            candidate = currentStrategy - 1;
-            if(candidate < 1) {
-                return 0;
-            } else {
-                return candidate;
-            }
-        } else {
-            return candidate;
-        }
-    }
-}
-
-/* experiment with playing with this and decay value */
-
-/* main fn called when using --optimize */
-/* Does strategy selection by benchmarking default compression levels
- * then optimizes by strategy, starting with the best one and moving
- * progressively moving further away by number
- * args:
- * fileNamesTable - list of files to benchmark
- * nbFiles - length of fileNamesTable
- * dictFileName - name of dictionary file if one, else NULL
- * target - performance constraints (cSpeed, dSpeed, cMem)
- * paramTarget - parameter constraints (i.e. restriction search space to where strategy = ZSTD_fast)
- * cLevel - compression level to exceed (all solutions must be > lvl in cSpeed + ratio)
- */
-
-static unsigned g_maxTries = 5;
-#define TRY_DECAY 1
-
-static int
-optimizeForSize(const char* const * const fileNamesTable, const size_t nbFiles,
-                const char* dictFileName,
-                constraint_t target, paramValues_t paramTarget,
-                const int cLevelOpt, const int cLevelRun,
-                const U32 memoTableLog)
-{
-    varInds_t varArray [NUM_PARAMS];
-    int ret = 0;
-    const size_t varLen = variableParams(paramTarget, varArray, dictFileName != NULL);
-    winnerInfo_t winner = initWinnerInfo(emptyParams());
-    memoTable_t* allMT = NULL;
-    paramValues_t paramBase;
-    contexts_t ctx;
-    buffers_t buf;
-    g_time = UTIL_getTime();
-
-    if (createBuffers(&buf, fileNamesTable, nbFiles)) {
-        DISPLAY("unable to load files\n");
-        return 1;
-    }
-
-    if (createContexts(&ctx, dictFileName)) {
-        DISPLAY("unable to load dictionary\n");
-        freeBuffers(buf);
-        return 2;
-    }
-
-    if (nbFiles == 1) {
-        DISPLAYLEVEL(2, "Loading %s...       \r", fileNamesTable[0]);
-    } else {
-        DISPLAYLEVEL(2, "Loading %lu Files...       \r", (unsigned long)nbFiles);
-    }
-
-    /* sanitize paramTarget */
-    optimizerAdjustInput(&paramTarget, buf.maxBlockSize);
-    paramBase = cParamUnsetMin(paramTarget);
-
-    allMT = createMemoTableArray(paramTarget, varArray, varLen, memoTableLog);
-
-    if (!allMT) {
-        DISPLAY("MemoTable Init Error\n");
-        ret = 2;
-        goto _cleanUp;
-    }
-
-    /* default strictnesses */
-    if (g_strictness == PARAM_UNSET) {
-        if(g_optmode) {
-            g_strictness = 100;
-        } else {
-            g_strictness = 90;
-        }
-    } else {
-        if(0 >= g_strictness || g_strictness > 100) {
-            DISPLAY("Strictness Outside of Bounds\n");
-            ret = 4;
-            goto _cleanUp;
-        }
-    }
-
-    /* use level'ing mode instead of normal target mode */
-    if (g_optmode) {
-        winner.params = cParamsToPVals(ZSTD_getCParams(cLevelOpt, buf.maxBlockSize, ctx.dictSize));
-        if(BMK_benchParam(&winner.result, buf, ctx, winner.params)) {
-            ret = 3;
-            goto _cleanUp;
-        }
-
-        g_lvltarget = winner.result;
-        g_lvltarget.cSpeed = (g_lvltarget.cSpeed * g_strictness) / 100;
-        g_lvltarget.dSpeed = (g_lvltarget.dSpeed * g_strictness) / 100;
-        g_lvltarget.cSize = (g_lvltarget.cSize * 100) / g_strictness;
-
-        target.cSpeed = (U32)g_lvltarget.cSpeed;
-        target.dSpeed = (U32)g_lvltarget.dSpeed;
-
-        BMK_printWinnerOpt(stdout, cLevelOpt, winner.result, winner.params, target, buf.srcSize);
-    }
-
-    /* Don't want it to return anything worse than the best known result */
-    if (g_singleRun) {
-        BMK_benchResult_t res;
-        g_params = adjustParams(overwriteParams(cParamsToPVals(ZSTD_getCParams(cLevelRun, buf.maxBlockSize, ctx.dictSize)), g_params), buf.maxBlockSize, ctx.dictSize);
-        if (BMK_benchParam(&res, buf, ctx, g_params)) {
-            ret = 45;
-            goto _cleanUp;
-        }
-        if(compareResultLT(winner.result, res, relaxTarget(target), buf.srcSize)) {
-            winner.result = res;
-            winner.params = g_params;
-        }
-    }
-
-    /* bench */
-    DISPLAYLEVEL(2, "\r%79s\r", "");
-    if(nbFiles == 1) {
-        DISPLAYLEVEL(2, "optimizing for %s", fileNamesTable[0]);
-    } else {
-        DISPLAYLEVEL(2, "optimizing for %lu Files", (unsigned long)nbFiles);
-    }
-
-    if(target.cSpeed != 0) { DISPLAYLEVEL(2," - limit compression speed %u MB/s", (unsigned)(target.cSpeed >> 20)); }
-    if(target.dSpeed != 0) { DISPLAYLEVEL(2, " - limit decompression speed %u MB/s", (unsigned)(target.dSpeed >> 20)); }
-    if(target.cMem != (U32)-1) { DISPLAYLEVEL(2, " - limit memory %u MB", (unsigned)(target.cMem >> 20)); }
-
-    DISPLAYLEVEL(2, "\n");
-    init_clockGranularity();
-
-    {   paramValues_t CParams;
-
-        /* find best solution from default params */
-        {   const int maxSeeds = g_noSeed ? 1 : ZSTD_maxCLevel();
-            DEBUGOUTPUT("Strategy Selection\n");
-            if (paramTarget.vals[strt_ind] == PARAM_UNSET) {
-                BMK_benchResult_t candidate;
-                int i;
-                for (i=1; i<=maxSeeds; i++) {
-                    int ec;
-                    CParams = overwriteParams(cParamsToPVals(ZSTD_getCParams(i, buf.maxBlockSize, ctx.dictSize)), paramTarget);
-                    ec = BMK_benchParam(&candidate, buf, ctx, CParams);
-                    BMK_printWinnerOpt(stdout, i, candidate, CParams, target, buf.srcSize);
-
-                    if(!ec && compareResultLT(winner.result, candidate, relaxTarget(target), buf.srcSize)) {
-                        winner.result = candidate;
-                        winner.params = CParams;
-                    }
-
-                    CHECKTIMEGT(ret, 0, _displayCleanUp); /* if pass time limit, stop */
-                    /* if the current params are too slow, just stop. */
-                    if(target.cSpeed > candidate.cSpeed * 3 / 2) { break; }
-                }
-
-                BMK_printWinnerOpt(stdout, CUSTOM_LEVEL, winner.result, winner.params, target, buf.srcSize);
-            }
-        }
-
-        DEBUGOUTPUT("Real Opt\n");
-        /* start 'real' optimization */
-        {   int bestStrategy = (int)winner.params.vals[strt_ind];
-            if (paramTarget.vals[strt_ind] == PARAM_UNSET) {
-                int st = bestStrategy;
-                int tries = g_maxTries;
-
-                /* one iterations of hill climbing with the level-defined parameters. */
-                {   winnerInfo_t const w1 = climbOnce(target, allMT, buf, ctx, winner.params);
-                    if (compareResultLT(winner.result, w1.result, target, buf.srcSize)) {
-                        winner = w1;
-                    }
-                    CHECKTIMEGT(ret, 0, _displayCleanUp);
-                }
-
-                while(st && tries > 0) {
-                    winnerInfo_t wc;
-                    DEBUGOUTPUT("StrategySwitch: %s\n", g_stratName[st]);
-
-                    wc = optimizeFixedStrategy(buf, ctx, target, paramBase, st, allMT, tries);
-
-                    if(compareResultLT(winner.result, wc.result, target, buf.srcSize)) {
-                        winner = wc;
-                        tries = g_maxTries;
-                        bestStrategy = st;
-                    } else {
-                        st = nextStrategy(st, bestStrategy);
-                        tries -= TRY_DECAY;
-                    }
-                    CHECKTIMEGT(ret, 0, _displayCleanUp);
-                }
-            } else {
-                winner = optimizeFixedStrategy(buf, ctx, target, paramBase, paramTarget.vals[strt_ind], allMT, g_maxTries);
-            }
-
-        }
-
-        /* no solution found */
-        if(winner.result.cSize == (size_t)-1) {
-            ret = 1;
-            DISPLAY("No feasible solution found\n");
-            goto _cleanUp;
-        }
-
-        /* end summary */
-_displayCleanUp:
-        if (g_displayLevel >= 0) {
-            BMK_displayOneResult(stdout, winner, buf.srcSize);
-        }
-        BMK_paramValues_into_commandLine(stdout, winner.params);
-        DISPLAYLEVEL(1, "grillParams size - optimizer completed \n");
-    }
-
-_cleanUp:
-    freeContexts(ctx);
-    freeBuffers(buf);
-    freeMemoTableArray(allMT);
-    return ret;
-}
-
-/*-************************************
-*  CLI parsing functions
-**************************************/
-
-/** longCommandWArg() :
- *  check if *stringPtr is the same as longCommand.
- *  If yes, @return 1 and advances *stringPtr to the position which immediately follows longCommand.
- * @return 0 and doesn't modify *stringPtr otherwise.
- * from zstdcli.c
- */
-static int longCommandWArg(const char** stringPtr, const char* longCommand)
-{
-    size_t const comSize = strlen(longCommand);
-    int const result = !strncmp(*stringPtr, longCommand, comSize);
-    if (result) *stringPtr += comSize;
-    return result;
-}
-
-static void errorOut(const char* msg)
-{
-    DISPLAY("%s \n", msg); exit(1);
-}
-
-/*! readU32FromChar() :
- * @return : unsigned integer value read from input in `char` format.
- *  allows and interprets K, KB, KiB, M, MB and MiB suffix.
- *  Will also modify `*stringPtr`, advancing it to position where it stopped reading.
- *  Note : function will exit() program if digit sequence overflows */
-static unsigned readU32FromChar(const char** stringPtr)
-{
-    const char errorMsg[] = "error: numeric value too large";
-    unsigned sign = 1;
-    unsigned result = 0;
-    if(**stringPtr == '-') { sign = (unsigned)-1; (*stringPtr)++; }
-    while ((**stringPtr >='0') && (**stringPtr <='9')) {
-        unsigned const max = (((unsigned)(-1)) / 10) - 1;
-        if (result > max) errorOut(errorMsg);
-        result *= 10;
-        assert(**stringPtr >= '0');
-        result += (unsigned)(**stringPtr - '0');
-        (*stringPtr)++ ;
-    }
-    if ((**stringPtr=='K') || (**stringPtr=='M')) {
-        unsigned const maxK = ((unsigned)(-1)) >> 10;
-        if (result > maxK) errorOut(errorMsg);
-        result <<= 10;
-        if (**stringPtr=='M') {
-            if (result > maxK) errorOut(errorMsg);
-            result <<= 10;
-        }
-        (*stringPtr)++;  /* skip `K` or `M` */
-        if (**stringPtr=='i') (*stringPtr)++;
-        if (**stringPtr=='B') (*stringPtr)++;
-    }
-    return result * sign;
-}
-
-static double readDoubleFromChar(const char** stringPtr)
-{
-    double result = 0, divide = 10;
-    while ((**stringPtr >='0') && (**stringPtr <='9')) {
-        result *= 10, result += **stringPtr - '0', (*stringPtr)++ ;
-    }
-    if(**stringPtr!='.') {
-        return result;
-    }
-    (*stringPtr)++;
-    while ((**stringPtr >='0') && (**stringPtr <='9')) {
-        result += (double)(**stringPtr - '0') / divide, divide *= 10, (*stringPtr)++ ;
-    }
-    return result;
-}
-
-static int usage(const char* exename)
-{
-    DISPLAY( "Usage :\n");
-    DISPLAY( "      %s [arg] file\n", exename);
-    DISPLAY( "Arguments :\n");
-    DISPLAY( " file : path to the file used as reference (if none, generates a compressible sample)\n");
-    DISPLAY( " -H/-h  : Help (this text + advanced options)\n");
-    return 0;
-}
-
-static int usage_advanced(void)
-{
-    DISPLAY( "\nAdvanced options :\n");
-    DISPLAY( " -T#          : set level 1 speed objective \n");
-    DISPLAY( " -B#          : cut input into blocks of size # (default : single block) \n");
-    DISPLAY( " --optimize=  : same as -O with more verbose syntax (see README.md)\n");
-    DISPLAY( " -S           : Single run \n");
-    DISPLAY( " --zstd       : Single run, parameter selection same as zstdcli \n");
-    DISPLAY( " -P#          : generated sample compressibility (default : %.1f%%) \n", COMPRESSIBILITY_DEFAULT * 100);
-    DISPLAY( " -t#          : Caps runtime of operation in seconds (default : %u seconds (%.1f hours)) \n",
-                                (unsigned)g_timeLimit_s, (double)g_timeLimit_s / 3600);
-    DISPLAY( " -v           : Prints Benchmarking output\n");
-    DISPLAY( " -D           : Next argument dictionary file\n");
-    DISPLAY( " -s           : Seperate Files\n");
-    return 0;
-}
-
-static int badusage(const char* exename)
-{
-    DISPLAY("Wrong parameters\n");
-    usage(exename);
-    return 1;
-}
-
-#define PARSE_SUB_ARGS(stringLong, stringShort, variable) { \
-    if ( longCommandWArg(&argument, stringLong)             \
-      || longCommandWArg(&argument, stringShort) ) {        \
-          variable = readU32FromChar(&argument);            \
-          if (argument[0]==',') {                           \
-              argument++; continue;                         \
-          } else break;                                     \
-}   }
-
-/* 1 if successful parse, 0 otherwise */
-static int parse_params(const char** argptr, paramValues_t* pv) {
-    int matched = 0;
-    const char* argOrig = *argptr;
-    varInds_t v;
-    for(v = 0; v < NUM_PARAMS; v++) {
-        if ( longCommandWArg(argptr,g_shortParamNames[v])
-          || longCommandWArg(argptr, g_paramNames[v]) ) {
-            if(**argptr == '=') {
-                (*argptr)++;
-                pv->vals[v] = readU32FromChar(argptr);
-                matched = 1;
-                break;
-            }
-        }
-        /* reset and try again */
-        *argptr = argOrig;
-    }
-    return matched;
-}
-
-/*-************************************
-*  Main
-**************************************/
-
-int main(int argc, const char** argv)
-{
-    int i,
-        filenamesStart=0,
-        result;
-    const char* exename=argv[0];
-    const char* input_filename = NULL;
-    const char* dictFileName = NULL;
-    U32 main_pause = 0;
-    int cLevelOpt = 0, cLevelRun = 0;
-    int seperateFiles = 0;
-    double compressibility = COMPRESSIBILITY_DEFAULT;
-    U32 memoTableLog = PARAM_UNSET;
-    constraint_t target = { 0, 0, (U32)-1 };
-
-    paramValues_t paramTarget = emptyParams();
-    g_params = emptyParams();
-
-    assert(argc>=1);   /* for exename */
-
-    for(i=1; i<argc; i++) {
-        const char* argument = argv[i];
-        DEBUGOUTPUT("%d: %s\n", i, argument);
-        assert(argument != NULL);
-
-        if(!strcmp(argument,"--no-seed")) { g_noSeed = 1; continue; }
-
-        if (longCommandWArg(&argument, "--optimize=")) {
-            g_optimizer = 1;
-            for ( ; ;) {
-                if(parse_params(&argument, &paramTarget)) { if(argument[0] == ',') { argument++; continue; } else break; }
-                PARSE_SUB_ARGS("compressionSpeed=" ,  "cSpeed=", target.cSpeed);
-                PARSE_SUB_ARGS("decompressionSpeed=", "dSpeed=", target.dSpeed);
-                PARSE_SUB_ARGS("compressionMemory=" , "cMem=", target.cMem);
-                PARSE_SUB_ARGS("strict=", "stc=", g_strictness);
-                PARSE_SUB_ARGS("maxTries=", "tries=", g_maxTries);
-                PARSE_SUB_ARGS("memoLimitLog=", "memLog=", memoTableLog);
-                if (longCommandWArg(&argument, "level=") || longCommandWArg(&argument, "lvl=")) { cLevelOpt = (int)readU32FromChar(&argument); g_optmode = 1; if (argument[0]==',') { argument++; continue; } else break; }
-                if (longCommandWArg(&argument, "speedForRatio=") || longCommandWArg(&argument, "speedRatio=")) { g_ratioMultiplier = readDoubleFromChar(&argument); if (argument[0]==',') { argument++; continue; } else break; }
-
-                DISPLAY("invalid optimization parameter \n");
-                return 1;
-            }
-
-            if (argument[0] != 0) {
-                DISPLAY("invalid --optimize= format\n");
-                return 1; /* check the end of string */
-            }
-            continue;
-        } else if (longCommandWArg(&argument, "--zstd=")) {
-        /* Decode command (note : aggregated commands are allowed) */
-            g_singleRun = 1;
-            for ( ; ;) {
-                if(parse_params(&argument, &g_params)) { if(argument[0] == ',') { argument++; continue; } else break; }
-                if (longCommandWArg(&argument, "level=") || longCommandWArg(&argument, "lvl=")) { cLevelRun = (int)readU32FromChar(&argument); g_params = emptyParams(); if (argument[0]==',') { argument++; continue; } else break; }
-
-                DISPLAY("invalid compression parameter \n");
-                return 1;
-            }
-
-            if (argument[0] != 0) {
-                DISPLAY("invalid --zstd= format\n");
-                return 1; /* check the end of string */
-            }
-            continue;
-            /* if not return, success */
-
-        } else if (longCommandWArg(&argument, "--display=")) {
-            /* Decode command (note : aggregated commands are allowed) */
-            memset(g_silenceParams, 1, sizeof(g_silenceParams));
-            for ( ; ;) {
-                int found = 0;
-                varInds_t v;
-                for(v = 0; v < NUM_PARAMS; v++) {
-                    if(longCommandWArg(&argument, g_shortParamNames[v]) || longCommandWArg(&argument, g_paramNames[v])) {
-                        g_silenceParams[v] = 0;
-                        found = 1;
-                    }
-                }
-                if(longCommandWArg(&argument, "compressionParameters") || longCommandWArg(&argument, "cParams")) {
-                    for(v = 0; v <= strt_ind; v++) {
-                        g_silenceParams[v] = 0;
-                    }
-                    found = 1;
-                }
-
-
-                if(found) {
-                    if(argument[0]==',') {
-                        continue;
-                    } else {
-                        break;
-                    }
-                }
-                DISPLAY("invalid parameter name parameter \n");
-                return 1;
-            }
-
-            if (argument[0] != 0) {
-                DISPLAY("invalid --display format\n");
-                return 1; /* check the end of string */
-            }
-            continue;
-        } else if (argument[0]=='-') {
-            argument++;
-
-            while (argument[0]!=0) {
-
-                switch(argument[0])
-                {
-                    /* Display help on usage */
-                case 'h' :
-                case 'H': usage(exename); usage_advanced(); return 0;
-
-                    /* Pause at the end (hidden option) */
-                case 'p': main_pause = 1; argument++; break;
-
-                    /* Sample compressibility (when no file provided) */
-                case 'P':
-                    argument++;
-                    {   U32 const proba32 = readU32FromChar(&argument);
-                        compressibility = (double)proba32 / 100.;
-                    }
-                    break;
-
-                    /* Run Single conf */
-                case 'S':
-                    g_singleRun = 1;
-                    argument++;
-                    for ( ; ; ) {
-                        switch(*argument)
-                        {
-                        case 'w':
-                            argument++;
-                            g_params.vals[wlog_ind] = readU32FromChar(&argument);
-                            continue;
-                        case 'c':
-                            argument++;
-                            g_params.vals[clog_ind] = readU32FromChar(&argument);
-                            continue;
-                        case 'h':
-                            argument++;
-                            g_params.vals[hlog_ind] = readU32FromChar(&argument);
-                            continue;
-                        case 's':
-                            argument++;
-                            g_params.vals[slog_ind] = readU32FromChar(&argument);
-                            continue;
-                        case 'l':  /* search length */
-                            argument++;
-                            g_params.vals[mml_ind] = readU32FromChar(&argument);
-                            continue;
-                        case 't':  /* target length */
-                            argument++;
-                            g_params.vals[tlen_ind] = readU32FromChar(&argument);
-                            continue;
-                        case 'S':  /* strategy */
-                            argument++;
-                            g_params.vals[strt_ind] = readU32FromChar(&argument);
-                            continue;
-                        case 'f':  /* forceAttachDict */
-                            argument++;
-                            g_params.vals[fadt_ind] = readU32FromChar(&argument);
-                            continue;
-                        case 'L':
-                            {   argument++;
-                                cLevelRun = (int)readU32FromChar(&argument);
-                                g_params = emptyParams();
-                                continue;
-                            }
-                        default : ;
-                        }
-                        break;
-                    }
-
-                    break;
-
-                    /* target level1 speed objective, in MB/s */
-                case 'T':
-                    argument++;
-                    g_target = readU32FromChar(&argument);
-                    break;
-
-                    /* cut input into blocks */
-                case 'B':
-                    argument++;
-                    g_blockSize = readU32FromChar(&argument);
-                    DISPLAY("using %u KB block size \n", (unsigned)(g_blockSize>>10));
-                    break;
-
-                    /* caps runtime (in seconds) */
-                case 't':
-                    argument++;
-                    g_timeLimit_s = readU32FromChar(&argument);
-                    break;
-
-                case 's':
-                    argument++;
-                    seperateFiles = 1;
-                    break;
-
-                case 'q':
-                    while (argument[0] == 'q') { argument++; g_displayLevel--; }
-                    break;
-
-                case 'v':
-                    while (argument[0] == 'v') { argument++; g_displayLevel++; }
-                    break;
-
-                /* load dictionary file (only applicable for optimizer rn) */
-                case 'D':
-                    if(i == argc - 1) { /* last argument, return error. */
-                        DISPLAY("Dictionary file expected but not given : %d\n", i);
-                        return 1;
-                    } else {
-                        i++;
-                        dictFileName = argv[i];
-                        argument += strlen(argument);
-                    }
-                    break;
-
-                    /* Unknown command */
-                default : return badusage(exename);
-                }
-            }
-            continue;
-        }   /* if (argument[0]=='-') */
-
-        /* first provided filename is input */
-        if (!input_filename) { input_filename=argument; filenamesStart=i; continue; }
-    }
-
-    /* Welcome message */
-    DISPLAYLEVEL(2, WELCOME_MESSAGE);
-
-    if (filenamesStart==0) {
-        if (g_optimizer) {
-            DISPLAY("Optimizer Expects File\n");
-            return 1;
-        } else {
-            result = benchSample(compressibility, cLevelRun);
-        }
-    } else {
-        if(seperateFiles) {
-            for(i = 0; i < argc - filenamesStart; i++) {
-                if (g_optimizer) {
-                    result = optimizeForSize(argv+filenamesStart + i, 1, dictFileName, target, paramTarget, cLevelOpt, cLevelRun, memoTableLog);
-                    if(result) { DISPLAY("Error on File %d", i); return result; }
-                } else {
-                    result = benchFiles(argv+filenamesStart + i, 1, dictFileName, cLevelRun);
-                    if(result) { DISPLAY("Error on File %d", i); return result; }
-                }
-            }
-        } else {
-            if (g_optimizer) {
-                assert(filenamesStart < argc);
-                result = optimizeForSize(argv+filenamesStart, (size_t)(argc-filenamesStart), dictFileName, target, paramTarget, cLevelOpt, cLevelRun, memoTableLog);
-            } else {
-                result = benchFiles(argv+filenamesStart, argc-filenamesStart, dictFileName, cLevelRun);
-            }
-        }
-    }
-
-    if (main_pause) { int unused; printf("press enter...\n"); unused = getchar(); (void)unused; }
-
-    return result;
-}
diff --git a/tests/playTests.sh b/tests/playTests.sh
deleted file mode 100755
index c09e0c21cc2b..000000000000
--- a/tests/playTests.sh
+++ /dev/null
@@ -1,1187 +0,0 @@
-#!/bin/sh
-
-set -e
-
-die() {
-    println "$@" 1>&2
-    exit 1
-}
-
-roundTripTest() {
-    if [ -n "$3" ]; then
-        cLevel="$3"
-        proba="$2"
-    else
-        cLevel="$2"
-        proba=""
-    fi
-    if [ -n "$4" ]; then
-        dLevel="$4"
-    else
-        dLevel="$cLevel"
-    fi
-
-    rm -f tmp1 tmp2
-    println "roundTripTest: ./datagen $1 $proba | $ZSTD -v$cLevel | $ZSTD -d$dLevel"
-    ./datagen $1 $proba | $MD5SUM > tmp1
-    ./datagen $1 $proba | $ZSTD --ultra -v$cLevel | $ZSTD -d$dLevel  | $MD5SUM > tmp2
-    $DIFF -q tmp1 tmp2
-}
-
-fileRoundTripTest() {
-    if [ -n "$3" ]; then
-        local_c="$3"
-        local_p="$2"
-    else
-        local_c="$2"
-        local_p=""
-    fi
-    if [ -n "$4" ]; then
-        local_d="$4"
-    else
-        local_d="$local_c"
-    fi
-
-    rm -f tmp.zstd tmp.md5.1 tmp.md5.2
-    println "fileRoundTripTest: ./datagen $1 $local_p > tmp && $ZSTD -v$local_c -c tmp | $ZSTD -d$local_d"
-    ./datagen $1 $local_p > tmp
-    < tmp $MD5SUM > tmp.md5.1
-    $ZSTD --ultra -v$local_c -c tmp | $ZSTD -d$local_d | $MD5SUM > tmp.md5.2
-    $DIFF -q tmp.md5.1 tmp.md5.2
-}
-
-truncateLastByte() {
-    dd bs=1 count=$(($(wc -c < "$1") - 1)) if="$1"
-}
-
-println() {
-    printf '%b\n' "${*}"
-}
-
-
-SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd)
-PRGDIR="$SCRIPT_DIR/../programs"
-TESTDIR="$SCRIPT_DIR/../tests"
-UNAME=$(uname)
-
-detectedTerminal=false
-if [ -t 0 ] && [ -t 1 ]
-then
-    detectedTerminal=true
-fi
-isTerminal=${isTerminal:-$detectedTerminal}
-
-isWindows=false
-INTOVOID="/dev/null"
-case "$UNAME" in
-  GNU) DEVDEVICE="/dev/random" ;;
-  *) DEVDEVICE="/dev/zero" ;;
-esac
-case "$OS" in
-  Windows*)
-    isWindows=true
-    INTOVOID="NUL"
-    DEVDEVICE="NUL"
-    ;;
-esac
-
-case "$UNAME" in
-  Darwin) MD5SUM="md5 -r" ;;
-  FreeBSD) MD5SUM="gmd5sum" ;;
-  OpenBSD) MD5SUM="md5" ;;
-  *) MD5SUM="md5sum" ;;
-esac
-
-DIFF="diff"
-case "$UNAME" in
-  SunOS) DIFF="gdiff" ;;
-esac
-
-println "\nStarting playTests.sh isWindows=$isWindows ZSTD='$ZSTD'"
-
-[ -n "$ZSTD" ] || die "ZSTD variable must be defined!"
-
-if echo hello | $ZSTD -v -T2 2>&1 > $INTOVOID | grep -q 'multi-threading is disabled'
-then
-    hasMT=""
-else
-    hasMT="true"
-fi
-
-
-println "\n===>  simple tests "
-
-./datagen > tmp
-println "test : basic compression "
-$ZSTD -f tmp                      # trivial compression case, creates tmp.zst
-println "test : basic decompression"
-$ZSTD -df tmp.zst                 # trivial decompression case (overwrites tmp)
-println "test : too large compression level => auto-fix"
-$ZSTD -99 -f tmp  # too large compression level, automatic sized down
-$ZSTD -5000000000 -f tmp && die "too large numeric value : must fail"
-println "test : --fast aka negative compression levels"
-$ZSTD --fast -f tmp  # == -1
-$ZSTD --fast=3 -f tmp  # == -3
-$ZSTD --fast=200000 -f tmp  # too low compression level, automatic fixed
-$ZSTD --fast=5000000000 -f tmp && die "too large numeric value : must fail"
-$ZSTD -c --fast=0 tmp > $INTOVOID && die "--fast must not accept value 0"
-println "test : too large numeric argument"
-$ZSTD --fast=9999999999 -f tmp  && die "should have refused numeric value"
-println "test : set compression level with environment variable ZSTD_CLEVEL"
-ZSTD_CLEVEL=12  $ZSTD -f tmp # positive compression level
-ZSTD_CLEVEL=-12 $ZSTD -f tmp # negative compression level
-ZSTD_CLEVEL=+12 $ZSTD -f tmp # valid: verbose '+' sign
-ZSTD_CLEVEL=''  $ZSTD -f tmp # empty env var, warn and revert to default setting
-ZSTD_CLEVEL=-   $ZSTD -f tmp # malformed env var, warn and revert to default setting
-ZSTD_CLEVEL=a   $ZSTD -f tmp # malformed env var, warn and revert to default setting
-ZSTD_CLEVEL=+a  $ZSTD -f tmp # malformed env var, warn and revert to default setting
-ZSTD_CLEVEL=3a7 $ZSTD -f tmp # malformed env var, warn and revert to default setting
-ZSTD_CLEVEL=50000000000  $ZSTD -f tmp # numeric value too large, warn and revert to default setting
-println "test : override ZSTD_CLEVEL with command line option"
-ZSTD_CLEVEL=12  $ZSTD --fast=3 -f tmp # overridden by command line option
-println "test : compress to stdout"
-$ZSTD tmp -c > tmpCompressed
-$ZSTD tmp --stdout > tmpCompressed       # long command format
-println "test : compress to named file"
-rm tmpCompressed
-$ZSTD tmp -o tmpCompressed
-test -f tmpCompressed   # file must be created
-println "test : -o must be followed by filename (must fail)"
-$ZSTD tmp -of tmpCompressed && die "-o must be followed by filename "
-println "test : force write, correct order"
-$ZSTD tmp -fo tmpCompressed
-println "test : forgotten argument"
-cp tmp tmp2
-$ZSTD tmp2 -fo && die "-o must be followed by filename "
-println "test : implied stdout when input is stdin"
-println bob | $ZSTD | $ZSTD -d
-if [ "$isTerminal" = true ]; then
-println "test : compressed data to terminal"
-println bob | $ZSTD && die "should have refused : compressed data to terminal"
-println "test : compressed data from terminal (a hang here is a test fail, zstd is wrongly waiting on data from terminal)"
-$ZSTD -d > $INTOVOID && die "should have refused : compressed data from terminal"
-fi
-println "test : null-length file roundtrip"
-println -n '' | $ZSTD - --stdout | $ZSTD -d --stdout
-println "test : ensure small file doesn't add 3-bytes null block"
-./datagen -g1 > tmp1
-$ZSTD tmp1 -c | wc -c | grep "14"
-$ZSTD < tmp1  | wc -c | grep "14"
-println "test : decompress file with wrong suffix (must fail)"
-$ZSTD -d tmpCompressed && die "wrong suffix error not detected!"
-$ZSTD -df tmp && die "should have refused : wrong extension"
-println "test : decompress into stdout"
-$ZSTD -d tmpCompressed -c > tmpResult    # decompression using stdout
-$ZSTD --decompress tmpCompressed -c > tmpResult
-$ZSTD --decompress tmpCompressed --stdout > tmpResult
-println "test : decompress from stdin into stdout"
-$ZSTD -dc   < tmp.zst > $INTOVOID   # combine decompression, stdin & stdout
-$ZSTD -dc - < tmp.zst > $INTOVOID
-$ZSTD -d    < tmp.zst > $INTOVOID   # implicit stdout when stdin is used
-$ZSTD -d  - < tmp.zst > $INTOVOID
-println "test : impose memory limitation (must fail)"
-$ZSTD -d -f tmp.zst -M2K -c > $INTOVOID && die "decompression needs more memory than allowed"
-$ZSTD -d -f tmp.zst --memlimit=2K -c > $INTOVOID && die "decompression needs more memory than allowed"  # long command
-$ZSTD -d -f tmp.zst --memory=2K -c > $INTOVOID && die "decompression needs more memory than allowed"  # long command
-$ZSTD -d -f tmp.zst --memlimit-decompress=2K -c > $INTOVOID && die "decompression needs more memory than allowed"  # long command
-println "test : overwrite protection"
-$ZSTD -q tmp && die "overwrite check failed!"
-println "test : force overwrite"
-$ZSTD -q -f tmp
-$ZSTD -q --force tmp
-println "test : overwrite readonly file"
-rm -f tmpro tmpro.zst
-println foo > tmpro.zst
-println foo > tmpro
-chmod 400 tmpro.zst
-$ZSTD -q tmpro && die "should have refused to overwrite read-only file"
-$ZSTD -q -f tmpro
-println "test: --no-progress flag"
-$ZSTD tmpro -c --no-progress | $ZSTD -d -f -o "$INTOVOID" --no-progress
-$ZSTD tmpro -cv --no-progress | $ZSTD -dv -f -o "$INTOVOID" --no-progress
-rm -f tmpro tmpro.zst
-println "test: overwrite input file (must fail)"
-$ZSTD tmp -fo tmp && die "zstd compression overwrote the input file"
-$ZSTD tmp.zst -dfo tmp.zst && die "zstd decompression overwrote the input file"
-println "test: detect that input file does not exist"
-$ZSTD nothere && die "zstd hasn't detected that input file does not exist"
-println "test: --[no-]compress-literals"
-$ZSTD tmp -c --no-compress-literals -1       | $ZSTD -t
-$ZSTD tmp -c --no-compress-literals --fast=1 | $ZSTD -t
-$ZSTD tmp -c --no-compress-literals -19      | $ZSTD -t
-$ZSTD tmp -c --compress-literals    -1       | $ZSTD -t
-$ZSTD tmp -c --compress-literals    --fast=1 | $ZSTD -t
-$ZSTD tmp -c --compress-literals    -19      | $ZSTD -t
-$ZSTD -b --fast=1 -i0e1 tmp --compress-literals
-$ZSTD -b --fast=1 -i0e1 tmp --no-compress-literals
-
-println "test: --exclude-compressed flag"
-rm -rf precompressedFilterTestDir
-mkdir -p precompressedFilterTestDir
-./datagen $size > precompressedFilterTestDir/input.5
-./datagen $size > precompressedFilterTestDir/input.6
-$ZSTD --exclude-compressed --long --rm -r precompressedFilterTestDir
-sleep 5
-./datagen $size > precompressedFilterTestDir/input.7
-./datagen $size > precompressedFilterTestDir/input.8
-$ZSTD --exclude-compressed --long --rm -r precompressedFilterTestDir
-test ! -f precompressedFilterTestDir/input.5.zst.zst
-test ! -f precompressedFilterTestDir/input.6.zst.zst
-file1timestamp=`date -r precompressedFilterTestDir/input.5.zst +%s`
-file2timestamp=`date -r precompressedFilterTestDir/input.7.zst +%s`
-if [[ $file2timestamp -ge $file1timestamp ]]; then
-  println "Test is successful. input.5.zst is precompressed and therefore not compressed/modified again."
-else
-  println "Test is not successful"
-fi
-#File Extension check.
-./datagen $size > precompressedFilterTestDir/input.zstbar
-$ZSTD --exclude-compressed --long --rm -r precompressedFilterTestDir
-#ZSTD should compress input.zstbar
-test -f precompressedFilterTestDir/input.zstbar.zst
-#Check without the --exclude-compressed flag
-$ZSTD --long --rm -r precompressedFilterTestDir
-#Files should get compressed again without the --exclude-compressed flag.
-test -f precompressedFilterTestDir/input.5.zst.zst
-test -f precompressedFilterTestDir/input.6.zst.zst
-println "Test completed"
-
-println "test : file removal"
-$ZSTD -f --rm tmp
-test ! -f tmp  # tmp should no longer be present
-$ZSTD -f -d --rm tmp.zst
-test ! -f tmp.zst   # tmp.zst should no longer be present
-println "test : should quietly not remove non-regular file"
-println hello > tmp
-$ZSTD tmp -f -o "$DEVDEVICE" 2>tmplog > "$INTOVOID"
-grep -v "Refusing to remove non-regular file" tmplog
-rm -f tmplog
-$ZSTD tmp -f -o "$INTOVOID" 2>&1 | grep -v "Refusing to remove non-regular file"
-println "test : --rm on stdin"
-println a | $ZSTD --rm > $INTOVOID   # --rm should remain silent
-rm tmp
-$ZSTD -f tmp && die "tmp not present : should have failed"
-test ! -f tmp.zst  # tmp.zst should not be created
-println "test : -d -f do not delete destination when source is not present"
-touch tmp    # create destination file
-$ZSTD -d -f tmp.zst && die "attempt to decompress a non existing file"
-test -f tmp  # destination file should still be present
-println "test : -f do not delete destination when source is not present"
-rm tmp         # erase source file
-touch tmp.zst  # create destination file
-$ZSTD -f tmp && die "attempt to compress a non existing file"
-test -f tmp.zst  # destination file should still be present
-rm -rf tmp*  # may also erase tmp* directory from previous failed run
-
-println "\n===> decompression only tests "
-head -c 1048576 /dev/zero > tmp
-$ZSTD -d -o tmp1 "$TESTDIR/golden-decompression/rle-first-block.zst"
-$DIFF -s tmp1 tmp
-rm tmp*
-
-println "test : compress multiple files"
-println hello > tmp1
-println world > tmp2
-$ZSTD tmp1 tmp2 -o "$INTOVOID" -f
-$ZSTD tmp1 tmp2 -c | $ZSTD -t
-$ZSTD tmp1 tmp2 -o tmp.zst
-test ! -f tmp1.zst
-test ! -f tmp2.zst
-$ZSTD tmp1 tmp2
-$ZSTD -t tmp1.zst tmp2.zst
-$ZSTD -dc tmp1.zst tmp2.zst
-$ZSTD tmp1.zst tmp2.zst -o "$INTOVOID" -f
-$ZSTD -d tmp1.zst tmp2.zst -o tmp
-touch tmpexists
-$ZSTD tmp1 tmp2 -f -o tmpexists
-$ZSTD tmp1 tmp2 -o tmpexists && die "should have refused to overwrite"
-# Bug: PR #972
-if [ "$?" -eq 139 ]; then
-  die "should not have segfaulted"
-fi
-rm tmp*
-
-println "test : compress multiple files into an output directory, --output-dir-flat"
-println henlo > tmp1
-mkdir tmpInputTestDir
-mkdir tmpInputTestDir/we
-mkdir tmpInputTestDir/we/must
-mkdir tmpInputTestDir/we/must/go
-mkdir tmpInputTestDir/we/must/go/deeper
-println cool > tmpInputTestDir/we/must/go/deeper/tmp2
-mkdir tmpOutDir
-$ZSTD tmp1 tmpInputTestDir/we/must/go/deeper/tmp2 --output-dir-flat tmpOutDir
-test -f tmpOutDir/tmp1.zst
-test -f tmpOutDir/tmp2.zst
-println "test : decompress multiple files into an output directory, --output-dir-flat"
-mkdir tmpOutDirDecomp
-$ZSTD tmpOutDir -r -d --output-dir-flat tmpOutDirDecomp
-test -f tmpOutDirDecomp/tmp2
-test -f tmpOutDirDecomp/tmp1
-rm -f tmpOutDirDecomp/*
-$ZSTD tmpOutDir -r -d --output-dir-flat=tmpOutDirDecomp
-test -f tmpOutDirDecomp/tmp2
-test -f tmpOutDirDecomp/tmp1
-rm -rf tmp*
-
-println "\n===>  Advanced compression parameters "
-println "Hello world!" | $ZSTD --zstd=windowLog=21,      - -o tmp.zst && die "wrong parameters not detected!"
-println "Hello world!" | $ZSTD --zstd=windowLo=21        - -o tmp.zst && die "wrong parameters not detected!"
-println "Hello world!" | $ZSTD --zstd=windowLog=21,slog  - -o tmp.zst && die "wrong parameters not detected!"
-println "Hello world!" | $ZSTD --zstd=strategy=10        - -o tmp.zst && die "parameter out of bound not detected!"  # > btultra2 : does not exist
-test ! -f tmp.zst  # tmp.zst should not be created
-roundTripTest -g512K
-roundTripTest -g512K " --zstd=mml=3,tlen=48,strat=6"
-roundTripTest -g512K " --zstd=strat=6,wlog=23,clog=23,hlog=22,slog=6"
-roundTripTest -g512K " --zstd=windowLog=23,chainLog=23,hashLog=22,searchLog=6,minMatch=3,targetLength=48,strategy=6"
-roundTripTest -g512K " --single-thread --long --zstd=ldmHashLog=20,ldmMinMatch=64,ldmBucketSizeLog=1,ldmHashRateLog=7"
-roundTripTest -g512K " --single-thread --long --zstd=lhlog=20,lmml=64,lblog=1,lhrlog=7"
-roundTripTest -g64K  "19 --zstd=strat=9"   # btultra2
-
-
-println "\n===>  Pass-Through mode "
-println "Hello world 1!" | $ZSTD -df
-println "Hello world 2!" | $ZSTD -dcf
-println "Hello world 3!" > tmp1
-$ZSTD -dcf tmp1
-
-
-println "\n===>  frame concatenation "
-
-println "hello " > hello.tmp
-println "world!" > world.tmp
-cat hello.tmp world.tmp > helloworld.tmp
-$ZSTD -c hello.tmp > hello.zstd
-$ZSTD -c world.tmp > world.zstd
-cat hello.zstd world.zstd > helloworld.zstd
-$ZSTD -dc helloworld.zstd > result.tmp
-cat result.tmp
-$DIFF helloworld.tmp result.tmp
-println "frame concatenation without checksum"
-$ZSTD -c hello.tmp > hello.zstd --no-check
-$ZSTD -c world.tmp > world.zstd --no-check
-cat hello.zstd world.zstd > helloworld.zstd
-$ZSTD -dc helloworld.zstd > result.tmp
-$DIFF helloworld.tmp result.tmp
-println "testing zstdcat symlink"
-ln -sf $ZSTD zstdcat
-./zstdcat helloworld.zstd > result.tmp
-$DIFF helloworld.tmp result.tmp
-ln -s helloworld.zstd helloworld.link.zstd
-./zstdcat helloworld.link.zstd > result.tmp
-$DIFF helloworld.tmp result.tmp
-rm zstdcat
-rm result.tmp
-println "testing zcat symlink"
-ln -sf $ZSTD zcat
-./zcat helloworld.zstd > result.tmp
-$DIFF helloworld.tmp result.tmp
-./zcat helloworld.link.zstd > result.tmp
-$DIFF helloworld.tmp result.tmp
-rm zcat
-rm ./*.tmp ./*.zstd
-println "frame concatenation tests completed"
-
-
-if [ "$isWindows" = false ] && [ "$UNAME" != 'SunOS' ] && [ "$UNAME" != "OpenBSD" ] ; then
-println "\n**** flush write error test **** "
-
-println "println foo | $ZSTD > /dev/full"
-println foo | $ZSTD > /dev/full && die "write error not detected!"
-println "println foo | $ZSTD | $ZSTD -d > /dev/full"
-println foo | $ZSTD | $ZSTD -d > /dev/full && die "write error not detected!"
-
-fi
-
-
-if [ "$isWindows" = false ] && [ "$UNAME" != 'SunOS' ] ; then
-
-println "\n===>  symbolic link test "
-
-rm -f hello.tmp world.tmp world2.tmp hello.tmp.zst world.tmp.zst
-println "hello world" > hello.tmp
-ln -s hello.tmp world.tmp
-ln -s hello.tmp world2.tmp
-$ZSTD world.tmp hello.tmp || true
-test -f hello.tmp.zst  # regular file should have been compressed!
-test ! -f world.tmp.zst  # symbolic link should not have been compressed!
-$ZSTD world.tmp || true
-test ! -f world.tmp.zst  # symbolic link should not have been compressed!
-$ZSTD world.tmp world2.tmp || true
-test ! -f world.tmp.zst  # symbolic link should not have been compressed!
-test ! -f world2.tmp.zst  # symbolic link should not have been compressed!
-$ZSTD world.tmp hello.tmp -f
-test -f world.tmp.zst  # symbolic link should have been compressed with --force
-rm -f hello.tmp world.tmp world2.tmp hello.tmp.zst world.tmp.zst
-
-fi
-
-
-println "\n===>  test sparse file support "
-
-./datagen -g5M  -P100 > tmpSparse
-$ZSTD tmpSparse -c | $ZSTD -dv -o tmpSparseRegen
-$DIFF -s tmpSparse tmpSparseRegen
-$ZSTD tmpSparse -c | $ZSTD -dv --sparse -c > tmpOutSparse
-$DIFF -s tmpSparse tmpOutSparse
-$ZSTD tmpSparse -c | $ZSTD -dv --no-sparse -c > tmpOutNoSparse
-$DIFF -s tmpSparse tmpOutNoSparse
-ls -ls tmpSparse*  # look at file size and block size on disk
-./datagen -s1 -g1200007 -P100 | $ZSTD | $ZSTD -dv --sparse -c > tmpSparseOdd   # Odd size file (to not finish on an exact nb of blocks)
-./datagen -s1 -g1200007 -P100 | $DIFF -s - tmpSparseOdd
-ls -ls tmpSparseOdd  # look at file size and block size on disk
-println "\n Sparse Compatibility with Console :"
-println "Hello World 1 !" | $ZSTD | $ZSTD -d -c
-println "Hello World 2 !" | $ZSTD | $ZSTD -d | cat
-println "\n Sparse Compatibility with Append :"
-./datagen -P100 -g1M > tmpSparse1M
-cat tmpSparse1M tmpSparse1M > tmpSparse2M
-$ZSTD -v -f tmpSparse1M -o tmpSparseCompressed
-$ZSTD -d -v -f tmpSparseCompressed -o tmpSparseRegenerated
-$ZSTD -d -v -f tmpSparseCompressed -c >> tmpSparseRegenerated
-ls -ls tmpSparse*  # look at file size and block size on disk
-$DIFF tmpSparse2M tmpSparseRegenerated
-rm tmpSparse*
-
-
-println "\n===>  multiple files tests "
-
-./datagen -s1        > tmp1 2> $INTOVOID
-./datagen -s2 -g100K > tmp2 2> $INTOVOID
-./datagen -s3 -g1M   > tmp3 2> $INTOVOID
-println "compress tmp* : "
-$ZSTD -f tmp*
-ls -ls tmp*
-rm tmp1 tmp2 tmp3
-println "decompress tmp* : "
-$ZSTD -df ./*.zst
-ls -ls tmp*
-println "compress tmp* into stdout > tmpall : "
-$ZSTD -c tmp1 tmp2 tmp3 > tmpall
-ls -ls tmp*  # check size of tmpall (should be tmp1.zst + tmp2.zst + tmp3.zst)
-println "decompress tmpall* into stdout > tmpdec : "
-cp tmpall tmpall2
-$ZSTD -dc tmpall* > tmpdec
-ls -ls tmp* # check size of tmpdec (should be 2*(tmp1 + tmp2 + tmp3))
-println "compress multiple files including a missing one (notHere) : "
-$ZSTD -f tmp1 notHere tmp2 && die "missing file not detected!"
-
-println "\n===>  stream-size mode"
-
-./datagen -g11000 > tmp
-println "test : basic file compression vs sized streaming compression"
-file_size=$($ZSTD -14 -f tmp -o tmp.zst && wc -c < tmp.zst)
-stream_size=$(cat tmp | $ZSTD -14 --stream-size=11000 | wc -c)
-if [ "$stream_size" -gt "$file_size" ]; then
-  die "hinted compression larger than expected"
-fi
-println "test : sized streaming compression and decompression"
-cat tmp | $ZSTD -14 -f tmp -o --stream-size=11000 tmp.zst
-$ZSTD -df tmp.zst -o tmp_decompress
-cmp tmp tmp_decompress || die "difference between original and decompressed file"
-println "test : incorrect stream size"
-cat tmp | $ZSTD -14 -f -o tmp.zst --stream-size=11001 && die "should fail with incorrect stream size"
-
-
-println "\n===>  size-hint mode"
-
-./datagen -g11000 > tmp
-./datagen -g11000 > tmp2
-./datagen > tmpDict
-println "test : basic file compression vs hinted streaming compression"
-file_size=$($ZSTD -14 -f tmp -o tmp.zst && wc -c < tmp.zst)
-stream_size=$(cat tmp | $ZSTD -14 --size-hint=11000 | wc -c)
-if [ "$stream_size" -ge "$file_size" ]; then
-  die "hinted compression larger than expected"
-fi
-println "test : hinted streaming compression and decompression"
-cat tmp | $ZSTD -14 -f -o tmp.zst --size-hint=11000
-$ZSTD -df tmp.zst -o tmp_decompress
-cmp tmp tmp_decompress || die "difference between original and decompressed file"
-println "test : hinted streaming compression with dictionary"
-cat tmp | $ZSTD -14 -f -D tmpDict --size-hint=11000 | $ZSTD -t -D tmpDict
-println "test : multiple file compression with hints and dictionary"
-$ZSTD -14 -f -D tmpDict --size-hint=11000 tmp tmp2
-$ZSTD -14 -f -o tmp1_.zst -D tmpDict --size-hint=11000 tmp
-$ZSTD -14 -f -o tmp2_.zst -D tmpDict --size-hint=11000 tmp2
-cmp tmp.zst tmp1_.zst || die "first file's output differs"
-cmp tmp2.zst tmp2_.zst || die "second file's output differs"
-println "test : incorrect hinted stream sizes"
-cat tmp | $ZSTD -14 -f --size-hint=11050 | $ZSTD -t  # slightly too high
-cat tmp | $ZSTD -14 -f --size-hint=10950 | $ZSTD -t  # slightly too low
-cat tmp | $ZSTD -14 -f --size-hint=22000 | $ZSTD -t  # considerably too high
-cat tmp | $ZSTD -14 -f --size-hint=5500  | $ZSTD -t  # considerably too low
-
-
-println "\n===>  dictionary tests "
-
-println "- test with raw dict (content only) "
-./datagen > tmpDict
-./datagen -g1M | $MD5SUM > tmp1
-./datagen -g1M | $ZSTD -D tmpDict | $ZSTD -D tmpDict -dvq | $MD5SUM > tmp2
-$DIFF -q tmp1 tmp2
-println "- Create first dictionary "
-TESTFILE="$PRGDIR"/zstdcli.c
-$ZSTD --train "$TESTDIR"/*.c "$PRGDIR"/*.c -o tmpDict
-cp "$TESTFILE" tmp
-println "- Test dictionary compression with tmpDict as an input file and dictionary"
-$ZSTD -f tmpDict -D tmpDict && die "compression error not detected!"
-println "- Dictionary compression roundtrip"
-$ZSTD -f tmp -D tmpDict
-$ZSTD -d tmp.zst -D tmpDict -fo result
-$DIFF "$TESTFILE" result
-println "- Dictionary compression with btlazy2 strategy"
-$ZSTD -f tmp -D tmpDict --zstd=strategy=6
-$ZSTD -d tmp.zst -D tmpDict -fo result
-$DIFF "$TESTFILE" result
-if [ -n "$hasMT" ]
-then
-    println "- Test dictionary compression with multithreading "
-    ./datagen -g5M | $ZSTD -T2 -D tmpDict | $ZSTD -t -D tmpDict   # fails with v1.3.2
-fi
-println "- Create second (different) dictionary "
-$ZSTD --train "$TESTDIR"/*.c "$PRGDIR"/*.c "$PRGDIR"/*.h -o tmpDictC
-$ZSTD -d tmp.zst -D tmpDictC -fo result && die "wrong dictionary not detected!"
-println "- Create dictionary with short dictID"
-$ZSTD --train "$TESTDIR"/*.c "$PRGDIR"/*.c --dictID=1 -o tmpDict1
-cmp tmpDict tmpDict1 && die "dictionaries should have different ID !"
-println "- Create dictionary with wrong dictID parameter order (must fail)"
-$ZSTD --train "$TESTDIR"/*.c "$PRGDIR"/*.c --dictID -o 1 tmpDict1 && die "wrong order : --dictID must be followed by argument "
-println "- Create dictionary with size limit"
-$ZSTD --train "$TESTDIR"/*.c "$PRGDIR"/*.c -o tmpDict2 --maxdict=4K -v
-println "- Create dictionary with small size limit"
-$ZSTD --train "$TESTDIR"/*.c "$PRGDIR"/*.c -o tmpDict3 --maxdict=1K -v
-println "- Create dictionary with wrong parameter order (must fail)"
-$ZSTD --train "$TESTDIR"/*.c "$PRGDIR"/*.c -o tmpDict3 --maxdict -v 4K && die "wrong order : --maxdict must be followed by argument "
-println "- Compress without dictID"
-$ZSTD -f tmp -D tmpDict1 --no-dictID
-$ZSTD -d tmp.zst -D tmpDict -fo result
-$DIFF "$TESTFILE" result
-println "- Compress with wrong argument order (must fail)"
-$ZSTD tmp -Df tmpDict1 -c > $INTOVOID && die "-D must be followed by dictionary name "
-println "- Compress multiple files with dictionary"
-rm -rf dirTestDict
-mkdir dirTestDict
-cp "$TESTDIR"/*.c dirTestDict
-cp "$PRGDIR"/*.c dirTestDict
-cp "$PRGDIR"/*.h dirTestDict
-$MD5SUM dirTestDict/* > tmph1
-$ZSTD -f --rm dirTestDict/* -D tmpDictC
-$ZSTD -d --rm dirTestDict/*.zst -D tmpDictC  # note : use internal checksum by default
-case "$UNAME" in
-  Darwin) println "md5sum -c not supported on OS-X : test skipped" ;;  # not compatible with OS-X's md5
-  *) $MD5SUM -c tmph1 ;;
-esac
-rm -rf dirTestDict
-println "- dictionary builder on bogus input"
-println "Hello World" > tmp
-$ZSTD --train-legacy -q tmp && die "Dictionary training should fail : not enough input source"
-./datagen -P0 -g10M > tmp
-$ZSTD --train-legacy -q tmp && die "Dictionary training should fail : source is pure noise"
-println "- Test -o before --train"
-rm -f tmpDict dictionary
-$ZSTD -o tmpDict --train "$TESTDIR"/*.c "$PRGDIR"/*.c
-test -f tmpDict
-$ZSTD --train "$TESTDIR"/*.c "$PRGDIR"/*.c
-test -f dictionary
-println "- Test dictionary training fails"
-echo "000000000000000000000000000000000" > tmpz
-$ZSTD --train tmpz tmpz tmpz tmpz tmpz tmpz tmpz tmpz tmpz && die "Dictionary training should fail : source is all zeros"
-if [ -n "$hasMT" ]
-then
-  $ZSTD --train -T0 tmpz tmpz tmpz tmpz tmpz tmpz tmpz tmpz tmpz && die "Dictionary training should fail : source is all zeros"
-  println "- Create dictionary with multithreading enabled"
-  $ZSTD --train -T0 "$TESTDIR"/*.c "$PRGDIR"/*.c -o tmpDict
-fi
-rm tmp* dictionary
-
-
-println "\n===>  fastCover dictionary builder : advanced options "
-TESTFILE="$PRGDIR"/zstdcli.c
-./datagen > tmpDict
-println "- Create first dictionary"
-$ZSTD --train-fastcover=k=46,d=8,f=15,split=80 "$TESTDIR"/*.c "$PRGDIR"/*.c -o tmpDict
-cp "$TESTFILE" tmp
-$ZSTD -f tmp -D tmpDict
-$ZSTD -d tmp.zst -D tmpDict -fo result
-$DIFF "$TESTFILE" result
-println "- Create second (different) dictionary"
-$ZSTD --train-fastcover=k=56,d=8 "$TESTDIR"/*.c "$PRGDIR"/*.c "$PRGDIR"/*.h -o tmpDictC
-$ZSTD -d tmp.zst -D tmpDictC -fo result && die "wrong dictionary not detected!"
-$ZSTD --train-fastcover=k=56,d=8 && die "Create dictionary without input file"
-println "- Create dictionary with short dictID"
-$ZSTD --train-fastcover=k=46,d=8,f=15,split=80 "$TESTDIR"/*.c "$PRGDIR"/*.c --dictID=1 -o tmpDict1
-cmp tmpDict tmpDict1 && die "dictionaries should have different ID !"
-println "- Create dictionaries with shrink-dict flag enabled"
-$ZSTD --train-fastcover=steps=1,shrink "$TESTDIR"/*.c "$PRGDIR"/*.c -o tmpShrinkDict
-$ZSTD --train-fastcover=steps=1,shrink=1 "$TESTDIR"/*.c "$PRGDIR"/*.c -o tmpShrinkDict1
-$ZSTD --train-fastcover=steps=1,shrink=5 "$TESTDIR"/*.c "$PRGDIR"/*.c -o tmpShrinkDict2
-println "- Create dictionary with size limit"
-$ZSTD --train-fastcover=steps=1 "$TESTDIR"/*.c "$PRGDIR"/*.c -o tmpDict2 --maxdict=4K
-println "- Create dictionary using all samples for both training and testing"
-$ZSTD --train-fastcover=k=56,d=8,split=100 -r "$TESTDIR"/*.c "$PRGDIR"/*.c
-println "- Create dictionary using f=16"
-$ZSTD --train-fastcover=k=56,d=8,f=16 -r "$TESTDIR"/*.c "$PRGDIR"/*.c
-$ZSTD --train-fastcover=k=56,d=8,accel=15 -r "$TESTDIR"/*.c "$PRGDIR"/*.c && die "Created dictionary using accel=15"
-println "- Create dictionary using accel=2"
-$ZSTD --train-fastcover=k=56,d=8,accel=2 -r "$TESTDIR"/*.c "$PRGDIR"/*.c
-println "- Create dictionary using accel=10"
-$ZSTD --train-fastcover=k=56,d=8,accel=10 -r "$TESTDIR"/*.c "$PRGDIR"/*.c
-println "- Create dictionary with multithreading"
-$ZSTD --train-fastcover -T4 -r "$TESTDIR"/*.c "$PRGDIR"/*.c
-println "- Test -o before --train-fastcover"
-rm -f tmpDict dictionary
-$ZSTD -o tmpDict --train-fastcover=k=56,d=8 "$TESTDIR"/*.c "$PRGDIR"/*.c
-test -f tmpDict
-$ZSTD --train-fastcover=k=56,d=8 "$TESTDIR"/*.c "$PRGDIR"/*.c
-test -f dictionary
-rm tmp* dictionary
-
-
-println "\n===>  legacy dictionary builder "
-
-TESTFILE="$PRGDIR"/zstdcli.c
-./datagen > tmpDict
-println "- Create first dictionary"
-$ZSTD --train-legacy=selectivity=8 "$TESTDIR"/*.c "$PRGDIR"/*.c -o tmpDict
-cp "$TESTFILE" tmp
-$ZSTD -f tmp -D tmpDict
-$ZSTD -d tmp.zst -D tmpDict -fo result
-$DIFF "$TESTFILE" result
-$ZSTD --train-legacy=s=8 && die "Create dictionary without input files (should error)"
-println "- Create second (different) dictionary"
-$ZSTD --train-legacy=s=5 "$TESTDIR"/*.c "$PRGDIR"/*.c "$PRGDIR"/*.h -o tmpDictC
-$ZSTD -d tmp.zst -D tmpDictC -fo result && die "wrong dictionary not detected!"
-println "- Create dictionary with short dictID"
-$ZSTD --train-legacy -s5 "$TESTDIR"/*.c "$PRGDIR"/*.c --dictID=1 -o tmpDict1
-cmp tmpDict tmpDict1 && die "dictionaries should have different ID !"
-println "- Create dictionary with size limit"
-$ZSTD --train-legacy -s9 "$TESTDIR"/*.c "$PRGDIR"/*.c -o tmpDict2 --maxdict=4K
-println "- Test -o before --train-legacy"
-rm -f tmpDict dictionary
-$ZSTD -o tmpDict --train-legacy "$TESTDIR"/*.c "$PRGDIR"/*.c
-test -f tmpDict
-$ZSTD --train-legacy "$TESTDIR"/*.c "$PRGDIR"/*.c
-test -f dictionary
-rm tmp* dictionary
-
-
-println "\n===>  integrity tests "
-
-println "test one file (tmp1.zst) "
-./datagen > tmp1
-$ZSTD tmp1
-$ZSTD -t tmp1.zst
-$ZSTD --test tmp1.zst
-println "test multiple files (*.zst) "
-$ZSTD -t ./*.zst
-println "test bad files (*) "
-$ZSTD -t ./* && die "bad files not detected !"
-$ZSTD -t tmp1 && die "bad file not detected !"
-cp tmp1 tmp2.zst
-$ZSTD -t tmp2.zst && die "bad file not detected !"
-./datagen -g0 > tmp3
-$ZSTD -t tmp3 && die "bad file not detected !"   # detects 0-sized files as bad
-println "test --rm and --test combined "
-$ZSTD -t --rm tmp1.zst
-test -f tmp1.zst   # check file is still present
-split -b16384 tmp1.zst tmpSplit.
-$ZSTD -t tmpSplit.* && die "bad file not detected !"
-./datagen | $ZSTD -c | $ZSTD -t
-
-
-
-println "\n===>  golden files tests "
-
-$ZSTD -t -r "$TESTDIR/golden-compression"
-$ZSTD -c -r "$TESTDIR/golden-compression" | $ZSTD -t
-
-
-println "\n===>  benchmark mode tests "
-
-println "bench one file"
-./datagen > tmp1
-$ZSTD -bi0 tmp1
-println "bench multiple levels"
-$ZSTD -i0b0e3 tmp1
-println "bench negative level"
-$ZSTD -bi0 --fast tmp1
-println "with recursive and quiet modes"
-$ZSTD -rqi0b1e2 tmp1
-println "benchmark decompression only"
-$ZSTD -f tmp1
-$ZSTD -b -d -i0 tmp1.zst
-
-println "\n===>  zstd compatibility tests "
-
-./datagen > tmp
-rm -f tmp.zst
-$ZSTD --format=zstd -f tmp
-test -f tmp.zst
-
-println "\n===>  gzip compatibility tests "
-
-GZIPMODE=1
-$ZSTD --format=gzip -V || GZIPMODE=0
-if [ $GZIPMODE -eq 1 ]; then
-    println "gzip support detected"
-    GZIPEXE=1
-    gzip -V || GZIPEXE=0
-    if [ $GZIPEXE -eq 1 ]; then
-        ./datagen > tmp
-        $ZSTD --format=gzip -f tmp
-        gzip -t -v tmp.gz
-        gzip -f tmp
-        $ZSTD -d -f -v tmp.gz
-        rm tmp*
-    else
-        println "gzip binary not detected"
-    fi
-else
-    println "gzip mode not supported"
-fi
-
-
-println "\n===>  gzip frame tests "
-
-if [ $GZIPMODE -eq 1 ]; then
-    ./datagen > tmp
-    $ZSTD -f --format=gzip tmp
-    $ZSTD -f tmp
-    cat tmp.gz tmp.zst tmp.gz tmp.zst | $ZSTD -d -f -o tmp
-    truncateLastByte tmp.gz | $ZSTD -t > $INTOVOID && die "incomplete frame not detected !"
-    rm tmp*
-else
-    println "gzip mode not supported"
-fi
-
-if [ $GZIPMODE -eq 1 ]; then
-    ./datagen > tmp
-    rm -f tmp.zst
-    $ZSTD --format=gzip --format=zstd -f tmp
-    test -f tmp.zst
-fi
-
-println "\n===>  xz compatibility tests "
-
-LZMAMODE=1
-$ZSTD --format=xz -V || LZMAMODE=0
-if [ $LZMAMODE -eq 1 ]; then
-    println "xz support detected"
-    XZEXE=1
-    xz -Q -V && lzma -Q -V || XZEXE=0
-    if [ $XZEXE -eq 1 ]; then
-        println "Testing zstd xz and lzma support"
-        ./datagen > tmp
-        $ZSTD --format=lzma -f tmp
-        $ZSTD --format=xz -f tmp
-        xz -Q -t -v tmp.xz
-        xz -Q -t -v tmp.lzma
-        xz -Q -f -k tmp
-        lzma -Q -f -k --lzma1 tmp
-        $ZSTD -d -f -v tmp.xz
-        $ZSTD -d -f -v tmp.lzma
-        rm tmp*
-        println "Creating symlinks"
-        ln -s $ZSTD ./xz
-        ln -s $ZSTD ./unxz
-        ln -s $ZSTD ./lzma
-        ln -s $ZSTD ./unlzma
-        println "Testing xz and lzma symlinks"
-        ./datagen > tmp
-        ./xz tmp
-        xz -Q -d tmp.xz
-        ./lzma tmp
-        lzma -Q -d tmp.lzma
-        println "Testing unxz and unlzma symlinks"
-        xz -Q tmp
-        ./xz -d tmp.xz
-        lzma -Q tmp
-        ./lzma -d tmp.lzma
-        rm xz unxz lzma unlzma
-        rm tmp*
-    else
-        println "xz binary not detected"
-    fi
-else
-    println "xz mode not supported"
-fi
-
-
-println "\n===>  xz frame tests "
-
-if [ $LZMAMODE -eq 1 ]; then
-    ./datagen > tmp
-    $ZSTD -f --format=xz tmp
-    $ZSTD -f --format=lzma tmp
-    $ZSTD -f tmp
-    cat tmp.xz tmp.lzma tmp.zst tmp.lzma tmp.xz tmp.zst | $ZSTD -d -f -o tmp
-    truncateLastByte tmp.xz | $ZSTD -t > $INTOVOID && die "incomplete frame not detected !"
-    truncateLastByte tmp.lzma | $ZSTD -t > $INTOVOID && die "incomplete frame not detected !"
-    rm tmp*
-else
-    println "xz mode not supported"
-fi
-
-println "\n===>  lz4 compatibility tests "
-
-LZ4MODE=1
-$ZSTD --format=lz4 -V || LZ4MODE=0
-if [ $LZ4MODE -eq 1 ]; then
-    println "lz4 support detected"
-    LZ4EXE=1
-    lz4 -V || LZ4EXE=0
-    if [ $LZ4EXE -eq 1 ]; then
-        ./datagen > tmp
-        $ZSTD --format=lz4 -f tmp
-        lz4 -t -v tmp.lz4
-        lz4 -f tmp
-        $ZSTD -d -f -v tmp.lz4
-        rm tmp*
-    else
-        println "lz4 binary not detected"
-    fi
-else
-    println "lz4 mode not supported"
-fi
-
-
-println "\n===>  lz4 frame tests "
-
-if [ $LZ4MODE -eq 1 ]; then
-    ./datagen > tmp
-    $ZSTD -f --format=lz4 tmp
-    $ZSTD -f tmp
-    cat tmp.lz4 tmp.zst tmp.lz4 tmp.zst | $ZSTD -d -f -o tmp
-    truncateLastByte tmp.lz4 | $ZSTD -t > $INTOVOID && die "incomplete frame not detected !"
-    rm tmp*
-else
-    println "lz4 mode not supported"
-fi
-
-println "\n===> suffix list test"
-
-! $ZSTD -d tmp.abc 2> tmplg
-
-if [ $GZIPMODE -ne 1 ]; then
-    grep ".gz" tmplg > $INTOVOID && die "Unsupported suffix listed"
-fi
-
-if [ $LZMAMODE -ne 1 ]; then
-    grep ".lzma" tmplg > $INTOVOID && die "Unsupported suffix listed"
-    grep ".xz" tmplg > $INTOVOID && die "Unsupported suffix listed"
-fi
-
-if [ $LZ4MODE -ne 1 ]; then
-    grep ".lz4" tmplg > $INTOVOID && die "Unsupported suffix listed"
-fi
-
-println "\n===>  tar extension tests "
-
-rm -f tmp tmp.tar tmp.tzst tmp.tgz tmp.txz tmp.tlz4
-
-./datagen > tmp
-tar cf tmp.tar tmp
-$ZSTD tmp.tar -o tmp.tzst
-rm tmp.tar
-$ZSTD -d tmp.tzst
-[ -e tmp.tar ] || die ".tzst failed to decompress to .tar!"
-rm -f tmp.tar tmp.tzst
-
-if [ $GZIPMODE -eq 1 ]; then
-    tar czf tmp.tgz tmp
-    $ZSTD -d tmp.tgz
-    [ -e tmp.tar ] || die ".tgz failed to decompress to .tar!"
-    rm -f tmp.tar tmp.tgz
-fi
-
-if [ $LZMAMODE -eq 1 ]; then
-    tar c tmp | $ZSTD --format=xz > tmp.txz
-    $ZSTD -d tmp.txz
-    [ -e tmp.tar ] || die ".txz failed to decompress to .tar!"
-    rm -f tmp.tar tmp.txz
-fi
-
-if [ $LZ4MODE -eq 1 ]; then
-    tar c tmp | $ZSTD --format=lz4 > tmp.tlz4
-    $ZSTD -d tmp.tlz4
-    [ -e tmp.tar ] || die ".tlz4 failed to decompress to .tar!"
-    rm -f tmp.tar tmp.tlz4
-fi
-
-touch tmp.t tmp.tz tmp.tzs
-! $ZSTD -d tmp.t
-! $ZSTD -d tmp.tz
-! $ZSTD -d tmp.tzs
-
-exit
-
-println "\n===>  zstd round-trip tests "
-
-roundTripTest
-roundTripTest -g15K       # TableID==3
-roundTripTest -g127K      # TableID==2
-roundTripTest -g255K      # TableID==1
-roundTripTest -g522K      # TableID==0
-roundTripTest -g519K 6    # greedy, hash chain
-roundTripTest -g517K 16   # btlazy2
-roundTripTest -g516K 19   # btopt
-
-fileRoundTripTest -g500K
-
-println "\n===>  zstd long distance matching round-trip tests "
-roundTripTest -g0 "2 --single-thread --long"
-roundTripTest -g1000K "1 --single-thread --long"
-roundTripTest -g517K "6 --single-thread --long"
-roundTripTest -g516K "16 --single-thread --long"
-roundTripTest -g518K "19 --single-thread --long"
-fileRoundTripTest -g5M "3 --single-thread --long"
-
-
-roundTripTest -g96K "5 --single-thread"
-if [ -n "$hasMT" ]
-then
-    println "\n===>  zstdmt round-trip tests "
-    roundTripTest -g4M "1 -T0"
-    roundTripTest -g8M "3 -T2"
-    roundTripTest -g8000K "2 --threads=2"
-    fileRoundTripTest -g4M "19 -T2 -B1M"
-
-    println "\n===>  zstdmt long distance matching round-trip tests "
-    roundTripTest -g8M "3 --long=24 -T2"
-
-    println "\n===>  ovLog tests "
-    ./datagen -g2MB > tmp
-    refSize=$($ZSTD tmp -6 -c --zstd=wlog=18         | wc -c)
-    ov9Size=$($ZSTD tmp -6 -c --zstd=wlog=18,ovlog=9 | wc -c)
-    ov1Size=$($ZSTD tmp -6 -c --zstd=wlog=18,ovlog=1 | wc -c)
-    if [ "$refSize" -eq "$ov9Size" ]; then
-        echo ov9Size should be different from refSize
-        exit 1
-    fi
-    if [ "$refSize" -eq "$ov1Size" ]; then
-        echo ov1Size should be different from refSize
-        exit 1
-    fi
-    if [ "$ov9Size" -ge "$ov1Size" ]; then
-        echo ov9Size="$ov9Size" should be smaller than ov1Size="$ov1Size"
-        exit 1
-    fi
-
-else
-    println "\n===>  no multithreading, skipping zstdmt tests "
-fi
-
-rm tmp*
-
-println "\n===>  zstd --list/-l single frame tests "
-./datagen > tmp1
-./datagen > tmp2
-./datagen > tmp3
-$ZSTD tmp*
-$ZSTD -l ./*.zst
-$ZSTD -lv ./*.zst | grep "Decompressed Size:"  # check that decompressed size is present in header
-$ZSTD --list ./*.zst
-$ZSTD --list -v ./*.zst
-
-println "\n===>  zstd --list/-l multiple frame tests "
-cat tmp1.zst tmp2.zst > tmp12.zst
-cat tmp12.zst tmp3.zst > tmp123.zst
-$ZSTD -l ./*.zst
-$ZSTD -lv ./*.zst
-
-println "\n===>  zstd --list/-l error detection tests "
-$ZSTD -l tmp1 tmp1.zst && die "-l must fail on non-zstd file"
-$ZSTD --list tmp* && die "-l must fail on non-zstd file"
-$ZSTD -lv tmp1* && die "-l must fail on non-zstd file"
-$ZSTD --list -v tmp2 tmp12.zst && die "-l must fail on non-zstd file"
-
-println "test : detect truncated compressed file "
-TEST_DATA_FILE=truncatable-input.txt
-FULL_COMPRESSED_FILE=${TEST_DATA_FILE}.zst
-TRUNCATED_COMPRESSED_FILE=truncated-input.txt.zst
-./datagen -g50000 > $TEST_DATA_FILE
-$ZSTD -f $TEST_DATA_FILE -o $FULL_COMPRESSED_FILE
-dd bs=1 count=100 if=$FULL_COMPRESSED_FILE of=$TRUNCATED_COMPRESSED_FILE
-$ZSTD --list $TRUNCATED_COMPRESSED_FILE && die "-l must fail on truncated file"
-
-rm $TEST_DATA_FILE
-rm $FULL_COMPRESSED_FILE
-rm $TRUNCATED_COMPRESSED_FILE
-
-println "\n===>  zstd --list/-l errors when presented with stdin / no files"
-$ZSTD -l && die "-l must fail on empty list of files"
-$ZSTD -l - && die "-l does not work on stdin"
-$ZSTD -l < tmp1.zst && die "-l does not work on stdin"
-$ZSTD -l - < tmp1.zst && die "-l does not work on stdin"
-$ZSTD -l - tmp1.zst && die "-l does not work on stdin"
-$ZSTD -l - tmp1.zst < tmp1.zst && die "-l does not work on stdin"
-$ZSTD -l tmp1.zst < tmp2.zst # this will check tmp1.zst, but not tmp2.zst, which is not an error : zstd simply doesn't read stdin in this case. It must not error just because stdin is not a tty
-
-println "\n===>  zstd --list/-l test with null files "
-./datagen -g0 > tmp5
-$ZSTD tmp5
-$ZSTD -l tmp5.zst
-$ZSTD -l tmp5* && die "-l must fail on non-zstd file"
-$ZSTD -lv tmp5.zst | grep "Decompressed Size: 0.00 KB (0 B)"  # check that 0 size is present in header
-$ZSTD -lv tmp5* && die "-l must fail on non-zstd file"
-
-println "\n===>  zstd --list/-l test with no content size field "
-./datagen -g513K | $ZSTD > tmp6.zst
-$ZSTD -l tmp6.zst
-$ZSTD -lv tmp6.zst | grep "Decompressed Size:"  && die "Field :Decompressed Size: should not be available in this compressed file"
-
-println "\n===>   zstd --list/-l test with no checksum "
-$ZSTD -f --no-check tmp1
-$ZSTD -l tmp1.zst
-$ZSTD -lv tmp1.zst
-
-rm tmp*
-
-
-println "\n===>   zstd long distance matching tests "
-roundTripTest -g0 " --single-thread --long"
-roundTripTest -g9M "2 --single-thread --long"
-# Test parameter parsing
-roundTripTest -g1M -P50 "1 --single-thread --long=29" " --memory=512MB"
-roundTripTest -g1M -P50 "1 --single-thread --long=29 --zstd=wlog=28" " --memory=256MB"
-roundTripTest -g1M -P50 "1 --single-thread --long=29" " --long=28 --memory=512MB"
-roundTripTest -g1M -P50 "1 --single-thread --long=29" " --zstd=wlog=28 --memory=512MB"
-
-
-if [ -n "$hasMT" ]
-then
-    println "\n===>   adaptive mode "
-    roundTripTest -g270000000 " --adapt"
-    roundTripTest -g27000000 " --adapt=min=1,max=4"
-    println "===>   test: --adapt must fail on incoherent bounds "
-    ./datagen > tmp
-    $ZSTD -f -vv --adapt=min=10,max=9 tmp && die "--adapt must fail on incoherent bounds"
-
-    println "\n===>   rsyncable mode "
-    roundTripTest -g10M " --rsyncable"
-    roundTripTest -g10M " --rsyncable -B100K"
-    println "===>   test: --rsyncable must fail with --single-thread"
-    $ZSTD -f -vv --rsyncable --single-thread tmp && die "--rsyncable must fail with --single-thread"
-fi
-
-
-if [ "$1" != "--test-large-data" ]; then
-    println "Skipping large data tests"
-    exit 0
-fi
-
-
-#############################################################################
-
-println "\n===>   large files tests "
-
-roundTripTest -g270000000 1
-roundTripTest -g250000000 2
-roundTripTest -g230000000 3
-
-roundTripTest -g140000000 -P60 4
-roundTripTest -g130000000 -P62 5
-roundTripTest -g120000000 -P65 6
-
-roundTripTest -g70000000 -P70 7
-roundTripTest -g60000000 -P71 8
-roundTripTest -g50000000 -P73 9
-
-roundTripTest -g35000000 -P75 10
-roundTripTest -g30000000 -P76 11
-roundTripTest -g25000000 -P78 12
-
-roundTripTest -g18000013 -P80 13
-roundTripTest -g18000014 -P80 14
-roundTripTest -g18000015 -P81 15
-roundTripTest -g18000016 -P84 16
-roundTripTest -g18000017 -P88 17
-roundTripTest -g18000018 -P94 18
-roundTripTest -g18000019 -P96 19
-
-roundTripTest -g5000000000 -P99 1
-roundTripTest -g1700000000 -P0 "1 --zstd=strategy=6"   # ensure btlazy2 can survive an overflow rescale
-
-fileRoundTripTest -g4193M -P99 1
-
-
-println "\n===>   zstd long, long distance matching round-trip tests "
-roundTripTest -g270000000 "1 --single-thread --long"
-roundTripTest -g130000000 -P60 "5 --single-thread --long"
-roundTripTest -g35000000 -P70 "8 --single-thread --long"
-roundTripTest -g18000001 -P80  "18 --single-thread --long"
-# Test large window logs
-roundTripTest -g700M -P50 "1 --single-thread --long=29"
-roundTripTest -g600M -P50 "1 --single-thread --long --zstd=wlog=29,clog=28"
-
-
-if [ -n "$hasMT" ]
-then
-    println "\n===>   zstdmt long round-trip tests "
-    roundTripTest -g80000000 -P99 "19 -T2" " "
-    roundTripTest -g5000000000 -P99 "1 -T2" " "
-    roundTripTest -g500000000 -P97 "1 -T999" " "
-    fileRoundTripTest -g4103M -P98 " -T0" " "
-    roundTripTest -g400000000 -P97 "1 --long=24 -T2" " "
-    # Exposes the bug in https://github.com/facebook/zstd/pull/1678
-    # This test fails on 4 different travis builds at the time of writing
-    # because it needs to allocate 8 GB of memory.
-    # roundTripTest -g10G -P99 "1 -T1 --long=31 --zstd=clog=27 --fast=1000"
-else
-    println "\n**** no multithreading, skipping zstdmt tests **** "
-fi
-
-
-println "\n===>  cover dictionary builder : advanced options "
-
-TESTFILE="$PRGDIR"/zstdcli.c
-./datagen > tmpDict
-println "- Create first dictionary"
-$ZSTD --train-cover=k=46,d=8,split=80 "$TESTDIR"/*.c "$PRGDIR"/*.c -o tmpDict
-cp "$TESTFILE" tmp
-$ZSTD -f tmp -D tmpDict
-$ZSTD -d tmp.zst -D tmpDict -fo result
-$DIFF "$TESTFILE" result
-$ZSTD --train-cover=k=56,d=8 && die "Create dictionary without input file (should error)"
-println "- Create second (different) dictionary"
-$ZSTD --train-cover=k=56,d=8 "$TESTDIR"/*.c "$PRGDIR"/*.c "$PRGDIR"/*.h -o tmpDictC
-$ZSTD -d tmp.zst -D tmpDictC -fo result && die "wrong dictionary not detected!"
-println "- Create dictionary using shrink-dict flag"
-$ZSTD --train-cover=steps=256,shrink "$TESTDIR"/*.c "$PRGDIR"/*.c --dictID=1 -o tmpShrinkDict
-$ZSTD --train-cover=steps=256,shrink=1 "$TESTDIR"/*.c "$PRGDIR"/*.c --dictID=1 -o tmpShrinkDict1
-$ZSTD --train-cover=steps=256,shrink=5 "$TESTDIR"/*.c "$PRGDIR"/*.c --dictID=1 -o tmpShrinkDict2
-println "- Create dictionary with short dictID"
-$ZSTD --train-cover=k=46,d=8,split=80 "$TESTDIR"/*.c "$PRGDIR"/*.c --dictID=1 -o tmpDict1
-cmp tmpDict tmpDict1 && die "dictionaries should have different ID !"
-println "- Create dictionary with size limit"
-$ZSTD --train-cover=steps=8 "$TESTDIR"/*.c "$PRGDIR"/*.c -o tmpDict2 --maxdict=4K
-println "- Compare size of dictionary from 90% training samples with 80% training samples"
-$ZSTD --train-cover=split=90 -r "$TESTDIR"/*.c "$PRGDIR"/*.c
-$ZSTD --train-cover=split=80 -r "$TESTDIR"/*.c "$PRGDIR"/*.c
-println "- Create dictionary using all samples for both training and testing"
-$ZSTD --train-cover=split=100 -r "$TESTDIR"/*.c "$PRGDIR"/*.c
-println "- Test -o before --train-cover"
-rm -f tmpDict dictionary
-$ZSTD -o tmpDict --train-cover "$TESTDIR"/*.c "$PRGDIR"/*.c
-test -f tmpDict
-$ZSTD --train-cover "$TESTDIR"/*.c "$PRGDIR"/*.c
-test -f dictionary
-rm -f tmp* dictionary
-
-
-if [ "$isWindows" = false ] ; then
-
-println "\n===>  zstd fifo named pipe test "
-head -c 10 /dev/zero > tmp_original
-mkfifo named_pipe
-head -c 10 /dev/zero > named_pipe &
-$ZSTD named_pipe -o tmp_compressed
-$ZSTD -d -o tmp_decompressed tmp_compressed
-$DIFF -s tmp_original tmp_decompressed
-rm -rf tmp*
-rm -rf named_pipe
-
-fi
-
-rm -f tmp*
diff --git a/tests/poolTests.c b/tests/poolTests.c
deleted file mode 100644
index 02ec62af9f37..000000000000
--- a/tests/poolTests.c
+++ /dev/null
@@ -1,271 +0,0 @@
-/*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-
-#include "pool.h"
-#include "threading.h"
-#include "util.h"
-#include "timefn.h"
-#include <stddef.h>
-#include <stdio.h>
-
-#define ASSERT_TRUE(p)                                                       \
-  do {                                                                       \
-    if (!(p)) {                                                              \
-      return 1;                                                              \
-    }                                                                        \
-  } while (0)
-#define ASSERT_FALSE(p) ASSERT_TRUE(!(p))
-#define ASSERT_EQ(lhs, rhs) ASSERT_TRUE((lhs) == (rhs))
-
-struct data {
-  ZSTD_pthread_mutex_t mutex;
-  unsigned data[16];
-  size_t i;
-};
-
-static void fn(void *opaque)
-{
-  struct data *data = (struct data *)opaque;
-  ZSTD_pthread_mutex_lock(&data->mutex);
-  data->data[data->i] = (unsigned)(data->i);
-  ++data->i;
-  ZSTD_pthread_mutex_unlock(&data->mutex);
-}
-
-static int testOrder(size_t numThreads, size_t queueSize)
-{
-  struct data data;
-  POOL_ctx* const ctx = POOL_create(numThreads, queueSize);
-  ASSERT_TRUE(ctx);
-  data.i = 0;
-  ASSERT_FALSE(ZSTD_pthread_mutex_init(&data.mutex, NULL));
-  { size_t i;
-    for (i = 0; i < 16; ++i) {
-      POOL_add(ctx, &fn, &data);
-    }
-  }
-  POOL_free(ctx);
-  ASSERT_EQ(16, data.i);
-  { size_t i;
-    for (i = 0; i < data.i; ++i) {
-      ASSERT_EQ(i, data.data[i]);
-    }
-  }
-  ZSTD_pthread_mutex_destroy(&data.mutex);
-  return 0;
-}
-
-
-/* --- test deadlocks --- */
-
-static void waitFn(void *opaque) {
-  (void)opaque;
-  UTIL_sleepMilli(1);
-}
-
-/* Tests for deadlock */
-static int testWait(size_t numThreads, size_t queueSize) {
-  struct data data;
-  POOL_ctx* const ctx = POOL_create(numThreads, queueSize);
-  ASSERT_TRUE(ctx);
-  { size_t i;
-    for (i = 0; i < 16; ++i) {
-        POOL_add(ctx, &waitFn, &data);
-    }
-  }
-  POOL_free(ctx);
-  return 0;
-}
-
-
-/* --- test POOL_resize() --- */
-
-typedef struct {
-    ZSTD_pthread_mutex_t mut;
-    int countdown;
-    int val;
-    int max;
-    ZSTD_pthread_cond_t cond;
-} poolTest_t;
-
-static void waitLongFn(void *opaque) {
-  poolTest_t* const test = (poolTest_t*) opaque;
-  ZSTD_pthread_mutex_lock(&test->mut);
-  test->val++;
-  if (test->val > test->max)
-      test->max = test->val;
-  ZSTD_pthread_mutex_unlock(&test->mut);
-
-  UTIL_sleepMilli(10);
-
-  ZSTD_pthread_mutex_lock(&test->mut);
-  test->val--;
-  test->countdown--;
-  if (test->countdown == 0)
-      ZSTD_pthread_cond_signal(&test->cond);
-  ZSTD_pthread_mutex_unlock(&test->mut);
-}
-
-static int testThreadReduction_internal(POOL_ctx* ctx, poolTest_t test)
-{
-    int const nbWaits = 16;
-
-    test.countdown = nbWaits;
-    test.val = 0;
-    test.max = 0;
-
-    {   int i;
-        for (i=0; i<nbWaits; i++)
-            POOL_add(ctx, &waitLongFn, &test);
-    }
-    ZSTD_pthread_mutex_lock(&test.mut);
-    while (test.countdown > 0)
-        ZSTD_pthread_cond_wait(&test.cond, &test.mut);
-    ASSERT_EQ(test.val, 0);
-    ASSERT_EQ(test.max, 4);
-    ZSTD_pthread_mutex_unlock(&test.mut);
-
-    ASSERT_EQ( POOL_resize(ctx, 2/*nbThreads*/) , 0 );
-    test.countdown = nbWaits;
-    test.val = 0;
-    test.max = 0;
-    {   int i;
-        for (i=0; i<nbWaits; i++)
-            POOL_add(ctx, &waitLongFn, &test);
-    }
-    ZSTD_pthread_mutex_lock(&test.mut);
-    while (test.countdown > 0)
-        ZSTD_pthread_cond_wait(&test.cond, &test.mut);
-    ASSERT_EQ(test.val, 0);
-    ASSERT_EQ(test.max, 2);
-    ZSTD_pthread_mutex_unlock(&test.mut);
-
-    return 0;
-}
-
-static int testThreadReduction(void) {
-    int result;
-    poolTest_t test;
-    POOL_ctx* const ctx = POOL_create(4 /*nbThreads*/, 2 /*queueSize*/);
-
-    ASSERT_TRUE(ctx);
-
-    memset(&test, 0, sizeof(test));
-    ASSERT_FALSE( ZSTD_pthread_mutex_init(&test.mut, NULL) );
-    ASSERT_FALSE( ZSTD_pthread_cond_init(&test.cond, NULL) );
-
-    result = testThreadReduction_internal(ctx, test);
-
-    ZSTD_pthread_mutex_destroy(&test.mut);
-    ZSTD_pthread_cond_destroy(&test.cond);
-    POOL_free(ctx);
-
-    return result;
-}
-
-
-/* --- test abrupt ending --- */
-
-typedef struct {
-    ZSTD_pthread_mutex_t mut;
-    int val;
-} abruptEndCanary_t;
-
-static void waitIncFn(void *opaque) {
-  abruptEndCanary_t* test = (abruptEndCanary_t*) opaque;
-  UTIL_sleepMilli(10);
-  ZSTD_pthread_mutex_lock(&test->mut);
-  test->val = test->val + 1;
-  ZSTD_pthread_mutex_unlock(&test->mut);
-}
-
-static int testAbruptEnding_internal(abruptEndCanary_t test)
-{
-    int const nbWaits = 16;
-
-    POOL_ctx* const ctx = POOL_create(3 /*numThreads*/, nbWaits /*queueSize*/);
-    ASSERT_TRUE(ctx);
-    test.val = 0;
-
-    {   int i;
-        for (i=0; i<nbWaits; i++)
-            POOL_add(ctx, &waitIncFn, &test);  /* all jobs pushed into queue */
-    }
-    ASSERT_EQ( POOL_resize(ctx, 1 /*numThreads*/) , 0 );   /* downsize numThreads, to try to break end condition */
-
-    POOL_free(ctx);  /* must finish all jobs in queue before giving back control */
-    ASSERT_EQ(test.val, nbWaits);
-    return 0;
-}
-
-static int testAbruptEnding(void) {
-    int result;
-    abruptEndCanary_t test;
-
-    memset(&test, 0, sizeof(test));
-    ASSERT_FALSE( ZSTD_pthread_mutex_init(&test.mut, NULL) );
-
-    result = testAbruptEnding_internal(test);
-
-    ZSTD_pthread_mutex_destroy(&test.mut);
-    return result;
-}
-
-
-
-/* --- test launcher --- */
-
-int main(int argc, const char **argv) {
-  size_t numThreads;
-  (void)argc;
-  (void)argv;
-
-  if (POOL_create(0, 1)) {   /* should not be possible */
-    printf("FAIL: should not create POOL with 0 threads\n");
-    return 1;
-  }
-
-  for (numThreads = 1; numThreads <= 4; ++numThreads) {
-    size_t queueSize;
-    for (queueSize = 0; queueSize <= 2; ++queueSize) {
-      printf("queueSize==%u, numThreads=%u \n",
-            (unsigned)queueSize, (unsigned)numThreads);
-      if (testOrder(numThreads, queueSize)) {
-        printf("FAIL: testOrder\n");
-        return 1;
-      }
-      printf("SUCCESS: testOrder\n");
-      if (testWait(numThreads, queueSize)) {
-        printf("FAIL: testWait\n");
-        return 1;
-      }
-      printf("SUCCESS: testWait\n");
-    }
-  }
-
-  if (testThreadReduction()) {
-      printf("FAIL: thread reduction not effective \n");
-      return 1;
-  } else {
-      printf("SUCCESS: thread reduction effective \n");
-  }
-
-  if (testAbruptEnding()) {
-      printf("FAIL: jobs in queue not completed on early end \n");
-      return 1;
-  } else {
-      printf("SUCCESS: all jobs in queue completed on early end \n");
-  }
-
-  printf("PASS: all POOL tests\n");
-
-  return 0;
-}
diff --git a/tests/rateLimiter.py b/tests/rateLimiter.py
deleted file mode 100755
index da0baf01464f..000000000000
--- a/tests/rateLimiter.py
+++ /dev/null
@@ -1,40 +0,0 @@
-#!/usr/bin/env python3
-
-# ################################################################
-# Copyright (c) 2018-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under both the BSD-style license (found in the
-# LICENSE file in the root directory of this source tree) and the GPLv2 (found
-# in the COPYING file in the root directory of this source tree).
-# ##########################################################################
-
-# Rate limiter, replacement for pv
-# this rate limiter does not "catch up" after a blocking period
-# Limitations:
-# - only accepts limit speed in MB/s
-
-import sys
-import time
-
-MB = 1024 * 1024
-rate = float(sys.argv[1]) * MB
-start = time.time()
-total_read = 0
-
-# sys.stderr.close()  # remove error message, for Ctrl+C
-
-try:
-  buf = " "
-  while len(buf):
-    now = time.time()
-    to_read = max(int(rate * (now - start)), 1)
-    max_buf_size = 1 * MB
-    to_read = min(to_read, max_buf_size)
-    start = now
-
-    buf = sys.stdin.buffer.read(to_read)
-    sys.stdout.buffer.write(buf)
-
-except (KeyboardInterrupt, BrokenPipeError) as e:
-    pass
diff --git a/tests/regression/Makefile b/tests/regression/Makefile
deleted file mode 100644
index 03e5b0af1d70..000000000000
--- a/tests/regression/Makefile
+++ /dev/null
@@ -1,58 +0,0 @@
-# ################################################################
-# Copyright (c) 2015-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under both the BSD-style license (found in the
-# LICENSE file in the root directory of this source tree) and the GPLv2 (found
-# in the COPYING file in the root directory of this source tree).
-# ################################################################
-
-CFLAGS ?= -O3
-
-CURL_CFLAGS := $(shell curl-config --cflags)
-CURL_LDFLAGS := $(shell curl-config --libs) -pthread
-
-PROGDIR := ../../programs
-LIBDIR := ../../lib
-ZSTD_CPPFLAGS := -I$(PROGDIR) -I$(LIBDIR) -I$(LIBDIR)/common
-
-REGRESSION_CFLAGS = $(CFLAGS) $(CURL_CFLAGS)
-REGRESSION_CPPFLAGS = $(CPPFLAGS) $(ZSTD_CPPFLAGS)
-REGRESSION_LDFLAGS = $(LDFLAGS) $(CURL_LDFLAGS)
-
-all: test
-
-xxhash.o: $(LIBDIR)/common/xxhash.c $(LIBDIR)/common/xxhash.h
-	$(CC) $(REGRESSION_CFLAGS) $(REGRESSION_CPPFLAGS) $< -c -o $@
-
-util.o: $(PROGDIR)/util.c $(PROGDIR)/util.h
-	$(CC) $(REGRESSION_CFLAGS) $(REGRESSION_CPPFLAGS) $< -c -o $@
-
-data.o: data.c data.h $(PROGDIR)/util.h $(LIBDIR)/common/xxhash.h
-	$(CC) $(REGRESSION_CFLAGS) $(REGRESSION_CPPFLAGS) $< -c -o $@
-
-config.o: config.c config.h levels.h
-	$(CC) $(REGRESSION_CFLAGS) $(REGRESSION_CPPFLAGS) $< -c -o $@
-
-method.h: data.h config.h result.h
-
-method.o: method.c method.h
-	$(CC) $(REGRESSION_CFLAGS) $(REGRESSION_CPPFLAGS) $< -c -o $@
-
-result.o: result.c result.h
-	$(CC) $(REGRESSION_CFLAGS) $(REGRESSION_CPPFLAGS) $< -c -o $@
-
-test.o: test.c data.h config.h method.h
-	$(CC) $(REGRESSION_CFLAGS) $(REGRESSION_CPPFLAGS) $< -c -o $@
-
-libzstd.a:
-	$(MAKE) -C $(LIBDIR) libzstd.a-mt
-	cp $(LIBDIR)/libzstd.a .
-
-test: test.o data.o config.o util.o method.o result.o xxhash.o libzstd.a
-	$(CC) $^ $(REGRESSION_LDFLAGS) -o $@
-
-.PHONY: clean
-clean:
-	$(MAKE) -C $(LIBDIR) clean
-	$(RM) *.o *.a test
diff --git a/tests/regression/config.c b/tests/regression/config.c
deleted file mode 100644
index b82482f46123..000000000000
--- a/tests/regression/config.c
+++ /dev/null
@@ -1,278 +0,0 @@
-/*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#include "config.h"
-
-/* Define a config for each fast level we want to test with. */
-#define FAST_LEVEL(x)                                               \
-    param_value_t const level_fast##x##_param_values[] = {          \
-        {.param = ZSTD_c_compressionLevel, .value = -x},            \
-    };                                                              \
-    config_t const level_fast##x = {                                \
-        .name = "level -" #x,                                       \
-        .cli_args = "--fast=" #x,                                   \
-        .param_values = PARAM_VALUES(level_fast##x##_param_values), \
-    };                                                              \
-    config_t const level_fast##x##_dict = {                         \
-        .name = "level -" #x " with dict",                          \
-        .cli_args = "--fast=" #x,                                   \
-        .param_values = PARAM_VALUES(level_fast##x##_param_values), \
-        .use_dictionary = 1,                                        \
-    };
-
-/* Define a config for each level we want to test with. */
-#define LEVEL(x)                                                \
-    param_value_t const level_##x##_param_values[] = {          \
-        {.param = ZSTD_c_compressionLevel, .value = x},         \
-    };                                                          \
-    config_t const level_##x = {                                \
-        .name = "level " #x,                                    \
-        .cli_args = "-" #x,                                     \
-        .param_values = PARAM_VALUES(level_##x##_param_values), \
-    };                                                          \
-    config_t const level_##x##_dict = {                         \
-        .name = "level " #x " with dict",                       \
-        .cli_args = "-" #x,                                     \
-        .param_values = PARAM_VALUES(level_##x##_param_values), \
-        .use_dictionary = 1,                                    \
-    };
-
-#define PARAM_VALUES(pv) \
-    { .data = pv, .size = sizeof(pv) / sizeof((pv)[0]) }
-
-#include "levels.h"
-
-#undef LEVEL
-#undef FAST_LEVEL
-
-static config_t no_pledged_src_size = {
-    .name = "no source size",
-    .cli_args = "",
-    .param_values = PARAM_VALUES(level_0_param_values),
-    .no_pledged_src_size = 1,
-};
-
-static param_value_t const ldm_param_values[] = {
-    {.param = ZSTD_c_enableLongDistanceMatching, .value = 1},
-};
-
-static config_t ldm = {
-    .name = "long distance mode",
-    .cli_args = "--long",
-    .param_values = PARAM_VALUES(ldm_param_values),
-};
-
-static param_value_t const mt_param_values[] = {
-    {.param = ZSTD_c_nbWorkers, .value = 2},
-};
-
-static config_t mt = {
-    .name = "multithreaded",
-    .cli_args = "-T2",
-    .param_values = PARAM_VALUES(mt_param_values),
-};
-
-static param_value_t const mt_ldm_param_values[] = {
-    {.param = ZSTD_c_nbWorkers, .value = 2},
-    {.param = ZSTD_c_enableLongDistanceMatching, .value = 1},
-};
-
-static config_t mt_ldm = {
-    .name = "multithreaded long distance mode",
-    .cli_args = "-T2 --long",
-    .param_values = PARAM_VALUES(mt_ldm_param_values),
-};
-
-static param_value_t mt_advanced_param_values[] = {
-    {.param = ZSTD_c_nbWorkers, .value = 2},
-    {.param = ZSTD_c_literalCompressionMode, .value = ZSTD_lcm_uncompressed},
-};
-
-static config_t mt_advanced = {
-    .name = "multithreaded with advanced params",
-    .cli_args = "-T2 --no-compress-literals",
-    .param_values = PARAM_VALUES(mt_advanced_param_values),
-};
-
-static param_value_t const small_wlog_param_values[] = {
-    {.param = ZSTD_c_windowLog, .value = 10},
-};
-
-static config_t small_wlog = {
-    .name = "small window log",
-    .cli_args = "--zstd=wlog=10",
-    .param_values = PARAM_VALUES(small_wlog_param_values),
-};
-
-static param_value_t const small_hlog_param_values[] = {
-    {.param = ZSTD_c_hashLog, .value = 6},
-    {.param = ZSTD_c_strategy, .value = (int)ZSTD_btopt},
-};
-
-static config_t small_hlog = {
-    .name = "small hash log",
-    .cli_args = "--zstd=hlog=6,strat=7",
-    .param_values = PARAM_VALUES(small_hlog_param_values),
-};
-
-static param_value_t const small_clog_param_values[] = {
-    {.param = ZSTD_c_chainLog, .value = 6},
-    {.param = ZSTD_c_strategy, .value = (int)ZSTD_btopt},
-};
-
-static config_t small_clog = {
-    .name = "small chain log",
-    .cli_args = "--zstd=clog=6,strat=7",
-    .param_values = PARAM_VALUES(small_clog_param_values),
-};
-
-static param_value_t const uncompressed_literals_param_values[] = {
-    {.param = ZSTD_c_compressionLevel, .value = 3},
-    {.param = ZSTD_c_literalCompressionMode, .value = ZSTD_lcm_uncompressed},
-};
-
-static config_t uncompressed_literals = {
-    .name = "uncompressed literals",
-    .cli_args = "-3 --no-compress-literals",
-    .param_values = PARAM_VALUES(uncompressed_literals_param_values),
-};
-
-static param_value_t const uncompressed_literals_opt_param_values[] = {
-    {.param = ZSTD_c_compressionLevel, .value = 19},
-    {.param = ZSTD_c_literalCompressionMode, .value = ZSTD_lcm_uncompressed},
-};
-
-static config_t uncompressed_literals_opt = {
-    .name = "uncompressed literals optimal",
-    .cli_args = "-19 --no-compress-literals",
-    .param_values = PARAM_VALUES(uncompressed_literals_opt_param_values),
-};
-
-static param_value_t const huffman_literals_param_values[] = {
-    {.param = ZSTD_c_compressionLevel, .value = -1},
-    {.param = ZSTD_c_literalCompressionMode, .value = ZSTD_lcm_huffman},
-};
-
-static config_t huffman_literals = {
-    .name = "huffman literals",
-    .cli_args = "--fast=1 --compress-literals",
-    .param_values = PARAM_VALUES(huffman_literals_param_values),
-};
-
-static param_value_t const explicit_params_param_values[] = {
-    {.param = ZSTD_c_checksumFlag, .value = 1},
-    {.param = ZSTD_c_contentSizeFlag, .value = 0},
-    {.param = ZSTD_c_dictIDFlag, .value = 0},
-    {.param = ZSTD_c_strategy, .value = (int)ZSTD_greedy},
-    {.param = ZSTD_c_windowLog, .value = 18},
-    {.param = ZSTD_c_hashLog, .value = 21},
-    {.param = ZSTD_c_chainLog, .value = 21},
-    {.param = ZSTD_c_targetLength, .value = 100},
-};
-
-static config_t explicit_params = {
-    .name = "explicit params",
-    .cli_args = "--no-check --no-dictID --zstd=strategy=3,wlog=18,hlog=21,clog=21,tlen=100",
-    .param_values = PARAM_VALUES(explicit_params_param_values),
-};
-
-static config_t const* g_configs[] = {
-
-#define FAST_LEVEL(x) &level_fast##x, &level_fast##x##_dict,
-#define LEVEL(x) &level_##x, &level_##x##_dict,
-#include "levels.h"
-#undef LEVEL
-#undef FAST_LEVEL
-
-    &no_pledged_src_size,
-    &ldm,
-    &mt,
-    &mt_ldm,
-    &small_wlog,
-    &small_hlog,
-    &small_clog,
-    &explicit_params,
-    &uncompressed_literals,
-    &uncompressed_literals_opt,
-    &huffman_literals,
-    &mt_advanced,
-    NULL,
-};
-
-config_t const* const* configs = g_configs;
-
-int config_skip_data(config_t const* config, data_t const* data) {
-    return config->use_dictionary && !data_has_dict(data);
-}
-
-int config_get_level(config_t const* config)
-{
-    param_values_t const params = config->param_values;
-    size_t i;
-    for (i = 0; i < params.size; ++i) {
-        if (params.data[i].param == ZSTD_c_compressionLevel)
-            return (int)params.data[i].value;
-    }
-    return CONFIG_NO_LEVEL;
-}
-
-ZSTD_parameters config_get_zstd_params(
-    config_t const* config,
-    uint64_t srcSize,
-    size_t dictSize)
-{
-    ZSTD_parameters zparams = {};
-    param_values_t const params = config->param_values;
-    int level = config_get_level(config);
-    if (level == CONFIG_NO_LEVEL)
-        level = 3;
-    zparams = ZSTD_getParams(
-        level,
-        config->no_pledged_src_size ? ZSTD_CONTENTSIZE_UNKNOWN : srcSize,
-        dictSize);
-    for (size_t i = 0; i < params.size; ++i) {
-        unsigned const value = params.data[i].value;
-        switch (params.data[i].param) {
-            case ZSTD_c_contentSizeFlag:
-                zparams.fParams.contentSizeFlag = value;
-                break;
-            case ZSTD_c_checksumFlag:
-                zparams.fParams.checksumFlag = value;
-                break;
-            case ZSTD_c_dictIDFlag:
-                zparams.fParams.noDictIDFlag = !value;
-                break;
-            case ZSTD_c_windowLog:
-                zparams.cParams.windowLog = value;
-                break;
-            case ZSTD_c_chainLog:
-                zparams.cParams.chainLog = value;
-                break;
-            case ZSTD_c_hashLog:
-                zparams.cParams.hashLog = value;
-                break;
-            case ZSTD_c_searchLog:
-                zparams.cParams.searchLog = value;
-                break;
-            case ZSTD_c_minMatch:
-                zparams.cParams.minMatch = value;
-                break;
-            case ZSTD_c_targetLength:
-                zparams.cParams.targetLength = value;
-                break;
-            case ZSTD_c_strategy:
-                zparams.cParams.strategy = (ZSTD_strategy)value;
-                break;
-            default:
-                break;
-        }
-    }
-    return zparams;
-}
diff --git a/tests/regression/config.h b/tests/regression/config.h
deleted file mode 100644
index 3cd0308a0987..000000000000
--- a/tests/regression/config.h
+++ /dev/null
@@ -1,86 +0,0 @@
-/*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#ifndef CONFIG_H
-#define CONFIG_H
-
-#include <stddef.h>
-
-#define ZSTD_STATIC_LINKING_ONLY
-#include <zstd.h>
-
-#include "data.h"
-
-typedef struct {
-    ZSTD_cParameter param;
-    int value;
-} param_value_t;
-
-typedef struct {
-    size_t size;
-    param_value_t const* data;
-} param_values_t;
-
-/**
- * The config tells the compression method what options to use.
- */
-typedef struct {
-    const char* name;  /**< Identifies the config in the results table */
-    /**
-     * Optional arguments to pass to the CLI. If not set, CLI-based methods
-     * will skip this config.
-     */
-    char const* cli_args;
-    /**
-     * Parameters to pass to the advanced API. If the advanced API isn't used,
-     * the parameters will be derived from these.
-     */
-    param_values_t param_values;
-    /**
-     * Boolean parameter that says if we should use a dictionary. If the data
-     * doesn't have a dictionary, this config is skipped. Defaults to no.
-     */
-    int use_dictionary;
-    /**
-     * Boolean parameter that says if we should pass the pledged source size
-     * when the method allows it. Defaults to yes.
-     */
-    int no_pledged_src_size;
-} config_t;
-
-/**
- * Returns true if the config should skip this data.
- * For instance, if the config requires a dictionary but the data doesn't have
- * one.
- */
-int config_skip_data(config_t const* config, data_t const* data);
-
-#define CONFIG_NO_LEVEL (-ZSTD_TARGETLENGTH_MAX - 1)
-/**
- * Returns the compression level specified by the config, or CONFIG_NO_LEVEL if
- * no level is specified. Note that 0 is a valid compression level, meaning
- * default.
- */
-int config_get_level(config_t const* config);
-
-/**
- * Returns the compression parameters specified by the config.
- */
-ZSTD_parameters config_get_zstd_params(
-    config_t const* config,
-    uint64_t srcSize,
-    size_t dictSize);
-
-/**
- * The NULL-terminated list of configs.
- */
-extern config_t const* const* configs;
-
-#endif
diff --git a/tests/regression/data.c b/tests/regression/data.c
deleted file mode 100644
index 86e7687de5ee..000000000000
--- a/tests/regression/data.c
+++ /dev/null
@@ -1,617 +0,0 @@
-/*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#include "data.h"
-
-#include <assert.h>
-#include <errno.h>
-#include <stdio.h>
-#include <string.h>
-
-#include <sys/stat.h>
-
-#include <curl/curl.h>
-
-#include "mem.h"
-#include "util.h"
-#define XXH_STATIC_LINKING_ONLY
-#include "xxhash.h"
-
-/**
- * Data objects
- */
-
-#define REGRESSION_RELEASE(x) \
-    "https://github.com/facebook/zstd/releases/download/regression-data/" x
-
-data_t silesia = {
-    .name = "silesia",
-    .type = data_type_dir,
-    .data =
-        {
-            .url = REGRESSION_RELEASE("silesia.tar.zst"),
-            .xxhash64 = 0x48a199f92f93e977LL,
-        },
-};
-
-data_t silesia_tar = {
-    .name = "silesia.tar",
-    .type = data_type_file,
-    .data =
-        {
-            .url = REGRESSION_RELEASE("silesia.tar.zst"),
-            .xxhash64 = 0x48a199f92f93e977LL,
-        },
-};
-
-data_t github = {
-    .name = "github",
-    .type = data_type_dir,
-    .data =
-        {
-            .url = REGRESSION_RELEASE("github.tar.zst"),
-            .xxhash64 = 0xa9b1b44b020df292LL,
-        },
-    .dict =
-        {
-            .url = REGRESSION_RELEASE("github.dict.zst"),
-            .xxhash64 = 0x1eddc6f737d3cb53LL,
-
-        },
-};
-
-static data_t* g_data[] = {
-    &silesia,
-    &silesia_tar,
-    &github,
-    NULL,
-};
-
-data_t const* const* data = (data_t const* const*)g_data;
-
-/**
- * data helpers.
- */
-
-int data_has_dict(data_t const* data) {
-    return data->dict.url != NULL;
-}
-
-/**
- * data buffer helper functions (documented in header).
- */
-
-data_buffer_t data_buffer_create(size_t const capacity) {
-    data_buffer_t buffer = {};
-
-    buffer.data = (uint8_t*)malloc(capacity);
-    if (buffer.data == NULL)
-        return buffer;
-    buffer.capacity = capacity;
-    return buffer;
-}
-
-data_buffer_t data_buffer_read(char const* filename) {
-    data_buffer_t buffer = {};
-
-    uint64_t const size = UTIL_getFileSize(filename);
-    if (size == UTIL_FILESIZE_UNKNOWN) {
-        fprintf(stderr, "unknown size for %s\n", filename);
-        return buffer;
-    }
-
-    buffer.data = (uint8_t*)malloc(size);
-    if (buffer.data == NULL) {
-        fprintf(stderr, "malloc failed\n");
-        return buffer;
-    }
-    buffer.capacity = size;
-
-    FILE* file = fopen(filename, "rb");
-    if (file == NULL) {
-        fprintf(stderr, "file null\n");
-        goto err;
-    }
-    buffer.size = fread(buffer.data, 1, buffer.capacity, file);
-    fclose(file);
-    if (buffer.size != buffer.capacity) {
-        fprintf(stderr, "read %zu != %zu\n", buffer.size, buffer.capacity);
-        goto err;
-    }
-
-    return buffer;
-err:
-    free(buffer.data);
-    memset(&buffer, 0, sizeof(buffer));
-    return buffer;
-}
-
-data_buffer_t data_buffer_get_data(data_t const* data) {
-    data_buffer_t const kEmptyBuffer = {};
-
-    if (data->type != data_type_file)
-        return kEmptyBuffer;
-
-    return data_buffer_read(data->data.path);
-}
-
-data_buffer_t data_buffer_get_dict(data_t const* data) {
-    data_buffer_t const kEmptyBuffer = {};
-
-    if (!data_has_dict(data))
-        return kEmptyBuffer;
-
-    return data_buffer_read(data->dict.path);
-}
-
-int data_buffer_compare(data_buffer_t buffer1, data_buffer_t buffer2) {
-    size_t const size =
-        buffer1.size < buffer2.size ? buffer1.size : buffer2.size;
-    int const cmp = memcmp(buffer1.data, buffer2.data, size);
-    if (cmp != 0)
-        return cmp;
-    if (buffer1.size < buffer2.size)
-        return -1;
-    if (buffer1.size == buffer2.size)
-        return 0;
-    assert(buffer1.size > buffer2.size);
-    return 1;
-}
-
-void data_buffer_free(data_buffer_t buffer) {
-    free(buffer.data);
-}
-
-/**
- * data filenames helpers.
- */
-
-data_filenames_t data_filenames_get(data_t const* data) {
-    data_filenames_t filenames = {.buffer = NULL, .size = 0};
-    char const* path = data->data.path;
-
-    filenames.filenames = UTIL_createFileList(
-        &path,
-        1,
-        &filenames.buffer,
-        &filenames.size,
-        /* followLinks */ 0);
-    return filenames;
-}
-
-void data_filenames_free(data_filenames_t filenames) {
-    UTIL_freeFileList(filenames.filenames, filenames.buffer);
-}
-
-/**
- * data buffers helpers.
- */
-
-data_buffers_t data_buffers_get(data_t const* data) {
-    data_buffers_t buffers = {.size = 0};
-    data_filenames_t filenames = data_filenames_get(data);
-    if (filenames.size == 0)
-        return buffers;
-
-    data_buffer_t* buffersPtr =
-        (data_buffer_t*)malloc(filenames.size * sizeof(data_buffer_t));
-    if (buffersPtr == NULL)
-        return buffers;
-    buffers.buffers = (data_buffer_t const*)buffersPtr;
-    buffers.size = filenames.size;
-
-    for (size_t i = 0; i < filenames.size; ++i) {
-        buffersPtr[i] = data_buffer_read(filenames.filenames[i]);
-        if (buffersPtr[i].data == NULL) {
-            data_buffers_t const kEmptyBuffer = {};
-            data_buffers_free(buffers);
-            return kEmptyBuffer;
-        }
-    }
-
-    return buffers;
-}
-
-/**
- * Frees the data buffers.
- */
-void data_buffers_free(data_buffers_t buffers) {
-    free((data_buffer_t*)buffers.buffers);
-}
-
-/**
- * Initialization and download functions.
- */
-
-static char* g_data_dir = NULL;
-
-/* mkdir -p */
-static int ensure_directory_exists(char const* indir) {
-    char* const dir = strdup(indir);
-    char* end = dir;
-    int ret = 0;
-    if (dir == NULL) {
-        ret = EINVAL;
-        goto out;
-    }
-    do {
-        /* Find the next directory level. */
-        for (++end; *end != '\0' && *end != '/'; ++end)
-            ;
-        /* End the string there, make the directory, and restore the string. */
-        char const save = *end;
-        *end = '\0';
-        int const isdir = UTIL_isDirectory(dir);
-        ret = mkdir(dir, S_IRWXU);
-        *end = save;
-        /* Its okay if the directory already exists. */
-        if (ret == 0 || (errno == EEXIST && isdir))
-            continue;
-        ret = errno;
-        fprintf(stderr, "mkdir() failed\n");
-        goto out;
-    } while (*end != '\0');
-
-    ret = 0;
-out:
-    free(dir);
-    return ret;
-}
-
-/** Concatenate 3 strings into a new buffer. */
-static char* cat3(char const* str1, char const* str2, char const* str3) {
-    size_t const size1 = strlen(str1);
-    size_t const size2 = strlen(str2);
-    size_t const size3 = str3 == NULL ? 0 : strlen(str3);
-    size_t const size = size1 + size2 + size3 + 1;
-    char* const dst = (char*)malloc(size);
-    if (dst == NULL)
-        return NULL;
-    strcpy(dst, str1);
-    strcpy(dst + size1, str2);
-    if (str3 != NULL)
-        strcpy(dst + size1 + size2, str3);
-    assert(strlen(dst) == size1 + size2 + size3);
-    return dst;
-}
-
-static char* cat2(char const* str1, char const* str2) {
-    return cat3(str1, str2, NULL);
-}
-
-/**
- * State needed by the curl callback.
- * It takes data from curl, hashes it, and writes it to the file.
- */
-typedef struct {
-    FILE* file;
-    XXH64_state_t xxhash64;
-    int error;
-} curl_data_t;
-
-/** Create the curl state. */
-static curl_data_t curl_data_create(
-    data_resource_t const* resource,
-    data_type_t type) {
-    curl_data_t cdata = {};
-
-    XXH64_reset(&cdata.xxhash64, 0);
-
-    assert(UTIL_isDirectory(g_data_dir));
-
-    if (type == data_type_file) {
-        /* Decompress the resource and store to the path. */
-        char* cmd = cat3("zstd -dqfo '", resource->path, "'");
-        if (cmd == NULL) {
-            cdata.error = ENOMEM;
-            return cdata;
-        }
-        cdata.file = popen(cmd, "w");
-        free(cmd);
-    } else {
-        /* Decompress and extract the resource to the cache directory. */
-        char* cmd = cat3("zstd -dc | tar -x -C '", g_data_dir, "'");
-        if (cmd == NULL) {
-            cdata.error = ENOMEM;
-            return cdata;
-        }
-        cdata.file = popen(cmd, "w");
-        free(cmd);
-    }
-    if (cdata.file == NULL) {
-        cdata.error = errno;
-    }
-
-    return cdata;
-}
-
-/** Free the curl state. */
-static int curl_data_free(curl_data_t cdata) {
-    return pclose(cdata.file);
-}
-
-/** curl callback. Updates the hash, and writes to the file. */
-static size_t curl_write(void* data, size_t size, size_t count, void* ptr) {
-    curl_data_t* cdata = (curl_data_t*)ptr;
-    size_t const written = fwrite(data, size, count, cdata->file);
-    XXH64_update(&cdata->xxhash64, data, written * size);
-    return written;
-}
-
-static int curl_download_resource(
-    CURL* curl,
-    data_resource_t const* resource,
-    data_type_t type) {
-    curl_data_t cdata;
-    /* Download the data. */
-    if (curl_easy_setopt(curl, CURLOPT_URL, resource->url) != 0)
-        return EINVAL;
-    if (curl_easy_setopt(curl, CURLOPT_WRITEDATA, &cdata) != 0)
-        return EINVAL;
-    cdata = curl_data_create(resource, type);
-    if (cdata.error != 0)
-        return cdata.error;
-    int const curl_err = curl_easy_perform(curl);
-    int const close_err = curl_data_free(cdata);
-    if (curl_err) {
-        fprintf(
-            stderr,
-            "downloading '%s' for '%s' failed\n",
-            resource->url,
-            resource->path);
-        return EIO;
-    }
-    if (close_err) {
-        fprintf(stderr, "writing data to '%s' failed\n", resource->path);
-        return EIO;
-    }
-    /* check that the file exists. */
-    if (type == data_type_file && !UTIL_isRegularFile(resource->path)) {
-        fprintf(stderr, "output file '%s' does not exist\n", resource->path);
-        return EIO;
-    }
-    if (type == data_type_dir && !UTIL_isDirectory(resource->path)) {
-        fprintf(
-            stderr, "output directory '%s' does not exist\n", resource->path);
-        return EIO;
-    }
-    /* Check that the hash matches. */
-    if (XXH64_digest(&cdata.xxhash64) != resource->xxhash64) {
-        fprintf(
-            stderr,
-            "checksum does not match: 0x%llxLL != 0x%llxLL\n",
-            (unsigned long long)XXH64_digest(&cdata.xxhash64),
-            (unsigned long long)resource->xxhash64);
-        return EINVAL;
-    }
-
-    return 0;
-}
-
-/** Download a single data object. */
-static int curl_download_datum(CURL* curl, data_t const* data) {
-    int ret;
-    ret = curl_download_resource(curl, &data->data, data->type);
-    if (ret != 0)
-        return ret;
-    if (data_has_dict(data)) {
-        ret = curl_download_resource(curl, &data->dict, data_type_file);
-        if (ret != 0)
-            return ret;
-    }
-    return ret;
-}
-
-/** Download all the data. */
-static int curl_download_data(data_t const* const* data) {
-    if (curl_global_init(CURL_GLOBAL_ALL) != 0)
-        return EFAULT;
-
-    curl_data_t cdata = {};
-    CURL* curl = curl_easy_init();
-    int err = EFAULT;
-
-    if (curl == NULL)
-        return EFAULT;
-
-    if (curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 1L) != 0)
-        goto out;
-    if (curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L) != 0)
-        goto out;
-    if (curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, curl_write) != 0)
-        goto out;
-
-    assert(data != NULL);
-    for (; *data != NULL; ++data) {
-        if (curl_download_datum(curl, *data) != 0)
-            goto out;
-    }
-
-    err = 0;
-out:
-    curl_easy_cleanup(curl);
-    curl_global_cleanup();
-    return err;
-}
-
-/** Fill the path member variable of the data objects. */
-static int data_create_paths(data_t* const* data, char const* dir) {
-    size_t const dirlen = strlen(dir);
-    assert(data != NULL);
-    for (; *data != NULL; ++data) {
-        data_t* const datum = *data;
-        datum->data.path = cat3(dir, "/", datum->name);
-        if (datum->data.path == NULL)
-            return ENOMEM;
-        if (data_has_dict(datum)) {
-            datum->dict.path = cat2(datum->data.path, ".dict");
-            if (datum->dict.path == NULL)
-                return ENOMEM;
-        }
-    }
-    return 0;
-}
-
-/** Free the path member variable of the data objects. */
-static void data_free_paths(data_t* const* data) {
-    assert(data != NULL);
-    for (; *data != NULL; ++data) {
-        data_t* datum = *data;
-        free((void*)datum->data.path);
-        free((void*)datum->dict.path);
-        datum->data.path = NULL;
-        datum->dict.path = NULL;
-    }
-}
-
-static char const kStampName[] = "STAMP";
-
-static void xxh_update_le(XXH64_state_t* state, uint64_t data) {
-    if (!MEM_isLittleEndian())
-        data = MEM_swap64(data);
-    XXH64_update(state, &data, sizeof(data));
-}
-
-/** Hash the data to create the stamp. */
-static uint64_t stamp_hash(data_t const* const* data) {
-    XXH64_state_t state;
-
-    XXH64_reset(&state, 0);
-    assert(data != NULL);
-    for (; *data != NULL; ++data) {
-        data_t const* datum = *data;
-        /* We don't care about the URL that we fetch from. */
-        /* The path is derived from the name. */
-        XXH64_update(&state, datum->name, strlen(datum->name));
-        xxh_update_le(&state, datum->data.xxhash64);
-        xxh_update_le(&state, datum->dict.xxhash64);
-        xxh_update_le(&state, datum->type);
-    }
-    return XXH64_digest(&state);
-}
-
-/** Check if the stamp matches the stamp in the cache directory. */
-static int stamp_check(char const* dir, data_t const* const* data) {
-    char* stamp = cat3(dir, "/", kStampName);
-    uint64_t const expected = stamp_hash(data);
-    XXH64_canonical_t actual;
-    FILE* stampfile = NULL;
-    int matches = 0;
-
-    if (stamp == NULL)
-        goto out;
-    if (!UTIL_isRegularFile(stamp)) {
-        fprintf(stderr, "stamp does not exist: recreating the data cache\n");
-        goto out;
-    }
-
-    stampfile = fopen(stamp, "rb");
-    if (stampfile == NULL) {
-        fprintf(stderr, "could not open stamp: recreating the data cache\n");
-        goto out;
-    }
-
-    size_t b;
-    if ((b = fread(&actual, sizeof(actual), 1, stampfile)) != 1) {
-        fprintf(stderr, "invalid stamp: recreating the data cache\n");
-        goto out;
-    }
-
-    matches = (expected == XXH64_hashFromCanonical(&actual));
-    if (matches)
-        fprintf(stderr, "stamp matches: reusing the cached data\n");
-    else
-        fprintf(stderr, "stamp does not match: recreating the data cache\n");
-
-out:
-    free(stamp);
-    if (stampfile != NULL)
-        fclose(stampfile);
-    return matches;
-}
-
-/** On success write a new stamp, on failure delete the old stamp. */
-static int
-stamp_write(char const* dir, data_t const* const* data, int const data_err) {
-    char* stamp = cat3(dir, "/", kStampName);
-    FILE* stampfile = NULL;
-    int err = EIO;
-
-    if (stamp == NULL)
-        return ENOMEM;
-
-    if (data_err != 0) {
-        err = data_err;
-        goto out;
-    }
-    XXH64_canonical_t hash;
-
-    XXH64_canonicalFromHash(&hash, stamp_hash(data));
-
-    stampfile = fopen(stamp, "wb");
-    if (stampfile == NULL)
-        goto out;
-    if (fwrite(&hash, sizeof(hash), 1, stampfile) != 1)
-        goto out;
-    err = 0;
-    fprintf(stderr, "stamped new data cache\n");
-out:
-    if (err != 0)
-        /* Ignore errors. */
-        unlink(stamp);
-    free(stamp);
-    if (stampfile != NULL)
-        fclose(stampfile);
-    return err;
-}
-
-int data_init(char const* dir) {
-    int err;
-
-    if (dir == NULL)
-        return EINVAL;
-
-    /* This must be first to simplify logic. */
-    err = ensure_directory_exists(dir);
-    if (err != 0)
-        return err;
-
-    /* Save the cache directory. */
-    g_data_dir = strdup(dir);
-    if (g_data_dir == NULL)
-        return ENOMEM;
-
-    err = data_create_paths(g_data, dir);
-    if (err != 0)
-        return err;
-
-    /* If the stamp matches then we are good to go.
-     * This must be called before any modifications to the data cache.
-     * After this point, we MUST call stamp_write() to update the STAMP,
-     * since we've updated the data cache.
-     */
-    if (stamp_check(dir, data))
-        return 0;
-
-    err = curl_download_data(data);
-    if (err != 0)
-        goto out;
-
-out:
-    /* This must be last, since it must know if data_init() succeeded. */
-    stamp_write(dir, data, err);
-    return err;
-}
-
-void data_finish(void) {
-    data_free_paths(g_data);
-    free(g_data_dir);
-    g_data_dir = NULL;
-}
diff --git a/tests/regression/data.h b/tests/regression/data.h
deleted file mode 100644
index 717fe1294c14..000000000000
--- a/tests/regression/data.h
+++ /dev/null
@@ -1,140 +0,0 @@
-/*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#ifndef DATA_H
-#define DATA_H
-
-#include <stddef.h>
-#include <stdint.h>
-
-typedef enum {
-    data_type_file = 1,  /**< This data is a file. *.zst */
-    data_type_dir = 2,   /**< This data is a directory. *.tar.zst */
-} data_type_t;
-
-typedef struct {
-    char const* url;   /**< Where to get this resource. */
-    uint64_t xxhash64; /**< Hash of the url contents. */
-    char const* path;  /**< The path of the unpacked resource (derived). */
-} data_resource_t;
-
-typedef struct {
-    data_resource_t data;
-    data_resource_t dict;
-    data_type_t type;  /**< The type of the data. */
-    char const* name;  /**< The logical name of the data (no extension). */
-} data_t;
-
-/**
- * The NULL-terminated list of data objects.
- */
-extern data_t const* const* data;
-
-
-int data_has_dict(data_t const* data);
-
-/**
- * Initializes the data module and downloads the data necessary.
- * Caches the downloads in dir. We add a stamp file in the directory after
- * a successful download. If a stamp file already exists, and matches our
- * current data stamp, we will use the cached data without downloading.
- *
- * @param dir The directory to cache the downloaded data into.
- *
- * @returns 0 on success.
- */
-int data_init(char const* dir);
-
-/**
- * Must be called at exit to free resources allocated by data_init().
- */
-void data_finish(void);
-
-typedef struct {
-    uint8_t* data;
-    size_t size;
-    size_t capacity;
-} data_buffer_t;
-
-/**
- * Read the file that data points to into a buffer.
- * NOTE: data must be a file, not a directory.
- *
- * @returns The buffer, which is NULL on failure.
- */
-data_buffer_t data_buffer_get_data(data_t const* data);
-
-/**
- * Read the dictionary that the data points to into a buffer.
- *
- * @returns The buffer, which is NULL on failure.
- */
-data_buffer_t data_buffer_get_dict(data_t const* data);
-
-/**
- * Read the contents of filename into a buffer.
- *
- * @returns The buffer, which is NULL on failure.
- */
-data_buffer_t data_buffer_read(char const* filename);
-
-/**
- * Create a buffer with the specified capacity.
- *
- * @returns The buffer, which is NULL on failure.
- */
-data_buffer_t data_buffer_create(size_t capacity);
-
-/**
- * Calls memcmp() on the contents [0, size) of both buffers.
- */
-int data_buffer_compare(data_buffer_t buffer1, data_buffer_t buffer2);
-
-/**
- * Frees an allocated buffer.
- */
-void data_buffer_free(data_buffer_t buffer);
-
-typedef struct {
-    char* buffer;
-    char const** filenames;
-    unsigned size;
-} data_filenames_t;
-
-/**
- * Get a recursive list of filenames in the data object. If it is a file, it
- * will only contain one entry. If it is a directory, it will recursively walk
- * the directory.
- *
- * @returns The list of filenames, which has size 0 and NULL pointers on error.
- */
-data_filenames_t data_filenames_get(data_t const* data);
-
-/**
- * Frees the filenames table.
- */
-void data_filenames_free(data_filenames_t filenames);
-
-typedef struct {
-    data_buffer_t const* buffers;
-    size_t size;
-} data_buffers_t;
-
-/**
- * @returns a list of buffers for every file in data. It is zero sized on error.
- */
-data_buffers_t data_buffers_get(data_t const* data);
-
-/**
- * Frees the data buffers.
- */
-void data_buffers_free(data_buffers_t buffers);
-
-#endif
diff --git a/tests/regression/levels.h b/tests/regression/levels.h
deleted file mode 100644
index f96689075126..000000000000
--- a/tests/regression/levels.h
+++ /dev/null
@@ -1,44 +0,0 @@
-/*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#ifndef LEVEL
-# error LEVEL(x) must be defined
-#endif
-#ifndef FAST_LEVEL
-# error FAST_LEVEL(x) must be defined
-#endif
-
-/**
- * The levels are chosen to trigger every strategy in every source size,
- * as well as some fast levels and the default level.
- * If you change the compression levels, you should probably update these.
- */
-
-FAST_LEVEL(5)
-
-FAST_LEVEL(3)
-
-FAST_LEVEL(1)
-LEVEL(0)
-LEVEL(1)
-
-LEVEL(3)
-LEVEL(4)
-LEVEL(5)
-LEVEL(6)
-LEVEL(7)
-
-LEVEL(9)
-
-LEVEL(13)
-
-LEVEL(16)
-
-LEVEL(19)
diff --git a/tests/regression/method.c b/tests/regression/method.c
deleted file mode 100644
index b74f548196fd..000000000000
--- a/tests/regression/method.c
+++ /dev/null
@@ -1,688 +0,0 @@
-/*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#include "method.h"
-
-#include <stdio.h>
-#include <stdlib.h>
-
-#define ZSTD_STATIC_LINKING_ONLY
-#include <zstd.h>
-
-#define MIN(x, y) ((x) < (y) ? (x) : (y))
-
-static char const* g_zstdcli = NULL;
-
-void method_set_zstdcli(char const* zstdcli) {
-    g_zstdcli = zstdcli;
-}
-
-/**
- * Macro to get a pointer of type, given ptr, which is a member variable with
- * the given name, member.
- *
- *     method_state_t* base = ...;
- *     buffer_state_t* state = container_of(base, buffer_state_t, base);
- */
-#define container_of(ptr, type, member) \
-    ((type*)(ptr == NULL ? NULL : (char*)(ptr)-offsetof(type, member)))
-
-/** State to reuse the same buffers between compression calls. */
-typedef struct {
-    method_state_t base;
-    data_buffers_t inputs; /**< The input buffer for each file. */
-    data_buffer_t dictionary; /**< The dictionary. */
-    data_buffer_t compressed; /**< The compressed data buffer. */
-    data_buffer_t decompressed; /**< The decompressed data buffer. */
-} buffer_state_t;
-
-static size_t buffers_max_size(data_buffers_t buffers) {
-    size_t max = 0;
-    for (size_t i = 0; i < buffers.size; ++i) {
-        if (buffers.buffers[i].size > max)
-            max = buffers.buffers[i].size;
-    }
-    return max;
-}
-
-static method_state_t* buffer_state_create(data_t const* data) {
-    buffer_state_t* state = (buffer_state_t*)calloc(1, sizeof(buffer_state_t));
-    if (state == NULL)
-        return NULL;
-    state->base.data = data;
-    state->inputs = data_buffers_get(data);
-    state->dictionary = data_buffer_get_dict(data);
-    size_t const max_size = buffers_max_size(state->inputs);
-    state->compressed = data_buffer_create(ZSTD_compressBound(max_size));
-    state->decompressed = data_buffer_create(max_size);
-    return &state->base;
-}
-
-static void buffer_state_destroy(method_state_t* base) {
-    if (base == NULL)
-        return;
-    buffer_state_t* state = container_of(base, buffer_state_t, base);
-    free(state);
-}
-
-static int buffer_state_bad(
-    buffer_state_t const* state,
-    config_t const* config) {
-    if (state == NULL) {
-        fprintf(stderr, "buffer_state_t is NULL\n");
-        return 1;
-    }
-    if (state->inputs.size == 0 || state->compressed.data == NULL ||
-        state->decompressed.data == NULL) {
-        fprintf(stderr, "buffer state allocation failure\n");
-        return 1;
-    }
-    if (config->use_dictionary && state->dictionary.data == NULL) {
-        fprintf(stderr, "dictionary loading failed\n");
-        return 1;
-    }
-    return 0;
-}
-
-static result_t simple_compress(method_state_t* base, config_t const* config) {
-    buffer_state_t* state = container_of(base, buffer_state_t, base);
-
-    if (buffer_state_bad(state, config))
-        return result_error(result_error_system_error);
-
-    /* Keep the tests short by skipping directories, since behavior shouldn't
-     * change.
-     */
-    if (base->data->type != data_type_file)
-        return result_error(result_error_skip);
-
-    if (config->use_dictionary || config->no_pledged_src_size)
-        return result_error(result_error_skip);
-
-    /* If the config doesn't specify a level, skip. */
-    int const level = config_get_level(config);
-    if (level == CONFIG_NO_LEVEL)
-        return result_error(result_error_skip);
-
-    data_buffer_t const input = state->inputs.buffers[0];
-
-    /* Compress, decompress, and check the result. */
-    state->compressed.size = ZSTD_compress(
-        state->compressed.data,
-        state->compressed.capacity,
-        input.data,
-        input.size,
-        level);
-    if (ZSTD_isError(state->compressed.size))
-        return result_error(result_error_compression_error);
-
-    state->decompressed.size = ZSTD_decompress(
-        state->decompressed.data,
-        state->decompressed.capacity,
-        state->compressed.data,
-        state->compressed.size);
-    if (ZSTD_isError(state->decompressed.size))
-        return result_error(result_error_decompression_error);
-    if (data_buffer_compare(input, state->decompressed))
-        return result_error(result_error_round_trip_error);
-
-    result_data_t data;
-    data.total_size = state->compressed.size;
-    return result_data(data);
-}
-
-static result_t compress_cctx_compress(
-    method_state_t* base,
-    config_t const* config) {
-    buffer_state_t* state = container_of(base, buffer_state_t, base);
-
-    if (buffer_state_bad(state, config))
-        return result_error(result_error_system_error);
-
-    if (config->no_pledged_src_size)
-        return result_error(result_error_skip);
-
-    if (base->data->type != data_type_dir)
-        return result_error(result_error_skip);
-
-    int const level = config_get_level(config);
-
-    ZSTD_CCtx* cctx = ZSTD_createCCtx();
-    ZSTD_DCtx* dctx = ZSTD_createDCtx();
-    if (cctx == NULL || dctx == NULL) {
-        fprintf(stderr, "context creation failed\n");
-        return result_error(result_error_system_error);
-    }
-
-    result_t result;
-    result_data_t data = {.total_size = 0};
-    for (size_t i = 0; i < state->inputs.size; ++i) {
-        data_buffer_t const input = state->inputs.buffers[i];
-        ZSTD_parameters const params =
-            config_get_zstd_params(config, input.size, state->dictionary.size);
-
-        if (level == CONFIG_NO_LEVEL)
-            state->compressed.size = ZSTD_compress_advanced(
-                cctx,
-                state->compressed.data,
-                state->compressed.capacity,
-                input.data,
-                input.size,
-                config->use_dictionary ? state->dictionary.data : NULL,
-                config->use_dictionary ? state->dictionary.size : 0,
-                params);
-        else if (config->use_dictionary)
-            state->compressed.size = ZSTD_compress_usingDict(
-                cctx,
-                state->compressed.data,
-                state->compressed.capacity,
-                input.data,
-                input.size,
-                state->dictionary.data,
-                state->dictionary.size,
-                level);
-        else
-            state->compressed.size = ZSTD_compressCCtx(
-                cctx,
-                state->compressed.data,
-                state->compressed.capacity,
-                input.data,
-                input.size,
-                level);
-
-        if (ZSTD_isError(state->compressed.size)) {
-            result = result_error(result_error_compression_error);
-            goto out;
-        }
-
-        if (config->use_dictionary)
-            state->decompressed.size = ZSTD_decompress_usingDict(
-                dctx,
-                state->decompressed.data,
-                state->decompressed.capacity,
-                state->compressed.data,
-                state->compressed.size,
-                state->dictionary.data,
-                state->dictionary.size);
-        else
-            state->decompressed.size = ZSTD_decompressDCtx(
-                dctx,
-                state->decompressed.data,
-                state->decompressed.capacity,
-                state->compressed.data,
-                state->compressed.size);
-        if (ZSTD_isError(state->decompressed.size)) {
-            result = result_error(result_error_decompression_error);
-            goto out;
-        }
-        if (data_buffer_compare(input, state->decompressed)) {
-            result = result_error(result_error_round_trip_error);
-            goto out;
-        }
-
-        data.total_size += state->compressed.size;
-    }
-
-    result = result_data(data);
-out:
-    ZSTD_freeCCtx(cctx);
-    ZSTD_freeDCtx(dctx);
-    return result;
-}
-
-/** Generic state creation function. */
-static method_state_t* method_state_create(data_t const* data) {
-    method_state_t* state = (method_state_t*)malloc(sizeof(method_state_t));
-    if (state == NULL)
-        return NULL;
-    state->data = data;
-    return state;
-}
-
-static void method_state_destroy(method_state_t* state) {
-    free(state);
-}
-
-static result_t cli_compress(method_state_t* state, config_t const* config) {
-    if (config->cli_args == NULL)
-        return result_error(result_error_skip);
-
-    /* We don't support no pledged source size with directories. Too slow. */
-    if (state->data->type == data_type_dir && config->no_pledged_src_size)
-        return result_error(result_error_skip);
-
-    if (g_zstdcli == NULL)
-        return result_error(result_error_system_error);
-
-    /* '<zstd>' -cqr <args> [-D '<dict>'] '<file/dir>' */
-    char cmd[1024];
-    size_t const cmd_size = snprintf(
-        cmd,
-        sizeof(cmd),
-        "'%s' -cqr %s %s%s%s %s '%s'",
-        g_zstdcli,
-        config->cli_args,
-        config->use_dictionary ? "-D '" : "",
-        config->use_dictionary ? state->data->dict.path : "",
-        config->use_dictionary ? "'" : "",
-        config->no_pledged_src_size ? "<" : "",
-        state->data->data.path);
-    if (cmd_size >= sizeof(cmd)) {
-        fprintf(stderr, "command too large: %s\n", cmd);
-        return result_error(result_error_system_error);
-    }
-    FILE* zstd = popen(cmd, "r");
-    if (zstd == NULL) {
-        fprintf(stderr, "failed to popen command: %s\n", cmd);
-        return result_error(result_error_system_error);
-    }
-
-    char out[4096];
-    size_t total_size = 0;
-    while (1) {
-        size_t const size = fread(out, 1, sizeof(out), zstd);
-        total_size += size;
-        if (size != sizeof(out))
-            break;
-    }
-    if (ferror(zstd) || pclose(zstd) != 0) {
-        fprintf(stderr, "zstd failed with command: %s\n", cmd);
-        return result_error(result_error_compression_error);
-    }
-
-    result_data_t const data = {.total_size = total_size};
-    return result_data(data);
-}
-
-static int advanced_config(
-    ZSTD_CCtx* cctx,
-    buffer_state_t* state,
-    config_t const* config) {
-    ZSTD_CCtx_reset(cctx, ZSTD_reset_session_and_parameters);
-    for (size_t p = 0; p < config->param_values.size; ++p) {
-        param_value_t const pv = config->param_values.data[p];
-        if (ZSTD_isError(ZSTD_CCtx_setParameter(cctx, pv.param, pv.value))) {
-            return 1;
-        }
-    }
-    if (config->use_dictionary) {
-        if (ZSTD_isError(ZSTD_CCtx_loadDictionary(
-                cctx, state->dictionary.data, state->dictionary.size))) {
-            return 1;
-        }
-    }
-    return 0;
-}
-
-static result_t advanced_one_pass_compress_output_adjustment(
-    method_state_t* base,
-    config_t const* config,
-    size_t const subtract) {
-    buffer_state_t* state = container_of(base, buffer_state_t, base);
-
-    if (buffer_state_bad(state, config))
-        return result_error(result_error_system_error);
-
-    ZSTD_CCtx* cctx = ZSTD_createCCtx();
-    result_t result;
-
-    if (!cctx || advanced_config(cctx, state, config)) {
-        result = result_error(result_error_compression_error);
-        goto out;
-    }
-
-    result_data_t data = {.total_size = 0};
-    for (size_t i = 0; i < state->inputs.size; ++i) {
-        data_buffer_t const input = state->inputs.buffers[i];
-
-        if (!config->no_pledged_src_size) {
-            if (ZSTD_isError(ZSTD_CCtx_setPledgedSrcSize(cctx, input.size))) {
-                result = result_error(result_error_compression_error);
-                goto out;
-            }
-        }
-        size_t const size = ZSTD_compress2(
-            cctx,
-            state->compressed.data,
-            ZSTD_compressBound(input.size) - subtract,
-            input.data,
-            input.size);
-        if (ZSTD_isError(size)) {
-            result = result_error(result_error_compression_error);
-            goto out;
-        }
-        data.total_size += size;
-    }
-
-    result = result_data(data);
-out:
-    ZSTD_freeCCtx(cctx);
-    return result;
-}
-
-static result_t advanced_one_pass_compress(
-    method_state_t* base,
-    config_t const* config) {
-  return advanced_one_pass_compress_output_adjustment(base, config, 0);
-}
-
-static result_t advanced_one_pass_compress_small_output(
-    method_state_t* base,
-    config_t const* config) {
-  return advanced_one_pass_compress_output_adjustment(base, config, 1);
-}
-
-static result_t advanced_streaming_compress(
-    method_state_t* base,
-    config_t const* config) {
-    buffer_state_t* state = container_of(base, buffer_state_t, base);
-
-    if (buffer_state_bad(state, config))
-        return result_error(result_error_system_error);
-
-    ZSTD_CCtx* cctx = ZSTD_createCCtx();
-    result_t result;
-
-    if (!cctx || advanced_config(cctx, state, config)) {
-        result = result_error(result_error_compression_error);
-        goto out;
-    }
-
-    result_data_t data = {.total_size = 0};
-    for (size_t i = 0; i < state->inputs.size; ++i) {
-        data_buffer_t input = state->inputs.buffers[i];
-
-        if (!config->no_pledged_src_size) {
-            if (ZSTD_isError(ZSTD_CCtx_setPledgedSrcSize(cctx, input.size))) {
-                result = result_error(result_error_compression_error);
-                goto out;
-            }
-        }
-
-        while (input.size > 0) {
-            ZSTD_inBuffer in = {input.data, MIN(input.size, 4096)};
-            input.data += in.size;
-            input.size -= in.size;
-            ZSTD_EndDirective const op =
-                input.size > 0 ? ZSTD_e_continue : ZSTD_e_end;
-            size_t ret = 0;
-            while (in.pos < in.size || (op == ZSTD_e_end && ret != 0)) {
-                ZSTD_outBuffer out = {state->compressed.data,
-                                      MIN(state->compressed.capacity, 1024)};
-                ret = ZSTD_compressStream2(cctx, &out, &in, op);
-                if (ZSTD_isError(ret)) {
-                    result = result_error(result_error_compression_error);
-                    goto out;
-                }
-                data.total_size += out.pos;
-            }
-        }
-    }
-
-    result = result_data(data);
-out:
-    ZSTD_freeCCtx(cctx);
-    return result;
-}
-
-static int init_cstream(
-    buffer_state_t* state,
-    ZSTD_CStream* zcs,
-    config_t const* config,
-    int const advanced,
-    ZSTD_CDict** cdict)
-{
-    size_t zret;
-    if (advanced) {
-        ZSTD_parameters const params = config_get_zstd_params(config, 0, 0);
-        ZSTD_CDict* dict = NULL;
-        if (cdict) {
-            if (!config->use_dictionary)
-              return 1;
-            *cdict = ZSTD_createCDict_advanced(
-                state->dictionary.data,
-                state->dictionary.size,
-                ZSTD_dlm_byRef,
-                ZSTD_dct_auto,
-                params.cParams,
-                ZSTD_defaultCMem);
-            if (!*cdict) {
-                return 1;
-            }
-            zret = ZSTD_initCStream_usingCDict_advanced(
-                zcs, *cdict, params.fParams, ZSTD_CONTENTSIZE_UNKNOWN);
-        } else {
-            zret = ZSTD_initCStream_advanced(
-                zcs,
-                config->use_dictionary ? state->dictionary.data : NULL,
-                config->use_dictionary ? state->dictionary.size : 0,
-                params,
-                ZSTD_CONTENTSIZE_UNKNOWN);
-        }
-    } else {
-        int const level = config_get_level(config);
-        if (level == CONFIG_NO_LEVEL)
-            return 1;
-        if (cdict) {
-            if (!config->use_dictionary)
-              return 1;
-            *cdict = ZSTD_createCDict(
-                state->dictionary.data,
-                state->dictionary.size,
-                level);
-            if (!*cdict) {
-                return 1;
-            }
-            zret = ZSTD_initCStream_usingCDict(zcs, *cdict);
-        } else if (config->use_dictionary) {
-            zret = ZSTD_initCStream_usingDict(
-                zcs,
-                state->dictionary.data,
-                state->dictionary.size,
-                level);
-        } else {
-            zret = ZSTD_initCStream(zcs, level);
-        }
-    }
-    if (ZSTD_isError(zret)) {
-        return 1;
-    }
-    return 0;
-}
-
-static result_t old_streaming_compress_internal(
-    method_state_t* base,
-    config_t const* config,
-    int const advanced,
-    int const cdict) {
-  buffer_state_t* state = container_of(base, buffer_state_t, base);
-
-  if (buffer_state_bad(state, config))
-    return result_error(result_error_system_error);
-
-
-  ZSTD_CStream* zcs = ZSTD_createCStream();
-  ZSTD_CDict* cd = NULL;
-  result_t result;
-  if (zcs == NULL) {
-    result = result_error(result_error_compression_error);
-    goto out;
-  }
-  if (!advanced && config_get_level(config) == CONFIG_NO_LEVEL) {
-    result = result_error(result_error_skip);
-    goto out;
-  }
-  if (cdict && !config->use_dictionary) {
-    result = result_error(result_error_skip);
-    goto out;
-  }
-  if (init_cstream(state, zcs, config, advanced, cdict ? &cd : NULL)) {
-    result = result_error(result_error_compression_error);
-    goto out;
-  }
-
-  result_data_t data = {.total_size = 0};
-  for (size_t i = 0; i < state->inputs.size; ++i) {
-    data_buffer_t input = state->inputs.buffers[i];
-    size_t zret = ZSTD_resetCStream(
-        zcs,
-        config->no_pledged_src_size ? ZSTD_CONTENTSIZE_UNKNOWN : input.size);
-    if (ZSTD_isError(zret)) {
-      result = result_error(result_error_compression_error);
-      goto out;
-    }
-
-    while (input.size > 0) {
-      ZSTD_inBuffer in = {input.data, MIN(input.size, 4096)};
-      input.data += in.size;
-      input.size -= in.size;
-      ZSTD_EndDirective const op =
-          input.size > 0 ? ZSTD_e_continue : ZSTD_e_end;
-      zret = 0;
-      while (in.pos < in.size || (op == ZSTD_e_end && zret != 0)) {
-        ZSTD_outBuffer out = {state->compressed.data,
-                              MIN(state->compressed.capacity, 1024)};
-        if (op == ZSTD_e_continue || in.pos < in.size)
-          zret = ZSTD_compressStream(zcs, &out, &in);
-        else
-          zret = ZSTD_endStream(zcs, &out);
-        if (ZSTD_isError(zret)) {
-          result = result_error(result_error_compression_error);
-          goto out;
-        }
-        data.total_size += out.pos;
-      }
-    }
-  }
-
-  result = result_data(data);
-out:
-    ZSTD_freeCStream(zcs);
-    ZSTD_freeCDict(cd);
-    return result;
-}
-
-static result_t old_streaming_compress(
-    method_state_t* base,
-    config_t const* config)
-{
-    return old_streaming_compress_internal(
-        base, config, /* advanced */ 0, /* cdict */ 0);
-}
-
-static result_t old_streaming_compress_advanced(
-    method_state_t* base,
-    config_t const* config)
-{
-    return old_streaming_compress_internal(
-        base, config, /* advanced */ 1, /* cdict */ 0);
-}
-
-static result_t old_streaming_compress_cdict(
-    method_state_t* base,
-    config_t const* config)
-{
-    return old_streaming_compress_internal(
-        base, config, /* advanced */ 0, /* cdict */ 1);
-}
-
-static result_t old_streaming_compress_cdict_advanced(
-    method_state_t* base,
-    config_t const* config)
-{
-    return old_streaming_compress_internal(
-        base, config, /* advanced */ 1, /* cdict */ 1);
-}
-
-method_t const simple = {
-    .name = "compress simple",
-    .create = buffer_state_create,
-    .compress = simple_compress,
-    .destroy = buffer_state_destroy,
-};
-
-method_t const compress_cctx = {
-    .name = "compress cctx",
-    .create = buffer_state_create,
-    .compress = compress_cctx_compress,
-    .destroy = buffer_state_destroy,
-};
-
-method_t const advanced_one_pass = {
-    .name = "advanced one pass",
-    .create = buffer_state_create,
-    .compress = advanced_one_pass_compress,
-    .destroy = buffer_state_destroy,
-};
-
-method_t const advanced_one_pass_small_out = {
-    .name = "advanced one pass small out",
-    .create = buffer_state_create,
-    .compress = advanced_one_pass_compress,
-    .destroy = buffer_state_destroy,
-};
-
-method_t const advanced_streaming = {
-    .name = "advanced streaming",
-    .create = buffer_state_create,
-    .compress = advanced_streaming_compress,
-    .destroy = buffer_state_destroy,
-};
-
-method_t const old_streaming = {
-    .name = "old streaming",
-    .create = buffer_state_create,
-    .compress = old_streaming_compress,
-    .destroy = buffer_state_destroy,
-};
-
-method_t const old_streaming_advanced = {
-    .name = "old streaming advanced",
-    .create = buffer_state_create,
-    .compress = old_streaming_compress_advanced,
-    .destroy = buffer_state_destroy,
-};
-
-method_t const old_streaming_cdict = {
-    .name = "old streaming cdcit",
-    .create = buffer_state_create,
-    .compress = old_streaming_compress_cdict,
-    .destroy = buffer_state_destroy,
-};
-
-method_t const old_streaming_advanced_cdict = {
-    .name = "old streaming advanced cdict",
-    .create = buffer_state_create,
-    .compress = old_streaming_compress_cdict_advanced,
-    .destroy = buffer_state_destroy,
-};
-
-method_t const cli = {
-    .name = "zstdcli",
-    .create = method_state_create,
-    .compress = cli_compress,
-    .destroy = method_state_destroy,
-};
-
-static method_t const* g_methods[] = {
-    &simple,
-    &compress_cctx,
-    &cli,
-    &advanced_one_pass,
-    &advanced_one_pass_small_out,
-    &advanced_streaming,
-    &old_streaming,
-    &old_streaming_advanced,
-    &old_streaming_cdict,
-    &old_streaming_advanced_cdict,
-    NULL,
-};
-
-method_t const* const* methods = g_methods;
diff --git a/tests/regression/method.h b/tests/regression/method.h
deleted file mode 100644
index d70b776b1b90..000000000000
--- a/tests/regression/method.h
+++ /dev/null
@@ -1,65 +0,0 @@
-/*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#ifndef METHOD_H
-#define METHOD_H
-
-#include <stddef.h>
-
-#include "data.h"
-#include "config.h"
-#include "result.h"
-
-/**
- * The base class for state that methods keep.
- * All derived method state classes must have a member of this type.
- */
-typedef struct {
-    data_t const* data;
-} method_state_t;
-
-/**
- * A method that compresses the data using config.
- */
-typedef struct {
-    char const* name;  /**< The identifier for this method in the results. */
-    /**
-     * Creates a state that must contain a member variable of method_state_t,
-     * and returns a pointer to that member variable.
-     *
-     * This method can be used to do expensive work that only depends on the
-     * data, like loading the data file into a buffer.
-     */
-    method_state_t* (*create)(data_t const* data);
-    /**
-     * Compresses the data in the state using the given config.
-     *
-     * @param state A pointer to the state returned by create().
-     *
-     * @returns The total compressed size on success, or an error code.
-     */
-    result_t (*compress)(method_state_t* state, config_t const* config);
-    /**
-     * Frees the state.
-     */
-    void (*destroy)(method_state_t* state);
-} method_t;
-
-/**
- * Set the zstd cli path. Must be called before any methods are used.
- */
-void method_set_zstdcli(char const* zstdcli);
-
-/**
- * A NULL-terminated list of methods.
- */
-extern method_t const* const* methods;
-
-#endif
diff --git a/tests/regression/result.c b/tests/regression/result.c
deleted file mode 100644
index 31439b08cde7..000000000000
--- a/tests/regression/result.c
+++ /dev/null
@@ -1,28 +0,0 @@
-/*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#include "result.h"
-
-char const* result_get_error_string(result_t result) {
-    switch (result_get_error(result)) {
-        case result_error_ok:
-            return "okay";
-        case result_error_skip:
-            return "skip";
-        case result_error_system_error:
-            return "system error";
-        case result_error_compression_error:
-            return "compression error";
-        case result_error_decompression_error:
-            return "decompression error";
-        case result_error_round_trip_error:
-            return "round trip error";
-    }
-}
diff --git a/tests/regression/result.h b/tests/regression/result.h
deleted file mode 100644
index 8c80cf85a966..000000000000
--- a/tests/regression/result.h
+++ /dev/null
@@ -1,103 +0,0 @@
-/*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#ifndef RESULT_H
-#define RESULT_H
-
-#include <stddef.h>
-
-/**
- * The error type enum.
- */
-typedef enum {
-    result_error_ok,                   /**< No error. */
-    result_error_skip,                 /**< This method was skipped. */
-    result_error_system_error,         /**< Some internal error happened. */
-    result_error_compression_error,    /**< Compression failed. */
-    result_error_decompression_error,  /**< Decompression failed. */
-    result_error_round_trip_error,     /**< Data failed to round trip. */
-} result_error_t;
-
-/**
- * The success type.
- */
-typedef struct {
-    size_t total_size;  /**< The total compressed size. */
-} result_data_t;
-
-/**
- * The result type.
- * Do not access the member variables directory, use the helper functions.
- */
-typedef struct {
-    result_error_t internal_error;
-    result_data_t internal_data;
-} result_t;
-
-/**
- * Create a result of the error type.
- */
-static result_t result_error(result_error_t error);
-/**
- * Create a result of the success type.
- */
-static result_t result_data(result_data_t data);
-
-/**
- * Check if the result is an error or skip.
- */
-static int result_is_error(result_t result);
-/**
- * Check if the result error is skip.
- */
-static int result_is_skip(result_t result);
-/**
- * Get the result error or okay.
- */
-static result_error_t result_get_error(result_t result);
-/**
- * Get the result data. The result MUST be checked with result_is_error() first.
- */
-static result_data_t result_get_data(result_t result);
-
-static result_t result_error(result_error_t error) {
-    result_t result = {
-        .internal_error = error,
-    };
-    return result;
-}
-
-static result_t result_data(result_data_t data) {
-    result_t result = {
-        .internal_error = result_error_ok,
-        .internal_data = data,
-    };
-    return result;
-}
-
-static int result_is_error(result_t result) {
-    return result_get_error(result) != result_error_ok;
-}
-
-static int result_is_skip(result_t result) {
-    return result_get_error(result) == result_error_skip;
-}
-
-static result_error_t result_get_error(result_t result) {
-    return result.internal_error;
-}
-
-char const* result_get_error_string(result_t result);
-
-static result_data_t result_get_data(result_t result) {
-    return result.internal_data;
-}
-
-#endif
diff --git a/tests/regression/results.csv b/tests/regression/results.csv
deleted file mode 100644
index a0e1566a5d50..000000000000
--- a/tests/regression/results.csv
+++ /dev/null
@@ -1,636 +0,0 @@
-Data,                               Config,                             Method,                             Total compressed size
-silesia.tar,                        level -5,                           compress simple,                    6738558
-silesia.tar,                        level -3,                           compress simple,                    6446362
-silesia.tar,                        level -1,                           compress simple,                    6186038
-silesia.tar,                        level 0,                            compress simple,                    4861374
-silesia.tar,                        level 1,                            compress simple,                    5334825
-silesia.tar,                        level 3,                            compress simple,                    4861374
-silesia.tar,                        level 4,                            compress simple,                    4799583
-silesia.tar,                        level 5,                            compress simple,                    4722271
-silesia.tar,                        level 6,                            compress simple,                    4672231
-silesia.tar,                        level 7,                            compress simple,                    4606657
-silesia.tar,                        level 9,                            compress simple,                    4554099
-silesia.tar,                        level 13,                           compress simple,                    4491706
-silesia.tar,                        level 16,                           compress simple,                    4381265
-silesia.tar,                        level 19,                           compress simple,                    4281551
-silesia.tar,                        uncompressed literals,              compress simple,                    4861374
-silesia.tar,                        uncompressed literals optimal,      compress simple,                    4281551
-silesia.tar,                        huffman literals,                   compress simple,                    6186038
-silesia,                            level -5,                           compress cctx,                      6737567
-silesia,                            level -3,                           compress cctx,                      6444663
-silesia,                            level -1,                           compress cctx,                      6178442
-silesia,                            level 0,                            compress cctx,                      4849491
-silesia,                            level 1,                            compress cctx,                      5313144
-silesia,                            level 3,                            compress cctx,                      4849491
-silesia,                            level 4,                            compress cctx,                      4786913
-silesia,                            level 5,                            compress cctx,                      4710178
-silesia,                            level 6,                            compress cctx,                      4659996
-silesia,                            level 7,                            compress cctx,                      4596234
-silesia,                            level 9,                            compress cctx,                      4543862
-silesia,                            level 13,                           compress cctx,                      4482073
-silesia,                            level 16,                           compress cctx,                      4377389
-silesia,                            level 19,                           compress cctx,                      4293262
-silesia,                            long distance mode,                 compress cctx,                      4849491
-silesia,                            multithreaded,                      compress cctx,                      4849491
-silesia,                            multithreaded long distance mode,   compress cctx,                      4849491
-silesia,                            small window log,                   compress cctx,                      7112472
-silesia,                            small hash log,                     compress cctx,                      6554898
-silesia,                            small chain log,                    compress cctx,                      4931093
-silesia,                            explicit params,                    compress cctx,                      4794609
-silesia,                            uncompressed literals,              compress cctx,                      4849491
-silesia,                            uncompressed literals optimal,      compress cctx,                      4293262
-silesia,                            huffman literals,                   compress cctx,                      6178442
-silesia,                            multithreaded with advanced params, compress cctx,                      4849491
-github,                             level -5,                           compress cctx,                      205285
-github,                             level -5 with dict,                 compress cctx,                      47294
-github,                             level -3,                           compress cctx,                      190643
-github,                             level -3 with dict,                 compress cctx,                      48047
-github,                             level -1,                           compress cctx,                      175568
-github,                             level -1 with dict,                 compress cctx,                      43527
-github,                             level 0,                            compress cctx,                      136311
-github,                             level 0 with dict,                  compress cctx,                      41534
-github,                             level 1,                            compress cctx,                      142450
-github,                             level 1 with dict,                  compress cctx,                      42157
-github,                             level 3,                            compress cctx,                      136311
-github,                             level 3 with dict,                  compress cctx,                      41534
-github,                             level 4,                            compress cctx,                      136144
-github,                             level 4 with dict,                  compress cctx,                      41725
-github,                             level 5,                            compress cctx,                      135106
-github,                             level 5 with dict,                  compress cctx,                      38934
-github,                             level 6,                            compress cctx,                      135108
-github,                             level 6 with dict,                  compress cctx,                      38628
-github,                             level 7,                            compress cctx,                      135108
-github,                             level 7 with dict,                  compress cctx,                      38741
-github,                             level 9,                            compress cctx,                      135108
-github,                             level 9 with dict,                  compress cctx,                      39335
-github,                             level 13,                           compress cctx,                      133717
-github,                             level 13 with dict,                 compress cctx,                      39923
-github,                             level 16,                           compress cctx,                      133717
-github,                             level 16 with dict,                 compress cctx,                      37568
-github,                             level 19,                           compress cctx,                      133717
-github,                             level 19 with dict,                 compress cctx,                      37567
-github,                             long distance mode,                 compress cctx,                      141101
-github,                             multithreaded,                      compress cctx,                      141101
-github,                             multithreaded long distance mode,   compress cctx,                      141101
-github,                             small window log,                   compress cctx,                      141101
-github,                             small hash log,                     compress cctx,                      138943
-github,                             small chain log,                    compress cctx,                      139239
-github,                             explicit params,                    compress cctx,                      140924
-github,                             uncompressed literals,              compress cctx,                      136311
-github,                             uncompressed literals optimal,      compress cctx,                      133717
-github,                             huffman literals,                   compress cctx,                      175568
-github,                             multithreaded with advanced params, compress cctx,                      141101
-silesia,                            level -5,                           zstdcli,                            6882514
-silesia,                            level -3,                           zstdcli,                            6568406
-silesia,                            level -1,                           zstdcli,                            6183433
-silesia,                            level 0,                            zstdcli,                            4849539
-silesia,                            level 1,                            zstdcli,                            5314157
-silesia,                            level 3,                            zstdcli,                            4849539
-silesia,                            level 4,                            zstdcli,                            4786961
-silesia,                            level 5,                            zstdcli,                            4710226
-silesia,                            level 6,                            zstdcli,                            4660044
-silesia,                            level 7,                            zstdcli,                            4596282
-silesia,                            level 9,                            zstdcli,                            4543910
-silesia,                            level 13,                           zstdcli,                            4482121
-silesia,                            level 16,                           zstdcli,                            4377437
-silesia,                            level 19,                           zstdcli,                            4293310
-silesia,                            long distance mode,                 zstdcli,                            4839698
-silesia,                            multithreaded,                      zstdcli,                            4849539
-silesia,                            multithreaded long distance mode,   zstdcli,                            4839698
-silesia,                            small window log,                   zstdcli,                            7123580
-silesia,                            small hash log,                     zstdcli,                            6554946
-silesia,                            small chain log,                    zstdcli,                            4931141
-silesia,                            explicit params,                    zstdcli,                            4797048
-silesia,                            uncompressed literals,              zstdcli,                            5128008
-silesia,                            uncompressed literals optimal,      zstdcli,                            4325482
-silesia,                            huffman literals,                   zstdcli,                            5331158
-silesia,                            multithreaded with advanced params, zstdcli,                            5128008
-silesia.tar,                        level -5,                           zstdcli,                            6738906
-silesia.tar,                        level -3,                           zstdcli,                            6448409
-silesia.tar,                        level -1,                           zstdcli,                            6186908
-silesia.tar,                        level 0,                            zstdcli,                            4861462
-silesia.tar,                        level 1,                            zstdcli,                            5336255
-silesia.tar,                        level 3,                            zstdcli,                            4861462
-silesia.tar,                        level 4,                            zstdcli,                            4800482
-silesia.tar,                        level 5,                            zstdcli,                            4723312
-silesia.tar,                        level 6,                            zstdcli,                            4673616
-silesia.tar,                        level 7,                            zstdcli,                            4608346
-silesia.tar,                        level 9,                            zstdcli,                            4554702
-silesia.tar,                        level 13,                           zstdcli,                            4491710
-silesia.tar,                        level 16,                           zstdcli,                            4381269
-silesia.tar,                        level 19,                           zstdcli,                            4281555
-silesia.tar,                        no source size,                     zstdcli,                            4861458
-silesia.tar,                        long distance mode,                 zstdcli,                            4853140
-silesia.tar,                        multithreaded,                      zstdcli,                            4861462
-silesia.tar,                        multithreaded long distance mode,   zstdcli,                            4853140
-silesia.tar,                        small window log,                   zstdcli,                            7127589
-silesia.tar,                        small hash log,                     zstdcli,                            6587841
-silesia.tar,                        small chain log,                    zstdcli,                            4943269
-silesia.tar,                        explicit params,                    zstdcli,                            4822318
-silesia.tar,                        uncompressed literals,              zstdcli,                            5129548
-silesia.tar,                        uncompressed literals optimal,      zstdcli,                            4320914
-silesia.tar,                        huffman literals,                   zstdcli,                            5347560
-silesia.tar,                        multithreaded with advanced params, zstdcli,                            5129548
-github,                             level -5,                           zstdcli,                            207285
-github,                             level -5 with dict,                 zstdcli,                            48718
-github,                             level -3,                           zstdcli,                            192643
-github,                             level -3 with dict,                 zstdcli,                            47395
-github,                             level -1,                           zstdcli,                            177568
-github,                             level -1 with dict,                 zstdcli,                            45170
-github,                             level 0,                            zstdcli,                            138311
-github,                             level 0 with dict,                  zstdcli,                            43148
-github,                             level 1,                            zstdcli,                            144450
-github,                             level 1 with dict,                  zstdcli,                            43682
-github,                             level 3,                            zstdcli,                            138311
-github,                             level 3 with dict,                  zstdcli,                            43148
-github,                             level 4,                            zstdcli,                            138144
-github,                             level 4 with dict,                  zstdcli,                            43251
-github,                             level 5,                            zstdcli,                            137106
-github,                             level 5 with dict,                  zstdcli,                            40938
-github,                             level 6,                            zstdcli,                            137108
-github,                             level 6 with dict,                  zstdcli,                            40632
-github,                             level 7,                            zstdcli,                            137108
-github,                             level 7 with dict,                  zstdcli,                            40766
-github,                             level 9,                            zstdcli,                            137108
-github,                             level 9 with dict,                  zstdcli,                            41326
-github,                             level 13,                           zstdcli,                            135717
-github,                             level 13 with dict,                 zstdcli,                            41716
-github,                             level 16,                           zstdcli,                            135717
-github,                             level 16 with dict,                 zstdcli,                            39577
-github,                             level 19,                           zstdcli,                            135717
-github,                             level 19 with dict,                 zstdcli,                            39576
-github,                             long distance mode,                 zstdcli,                            138311
-github,                             multithreaded,                      zstdcli,                            138311
-github,                             multithreaded long distance mode,   zstdcli,                            138311
-github,                             small window log,                   zstdcli,                            138311
-github,                             small hash log,                     zstdcli,                            137467
-github,                             small chain log,                    zstdcli,                            138314
-github,                             explicit params,                    zstdcli,                            136140
-github,                             uncompressed literals,              zstdcli,                            167915
-github,                             uncompressed literals optimal,      zstdcli,                            158824
-github,                             huffman literals,                   zstdcli,                            144450
-github,                             multithreaded with advanced params, zstdcli,                            167915
-silesia,                            level -5,                           advanced one pass,                  6737567
-silesia,                            level -3,                           advanced one pass,                  6444663
-silesia,                            level -1,                           advanced one pass,                  6178442
-silesia,                            level 0,                            advanced one pass,                  4849491
-silesia,                            level 1,                            advanced one pass,                  5313144
-silesia,                            level 3,                            advanced one pass,                  4849491
-silesia,                            level 4,                            advanced one pass,                  4786913
-silesia,                            level 5,                            advanced one pass,                  4710178
-silesia,                            level 6,                            advanced one pass,                  4659996
-silesia,                            level 7,                            advanced one pass,                  4596234
-silesia,                            level 9,                            advanced one pass,                  4543862
-silesia,                            level 13,                           advanced one pass,                  4482073
-silesia,                            level 16,                           advanced one pass,                  4377389
-silesia,                            level 19,                           advanced one pass,                  4293262
-silesia,                            no source size,                     advanced one pass,                  4849491
-silesia,                            long distance mode,                 advanced one pass,                  4839650
-silesia,                            multithreaded,                      advanced one pass,                  4849491
-silesia,                            multithreaded long distance mode,   advanced one pass,                  4839650
-silesia,                            small window log,                   advanced one pass,                  7123532
-silesia,                            small hash log,                     advanced one pass,                  6554898
-silesia,                            small chain log,                    advanced one pass,                  4931093
-silesia,                            explicit params,                    advanced one pass,                  4797035
-silesia,                            uncompressed literals,              advanced one pass,                  5127960
-silesia,                            uncompressed literals optimal,      advanced one pass,                  4325434
-silesia,                            huffman literals,                   advanced one pass,                  5326210
-silesia,                            multithreaded with advanced params, advanced one pass,                  5127960
-silesia.tar,                        level -5,                           advanced one pass,                  6738558
-silesia.tar,                        level -3,                           advanced one pass,                  6446362
-silesia.tar,                        level -1,                           advanced one pass,                  6186038
-silesia.tar,                        level 0,                            advanced one pass,                  4861374
-silesia.tar,                        level 1,                            advanced one pass,                  5334825
-silesia.tar,                        level 3,                            advanced one pass,                  4861374
-silesia.tar,                        level 4,                            advanced one pass,                  4799583
-silesia.tar,                        level 5,                            advanced one pass,                  4722271
-silesia.tar,                        level 6,                            advanced one pass,                  4672231
-silesia.tar,                        level 7,                            advanced one pass,                  4606657
-silesia.tar,                        level 9,                            advanced one pass,                  4554099
-silesia.tar,                        level 13,                           advanced one pass,                  4491706
-silesia.tar,                        level 16,                           advanced one pass,                  4381265
-silesia.tar,                        level 19,                           advanced one pass,                  4281551
-silesia.tar,                        no source size,                     advanced one pass,                  4861374
-silesia.tar,                        long distance mode,                 advanced one pass,                  4848046
-silesia.tar,                        multithreaded,                      advanced one pass,                  4860726
-silesia.tar,                        multithreaded long distance mode,   advanced one pass,                  4847343
-silesia.tar,                        small window log,                   advanced one pass,                  7127549
-silesia.tar,                        small hash log,                     advanced one pass,                  6587833
-silesia.tar,                        small chain log,                    advanced one pass,                  4943266
-silesia.tar,                        explicit params,                    advanced one pass,                  4808543
-silesia.tar,                        uncompressed literals,              advanced one pass,                  5129447
-silesia.tar,                        uncompressed literals optimal,      advanced one pass,                  4320910
-silesia.tar,                        huffman literals,                   advanced one pass,                  5347283
-silesia.tar,                        multithreaded with advanced params, advanced one pass,                  5129766
-github,                             level -5,                           advanced one pass,                  205285
-github,                             level -5 with dict,                 advanced one pass,                  46718
-github,                             level -3,                           advanced one pass,                  190643
-github,                             level -3 with dict,                 advanced one pass,                  45395
-github,                             level -1,                           advanced one pass,                  175568
-github,                             level -1 with dict,                 advanced one pass,                  43170
-github,                             level 0,                            advanced one pass,                  136311
-github,                             level 0 with dict,                  advanced one pass,                  41148
-github,                             level 1,                            advanced one pass,                  142450
-github,                             level 1 with dict,                  advanced one pass,                  41682
-github,                             level 3,                            advanced one pass,                  136311
-github,                             level 3 with dict,                  advanced one pass,                  41148
-github,                             level 4,                            advanced one pass,                  136144
-github,                             level 4 with dict,                  advanced one pass,                  41251
-github,                             level 5,                            advanced one pass,                  135106
-github,                             level 5 with dict,                  advanced one pass,                  38938
-github,                             level 6,                            advanced one pass,                  135108
-github,                             level 6 with dict,                  advanced one pass,                  38632
-github,                             level 7,                            advanced one pass,                  135108
-github,                             level 7 with dict,                  advanced one pass,                  38766
-github,                             level 9,                            advanced one pass,                  135108
-github,                             level 9 with dict,                  advanced one pass,                  39326
-github,                             level 13,                           advanced one pass,                  133717
-github,                             level 13 with dict,                 advanced one pass,                  39716
-github,                             level 16,                           advanced one pass,                  133717
-github,                             level 16 with dict,                 advanced one pass,                  37577
-github,                             level 19,                           advanced one pass,                  133717
-github,                             level 19 with dict,                 advanced one pass,                  37576
-github,                             no source size,                     advanced one pass,                  136311
-github,                             long distance mode,                 advanced one pass,                  136311
-github,                             multithreaded,                      advanced one pass,                  136311
-github,                             multithreaded long distance mode,   advanced one pass,                  136311
-github,                             small window log,                   advanced one pass,                  136311
-github,                             small hash log,                     advanced one pass,                  135467
-github,                             small chain log,                    advanced one pass,                  136314
-github,                             explicit params,                    advanced one pass,                  137670
-github,                             uncompressed literals,              advanced one pass,                  165915
-github,                             uncompressed literals optimal,      advanced one pass,                  156824
-github,                             huffman literals,                   advanced one pass,                  142450
-github,                             multithreaded with advanced params, advanced one pass,                  165915
-silesia,                            level -5,                           advanced one pass small out,        6737567
-silesia,                            level -3,                           advanced one pass small out,        6444663
-silesia,                            level -1,                           advanced one pass small out,        6178442
-silesia,                            level 0,                            advanced one pass small out,        4849491
-silesia,                            level 1,                            advanced one pass small out,        5313144
-silesia,                            level 3,                            advanced one pass small out,        4849491
-silesia,                            level 4,                            advanced one pass small out,        4786913
-silesia,                            level 5,                            advanced one pass small out,        4710178
-silesia,                            level 6,                            advanced one pass small out,        4659996
-silesia,                            level 7,                            advanced one pass small out,        4596234
-silesia,                            level 9,                            advanced one pass small out,        4543862
-silesia,                            level 13,                           advanced one pass small out,        4482073
-silesia,                            level 16,                           advanced one pass small out,        4377389
-silesia,                            level 19,                           advanced one pass small out,        4293262
-silesia,                            no source size,                     advanced one pass small out,        4849491
-silesia,                            long distance mode,                 advanced one pass small out,        4839650
-silesia,                            multithreaded,                      advanced one pass small out,        4849491
-silesia,                            multithreaded long distance mode,   advanced one pass small out,        4839650
-silesia,                            small window log,                   advanced one pass small out,        7123532
-silesia,                            small hash log,                     advanced one pass small out,        6554898
-silesia,                            small chain log,                    advanced one pass small out,        4931093
-silesia,                            explicit params,                    advanced one pass small out,        4797035
-silesia,                            uncompressed literals,              advanced one pass small out,        5127960
-silesia,                            uncompressed literals optimal,      advanced one pass small out,        4325434
-silesia,                            huffman literals,                   advanced one pass small out,        5326210
-silesia,                            multithreaded with advanced params, advanced one pass small out,        5127960
-silesia.tar,                        level -5,                           advanced one pass small out,        6738558
-silesia.tar,                        level -3,                           advanced one pass small out,        6446362
-silesia.tar,                        level -1,                           advanced one pass small out,        6186038
-silesia.tar,                        level 0,                            advanced one pass small out,        4861374
-silesia.tar,                        level 1,                            advanced one pass small out,        5334825
-silesia.tar,                        level 3,                            advanced one pass small out,        4861374
-silesia.tar,                        level 4,                            advanced one pass small out,        4799583
-silesia.tar,                        level 5,                            advanced one pass small out,        4722271
-silesia.tar,                        level 6,                            advanced one pass small out,        4672231
-silesia.tar,                        level 7,                            advanced one pass small out,        4606657
-silesia.tar,                        level 9,                            advanced one pass small out,        4554099
-silesia.tar,                        level 13,                           advanced one pass small out,        4491706
-silesia.tar,                        level 16,                           advanced one pass small out,        4381265
-silesia.tar,                        level 19,                           advanced one pass small out,        4281551
-silesia.tar,                        no source size,                     advanced one pass small out,        4861374
-silesia.tar,                        long distance mode,                 advanced one pass small out,        4848046
-silesia.tar,                        multithreaded,                      advanced one pass small out,        4860726
-silesia.tar,                        multithreaded long distance mode,   advanced one pass small out,        4847343
-silesia.tar,                        small window log,                   advanced one pass small out,        7127549
-silesia.tar,                        small hash log,                     advanced one pass small out,        6587833
-silesia.tar,                        small chain log,                    advanced one pass small out,        4943266
-silesia.tar,                        explicit params,                    advanced one pass small out,        4808543
-silesia.tar,                        uncompressed literals,              advanced one pass small out,        5129447
-silesia.tar,                        uncompressed literals optimal,      advanced one pass small out,        4320910
-silesia.tar,                        huffman literals,                   advanced one pass small out,        5347283
-silesia.tar,                        multithreaded with advanced params, advanced one pass small out,        5129766
-github,                             level -5,                           advanced one pass small out,        205285
-github,                             level -5 with dict,                 advanced one pass small out,        46718
-github,                             level -3,                           advanced one pass small out,        190643
-github,                             level -3 with dict,                 advanced one pass small out,        45395
-github,                             level -1,                           advanced one pass small out,        175568
-github,                             level -1 with dict,                 advanced one pass small out,        43170
-github,                             level 0,                            advanced one pass small out,        136311
-github,                             level 0 with dict,                  advanced one pass small out,        41148
-github,                             level 1,                            advanced one pass small out,        142450
-github,                             level 1 with dict,                  advanced one pass small out,        41682
-github,                             level 3,                            advanced one pass small out,        136311
-github,                             level 3 with dict,                  advanced one pass small out,        41148
-github,                             level 4,                            advanced one pass small out,        136144
-github,                             level 4 with dict,                  advanced one pass small out,        41251
-github,                             level 5,                            advanced one pass small out,        135106
-github,                             level 5 with dict,                  advanced one pass small out,        38938
-github,                             level 6,                            advanced one pass small out,        135108
-github,                             level 6 with dict,                  advanced one pass small out,        38632
-github,                             level 7,                            advanced one pass small out,        135108
-github,                             level 7 with dict,                  advanced one pass small out,        38766
-github,                             level 9,                            advanced one pass small out,        135108
-github,                             level 9 with dict,                  advanced one pass small out,        39326
-github,                             level 13,                           advanced one pass small out,        133717
-github,                             level 13 with dict,                 advanced one pass small out,        39716
-github,                             level 16,                           advanced one pass small out,        133717
-github,                             level 16 with dict,                 advanced one pass small out,        37577
-github,                             level 19,                           advanced one pass small out,        133717
-github,                             level 19 with dict,                 advanced one pass small out,        37576
-github,                             no source size,                     advanced one pass small out,        136311
-github,                             long distance mode,                 advanced one pass small out,        136311
-github,                             multithreaded,                      advanced one pass small out,        136311
-github,                             multithreaded long distance mode,   advanced one pass small out,        136311
-github,                             small window log,                   advanced one pass small out,        136311
-github,                             small hash log,                     advanced one pass small out,        135467
-github,                             small chain log,                    advanced one pass small out,        136314
-github,                             explicit params,                    advanced one pass small out,        137670
-github,                             uncompressed literals,              advanced one pass small out,        165915
-github,                             uncompressed literals optimal,      advanced one pass small out,        156824
-github,                             huffman literals,                   advanced one pass small out,        142450
-github,                             multithreaded with advanced params, advanced one pass small out,        165915
-silesia,                            level -5,                           advanced streaming,                 6882466
-silesia,                            level -3,                           advanced streaming,                 6568358
-silesia,                            level -1,                           advanced streaming,                 6183385
-silesia,                            level 0,                            advanced streaming,                 4849491
-silesia,                            level 1,                            advanced streaming,                 5314109
-silesia,                            level 3,                            advanced streaming,                 4849491
-silesia,                            level 4,                            advanced streaming,                 4786913
-silesia,                            level 5,                            advanced streaming,                 4710178
-silesia,                            level 6,                            advanced streaming,                 4659996
-silesia,                            level 7,                            advanced streaming,                 4596234
-silesia,                            level 9,                            advanced streaming,                 4543862
-silesia,                            level 13,                           advanced streaming,                 4482073
-silesia,                            level 16,                           advanced streaming,                 4377389
-silesia,                            level 19,                           advanced streaming,                 4293262
-silesia,                            no source size,                     advanced streaming,                 4849455
-silesia,                            long distance mode,                 advanced streaming,                 4839650
-silesia,                            multithreaded,                      advanced streaming,                 4849491
-silesia,                            multithreaded long distance mode,   advanced streaming,                 4839650
-silesia,                            small window log,                   advanced streaming,                 7123534
-silesia,                            small hash log,                     advanced streaming,                 6554898
-silesia,                            small chain log,                    advanced streaming,                 4931093
-silesia,                            explicit params,                    advanced streaming,                 4797048
-silesia,                            uncompressed literals,              advanced streaming,                 5127960
-silesia,                            uncompressed literals optimal,      advanced streaming,                 4325434
-silesia,                            huffman literals,                   advanced streaming,                 5331110
-silesia,                            multithreaded with advanced params, advanced streaming,                 5127960
-silesia.tar,                        level -5,                           advanced streaming,                 6982738
-silesia.tar,                        level -3,                           advanced streaming,                 6641264
-silesia.tar,                        level -1,                           advanced streaming,                 6190789
-silesia.tar,                        level 0,                            advanced streaming,                 4861376
-silesia.tar,                        level 1,                            advanced streaming,                 5336879
-silesia.tar,                        level 3,                            advanced streaming,                 4861376
-silesia.tar,                        level 4,                            advanced streaming,                 4799583
-silesia.tar,                        level 5,                            advanced streaming,                 4722276
-silesia.tar,                        level 6,                            advanced streaming,                 4672240
-silesia.tar,                        level 7,                            advanced streaming,                 4606657
-silesia.tar,                        level 9,                            advanced streaming,                 4554106
-silesia.tar,                        level 13,                           advanced streaming,                 4491707
-silesia.tar,                        level 16,                           advanced streaming,                 4381284
-silesia.tar,                        level 19,                           advanced streaming,                 4281511
-silesia.tar,                        no source size,                     advanced streaming,                 4861372
-silesia.tar,                        long distance mode,                 advanced streaming,                 4848046
-silesia.tar,                        multithreaded,                      advanced streaming,                 4861458
-silesia.tar,                        multithreaded long distance mode,   advanced streaming,                 4853136
-silesia.tar,                        small window log,                   advanced streaming,                 7127549
-silesia.tar,                        small hash log,                     advanced streaming,                 6587834
-silesia.tar,                        small chain log,                    advanced streaming,                 4943271
-silesia.tar,                        explicit params,                    advanced streaming,                 4808570
-silesia.tar,                        uncompressed literals,              advanced streaming,                 5129450
-silesia.tar,                        uncompressed literals optimal,      advanced streaming,                 4320841
-silesia.tar,                        huffman literals,                   advanced streaming,                 5352306
-silesia.tar,                        multithreaded with advanced params, advanced streaming,                 5129544
-github,                             level -5,                           advanced streaming,                 205285
-github,                             level -5 with dict,                 advanced streaming,                 46718
-github,                             level -3,                           advanced streaming,                 190643
-github,                             level -3 with dict,                 advanced streaming,                 45395
-github,                             level -1,                           advanced streaming,                 175568
-github,                             level -1 with dict,                 advanced streaming,                 43170
-github,                             level 0,                            advanced streaming,                 136311
-github,                             level 0 with dict,                  advanced streaming,                 41148
-github,                             level 1,                            advanced streaming,                 142450
-github,                             level 1 with dict,                  advanced streaming,                 41682
-github,                             level 3,                            advanced streaming,                 136311
-github,                             level 3 with dict,                  advanced streaming,                 41148
-github,                             level 4,                            advanced streaming,                 136144
-github,                             level 4 with dict,                  advanced streaming,                 41251
-github,                             level 5,                            advanced streaming,                 135106
-github,                             level 5 with dict,                  advanced streaming,                 38938
-github,                             level 6,                            advanced streaming,                 135108
-github,                             level 6 with dict,                  advanced streaming,                 38632
-github,                             level 7,                            advanced streaming,                 135108
-github,                             level 7 with dict,                  advanced streaming,                 38766
-github,                             level 9,                            advanced streaming,                 135108
-github,                             level 9 with dict,                  advanced streaming,                 39326
-github,                             level 13,                           advanced streaming,                 133717
-github,                             level 13 with dict,                 advanced streaming,                 39716
-github,                             level 16,                           advanced streaming,                 133717
-github,                             level 16 with dict,                 advanced streaming,                 37577
-github,                             level 19,                           advanced streaming,                 133717
-github,                             level 19 with dict,                 advanced streaming,                 37576
-github,                             no source size,                     advanced streaming,                 136311
-github,                             long distance mode,                 advanced streaming,                 136311
-github,                             multithreaded,                      advanced streaming,                 136311
-github,                             multithreaded long distance mode,   advanced streaming,                 136311
-github,                             small window log,                   advanced streaming,                 136311
-github,                             small hash log,                     advanced streaming,                 135467
-github,                             small chain log,                    advanced streaming,                 136314
-github,                             explicit params,                    advanced streaming,                 137670
-github,                             uncompressed literals,              advanced streaming,                 165915
-github,                             uncompressed literals optimal,      advanced streaming,                 156824
-github,                             huffman literals,                   advanced streaming,                 142450
-github,                             multithreaded with advanced params, advanced streaming,                 165915
-silesia,                            level -5,                           old streaming,                      6882466
-silesia,                            level -3,                           old streaming,                      6568358
-silesia,                            level -1,                           old streaming,                      6183385
-silesia,                            level 0,                            old streaming,                      4849491
-silesia,                            level 1,                            old streaming,                      5314109
-silesia,                            level 3,                            old streaming,                      4849491
-silesia,                            level 4,                            old streaming,                      4786913
-silesia,                            level 5,                            old streaming,                      4710178
-silesia,                            level 6,                            old streaming,                      4659996
-silesia,                            level 7,                            old streaming,                      4596234
-silesia,                            level 9,                            old streaming,                      4543862
-silesia,                            level 13,                           old streaming,                      4482073
-silesia,                            level 16,                           old streaming,                      4377389
-silesia,                            level 19,                           old streaming,                      4293262
-silesia,                            no source size,                     old streaming,                      4849455
-silesia,                            uncompressed literals,              old streaming,                      4849491
-silesia,                            uncompressed literals optimal,      old streaming,                      4293262
-silesia,                            huffman literals,                   old streaming,                      6183385
-silesia.tar,                        level -5,                           old streaming,                      6982738
-silesia.tar,                        level -3,                           old streaming,                      6641264
-silesia.tar,                        level -1,                           old streaming,                      6190789
-silesia.tar,                        level 0,                            old streaming,                      4861376
-silesia.tar,                        level 1,                            old streaming,                      5336879
-silesia.tar,                        level 3,                            old streaming,                      4861376
-silesia.tar,                        level 4,                            old streaming,                      4799583
-silesia.tar,                        level 5,                            old streaming,                      4722276
-silesia.tar,                        level 6,                            old streaming,                      4672240
-silesia.tar,                        level 7,                            old streaming,                      4606657
-silesia.tar,                        level 9,                            old streaming,                      4554106
-silesia.tar,                        level 13,                           old streaming,                      4491707
-silesia.tar,                        level 16,                           old streaming,                      4381284
-silesia.tar,                        level 19,                           old streaming,                      4281511
-silesia.tar,                        no source size,                     old streaming,                      4861372
-silesia.tar,                        uncompressed literals,              old streaming,                      4861376
-silesia.tar,                        uncompressed literals optimal,      old streaming,                      4281511
-silesia.tar,                        huffman literals,                   old streaming,                      6190789
-github,                             level -5,                           old streaming,                      205285
-github,                             level -5 with dict,                 old streaming,                      46718
-github,                             level -3,                           old streaming,                      190643
-github,                             level -3 with dict,                 old streaming,                      45395
-github,                             level -1,                           old streaming,                      175568
-github,                             level -1 with dict,                 old streaming,                      43170
-github,                             level 0,                            old streaming,                      136311
-github,                             level 0 with dict,                  old streaming,                      41148
-github,                             level 1,                            old streaming,                      142450
-github,                             level 1 with dict,                  old streaming,                      41682
-github,                             level 3,                            old streaming,                      136311
-github,                             level 3 with dict,                  old streaming,                      41148
-github,                             level 4,                            old streaming,                      136144
-github,                             level 4 with dict,                  old streaming,                      41251
-github,                             level 5,                            old streaming,                      135106
-github,                             level 5 with dict,                  old streaming,                      38938
-github,                             level 6,                            old streaming,                      135108
-github,                             level 6 with dict,                  old streaming,                      38632
-github,                             level 7,                            old streaming,                      135108
-github,                             level 7 with dict,                  old streaming,                      38766
-github,                             level 9,                            old streaming,                      135108
-github,                             level 9 with dict,                  old streaming,                      39326
-github,                             level 13,                           old streaming,                      133717
-github,                             level 13 with dict,                 old streaming,                      39716
-github,                             level 16,                           old streaming,                      133717
-github,                             level 16 with dict,                 old streaming,                      37577
-github,                             level 19,                           old streaming,                      133717
-github,                             level 19 with dict,                 old streaming,                      37576
-github,                             no source size,                     old streaming,                      140631
-github,                             uncompressed literals,              old streaming,                      136311
-github,                             uncompressed literals optimal,      old streaming,                      133717
-github,                             huffman literals,                   old streaming,                      175568
-silesia,                            level -5,                           old streaming advanced,             6882466
-silesia,                            level -3,                           old streaming advanced,             6568358
-silesia,                            level -1,                           old streaming advanced,             6183385
-silesia,                            level 0,                            old streaming advanced,             4849491
-silesia,                            level 1,                            old streaming advanced,             5314109
-silesia,                            level 3,                            old streaming advanced,             4849491
-silesia,                            level 4,                            old streaming advanced,             4786913
-silesia,                            level 5,                            old streaming advanced,             4710178
-silesia,                            level 6,                            old streaming advanced,             4659996
-silesia,                            level 7,                            old streaming advanced,             4596234
-silesia,                            level 9,                            old streaming advanced,             4543862
-silesia,                            level 13,                           old streaming advanced,             4482073
-silesia,                            level 16,                           old streaming advanced,             4377389
-silesia,                            level 19,                           old streaming advanced,             4293262
-silesia,                            no source size,                     old streaming advanced,             4849455
-silesia,                            long distance mode,                 old streaming advanced,             4849491
-silesia,                            multithreaded,                      old streaming advanced,             4849491
-silesia,                            multithreaded long distance mode,   old streaming advanced,             4849491
-silesia,                            small window log,                   old streaming advanced,             7123534
-silesia,                            small hash log,                     old streaming advanced,             6554898
-silesia,                            small chain log,                    old streaming advanced,             4931093
-silesia,                            explicit params,                    old streaming advanced,             4797048
-silesia,                            uncompressed literals,              old streaming advanced,             4849491
-silesia,                            uncompressed literals optimal,      old streaming advanced,             4293262
-silesia,                            huffman literals,                   old streaming advanced,             6183385
-silesia,                            multithreaded with advanced params, old streaming advanced,             4849491
-silesia.tar,                        level -5,                           old streaming advanced,             6982738
-silesia.tar,                        level -3,                           old streaming advanced,             6641264
-silesia.tar,                        level -1,                           old streaming advanced,             6190789
-silesia.tar,                        level 0,                            old streaming advanced,             4861376
-silesia.tar,                        level 1,                            old streaming advanced,             5336879
-silesia.tar,                        level 3,                            old streaming advanced,             4861376
-silesia.tar,                        level 4,                            old streaming advanced,             4799583
-silesia.tar,                        level 5,                            old streaming advanced,             4722276
-silesia.tar,                        level 6,                            old streaming advanced,             4672240
-silesia.tar,                        level 7,                            old streaming advanced,             4606657
-silesia.tar,                        level 9,                            old streaming advanced,             4554106
-silesia.tar,                        level 13,                           old streaming advanced,             4491707
-silesia.tar,                        level 16,                           old streaming advanced,             4381284
-silesia.tar,                        level 19,                           old streaming advanced,             4281511
-silesia.tar,                        no source size,                     old streaming advanced,             4861372
-silesia.tar,                        long distance mode,                 old streaming advanced,             4861376
-silesia.tar,                        multithreaded,                      old streaming advanced,             4861376
-silesia.tar,                        multithreaded long distance mode,   old streaming advanced,             4861376
-silesia.tar,                        small window log,                   old streaming advanced,             7127552
-silesia.tar,                        small hash log,                     old streaming advanced,             6587834
-silesia.tar,                        small chain log,                    old streaming advanced,             4943271
-silesia.tar,                        explicit params,                    old streaming advanced,             4808570
-silesia.tar,                        uncompressed literals,              old streaming advanced,             4861376
-silesia.tar,                        uncompressed literals optimal,      old streaming advanced,             4281511
-silesia.tar,                        huffman literals,                   old streaming advanced,             6190789
-silesia.tar,                        multithreaded with advanced params, old streaming advanced,             4861376
-github,                             level -5,                           old streaming advanced,             216734
-github,                             level -5 with dict,                 old streaming advanced,             49562
-github,                             level -3,                           old streaming advanced,             192160
-github,                             level -3 with dict,                 old streaming advanced,             44956
-github,                             level -1,                           old streaming advanced,             181108
-github,                             level -1 with dict,                 old streaming advanced,             42383
-github,                             level 0,                            old streaming advanced,             141090
-github,                             level 0 with dict,                  old streaming advanced,             41113
-github,                             level 1,                            old streaming advanced,             143682
-github,                             level 1 with dict,                  old streaming advanced,             42430
-github,                             level 3,                            old streaming advanced,             141090
-github,                             level 3 with dict,                  old streaming advanced,             41113
-github,                             level 4,                            old streaming advanced,             141090
-github,                             level 4 with dict,                  old streaming advanced,             41084
-github,                             level 5,                            old streaming advanced,             139391
-github,                             level 5 with dict,                  old streaming advanced,             39159
-github,                             level 6,                            old streaming advanced,             139394
-github,                             level 6 with dict,                  old streaming advanced,             38749
-github,                             level 7,                            old streaming advanced,             138675
-github,                             level 7 with dict,                  old streaming advanced,             38746
-github,                             level 9,                            old streaming advanced,             138675
-github,                             level 9 with dict,                  old streaming advanced,             38987
-github,                             level 13,                           old streaming advanced,             138675
-github,                             level 13 with dict,                 old streaming advanced,             39724
-github,                             level 16,                           old streaming advanced,             138675
-github,                             level 16 with dict,                 old streaming advanced,             40771
-github,                             level 19,                           old streaming advanced,             133717
-github,                             level 19 with dict,                 old streaming advanced,             37576
-github,                             no source size,                     old streaming advanced,             140631
-github,                             long distance mode,                 old streaming advanced,             141090
-github,                             multithreaded,                      old streaming advanced,             141090
-github,                             multithreaded long distance mode,   old streaming advanced,             141090
-github,                             small window log,                   old streaming advanced,             141090
-github,                             small hash log,                     old streaming advanced,             141578
-github,                             small chain log,                    old streaming advanced,             139258
-github,                             explicit params,                    old streaming advanced,             140930
-github,                             uncompressed literals,              old streaming advanced,             141090
-github,                             uncompressed literals optimal,      old streaming advanced,             133717
-github,                             huffman literals,                   old streaming advanced,             181108
-github,                             multithreaded with advanced params, old streaming advanced,             141090
-github,                             level -5 with dict,                 old streaming cdcit,                46718
-github,                             level -3 with dict,                 old streaming cdcit,                45395
-github,                             level -1 with dict,                 old streaming cdcit,                43170
-github,                             level 0 with dict,                  old streaming cdcit,                41148
-github,                             level 1 with dict,                  old streaming cdcit,                41682
-github,                             level 3 with dict,                  old streaming cdcit,                41148
-github,                             level 4 with dict,                  old streaming cdcit,                41251
-github,                             level 5 with dict,                  old streaming cdcit,                38938
-github,                             level 6 with dict,                  old streaming cdcit,                38632
-github,                             level 7 with dict,                  old streaming cdcit,                38766
-github,                             level 9 with dict,                  old streaming cdcit,                39326
-github,                             level 13 with dict,                 old streaming cdcit,                39716
-github,                             level 16 with dict,                 old streaming cdcit,                37577
-github,                             level 19 with dict,                 old streaming cdcit,                37576
-github,                             level -5 with dict,                 old streaming advanced cdict,       49562
-github,                             level -3 with dict,                 old streaming advanced cdict,       44956
-github,                             level -1 with dict,                 old streaming advanced cdict,       42383
-github,                             level 0 with dict,                  old streaming advanced cdict,       41113
-github,                             level 1 with dict,                  old streaming advanced cdict,       42430
-github,                             level 3 with dict,                  old streaming advanced cdict,       41113
-github,                             level 4 with dict,                  old streaming advanced cdict,       41084
-github,                             level 5 with dict,                  old streaming advanced cdict,       39158
-github,                             level 6 with dict,                  old streaming advanced cdict,       38748
-github,                             level 7 with dict,                  old streaming advanced cdict,       38744
-github,                             level 9 with dict,                  old streaming advanced cdict,       38986
-github,                             level 13 with dict,                 old streaming advanced cdict,       39724
-github,                             level 16 with dict,                 old streaming advanced cdict,       40771
-github,                             level 19 with dict,                 old streaming advanced cdict,       37576
diff --git a/tests/regression/test.c b/tests/regression/test.c
deleted file mode 100644
index 812893b2730e..000000000000
--- a/tests/regression/test.c
+++ /dev/null
@@ -1,362 +0,0 @@
-/*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#include <assert.h>
-#include <getopt.h>
-#include <stdio.h>
-#include <string.h>
-
-#include "config.h"
-#include "data.h"
-#include "method.h"
-
-static int g_max_name_len = 0;
-
-/** Check if a name contains a comma or is too long. */
-static int is_name_bad(char const* name) {
-    if (name == NULL)
-        return 1;
-    int const len = strlen(name);
-    if (len > g_max_name_len)
-        g_max_name_len = len;
-    for (; *name != '\0'; ++name)
-        if (*name == ',')
-            return 1;
-    return 0;
-}
-
-/** Check if any of the names contain a comma. */
-static int are_names_bad() {
-    for (size_t method = 0; methods[method] != NULL; ++method)
-        if (is_name_bad(methods[method]->name)) {
-            fprintf(stderr, "method name %s is bad\n", methods[method]->name);
-            return 1;
-        }
-    for (size_t datum = 0; data[datum] != NULL; ++datum)
-        if (is_name_bad(data[datum]->name)) {
-            fprintf(stderr, "data name %s is bad\n", data[datum]->name);
-            return 1;
-        }
-    for (size_t config = 0; configs[config] != NULL; ++config)
-        if (is_name_bad(configs[config]->name)) {
-            fprintf(stderr, "config name %s is bad\n", configs[config]->name);
-            return 1;
-        }
-    return 0;
-}
-
-/**
- * Option parsing using getopt.
- * When you add a new option update: long_options, long_extras, and
- * short_options.
- */
-
-/** Option variables filled by parse_args. */
-static char const* g_output = NULL;
-static char const* g_diff = NULL;
-static char const* g_cache = NULL;
-static char const* g_zstdcli = NULL;
-static char const* g_config = NULL;
-static char const* g_data = NULL;
-static char const* g_method = NULL;
-
-typedef enum {
-    required_option,
-    optional_option,
-    help_option,
-} option_type;
-
-/**
- * Extra state that we need to keep per-option that we can't store in getopt.
- */
-struct option_extra {
-    int id; /**< The short option name, used as an id. */
-    char const* help; /**< The help message. */
-    option_type opt_type; /**< The option type: required, optional, or help. */
-    char const** value; /**< The value to set or NULL if no_argument. */
-};
-
-/** The options. */
-static struct option long_options[] = {
-    {"cache", required_argument, NULL, 'c'},
-    {"output", required_argument, NULL, 'o'},
-    {"zstd", required_argument, NULL, 'z'},
-    {"config", required_argument, NULL, 128},
-    {"data", required_argument, NULL, 129},
-    {"method", required_argument, NULL, 130},
-    {"diff", required_argument, NULL, 'd'},
-    {"help", no_argument, NULL, 'h'},
-};
-
-static size_t const nargs = sizeof(long_options) / sizeof(long_options[0]);
-
-/** The extra info for the options. Must be in the same order as the options. */
-static struct option_extra long_extras[] = {
-    {'c', "the cache directory", required_option, &g_cache},
-    {'o', "write the results here", required_option, &g_output},
-    {'z', "zstd cli tool", required_option, &g_zstdcli},
-    {128, "use this config", optional_option, &g_config},
-    {129, "use this data", optional_option, &g_data},
-    {130, "use this method", optional_option, &g_method},
-    {'d', "compare the results to this file", optional_option, &g_diff},
-    {'h', "display this message", help_option, NULL},
-};
-
-/** The short options. Must correspond to the options. */
-static char const short_options[] = "c:d:ho:z:";
-
-/** Return the help string for the option type. */
-static char const* required_message(option_type opt_type) {
-    switch (opt_type) {
-        case required_option:
-            return "[required]";
-        case optional_option:
-            return "[optional]";
-        case help_option:
-            return "";
-        default:
-            assert(0);
-            return NULL;
-    }
-}
-
-/** Print the help for the program. */
-static void print_help(void) {
-    fprintf(stderr, "regression test runner\n");
-    size_t const nargs = sizeof(long_options) / sizeof(long_options[0]);
-    for (size_t i = 0; i < nargs; ++i) {
-        if (long_options[i].val < 128) {
-            /* Long / short  - help [option type] */
-            fprintf(
-                stderr,
-                "--%s / -%c \t- %s %s\n",
-                long_options[i].name,
-                long_options[i].val,
-                long_extras[i].help,
-                required_message(long_extras[i].opt_type));
-        } else {
-            /* Short / long  - help [option type] */
-            fprintf(
-                stderr,
-                "--%s      \t- %s %s\n",
-                long_options[i].name,
-                long_extras[i].help,
-                required_message(long_extras[i].opt_type));
-        }
-    }
-}
-
-/** Parse the arguments. Return 0 on success. Print help on failure. */
-static int parse_args(int argc, char** argv) {
-    int option_index = 0;
-    int c;
-
-    while (1) {
-        c = getopt_long(argc, argv, short_options, long_options, &option_index);
-        if (c == -1)
-            break;
-
-        int found = 0;
-        for (size_t i = 0; i < nargs; ++i) {
-            if (c == long_extras[i].id && long_extras[i].value != NULL) {
-                *long_extras[i].value = optarg;
-                found = 1;
-                break;
-            }
-        }
-        if (found)
-            continue;
-
-        switch (c) {
-            case 'h':
-            case '?':
-            default:
-                print_help();
-                return 1;
-        }
-    }
-
-    int bad = 0;
-    for (size_t i = 0; i < nargs; ++i) {
-        if (long_extras[i].opt_type != required_option)
-            continue;
-        if (long_extras[i].value == NULL)
-            continue;
-        if (*long_extras[i].value != NULL)
-            continue;
-        fprintf(
-            stderr,
-            "--%s is a required argument but is not set\n",
-            long_options[i].name);
-        bad = 1;
-    }
-    if (bad) {
-        fprintf(stderr, "\n");
-        print_help();
-        return 1;
-    }
-
-    return 0;
-}
-
-/** Helper macro to print to stderr and a file. */
-#define tprintf(file, ...)            \
-    do {                              \
-        fprintf(file, __VA_ARGS__);   \
-        fprintf(stderr, __VA_ARGS__); \
-    } while (0)
-/** Helper macro to flush stderr and a file. */
-#define tflush(file)    \
-    do {                \
-        fflush(file);   \
-        fflush(stderr); \
-    } while (0)
-
-void tprint_names(
-    FILE* results,
-    char const* data_name,
-    char const* config_name,
-    char const* method_name) {
-    int const data_padding = g_max_name_len - strlen(data_name);
-    int const config_padding = g_max_name_len - strlen(config_name);
-    int const method_padding = g_max_name_len - strlen(method_name);
-
-    tprintf(
-        results,
-        "%s, %*s%s, %*s%s, %*s",
-        data_name,
-        data_padding,
-        "",
-        config_name,
-        config_padding,
-        "",
-        method_name,
-        method_padding,
-        "");
-}
-
-/**
- * Run all the regression tests and record the results table to results and
- * stderr progressively.
- */
-static int run_all(FILE* results) {
-    tprint_names(results, "Data", "Config", "Method");
-    tprintf(results, "Total compressed size\n");
-    for (size_t method = 0; methods[method] != NULL; ++method) {
-        if (g_method != NULL && strcmp(methods[method]->name, g_method))
-            continue;
-        for (size_t datum = 0; data[datum] != NULL; ++datum) {
-            if (g_data != NULL && strcmp(data[datum]->name, g_data))
-                continue;
-            /* Create the state common to all configs */
-            method_state_t* state = methods[method]->create(data[datum]);
-            for (size_t config = 0; configs[config] != NULL; ++config) {
-                if (g_config != NULL && strcmp(configs[config]->name, g_config))
-                    continue;
-                if (config_skip_data(configs[config], data[datum]))
-                    continue;
-                /* Print the result for the (method, data, config) tuple. */
-                result_t const result =
-                    methods[method]->compress(state, configs[config]);
-                if (result_is_skip(result))
-                    continue;
-                tprint_names(
-                    results,
-                    data[datum]->name,
-                    configs[config]->name,
-                    methods[method]->name);
-                if (result_is_error(result)) {
-                    tprintf(results, "%s\n", result_get_error_string(result));
-                } else {
-                    tprintf(
-                        results,
-                        "%llu\n",
-                        (unsigned long long)result_get_data(result).total_size);
-                }
-                tflush(results);
-            }
-            methods[method]->destroy(state);
-        }
-    }
-    return 0;
-}
-
-/** memcmp() the old results file and the new results file. */
-static int diff_results(char const* actual_file, char const* expected_file) {
-    data_buffer_t const actual = data_buffer_read(actual_file);
-    data_buffer_t const expected = data_buffer_read(expected_file);
-    int ret = 1;
-
-    if (actual.data == NULL) {
-        fprintf(stderr, "failed to open results '%s' for diff\n", actual_file);
-        goto out;
-    }
-    if (expected.data == NULL) {
-        fprintf(
-            stderr,
-            "failed to open previous results '%s' for diff\n",
-            expected_file);
-        goto out;
-    }
-
-    ret = data_buffer_compare(actual, expected);
-    if (ret != 0) {
-        fprintf(
-            stderr,
-            "actual results '%s' does not match expected results '%s'\n",
-            actual_file,
-            expected_file);
-    } else {
-        fprintf(stderr, "actual results match expected results\n");
-    }
-out:
-    data_buffer_free(actual);
-    data_buffer_free(expected);
-    return ret;
-}
-
-int main(int argc, char** argv) {
-    /* Parse args and validate modules. */
-    int ret = parse_args(argc, argv);
-    if (ret != 0)
-        return ret;
-
-    if (are_names_bad())
-        return 1;
-
-    /* Initialize modules. */
-    method_set_zstdcli(g_zstdcli);
-    ret = data_init(g_cache);
-    if (ret != 0) {
-        fprintf(stderr, "data_init() failed with error=%s\n", strerror(ret));
-        return 1;
-    }
-
-    /* Run the regression tests. */
-    ret = 1;
-    FILE* results = fopen(g_output, "w");
-    if (results == NULL) {
-        fprintf(stderr, "Failed to open the output file\n");
-        goto out;
-    }
-    ret = run_all(results);
-    fclose(results);
-
-    if (ret != 0)
-        goto out;
-
-    if (g_diff)
-        /* Diff the new results with the previous results. */
-        ret = diff_results(g_output, g_diff);
-
-out:
-    data_finish();
-    return ret;
-}
diff --git a/tests/roundTripCrash.c b/tests/roundTripCrash.c
deleted file mode 100644
index 3de5933185d5..000000000000
--- a/tests/roundTripCrash.c
+++ /dev/null
@@ -1,241 +0,0 @@
-/*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-/*
-  This program takes a file in input,
-  performs a zstd round-trip test (compression - decompress)
-  compares the result with original
-  and generates a crash (double free) on corruption detection.
-*/
-
-/*===========================================
-*   Dependencies
-*==========================================*/
-#include <stddef.h>     /* size_t */
-#include <stdlib.h>     /* malloc, free, exit */
-#include <stdio.h>      /* fprintf */
-#include <string.h>     /* strcmp */
-#include <sys/types.h>  /* stat */
-#include <sys/stat.h>   /* stat */
-#include "xxhash.h"
-
-#define ZSTD_STATIC_LINKING_ONLY
-#include "zstd.h"
-
-/*===========================================
-*   Macros
-*==========================================*/
-#define MIN(a,b)  ( (a) < (b) ? (a) : (b) )
-
-static void crash(int errorCode){
-    /* abort if AFL/libfuzzer, exit otherwise */
-    #ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION /* could also use __AFL_COMPILER */
-        abort();
-    #else
-        exit(errorCode);
-    #endif
-}
-
-#define CHECK_Z(f) {                            \
-    size_t const err = f;                       \
-    if (ZSTD_isError(err)) {                    \
-        fprintf(stderr,                         \
-                "Error=> %s: %s",               \
-                #f, ZSTD_getErrorName(err));    \
-        crash(1);                                \
-}   }
-
-/** roundTripTest() :
-*   Compresses `srcBuff` into `compressedBuff`,
-*   then decompresses `compressedBuff` into `resultBuff`.
-*   Compression level used is derived from first content byte.
-*   @return : result of decompression, which should be == `srcSize`
-*          or an error code if either compression or decompression fails.
-*   Note : `compressedBuffCapacity` should be `>= ZSTD_compressBound(srcSize)`
-*          for compression to be guaranteed to work */
-static size_t roundTripTest(void* resultBuff, size_t resultBuffCapacity,
-                            void* compressedBuff, size_t compressedBuffCapacity,
-                      const void* srcBuff, size_t srcBuffSize)
-{
-    static const int maxClevel = 19;
-    size_t const hashLength = MIN(128, srcBuffSize);
-    unsigned const h32 = XXH32(srcBuff, hashLength, 0);
-    int const cLevel = h32 % maxClevel;
-    size_t const cSize = ZSTD_compress(compressedBuff, compressedBuffCapacity, srcBuff, srcBuffSize, cLevel);
-    if (ZSTD_isError(cSize)) {
-        fprintf(stderr, "Compression error : %s \n", ZSTD_getErrorName(cSize));
-        return cSize;
-    }
-    return ZSTD_decompress(resultBuff, resultBuffCapacity, compressedBuff, cSize);
-}
-
-/** cctxParamRoundTripTest() :
- *  Same as roundTripTest() except allows experimenting with ZSTD_CCtx_params. */
-static size_t cctxParamRoundTripTest(void* resultBuff, size_t resultBuffCapacity,
-                            void* compressedBuff, size_t compressedBuffCapacity,
-                      const void* srcBuff, size_t srcBuffSize)
-{
-    ZSTD_CCtx* const cctx = ZSTD_createCCtx();
-    ZSTD_CCtx_params* const cctxParams = ZSTD_createCCtxParams();
-    ZSTD_inBuffer inBuffer = { srcBuff, srcBuffSize, 0 };
-    ZSTD_outBuffer outBuffer = { compressedBuff, compressedBuffCapacity, 0 };
-
-    static const int maxClevel = 19;
-    size_t const hashLength = MIN(128, srcBuffSize);
-    unsigned const h32 = XXH32(srcBuff, hashLength, 0);
-    int const cLevel = h32 % maxClevel;
-
-    /* Set parameters */
-    CHECK_Z( ZSTD_CCtxParams_setParameter(cctxParams, ZSTD_c_compressionLevel, cLevel) );
-    CHECK_Z( ZSTD_CCtxParams_setParameter(cctxParams, ZSTD_c_nbWorkers, 2) );
-    CHECK_Z( ZSTD_CCtxParams_setParameter(cctxParams, ZSTD_c_overlapLog, 5) );
-
-
-    /* Apply parameters */
-    CHECK_Z( ZSTD_CCtx_setParametersUsingCCtxParams(cctx, cctxParams) );
-
-    CHECK_Z (ZSTD_compressStream2(cctx, &outBuffer, &inBuffer, ZSTD_e_end) );
-
-    ZSTD_freeCCtxParams(cctxParams);
-    ZSTD_freeCCtx(cctx);
-
-    return ZSTD_decompress(resultBuff, resultBuffCapacity, compressedBuff, outBuffer.pos);
-}
-
-static size_t checkBuffers(const void* buff1, const void* buff2, size_t buffSize)
-{
-    const char* ip1 = (const char*)buff1;
-    const char* ip2 = (const char*)buff2;
-    size_t pos;
-
-    for (pos=0; pos<buffSize; pos++)
-        if (ip1[pos]!=ip2[pos])
-            break;
-
-    return pos;
-}
-
-static void roundTripCheck(const void* srcBuff, size_t srcBuffSize, int testCCtxParams)
-{
-    size_t const cBuffSize = ZSTD_compressBound(srcBuffSize);
-    void* cBuff = malloc(cBuffSize);
-    void* rBuff = malloc(cBuffSize);
-
-    if (!cBuff || !rBuff) {
-        fprintf(stderr, "not enough memory ! \n");
-        exit (1);
-    }
-
-    {   size_t const result = testCCtxParams ?
-                  cctxParamRoundTripTest(rBuff, cBuffSize, cBuff, cBuffSize, srcBuff, srcBuffSize)
-                : roundTripTest(rBuff, cBuffSize, cBuff, cBuffSize, srcBuff, srcBuffSize);
-        if (ZSTD_isError(result)) {
-            fprintf(stderr, "roundTripTest error : %s \n", ZSTD_getErrorName(result));
-            crash(1);
-        }
-        if (result != srcBuffSize) {
-            fprintf(stderr, "Incorrect regenerated size : %u != %u\n", (unsigned)result, (unsigned)srcBuffSize);
-            crash(1);
-        }
-        if (checkBuffers(srcBuff, rBuff, srcBuffSize) != srcBuffSize) {
-            fprintf(stderr, "Silent decoding corruption !!!");
-            crash(1);
-        }
-    }
-
-    free(cBuff);
-    free(rBuff);
-}
-
-
-static size_t getFileSize(const char* infilename)
-{
-    int r;
-#if defined(_MSC_VER)
-    struct _stat64 statbuf;
-    r = _stat64(infilename, &statbuf);
-    if (r || !(statbuf.st_mode & S_IFREG)) return 0;   /* No good... */
-#else
-    struct stat statbuf;
-    r = stat(infilename, &statbuf);
-    if (r || !S_ISREG(statbuf.st_mode)) return 0;   /* No good... */
-#endif
-    return (size_t)statbuf.st_size;
-}
-
-
-static int isDirectory(const char* infilename)
-{
-    int r;
-#if defined(_MSC_VER)
-    struct _stat64 statbuf;
-    r = _stat64(infilename, &statbuf);
-    if (!r && (statbuf.st_mode & _S_IFDIR)) return 1;
-#else
-    struct stat statbuf;
-    r = stat(infilename, &statbuf);
-    if (!r && S_ISDIR(statbuf.st_mode)) return 1;
-#endif
-    return 0;
-}
-
-
-/** loadFile() :
-*   requirement : `buffer` size >= `fileSize` */
-static void loadFile(void* buffer, const char* fileName, size_t fileSize)
-{
-    FILE* const f = fopen(fileName, "rb");
-    if (isDirectory(fileName)) {
-        fprintf(stderr, "Ignoring %s directory \n", fileName);
-        exit(2);
-    }
-    if (f==NULL) {
-        fprintf(stderr, "Impossible to open %s \n", fileName);
-        exit(3);
-    }
-    {   size_t const readSize = fread(buffer, 1, fileSize, f);
-        if (readSize != fileSize) {
-            fprintf(stderr, "Error reading %s \n", fileName);
-            exit(5);
-    }   }
-    fclose(f);
-}
-
-
-static void fileCheck(const char* fileName, int testCCtxParams)
-{
-    size_t const fileSize = getFileSize(fileName);
-    void* const buffer = malloc(fileSize + !fileSize /* avoid 0 */);
-    if (!buffer) {
-        fprintf(stderr, "not enough memory \n");
-        exit(4);
-    }
-    loadFile(buffer, fileName, fileSize);
-    roundTripCheck(buffer, fileSize, testCCtxParams);
-    free (buffer);
-}
-
-int main(int argCount, const char** argv) {
-    int argNb = 1;
-    int testCCtxParams = 0;
-    if (argCount < 2) {
-        fprintf(stderr, "Error : no argument : need input file \n");
-        exit(9);
-    }
-
-    if (!strcmp(argv[argNb], "--cctxParams")) {
-      testCCtxParams = 1;
-      argNb++;
-    }
-
-    fileCheck(argv[argNb], testCCtxParams);
-    fprintf(stderr, "no pb detected\n");
-    return 0;
-}
diff --git a/tests/seqgen.c b/tests/seqgen.c
deleted file mode 100644
index 06e3ed5ba0b0..000000000000
--- a/tests/seqgen.c
+++ /dev/null
@@ -1,260 +0,0 @@
-/*
- * Copyright (c) 2017-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#include "seqgen.h"
-#include "mem.h"
-#include <string.h>
-
-#define MIN(a, b)  ((a) < (b) ? (a) : (b))
-
-static const size_t kMatchBytes = 128;
-
-#define SEQ_rotl32(x,r) ((x << r) | (x >> (32 - r)))
-static BYTE SEQ_randByte(unsigned* src)
-{
-    static const U32 prime1 = 2654435761U;
-    static const U32 prime2 = 2246822519U;
-    U32 rand32 = *src;
-    rand32 *= prime1;
-    rand32 ^= prime2;
-    rand32  = SEQ_rotl32(rand32, 13);
-    *src = rand32;
-    return (BYTE)(rand32 >> 5);
-}
-
-SEQ_stream SEQ_initStream(unsigned seed)
-{
-    SEQ_stream stream;
-    stream.state = 0;
-    XXH64_reset(&stream.xxh, 0);
-    stream.seed = seed;
-    return stream;
-}
-
-/* Generates a single guard byte, then match length + 1 of a different byte,
- * then another guard byte.
- */
-static size_t SEQ_gen_matchLength(SEQ_stream* stream, unsigned value,
-                                  SEQ_outBuffer* out)
-{
-    typedef enum {
-        ml_first_byte = 0,
-        ml_match_bytes,
-        ml_last_byte,
-    } ml_state;
-    BYTE* const ostart = (BYTE*)out->dst;
-    BYTE* const oend = ostart + out->size;
-    BYTE* op = ostart + out->pos;
-
-    switch ((ml_state)stream->state) {
-    case ml_first_byte:
-        /* Generate a single byte and pick a different byte for the match */
-        if (op >= oend) {
-            stream->bytesLeft = 1;
-            break;
-        }
-        *op = SEQ_randByte(&stream->seed) & 0xFF;
-        do {
-            stream->saved = SEQ_randByte(&stream->seed) & 0xFF;
-        } while (*op == stream->saved);
-        ++op;
-        /* State transition */
-        stream->state = ml_match_bytes;
-        stream->bytesLeft = value + 1;
-    /* fall-through */
-    case ml_match_bytes: {
-        /* Copy matchLength + 1 bytes to the output buffer */
-        size_t const setLength = MIN(stream->bytesLeft, (size_t)(oend - op));
-        if (setLength > 0) {
-            memset(op, stream->saved, setLength);
-            op += setLength;
-            stream->bytesLeft -= setLength;
-        }
-        if (stream->bytesLeft > 0)
-            break;
-        /* State transition */
-        stream->state = ml_last_byte;
-    }
-    /* fall-through */
-    case ml_last_byte:
-        /* Generate a single byte and pick a different byte for the match */
-        if (op >= oend) {
-            stream->bytesLeft = 1;
-            break;
-        }
-        do {
-            *op = SEQ_randByte(&stream->seed) & 0xFF;
-        } while (*op == stream->saved);
-        ++op;
-        /* State transition */
-    /* fall-through */
-    default:
-        stream->state = 0;
-        stream->bytesLeft = 0;
-        break;
-    }
-    XXH64_update(&stream->xxh, ostart + out->pos, (op - ostart) - out->pos);
-    out->pos = op - ostart;
-    return stream->bytesLeft;
-}
-
-/* Saves the current seed then generates kMatchBytes random bytes >= 128.
- * Generates literal length - kMatchBytes random bytes < 128.
- * Generates another kMatchBytes using the saved seed to generate a match.
- * This way the match is easy to find for the compressors.
- */
-static size_t SEQ_gen_litLength(SEQ_stream* stream, unsigned value, SEQ_outBuffer* out)
-{
-    typedef enum {
-        ll_start = 0,
-        ll_run_bytes,
-        ll_literals,
-        ll_run_match,
-    } ll_state;
-    BYTE* const ostart = (BYTE*)out->dst;
-    BYTE* const oend = ostart + out->size;
-    BYTE* op = ostart + out->pos;
-
-    switch ((ll_state)stream->state) {
-    case ll_start:
-        stream->state = ll_run_bytes;
-        stream->saved = stream->seed;
-        stream->bytesLeft = MIN(kMatchBytes, value);
-    /* fall-through */
-    case ll_run_bytes:
-        while (stream->bytesLeft > 0 && op < oend) {
-            *op++ = SEQ_randByte(&stream->seed) | 0x80;
-            --stream->bytesLeft;
-        }
-        if (stream->bytesLeft > 0)
-            break;
-        /* State transition */
-        stream->state = ll_literals;
-        stream->bytesLeft = value - MIN(kMatchBytes, value);
-    /* fall-through */
-    case ll_literals:
-        while (stream->bytesLeft > 0 && op < oend) {
-            *op++ = SEQ_randByte(&stream->seed) & 0x7F;
-            --stream->bytesLeft;
-        }
-        if (stream->bytesLeft > 0)
-            break;
-        /* State transition */
-        stream->state = ll_run_match;
-        stream->bytesLeft = MIN(kMatchBytes, value);
-    /* fall-through */
-    case ll_run_match: {
-        while (stream->bytesLeft > 0 && op < oend) {
-            *op++ = SEQ_randByte(&stream->saved) | 0x80;
-            --stream->bytesLeft;
-        }
-        if (stream->bytesLeft > 0)
-            break;
-    }
-    /* fall-through */
-    default:
-        stream->state = 0;
-        stream->bytesLeft = 0;
-        break;
-    }
-    XXH64_update(&stream->xxh, ostart + out->pos, (op - ostart) - out->pos);
-    out->pos = op - ostart;
-    return stream->bytesLeft;
-}
-
-/* Saves the current seed then generates kMatchBytes random bytes >= 128.
- * Generates offset - kMatchBytes of zeros to get a large offset without
- * polluting the hash tables.
- * Generates another kMatchBytes using the saved seed to generate a with the
- * required offset.
- */
-static size_t SEQ_gen_offset(SEQ_stream* stream, unsigned value, SEQ_outBuffer* out)
-{
-    typedef enum {
-        of_start = 0,
-        of_run_bytes,
-        of_offset,
-        of_run_match,
-    } of_state;
-    BYTE* const ostart = (BYTE*)out->dst;
-    BYTE* const oend = ostart + out->size;
-    BYTE* op = ostart + out->pos;
-
-    switch ((of_state)stream->state) {
-    case of_start:
-        stream->state = of_run_bytes;
-        stream->saved = stream->seed;
-        stream->bytesLeft = MIN(value, kMatchBytes);
-    /* fall-through */
-    case of_run_bytes: {
-        while (stream->bytesLeft > 0 && op < oend) {
-            *op++ = SEQ_randByte(&stream->seed) | 0x80;
-            --stream->bytesLeft;
-        }
-        if (stream->bytesLeft > 0)
-            break;
-        /* State transition */
-        stream->state = of_offset;
-        stream->bytesLeft = value - MIN(value, kMatchBytes);
-    }
-    /* fall-through */
-    case of_offset: {
-        /* Copy matchLength + 1 bytes to the output buffer */
-        size_t const setLength = MIN(stream->bytesLeft, (size_t)(oend - op));
-        if (setLength > 0) {
-            memset(op, 0, setLength);
-            op += setLength;
-            stream->bytesLeft -= setLength;
-        }
-        if (stream->bytesLeft > 0)
-            break;
-        /* State transition */
-        stream->state = of_run_match;
-        stream->bytesLeft = MIN(value, kMatchBytes);
-    }
-    /* fall-through */
-    case of_run_match: {
-        while (stream->bytesLeft > 0 && op < oend) {
-            *op++ = SEQ_randByte(&stream->saved) | 0x80;
-            --stream->bytesLeft;
-        }
-        if (stream->bytesLeft > 0)
-            break;
-    }
-    /* fall-through */
-    default:
-        stream->state = 0;
-        stream->bytesLeft = 0;
-        break;
-    }
-    XXH64_update(&stream->xxh, ostart + out->pos, (op - ostart) - out->pos);
-    out->pos = op - ostart;
-    return stream->bytesLeft;
-}
-
-/* Returns the number of bytes left to generate.
- * Must pass the same type/value until it returns 0.
- */
-size_t SEQ_gen(SEQ_stream* stream, SEQ_gen_type type, unsigned value, SEQ_outBuffer* out)
-{
-    switch (type) {
-        case SEQ_gen_ml: return SEQ_gen_matchLength(stream, value, out);
-        case SEQ_gen_ll: return SEQ_gen_litLength(stream, value, out);
-        case SEQ_gen_of: return SEQ_gen_offset(stream, value, out);
-        case SEQ_gen_max: /* fall-through */
-        default: return 0;
-    }
-}
-
-/* Returns the xxhash of the data produced so far */
-XXH64_hash_t SEQ_digest(SEQ_stream const* stream)
-{
-    return XXH64_digest(&stream->xxh);
-}
diff --git a/tests/seqgen.h b/tests/seqgen.h
deleted file mode 100644
index 72d798846aa8..000000000000
--- a/tests/seqgen.h
+++ /dev/null
@@ -1,58 +0,0 @@
-/*
- * Copyright (c) 2017-present, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#ifndef SEQGEN_H
-#define SEQGEN_H
-
-#define XXH_STATIC_LINKING_ONLY
-
-#include "xxhash.h"
-#include <stddef.h>   /* size_t */
-
-typedef enum {
-    SEQ_gen_ml = 0,
-    SEQ_gen_ll,
-    SEQ_gen_of,
-    SEQ_gen_max /* Must be the last value */
-} SEQ_gen_type;
-
-/* Internal state, do not use */
-typedef struct {
-    XXH64_state_t xxh; /* xxh state for all the data produced so far (seed=0) */
-    unsigned seed;
-    int state; /* enum to control state machine (clean=0) */
-    unsigned saved;
-    size_t bytesLeft;
-} SEQ_stream;
-
-SEQ_stream SEQ_initStream(unsigned seed);
-
-typedef struct {
-    void* dst;
-    size_t size;
-    size_t pos;
-} SEQ_outBuffer;
-
-/* Returns non-zero until the current type/value has been generated.
- * Must pass the same type/value until it returns 0.
- *
- * Recommended to pick a value in the middle of the range you want, since there
- * may be some noise that causes actual results to be slightly different.
- * We try to be more accurate for smaller values.
- *
- * NOTE: Very small values don't work well (< 6).
- */
-size_t SEQ_gen(SEQ_stream* stream, SEQ_gen_type type, unsigned value,
-               SEQ_outBuffer* out);
-
-/* Returns the xxhash of the data produced so far */
-XXH64_hash_t SEQ_digest(SEQ_stream const* stream);
-
-#endif /* SEQGEN_H */
diff --git a/tests/symbols.c b/tests/symbols.c
deleted file mode 100644
index 4d9c6fc0c9e2..000000000000
--- a/tests/symbols.c
+++ /dev/null
@@ -1,164 +0,0 @@
-/*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-
-#include <stdio.h>
-#include "zstd_errors.h"
-#define ZSTD_STATIC_LINKING_ONLY
-#include "zstd.h"
-#define ZBUFF_DISABLE_DEPRECATE_WARNINGS
-#define ZBUFF_STATIC_LINKING_ONLY
-#include "zbuff.h"
-#define ZDICT_DISABLE_DEPRECATE_WARNINGS
-#define ZDICT_STATIC_LINKING_ONLY
-#include "zdict.h"
-
-static const void *symbols[] = {
-/* zstd.h */
-  &ZSTD_versionNumber,
-  &ZSTD_compress,
-  &ZSTD_decompress,
-  &ZSTD_getDecompressedSize,
-  &ZSTD_findDecompressedSize,
-  &ZSTD_findFrameCompressedSize,
-  &ZSTD_getFrameContentSize,
-  &ZSTD_maxCLevel,
-  &ZSTD_compressBound,
-  &ZSTD_decompressBound,
-  &ZSTD_isError,
-  &ZSTD_getErrorName,
-  &ZSTD_createCCtx,
-  &ZSTD_freeCCtx,
-  &ZSTD_compressCCtx,
-  &ZSTD_createDCtx,
-  &ZSTD_freeDCtx,
-  &ZSTD_decompressDCtx,
-  &ZSTD_compress_usingDict,
-  &ZSTD_decompress_usingDict,
-  &ZSTD_createCDict,
-  &ZSTD_freeCDict,
-  &ZSTD_compress_usingCDict,
-  &ZSTD_createDDict,
-  &ZSTD_freeDDict,
-  &ZSTD_decompress_usingDDict,
-  &ZSTD_createCStream,
-  &ZSTD_freeCStream,
-  &ZSTD_initCStream,
-  &ZSTD_compressStream,
-  &ZSTD_flushStream,
-  &ZSTD_endStream,
-  &ZSTD_CStreamInSize,
-  &ZSTD_CStreamOutSize,
-  &ZSTD_createDStream,
-  &ZSTD_freeDStream,
-  &ZSTD_initDStream,
-  &ZSTD_decompressStream,
-  &ZSTD_DStreamInSize,
-  &ZSTD_DStreamOutSize,
-/* zstd.h: advanced functions */
-  &ZSTD_estimateCCtxSize,
-  &ZSTD_createCCtx_advanced,
-  &ZSTD_sizeof_CCtx,
-  &ZSTD_createCDict_advanced,
-  &ZSTD_sizeof_CDict,
-  &ZSTD_getCParams,
-  &ZSTD_getParams,
-  &ZSTD_checkCParams,
-  &ZSTD_adjustCParams,
-  &ZSTD_compress_advanced,
-  &ZSTD_isFrame,
-  &ZSTD_estimateDCtxSize,
-  &ZSTD_createDCtx_advanced,
-  &ZSTD_sizeof_DCtx,
-  &ZSTD_sizeof_DDict,
-  &ZSTD_getDictID_fromDict,
-  &ZSTD_getDictID_fromDDict,
-  &ZSTD_getDictID_fromFrame,
-  &ZSTD_createCStream_advanced,
-  &ZSTD_initCStream_srcSize,
-  &ZSTD_initCStream_usingDict,
-  &ZSTD_initCStream_advanced,
-  &ZSTD_initCStream_usingCDict,
-  &ZSTD_resetCStream,
-  &ZSTD_sizeof_CStream,
-  &ZSTD_createDStream_advanced,
-  &ZSTD_initDStream_usingDict,
-  &ZSTD_initDStream_usingDDict,
-  &ZSTD_resetDStream,
-  &ZSTD_sizeof_DStream,
-  &ZSTD_compressBegin,
-  &ZSTD_compressBegin_usingDict,
-  &ZSTD_compressBegin_advanced,
-  &ZSTD_copyCCtx,
-  &ZSTD_compressContinue,
-  &ZSTD_compressEnd,
-  &ZSTD_getFrameHeader,
-  &ZSTD_decompressBegin,
-  &ZSTD_decompressBegin_usingDict,
-  &ZSTD_copyDCtx,
-  &ZSTD_nextSrcSizeToDecompress,
-  &ZSTD_decompressContinue,
-  &ZSTD_nextInputType,
-  &ZSTD_getBlockSize,
-  &ZSTD_compressBlock,
-  &ZSTD_decompressBlock,
-  &ZSTD_insertBlock,
-/* zstd_errors.h */
-  &ZSTD_getErrorCode,
-  &ZSTD_getErrorString,
-/* zbuff.h */
-  &ZBUFF_createCCtx,
-  &ZBUFF_freeCCtx,
-  &ZBUFF_compressInit,
-  &ZBUFF_compressInitDictionary,
-  &ZBUFF_compressContinue,
-  &ZBUFF_compressFlush,
-  &ZBUFF_compressEnd,
-  &ZBUFF_createDCtx,
-  &ZBUFF_freeDCtx,
-  &ZBUFF_decompressInit,
-  &ZBUFF_decompressInitDictionary,
-  &ZBUFF_decompressContinue,
-  &ZBUFF_isError,
-  &ZBUFF_getErrorName,
-  &ZBUFF_recommendedCInSize,
-  &ZBUFF_recommendedCOutSize,
-  &ZBUFF_recommendedDInSize,
-  &ZBUFF_recommendedDOutSize,
-/* zbuff.h: advanced functions */
-  &ZBUFF_createCCtx_advanced,
-  &ZBUFF_createDCtx_advanced,
-  &ZBUFF_compressInit_advanced,
-/* zdict.h */
-  &ZDICT_trainFromBuffer,
-  &ZDICT_getDictID,
-  &ZDICT_isError,
-  &ZDICT_getErrorName,
-/* zdict.h: advanced functions */
-  &ZDICT_trainFromBuffer_cover,
-  &ZDICT_optimizeTrainFromBuffer_cover,
-  &ZDICT_trainFromBuffer_fastCover,
-  &ZDICT_optimizeTrainFromBuffer_fastCover,
-  &ZDICT_finalizeDictionary,
-  &ZDICT_trainFromBuffer_legacy,
-  &ZDICT_addEntropyTablesFromBuffer,
-  NULL,
-};
-
-int main(int argc, const char** argv) {
-  const void **symbol;
-  (void)argc;
-  (void)argv;
-
-  for (symbol = symbols; *symbol != NULL; ++symbol) {
-    printf("%p\n", *symbol);
-  }
-  return 0;
-}
diff --git a/tests/test-zstd-speed.py b/tests/test-zstd-speed.py
deleted file mode 100755
index 1096d5e4e224..000000000000
--- a/tests/test-zstd-speed.py
+++ /dev/null
@@ -1,376 +0,0 @@
-#! /usr/bin/env python3
-
-# ################################################################
-# Copyright (c) 2016-present, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under both the BSD-style license (found in the
-# LICENSE file in the root directory of this source tree) and the GPLv2 (found
-# in the COPYING file in the root directory of this source tree).
-# ##########################################################################
-
-# Limitations:
-# - doesn't support filenames with spaces
-# - dir1/zstd and dir2/zstd will be merged in a single results file
-
-import argparse
-import os           # getloadavg
-import string
-import subprocess
-import time         # strftime
-import traceback
-import hashlib
-import platform     # system
-
-script_version = 'v1.1.2 (2017-03-26)'
-default_repo_url = 'https://github.com/facebook/zstd.git'
-working_dir_name = 'speedTest'
-working_path = os.getcwd() + '/' + working_dir_name     # /path/to/zstd/tests/speedTest
-clone_path = working_path + '/' + 'zstd'                # /path/to/zstd/tests/speedTest/zstd
-email_header = 'ZSTD_speedTest'
-pid = str(os.getpid())
-verbose = False
-clang_version = "unknown"
-gcc_version = "unknown"
-args = None
-
-
-def hashfile(hasher, fname, blocksize=65536):
-    with open(fname, "rb") as f:
-        for chunk in iter(lambda: f.read(blocksize), b""):
-            hasher.update(chunk)
-    return hasher.hexdigest()
-
-
-def log(text):
-    print(time.strftime("%Y/%m/%d %H:%M:%S") + ' - ' + text)
-
-
-def execute(command, print_command=True, print_output=False, print_error=True, param_shell=True):
-    if print_command:
-        log("> " + command)
-    popen = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=param_shell, cwd=execute.cwd)
-    stdout_lines, stderr_lines = popen.communicate(timeout=args.timeout)
-    stderr_lines = stderr_lines.decode("utf-8")
-    stdout_lines = stdout_lines.decode("utf-8")
-    if print_output:
-        if stdout_lines:
-            print(stdout_lines)
-        if stderr_lines:
-            print(stderr_lines)
-    if popen.returncode is not None and popen.returncode != 0:
-        if stderr_lines and not print_output and print_error:
-            print(stderr_lines)
-        raise RuntimeError(stdout_lines + stderr_lines)
-    return (stdout_lines + stderr_lines).splitlines()
-execute.cwd = None
-
-
-def does_command_exist(command):
-    try:
-        execute(command, verbose, False, False)
-    except Exception:
-        return False
-    return True
-
-
-def send_email(emails, topic, text, have_mutt, have_mail):
-    logFileName = working_path + '/' + 'tmpEmailContent'
-    with open(logFileName, "w") as myfile:
-        myfile.writelines(text)
-        myfile.close()
-        if have_mutt:
-            execute('mutt -s "' + topic + '" ' + emails + ' < ' + logFileName, verbose)
-        elif have_mail:
-            execute('mail -s "' + topic + '" ' + emails + ' < ' + logFileName, verbose)
-        else:
-            log("e-mail cannot be sent (mail or mutt not found)")
-
-
-def send_email_with_attachments(branch, commit, last_commit, args, text, results_files,
-                                logFileName, have_mutt, have_mail):
-    with open(logFileName, "w") as myfile:
-        myfile.writelines(text)
-        myfile.close()
-        email_topic = '[%s:%s] Warning for %s:%s last_commit=%s speed<%s ratio<%s' \
-                      % (email_header, pid, branch, commit, last_commit,
-                         args.lowerLimit, args.ratioLimit)
-        if have_mutt:
-            execute('mutt -s "' + email_topic + '" ' + args.emails + ' -a ' + results_files
-                    + ' < ' + logFileName)
-        elif have_mail:
-            execute('mail -s "' + email_topic + '" ' + args.emails + ' < ' + logFileName)
-        else:
-            log("e-mail cannot be sent (mail or mutt not found)")
-
-
-def git_get_branches():
-    execute('git fetch -p', verbose)
-    branches = execute('git branch -rl', verbose)
-    output = []
-    for line in branches:
-        if ("HEAD" not in line) and ("coverity_scan" not in line) and ("gh-pages" not in line):
-            output.append(line.strip())
-    return output
-
-
-def git_get_changes(branch, commit, last_commit):
-    fmt = '--format="%h: (%an) %s, %ar"'
-    if last_commit is None:
-        commits = execute('git log -n 10 %s %s' % (fmt, commit))
-    else:
-        commits = execute('git --no-pager log %s %s..%s' % (fmt, last_commit, commit))
-    return str('Changes in %s since %s:\n' % (branch, last_commit)) + '\n'.join(commits)
-
-
-def get_last_results(resultsFileName):
-    if not os.path.isfile(resultsFileName):
-        return None, None, None, None
-    commit = None
-    csize = []
-    cspeed = []
-    dspeed = []
-    with open(resultsFileName, 'r') as f:
-        for line in f:
-            words = line.split()
-            if len(words) <= 4:   # branch + commit + compilerVer + md5
-                commit = words[1]
-                csize = []
-                cspeed = []
-                dspeed = []
-            if (len(words) == 8) or (len(words) == 9):  # results: "filename" or "XX files"
-                csize.append(int(words[1]))
-                cspeed.append(float(words[3]))
-                dspeed.append(float(words[5]))
-    return commit, csize, cspeed, dspeed
-
-
-def benchmark_and_compare(branch, commit, last_commit, args, executableName, md5sum, compilerVersion, resultsFileName,
-                          testFilePath, fileName, last_csize, last_cspeed, last_dspeed):
-    sleepTime = 30
-    while os.getloadavg()[0] > args.maxLoadAvg:
-        log("WARNING: bench loadavg=%.2f is higher than %s, sleeping for %s seconds"
-            % (os.getloadavg()[0], args.maxLoadAvg, sleepTime))
-        time.sleep(sleepTime)
-    start_load = str(os.getloadavg())
-    osType = platform.system()
-    if osType == 'Linux':
-        cpuSelector = "taskset --cpu-list 0"
-    else:
-        cpuSelector = ""
-    if args.dictionary:
-        result = execute('%s programs/%s -rqi5b1e%s -D %s %s' % (cpuSelector, executableName, args.lastCLevel, args.dictionary, testFilePath), print_output=True)
-    else:
-        result = execute('%s programs/%s -rqi5b1e%s %s' % (cpuSelector, executableName, args.lastCLevel, testFilePath), print_output=True)
-    end_load = str(os.getloadavg())
-    linesExpected = args.lastCLevel + 1
-    if len(result) != linesExpected:
-        raise RuntimeError("ERROR: number of result lines=%d is different that expected %d\n%s" % (len(result), linesExpected, '\n'.join(result)))
-    with open(resultsFileName, "a") as myfile:
-        myfile.write('%s %s %s md5=%s\n' % (branch, commit, compilerVersion, md5sum))
-        myfile.write('\n'.join(result) + '\n')
-        myfile.close()
-        if (last_cspeed == None):
-            log("WARNING: No data for comparison for branch=%s file=%s " % (branch, fileName))
-            return ""
-        commit, csize, cspeed, dspeed = get_last_results(resultsFileName)
-        text = ""
-        for i in range(0, min(len(cspeed), len(last_cspeed))):
-            print("%s:%s -%d cSpeed=%6.2f cLast=%6.2f cDiff=%1.4f dSpeed=%6.2f dLast=%6.2f dDiff=%1.4f ratioDiff=%1.4f %s" % (branch, commit, i+1, cspeed[i], last_cspeed[i], cspeed[i]/last_cspeed[i], dspeed[i], last_dspeed[i], dspeed[i]/last_dspeed[i], float(last_csize[i])/csize[i], fileName))
-            if (cspeed[i]/last_cspeed[i] < args.lowerLimit):
-                text += "WARNING: %s -%d cSpeed=%.2f cLast=%.2f cDiff=%.4f %s\n" % (executableName, i+1, cspeed[i], last_cspeed[i], cspeed[i]/last_cspeed[i], fileName)
-            if (dspeed[i]/last_dspeed[i] < args.lowerLimit):
-                text += "WARNING: %s -%d dSpeed=%.2f dLast=%.2f dDiff=%.4f %s\n" % (executableName, i+1, dspeed[i], last_dspeed[i], dspeed[i]/last_dspeed[i], fileName)
-            if (float(last_csize[i])/csize[i] < args.ratioLimit):
-                text += "WARNING: %s -%d cSize=%d last_cSize=%d diff=%.4f %s\n" % (executableName, i+1, csize[i], last_csize[i], float(last_csize[i])/csize[i], fileName)
-        if text:
-            text = args.message + ("\nmaxLoadAvg=%s  load average at start=%s end=%s\n%s  last_commit=%s  md5=%s\n" % (args.maxLoadAvg, start_load, end_load, compilerVersion, last_commit, md5sum)) + text
-        return text
-
-
-def update_config_file(branch, commit):
-    last_commit = None
-    commitFileName = working_path + "/commit_" + branch.replace("/", "_") + ".txt"
-    if os.path.isfile(commitFileName):
-        with open(commitFileName, 'r') as infile:
-            last_commit = infile.read()
-    with open(commitFileName, 'w') as outfile:
-        outfile.write(commit)
-    return last_commit
-
-
-def double_check(branch, commit, args, executableName, md5sum, compilerVersion, resultsFileName, filePath, fileName):
-    last_commit, csize, cspeed, dspeed = get_last_results(resultsFileName)
-    if not args.dry_run:
-        text = benchmark_and_compare(branch, commit, last_commit, args, executableName, md5sum, compilerVersion, resultsFileName, filePath, fileName, csize, cspeed, dspeed)
-        if text:
-            log("WARNING: redoing tests for branch %s: commit %s" % (branch, commit))
-            text = benchmark_and_compare(branch, commit, last_commit, args, executableName, md5sum, compilerVersion, resultsFileName, filePath, fileName, csize, cspeed, dspeed)
-    return text
-
-
-def test_commit(branch, commit, last_commit, args, testFilePaths, have_mutt, have_mail):
-    local_branch = branch.split('/')[1]
-    version = local_branch.rpartition('-')[2] + '_' + commit
-    if not args.dry_run:
-        execute('make -C programs clean zstd CC=clang MOREFLAGS="-Werror -Wconversion -Wno-sign-conversion -DZSTD_GIT_COMMIT=%s" && ' % version +
-                'mv programs/zstd programs/zstd_clang && ' +
-                'make -C programs clean zstd zstd32 MOREFLAGS="-DZSTD_GIT_COMMIT=%s"' % version)
-    md5_zstd = hashfile(hashlib.md5(), clone_path + '/programs/zstd')
-    md5_zstd32 = hashfile(hashlib.md5(), clone_path + '/programs/zstd32')
-    md5_zstd_clang = hashfile(hashlib.md5(), clone_path + '/programs/zstd_clang')
-    print("md5(zstd)=%s\nmd5(zstd32)=%s\nmd5(zstd_clang)=%s" % (md5_zstd, md5_zstd32, md5_zstd_clang))
-    print("gcc_version=%s clang_version=%s" % (gcc_version, clang_version))
-
-    logFileName = working_path + "/log_" + branch.replace("/", "_") + ".txt"
-    text_to_send = []
-    results_files = ""
-    if args.dictionary:
-        dictName = args.dictionary.rpartition('/')[2]
-    else:
-        dictName = None
-
-    for filePath in testFilePaths:
-        fileName = filePath.rpartition('/')[2]
-        if dictName:
-            resultsFileName = working_path + "/" + dictName.replace(".", "_") + "_" + branch.replace("/", "_") + "_" + fileName.replace(".", "_") + ".txt"
-        else:
-            resultsFileName = working_path + "/results_" + branch.replace("/", "_") + "_" + fileName.replace(".", "_") + ".txt"
-        text = double_check(branch, commit, args, 'zstd', md5_zstd, 'gcc_version='+gcc_version, resultsFileName, filePath, fileName)
-        if text:
-            text_to_send.append(text)
-            results_files += resultsFileName + " "
-        resultsFileName = working_path + "/results32_" + branch.replace("/", "_") + "_" + fileName.replace(".", "_") + ".txt"
-        text = double_check(branch, commit, args, 'zstd32', md5_zstd32, 'gcc_version='+gcc_version, resultsFileName, filePath, fileName)
-        if text:
-            text_to_send.append(text)
-            results_files += resultsFileName + " "
-        resultsFileName = working_path + "/resultsClang_" + branch.replace("/", "_") + "_" + fileName.replace(".", "_") + ".txt"
-        text = double_check(branch, commit, args, 'zstd_clang', md5_zstd_clang, 'clang_version='+clang_version, resultsFileName, filePath, fileName)
-        if text:
-            text_to_send.append(text)
-            results_files += resultsFileName + " "
-    if text_to_send:
-        send_email_with_attachments(branch, commit, last_commit, args, text_to_send, results_files, logFileName, have_mutt, have_mail)
-
-
-if __name__ == '__main__':
-    parser = argparse.ArgumentParser()
-    parser.add_argument('testFileNames', help='file or directory names list for speed benchmark')
-    parser.add_argument('emails', help='list of e-mail addresses to send warnings')
-    parser.add_argument('--dictionary', '-D', help='path to the dictionary')
-    parser.add_argument('--message', '-m', help='attach an additional message to e-mail', default="")
-    parser.add_argument('--repoURL', help='changes default repository URL', default=default_repo_url)
-    parser.add_argument('--lowerLimit', '-l', type=float, help='send email if speed is lower than given limit', default=0.98)
-    parser.add_argument('--ratioLimit', '-r', type=float, help='send email if ratio is lower than given limit', default=0.999)
-    parser.add_argument('--maxLoadAvg', type=float, help='maximum load average to start testing', default=0.75)
-    parser.add_argument('--lastCLevel', type=int, help='last compression level for testing', default=5)
-    parser.add_argument('--sleepTime', '-s', type=int, help='frequency of repository checking in seconds', default=300)
-    parser.add_argument('--timeout', '-t', type=int, help='timeout for executing shell commands', default=1800)
-    parser.add_argument('--dry-run', dest='dry_run', action='store_true', help='not build', default=False)
-    parser.add_argument('--verbose', '-v', action='store_true', help='more verbose logs', default=False)
-    args = parser.parse_args()
-    verbose = args.verbose
-
-    # check if test files are accessible
-    testFileNames = args.testFileNames.split()
-    testFilePaths = []
-    for fileName in testFileNames:
-        fileName = os.path.expanduser(fileName)
-        if os.path.isfile(fileName) or os.path.isdir(fileName):
-            testFilePaths.append(os.path.abspath(fileName))
-        else:
-            log("ERROR: File/directory not found: " + fileName)
-            exit(1)
-
-    # check if dictionary is accessible
-    if args.dictionary:
-        args.dictionary = os.path.abspath(os.path.expanduser(args.dictionary))
-        if not os.path.isfile(args.dictionary):
-            log("ERROR: Dictionary not found: " + args.dictionary)
-            exit(1)
-
-    # check availability of e-mail senders
-    have_mutt = does_command_exist("mutt -h")
-    have_mail = does_command_exist("mail -V")
-    if not have_mutt and not have_mail:
-        log("ERROR: e-mail senders 'mail' or 'mutt' not found")
-        exit(1)
-
-    clang_version = execute("clang -v 2>&1 | grep ' version ' | sed -e 's:.*version \\([0-9.]*\\).*:\\1:' -e 's:\\.\\([0-9][0-9]\\):\\1:g'", verbose)[0];
-    gcc_version = execute("gcc -dumpversion", verbose)[0];
-
-    if verbose:
-        print("PARAMETERS:\nrepoURL=%s" % args.repoURL)
-        print("working_path=%s" % working_path)
-        print("clone_path=%s" % clone_path)
-        print("testFilePath(%s)=%s" % (len(testFilePaths), testFilePaths))
-        print("message=%s" % args.message)
-        print("emails=%s" % args.emails)
-        print("dictionary=%s" % args.dictionary)
-        print("maxLoadAvg=%s" % args.maxLoadAvg)
-        print("lowerLimit=%s" % args.lowerLimit)
-        print("ratioLimit=%s" % args.ratioLimit)
-        print("lastCLevel=%s" % args.lastCLevel)
-        print("sleepTime=%s" % args.sleepTime)
-        print("timeout=%s" % args.timeout)
-        print("dry_run=%s" % args.dry_run)
-        print("verbose=%s" % args.verbose)
-        print("have_mutt=%s have_mail=%s" % (have_mutt, have_mail))
-
-    # clone ZSTD repo if needed
-    if not os.path.isdir(working_path):
-        os.mkdir(working_path)
-    if not os.path.isdir(clone_path):
-        execute.cwd = working_path
-        execute('git clone ' + args.repoURL)
-    if not os.path.isdir(clone_path):
-        log("ERROR: ZSTD clone not found: " + clone_path)
-        exit(1)
-    execute.cwd = clone_path
-
-    # check if speedTest.pid already exists
-    pidfile = "./speedTest.pid"
-    if os.path.isfile(pidfile):
-        log("ERROR: %s already exists, exiting" % pidfile)
-        exit(1)
-
-    send_email(args.emails, '[%s:%s] test-zstd-speed.py %s has been started' % (email_header, pid, script_version), args.message, have_mutt, have_mail)
-    with open(pidfile, 'w') as the_file:
-        the_file.write(pid)
-
-    branch = ""
-    commit = ""
-    first_time = True
-    while True:
-        try:
-            if first_time:
-                first_time = False
-            else:
-                time.sleep(args.sleepTime)
-            loadavg = os.getloadavg()[0]
-            if (loadavg <= args.maxLoadAvg):
-                branches = git_get_branches()
-                for branch in branches:
-                    commit = execute('git show -s --format=%h ' + branch, verbose)[0]
-                    last_commit = update_config_file(branch, commit)
-                    if commit == last_commit:
-                        log("skipping branch %s: head %s already processed" % (branch, commit))
-                    else:
-                        log("build branch %s: head %s is different from prev %s" % (branch, commit, last_commit))
-                        execute('git checkout -- . && git checkout ' + branch)
-                        print(git_get_changes(branch, commit, last_commit))
-                        test_commit(branch, commit, last_commit, args, testFilePaths, have_mutt, have_mail)
-            else:
-                log("WARNING: main loadavg=%.2f is higher than %s" % (loadavg, args.maxLoadAvg))
-            if verbose:
-                log("sleep for %s seconds" % args.sleepTime)
-        except Exception as e:
-            stack = traceback.format_exc()
-            email_topic = '[%s:%s] ERROR in %s:%s' % (email_header, pid, branch, commit)
-            send_email(args.emails, email_topic, stack, have_mutt, have_mail)
-            print(stack)
-        except KeyboardInterrupt:
-            os.unlink(pidfile)
-            send_email(args.emails, '[%s:%s] test-zstd-speed.py %s has been stopped' % (email_header, pid, script_version), args.message, have_mutt, have_mail)
-            exit(0)
diff --git a/tests/test-zstd-versions.py b/tests/test-zstd-versions.py
deleted file mode 100755
index 8e88b869b0d2..000000000000
--- a/tests/test-zstd-versions.py
+++ /dev/null
@@ -1,276 +0,0 @@
-#!/usr/bin/env python3
-"""Test zstd interoperability between versions"""
-
-# ################################################################
-# Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under both the BSD-style license (found in the
-# LICENSE file in the root directory of this source tree) and the GPLv2 (found
-# in the COPYING file in the root directory of this source tree).
-# ################################################################
-
-import filecmp
-import glob
-import hashlib
-import os
-import shutil
-import sys
-import subprocess
-from subprocess import Popen, PIPE
-
-repo_url = 'https://github.com/facebook/zstd.git'
-tmp_dir_name = 'tests/versionsTest'
-make_cmd = 'make'
-git_cmd = 'git'
-test_dat_src = 'README.md'
-test_dat = 'test_dat'
-head = 'vdevel'
-dict_source = 'dict_source'
-dict_files = './zstd/programs/*.c ./zstd/lib/common/*.c ./zstd/lib/compress/*.c ./zstd/lib/decompress/*.c ./zstd/lib/dictBuilder/*.c ./zstd/lib/legacy/*.c '
-dict_files += './zstd/programs/*.h ./zstd/lib/common/*.h ./zstd/lib/compress/*.h ./zstd/lib/dictBuilder/*.h ./zstd/lib/legacy/*.h'
-
-
-def execute(command, print_output=False, print_error=True, param_shell=False):
-    popen = Popen(command, stdout=PIPE, stderr=PIPE, shell=param_shell)
-    stdout_lines, stderr_lines = popen.communicate()
-    stderr_lines = stderr_lines.decode("utf-8")
-    stdout_lines = stdout_lines.decode("utf-8")
-    if print_output:
-        print(stdout_lines)
-        print(stderr_lines)
-    if popen.returncode is not None and popen.returncode != 0:
-        if not print_output and print_error:
-            print(stderr_lines)
-    return popen.returncode
-
-
-def proc(cmd_args, pipe=True, dummy=False):
-    if dummy:
-        return
-    if pipe:
-        subproc = Popen(cmd_args, stdout=PIPE, stderr=PIPE)
-    else:
-        subproc = Popen(cmd_args)
-    return subproc.communicate()
-
-
-def make(args, pipe=True):
-    return proc([make_cmd] + args, pipe)
-
-
-def git(args, pipe=True):
-    return proc([git_cmd] + args, pipe)
-
-
-def get_git_tags():
-    stdout, stderr = git(['tag', '-l', 'v[0-9].[0-9].[0-9]'])
-    tags = stdout.decode('utf-8').split()
-    return tags
-
-
-def create_dict(tag, dict_source_path):
-    dict_name = 'dict.' + tag
-    if not os.path.isfile(dict_name):
-        cFiles = glob.glob(dict_source_path + "/*.c")
-        hFiles = glob.glob(dict_source_path + "/*.h")
-        if tag == 'v0.5.0':
-            result = execute('./dictBuilder.' + tag + ' ' + ' '.join(cFiles) + ' ' + ' '.join(hFiles) + ' -o ' + dict_name, print_output=False, param_shell=True)
-        else:
-            result = execute('./zstd.' + tag + ' -f --train ' + ' '.join(cFiles) + ' ' + ' '.join(hFiles) + ' -o ' + dict_name, print_output=False, param_shell=True)
-        if result == 0:
-            print(dict_name + ' created')
-        else:
-            print('ERROR: creating of ' + dict_name + ' failed')
-    else:
-        print(dict_name + ' already exists')
-
-
-def dict_compress_sample(tag, sample):
-    dict_name = 'dict.' + tag
-    DEVNULL = open(os.devnull, 'wb')
-    if subprocess.call(['./zstd.' + tag, '-D', dict_name, '-f',   sample], stderr=DEVNULL) == 0:
-        os.rename(sample + '.zst', sample + '_01_64_' + tag + '_dictio.zst')
-    if subprocess.call(['./zstd.' + tag, '-D', dict_name, '-5f',  sample], stderr=DEVNULL) == 0:
-        os.rename(sample + '.zst', sample + '_05_64_' + tag + '_dictio.zst')
-    if subprocess.call(['./zstd.' + tag, '-D', dict_name, '-9f',  sample], stderr=DEVNULL) == 0:
-        os.rename(sample + '.zst', sample + '_09_64_' + tag + '_dictio.zst')
-    if subprocess.call(['./zstd.' + tag, '-D', dict_name, '-15f', sample], stderr=DEVNULL) == 0:
-        os.rename(sample + '.zst', sample + '_15_64_' + tag + '_dictio.zst')
-    if subprocess.call(['./zstd.' + tag, '-D', dict_name, '-18f', sample], stderr=DEVNULL) == 0:
-        os.rename(sample + '.zst', sample + '_18_64_' + tag + '_dictio.zst')
-    # zstdFiles = glob.glob("*.zst*")
-    # print(zstdFiles)
-    print(tag + " : dict compression completed")
-
-
-def compress_sample(tag, sample):
-    DEVNULL = open(os.devnull, 'wb')
-    if subprocess.call(['./zstd.' + tag, '-f',   sample], stderr=DEVNULL) == 0:
-        os.rename(sample + '.zst', sample + '_01_64_' + tag + '_nodict.zst')
-    if subprocess.call(['./zstd.' + tag, '-5f',  sample], stderr=DEVNULL) == 0:
-        os.rename(sample + '.zst', sample + '_05_64_' + tag + '_nodict.zst')
-    if subprocess.call(['./zstd.' + tag, '-9f',  sample], stderr=DEVNULL) == 0:
-        os.rename(sample + '.zst', sample + '_09_64_' + tag + '_nodict.zst')
-    if subprocess.call(['./zstd.' + tag, '-15f', sample], stderr=DEVNULL) == 0:
-        os.rename(sample + '.zst', sample + '_15_64_' + tag + '_nodict.zst')
-    if subprocess.call(['./zstd.' + tag, '-18f', sample], stderr=DEVNULL) == 0:
-        os.rename(sample + '.zst', sample + '_18_64_' + tag + '_nodict.zst')
-    # zstdFiles = glob.glob("*.zst*")
-    # print(zstdFiles)
-    print(tag + " : compression completed")
-
-
-# http://stackoverflow.com/a/19711609/2132223
-def sha1_of_file(filepath):
-    with open(filepath, 'rb') as f:
-        return hashlib.sha1(f.read()).hexdigest()
-
-
-def remove_duplicates():
-    list_of_zst = sorted(glob.glob('*.zst'))
-    for i, ref_zst in enumerate(list_of_zst):
-        if not os.path.isfile(ref_zst):
-            continue
-        for j in range(i + 1, len(list_of_zst)):
-            compared_zst = list_of_zst[j]
-            if not os.path.isfile(compared_zst):
-                continue
-            if filecmp.cmp(ref_zst, compared_zst):
-                os.remove(compared_zst)
-                print('duplicated : {} == {}'.format(ref_zst, compared_zst))
-
-
-def decompress_zst(tag):
-    dec_error = 0
-    list_zst = sorted(glob.glob('*_nodict.zst'))
-    for file_zst in list_zst:
-        print(file_zst, end=' ')
-        print(tag, end=' ')
-        file_dec = file_zst + '_d64_' + tag + '.dec'
-        if tag <= 'v0.5.0':
-            params = ['./zstd.' + tag, '-df', file_zst, file_dec]
-        else:
-            params = ['./zstd.' + tag, '-df', file_zst, '-o', file_dec]
-        if execute(params) == 0:
-            if not filecmp.cmp(file_dec, test_dat):
-                print('ERR !! ')
-                dec_error = 1
-            else:
-                print('OK     ')
-        else:
-            print('command does not work')
-            dec_error = 1
-    return dec_error
-
-
-def decompress_dict(tag):
-    dec_error = 0
-    list_zst = sorted(glob.glob('*_dictio.zst'))
-    for file_zst in list_zst:
-        dict_tag = file_zst[0:len(file_zst)-11]  # remove "_dictio.zst"
-        if head in dict_tag: # find vdevel
-            dict_tag = head
-        else:
-            dict_tag = dict_tag[dict_tag.rfind('v'):]
-        if tag == 'v0.6.0' and dict_tag < 'v0.6.0':
-            continue
-        dict_name = 'dict.' + dict_tag
-        print(file_zst + ' ' + tag + ' dict=' + dict_tag, end=' ')
-        file_dec = file_zst + '_d64_' + tag + '.dec'
-        if tag <= 'v0.5.0':
-            params = ['./zstd.' + tag, '-D', dict_name, '-df', file_zst, file_dec]
-        else:
-            params = ['./zstd.' + tag, '-D', dict_name, '-df', file_zst, '-o', file_dec]
-        if execute(params) == 0:
-            if not filecmp.cmp(file_dec, test_dat):
-                print('ERR !! ')
-                dec_error = 1
-            else:
-                print('OK     ')
-        else:
-            print('command does not work')
-            dec_error = 1
-    return dec_error
-
-
-if __name__ == '__main__':
-    error_code = 0
-    base_dir = os.getcwd() + '/..'                  # /path/to/zstd
-    tmp_dir = base_dir + '/' + tmp_dir_name         # /path/to/zstd/tests/versionsTest
-    clone_dir = tmp_dir + '/' + 'zstd'              # /path/to/zstd/tests/versionsTest/zstd
-    dict_source_path = tmp_dir + '/' + dict_source  # /path/to/zstd/tests/versionsTest/dict_source
-    programs_dir = base_dir + '/programs'           # /path/to/zstd/programs
-    os.makedirs(tmp_dir, exist_ok=True)
-
-    # since Travis clones limited depth, we should clone full repository
-    if not os.path.isdir(clone_dir):
-        git(['clone', repo_url, clone_dir])
-
-    shutil.copy2(base_dir + '/' + test_dat_src, tmp_dir + '/' + test_dat)
-
-    # Retrieve all release tags
-    print('Retrieve all release tags :')
-    os.chdir(clone_dir)
-    alltags = get_git_tags() + [head]
-    tags = [t for t in alltags if t >= 'v0.5.0']
-    print(tags)
-
-    # Build all release zstd
-    for tag in tags:
-        os.chdir(base_dir)
-        dst_zstd = '{}/zstd.{}'.format(tmp_dir, tag)  # /path/to/zstd/tests/versionsTest/zstd.<TAG>
-        if not os.path.isfile(dst_zstd) or tag == head:
-            if tag != head:
-                r_dir = '{}/{}'.format(tmp_dir, tag)  # /path/to/zstd/tests/versionsTest/<TAG>
-                os.makedirs(r_dir, exist_ok=True)
-                os.chdir(clone_dir)
-                git(['--work-tree=' + r_dir, 'checkout', tag, '--', '.'], False)
-                if tag == 'v0.5.0':
-                    os.chdir(r_dir + '/dictBuilder')  # /path/to/zstd/tests/versionsTest/v0.5.0/dictBuilder
-                    make(['clean', 'dictBuilder'], False)
-                    shutil.copy2('dictBuilder', '{}/dictBuilder.{}'.format(tmp_dir, tag))
-                os.chdir(r_dir + '/programs')  # /path/to/zstd/tests/versionsTest/<TAG>/programs
-                make(['clean', 'zstd'], False)
-            else:
-                os.chdir(programs_dir)
-                make(['zstd'], False)
-            shutil.copy2('zstd',   dst_zstd)
-
-    # remove any remaining *.zst and *.dec from previous test
-    os.chdir(tmp_dir)
-    for compressed in glob.glob("*.zst"):
-        os.remove(compressed)
-    for dec in glob.glob("*.dec"):
-        os.remove(dec)
-
-    # copy *.c and *.h to a temporary directory ("dict_source")
-    if not os.path.isdir(dict_source_path):
-        os.mkdir(dict_source_path)
-        print('cp ' + dict_files + ' ' + dict_source_path)
-        execute('cp ' + dict_files + ' ' + dict_source_path, param_shell=True)
-
-    print('Compress test.dat by all released zstd')
-
-    error_code = 0
-    for tag in tags:
-        print(tag)
-        if tag >= 'v0.5.0':
-            create_dict(tag, dict_source_path)
-            dict_compress_sample(tag, test_dat)
-            remove_duplicates()
-            error_code += decompress_dict(tag)
-        compress_sample(tag, test_dat)
-        remove_duplicates()
-        error_code += decompress_zst(tag)
-
-    print('')
-    print('Enumerate different compressed files')
-    zstds = sorted(glob.glob('*.zst'))
-    for zstd in zstds:
-        print(zstd + ' : ' + repr(os.path.getsize(zstd)) + ', ' + sha1_of_file(zstd))
-
-    if error_code != 0:
-        print('======  ERROR !!!  =======')
-
-    sys.exit(error_code)
diff --git a/tests/zbufftest.c b/tests/zbufftest.c
deleted file mode 100644
index 8a4a27907b0e..000000000000
--- a/tests/zbufftest.c
+++ /dev/null
@@ -1,619 +0,0 @@
-/*
- * Copyright (c) 2015-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-
-/*-************************************
-*  Compiler specific
-**************************************/
-#ifdef _MSC_VER    /* Visual Studio */
-#  define _CRT_SECURE_NO_WARNINGS     /* fgets */
-#  pragma warning(disable : 4127)     /* disable: C4127: conditional expression is constant */
-#  pragma warning(disable : 4146)     /* disable: C4146: minus unsigned expression */
-#endif
-
-
-/*-************************************
-*  Includes
-**************************************/
-#include <stdlib.h>       /* free */
-#include <stdio.h>        /* fgets, sscanf */
-#include <string.h>       /* strcmp */
-#include "timefn.h"       /* UTIL_time_t */
-#include "mem.h"
-#define ZSTD_STATIC_LINKING_ONLY   /* ZSTD_maxCLevel */
-#include "zstd.h"         /* ZSTD_compressBound */
-#define ZBUFF_STATIC_LINKING_ONLY  /* ZBUFF_createCCtx_advanced */
-#include "zbuff.h"        /* ZBUFF_isError */
-#include "datagen.h"      /* RDG_genBuffer */
-#define XXH_STATIC_LINKING_ONLY
-#include "xxhash.h"       /* XXH64_* */
-#include "util.h"
-
-
-/*-************************************
-*  Constants
-**************************************/
-#define KB *(1U<<10)
-#define MB *(1U<<20)
-#define GB *(1U<<30)
-
-static const U32 nbTestsDefault = 10000;
-#define COMPRESSIBLE_NOISE_LENGTH (10 MB)
-#define FUZ_COMPRESSIBILITY_DEFAULT 50
-static const U32 prime1 = 2654435761U;
-static const U32 prime2 = 2246822519U;
-
-
-
-/*-************************************
-*  Display Macros
-**************************************/
-#define DISPLAY(...)          fprintf(stderr, __VA_ARGS__)
-#define DISPLAYLEVEL(l, ...)  if (g_displayLevel>=l) { DISPLAY(__VA_ARGS__); }
-static U32 g_displayLevel = 2;
-
-static const U64 g_refreshRate = SEC_TO_MICRO / 6;
-static UTIL_time_t g_displayClock = UTIL_TIME_INITIALIZER;
-
-#define DISPLAYUPDATE(l, ...) if (g_displayLevel>=l) { \
-            if ((UTIL_clockSpanMicro(g_displayClock) > g_refreshRate) || (g_displayLevel>=4)) \
-            { g_displayClock = UTIL_getTime(); DISPLAY(__VA_ARGS__); \
-            if (g_displayLevel>=4) fflush(stderr); } }
-
-static U64 g_clockTime = 0;
-
-
-/*-*******************************************************
-*  Fuzzer functions
-*********************************************************/
-#undef MIN
-#undef MAX
-#define MIN(a,b) ((a)<(b)?(a):(b))
-#define MAX(a,b) ((a)>(b)?(a):(b))
-/*! FUZ_rand() :
-    @return : a 27 bits random value, from a 32-bits `seed`.
-    `seed` is also modified */
-#  define FUZ_rotl32(x,r) ((x << r) | (x >> (32 - r)))
-static unsigned int FUZ_rand(unsigned int* seedPtr)
-{
-    U32 rand32 = *seedPtr;
-    rand32 *= prime1;
-    rand32 += prime2;
-    rand32  = FUZ_rotl32(rand32, 13);
-    *seedPtr = rand32;
-    return rand32 >> 5;
-}
-
-
-/*
-static unsigned FUZ_highbit32(U32 v32)
-{
-    unsigned nbBits = 0;
-    if (v32==0) return 0;
-    for ( ; v32 ; v32>>=1) nbBits++;
-    return nbBits;
-}
-*/
-
-static void* ZBUFF_allocFunction(void* opaque, size_t size)
-{
-    void* address = malloc(size);
-    (void)opaque;
-    /* DISPLAYLEVEL(4, "alloc %p, %d opaque=%p \n", address, (int)size, opaque); */
-    return address;
-}
-
-static void ZBUFF_freeFunction(void* opaque, void* address)
-{
-    (void)opaque;
-    /* if (address) DISPLAYLEVEL(4, "free %p opaque=%p \n", address, opaque); */
-    free(address);
-}
-
-static int basicUnitTests(U32 seed, double compressibility, ZSTD_customMem customMem)
-{
-    int testResult = 0;
-    size_t CNBufferSize = COMPRESSIBLE_NOISE_LENGTH;
-    void* CNBuffer = malloc(CNBufferSize);
-    size_t const skippableFrameSize = 11;
-    size_t const compressedBufferSize = (8 + skippableFrameSize) + ZSTD_compressBound(COMPRESSIBLE_NOISE_LENGTH);
-    void* compressedBuffer = malloc(compressedBufferSize);
-    size_t const decodedBufferSize = CNBufferSize;
-    void* decodedBuffer = malloc(decodedBufferSize);
-    size_t cSize, readSize, readSkipSize, genSize;
-    U32 testNb=0;
-    ZBUFF_CCtx* zc = ZBUFF_createCCtx_advanced(customMem);
-    ZBUFF_DCtx* zd = ZBUFF_createDCtx_advanced(customMem);
-
-    /* Create compressible test buffer */
-    if (!CNBuffer || !compressedBuffer || !decodedBuffer || !zc || !zd) {
-        DISPLAY("Not enough memory, aborting\n");
-        goto _output_error;
-    }
-    RDG_genBuffer(CNBuffer, CNBufferSize, compressibility, 0., seed);
-
-    /* generate skippable frame */
-    MEM_writeLE32(compressedBuffer, ZSTD_MAGIC_SKIPPABLE_START);
-    MEM_writeLE32(((char*)compressedBuffer)+4, (U32)skippableFrameSize);
-    cSize = skippableFrameSize + 8;
-
-    /* Basic compression test */
-    DISPLAYLEVEL(4, "test%3i : compress %u bytes : ", testNb++, COMPRESSIBLE_NOISE_LENGTH);
-    ZBUFF_compressInitDictionary(zc, CNBuffer, 128 KB, 1);
-    readSize = CNBufferSize;
-    genSize = compressedBufferSize;
-    { size_t const r = ZBUFF_compressContinue(zc, ((char*)compressedBuffer)+cSize, &genSize, CNBuffer, &readSize);
-      if (ZBUFF_isError(r)) goto _output_error; }
-    if (readSize != CNBufferSize) goto _output_error;   /* entire input should be consumed */
-    cSize += genSize;
-    genSize = compressedBufferSize - cSize;
-    { size_t const r = ZBUFF_compressEnd(zc, ((char*)compressedBuffer)+cSize, &genSize);
-      if (r != 0) goto _output_error; }  /* error, or some data not flushed */
-    cSize += genSize;
-    DISPLAYLEVEL(4, "OK (%u bytes : %.2f%%)\n", (U32)cSize, (double)cSize/COMPRESSIBLE_NOISE_LENGTH*100);
-
-    /* skippable frame test */
-    DISPLAYLEVEL(4, "test%3i : decompress skippable frame : ", testNb++);
-    ZBUFF_decompressInitDictionary(zd, CNBuffer, 128 KB);
-    readSkipSize = cSize;
-    genSize = CNBufferSize;
-    { size_t const r = ZBUFF_decompressContinue(zd, decodedBuffer, &genSize, compressedBuffer, &readSkipSize);
-      if (r != 0) goto _output_error; }
-    if (genSize != 0) goto _output_error;   /* skippable frame len is 0 */
-    DISPLAYLEVEL(4, "OK \n");
-
-    /* Basic decompression test */
-    DISPLAYLEVEL(4, "test%3i : decompress %u bytes : ", testNb++, COMPRESSIBLE_NOISE_LENGTH);
-    ZBUFF_decompressInitDictionary(zd, CNBuffer, 128 KB);
-    readSize = cSize - readSkipSize;
-    genSize = CNBufferSize;
-    { size_t const r = ZBUFF_decompressContinue(zd, decodedBuffer, &genSize, ((char*)compressedBuffer)+readSkipSize, &readSize);
-      if (r != 0) goto _output_error; }  /* should reach end of frame == 0; otherwise, some data left, or an error */
-    if (genSize != CNBufferSize) goto _output_error;   /* should regenerate the same amount */
-    if (readSize+readSkipSize != cSize) goto _output_error;   /* should have read the entire frame */
-    DISPLAYLEVEL(4, "OK \n");
-
-    /* check regenerated data is byte exact */
-    DISPLAYLEVEL(4, "test%3i : check decompressed result : ", testNb++);
-    {   size_t i;
-        for (i=0; i<CNBufferSize; i++) {
-            if (((BYTE*)decodedBuffer)[i] != ((BYTE*)CNBuffer)[i]) goto _output_error;
-    }   }
-    DISPLAYLEVEL(4, "OK \n");
-
-    /* Byte-by-byte decompression test */
-    DISPLAYLEVEL(4, "test%3i : decompress byte-by-byte : ", testNb++);
-    {   size_t r, pIn=0, pOut=0;
-        do
-        {   ZBUFF_decompressInitDictionary(zd, CNBuffer, 128 KB);
-            r = 1;
-            while (r) {
-                size_t inS = 1;
-                size_t outS = 1;
-                r = ZBUFF_decompressContinue(zd, ((BYTE*)decodedBuffer)+pOut, &outS, ((BYTE*)compressedBuffer)+pIn, &inS);
-                pIn += inS;
-                pOut += outS;
-            }
-            readSize = pIn;
-            genSize = pOut;
-        } while (genSize==0);
-    }
-    if (genSize != CNBufferSize) goto _output_error;   /* should regenerate the same amount */
-    if (readSize != cSize) goto _output_error;   /* should have read the entire frame */
-    DISPLAYLEVEL(4, "OK \n");
-
-    /* check regenerated data is byte exact */
-    DISPLAYLEVEL(4, "test%3i : check decompressed result : ", testNb++);
-    {   size_t i;
-        for (i=0; i<CNBufferSize; i++) {
-            if (((BYTE*)decodedBuffer)[i] != ((BYTE*)CNBuffer)[i]) goto _output_error;
-    }   }
-    DISPLAYLEVEL(4, "OK \n");
-
-_end:
-    ZBUFF_freeCCtx(zc);
-    ZBUFF_freeDCtx(zd);
-    free(CNBuffer);
-    free(compressedBuffer);
-    free(decodedBuffer);
-    return testResult;
-
-_output_error:
-    testResult = 1;
-    DISPLAY("Error detected in Unit tests ! \n");
-    goto _end;
-}
-
-
-static size_t findDiff(const void* buf1, const void* buf2, size_t max)
-{
-    const BYTE* b1 = (const BYTE*)buf1;
-    const BYTE* b2 = (const BYTE*)buf2;
-    size_t u;
-    for (u=0; u<max; u++) {
-        if (b1[u] != b2[u]) break;
-    }
-    return u;
-}
-
-static size_t FUZ_rLogLength(U32* seed, U32 logLength)
-{
-    size_t const lengthMask = ((size_t)1 << logLength) - 1;
-    return (lengthMask+1) + (FUZ_rand(seed) & lengthMask);
-}
-
-static size_t FUZ_randomLength(U32* seed, U32 maxLog)
-{
-    U32 const logLength = FUZ_rand(seed) % maxLog;
-    return FUZ_rLogLength(seed, logLength);
-}
-
-#define CHECK(cond, ...) if (cond) { DISPLAY("Error => "); DISPLAY(__VA_ARGS__); \
-                         DISPLAY(" (seed %u, test nb %u)  \n", seed, testNb); goto _output_error; }
-
-static int fuzzerTests(U32 seed, U32 nbTests, unsigned startTest, double compressibility)
-{
-    static const U32 maxSrcLog = 24;
-    static const U32 maxSampleLog = 19;
-    BYTE* cNoiseBuffer[5];
-    size_t const srcBufferSize = (size_t)1<<maxSrcLog;
-    BYTE* copyBuffer;
-    size_t const copyBufferSize= srcBufferSize + (1<<maxSampleLog);
-    BYTE* cBuffer;
-    size_t const cBufferSize   = ZSTD_compressBound(srcBufferSize);
-    BYTE* dstBuffer;
-    size_t dstBufferSize = srcBufferSize;
-    U32 result = 0;
-    U32 testNb = 0;
-    U32 coreSeed = seed;
-    ZBUFF_CCtx* zc;
-    ZBUFF_DCtx* zd;
-    UTIL_time_t startClock = UTIL_getTime();
-
-    /* allocations */
-    zc = ZBUFF_createCCtx();
-    zd = ZBUFF_createDCtx();
-    cNoiseBuffer[0] = (BYTE*)malloc (srcBufferSize);
-    cNoiseBuffer[1] = (BYTE*)malloc (srcBufferSize);
-    cNoiseBuffer[2] = (BYTE*)malloc (srcBufferSize);
-    cNoiseBuffer[3] = (BYTE*)malloc (srcBufferSize);
-    cNoiseBuffer[4] = (BYTE*)malloc (srcBufferSize);
-    copyBuffer= (BYTE*)malloc (copyBufferSize);
-    dstBuffer = (BYTE*)malloc (dstBufferSize);
-    cBuffer   = (BYTE*)malloc (cBufferSize);
-    CHECK (!cNoiseBuffer[0] || !cNoiseBuffer[1] || !cNoiseBuffer[2] || !cNoiseBuffer[3] || !cNoiseBuffer[4] ||
-           !copyBuffer || !dstBuffer || !cBuffer || !zc || !zd,
-           "Not enough memory, fuzzer tests cancelled");
-
-    /* Create initial samples */
-    RDG_genBuffer(cNoiseBuffer[0], srcBufferSize, 0.00, 0., coreSeed);    /* pure noise */
-    RDG_genBuffer(cNoiseBuffer[1], srcBufferSize, 0.05, 0., coreSeed);    /* barely compressible */
-    RDG_genBuffer(cNoiseBuffer[2], srcBufferSize, compressibility, 0., coreSeed);
-    RDG_genBuffer(cNoiseBuffer[3], srcBufferSize, 0.95, 0., coreSeed);    /* highly compressible */
-    RDG_genBuffer(cNoiseBuffer[4], srcBufferSize, 1.00, 0., coreSeed);    /* sparse content */
-    memset(copyBuffer, 0x65, copyBufferSize);                             /* make copyBuffer considered initialized */
-
-    /* catch up testNb */
-    for (testNb=1; testNb < startTest; testNb++)
-        FUZ_rand(&coreSeed);
-
-    /* test loop */
-    for ( ; (testNb <= nbTests) || (UTIL_clockSpanMicro(startClock) < g_clockTime) ; testNb++ ) {
-        U32 lseed;
-        const BYTE* srcBuffer;
-        const BYTE* dict;
-        size_t maxTestSize, dictSize;
-        size_t cSize, totalTestSize, totalCSize, totalGenSize;
-        size_t errorCode;
-        U32 n, nbChunks;
-        XXH64_state_t xxhState;
-        U64 crcOrig;
-
-        /* init */
-        DISPLAYUPDATE(2, "\r%6u", testNb);
-        if (nbTests >= testNb) DISPLAYUPDATE(2, "/%6u   ", nbTests);
-        FUZ_rand(&coreSeed);
-        lseed = coreSeed ^ prime1;
-
-        /* states full reset (unsynchronized) */
-        /* some issues only happen when reusing states in a specific sequence of parameters */
-        if ((FUZ_rand(&lseed) & 0xFF) == 131) { ZBUFF_freeCCtx(zc); zc = ZBUFF_createCCtx(); }
-        if ((FUZ_rand(&lseed) & 0xFF) == 132) { ZBUFF_freeDCtx(zd); zd = ZBUFF_createDCtx(); }
-
-        /* srcBuffer selection [0-4] */
-        {   U32 buffNb = FUZ_rand(&lseed) & 0x7F;
-            if (buffNb & 7) buffNb=2;   /* most common : compressible (P) */
-            else {
-                buffNb >>= 3;
-                if (buffNb & 7) {
-                    const U32 tnb[2] = { 1, 3 };   /* barely/highly compressible */
-                    buffNb = tnb[buffNb >> 3];
-                } else {
-                    const U32 tnb[2] = { 0, 4 };   /* not compressible / sparse */
-                    buffNb = tnb[buffNb >> 3];
-            }   }
-            srcBuffer = cNoiseBuffer[buffNb];
-        }
-
-        /* compression init */
-        {   U32 const testLog = FUZ_rand(&lseed) % maxSrcLog;
-            U32 const cLevel = (FUZ_rand(&lseed) % (ZSTD_maxCLevel() - (testLog/3))) + 1;
-            maxTestSize = FUZ_rLogLength(&lseed, testLog);
-            dictSize  = (FUZ_rand(&lseed)==1) ? FUZ_randomLength(&lseed, maxSampleLog) : 0;
-            /* random dictionary selection */
-            {   size_t const dictStart = FUZ_rand(&lseed) % (srcBufferSize - dictSize);
-                dict = srcBuffer + dictStart;
-            }
-            {   ZSTD_parameters params = ZSTD_getParams(cLevel, 0, dictSize);
-                params.fParams.checksumFlag = FUZ_rand(&lseed) & 1;
-                params.fParams.noDictIDFlag = FUZ_rand(&lseed) & 1;
-                {   size_t const initError = ZBUFF_compressInit_advanced(zc, dict, dictSize, params, ZSTD_CONTENTSIZE_UNKNOWN);
-                    CHECK (ZBUFF_isError(initError),"init error : %s", ZBUFF_getErrorName(initError));
-        }   }   }
-
-        /* multi-segments compression test */
-        XXH64_reset(&xxhState, 0);
-        nbChunks    = (FUZ_rand(&lseed) & 127) + 2;
-        for (n=0, cSize=0, totalTestSize=0 ; (n<nbChunks) && (totalTestSize < maxTestSize) ; n++) {
-            /* compress random chunk into random size dst buffer */
-            {   size_t readChunkSize = FUZ_randomLength(&lseed, maxSampleLog);
-                size_t const randomDstSize = FUZ_randomLength(&lseed, maxSampleLog);
-                size_t dstBuffSize = MIN(cBufferSize - cSize, randomDstSize);
-                size_t const srcStart = FUZ_rand(&lseed) % (srcBufferSize - readChunkSize);
-
-                size_t const compressionError = ZBUFF_compressContinue(zc, cBuffer+cSize, &dstBuffSize, srcBuffer+srcStart, &readChunkSize);
-                CHECK (ZBUFF_isError(compressionError), "compression error : %s", ZBUFF_getErrorName(compressionError));
-
-                XXH64_update(&xxhState, srcBuffer+srcStart, readChunkSize);
-                memcpy(copyBuffer+totalTestSize, srcBuffer+srcStart, readChunkSize);
-                cSize += dstBuffSize;
-                totalTestSize += readChunkSize;
-            }
-
-            /* random flush operation, to mess around */
-            if ((FUZ_rand(&lseed) & 15) == 0) {
-                size_t const randomDstSize = FUZ_randomLength(&lseed, maxSampleLog);
-                size_t dstBuffSize = MIN(cBufferSize - cSize, randomDstSize);
-                size_t const flushError = ZBUFF_compressFlush(zc, cBuffer+cSize, &dstBuffSize);
-                CHECK (ZBUFF_isError(flushError), "flush error : %s", ZBUFF_getErrorName(flushError));
-                cSize += dstBuffSize;
-        }   }
-
-        /* final frame epilogue */
-        {   size_t remainingToFlush = (size_t)(-1);
-            while (remainingToFlush) {
-                size_t const randomDstSize = FUZ_randomLength(&lseed, maxSampleLog);
-                size_t dstBuffSize = MIN(cBufferSize - cSize, randomDstSize);
-                U32 const enoughDstSize = dstBuffSize >= remainingToFlush;
-                remainingToFlush = ZBUFF_compressEnd(zc, cBuffer+cSize, &dstBuffSize);
-                CHECK (ZBUFF_isError(remainingToFlush), "flush error : %s", ZBUFF_getErrorName(remainingToFlush));
-                CHECK (enoughDstSize && remainingToFlush, "ZBUFF_compressEnd() not fully flushed (%u remaining), but enough space available", (U32)remainingToFlush);
-                cSize += dstBuffSize;
-        }   }
-        crcOrig = XXH64_digest(&xxhState);
-
-        /* multi - fragments decompression test */
-        ZBUFF_decompressInitDictionary(zd, dict, dictSize);
-        errorCode = 1;
-        for (totalCSize = 0, totalGenSize = 0 ; errorCode ; ) {
-            size_t readCSrcSize = FUZ_randomLength(&lseed, maxSampleLog);
-            size_t const randomDstSize = FUZ_randomLength(&lseed, maxSampleLog);
-            size_t dstBuffSize = MIN(dstBufferSize - totalGenSize, randomDstSize);
-            errorCode = ZBUFF_decompressContinue(zd, dstBuffer+totalGenSize, &dstBuffSize, cBuffer+totalCSize, &readCSrcSize);
-            CHECK (ZBUFF_isError(errorCode), "decompression error : %s", ZBUFF_getErrorName(errorCode));
-            totalGenSize += dstBuffSize;
-            totalCSize += readCSrcSize;
-        }
-        CHECK (errorCode != 0, "frame not fully decoded");
-        CHECK (totalGenSize != totalTestSize, "decompressed data : wrong size")
-        CHECK (totalCSize != cSize, "compressed data should be fully read")
-        { U64 const crcDest = XXH64(dstBuffer, totalTestSize, 0);
-          if (crcDest!=crcOrig) findDiff(copyBuffer, dstBuffer, totalTestSize);
-          CHECK (crcDest!=crcOrig, "decompressed data corrupted"); }
-
-        /*=====   noisy/erroneous src decompression test   =====*/
-
-        /* add some noise */
-        {   U32 const nbNoiseChunks = (FUZ_rand(&lseed) & 7) + 2;
-            U32 nn; for (nn=0; nn<nbNoiseChunks; nn++) {
-                size_t const randomNoiseSize = FUZ_randomLength(&lseed, maxSampleLog);
-                size_t const noiseSize  = MIN((cSize/3) , randomNoiseSize);
-                size_t const noiseStart = FUZ_rand(&lseed) % (srcBufferSize - noiseSize);
-                size_t const cStart = FUZ_rand(&lseed) % (cSize - noiseSize);
-                memcpy(cBuffer+cStart, srcBuffer+noiseStart, noiseSize);
-        }   }
-
-        /* try decompression on noisy data */
-        ZBUFF_decompressInit(zd);
-        totalCSize = 0;
-        totalGenSize = 0;
-        while ( (totalCSize < cSize) && (totalGenSize < dstBufferSize) ) {
-            size_t readCSrcSize = FUZ_randomLength(&lseed, maxSampleLog);
-            size_t const randomDstSize = FUZ_randomLength(&lseed, maxSampleLog);
-            size_t dstBuffSize = MIN(dstBufferSize - totalGenSize, randomDstSize);
-            size_t const decompressError = ZBUFF_decompressContinue(zd, dstBuffer+totalGenSize, &dstBuffSize, cBuffer+totalCSize, &readCSrcSize);
-            if (ZBUFF_isError(decompressError)) break;   /* error correctly detected */
-            totalGenSize += dstBuffSize;
-            totalCSize += readCSrcSize;
-    }   }
-    DISPLAY("\r%u fuzzer tests completed   \n", testNb);
-
-_cleanup:
-    ZBUFF_freeCCtx(zc);
-    ZBUFF_freeDCtx(zd);
-    free(cNoiseBuffer[0]);
-    free(cNoiseBuffer[1]);
-    free(cNoiseBuffer[2]);
-    free(cNoiseBuffer[3]);
-    free(cNoiseBuffer[4]);
-    free(copyBuffer);
-    free(cBuffer);
-    free(dstBuffer);
-    return result;
-
-_output_error:
-    result = 1;
-    goto _cleanup;
-}
-
-
-/*-*******************************************************
-*  Command line
-*********************************************************/
-static int FUZ_usage(const char* programName)
-{
-    DISPLAY( "Usage :\n");
-    DISPLAY( "      %s [args]\n", programName);
-    DISPLAY( "\n");
-    DISPLAY( "Arguments :\n");
-    DISPLAY( " -i#    : Nb of tests (default:%u) \n", nbTestsDefault);
-    DISPLAY( " -s#    : Select seed (default:prompt user)\n");
-    DISPLAY( " -t#    : Select starting test number (default:0)\n");
-    DISPLAY( " -P#    : Select compressibility in %% (default:%i%%)\n", FUZ_COMPRESSIBILITY_DEFAULT);
-    DISPLAY( " -v     : verbose\n");
-    DISPLAY( " -p     : pause at the end\n");
-    DISPLAY( " -h     : display help and exit\n");
-    return 0;
-}
-
-
-int main(int argc, const char** argv)
-{
-    U32 seed=0;
-    int seedset=0;
-    int argNb;
-    int nbTests = nbTestsDefault;
-    int testNb = 0;
-    int proba = FUZ_COMPRESSIBILITY_DEFAULT;
-    int result=0;
-    U32 mainPause = 0;
-    const char* programName = argv[0];
-    ZSTD_customMem customMem = { ZBUFF_allocFunction, ZBUFF_freeFunction, NULL };
-    ZSTD_customMem customNULL = { NULL, NULL, NULL };
-
-    /* Check command line */
-    for(argNb=1; argNb<argc; argNb++) {
-        const char* argument = argv[argNb];
-        if(!argument) continue;   /* Protection if argument empty */
-
-        /* Parsing commands. Aggregated commands are allowed */
-        if (argument[0]=='-') {
-            argument++;
-
-            while (*argument!=0) {
-                switch(*argument)
-                {
-                case 'h':
-                    return FUZ_usage(programName);
-                case 'v':
-                    argument++;
-                    g_displayLevel=4;
-                    break;
-                case 'q':
-                    argument++;
-                    g_displayLevel--;
-                    break;
-                case 'p': /* pause at the end */
-                    argument++;
-                    mainPause = 1;
-                    break;
-
-                case 'i':
-                    argument++;
-                    nbTests=0; g_clockTime=0;
-                    while ((*argument>='0') && (*argument<='9')) {
-                        nbTests *= 10;
-                        nbTests += *argument - '0';
-                        argument++;
-                    }
-                    break;
-
-                case 'T':
-                    argument++;
-                    nbTests=0; g_clockTime=0;
-                    while ((*argument>='0') && (*argument<='9')) {
-                        g_clockTime *= 10;
-                        g_clockTime += *argument - '0';
-                        argument++;
-                    }
-                    if (*argument=='m') g_clockTime *=60, argument++;
-                    if (*argument=='n') argument++;
-                    g_clockTime *= SEC_TO_MICRO;
-                    break;
-
-                case 's':
-                    argument++;
-                    seed=0;
-                    seedset=1;
-                    while ((*argument>='0') && (*argument<='9')) {
-                        seed *= 10;
-                        seed += *argument - '0';
-                        argument++;
-                    }
-                    break;
-
-                case 't':
-                    argument++;
-                    testNb=0;
-                    while ((*argument>='0') && (*argument<='9')) {
-                        testNb *= 10;
-                        testNb += *argument - '0';
-                        argument++;
-                    }
-                    break;
-
-                case 'P':   /* compressibility % */
-                    argument++;
-                    proba=0;
-                    while ((*argument>='0') && (*argument<='9')) {
-                        proba *= 10;
-                        proba += *argument - '0';
-                        argument++;
-                    }
-                    if (proba<0) proba=0;
-                    if (proba>100) proba=100;
-                    break;
-
-                default:
-                    return FUZ_usage(programName);
-                }
-    }   }   }   /* for(argNb=1; argNb<argc; argNb++) */
-
-    /* Get Seed */
-    DISPLAY("Starting zstd_buffered tester (%i-bits, %s)\n", (int)(sizeof(size_t)*8), ZSTD_VERSION_STRING);
-
-    if (!seedset) {
-        time_t const t = time(NULL);
-        U32 const h = XXH32(&t, sizeof(t), 1);
-        seed = h % 10000;
-    }
-    DISPLAY("Seed = %u\n", seed);
-    if (proba!=FUZ_COMPRESSIBILITY_DEFAULT) DISPLAY("Compressibility : %i%%\n", proba);
-
-    if (nbTests<=0) nbTests=1;
-
-    if (testNb==0) {
-        result = basicUnitTests(0, ((double)proba) / 100, customNULL);  /* constant seed for predictability */
-        if (!result) {
-            DISPLAYLEVEL(4, "Unit tests using customMem :\n")
-            result = basicUnitTests(0, ((double)proba) / 100, customMem);  /* use custom memory allocation functions */
-    }   }
-
-    if (!result)
-        result = fuzzerTests(seed, nbTests, testNb, ((double)proba) / 100);
-
-    if (mainPause) {
-        int unused;
-        DISPLAY("Press Enter \n");
-        unused = getchar();
-        (void)unused;
-    }
-    return result;
-}
diff --git a/tests/zstreamtest.c b/tests/zstreamtest.c
deleted file mode 100644
index 2047d4bd85cc..000000000000
--- a/tests/zstreamtest.c
+++ /dev/null
@@ -1,2477 +0,0 @@
-/*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-
-/*-************************************
- *  Compiler specific
- **************************************/
-#ifdef _MSC_VER    /* Visual Studio */
-#  define _CRT_SECURE_NO_WARNINGS   /* fgets */
-#  pragma warning(disable : 4127)   /* disable: C4127: conditional expression is constant */
-#  pragma warning(disable : 4146)   /* disable: C4146: minus unsigned expression */
-#endif
-
-
-/*-************************************
- *  Includes
- **************************************/
-#include <stdlib.h>       /* free */
-#include <stdio.h>        /* fgets, sscanf */
-#include <string.h>       /* strcmp */
-#include <assert.h>       /* assert */
-#include "timefn.h"       /* UTIL_time_t, UTIL_getTime */
-#include "mem.h"
-#define ZSTD_STATIC_LINKING_ONLY  /* ZSTD_maxCLevel, ZSTD_customMem, ZSTD_getDictID_fromFrame */
-#include "zstd.h"         /* ZSTD_compressBound */
-#include "zstd_errors.h"  /* ZSTD_error_srcSize_wrong */
-#include "zstdmt_compress.h"
-#include "zdict.h"        /* ZDICT_trainFromBuffer */
-#include "datagen.h"      /* RDG_genBuffer */
-#define XXH_STATIC_LINKING_ONLY   /* XXH64_state_t */
-#include "xxhash.h"       /* XXH64_* */
-#include "seqgen.h"
-#include "util.h"
-#include "timefn.h"       /* UTIL_time_t, UTIL_clockSpanMicro, UTIL_getTime */
-
-
-/*-************************************
- *  Constants
- **************************************/
-#define KB *(1U<<10)
-#define MB *(1U<<20)
-#define GB *(1U<<30)
-
-static const int nbTestsDefault = 10000;
-static const U32 g_cLevelMax_smallTests = 10;
-#define COMPRESSIBLE_NOISE_LENGTH (10 MB)
-#define FUZ_COMPRESSIBILITY_DEFAULT 50
-static const U32 prime32 = 2654435761U;
-
-
-/*-************************************
- *  Display Macros
- **************************************/
-#define DISPLAY(...)          fprintf(stderr, __VA_ARGS__)
-#define DISPLAYLEVEL(l, ...)  if (g_displayLevel>=l) {                     \
-                                  DISPLAY(__VA_ARGS__);                    \
-                                  if (g_displayLevel>=4) fflush(stderr); }
-static U32 g_displayLevel = 2;
-
-static const U64 g_refreshRate = SEC_TO_MICRO / 6;
-static UTIL_time_t g_displayClock = UTIL_TIME_INITIALIZER;
-
-#define DISPLAYUPDATE(l, ...) if (g_displayLevel>=l) { \
-            if ((UTIL_clockSpanMicro(g_displayClock) > g_refreshRate) || (g_displayLevel>=4)) \
-            { g_displayClock = UTIL_getTime(); DISPLAY(__VA_ARGS__); \
-            if (g_displayLevel>=4) fflush(stderr); } }
-
-static U64 g_clockTime = 0;
-
-
-/*-*******************************************************
- *  Check macros
- *********************************************************/
-#undef MIN
-#undef MAX
-#define MIN(a,b) ((a)<(b)?(a):(b))
-#define MAX(a,b) ((a)>(b)?(a):(b))
-/*! FUZ_rand() :
-    @return : a 27 bits random value, from a 32-bits `seed`.
-    `seed` is also modified */
-#define FUZ_rotl32(x,r) ((x << r) | (x >> (32 - r)))
-static U32 FUZ_rand(U32* seedPtr)
-{
-    static const U32 prime2 = 2246822519U;
-    U32 rand32 = *seedPtr;
-    rand32 *= prime32;
-    rand32 += prime2;
-    rand32  = FUZ_rotl32(rand32, 13);
-    *seedPtr = rand32;
-    return rand32 >> 5;
-}
-
-#define CHECK(cond, ...) {                                   \
-    if (cond) {                                              \
-        DISPLAY("Error => ");                                \
-        DISPLAY(__VA_ARGS__);                                \
-        DISPLAY(" (seed %u, test nb %u, line %u) \n",        \
-                (unsigned)seed, testNb, __LINE__);           \
-        goto _output_error;                                  \
-}   }
-
-#define CHECK_Z(f) {                                         \
-    size_t const err = f;                                    \
-    CHECK(ZSTD_isError(err), "%s : %s ",                     \
-          #f, ZSTD_getErrorName(err));                       \
-}
-
-#define CHECK_RET(ret, cond, ...) {                          \
-    if (cond) {                                              \
-        DISPLAY("Error %llu => ", (unsigned long long)ret);  \
-        DISPLAY(__VA_ARGS__);                                \
-        DISPLAY(" (line %u)\n", __LINE__);                   \
-        return ret;                                          \
-}   }
-
-#define CHECK_RET_Z(f) {                                     \
-    size_t const err = f;                                    \
-    CHECK_RET(err, ZSTD_isError(err), "%s : %s ",            \
-          #f, ZSTD_getErrorName(err));                       \
-}
-
-
-/*======================================================
- *   Basic Unit tests
- *======================================================*/
-
-typedef struct {
-    void* start;
-    size_t size;
-    size_t filled;
-} buffer_t;
-
-static const buffer_t kBuffNull = { NULL, 0 , 0 };
-
-static void FUZ_freeDictionary(buffer_t dict)
-{
-    free(dict.start);
-}
-
-static buffer_t FUZ_createDictionary(const void* src, size_t srcSize, size_t blockSize, size_t requestedDictSize)
-{
-    buffer_t dict = kBuffNull;
-    size_t const nbBlocks = (srcSize + (blockSize-1)) / blockSize;
-    size_t* const blockSizes = (size_t*)malloc(nbBlocks * sizeof(size_t));
-    if (!blockSizes) return kBuffNull;
-    dict.start = malloc(requestedDictSize);
-    if (!dict.start) { free(blockSizes); return kBuffNull; }
-    {   size_t nb;
-        for (nb=0; nb<nbBlocks-1; nb++) blockSizes[nb] = blockSize;
-        blockSizes[nbBlocks-1] = srcSize - (blockSize * (nbBlocks-1));
-    }
-    {   size_t const dictSize = ZDICT_trainFromBuffer(dict.start, requestedDictSize, src, blockSizes, (unsigned)nbBlocks);
-        free(blockSizes);
-        if (ZDICT_isError(dictSize)) { FUZ_freeDictionary(dict); return kBuffNull; }
-        dict.size = requestedDictSize;
-        dict.filled = dictSize;
-        return dict;
-    }
-}
-
-/* Round trips data and updates xxh with the decompressed data produced */
-static size_t SEQ_roundTrip(ZSTD_CCtx* cctx, ZSTD_DCtx* dctx,
-                            XXH64_state_t* xxh, void* data, size_t size,
-                            ZSTD_EndDirective endOp)
-{
-    static BYTE compressed[1024];
-    static BYTE uncompressed[1024];
-
-    ZSTD_inBuffer cin = {data, size, 0};
-    size_t cret;
-
-    do {
-        ZSTD_outBuffer cout = { compressed, sizeof(compressed), 0 };
-        ZSTD_inBuffer din   = { compressed, 0, 0 };
-        ZSTD_outBuffer dout = { uncompressed, 0, 0 };
-
-        cret = ZSTD_compressStream2(cctx, &cout, &cin, endOp);
-        if (ZSTD_isError(cret))
-            return cret;
-
-        din.size = cout.pos;
-        while (din.pos < din.size || (endOp == ZSTD_e_end && cret == 0)) {
-            size_t dret;
-
-            dout.pos = 0;
-            dout.size = sizeof(uncompressed);
-            dret = ZSTD_decompressStream(dctx, &dout, &din);
-            if (ZSTD_isError(dret))
-                return dret;
-            XXH64_update(xxh, dout.dst, dout.pos);
-            if (dret == 0)
-                break;
-        }
-    } while (cin.pos < cin.size || (endOp != ZSTD_e_continue && cret != 0));
-    return 0;
-}
-
-/* Generates some data and round trips it */
-static size_t SEQ_generateRoundTrip(ZSTD_CCtx* cctx, ZSTD_DCtx* dctx,
-                                    XXH64_state_t* xxh, SEQ_stream* seq,
-                                    SEQ_gen_type type, unsigned value)
-{
-    static BYTE data[1024];
-    size_t gen;
-
-    do {
-        SEQ_outBuffer sout = {data, sizeof(data), 0};
-        size_t ret;
-        gen = SEQ_gen(seq, type, value, &sout);
-
-        ret = SEQ_roundTrip(cctx, dctx, xxh, sout.dst, sout.pos, ZSTD_e_continue);
-        if (ZSTD_isError(ret))
-            return ret;
-    } while (gen != 0);
-
-    return 0;
-}
-
-static size_t getCCtxParams(ZSTD_CCtx* zc, ZSTD_parameters* savedParams)
-{
-    int value;
-    CHECK_RET_Z(ZSTD_CCtx_getParameter(zc, ZSTD_c_windowLog, (int*)&savedParams->cParams.windowLog));
-    CHECK_RET_Z(ZSTD_CCtx_getParameter(zc, ZSTD_c_hashLog, (int*)&savedParams->cParams.hashLog));
-    CHECK_RET_Z(ZSTD_CCtx_getParameter(zc, ZSTD_c_chainLog, (int*)&savedParams->cParams.chainLog));
-    CHECK_RET_Z(ZSTD_CCtx_getParameter(zc, ZSTD_c_searchLog, (int*)&savedParams->cParams.searchLog));
-    CHECK_RET_Z(ZSTD_CCtx_getParameter(zc, ZSTD_c_minMatch, (int*)&savedParams->cParams.minMatch));
-    CHECK_RET_Z(ZSTD_CCtx_getParameter(zc, ZSTD_c_targetLength, (int*)&savedParams->cParams.targetLength));
-    CHECK_RET_Z(ZSTD_CCtx_getParameter(zc, ZSTD_c_strategy, &value));
-    savedParams->cParams.strategy = value;
-
-    CHECK_RET_Z(ZSTD_CCtx_getParameter(zc, ZSTD_c_checksumFlag, &savedParams->fParams.checksumFlag));
-    CHECK_RET_Z(ZSTD_CCtx_getParameter(zc, ZSTD_c_contentSizeFlag, &savedParams->fParams.contentSizeFlag));
-    CHECK_RET_Z(ZSTD_CCtx_getParameter(zc, ZSTD_c_dictIDFlag, &value));
-    savedParams->fParams.noDictIDFlag = !value;
-    return 0;
-}
-
-static U32 badParameters(ZSTD_CCtx* zc, ZSTD_parameters const savedParams)
-{
-    ZSTD_parameters params;
-    if (ZSTD_isError(getCCtxParams(zc, &params))) return 10;
-    CHECK_RET(1, params.cParams.windowLog != savedParams.cParams.windowLog, "windowLog");
-    CHECK_RET(2, params.cParams.hashLog != savedParams.cParams.hashLog, "hashLog");
-    CHECK_RET(3, params.cParams.chainLog != savedParams.cParams.chainLog, "chainLog");
-    CHECK_RET(4, params.cParams.searchLog != savedParams.cParams.searchLog, "searchLog");
-    CHECK_RET(5, params.cParams.minMatch != savedParams.cParams.minMatch, "minMatch");
-    CHECK_RET(6, params.cParams.targetLength != savedParams.cParams.targetLength, "targetLength");
-
-    CHECK_RET(7, params.fParams.checksumFlag != savedParams.fParams.checksumFlag, "checksumFlag");
-    CHECK_RET(8, params.fParams.contentSizeFlag != savedParams.fParams.contentSizeFlag, "contentSizeFlag");
-    CHECK_RET(9, params.fParams.noDictIDFlag != savedParams.fParams.noDictIDFlag, "noDictIDFlag");
-    return 0;
-}
-
-static int basicUnitTests(U32 seed, double compressibility)
-{
-    size_t const CNBufferSize = COMPRESSIBLE_NOISE_LENGTH;
-    void* CNBuffer = malloc(CNBufferSize);
-    size_t const skippableFrameSize = 200 KB;
-    size_t const compressedBufferSize = (8 + skippableFrameSize) + ZSTD_compressBound(COMPRESSIBLE_NOISE_LENGTH);
-    void* compressedBuffer = malloc(compressedBufferSize);
-    size_t const decodedBufferSize = CNBufferSize;
-    void* decodedBuffer = malloc(decodedBufferSize);
-    size_t cSize;
-    int testResult = 0;
-    int testNb = 1;
-    U32 coreSeed = 0;  /* this name to conform with CHECK_Z macro display */
-    ZSTD_CStream* zc = ZSTD_createCStream();
-    ZSTD_DStream* zd = ZSTD_createDStream();
-    ZSTDMT_CCtx* mtctx = ZSTDMT_createCCtx(2);
-
-    ZSTD_inBuffer  inBuff, inBuff2;
-    ZSTD_outBuffer outBuff;
-    buffer_t dictionary = kBuffNull;
-    size_t const dictSize = 128 KB;
-    unsigned dictID = 0;
-
-    /* Create compressible test buffer */
-    if (!CNBuffer || !compressedBuffer || !decodedBuffer || !zc || !zd) {
-        DISPLAY("Not enough memory, aborting \n");
-        goto _output_error;
-    }
-    RDG_genBuffer(CNBuffer, CNBufferSize, compressibility, 0., seed);
-
-    /* Create dictionary */
-    DISPLAYLEVEL(3, "creating dictionary for unit tests \n");
-    dictionary = FUZ_createDictionary(CNBuffer, CNBufferSize / 3, 16 KB, 48 KB);
-    if (!dictionary.start) {
-        DISPLAY("Error creating dictionary, aborting \n");
-        goto _output_error;
-    }
-    dictID = ZDICT_getDictID(dictionary.start, dictionary.filled);
-
-    /* Basic compression test */
-    DISPLAYLEVEL(3, "test%3i : compress %u bytes : ", testNb++, COMPRESSIBLE_NOISE_LENGTH);
-    CHECK_Z( ZSTD_initCStream(zc, 1 /* cLevel */) );
-    outBuff.dst = (char*)(compressedBuffer);
-    outBuff.size = compressedBufferSize;
-    outBuff.pos = 0;
-    inBuff.src = CNBuffer;
-    inBuff.size = CNBufferSize;
-    inBuff.pos = 0;
-    CHECK_Z( ZSTD_compressStream(zc, &outBuff, &inBuff) );
-    if (inBuff.pos != inBuff.size) goto _output_error;   /* entire input should be consumed */
-    { size_t const r = ZSTD_endStream(zc, &outBuff);
-      if (r != 0) goto _output_error; }  /* error, or some data not flushed */
-    DISPLAYLEVEL(3, "OK (%u bytes)\n", (unsigned)outBuff.pos);
-
-    /* generate skippable frame */
-    MEM_writeLE32(compressedBuffer, ZSTD_MAGIC_SKIPPABLE_START);
-    MEM_writeLE32(((char*)compressedBuffer)+4, (U32)skippableFrameSize);
-    cSize = skippableFrameSize + 8;
-
-    /* Basic compression test using dict */
-    DISPLAYLEVEL(3, "test%3i : skipframe + compress %u bytes : ", testNb++, COMPRESSIBLE_NOISE_LENGTH);
-    CHECK_Z( ZSTD_initCStream_usingDict(zc, CNBuffer, dictSize, 1 /* cLevel */) );
-    outBuff.dst = (char*)(compressedBuffer)+cSize;
-    assert(compressedBufferSize > cSize);
-    outBuff.size = compressedBufferSize - cSize;
-    outBuff.pos = 0;
-    inBuff.src = CNBuffer;
-    inBuff.size = CNBufferSize;
-    inBuff.pos = 0;
-    CHECK_Z( ZSTD_compressStream(zc, &outBuff, &inBuff) );
-    if (inBuff.pos != inBuff.size) goto _output_error;   /* entire input should be consumed */
-    { size_t const r = ZSTD_endStream(zc, &outBuff);
-      if (r != 0) goto _output_error; }  /* error, or some data not flushed */
-    cSize += outBuff.pos;
-    DISPLAYLEVEL(3, "OK (%u bytes : %.2f%%)\n",
-                    (unsigned)cSize, (double)cSize/COMPRESSIBLE_NOISE_LENGTH*100);
-
-    /* context size functions */
-    DISPLAYLEVEL(3, "test%3i : estimate CStream size : ", testNb++);
-    {   ZSTD_compressionParameters const cParams = ZSTD_getCParams(1, CNBufferSize, dictSize);
-        size_t const cstreamSize = ZSTD_estimateCStreamSize_usingCParams(cParams);
-        size_t const cdictSize = ZSTD_estimateCDictSize_advanced(dictSize, cParams, ZSTD_dlm_byCopy); /* uses ZSTD_initCStream_usingDict() */
-        if (ZSTD_isError(cstreamSize)) goto _output_error;
-        if (ZSTD_isError(cdictSize)) goto _output_error;
-        DISPLAYLEVEL(3, "OK (%u bytes) \n", (unsigned)(cstreamSize + cdictSize));
-    }
-
-    /* context size functions */
-    DISPLAYLEVEL(3, "test%3i : estimate CStream size using CCtxParams : ", testNb++);
-    {   ZSTD_CCtx_params* const params = ZSTD_createCCtxParams();
-        size_t cstreamSize, cctxSize;
-        CHECK_Z( ZSTD_CCtxParams_setParameter(params, ZSTD_c_compressionLevel, 19) );
-        cstreamSize = ZSTD_estimateCStreamSize_usingCCtxParams(params);
-        CHECK_Z(cstreamSize);
-        cctxSize = ZSTD_estimateCCtxSize_usingCCtxParams(params);
-        CHECK_Z(cctxSize);
-        if (cstreamSize <= cctxSize + 2 * ZSTD_BLOCKSIZE_MAX) goto _output_error;
-        ZSTD_freeCCtxParams(params);
-        DISPLAYLEVEL(3, "OK \n");
-    }
-
-    DISPLAYLEVEL(3, "test%3i : check actual CStream size : ", testNb++);
-    {   size_t const s = ZSTD_sizeof_CStream(zc);
-        if (ZSTD_isError(s)) goto _output_error;
-        DISPLAYLEVEL(3, "OK (%u bytes) \n", (unsigned)s);
-    }
-
-    /* Attempt bad compression parameters */
-    DISPLAYLEVEL(3, "test%3i : use bad compression parameters : ", testNb++);
-    {   size_t r;
-        ZSTD_parameters params = ZSTD_getParams(1, 0, 0);
-        params.cParams.minMatch = 2;
-        r = ZSTD_initCStream_advanced(zc, NULL, 0, params, 0);
-        if (!ZSTD_isError(r)) goto _output_error;
-        DISPLAYLEVEL(3, "init error : %s \n", ZSTD_getErrorName(r));
-    }
-
-    /* skippable frame test */
-    DISPLAYLEVEL(3, "test%3i : decompress skippable frame : ", testNb++);
-    CHECK_Z( ZSTD_initDStream_usingDict(zd, CNBuffer, dictSize) );
-    inBuff.src = compressedBuffer;
-    inBuff.size = cSize;
-    inBuff.pos = 0;
-    outBuff.dst = decodedBuffer;
-    outBuff.size = CNBufferSize;
-    outBuff.pos = 0;
-    {   size_t const r = ZSTD_decompressStream(zd, &outBuff, &inBuff);
-        DISPLAYLEVEL(5, " ( ZSTD_decompressStream => %u ) ", (unsigned)r);
-        if (r != 0) goto _output_error;
-    }
-    if (outBuff.pos != 0) goto _output_error;   /* skippable frame output len is 0 */
-    DISPLAYLEVEL(3, "OK \n");
-
-    /* Basic decompression test */
-    inBuff2 = inBuff;
-    DISPLAYLEVEL(3, "test%3i : decompress %u bytes : ", testNb++, COMPRESSIBLE_NOISE_LENGTH);
-    ZSTD_initDStream_usingDict(zd, CNBuffer, dictSize);
-    CHECK_Z( ZSTD_DCtx_setParameter(zd, ZSTD_d_windowLogMax, ZSTD_WINDOWLOG_LIMIT_DEFAULT+1) );  /* large limit */
-    { size_t const remaining = ZSTD_decompressStream(zd, &outBuff, &inBuff);
-      if (remaining != 0) goto _output_error; }  /* should reach end of frame == 0; otherwise, some data left, or an error */
-    if (outBuff.pos != CNBufferSize) goto _output_error;   /* should regenerate the same amount */
-    if (inBuff.pos != inBuff.size) goto _output_error;   /* should have read the entire frame */
-    DISPLAYLEVEL(3, "OK \n");
-
-    /* Re-use without init */
-    DISPLAYLEVEL(3, "test%3i : decompress again without init (re-use previous settings): ", testNb++);
-    outBuff.pos = 0;
-    { size_t const remaining = ZSTD_decompressStream(zd, &outBuff, &inBuff2);
-      if (remaining != 0) goto _output_error; }  /* should reach end of frame == 0; otherwise, some data left, or an error */
-    if (outBuff.pos != CNBufferSize) goto _output_error;   /* should regenerate the same amount */
-    if (inBuff.pos != inBuff.size) goto _output_error;   /* should have read the entire frame */
-    DISPLAYLEVEL(3, "OK \n");
-
-    /* check regenerated data is byte exact */
-    DISPLAYLEVEL(3, "test%3i : check decompressed result : ", testNb++);
-    {   size_t i;
-        for (i=0; i<CNBufferSize; i++) {
-            if (((BYTE*)decodedBuffer)[i] != ((BYTE*)CNBuffer)[i]) goto _output_error;
-    }   }
-    DISPLAYLEVEL(3, "OK \n");
-
-    /* context size functions */
-    DISPLAYLEVEL(3, "test%3i : estimate DStream size : ", testNb++);
-    {   ZSTD_frameHeader fhi;
-        const void* cStart = (char*)compressedBuffer + (skippableFrameSize + 8);
-        size_t const gfhError = ZSTD_getFrameHeader(&fhi, cStart, cSize);
-        if (gfhError!=0) goto _output_error;
-        DISPLAYLEVEL(5, " (windowSize : %u) ", (unsigned)fhi.windowSize);
-        {   size_t const s = ZSTD_estimateDStreamSize(fhi.windowSize)
-                            /* uses ZSTD_initDStream_usingDict() */
-                           + ZSTD_estimateDDictSize(dictSize, ZSTD_dlm_byCopy);
-            if (ZSTD_isError(s)) goto _output_error;
-            DISPLAYLEVEL(3, "OK (%u bytes) \n", (unsigned)s);
-    }   }
-
-    DISPLAYLEVEL(3, "test%3i : check actual DStream size : ", testNb++);
-    { size_t const s = ZSTD_sizeof_DStream(zd);
-      if (ZSTD_isError(s)) goto _output_error;
-      DISPLAYLEVEL(3, "OK (%u bytes) \n", (unsigned)s);
-    }
-
-    /* Decompression by small increment */
-    DISPLAYLEVEL(3, "test%3i : decompress byte-by-byte : ", testNb++);
-    {   /* skippable frame */
-        size_t r = 1;
-        ZSTD_initDStream_usingDict(zd, CNBuffer, dictSize);
-        inBuff.src = compressedBuffer;
-        outBuff.dst = decodedBuffer;
-        inBuff.pos = 0;
-        outBuff.pos = 0;
-        while (r) {   /* skippable frame */
-            size_t const inSize = (FUZ_rand(&coreSeed) & 15) + 1;
-            size_t const outSize = (FUZ_rand(&coreSeed) & 15) + 1;
-            inBuff.size = inBuff.pos + inSize;
-            outBuff.size = outBuff.pos + outSize;
-            r = ZSTD_decompressStream(zd, &outBuff, &inBuff);
-            if (ZSTD_isError(r)) DISPLAYLEVEL(4, "ZSTD_decompressStream on skippable frame error : %s \n", ZSTD_getErrorName(r));
-            if (ZSTD_isError(r)) goto _output_error;
-        }
-        /* normal frame */
-        ZSTD_initDStream_usingDict(zd, CNBuffer, dictSize);
-        r=1;
-        while (r) {
-            size_t const inSize = FUZ_rand(&coreSeed) & 15;
-            size_t const outSize = (FUZ_rand(&coreSeed) & 15) + (!inSize);   /* avoid having both sizes at 0 => would trigger a no_forward_progress error */
-            inBuff.size = inBuff.pos + inSize;
-            outBuff.size = outBuff.pos + outSize;
-            r = ZSTD_decompressStream(zd, &outBuff, &inBuff);
-            if (ZSTD_isError(r)) DISPLAYLEVEL(4, "ZSTD_decompressStream error : %s \n", ZSTD_getErrorName(r));
-            if (ZSTD_isError(r)) goto _output_error;
-        }
-    }
-    if (outBuff.pos != CNBufferSize) DISPLAYLEVEL(4, "outBuff.pos != CNBufferSize : should have regenerated same amount ! \n");
-    if (outBuff.pos != CNBufferSize) goto _output_error;   /* should regenerate the same amount */
-    if (inBuff.pos != cSize) DISPLAYLEVEL(4, "inBuff.pos != cSize : should have real all input ! \n");
-    if (inBuff.pos != cSize) goto _output_error;   /* should have read the entire frame */
-    DISPLAYLEVEL(3, "OK \n");
-
-    /* check regenerated data is byte exact */
-    DISPLAYLEVEL(3, "test%3i : check decompressed result : ", testNb++);
-    {   size_t i;
-        for (i=0; i<CNBufferSize; i++) {
-            if (((BYTE*)decodedBuffer)[i] != ((BYTE*)CNBuffer)[i]) goto _output_error;
-    }   }
-    DISPLAYLEVEL(3, "OK \n");
-
-    /* Decompression forward progress */
-    DISPLAYLEVEL(3, "test%3i : generate error when ZSTD_decompressStream() doesn't progress : ", testNb++);
-    {   /* skippable frame */
-        size_t r = 0;
-        int decNb = 0;
-        int const maxDec = 100;
-        inBuff.src = compressedBuffer;
-        inBuff.size = cSize;
-        inBuff.pos = 0;
-
-        outBuff.dst = decodedBuffer;
-        outBuff.pos = 0;
-        outBuff.size = CNBufferSize-1;   /* 1 byte missing */
-
-        for (decNb=0; decNb<maxDec; decNb++) {
-            if (r==0) ZSTD_initDStream_usingDict(zd, CNBuffer, dictSize);
-            r = ZSTD_decompressStream(zd, &outBuff, &inBuff);
-            if (ZSTD_isError(r)) break;
-        }
-        if (!ZSTD_isError(r)) DISPLAYLEVEL(4, "ZSTD_decompressStream should have triggered a no_forward_progress error \n");
-        if (!ZSTD_isError(r)) goto _output_error;   /* should have triggered no_forward_progress error */
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    /* _srcSize compression test */
-    DISPLAYLEVEL(3, "test%3i : compress_srcSize %u bytes : ", testNb++, COMPRESSIBLE_NOISE_LENGTH);
-    CHECK_Z( ZSTD_initCStream_srcSize(zc, 1, CNBufferSize) );
-    outBuff.dst = (char*)(compressedBuffer);
-    outBuff.size = compressedBufferSize;
-    outBuff.pos = 0;
-    inBuff.src = CNBuffer;
-    inBuff.size = CNBufferSize;
-    inBuff.pos = 0;
-    CHECK_Z( ZSTD_compressStream(zc, &outBuff, &inBuff) );
-    CHECK(inBuff.pos != inBuff.size, "Entire input should be consumed");
-    {   size_t const r = ZSTD_endStream(zc, &outBuff);
-        CHECK(r != 0, "Error or some data not flushed (ret=%zu)", r);
-    }
-    {   unsigned long long origSize = ZSTD_findDecompressedSize(outBuff.dst, outBuff.pos);
-        CHECK(origSize == ZSTD_CONTENTSIZE_UNKNOWN, "Unknown!");
-        CHECK((size_t)origSize != CNBufferSize, "Exact original size must be present (got %llu)", origSize);
-    }
-    DISPLAYLEVEL(3, "OK (%u bytes : %.2f%%)\n", (unsigned)cSize, (double)cSize/COMPRESSIBLE_NOISE_LENGTH*100);
-
-    /* wrong _srcSize compression test */
-    DISPLAYLEVEL(3, "test%3i : too large srcSize : %u bytes : ", testNb++, COMPRESSIBLE_NOISE_LENGTH-1);
-    ZSTD_initCStream_srcSize(zc, 1, CNBufferSize+1);
-    outBuff.dst = (char*)(compressedBuffer);
-    outBuff.size = compressedBufferSize;
-    outBuff.pos = 0;
-    inBuff.src = CNBuffer;
-    inBuff.size = CNBufferSize;
-    inBuff.pos = 0;
-    CHECK_Z( ZSTD_compressStream(zc, &outBuff, &inBuff) );
-    if (inBuff.pos != inBuff.size) goto _output_error;   /* entire input should be consumed */
-    { size_t const r = ZSTD_endStream(zc, &outBuff);
-      if (ZSTD_getErrorCode(r) != ZSTD_error_srcSize_wrong) goto _output_error;    /* must fail : wrong srcSize */
-      DISPLAYLEVEL(3, "OK (error detected : %s) \n", ZSTD_getErrorName(r)); }
-
-    /* wrong _srcSize compression test */
-    DISPLAYLEVEL(3, "test%3i : too small srcSize : %u bytes : ", testNb++, COMPRESSIBLE_NOISE_LENGTH-1);
-    ZSTD_initCStream_srcSize(zc, 1, CNBufferSize-1);
-    outBuff.dst = (char*)(compressedBuffer);
-    outBuff.size = compressedBufferSize;
-    outBuff.pos = 0;
-    inBuff.src = CNBuffer;
-    inBuff.size = CNBufferSize;
-    inBuff.pos = 0;
-    {   size_t const r = ZSTD_compressStream(zc, &outBuff, &inBuff);
-        if (ZSTD_getErrorCode(r) != ZSTD_error_srcSize_wrong) goto _output_error;    /* must fail : wrong srcSize */
-        DISPLAYLEVEL(3, "OK (error detected : %s) \n", ZSTD_getErrorName(r));
-    }
-
-    DISPLAYLEVEL(3, "test%3i : wrong srcSize !contentSizeFlag : %u bytes : ", testNb++, COMPRESSIBLE_NOISE_LENGTH-1);
-    {   ZSTD_parameters params = ZSTD_getParams(1, CNBufferSize, 0);
-        params.fParams.contentSizeFlag = 0;
-        CHECK_Z(ZSTD_initCStream_advanced(zc, NULL, 0, params, CNBufferSize - MIN(CNBufferSize, 200 KB)));
-        outBuff.dst = (char*)compressedBuffer;
-        outBuff.size = compressedBufferSize;
-        outBuff.pos = 0;
-        inBuff.src = CNBuffer;
-        inBuff.size = CNBufferSize;
-        inBuff.pos = 0;
-        {   size_t const r = ZSTD_compressStream(zc, &outBuff, &inBuff);
-            if (ZSTD_getErrorCode(r) != ZSTD_error_srcSize_wrong) goto _output_error;    /* must fail : wrong srcSize */
-            DISPLAYLEVEL(3, "OK (error detected : %s) \n", ZSTD_getErrorName(r));
-    }   }
-
-    /* Complex context re-use scenario */
-    DISPLAYLEVEL(3, "test%3i : context re-use : ", testNb++);
-    ZSTD_freeCStream(zc);
-    zc = ZSTD_createCStream();
-    if (zc==NULL) goto _output_error;   /* memory allocation issue */
-    /* use 1 */
-    {   size_t const inSize = 513;
-        DISPLAYLEVEL(5, "use1 ");
-        ZSTD_initCStream_advanced(zc, NULL, 0, ZSTD_getParams(19, inSize, 0), inSize);   /* needs btopt + search3 to trigger hashLog3 */
-        inBuff.src = CNBuffer;
-        inBuff.size = inSize;
-        inBuff.pos = 0;
-        outBuff.dst = (char*)(compressedBuffer)+cSize;
-        outBuff.size = ZSTD_compressBound(inSize);
-        outBuff.pos = 0;
-        DISPLAYLEVEL(5, "compress1 ");
-        CHECK_Z( ZSTD_compressStream(zc, &outBuff, &inBuff) );
-        if (inBuff.pos != inBuff.size) goto _output_error;   /* entire input should be consumed */
-        DISPLAYLEVEL(5, "end1 ");
-        { size_t const r = ZSTD_endStream(zc, &outBuff);
-            if (r != 0) goto _output_error; }  /* error, or some data not flushed */
-    }
-    /* use 2 */
-    {   size_t const inSize = 1025;   /* will not continue, because tables auto-adjust and are therefore different size */
-        DISPLAYLEVEL(5, "use2 ");
-        ZSTD_initCStream_advanced(zc, NULL, 0, ZSTD_getParams(19, inSize, 0), inSize);   /* needs btopt + search3 to trigger hashLog3 */
-        inBuff.src = CNBuffer;
-        inBuff.size = inSize;
-        inBuff.pos = 0;
-        outBuff.dst = (char*)(compressedBuffer)+cSize;
-        outBuff.size = ZSTD_compressBound(inSize);
-        outBuff.pos = 0;
-        DISPLAYLEVEL(5, "compress2 ");
-        CHECK_Z( ZSTD_compressStream(zc, &outBuff, &inBuff) );
-        if (inBuff.pos != inBuff.size) goto _output_error;   /* entire input should be consumed */
-        DISPLAYLEVEL(5, "end2 ");
-        { size_t const r = ZSTD_endStream(zc, &outBuff);
-            if (r != 0) goto _output_error; }  /* error, or some data not flushed */
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    /* CDict scenario */
-    DISPLAYLEVEL(3, "test%3i : digested dictionary : ", testNb++);
-    {   ZSTD_CDict* const cdict = ZSTD_createCDict(dictionary.start, dictionary.filled, 1 /*byRef*/ );
-        size_t const initError = ZSTD_initCStream_usingCDict(zc, cdict);
-        DISPLAYLEVEL(5, "ZSTD_initCStream_usingCDict result : %u ", (unsigned)initError);
-        if (ZSTD_isError(initError)) goto _output_error;
-        outBuff.dst = compressedBuffer;
-        outBuff.size = compressedBufferSize;
-        outBuff.pos = 0;
-        inBuff.src = CNBuffer;
-        inBuff.size = CNBufferSize;
-        inBuff.pos = 0;
-        DISPLAYLEVEL(5, "- starting ZSTD_compressStream ");
-        CHECK_Z( ZSTD_compressStream(zc, &outBuff, &inBuff) );
-        if (inBuff.pos != inBuff.size) goto _output_error;   /* entire input should be consumed */
-        {   size_t const r = ZSTD_endStream(zc, &outBuff);
-            DISPLAYLEVEL(5, "- ZSTD_endStream result : %u ", (unsigned)r);
-            if (r != 0) goto _output_error;  /* error, or some data not flushed */
-        }
-        cSize = outBuff.pos;
-        ZSTD_freeCDict(cdict);
-        DISPLAYLEVEL(3, "OK (%u bytes : %.2f%%)\n", (unsigned)cSize, (double)cSize/CNBufferSize*100);
-    }
-
-    DISPLAYLEVEL(3, "test%3i : check CStream size : ", testNb++);
-    { size_t const s = ZSTD_sizeof_CStream(zc);
-      if (ZSTD_isError(s)) goto _output_error;
-      DISPLAYLEVEL(3, "OK (%u bytes) \n", (unsigned)s);
-    }
-
-    DISPLAYLEVEL(4, "test%3i : check Dictionary ID : ", testNb++);
-    { unsigned const dID = ZSTD_getDictID_fromFrame(compressedBuffer, cSize);
-      if (dID != dictID) goto _output_error;
-      DISPLAYLEVEL(4, "OK (%u) \n", dID);
-    }
-
-    /* DDict scenario */
-    DISPLAYLEVEL(3, "test%3i : decompress %u bytes with digested dictionary : ", testNb++, (unsigned)CNBufferSize);
-    {   ZSTD_DDict* const ddict = ZSTD_createDDict(dictionary.start, dictionary.filled);
-        size_t const initError = ZSTD_initDStream_usingDDict(zd, ddict);
-        if (ZSTD_isError(initError)) goto _output_error;
-        outBuff.dst = decodedBuffer;
-        outBuff.size = CNBufferSize;
-        outBuff.pos = 0;
-        inBuff.src = compressedBuffer;
-        inBuff.size = cSize;
-        inBuff.pos = 0;
-        { size_t const r = ZSTD_decompressStream(zd, &outBuff, &inBuff);
-          if (r != 0) goto _output_error; }  /* should reach end of frame == 0; otherwise, some data left, or an error */
-        if (outBuff.pos != CNBufferSize) goto _output_error;   /* should regenerate the same amount */
-        if (inBuff.pos != inBuff.size) goto _output_error;   /* should have read the entire frame */
-        ZSTD_freeDDict(ddict);
-        DISPLAYLEVEL(3, "OK \n");
-    }
-
-    /* Memory restriction */
-    DISPLAYLEVEL(3, "test%3i : maxWindowSize < frame requirement : ", testNb++);
-    ZSTD_initDStream_usingDict(zd, CNBuffer, dictSize);
-    CHECK_Z( ZSTD_DCtx_setParameter(zd, ZSTD_d_windowLogMax, 10) );  /* too small limit */
-    outBuff.dst = decodedBuffer;
-    outBuff.size = CNBufferSize;
-    outBuff.pos = 0;
-    inBuff.src = compressedBuffer;
-    inBuff.size = cSize;
-    inBuff.pos = 0;
-    { size_t const r = ZSTD_decompressStream(zd, &outBuff, &inBuff);
-      if (!ZSTD_isError(r)) goto _output_error;  /* must fail : frame requires > 100 bytes */
-      DISPLAYLEVEL(3, "OK (%s)\n", ZSTD_getErrorName(r)); }
-    ZSTD_DCtx_reset(zd, ZSTD_reset_session_and_parameters);   /* leave zd in good shape for next tests */
-
-    DISPLAYLEVEL(3, "test%3i : dictionary source size and level : ", testNb++);
-    {   ZSTD_DCtx* const dctx = ZSTD_createDCtx();
-        int const maxLevel = 16;   /* first level with zstd_opt */
-        int level;
-        assert(maxLevel < ZSTD_maxCLevel());
-        CHECK_Z( ZSTD_DCtx_loadDictionary_byReference(dctx, dictionary.start, dictionary.filled) );
-        for (level = 1; level <= maxLevel; ++level) {
-            ZSTD_CDict* const cdict = ZSTD_createCDict(dictionary.start, dictionary.filled, level);
-            size_t const maxSize = MIN(1 MB, CNBufferSize);
-            size_t size;
-            for (size = 512; size <= maxSize; size <<= 1) {
-                U64 const crcOrig = XXH64(CNBuffer, size, 0);
-                ZSTD_CCtx* const cctx = ZSTD_createCCtx();
-                ZSTD_parameters savedParams;
-                getCCtxParams(cctx, &savedParams);
-                outBuff.dst = compressedBuffer;
-                outBuff.size = compressedBufferSize;
-                outBuff.pos = 0;
-                inBuff.src = CNBuffer;
-                inBuff.size = size;
-                inBuff.pos = 0;
-                CHECK_Z(ZSTD_CCtx_refCDict(cctx, cdict));
-                CHECK_Z(ZSTD_compressStream2(cctx, &outBuff, &inBuff, ZSTD_e_end));
-                CHECK(badParameters(cctx, savedParams), "Bad CCtx params");
-                if (inBuff.pos != inBuff.size) goto _output_error;
-                {   ZSTD_outBuffer decOut = {decodedBuffer, size, 0};
-                    ZSTD_inBuffer decIn = {outBuff.dst, outBuff.pos, 0};
-                    CHECK_Z( ZSTD_decompressStream(dctx, &decOut, &decIn) );
-                    if (decIn.pos != decIn.size) goto _output_error;
-                    if (decOut.pos != size) goto _output_error;
-                    {   U64 const crcDec = XXH64(decOut.dst, decOut.pos, 0);
-                        if (crcDec != crcOrig) goto _output_error;
-                }   }
-                ZSTD_freeCCtx(cctx);
-            }
-            ZSTD_freeCDict(cdict);
-        }
-        ZSTD_freeDCtx(dctx);
-    }
-    DISPLAYLEVEL(3, "OK\n");
-
-    ZSTD_CCtx_reset(zc, ZSTD_reset_session_and_parameters);
-    CHECK_Z( ZSTD_CCtx_loadDictionary(zc, dictionary.start, dictionary.filled) );
-    cSize = ZSTD_compress2(zc, compressedBuffer, compressedBufferSize, CNBuffer, MIN(CNBufferSize, 100 KB));
-    CHECK_Z(cSize);
-    DISPLAYLEVEL(3, "test%3i : ZSTD_decompressStream() with dictionary : ", testNb++);
-    {
-        ZSTD_DCtx* dctx = ZSTD_createDCtx();
-        /* We should fail to decompress without a dictionary. */
-        ZSTD_DCtx_reset(dctx, ZSTD_reset_session_and_parameters);
-        {   ZSTD_outBuffer out = {decodedBuffer, decodedBufferSize, 0};
-            ZSTD_inBuffer in = {compressedBuffer, cSize, 0};
-            size_t const ret = ZSTD_decompressStream(dctx, &out, &in);
-            if (!ZSTD_isError(ret)) goto _output_error;
-        }
-        /* We should succeed to decompress with the dictionary. */
-        ZSTD_DCtx_reset(dctx, ZSTD_reset_session_and_parameters);
-        CHECK_Z( ZSTD_DCtx_loadDictionary(dctx, dictionary.start, dictionary.filled) );
-        {   ZSTD_outBuffer out = {decodedBuffer, decodedBufferSize, 0};
-            ZSTD_inBuffer in = {compressedBuffer, cSize, 0};
-            if (ZSTD_decompressStream(dctx, &out, &in) != 0) goto _output_error;
-            if (in.pos != in.size) goto _output_error;
-        }
-        /* The dictionary should presist across calls. */
-        {   ZSTD_outBuffer out = {decodedBuffer, decodedBufferSize, 0};
-            ZSTD_inBuffer in = {compressedBuffer, cSize, 0};
-            if (ZSTD_decompressStream(dctx, &out, &in) != 0) goto _output_error;
-            if (in.pos != in.size) goto _output_error;
-        }
-        /* The dictionary should not be cleared by ZSTD_reset_session_only. */
-        ZSTD_DCtx_reset(dctx, ZSTD_reset_session_only);
-        {   ZSTD_outBuffer out = {decodedBuffer, decodedBufferSize, 0};
-            ZSTD_inBuffer in = {compressedBuffer, cSize, 0};
-            if (ZSTD_decompressStream(dctx, &out, &in) != 0) goto _output_error;
-            if (in.pos != in.size) goto _output_error;
-        }
-        /* When we reset the context the dictionary is cleared. */
-        ZSTD_DCtx_reset(dctx, ZSTD_reset_session_and_parameters);
-        {   ZSTD_outBuffer out = {decodedBuffer, decodedBufferSize, 0};
-            ZSTD_inBuffer in = {compressedBuffer, cSize, 0};
-            size_t const ret = ZSTD_decompressStream(dctx, &out, &in);
-            if (!ZSTD_isError(ret)) goto _output_error;
-        }
-        ZSTD_freeDCtx(dctx);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : ZSTD_resetDStream() with dictionary : ", testNb++);
-    {
-        ZSTD_DCtx* dctx = ZSTD_createDCtx();
-        /* We should succeed to decompress with the dictionary. */
-        ZSTD_resetDStream(dctx);
-        CHECK_Z( ZSTD_DCtx_loadDictionary(dctx, dictionary.start, dictionary.filled) );
-        {   ZSTD_outBuffer out = {decodedBuffer, decodedBufferSize, 0};
-            ZSTD_inBuffer in = {compressedBuffer, cSize, 0};
-            if (ZSTD_decompressStream(dctx, &out, &in) != 0) goto _output_error;
-            if (in.pos != in.size) goto _output_error;
-        }
-        /* The dictionary should not be cleared by ZSTD_resetDStream(). */
-        ZSTD_resetDStream(dctx);
-        {   ZSTD_outBuffer out = {decodedBuffer, decodedBufferSize, 0};
-            ZSTD_inBuffer in = {compressedBuffer, cSize, 0};
-            if (ZSTD_decompressStream(dctx, &out, &in) != 0) goto _output_error;
-            if (in.pos != in.size) goto _output_error;
-        }
-        /* The dictionary should be cleared by ZSTD_initDStream(). */
-        CHECK_Z( ZSTD_initDStream(dctx) );
-        {   ZSTD_outBuffer out = {decodedBuffer, decodedBufferSize, 0};
-            ZSTD_inBuffer in = {compressedBuffer, cSize, 0};
-            size_t const ret = ZSTD_decompressStream(dctx, &out, &in);
-            if (!ZSTD_isError(ret)) goto _output_error;
-        }
-        ZSTD_freeDCtx(dctx);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : ZSTD_decompressStream() with ddict : ", testNb++);
-    {
-        ZSTD_DCtx* dctx = ZSTD_createDCtx();
-        ZSTD_DDict* ddict = ZSTD_createDDict(dictionary.start, dictionary.filled);
-        /* We should succeed to decompress with the ddict. */
-        ZSTD_DCtx_reset(dctx, ZSTD_reset_session_and_parameters);
-        CHECK_Z( ZSTD_DCtx_refDDict(dctx, ddict) );
-        {   ZSTD_outBuffer out = {decodedBuffer, decodedBufferSize, 0};
-            ZSTD_inBuffer in = {compressedBuffer, cSize, 0};
-            if (ZSTD_decompressStream(dctx, &out, &in) != 0) goto _output_error;
-            if (in.pos != in.size) goto _output_error;
-        }
-        /* The ddict should presist across calls. */
-        {   ZSTD_outBuffer out = {decodedBuffer, decodedBufferSize, 0};
-            ZSTD_inBuffer in = {compressedBuffer, cSize, 0};
-            if (ZSTD_decompressStream(dctx, &out, &in) != 0) goto _output_error;
-            if (in.pos != in.size) goto _output_error;
-        }
-        /* When we reset the context the ddict is cleared. */
-        ZSTD_DCtx_reset(dctx, ZSTD_reset_session_and_parameters);
-        {   ZSTD_outBuffer out = {decodedBuffer, decodedBufferSize, 0};
-            ZSTD_inBuffer in = {compressedBuffer, cSize, 0};
-            size_t const ret = ZSTD_decompressStream(dctx, &out, &in);
-            if (!ZSTD_isError(ret)) goto _output_error;
-        }
-        ZSTD_freeDCtx(dctx);
-        ZSTD_freeDDict(ddict);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : ZSTD_decompressDCtx() with prefix : ", testNb++);
-    {
-        ZSTD_DCtx* dctx = ZSTD_createDCtx();
-        /* We should succeed to decompress with the prefix. */
-        ZSTD_DCtx_reset(dctx, ZSTD_reset_session_and_parameters);
-        CHECK_Z( ZSTD_DCtx_refPrefix_advanced(dctx, dictionary.start, dictionary.filled, ZSTD_dct_auto) );
-        {   ZSTD_outBuffer out = {decodedBuffer, decodedBufferSize, 0};
-            ZSTD_inBuffer in = {compressedBuffer, cSize, 0};
-            if (ZSTD_decompressStream(dctx, &out, &in) != 0) goto _output_error;
-            if (in.pos != in.size) goto _output_error;
-        }
-        /* The prefix should be cleared after the first compression. */
-        {   ZSTD_outBuffer out = {decodedBuffer, decodedBufferSize, 0};
-            ZSTD_inBuffer in = {compressedBuffer, cSize, 0};
-            size_t const ret = ZSTD_decompressStream(dctx, &out, &in);
-            if (!ZSTD_isError(ret)) goto _output_error;
-        }
-        ZSTD_freeDCtx(dctx);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : ZSTD_initDStream*() with dictionary : ", testNb++);
-    {
-        ZSTD_DCtx* dctx = ZSTD_createDCtx();
-        ZSTD_DDict* ddict = ZSTD_createDDict(dictionary.start, dictionary.filled);
-        size_t ret;
-        /* We should succeed to decompress with the dictionary. */
-        CHECK_Z( ZSTD_initDStream_usingDict(dctx, dictionary.start, dictionary.filled) );
-        CHECK_Z( ZSTD_decompressDCtx(dctx, decodedBuffer, decodedBufferSize, compressedBuffer, cSize) );
-        /* The dictionary should presist across calls. */
-        CHECK_Z( ZSTD_decompressDCtx(dctx, decodedBuffer, decodedBufferSize, compressedBuffer, cSize) );
-        /* We should succeed to decompress with the ddict. */
-        CHECK_Z( ZSTD_initDStream_usingDDict(dctx, ddict) );
-        CHECK_Z( ZSTD_decompressDCtx(dctx, decodedBuffer, decodedBufferSize, compressedBuffer, cSize) );
-        /* The ddict should presist across calls. */
-        CHECK_Z( ZSTD_decompressDCtx(dctx, decodedBuffer, decodedBufferSize, compressedBuffer, cSize) );
-        /* When we reset the context the ddict is cleared. */
-        CHECK_Z( ZSTD_initDStream(dctx) );
-        ret = ZSTD_decompressDCtx(dctx, decodedBuffer, decodedBufferSize, compressedBuffer, cSize);
-        if (!ZSTD_isError(ret)) goto _output_error;
-        ZSTD_freeDCtx(dctx);
-        ZSTD_freeDDict(ddict);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : ZSTD_initCStream_usingCDict_advanced with masked dictID : ", testNb++);
-    {   ZSTD_compressionParameters const cParams = ZSTD_getCParams(1, CNBufferSize, dictionary.filled);
-        ZSTD_frameParameters const fParams = { 1 /* contentSize */, 1 /* checksum */, 1 /* noDictID */};
-        ZSTD_CDict* const cdict = ZSTD_createCDict_advanced(dictionary.start, dictionary.filled, ZSTD_dlm_byRef, ZSTD_dct_auto, cParams, ZSTD_defaultCMem);
-        size_t const initError = ZSTD_initCStream_usingCDict_advanced(zc, cdict, fParams, CNBufferSize);
-        if (ZSTD_isError(initError)) goto _output_error;
-        outBuff.dst = compressedBuffer;
-        outBuff.size = compressedBufferSize;
-        outBuff.pos = 0;
-        inBuff.src = CNBuffer;
-        inBuff.size = CNBufferSize;
-        inBuff.pos = 0;
-        CHECK_Z( ZSTD_compressStream(zc, &outBuff, &inBuff) );
-        if (inBuff.pos != inBuff.size) goto _output_error;  /* entire input should be consumed */
-        { size_t const r = ZSTD_endStream(zc, &outBuff);
-          if (r != 0) goto _output_error; }  /* error, or some data not flushed */
-        cSize = outBuff.pos;
-        ZSTD_freeCDict(cdict);
-        DISPLAYLEVEL(3, "OK (%u bytes : %.2f%%)\n", (unsigned)cSize, (double)cSize/CNBufferSize*100);
-    }
-
-    DISPLAYLEVEL(3, "test%3i : try retrieving dictID from frame : ", testNb++);
-    {   U32 const did = ZSTD_getDictID_fromFrame(compressedBuffer, cSize);
-        if (did != 0) goto _output_error;
-    }
-    DISPLAYLEVEL(3, "OK (not detected) \n");
-
-    DISPLAYLEVEL(3, "test%3i : decompress without dictionary : ", testNb++);
-    {   size_t const r = ZSTD_decompress(decodedBuffer, CNBufferSize, compressedBuffer, cSize);
-        if (!ZSTD_isError(r)) goto _output_error;  /* must fail : dictionary not used */
-        DISPLAYLEVEL(3, "OK (%s)\n", ZSTD_getErrorName(r));
-    }
-
-    DISPLAYLEVEL(3, "test%3i : compress with ZSTD_CCtx_refPrefix : ", testNb++);
-    CHECK_Z( ZSTD_CCtx_refPrefix(zc, dictionary.start, dictionary.filled) );
-    outBuff.dst = compressedBuffer;
-    outBuff.size = compressedBufferSize;
-    outBuff.pos = 0;
-    inBuff.src = CNBuffer;
-    inBuff.size = CNBufferSize;
-    inBuff.pos = 0;
-    CHECK_Z( ZSTD_compressStream2(zc, &outBuff, &inBuff, ZSTD_e_end) );
-    if (inBuff.pos != inBuff.size) goto _output_error;  /* entire input should be consumed */
-    cSize = outBuff.pos;
-    DISPLAYLEVEL(3, "OK (%u bytes : %.2f%%)\n", (unsigned)cSize, (double)cSize/CNBufferSize*100);
-
-    DISPLAYLEVEL(3, "test%3i : decompress with ZSTD_DCtx_refPrefix : ", testNb++);
-    CHECK_Z( ZSTD_DCtx_refPrefix(zd, dictionary.start, dictionary.filled) );
-    outBuff.dst = decodedBuffer;
-    outBuff.size = CNBufferSize;
-    outBuff.pos = 0;
-    inBuff.src = compressedBuffer;
-    inBuff.size = cSize;
-    inBuff.pos = 0;
-    CHECK_Z( ZSTD_decompressStream(zd, &outBuff, &inBuff) );
-    if (inBuff.pos != inBuff.size) goto _output_error;  /* entire input should be consumed */
-    if (outBuff.pos != CNBufferSize) goto _output_error;  /* must regenerate whole input */
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : decompress without dictionary (should fail): ", testNb++);
-    {   size_t const r = ZSTD_decompress(decodedBuffer, CNBufferSize, compressedBuffer, cSize);
-        if (!ZSTD_isError(r)) goto _output_error;  /* must fail : dictionary not used */
-        DISPLAYLEVEL(3, "OK (%s)\n", ZSTD_getErrorName(r));
-    }
-
-    DISPLAYLEVEL(3, "test%3i : compress again with ZSTD_compressStream2 : ", testNb++);
-    outBuff.dst = compressedBuffer;
-    outBuff.size = compressedBufferSize;
-    outBuff.pos = 0;
-    inBuff.src = CNBuffer;
-    inBuff.size = CNBufferSize;
-    inBuff.pos = 0;
-    CHECK_Z( ZSTD_compressStream2(zc, &outBuff, &inBuff, ZSTD_e_end) );
-    if (inBuff.pos != inBuff.size) goto _output_error;  /* entire input should be consumed */
-    cSize = outBuff.pos;
-    DISPLAYLEVEL(3, "OK (%u bytes : %.2f%%)\n", (unsigned)cSize, (double)cSize/CNBufferSize*100);
-
-    DISPLAYLEVEL(3, "test%3i : decompress without dictionary (should work): ", testNb++);
-    CHECK_Z( ZSTD_decompress(decodedBuffer, CNBufferSize, compressedBuffer, cSize) );
-    DISPLAYLEVEL(3, "OK \n");
-
-    /* Empty srcSize */
-    DISPLAYLEVEL(3, "test%3i : ZSTD_initCStream_advanced with pledgedSrcSize=0 and dict : ", testNb++);
-    {   ZSTD_parameters params = ZSTD_getParams(5, 0, 0);
-        params.fParams.contentSizeFlag = 1;
-        CHECK_Z( ZSTD_initCStream_advanced(zc, dictionary.start, dictionary.filled, params, 0 /* pledgedSrcSize==0 means "empty" when params.fParams.contentSizeFlag is set */) );
-    } /* cstream advanced shall write content size = 0 */
-    outBuff.dst = compressedBuffer;
-    outBuff.size = compressedBufferSize;
-    outBuff.pos = 0;
-    inBuff.src = CNBuffer;
-    inBuff.size = 0;
-    inBuff.pos = 0;
-    CHECK_Z( ZSTD_compressStream(zc, &outBuff, &inBuff) );
-    if (ZSTD_endStream(zc, &outBuff) != 0) goto _output_error;
-    cSize = outBuff.pos;
-    if (ZSTD_findDecompressedSize(compressedBuffer, cSize) != 0) goto _output_error;
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : pledgedSrcSize == 0 behaves properly : ", testNb++);
-    {   ZSTD_parameters params = ZSTD_getParams(5, 0, 0);
-        params.fParams.contentSizeFlag = 1;
-        CHECK_Z( ZSTD_initCStream_advanced(zc, NULL, 0, params, 0) );
-    } /* cstream advanced shall write content size = 0 */
-    inBuff.src = CNBuffer;
-    inBuff.size = 0;
-    inBuff.pos = 0;
-    outBuff.dst = compressedBuffer;
-    outBuff.size = compressedBufferSize;
-    outBuff.pos = 0;
-    CHECK_Z( ZSTD_compressStream(zc, &outBuff, &inBuff) );
-    if (ZSTD_endStream(zc, &outBuff) != 0) goto _output_error;
-    cSize = outBuff.pos;
-    if (ZSTD_findDecompressedSize(compressedBuffer, cSize) != 0) goto _output_error;
-
-    ZSTD_resetCStream(zc, 0); /* resetCStream should treat 0 as unknown */
-    outBuff.dst = compressedBuffer;
-    outBuff.size = compressedBufferSize;
-    outBuff.pos = 0;
-    inBuff.src = CNBuffer;
-    inBuff.size = 0;
-    inBuff.pos = 0;
-    CHECK_Z( ZSTD_compressStream(zc, &outBuff, &inBuff) );
-    if (ZSTD_endStream(zc, &outBuff) != 0) goto _output_error;
-    cSize = outBuff.pos;
-    if (ZSTD_findDecompressedSize(compressedBuffer, cSize) != ZSTD_CONTENTSIZE_UNKNOWN) goto _output_error;
-    DISPLAYLEVEL(3, "OK \n");
-
-    /* Basic multithreading compression test */
-    DISPLAYLEVEL(3, "test%3i : compress %u bytes with multiple threads : ", testNb++, COMPRESSIBLE_NOISE_LENGTH);
-    {   ZSTD_parameters const params = ZSTD_getParams(1, 0, 0);
-        int jobSize;
-        CHECK_Z( ZSTDMT_getMTCtxParameter(mtctx, ZSTDMT_p_jobSize, &jobSize));
-        CHECK(jobSize != 0, "job size non-zero");
-        CHECK_Z( ZSTDMT_initCStream_advanced(mtctx, CNBuffer, dictSize, params, CNBufferSize) );
-        CHECK_Z( ZSTDMT_getMTCtxParameter(mtctx, ZSTDMT_p_jobSize, &jobSize));
-        CHECK(jobSize != 0, "job size non-zero");
-    }
-    outBuff.dst = compressedBuffer;
-    outBuff.size = compressedBufferSize;
-    outBuff.pos = 0;
-    inBuff.src = CNBuffer;
-    inBuff.size = CNBufferSize;
-    inBuff.pos = 0;
-    {   size_t const compressResult = ZSTDMT_compressStream_generic(mtctx, &outBuff, &inBuff, ZSTD_e_end);
-        if (compressResult != 0) goto _output_error;  /* compression must be completed in a single round */
-    }
-    if (inBuff.pos != inBuff.size) goto _output_error;   /* entire input should be consumed */
-    {   size_t const compressedSize = ZSTD_findFrameCompressedSize(compressedBuffer, outBuff.pos);
-        if (compressedSize != outBuff.pos) goto _output_error;  /* must be a full valid frame */
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    /* Complex multithreading + dictionary test */
-    {   U32 const nbWorkers = 2;
-        size_t const jobSize = 4 * 1 MB;
-        size_t const srcSize = jobSize * nbWorkers;  /* we want each job to have predictable size */
-        size_t const segLength = 2 KB;
-        size_t const offset = 600 KB;   /* must be larger than window defined in cdict */
-        size_t const start = jobSize + (offset-1);
-        const BYTE* const srcToCopy = (const BYTE*)CNBuffer + start;
-        BYTE* const dst = (BYTE*)CNBuffer + start - offset;
-        DISPLAYLEVEL(3, "test%3i : compress %u bytes with multiple threads + dictionary : ", testNb++, (unsigned)srcSize);
-        CHECK_Z( ZSTD_CCtx_setParameter(zc, ZSTD_c_compressionLevel, 3) );
-        CHECK_Z( ZSTD_CCtx_setParameter(zc, ZSTD_c_nbWorkers, nbWorkers) );
-        CHECK_Z( ZSTD_CCtx_setParameter(zc, ZSTD_c_jobSize, jobSize) );
-        assert(start > offset);
-        assert(start + segLength < COMPRESSIBLE_NOISE_LENGTH);
-        memcpy(dst, srcToCopy, segLength);   /* create a long repetition at long distance for job 2 */
-        outBuff.dst = compressedBuffer;
-        outBuff.size = compressedBufferSize;
-        outBuff.pos = 0;
-        inBuff.src = CNBuffer;
-        inBuff.size = srcSize; assert(srcSize < COMPRESSIBLE_NOISE_LENGTH);
-        inBuff.pos = 0;
-    }
-    {   ZSTD_compressionParameters const cParams = ZSTD_getCParams(1, 4 KB, dictionary.filled);   /* intentionally lies on estimatedSrcSize, to push cdict into targeting a small window size */
-        ZSTD_CDict* const cdict = ZSTD_createCDict_advanced(dictionary.start, dictionary.filled, ZSTD_dlm_byRef, ZSTD_dct_fullDict, cParams, ZSTD_defaultCMem);
-        DISPLAYLEVEL(5, "cParams.windowLog = %u : ", cParams.windowLog);
-        CHECK_Z( ZSTD_CCtx_refCDict(zc, cdict) );
-        CHECK_Z( ZSTD_compressStream2(zc, &outBuff, &inBuff, ZSTD_e_end) );
-        CHECK_Z( ZSTD_CCtx_refCDict(zc, NULL) );  /* do not keep a reference to cdict, as its lifetime ends */
-        ZSTD_freeCDict(cdict);
-    }
-    if (inBuff.pos != inBuff.size) goto _output_error;   /* entire input should be consumed */
-    cSize = outBuff.pos;
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : decompress large frame created from multiple threads + dictionary : ", testNb++);
-    {   ZSTD_DStream* const dstream = ZSTD_createDCtx();
-        ZSTD_frameHeader zfh;
-        ZSTD_getFrameHeader(&zfh, compressedBuffer, cSize);
-        DISPLAYLEVEL(5, "frame windowsize = %u : ", (unsigned)zfh.windowSize);
-        outBuff.dst = decodedBuffer;
-        outBuff.size = CNBufferSize;
-        outBuff.pos = 0;
-        inBuff.src = compressedBuffer;
-        inBuff.pos = 0;
-        CHECK_Z( ZSTD_initDStream_usingDict(dstream, dictionary.start, dictionary.filled) );
-        inBuff.size = 1;  /* avoid shortcut to single-pass mode */
-        CHECK_Z( ZSTD_decompressStream(dstream, &outBuff, &inBuff) );
-        inBuff.size = cSize;
-        CHECK_Z( ZSTD_decompressStream(dstream, &outBuff, &inBuff) );
-        if (inBuff.pos != inBuff.size) goto _output_error;   /* entire input should be consumed */
-        ZSTD_freeDStream(dstream);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : check dictionary FSE tables can represent every code : ", testNb++);
-    {   unsigned const kMaxWindowLog = 24;
-        unsigned value;
-        ZSTD_compressionParameters cParams = ZSTD_getCParams(3, 1U << kMaxWindowLog, 1024);
-        ZSTD_CDict* cdict;
-        ZSTD_DDict* ddict;
-        SEQ_stream seq = SEQ_initStream(0x87654321);
-        SEQ_gen_type type;
-        XXH64_state_t xxh;
-
-        XXH64_reset(&xxh, 0);
-        cParams.windowLog = kMaxWindowLog;
-        cdict = ZSTD_createCDict_advanced(dictionary.start, dictionary.filled, ZSTD_dlm_byRef, ZSTD_dct_fullDict, cParams, ZSTD_defaultCMem);
-        ddict = ZSTD_createDDict(dictionary.start, dictionary.filled);
-
-        if (!cdict || !ddict) goto _output_error;
-
-        ZSTD_CCtx_reset(zc, ZSTD_reset_session_only);
-        ZSTD_resetDStream(zd);
-        CHECK_Z(ZSTD_CCtx_refCDict(zc, cdict));
-        CHECK_Z(ZSTD_initDStream_usingDDict(zd, ddict));
-        CHECK_Z(ZSTD_DCtx_setParameter(zd, ZSTD_d_windowLogMax, kMaxWindowLog));
-        /* Test all values < 300 */
-        for (value = 0; value < 300; ++value) {
-            for (type = (SEQ_gen_type)0; type < SEQ_gen_max; ++type) {
-                CHECK_Z(SEQ_generateRoundTrip(zc, zd, &xxh, &seq, type, value));
-            }
-        }
-        /* Test values 2^8 to 2^17 */
-        for (value = (1 << 8); value < (1 << 17); value <<= 1) {
-            for (type = (SEQ_gen_type)0; type < SEQ_gen_max; ++type) {
-                CHECK_Z(SEQ_generateRoundTrip(zc, zd, &xxh, &seq, type, value));
-                CHECK_Z(SEQ_generateRoundTrip(zc, zd, &xxh, &seq, type, value + (value >> 2)));
-            }
-        }
-        /* Test offset values up to the max window log */
-        for (value = 8; value <= kMaxWindowLog; ++value) {
-            CHECK_Z(SEQ_generateRoundTrip(zc, zd, &xxh, &seq, SEQ_gen_of, (1U << value) - 1));
-        }
-
-        CHECK_Z(SEQ_roundTrip(zc, zd, &xxh, NULL, 0, ZSTD_e_end));
-        CHECK(SEQ_digest(&seq) != XXH64_digest(&xxh), "SEQ XXH64 does not match");
-
-        ZSTD_freeCDict(cdict);
-        ZSTD_freeDDict(ddict);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : ZSTD_initCStream_srcSize sets requestedParams : ", testNb++);
-    {   int level;
-        CHECK_Z(ZSTD_initCStream_srcSize(zc, 11, ZSTD_CONTENTSIZE_UNKNOWN));
-        CHECK_Z(ZSTD_CCtx_getParameter(zc, ZSTD_c_compressionLevel, &level));
-        CHECK(level != 11, "Compression level does not match");
-        ZSTD_resetCStream(zc, ZSTD_CONTENTSIZE_UNKNOWN);
-        CHECK_Z(ZSTD_CCtx_getParameter(zc, ZSTD_c_compressionLevel, &level));
-        CHECK(level != 11, "Compression level does not match");
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : ZSTD_initCStream_advanced sets requestedParams : ", testNb++);
-    {   ZSTD_parameters const params = ZSTD_getParams(9, 0, 0);
-        CHECK_Z(ZSTD_initCStream_advanced(zc, NULL, 0, params, ZSTD_CONTENTSIZE_UNKNOWN));
-        CHECK(badParameters(zc, params), "Compression parameters do not match");
-        ZSTD_resetCStream(zc, ZSTD_CONTENTSIZE_UNKNOWN);
-        CHECK(badParameters(zc, params), "Compression parameters do not match");
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : ZSTD_c_srcSizeHint bounds : ", testNb++);
-    ZSTD_CCtx_reset(zc, ZSTD_reset_session_and_parameters);
-    CHECK_Z(ZSTD_CCtx_setParameter(zc, ZSTD_c_srcSizeHint, INT_MAX));
-    {   int srcSizeHint;
-        CHECK_Z(ZSTD_CCtx_getParameter(zc, ZSTD_c_srcSizeHint, &srcSizeHint));
-        CHECK(!(srcSizeHint == INT_MAX), "srcSizeHint doesn't match");
-    }
-    CHECK(!ZSTD_isError(ZSTD_CCtx_setParameter(zc, ZSTD_c_srcSizeHint, -1)), "Out of range doesn't error");
-    DISPLAYLEVEL(3, "OK \n");
-
-    /* Overlen overwriting window data bug */
-    DISPLAYLEVEL(3, "test%3i : wildcopy doesn't overwrite potential match data : ", testNb++);
-    {   /* This test has a window size of 1024 bytes and consists of 3 blocks:
-            1. 'a' repeated 517 times
-            2. 'b' repeated 516 times
-            3. a compressed block with no literals and 3 sequence commands:
-                litlength = 0, offset = 24, match length = 24
-                litlength = 0, offset = 24, match length = 3 (this one creates an overlength write of length 2*WILDCOPY_OVERLENGTH - 3)
-                litlength = 0, offset = 1021, match length = 3 (this one will try to read from overwritten data if the buffer is too small) */
-
-        const char* testCase =
-            "\x28\xB5\x2F\xFD\x04\x00\x4C\x00\x00\x10\x61\x61\x01\x00\x00\x2A"
-            "\x80\x05\x44\x00\x00\x08\x62\x01\x00\x00\x2A\x20\x04\x5D\x00\x00"
-            "\x00\x03\x40\x00\x00\x64\x60\x27\xB0\xE0\x0C\x67\x62\xCE\xE0";
-        ZSTD_DStream* const zds = ZSTD_createDStream();
-        if (zds==NULL) goto _output_error;
-
-        CHECK_Z( ZSTD_initDStream(zds) );
-        inBuff.src = testCase;
-        inBuff.size = 47;
-        inBuff.pos = 0;
-        outBuff.dst = decodedBuffer;
-        outBuff.size = CNBufferSize;
-        outBuff.pos = 0;
-
-        while (inBuff.pos < inBuff.size) {
-            CHECK_Z( ZSTD_decompressStream(zds, &outBuff, &inBuff) );
-        }
-
-        ZSTD_freeDStream(zds);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    /* Small Sequence Section bug */
-    DISPLAYLEVEL(3, "test%3i : decompress blocks with small sequences section : ", testNb++);
-    {   /* This test consists of 3 blocks. Each block has one sequence.
-            The sequence has literal length of 10, match length of 10 and offset of 10.
-            The sequence value and compression mode for the blocks are following:
-            The order of values are ll, ml, of.
-              - First block  : (10, 7, 13) (rle, rle, rle)
-                 - size of sequences section: 6 bytes (1 byte for nbSeq, 1 byte for encoding mode, 3 bytes for rle, 1 byte bitstream)
-              - Second block : (10, 7, 1) (repeat, repeat, rle)
-                 - size of sequences section: 4 bytes (1 byte for nbSeq, 1 byte for encoding mode, 1 bytes for rle, 1 byte bitstream)
-              - Third block  : (10, 7, 1) (repeat, repeat, repeat)
-                 - size of sequences section: 3 bytes (1 byte for nbSeq, 1 byte for encoding mode, 1 byte bitstream) */
-
-        unsigned char compressed[] = {
-            0x28, 0xb5, 0x2f, 0xfd, 0x24, 0x3c, 0x35, 0x01, 0x00, 0xf0, 0x85, 0x08,
-            0xc2, 0xc4, 0x70, 0xcf, 0xd7, 0xc0, 0x96, 0x7e, 0x4c, 0x6b, 0xa9, 0x8b,
-            0xbc, 0xc5, 0xb6, 0xd9, 0x7f, 0x4c, 0xf1, 0x05, 0xa6, 0x54, 0xef, 0xac,
-            0x69, 0x94, 0x89, 0x1c, 0x03, 0x44, 0x0a, 0x07, 0x00, 0xb4, 0x04, 0x80,
-            0x40, 0x0a, 0xa4
-        };
-        unsigned int compressedSize = 51;
-        unsigned char decompressed[] = {
-            0x85, 0x08, 0xc2, 0xc4, 0x70, 0xcf, 0xd7, 0xc0, 0x96, 0x7e, 0x85, 0x08,
-            0xc2, 0xc4, 0x70, 0xcf, 0xd7, 0xc0, 0x96, 0x7e, 0x4c, 0x6b, 0xa9, 0x8b,
-            0xbc, 0xc5, 0xb6, 0xd9, 0x7f, 0x4c, 0x4c, 0x6b, 0xa9, 0x8b, 0xbc, 0xc5,
-            0xb6, 0xd9, 0x7f, 0x4c, 0xf1, 0x05, 0xa6, 0x54, 0xef, 0xac, 0x69, 0x94,
-            0x89, 0x1c, 0xf1, 0x05, 0xa6, 0x54, 0xef, 0xac, 0x69, 0x94, 0x89, 0x1c
-        };
-        unsigned int decompressedSize = 60;
-
-        ZSTD_DStream* const zds = ZSTD_createDStream();
-        if (zds==NULL) goto _output_error;
-
-        CHECK_Z( ZSTD_initDStream(zds) );
-        inBuff.src = compressed;
-        inBuff.size = compressedSize;
-        inBuff.pos = 0;
-        outBuff.dst = decodedBuffer;
-        outBuff.size = CNBufferSize;
-        outBuff.pos = 0;
-
-        CHECK(ZSTD_decompressStream(zds, &outBuff, &inBuff) != 0,
-              "Decompress did not reach the end of frame");
-        CHECK(inBuff.pos != inBuff.size, "Decompress did not fully consume input");
-        CHECK(outBuff.pos != decompressedSize, "Decompressed size does not match");
-        CHECK(memcmp(outBuff.dst, decompressed, decompressedSize) != 0,
-              "Decompressed data does not match");
-
-        ZSTD_freeDStream(zds);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : dictionary + uncompressible block + reusing tables checks offset table validity: ", testNb++);
-    {   ZSTD_CDict* const cdict = ZSTD_createCDict_advanced(
-            dictionary.start, dictionary.filled,
-            ZSTD_dlm_byRef, ZSTD_dct_fullDict,
-            ZSTD_getCParams(3, 0, dictionary.filled),
-            ZSTD_defaultCMem);
-        const size_t inbufsize = 2 * 128 * 1024; /* 2 blocks */
-        const size_t outbufsize = ZSTD_compressBound(inbufsize);
-        size_t inbufpos = 0;
-        size_t cursegmentlen;
-        BYTE *inbuf = (BYTE *)malloc(inbufsize);
-        BYTE *outbuf = (BYTE *)malloc(outbufsize);
-        BYTE *checkbuf = (BYTE *)malloc(inbufsize);
-        size_t ret;
-
-        CHECK(cdict == NULL, "failed to alloc cdict");
-        CHECK(inbuf == NULL, "failed to alloc input buffer");
-
-        /* first block is uncompressible */
-        cursegmentlen = 128 * 1024;
-        RDG_genBuffer(inbuf + inbufpos, cursegmentlen, 0., 0., seed);
-        inbufpos += cursegmentlen;
-
-        /* second block is compressible */
-        cursegmentlen = 128 * 1024 - 256;
-        RDG_genBuffer(inbuf + inbufpos, cursegmentlen, 0.05, 0., seed);
-        inbufpos += cursegmentlen;
-
-        /* and includes a very long backref */
-        cursegmentlen = 128;
-        memcpy(inbuf + inbufpos, dictionary.start + 256, cursegmentlen);
-        inbufpos += cursegmentlen;
-
-        /* and includes a very long backref */
-        cursegmentlen = 128;
-        memcpy(inbuf + inbufpos, dictionary.start + 128, cursegmentlen);
-        inbufpos += cursegmentlen;
-
-        ret = ZSTD_compress_usingCDict(zc, outbuf, outbufsize, inbuf, inbufpos, cdict);
-        CHECK_Z(ret);
-
-        ret = ZSTD_decompress_usingDict(zd, checkbuf, inbufsize, outbuf, ret, dictionary.start, dictionary.filled);
-        CHECK_Z(ret);
-
-        CHECK(memcmp(inbuf, checkbuf, inbufpos), "start and finish buffers don't match");
-
-        ZSTD_freeCDict(cdict);
-        free(inbuf);
-        free(outbuf);
-        free(checkbuf);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-    DISPLAYLEVEL(3, "test%3i : dictionary + small blocks + reusing tables checks offset table validity: ", testNb++);
-    {   ZSTD_CDict* const cdict = ZSTD_createCDict_advanced(
-            dictionary.start, dictionary.filled,
-            ZSTD_dlm_byRef, ZSTD_dct_fullDict,
-            ZSTD_getCParams(3, 0, dictionary.filled),
-            ZSTD_defaultCMem);
-        ZSTD_outBuffer out = {compressedBuffer, compressedBufferSize, 0};
-        int remainingInput = 256 * 1024;
-        int offset;
-
-        CHECK_Z(ZSTD_CCtx_reset(zc, ZSTD_reset_session_and_parameters));
-        CHECK_Z(ZSTD_CCtx_refCDict(zc, cdict));
-        CHECK_Z(ZSTD_CCtx_setParameter(zc, ZSTD_c_checksumFlag, 1));
-        /* Write a bunch of 6 byte blocks */
-        while (remainingInput > 0) {
-          char testBuffer[6] = "\xAA\xAA\xAA\xAA\xAA\xAA";
-          const size_t kSmallBlockSize = sizeof(testBuffer);
-          ZSTD_inBuffer in = {testBuffer, kSmallBlockSize, 0};
-
-          CHECK_Z(ZSTD_compressStream2(zc, &out, &in, ZSTD_e_flush));
-          CHECK(in.pos != in.size, "input not fully consumed");
-          remainingInput -= kSmallBlockSize;
-        }
-        /* Write several very long offset matches into the dictionary */
-        for (offset = 1024; offset >= 0; offset -= 128) {
-          ZSTD_inBuffer in = {dictionary.start + offset, 128, 0};
-          ZSTD_EndDirective flush = offset > 0 ? ZSTD_e_continue : ZSTD_e_end;
-          CHECK_Z(ZSTD_compressStream2(zc, &out, &in, flush));
-          CHECK(in.pos != in.size, "input not fully consumed");
-        }
-        /* Ensure decompression works */
-        CHECK_Z(ZSTD_decompress_usingDict(zd, decodedBuffer, CNBufferSize, out.dst, out.pos, dictionary.start, dictionary.filled));
-
-        ZSTD_freeCDict(cdict);
-    }
-    DISPLAYLEVEL(3, "OK \n");
-
-_end:
-    FUZ_freeDictionary(dictionary);
-    ZSTD_freeCStream(zc);
-    ZSTD_freeDStream(zd);
-    ZSTDMT_freeCCtx(mtctx);
-    free(CNBuffer);
-    free(compressedBuffer);
-    free(decodedBuffer);
-    return testResult;
-
-_output_error:
-    testResult = 1;
-    DISPLAY("Error detected in Unit tests ! \n");
-    goto _end;
-}
-
-
-/* ======   Fuzzer tests   ====== */
-
-static size_t findDiff(const void* buf1, const void* buf2, size_t max)
-{
-    const BYTE* b1 = (const BYTE*)buf1;
-    const BYTE* b2 = (const BYTE*)buf2;
-    size_t u;
-    for (u=0; u<max; u++) {
-        if (b1[u] != b2[u]) break;
-    }
-    if (u==max) {
-        DISPLAY("=> No difference detected within %u bytes \n", (unsigned)max);
-        return u;
-    }
-    DISPLAY("Error at position %u / %u \n", (unsigned)u, (unsigned)max);
-    if (u>=3)
-        DISPLAY(" %02X %02X %02X ",
-                b1[u-3], b1[u-2], b1[u-1]);
-    DISPLAY(" :%02X:  %02X %02X %02X %02X %02X \n",
-            b1[u], b1[u+1], b1[u+2], b1[u+3], b1[u+4], b1[u+5]);
-    if (u>=3)
-        DISPLAY(" %02X %02X %02X ",
-                b2[u-3], b2[u-2], b2[u-1]);
-    DISPLAY(" :%02X:  %02X %02X %02X %02X %02X \n",
-            b2[u], b2[u+1], b2[u+2], b2[u+3], b2[u+4], b2[u+5]);
-    return u;
-}
-
-static size_t FUZ_rLogLength(U32* seed, U32 logLength)
-{
-    size_t const lengthMask = ((size_t)1 << logLength) - 1;
-    return (lengthMask+1) + (FUZ_rand(seed) & lengthMask);
-}
-
-static size_t FUZ_randomLength(U32* seed, U32 maxLog)
-{
-    U32 const logLength = FUZ_rand(seed) % maxLog;
-    return FUZ_rLogLength(seed, logLength);
-}
-
-/* Return value in range minVal <= v <= maxVal */
-static U32 FUZ_randomClampedLength(U32* seed, U32 minVal, U32 maxVal)
-{
-    U32 const mod = maxVal < minVal ? 1 : (maxVal + 1) - minVal;
-    return (U32)((FUZ_rand(seed) % mod) + minVal);
-}
-
-static int fuzzerTests(U32 seed, unsigned nbTests, unsigned startTest, double compressibility, int bigTests)
-{
-    U32 const maxSrcLog = bigTests ? 24 : 22;
-    static const U32 maxSampleLog = 19;
-    size_t const srcBufferSize = (size_t)1<<maxSrcLog;
-    BYTE* cNoiseBuffer[5];
-    size_t const copyBufferSize = srcBufferSize + (1<<maxSampleLog);
-    BYTE*  const copyBuffer = (BYTE*)malloc (copyBufferSize);
-    size_t const cBufferSize = ZSTD_compressBound(srcBufferSize);
-    BYTE*  const cBuffer = (BYTE*)malloc (cBufferSize);
-    size_t const dstBufferSize = srcBufferSize;
-    BYTE*  const dstBuffer = (BYTE*)malloc (dstBufferSize);
-    U32 result = 0;
-    unsigned testNb = 0;
-    U32 coreSeed = seed;
-    ZSTD_CStream* zc = ZSTD_createCStream();   /* will be re-created sometimes */
-    ZSTD_DStream* zd = ZSTD_createDStream();   /* will be re-created sometimes */
-    ZSTD_DStream* const zd_noise = ZSTD_createDStream();
-    UTIL_time_t const startClock = UTIL_getTime();
-    const BYTE* dict = NULL;  /* can keep same dict on 2 consecutive tests */
-    size_t dictSize = 0;
-    U32 oldTestLog = 0;
-    U32 const cLevelMax = bigTests ? (U32)ZSTD_maxCLevel() : g_cLevelMax_smallTests;
-
-    /* allocations */
-    cNoiseBuffer[0] = (BYTE*)malloc (srcBufferSize);
-    cNoiseBuffer[1] = (BYTE*)malloc (srcBufferSize);
-    cNoiseBuffer[2] = (BYTE*)malloc (srcBufferSize);
-    cNoiseBuffer[3] = (BYTE*)malloc (srcBufferSize);
-    cNoiseBuffer[4] = (BYTE*)malloc (srcBufferSize);
-    CHECK (!cNoiseBuffer[0] || !cNoiseBuffer[1] || !cNoiseBuffer[2] || !cNoiseBuffer[3] || !cNoiseBuffer[4] ||
-           !copyBuffer || !dstBuffer || !cBuffer || !zc || !zd || !zd_noise ,
-           "Not enough memory, fuzzer tests cancelled");
-
-    /* Create initial samples */
-    RDG_genBuffer(cNoiseBuffer[0], srcBufferSize, 0.00, 0., coreSeed);    /* pure noise */
-    RDG_genBuffer(cNoiseBuffer[1], srcBufferSize, 0.05, 0., coreSeed);    /* barely compressible */
-    RDG_genBuffer(cNoiseBuffer[2], srcBufferSize, compressibility, 0., coreSeed);
-    RDG_genBuffer(cNoiseBuffer[3], srcBufferSize, 0.95, 0., coreSeed);    /* highly compressible */
-    RDG_genBuffer(cNoiseBuffer[4], srcBufferSize, 1.00, 0., coreSeed);    /* sparse content */
-    memset(copyBuffer, 0x65, copyBufferSize);                             /* make copyBuffer considered initialized */
-    ZSTD_initDStream_usingDict(zd, NULL, 0);  /* ensure at least one init */
-
-    /* catch up testNb */
-    for (testNb=1; testNb < startTest; testNb++)
-        FUZ_rand(&coreSeed);
-
-    /* test loop */
-    for ( ; (testNb <= nbTests) || (UTIL_clockSpanMicro(startClock) < g_clockTime) ; testNb++ ) {
-        U32 lseed;
-        const BYTE* srcBuffer;
-        size_t totalTestSize, totalGenSize, cSize;
-        XXH64_state_t xxhState;
-        U64 crcOrig;
-        U32 resetAllowed = 1;
-        size_t maxTestSize;
-
-        /* init */
-        FUZ_rand(&coreSeed);
-        lseed = coreSeed ^ prime32;
-        if (nbTests >= testNb) {
-            DISPLAYUPDATE(2, "\r%6u/%6u    ", testNb, nbTests);
-        } else {
-            DISPLAYUPDATE(2, "\r%6u        ", testNb);
-        }
-
-        /* states full reset (deliberately not synchronized) */
-        /* some issues can only happen when reusing states */
-        if ((FUZ_rand(&lseed) & 0xFF) == 131) {
-            ZSTD_freeCStream(zc);
-            zc = ZSTD_createCStream();
-            CHECK(zc==NULL, "ZSTD_createCStream : allocation error");
-            resetAllowed=0;
-        }
-        if ((FUZ_rand(&lseed) & 0xFF) == 132) {
-            ZSTD_freeDStream(zd);
-            zd = ZSTD_createDStream();
-            CHECK(zd==NULL, "ZSTD_createDStream : allocation error");
-            CHECK_Z( ZSTD_initDStream_usingDict(zd, NULL, 0) );  /* ensure at least one init */
-        }
-
-        /* srcBuffer selection [0-4] */
-        {   U32 buffNb = FUZ_rand(&lseed) & 0x7F;
-            if (buffNb & 7) buffNb=2;   /* most common : compressible (P) */
-            else {
-                buffNb >>= 3;
-                if (buffNb & 7) {
-                    const U32 tnb[2] = { 1, 3 };   /* barely/highly compressible */
-                    buffNb = tnb[buffNb >> 3];
-                } else {
-                    const U32 tnb[2] = { 0, 4 };   /* not compressible / sparse */
-                    buffNb = tnb[buffNb >> 3];
-            }   }
-            srcBuffer = cNoiseBuffer[buffNb];
-        }
-
-        /* compression init */
-        if ((FUZ_rand(&lseed)&1) /* at beginning, to keep same nb of rand */
-            && oldTestLog /* at least one test happened */ && resetAllowed) {
-            maxTestSize = FUZ_randomLength(&lseed, oldTestLog+2);
-            maxTestSize = MIN(maxTestSize, srcBufferSize-16);
-            {   U64 const pledgedSrcSize = (FUZ_rand(&lseed) & 3) ? 0 : maxTestSize;
-                CHECK_Z( ZSTD_resetCStream(zc, pledgedSrcSize) );
-            }
-        } else {
-            U32 const testLog = FUZ_rand(&lseed) % maxSrcLog;
-            U32 const dictLog = FUZ_rand(&lseed) % maxSrcLog;
-            U32 const cLevelCandidate = ( FUZ_rand(&lseed) %
-                                (ZSTD_maxCLevel() -
-                                (MAX(testLog, dictLog) / 3)))
-                                 + 1;
-            U32 const cLevel = MIN(cLevelCandidate, cLevelMax);
-            maxTestSize = FUZ_rLogLength(&lseed, testLog);
-            oldTestLog = testLog;
-            /* random dictionary selection */
-            dictSize  = ((FUZ_rand(&lseed)&7)==1) ? FUZ_rLogLength(&lseed, dictLog) : 0;
-            {   size_t const dictStart = FUZ_rand(&lseed) % (srcBufferSize - dictSize);
-                dict = srcBuffer + dictStart;
-            }
-            {   U64 const pledgedSrcSize = (FUZ_rand(&lseed) & 3) ? ZSTD_CONTENTSIZE_UNKNOWN : maxTestSize;
-                ZSTD_parameters params = ZSTD_getParams(cLevel, pledgedSrcSize, dictSize);
-                params.fParams.checksumFlag = FUZ_rand(&lseed) & 1;
-                params.fParams.noDictIDFlag = FUZ_rand(&lseed) & 1;
-                params.fParams.contentSizeFlag = FUZ_rand(&lseed) & 1;
-                CHECK_Z ( ZSTD_initCStream_advanced(zc, dict, dictSize, params, pledgedSrcSize) );
-        }   }
-
-        /* multi-segments compression test */
-        XXH64_reset(&xxhState, 0);
-        {   ZSTD_outBuffer outBuff = { cBuffer, cBufferSize, 0 } ;
-            U32 n;
-            for (n=0, cSize=0, totalTestSize=0 ; totalTestSize < maxTestSize ; n++) {
-                /* compress random chunks into randomly sized dst buffers */
-                {   size_t const randomSrcSize = FUZ_randomLength(&lseed, maxSampleLog);
-                    size_t const srcSize = MIN(maxTestSize-totalTestSize, randomSrcSize);
-                    size_t const srcStart = FUZ_rand(&lseed) % (srcBufferSize - srcSize);
-                    size_t const randomDstSize = FUZ_randomLength(&lseed, maxSampleLog);
-                    size_t const dstBuffSize = MIN(cBufferSize - cSize, randomDstSize);
-                    ZSTD_inBuffer inBuff = { srcBuffer+srcStart, srcSize, 0 };
-                    outBuff.size = outBuff.pos + dstBuffSize;
-
-                    CHECK_Z( ZSTD_compressStream(zc, &outBuff, &inBuff) );
-
-                    XXH64_update(&xxhState, srcBuffer+srcStart, inBuff.pos);
-                    memcpy(copyBuffer+totalTestSize, srcBuffer+srcStart, inBuff.pos);
-                    totalTestSize += inBuff.pos;
-                }
-
-                /* random flush operation, to mess around */
-                if ((FUZ_rand(&lseed) & 15) == 0) {
-                    size_t const randomDstSize = FUZ_randomLength(&lseed, maxSampleLog);
-                    size_t const adjustedDstSize = MIN(cBufferSize - cSize, randomDstSize);
-                    outBuff.size = outBuff.pos + adjustedDstSize;
-                    CHECK_Z( ZSTD_flushStream(zc, &outBuff) );
-            }   }
-
-            /* final frame epilogue */
-            {   size_t remainingToFlush = (size_t)(-1);
-                while (remainingToFlush) {
-                    size_t const randomDstSize = FUZ_randomLength(&lseed, maxSampleLog);
-                    size_t const adjustedDstSize = MIN(cBufferSize - cSize, randomDstSize);
-                    outBuff.size = outBuff.pos + adjustedDstSize;
-                    remainingToFlush = ZSTD_endStream(zc, &outBuff);
-                    CHECK (ZSTD_isError(remainingToFlush), "end error : %s", ZSTD_getErrorName(remainingToFlush));
-            }   }
-            crcOrig = XXH64_digest(&xxhState);
-            cSize = outBuff.pos;
-        }
-
-        /* multi - fragments decompression test */
-        if (!dictSize /* don't reset if dictionary : could be different */ && (FUZ_rand(&lseed) & 1)) {
-            CHECK_Z ( ZSTD_resetDStream(zd) );
-        } else {
-            CHECK_Z ( ZSTD_initDStream_usingDict(zd, dict, dictSize) );
-        }
-        {   size_t decompressionResult = 1;
-            ZSTD_inBuffer  inBuff = { cBuffer, cSize, 0 };
-            ZSTD_outBuffer outBuff= { dstBuffer, dstBufferSize, 0 };
-            for (totalGenSize = 0 ; decompressionResult ; ) {
-                size_t const readCSrcSize = FUZ_randomLength(&lseed, maxSampleLog);
-                size_t const randomDstSize = FUZ_randomLength(&lseed, maxSampleLog);
-                size_t const dstBuffSize = MIN(dstBufferSize - totalGenSize, randomDstSize);
-                inBuff.size = inBuff.pos + readCSrcSize;
-                outBuff.size = outBuff.pos + dstBuffSize;
-                decompressionResult = ZSTD_decompressStream(zd, &outBuff, &inBuff);
-                if (ZSTD_getErrorCode(decompressionResult) == ZSTD_error_checksum_wrong) {
-                    DISPLAY("checksum error : \n");
-                    findDiff(copyBuffer, dstBuffer, totalTestSize);
-                }
-                CHECK( ZSTD_isError(decompressionResult), "decompression error : %s",
-                       ZSTD_getErrorName(decompressionResult) );
-            }
-            CHECK (decompressionResult != 0, "frame not fully decoded");
-            CHECK (outBuff.pos != totalTestSize, "decompressed data : wrong size (%u != %u)",
-                    (unsigned)outBuff.pos, (unsigned)totalTestSize);
-            CHECK (inBuff.pos != cSize, "compressed data should be fully read")
-            {   U64 const crcDest = XXH64(dstBuffer, totalTestSize, 0);
-                if (crcDest!=crcOrig) findDiff(copyBuffer, dstBuffer, totalTestSize);
-                CHECK (crcDest!=crcOrig, "decompressed data corrupted");
-        }   }
-
-        /*=====   noisy/erroneous src decompression test   =====*/
-
-        /* add some noise */
-        {   U32 const nbNoiseChunks = (FUZ_rand(&lseed) & 7) + 2;
-            U32 nn; for (nn=0; nn<nbNoiseChunks; nn++) {
-                size_t const randomNoiseSize = FUZ_randomLength(&lseed, maxSampleLog);
-                size_t const noiseSize  = MIN((cSize/3) , randomNoiseSize);
-                size_t const noiseStart = FUZ_rand(&lseed) % (srcBufferSize - noiseSize);
-                size_t const cStart = FUZ_rand(&lseed) % (cSize - noiseSize);
-                memcpy(cBuffer+cStart, srcBuffer+noiseStart, noiseSize);
-        }   }
-
-        /* try decompression on noisy data */
-        CHECK_Z( ZSTD_initDStream(zd_noise) );   /* note : no dictionary */
-        {   ZSTD_inBuffer  inBuff = { cBuffer, cSize, 0 };
-            ZSTD_outBuffer outBuff= { dstBuffer, dstBufferSize, 0 };
-            while (outBuff.pos < dstBufferSize) {
-                size_t const randomCSrcSize = FUZ_randomLength(&lseed, maxSampleLog);
-                size_t const randomDstSize = FUZ_randomLength(&lseed, maxSampleLog);
-                size_t const adjustedDstSize = MIN(dstBufferSize - outBuff.pos, randomDstSize);
-                size_t const adjustedCSrcSize = MIN(cSize - inBuff.pos, randomCSrcSize);
-                outBuff.size = outBuff.pos + adjustedDstSize;
-                inBuff.size  = inBuff.pos + adjustedCSrcSize;
-                {   size_t const decompressError = ZSTD_decompressStream(zd, &outBuff, &inBuff);
-                    if (ZSTD_isError(decompressError)) break;   /* error correctly detected */
-                    /* No forward progress possible */
-                    if (outBuff.pos < outBuff.size && inBuff.pos == cSize) break;
-    }   }   }   }
-    DISPLAY("\r%u fuzzer tests completed   \n", testNb);
-
-_cleanup:
-    ZSTD_freeCStream(zc);
-    ZSTD_freeDStream(zd);
-    ZSTD_freeDStream(zd_noise);
-    free(cNoiseBuffer[0]);
-    free(cNoiseBuffer[1]);
-    free(cNoiseBuffer[2]);
-    free(cNoiseBuffer[3]);
-    free(cNoiseBuffer[4]);
-    free(copyBuffer);
-    free(cBuffer);
-    free(dstBuffer);
-    return result;
-
-_output_error:
-    result = 1;
-    goto _cleanup;
-}
-
-
-/* fuzzing ZSTDMT_* interface */
-static int fuzzerTests_MT(U32 seed, int nbTests, int startTest,
-                          double compressibility, int bigTests)
-{
-    const U32 maxSrcLog = bigTests ? 24 : 22;
-    static const U32 maxSampleLog = 19;
-    size_t const srcBufferSize = (size_t)1<<maxSrcLog;
-    BYTE* cNoiseBuffer[5];
-    size_t const copyBufferSize= srcBufferSize + (1<<maxSampleLog);
-    BYTE*  const copyBuffer = (BYTE*)malloc (copyBufferSize);
-    size_t const cBufferSize   = ZSTD_compressBound(srcBufferSize);
-    BYTE*  const cBuffer = (BYTE*)malloc (cBufferSize);
-    size_t const dstBufferSize = srcBufferSize;
-    BYTE*  const dstBuffer = (BYTE*)malloc (dstBufferSize);
-    U32 result = 0;
-    int testNb = 0;
-    U32 coreSeed = seed;
-    int nbThreads = 2;
-    ZSTDMT_CCtx* zc = ZSTDMT_createCCtx(nbThreads);   /* will be reset sometimes */
-    ZSTD_DStream* zd = ZSTD_createDStream();   /* will be reset sometimes */
-    ZSTD_DStream* const zd_noise = ZSTD_createDStream();
-    UTIL_time_t const startClock = UTIL_getTime();
-    const BYTE* dict=NULL;   /* can keep same dict on 2 consecutive tests */
-    size_t dictSize = 0;
-    int const cLevelMax = bigTests ? (U32)ZSTD_maxCLevel()-1 : g_cLevelMax_smallTests;
-    U32 const nbThreadsMax = bigTests ? 4 : 2;
-
-    /* allocations */
-    cNoiseBuffer[0] = (BYTE*)malloc (srcBufferSize);
-    cNoiseBuffer[1] = (BYTE*)malloc (srcBufferSize);
-    cNoiseBuffer[2] = (BYTE*)malloc (srcBufferSize);
-    cNoiseBuffer[3] = (BYTE*)malloc (srcBufferSize);
-    cNoiseBuffer[4] = (BYTE*)malloc (srcBufferSize);
-    CHECK (!cNoiseBuffer[0] || !cNoiseBuffer[1] || !cNoiseBuffer[2] || !cNoiseBuffer[3] || !cNoiseBuffer[4] ||
-           !copyBuffer || !dstBuffer || !cBuffer || !zc || !zd || !zd_noise ,
-           "Not enough memory, fuzzer tests cancelled");
-
-    /* Create initial samples */
-    RDG_genBuffer(cNoiseBuffer[0], srcBufferSize, 0.00, 0., coreSeed);    /* pure noise */
-    RDG_genBuffer(cNoiseBuffer[1], srcBufferSize, 0.05, 0., coreSeed);    /* barely compressible */
-    RDG_genBuffer(cNoiseBuffer[2], srcBufferSize, compressibility, 0., coreSeed);
-    RDG_genBuffer(cNoiseBuffer[3], srcBufferSize, 0.95, 0., coreSeed);    /* highly compressible */
-    RDG_genBuffer(cNoiseBuffer[4], srcBufferSize, 1.00, 0., coreSeed);    /* sparse content */
-    memset(copyBuffer, 0x65, copyBufferSize);                             /* make copyBuffer considered initialized */
-    ZSTD_initDStream_usingDict(zd, NULL, 0);  /* ensure at least one init */
-    DISPLAYLEVEL(6, "Creating initial context with %i threads \n", nbThreads);
-
-    /* catch up testNb */
-    for (testNb=1; testNb < startTest; testNb++)
-        FUZ_rand(&coreSeed);
-
-    /* test loop */
-    for ( ; (testNb <= nbTests) || (UTIL_clockSpanMicro(startClock) < g_clockTime) ; testNb++ ) {
-        U32 lseed;
-        const BYTE* srcBuffer;
-        size_t totalTestSize, totalGenSize, cSize;
-        XXH64_state_t xxhState;
-        U64 crcOrig;
-        size_t maxTestSize;
-
-        FUZ_rand(&coreSeed);
-        if (nbTests >= testNb) {
-            DISPLAYUPDATE(2, "\r%6u/%6u    ", testNb, nbTests);
-        } else {
-            DISPLAYUPDATE(2, "\r%6u         ", testNb);
-        }
-        lseed = coreSeed ^ prime32;
-
-        /* states full reset (deliberately not synchronized) */
-        /* some issues can only happen when reusing states */
-        if ((FUZ_rand(&lseed) & 0xFF) == 131) {
-            nbThreads = (FUZ_rand(&lseed) % nbThreadsMax) + 1;
-            DISPLAYLEVEL(5, "Creating new context with %u threads \n", nbThreads);
-            ZSTDMT_freeCCtx(zc);
-            zc = ZSTDMT_createCCtx(nbThreads);
-            CHECK(zc==NULL, "ZSTDMT_createCCtx allocation error")
-        }
-        if ((FUZ_rand(&lseed) & 0xFF) == 132) {
-            ZSTD_freeDStream(zd);
-            zd = ZSTD_createDStream();
-            CHECK(zd==NULL, "ZSTDMT_createCCtx allocation error")
-            ZSTD_initDStream_usingDict(zd, NULL, 0);  /* ensure at least one init */
-        }
-
-        /* srcBuffer selection [0-4] */
-        {   U32 buffNb = FUZ_rand(&lseed) & 0x7F;
-            if (buffNb & 7) buffNb=2;   /* most common : compressible (P) */
-            else {
-                buffNb >>= 3;
-                if (buffNb & 7) {
-                    const U32 tnb[2] = { 1, 3 };   /* barely/highly compressible */
-                    buffNb = tnb[buffNb >> 3];
-                } else {
-                    const U32 tnb[2] = { 0, 4 };   /* not compressible / sparse */
-                    buffNb = tnb[buffNb >> 3];
-            }   }
-            srcBuffer = cNoiseBuffer[buffNb];
-        }
-
-        /* compression init */
-        {   U32 const testLog = FUZ_rand(&lseed) % maxSrcLog;
-            U32 const dictLog = FUZ_rand(&lseed) % maxSrcLog;
-            int const cLevelCandidate = ( FUZ_rand(&lseed)
-                            % (ZSTD_maxCLevel() - (MAX(testLog, dictLog) / 2)) )
-                            + 1;
-            int const cLevelThreadAdjusted = cLevelCandidate - (nbThreads * 2) + 2;  /* reduce cLevel when multiple threads to reduce memory consumption */
-            int const cLevelMin = MAX(cLevelThreadAdjusted, 1);  /* no negative cLevel yet */
-            int const cLevel = MIN(cLevelMin, cLevelMax);
-            maxTestSize = FUZ_rLogLength(&lseed, testLog);
-
-            if (FUZ_rand(&lseed)&1) {   /* simple init */
-                int const compressionLevel = (FUZ_rand(&lseed) % 5) + 1;
-                DISPLAYLEVEL(5, "Init with compression level = %i \n", compressionLevel);
-                CHECK_Z( ZSTDMT_initCStream(zc, compressionLevel) );
-            } else {   /* advanced init */
-                /* random dictionary selection */
-                dictSize  = ((FUZ_rand(&lseed)&63)==1) ? FUZ_rLogLength(&lseed, dictLog) : 0;
-                {   size_t const dictStart = FUZ_rand(&lseed) % (srcBufferSize - dictSize);
-                    dict = srcBuffer + dictStart;
-                }
-                {   U64 const pledgedSrcSize = (FUZ_rand(&lseed) & 3) ? ZSTD_CONTENTSIZE_UNKNOWN : maxTestSize;
-                    ZSTD_parameters params = ZSTD_getParams(cLevel, pledgedSrcSize, dictSize);
-                    DISPLAYLEVEL(5, "Init with windowLog = %u, pledgedSrcSize = %u, dictSize = %u \n",
-                        params.cParams.windowLog, (unsigned)pledgedSrcSize, (unsigned)dictSize);
-                    params.fParams.checksumFlag = FUZ_rand(&lseed) & 1;
-                    params.fParams.noDictIDFlag = FUZ_rand(&lseed) & 1;
-                    params.fParams.contentSizeFlag = FUZ_rand(&lseed) & 1;
-                    DISPLAYLEVEL(5, "checksumFlag : %u \n", params.fParams.checksumFlag);
-                    CHECK_Z( ZSTDMT_setMTCtxParameter(zc, ZSTDMT_p_overlapLog, FUZ_rand(&lseed) % 12) );
-                    CHECK_Z( ZSTDMT_setMTCtxParameter(zc, ZSTDMT_p_jobSize, FUZ_rand(&lseed) % (2*maxTestSize+1)) );   /* custom job size */
-                    CHECK_Z( ZSTDMT_initCStream_advanced(zc, dict, dictSize, params, pledgedSrcSize) );
-        }   }   }
-
-        /* multi-segments compression test */
-        XXH64_reset(&xxhState, 0);
-        {   ZSTD_outBuffer outBuff = { cBuffer, cBufferSize, 0 } ;
-            U32 n;
-            for (n=0, cSize=0, totalTestSize=0 ; totalTestSize < maxTestSize ; n++) {
-                /* compress random chunks into randomly sized dst buffers */
-                {   size_t const randomSrcSize = FUZ_randomLength(&lseed, maxSampleLog);
-                    size_t const srcSize = MIN (maxTestSize-totalTestSize, randomSrcSize);
-                    size_t const srcStart = FUZ_rand(&lseed) % (srcBufferSize - srcSize);
-                    size_t const randomDstSize = FUZ_randomLength(&lseed, maxSampleLog);
-                    size_t const dstBuffSize = MIN(cBufferSize - cSize, randomDstSize);
-                    ZSTD_inBuffer inBuff = { srcBuffer+srcStart, srcSize, 0 };
-                    outBuff.size = outBuff.pos + dstBuffSize;
-
-                    DISPLAYLEVEL(6, "Sending %u bytes to compress \n", (unsigned)srcSize);
-                    CHECK_Z( ZSTDMT_compressStream(zc, &outBuff, &inBuff) );
-                    DISPLAYLEVEL(6, "%u bytes read by ZSTDMT_compressStream \n", (unsigned)inBuff.pos);
-
-                    XXH64_update(&xxhState, srcBuffer+srcStart, inBuff.pos);
-                    memcpy(copyBuffer+totalTestSize, srcBuffer+srcStart, inBuff.pos);
-                    totalTestSize += inBuff.pos;
-                }
-
-                /* random flush operation, to mess around */
-                if ((FUZ_rand(&lseed) & 15) == 0) {
-                    size_t const randomDstSize = FUZ_randomLength(&lseed, maxSampleLog);
-                    size_t const adjustedDstSize = MIN(cBufferSize - cSize, randomDstSize);
-                    size_t const previousPos = outBuff.pos;
-                    outBuff.size = outBuff.pos + adjustedDstSize;
-                    DISPLAYLEVEL(5, "Flushing into dst buffer of size %u \n", (unsigned)adjustedDstSize);
-                    CHECK_Z( ZSTDMT_flushStream(zc, &outBuff) );
-                    assert(outBuff.pos >= previousPos);
-                    DISPLAYLEVEL(6, "%u bytes flushed by ZSTDMT_flushStream \n", (unsigned)(outBuff.pos-previousPos));
-            }   }
-
-            /* final frame epilogue */
-            {   size_t remainingToFlush = (size_t)(-1);
-                while (remainingToFlush) {
-                    size_t const randomDstSize = FUZ_randomLength(&lseed, maxSampleLog);
-                    size_t const adjustedDstSize = MIN(cBufferSize - cSize, randomDstSize);
-                    size_t const previousPos = outBuff.pos;
-                    outBuff.size = outBuff.pos + adjustedDstSize;
-                    DISPLAYLEVEL(5, "Ending into dst buffer of size %u \n", (unsigned)adjustedDstSize);
-                    remainingToFlush = ZSTDMT_endStream(zc, &outBuff);
-                    CHECK (ZSTD_isError(remainingToFlush), "ZSTDMT_endStream error : %s", ZSTD_getErrorName(remainingToFlush));
-                    assert(outBuff.pos >= previousPos);
-                    DISPLAYLEVEL(6, "%u bytes flushed by ZSTDMT_endStream \n", (unsigned)(outBuff.pos-previousPos));
-                    DISPLAYLEVEL(5, "endStream : remainingToFlush : %u \n", (unsigned)remainingToFlush);
-            }   }
-            crcOrig = XXH64_digest(&xxhState);
-            cSize = outBuff.pos;
-            DISPLAYLEVEL(5, "Frame completed : %u bytes compressed into %u bytes \n",
-                            (unsigned)totalTestSize, (unsigned)cSize);
-        }
-
-        /* multi - fragments decompression test */
-        assert(totalTestSize < dstBufferSize);
-        memset(dstBuffer, 170, totalTestSize);   /* init dest area */
-        if (!dictSize /* don't reset if dictionary : could be different */ && (FUZ_rand(&lseed) & 1)) {
-            CHECK_Z( ZSTD_resetDStream(zd) );
-        } else {
-            CHECK_Z( ZSTD_initDStream_usingDict(zd, dict, dictSize) );
-        }
-        {   size_t decompressionResult = 1;
-            ZSTD_inBuffer  inBuff = { cBuffer, cSize, 0 };
-            ZSTD_outBuffer outBuff= { dstBuffer, dstBufferSize, 0 };
-            for (totalGenSize = 0 ; decompressionResult ; ) {
-                size_t const readCSrcSize = FUZ_randomLength(&lseed, maxSampleLog);
-                size_t const randomDstSize = FUZ_randomLength(&lseed, maxSampleLog);
-                size_t const dstBuffSize = MIN(dstBufferSize - totalGenSize, randomDstSize);
-                inBuff.size = inBuff.pos + readCSrcSize;
-                outBuff.size = outBuff.pos + dstBuffSize;
-                DISPLAYLEVEL(6, "ZSTD_decompressStream input %u bytes into outBuff %u bytes \n",
-                                (unsigned)readCSrcSize, (unsigned)dstBuffSize);
-                decompressionResult = ZSTD_decompressStream(zd, &outBuff, &inBuff);
-                if (ZSTD_isError(decompressionResult)) {
-                    DISPLAY("ZSTD_decompressStream error : %s \n", ZSTD_getErrorName(decompressionResult));
-                    findDiff(copyBuffer, dstBuffer, totalTestSize);
-                }
-                CHECK (ZSTD_isError(decompressionResult), "decompression error : %s", ZSTD_getErrorName(decompressionResult));
-                DISPLAYLEVEL(6, "total ingested (inBuff.pos) = %u and produced (outBuff.pos) = %u \n",
-                                (unsigned)inBuff.pos, (unsigned)outBuff.pos);
-            }
-            CHECK (outBuff.pos != totalTestSize,
-                    "decompressed data : wrong size (%u != %u)",
-                    (unsigned)outBuff.pos, (unsigned)totalTestSize );
-            CHECK (inBuff.pos != cSize,
-                    "compressed data should be fully read (%u != %u)",
-                    (unsigned)inBuff.pos, (unsigned)cSize );
-            {   U64 const crcDest = XXH64(dstBuffer, totalTestSize, 0);
-                if (crcDest!=crcOrig) findDiff(copyBuffer, dstBuffer, totalTestSize);
-                CHECK (crcDest!=crcOrig, "decompressed data corrupted");
-        }   }
-
-        /*=====   noisy/erroneous src decompression test   =====*/
-
-        /* add some noise */
-        {   U32 const nbNoiseChunks = (FUZ_rand(&lseed) & 7) + 2;
-            U32 nn; for (nn=0; nn<nbNoiseChunks; nn++) {
-                size_t const randomNoiseSize = FUZ_randomLength(&lseed, maxSampleLog);
-                size_t const noiseSize  = MIN((cSize/3) , randomNoiseSize);
-                size_t const noiseStart = FUZ_rand(&lseed) % (srcBufferSize - noiseSize);
-                size_t const cStart = FUZ_rand(&lseed) % (cSize - noiseSize);
-                memcpy(cBuffer+cStart, srcBuffer+noiseStart, noiseSize);
-        }   }
-
-        /* try decompression on noisy data */
-        CHECK_Z( ZSTD_initDStream(zd_noise) );   /* note : no dictionary */
-        {   ZSTD_inBuffer  inBuff = { cBuffer, cSize, 0 };
-            ZSTD_outBuffer outBuff= { dstBuffer, dstBufferSize, 0 };
-            while (outBuff.pos < dstBufferSize) {
-                size_t const randomCSrcSize = FUZ_randomLength(&lseed, maxSampleLog);
-                size_t const randomDstSize = FUZ_randomLength(&lseed, maxSampleLog);
-                size_t const adjustedDstSize = MIN(dstBufferSize - outBuff.pos, randomDstSize);
-                size_t const adjustedCSrcSize = MIN(cSize - inBuff.pos, randomCSrcSize);
-                outBuff.size = outBuff.pos + adjustedDstSize;
-                inBuff.size  = inBuff.pos + adjustedCSrcSize;
-                {   size_t const decompressError = ZSTD_decompressStream(zd, &outBuff, &inBuff);
-                    if (ZSTD_isError(decompressError)) break;   /* error correctly detected */
-                    /* No forward progress possible */
-                    if (outBuff.pos < outBuff.size && inBuff.pos == cSize) break;
-    }   }   }   }
-    DISPLAY("\r%u fuzzer tests completed   \n", testNb);
-
-_cleanup:
-    ZSTDMT_freeCCtx(zc);
-    ZSTD_freeDStream(zd);
-    ZSTD_freeDStream(zd_noise);
-    free(cNoiseBuffer[0]);
-    free(cNoiseBuffer[1]);
-    free(cNoiseBuffer[2]);
-    free(cNoiseBuffer[3]);
-    free(cNoiseBuffer[4]);
-    free(copyBuffer);
-    free(cBuffer);
-    free(dstBuffer);
-    return result;
-
-_output_error:
-    result = 1;
-    goto _cleanup;
-}
-
-/** If useOpaqueAPI, sets param in cctxParams.
- *  Otherwise, sets the param in zc. */
-static size_t setCCtxParameter(ZSTD_CCtx* zc, ZSTD_CCtx_params* cctxParams,
-                               ZSTD_cParameter param, unsigned value,
-                               int useOpaqueAPI)
-{
-    if (useOpaqueAPI) {
-        return ZSTD_CCtxParams_setParameter(cctxParams, param, value);
-    } else {
-        return ZSTD_CCtx_setParameter(zc, param, value);
-    }
-}
-
-/* Tests for ZSTD_compress_generic() API */
-static int fuzzerTests_newAPI(U32 seed, int nbTests, int startTest,
-                              double compressibility, int bigTests)
-{
-    U32 const maxSrcLog = bigTests ? 24 : 22;
-    static const U32 maxSampleLog = 19;
-    size_t const srcBufferSize = (size_t)1<<maxSrcLog;
-    BYTE* cNoiseBuffer[5];
-    size_t const copyBufferSize= srcBufferSize + (1<<maxSampleLog);
-    BYTE*  const copyBuffer = (BYTE*)malloc (copyBufferSize);
-    size_t const cBufferSize   = ZSTD_compressBound(srcBufferSize);
-    BYTE*  const cBuffer = (BYTE*)malloc (cBufferSize);
-    size_t const dstBufferSize = srcBufferSize;
-    BYTE*  const dstBuffer = (BYTE*)malloc (dstBufferSize);
-    U32 result = 0;
-    int testNb = 0;
-    U32 coreSeed = seed;
-    ZSTD_CCtx* zc = ZSTD_createCCtx();   /* will be reset sometimes */
-    ZSTD_DStream* zd = ZSTD_createDStream();   /* will be reset sometimes */
-    ZSTD_DStream* const zd_noise = ZSTD_createDStream();
-    UTIL_time_t const startClock = UTIL_getTime();
-    const BYTE* dict = NULL;   /* can keep same dict on 2 consecutive tests */
-    size_t dictSize = 0;
-    U32 oldTestLog = 0;
-    U32 windowLogMalus = 0;   /* can survive between 2 loops */
-    U32 const cLevelMax = bigTests ? (U32)ZSTD_maxCLevel()-1 : g_cLevelMax_smallTests;
-    U32 const nbThreadsMax = bigTests ? 4 : 2;
-    ZSTD_CCtx_params* cctxParams = ZSTD_createCCtxParams();
-
-    /* allocations */
-    cNoiseBuffer[0] = (BYTE*)malloc (srcBufferSize);
-    cNoiseBuffer[1] = (BYTE*)malloc (srcBufferSize);
-    cNoiseBuffer[2] = (BYTE*)malloc (srcBufferSize);
-    cNoiseBuffer[3] = (BYTE*)malloc (srcBufferSize);
-    cNoiseBuffer[4] = (BYTE*)malloc (srcBufferSize);
-    CHECK (!cNoiseBuffer[0] || !cNoiseBuffer[1] || !cNoiseBuffer[2] || !cNoiseBuffer[3] || !cNoiseBuffer[4] ||
-           !copyBuffer || !dstBuffer || !cBuffer || !zc || !zd || !zd_noise ,
-           "Not enough memory, fuzzer tests cancelled");
-
-    /* Create initial samples */
-    RDG_genBuffer(cNoiseBuffer[0], srcBufferSize, 0.00, 0., coreSeed);    /* pure noise */
-    RDG_genBuffer(cNoiseBuffer[1], srcBufferSize, 0.05, 0., coreSeed);    /* barely compressible */
-    RDG_genBuffer(cNoiseBuffer[2], srcBufferSize, compressibility, 0., coreSeed);
-    RDG_genBuffer(cNoiseBuffer[3], srcBufferSize, 0.95, 0., coreSeed);    /* highly compressible */
-    RDG_genBuffer(cNoiseBuffer[4], srcBufferSize, 1.00, 0., coreSeed);    /* sparse content */
-    memset(copyBuffer, 0x65, copyBufferSize);                             /* make copyBuffer considered initialized */
-    CHECK_Z( ZSTD_initDStream_usingDict(zd, NULL, 0) );   /* ensure at least one init */
-
-    /* catch up testNb */
-    for (testNb=1; testNb < startTest; testNb++)
-        FUZ_rand(&coreSeed);
-
-    /* test loop */
-    for ( ; (testNb <= nbTests) || (UTIL_clockSpanMicro(startClock) < g_clockTime) ; testNb++ ) {
-        U32 lseed;
-        int opaqueAPI;
-        const BYTE* srcBuffer;
-        size_t totalTestSize, totalGenSize, cSize;
-        XXH64_state_t xxhState;
-        U64 crcOrig;
-        U32 resetAllowed = 1;
-        size_t maxTestSize;
-        ZSTD_parameters savedParams;
-
-        /* init */
-        if (nbTests >= testNb) { DISPLAYUPDATE(2, "\r%6u/%6u    ", testNb, nbTests); }
-        else { DISPLAYUPDATE(2, "\r%6u          ", testNb); }
-        FUZ_rand(&coreSeed);
-        lseed = coreSeed ^ prime32;
-        DISPLAYLEVEL(5, " ***  Test %u  *** \n", testNb);
-        opaqueAPI = FUZ_rand(&lseed) & 1;
-
-        /* states full reset (deliberately not synchronized) */
-        /* some issues can only happen when reusing states */
-        if ((FUZ_rand(&lseed) & 0xFF) == 131) {
-            DISPLAYLEVEL(5, "Creating new context \n");
-            ZSTD_freeCCtx(zc);
-            zc = ZSTD_createCCtx();
-            CHECK(zc == NULL, "ZSTD_createCCtx allocation error");
-            resetAllowed = 0;
-        }
-        if ((FUZ_rand(&lseed) & 0xFF) == 132) {
-            ZSTD_freeDStream(zd);
-            zd = ZSTD_createDStream();
-            CHECK(zd == NULL, "ZSTD_createDStream allocation error");
-            ZSTD_initDStream_usingDict(zd, NULL, 0);  /* ensure at least one init */
-        }
-
-        /* srcBuffer selection [0-4] */
-        {   U32 buffNb = FUZ_rand(&lseed) & 0x7F;
-            if (buffNb & 7) buffNb=2;   /* most common : compressible (P) */
-            else {
-                buffNb >>= 3;
-                if (buffNb & 7) {
-                    const U32 tnb[2] = { 1, 3 };   /* barely/highly compressible */
-                    buffNb = tnb[buffNb >> 3];
-                } else {
-                    const U32 tnb[2] = { 0, 4 };   /* not compressible / sparse */
-                    buffNb = tnb[buffNb >> 3];
-            }   }
-            srcBuffer = cNoiseBuffer[buffNb];
-        }
-
-        /* compression init */
-        CHECK_Z( ZSTD_CCtx_loadDictionary(zc, NULL, 0) );   /* cancel previous dict /*/
-        if ((FUZ_rand(&lseed)&1) /* at beginning, to keep same nb of rand */
-          && oldTestLog   /* at least one test happened */
-          && resetAllowed) {
-            /* just set a compression level */
-            maxTestSize = FUZ_randomLength(&lseed, oldTestLog+2);
-            if (maxTestSize >= srcBufferSize) maxTestSize = srcBufferSize-1;
-            {   int const compressionLevel = (FUZ_rand(&lseed) % 5) + 1;
-                DISPLAYLEVEL(5, "t%u : compression level : %i \n", testNb, compressionLevel);
-                CHECK_Z (setCCtxParameter(zc, cctxParams, ZSTD_c_compressionLevel, compressionLevel, opaqueAPI) );
-            }
-        } else {
-            U32 const testLog = FUZ_rand(&lseed) % maxSrcLog;
-            U32 const dictLog = FUZ_rand(&lseed) % maxSrcLog;
-            U32 const cLevelCandidate = (FUZ_rand(&lseed) %
-                               (ZSTD_maxCLevel() -
-                               (MAX(testLog, dictLog) / 2))) +
-                               1;
-            int const cLevel = MIN(cLevelCandidate, cLevelMax);
-            DISPLAYLEVEL(5, "t%i: base cLevel : %u \n", testNb, cLevel);
-            maxTestSize = FUZ_rLogLength(&lseed, testLog);
-            DISPLAYLEVEL(5, "t%i: maxTestSize : %u \n", testNb, (unsigned)maxTestSize);
-            oldTestLog = testLog;
-            /* random dictionary selection */
-            dictSize  = ((FUZ_rand(&lseed)&63)==1) ? FUZ_rLogLength(&lseed, dictLog) : 0;
-            {   size_t const dictStart = FUZ_rand(&lseed) % (srcBufferSize - dictSize);
-                dict = srcBuffer + dictStart;
-                if (!dictSize) dict=NULL;
-            }
-            {   U64 const pledgedSrcSize = (FUZ_rand(&lseed) & 3) ? ZSTD_CONTENTSIZE_UNKNOWN : maxTestSize;
-                ZSTD_compressionParameters cParams = ZSTD_getCParams(cLevel, pledgedSrcSize, dictSize);
-                const U32 windowLogMax = bigTests ? 24 : 20;
-                const U32 searchLogMax = bigTests ? 15 : 13;
-                if (dictSize)
-                    DISPLAYLEVEL(5, "t%u: with dictionary of size : %zu \n", testNb, dictSize);
-
-                /* mess with compression parameters */
-                cParams.windowLog += (FUZ_rand(&lseed) & 3) - 1;
-                cParams.windowLog = MIN(windowLogMax, cParams.windowLog);
-                cParams.hashLog += (FUZ_rand(&lseed) & 3) - 1;
-                cParams.chainLog += (FUZ_rand(&lseed) & 3) - 1;
-                cParams.searchLog += (FUZ_rand(&lseed) & 3) - 1;
-                cParams.searchLog = MIN(searchLogMax, cParams.searchLog);
-                cParams.minMatch += (FUZ_rand(&lseed) & 3) - 1;
-                cParams.targetLength = (U32)((cParams.targetLength + 1 ) * (0.5 + ((double)(FUZ_rand(&lseed) & 127) / 128)));
-                cParams = ZSTD_adjustCParams(cParams, pledgedSrcSize, dictSize);
-
-                if (FUZ_rand(&lseed) & 1) {
-                    DISPLAYLEVEL(5, "t%u: windowLog : %u \n", testNb, cParams.windowLog);
-                    CHECK_Z( setCCtxParameter(zc, cctxParams, ZSTD_c_windowLog, cParams.windowLog, opaqueAPI) );
-                    assert(cParams.windowLog >= ZSTD_WINDOWLOG_MIN);   /* guaranteed by ZSTD_adjustCParams() */
-                    windowLogMalus = (cParams.windowLog - ZSTD_WINDOWLOG_MIN) / 5;
-                }
-                if (FUZ_rand(&lseed) & 1) {
-                    DISPLAYLEVEL(5, "t%u: hashLog : %u \n", testNb, cParams.hashLog);
-                    CHECK_Z( setCCtxParameter(zc, cctxParams, ZSTD_c_hashLog, cParams.hashLog, opaqueAPI) );
-                }
-                if (FUZ_rand(&lseed) & 1) {
-                    DISPLAYLEVEL(5, "t%u: chainLog : %u \n", testNb, cParams.chainLog);
-                    CHECK_Z( setCCtxParameter(zc, cctxParams, ZSTD_c_chainLog, cParams.chainLog, opaqueAPI) );
-                }
-                if (FUZ_rand(&lseed) & 1) CHECK_Z( setCCtxParameter(zc, cctxParams, ZSTD_c_searchLog, cParams.searchLog, opaqueAPI) );
-                if (FUZ_rand(&lseed) & 1) CHECK_Z( setCCtxParameter(zc, cctxParams, ZSTD_c_minMatch, cParams.minMatch, opaqueAPI) );
-                if (FUZ_rand(&lseed) & 1) CHECK_Z( setCCtxParameter(zc, cctxParams, ZSTD_c_targetLength, cParams.targetLength, opaqueAPI) );
-
-                /* mess with long distance matching parameters */
-                if (bigTests) {
-                    if (FUZ_rand(&lseed) & 1) CHECK_Z( setCCtxParameter(zc, cctxParams, ZSTD_c_enableLongDistanceMatching, FUZ_rand(&lseed) & 63, opaqueAPI) );
-                    if (FUZ_rand(&lseed) & 3) CHECK_Z( setCCtxParameter(zc, cctxParams, ZSTD_c_ldmHashLog, FUZ_randomClampedLength(&lseed, ZSTD_HASHLOG_MIN, 23), opaqueAPI) );
-                    if (FUZ_rand(&lseed) & 3) CHECK_Z( setCCtxParameter(zc, cctxParams, ZSTD_c_ldmMinMatch, FUZ_randomClampedLength(&lseed, ZSTD_LDM_MINMATCH_MIN, ZSTD_LDM_MINMATCH_MAX), opaqueAPI) );
-                    if (FUZ_rand(&lseed) & 3) CHECK_Z( setCCtxParameter(zc, cctxParams, ZSTD_c_ldmBucketSizeLog, FUZ_randomClampedLength(&lseed, ZSTD_LDM_BUCKETSIZELOG_MIN, ZSTD_LDM_BUCKETSIZELOG_MAX), opaqueAPI) );
-                    if (FUZ_rand(&lseed) & 3) CHECK_Z( setCCtxParameter(zc, cctxParams, ZSTD_c_ldmHashRateLog, FUZ_randomClampedLength(&lseed, ZSTD_LDM_HASHRATELOG_MIN, ZSTD_LDM_HASHRATELOG_MAX), opaqueAPI) );
-                    if (FUZ_rand(&lseed) & 3) CHECK_Z( setCCtxParameter(zc, cctxParams, ZSTD_c_srcSizeHint, FUZ_randomClampedLength(&lseed, ZSTD_SRCSIZEHINT_MIN, ZSTD_SRCSIZEHINT_MAX), opaqueAPI) );
-                }
-
-                /* mess with frame parameters */
-                if (FUZ_rand(&lseed) & 1) {
-                    int const checksumFlag = FUZ_rand(&lseed) & 1;
-                    DISPLAYLEVEL(5, "t%u: frame checksum : %u \n", testNb, checksumFlag);
-                    CHECK_Z( setCCtxParameter(zc, cctxParams, ZSTD_c_checksumFlag, checksumFlag, opaqueAPI) );
-                }
-                if (FUZ_rand(&lseed) & 1) CHECK_Z( setCCtxParameter(zc, cctxParams, ZSTD_c_dictIDFlag, FUZ_rand(&lseed) & 1, opaqueAPI) );
-                if (FUZ_rand(&lseed) & 1) CHECK_Z( setCCtxParameter(zc, cctxParams, ZSTD_c_contentSizeFlag, FUZ_rand(&lseed) & 1, opaqueAPI) );
-                if (FUZ_rand(&lseed) & 1) {
-                    DISPLAYLEVEL(5, "t%u: pledgedSrcSize : %u \n", testNb, (unsigned)pledgedSrcSize);
-                    CHECK_Z( ZSTD_CCtx_setPledgedSrcSize(zc, pledgedSrcSize) );
-                }
-
-                /* multi-threading parameters. Only adjust occasionally for small tests. */
-                if (bigTests || (FUZ_rand(&lseed) & 0xF) == 0xF) {
-                    U32 const nbThreadsCandidate = (FUZ_rand(&lseed) & 4) + 1;
-                    U32 const nbThreadsAdjusted = (windowLogMalus < nbThreadsCandidate) ? nbThreadsCandidate - windowLogMalus : 1;
-                    int const nbThreads = MIN(nbThreadsAdjusted, nbThreadsMax);
-                    DISPLAYLEVEL(5, "t%i: nbThreads : %u \n", testNb, nbThreads);
-                    CHECK_Z( setCCtxParameter(zc, cctxParams, ZSTD_c_nbWorkers, nbThreads, opaqueAPI) );
-                    if (nbThreads > 1) {
-                        U32 const jobLog = FUZ_rand(&lseed) % (testLog+1);
-                        CHECK_Z( setCCtxParameter(zc, cctxParams, ZSTD_c_overlapLog, FUZ_rand(&lseed) % 10, opaqueAPI) );
-                        CHECK_Z( setCCtxParameter(zc, cctxParams, ZSTD_c_jobSize, (U32)FUZ_rLogLength(&lseed, jobLog), opaqueAPI) );
-                    }
-                }
-                /* Enable rsyncable mode 1 in 4 times. */
-                setCCtxParameter(zc, cctxParams, ZSTD_c_rsyncable, (FUZ_rand(&lseed) % 4 == 0), opaqueAPI);
-
-                if (FUZ_rand(&lseed) & 1) CHECK_Z( setCCtxParameter(zc, cctxParams, ZSTD_c_forceMaxWindow, FUZ_rand(&lseed) & 1, opaqueAPI) );
-
-                /* Apply parameters */
-                if (opaqueAPI) {
-                    DISPLAYLEVEL(5, "t%u: applying CCtxParams \n", testNb);
-                    CHECK_Z (ZSTD_CCtx_setParametersUsingCCtxParams(zc, cctxParams) );
-                }
-
-                if (FUZ_rand(&lseed) & 1) {
-                    if (FUZ_rand(&lseed) & 1) {
-                        CHECK_Z( ZSTD_CCtx_loadDictionary(zc, dict, dictSize) );
-                    } else {
-                        CHECK_Z( ZSTD_CCtx_loadDictionary_byReference(zc, dict, dictSize) );
-                    }
-                } else {
-                    CHECK_Z( ZSTD_CCtx_refPrefix(zc, dict, dictSize) );
-                }
-        }   }
-
-        CHECK_Z(getCCtxParams(zc, &savedParams));
-
-        /* multi-segments compression test */
-        XXH64_reset(&xxhState, 0);
-        {   ZSTD_outBuffer outBuff = { cBuffer, cBufferSize, 0 } ;
-            for (cSize=0, totalTestSize=0 ; (totalTestSize < maxTestSize) ; ) {
-                /* compress random chunks into randomly sized dst buffers */
-                size_t const randomSrcSize = FUZ_randomLength(&lseed, maxSampleLog);
-                size_t const srcSize = MIN(maxTestSize-totalTestSize, randomSrcSize);
-                size_t const srcStart = FUZ_rand(&lseed) % (srcBufferSize - srcSize);
-                size_t const randomDstSize = FUZ_randomLength(&lseed, maxSampleLog+1);
-                size_t const dstBuffSize = MIN(cBufferSize - cSize, randomDstSize);
-                ZSTD_EndDirective const flush = (FUZ_rand(&lseed) & 15) ? ZSTD_e_continue : ZSTD_e_flush;
-                ZSTD_inBuffer inBuff = { srcBuffer+srcStart, srcSize, 0 };
-                outBuff.size = outBuff.pos + dstBuffSize;
-
-                CHECK_Z( ZSTD_compressStream2(zc, &outBuff, &inBuff, flush) );
-                DISPLAYLEVEL(6, "t%u: compress consumed %u bytes (total : %u) ; flush: %u (total : %u) \n",
-                    testNb, (unsigned)inBuff.pos, (unsigned)(totalTestSize + inBuff.pos), (unsigned)flush, (unsigned)outBuff.pos);
-
-                XXH64_update(&xxhState, srcBuffer+srcStart, inBuff.pos);
-                memcpy(copyBuffer+totalTestSize, srcBuffer+srcStart, inBuff.pos);
-                totalTestSize += inBuff.pos;
-            }
-
-            /* final frame epilogue */
-            {   size_t remainingToFlush = 1;
-                while (remainingToFlush) {
-                    ZSTD_inBuffer inBuff = { NULL, 0, 0 };
-                    size_t const randomDstSize = FUZ_randomLength(&lseed, maxSampleLog+1);
-                    size_t const adjustedDstSize = MIN(cBufferSize - cSize, randomDstSize);
-                    outBuff.size = outBuff.pos + adjustedDstSize;
-                    DISPLAYLEVEL(6, "t%u: End-flush into dst buffer of size %u \n", testNb, (unsigned)adjustedDstSize);
-                    remainingToFlush = ZSTD_compressStream2(zc, &outBuff, &inBuff, ZSTD_e_end);
-                    DISPLAYLEVEL(6, "t%u: Total flushed so far : %u bytes \n", testNb, (unsigned)outBuff.pos);
-                    CHECK( ZSTD_isError(remainingToFlush),
-                          "ZSTD_compressStream2 w/ ZSTD_e_end error : %s",
-                           ZSTD_getErrorName(remainingToFlush) );
-            }   }
-            crcOrig = XXH64_digest(&xxhState);
-            cSize = outBuff.pos;
-            DISPLAYLEVEL(5, "Frame completed : %zu bytes \n", cSize);
-        }
-
-        CHECK(badParameters(zc, savedParams), "CCtx params are wrong");
-
-        /* multi - fragments decompression test */
-        if (!dictSize /* don't reset if dictionary : could be different */ && (FUZ_rand(&lseed) & 1)) {
-            DISPLAYLEVEL(5, "resetting DCtx (dict:%p) \n", dict);
-            CHECK_Z( ZSTD_resetDStream(zd) );
-        } else {
-            if (dictSize)
-                DISPLAYLEVEL(5, "using dictionary of size %zu \n", dictSize);
-            CHECK_Z( ZSTD_initDStream_usingDict(zd, dict, dictSize) );
-        }
-        {   size_t decompressionResult = 1;
-            ZSTD_inBuffer  inBuff = { cBuffer, cSize, 0 };
-            ZSTD_outBuffer outBuff= { dstBuffer, dstBufferSize, 0 };
-            for (totalGenSize = 0 ; decompressionResult ; ) {
-                size_t const readCSrcSize = FUZ_randomLength(&lseed, maxSampleLog);
-                size_t const randomDstSize = FUZ_randomLength(&lseed, maxSampleLog);
-                size_t const dstBuffSize = MIN(dstBufferSize - totalGenSize, randomDstSize);
-                inBuff.size = inBuff.pos + readCSrcSize;
-                outBuff.size = outBuff.pos + dstBuffSize;
-                DISPLAYLEVEL(6, "decompression presented %u new bytes (pos:%u/%u)\n",
-                                (unsigned)readCSrcSize, (unsigned)inBuff.pos, (unsigned)cSize);
-                decompressionResult = ZSTD_decompressStream(zd, &outBuff, &inBuff);
-                DISPLAYLEVEL(6, "so far: consumed = %u, produced = %u \n",
-                                (unsigned)inBuff.pos, (unsigned)outBuff.pos);
-                if (ZSTD_isError(decompressionResult)) {
-                    DISPLAY("ZSTD_decompressStream error : %s \n", ZSTD_getErrorName(decompressionResult));
-                    findDiff(copyBuffer, dstBuffer, totalTestSize);
-                }
-                CHECK (ZSTD_isError(decompressionResult), "decompression error : %s", ZSTD_getErrorName(decompressionResult));
-                CHECK (inBuff.pos > cSize, "ZSTD_decompressStream consumes too much input : %u > %u ", (unsigned)inBuff.pos, (unsigned)cSize);
-            }
-            CHECK (inBuff.pos != cSize, "compressed data should be fully read (%u != %u)", (unsigned)inBuff.pos, (unsigned)cSize);
-            CHECK (outBuff.pos != totalTestSize, "decompressed data : wrong size (%u != %u)", (unsigned)outBuff.pos, (unsigned)totalTestSize);
-            {   U64 const crcDest = XXH64(dstBuffer, totalTestSize, 0);
-                if (crcDest!=crcOrig) findDiff(copyBuffer, dstBuffer, totalTestSize);
-                CHECK (crcDest!=crcOrig, "decompressed data corrupted");
-        }   }
-
-        /*=====   noisy/erroneous src decompression test   =====*/
-
-        /* add some noise */
-        {   U32 const nbNoiseChunks = (FUZ_rand(&lseed) & 7) + 2;
-            U32 nn; for (nn=0; nn<nbNoiseChunks; nn++) {
-                size_t const randomNoiseSize = FUZ_randomLength(&lseed, maxSampleLog);
-                size_t const noiseSize  = MIN((cSize/3) , randomNoiseSize);
-                size_t const noiseStart = FUZ_rand(&lseed) % (srcBufferSize - noiseSize);
-                size_t const cStart = FUZ_rand(&lseed) % (cSize - noiseSize);
-                memcpy(cBuffer+cStart, srcBuffer+noiseStart, noiseSize);
-        }   }
-
-        /* try decompression on noisy data */
-        CHECK_Z( ZSTD_initDStream(zd_noise) );   /* note : no dictionary */
-        {   ZSTD_inBuffer  inBuff = { cBuffer, cSize, 0 };
-            ZSTD_outBuffer outBuff= { dstBuffer, dstBufferSize, 0 };
-            while (outBuff.pos < dstBufferSize) {
-                size_t const randomCSrcSize = FUZ_randomLength(&lseed, maxSampleLog);
-                size_t const randomDstSize = FUZ_randomLength(&lseed, maxSampleLog);
-                size_t const adjustedDstSize = MIN(dstBufferSize - outBuff.pos, randomDstSize);
-                size_t const adjustedCSrcSize = MIN(cSize - inBuff.pos, randomCSrcSize);
-                outBuff.size = outBuff.pos + adjustedDstSize;
-                inBuff.size  = inBuff.pos + adjustedCSrcSize;
-                {   size_t const decompressError = ZSTD_decompressStream(zd, &outBuff, &inBuff);
-                    if (ZSTD_isError(decompressError)) break;   /* error correctly detected */
-                    /* Good so far, but no more progress possible */
-                    if (outBuff.pos < outBuff.size && inBuff.pos == cSize) break;
-    }   }   }   }
-    DISPLAY("\r%u fuzzer tests completed   \n", testNb-1);
-
-_cleanup:
-    ZSTD_freeCCtx(zc);
-    ZSTD_freeDStream(zd);
-    ZSTD_freeDStream(zd_noise);
-    ZSTD_freeCCtxParams(cctxParams);
-    free(cNoiseBuffer[0]);
-    free(cNoiseBuffer[1]);
-    free(cNoiseBuffer[2]);
-    free(cNoiseBuffer[3]);
-    free(cNoiseBuffer[4]);
-    free(copyBuffer);
-    free(cBuffer);
-    free(dstBuffer);
-    return result;
-
-_output_error:
-    result = 1;
-    goto _cleanup;
-}
-
-/*-*******************************************************
-*  Command line
-*********************************************************/
-static int FUZ_usage(const char* programName)
-{
-    DISPLAY( "Usage :\n");
-    DISPLAY( "      %s [args]\n", programName);
-    DISPLAY( "\n");
-    DISPLAY( "Arguments :\n");
-    DISPLAY( " -i#    : Nb of tests (default:%u) \n", nbTestsDefault);
-    DISPLAY( " -s#    : Select seed (default:prompt user)\n");
-    DISPLAY( " -t#    : Select starting test number (default:0)\n");
-    DISPLAY( " -P#    : Select compressibility in %% (default:%i%%)\n", FUZ_COMPRESSIBILITY_DEFAULT);
-    DISPLAY( " -v     : verbose\n");
-    DISPLAY( " -p     : pause at the end\n");
-    DISPLAY( " -h     : display help and exit\n");
-    return 0;
-}
-
-typedef enum { simple_api, mt_api, advanced_api } e_api;
-
-int main(int argc, const char** argv)
-{
-    U32 seed = 0;
-    int seedset = 0;
-    int nbTests = nbTestsDefault;
-    int testNb = 0;
-    int proba = FUZ_COMPRESSIBILITY_DEFAULT;
-    int result = 0;
-    int mainPause = 0;
-    int bigTests = (sizeof(size_t) == 8);
-    e_api selected_api = simple_api;
-    const char* const programName = argv[0];
-    int argNb;
-
-    /* Check command line */
-    for(argNb=1; argNb<argc; argNb++) {
-        const char* argument = argv[argNb];
-        assert(argument != NULL);
-
-        /* Parsing commands. Aggregated commands are allowed */
-        if (argument[0]=='-') {
-
-            if (!strcmp(argument, "--mt")) { selected_api=mt_api; testNb += !testNb; continue; }
-            if (!strcmp(argument, "--newapi")) { selected_api=advanced_api; testNb += !testNb; continue; }
-            if (!strcmp(argument, "--no-big-tests")) { bigTests=0; continue; }
-
-            argument++;
-            while (*argument!=0) {
-                switch(*argument)
-                {
-                case 'h':
-                    return FUZ_usage(programName);
-
-                case 'v':
-                    argument++;
-                    g_displayLevel++;
-                    break;
-
-                case 'q':
-                    argument++;
-                    g_displayLevel--;
-                    break;
-
-                case 'p': /* pause at the end */
-                    argument++;
-                    mainPause = 1;
-                    break;
-
-                case 'i':   /* limit tests by nb of iterations (default) */
-                    argument++;
-                    nbTests=0; g_clockTime=0;
-                    while ((*argument>='0') && (*argument<='9')) {
-                        nbTests *= 10;
-                        nbTests += *argument - '0';
-                        argument++;
-                    }
-                    break;
-
-                case 'T':   /* limit tests by time */
-                    argument++;
-                    nbTests=0; g_clockTime=0;
-                    while ((*argument>='0') && (*argument<='9')) {
-                        g_clockTime *= 10;
-                        g_clockTime += *argument - '0';
-                        argument++;
-                    }
-                    if (*argument=='m') {    /* -T1m == -T60 */
-                        g_clockTime *=60, argument++;
-                        if (*argument=='n') argument++; /* -T1mn == -T60 */
-                    } else if (*argument=='s') argument++; /* -T10s == -T10 */
-                    g_clockTime *= SEC_TO_MICRO;
-                    break;
-
-                case 's':   /* manually select seed */
-                    argument++;
-                    seedset=1;
-                    seed=0;
-                    while ((*argument>='0') && (*argument<='9')) {
-                        seed *= 10;
-                        seed += *argument - '0';
-                        argument++;
-                    }
-                    break;
-
-                case 't':   /* select starting test number */
-                    argument++;
-                    testNb=0;
-                    while ((*argument>='0') && (*argument<='9')) {
-                        testNb *= 10;
-                        testNb += *argument - '0';
-                        argument++;
-                    }
-                    break;
-
-                case 'P':   /* compressibility % */
-                    argument++;
-                    proba=0;
-                    while ((*argument>='0') && (*argument<='9')) {
-                        proba *= 10;
-                        proba += *argument - '0';
-                        argument++;
-                    }
-                    if (proba<0) proba=0;
-                    if (proba>100) proba=100;
-                    break;
-
-                default:
-                    return FUZ_usage(programName);
-                }
-    }   }   }   /* for(argNb=1; argNb<argc; argNb++) */
-
-    /* Get Seed */
-    DISPLAY("Starting zstream tester (%i-bits, %s)\n", (int)(sizeof(size_t)*8), ZSTD_VERSION_STRING);
-
-    if (!seedset) {
-        time_t const t = time(NULL);
-        U32 const h = XXH32(&t, sizeof(t), 1);
-        seed = h % 10000;
-    }
-
-    DISPLAY("Seed = %u\n", (unsigned)seed);
-    if (proba!=FUZ_COMPRESSIBILITY_DEFAULT) DISPLAY("Compressibility : %i%%\n", proba);
-
-    if (nbTests<=0) nbTests=1;
-
-    if (testNb==0) {
-        result = basicUnitTests(0, ((double)proba) / 100);  /* constant seed for predictability */
-    }
-
-    if (!result) {
-        switch(selected_api)
-        {
-        case simple_api :
-            result = fuzzerTests(seed, nbTests, testNb, ((double)proba) / 100, bigTests);
-            break;
-        case mt_api :
-            result = fuzzerTests_MT(seed, nbTests, testNb, ((double)proba) / 100, bigTests);
-            break;
-        case advanced_api :
-            result = fuzzerTests_newAPI(seed, nbTests, testNb, ((double)proba) / 100, bigTests);
-            break;
-        default :
-            assert(0);   /* impossible */
-        }
-    }
-
-    if (mainPause) {
-        int unused;
-        DISPLAY("Press Enter \n");
-        unused = getchar();
-        (void)unused;
-    }
-    return result;
-}
diff --git a/zlibWrapper/Makefile b/zlibWrapper/Makefile
index 6addb743cbe6..feed5b84a34f 100644
--- a/zlibWrapper/Makefile
+++ b/zlibWrapper/Makefile
@@ -20,8 +20,8 @@ TEST_FILE = ../doc/zstd_compression_format.md
 
 CPPFLAGS += -DXXH_NAMESPACE=ZSTD_ -I$(ZLIB_PATH) -I$(PROGRAMS_PATH)       \
             -I$(ZSTDLIBDIR) -I$(ZSTDLIBDIR)/common -I$(ZLIBWRAPPER_PATH)
-STDFLAGS  = -std=c90 -pedantic -Wno-long-long -Wno-variadic-macros -Wc++-compat \
-            -DNO_snprintf -DNO_vsnprintf  # strict ISO C90 is missing these prototypes
+STDFLAGS  = -std=c89 -pedantic -Wno-long-long -Wno-variadic-macros -Wc++-compat \
+            -DNO_snprintf -DNO_vsnprintf  # strict ANSI C89 is missing these prototypes
 DEBUGFLAGS= -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wswitch-enum \
             -Wdeclaration-after-statement -Wstrict-prototypes -Wundef     \
             -Wstrict-aliasing=1
diff --git a/zlibWrapper/examples/zwrapbench.c b/zlibWrapper/examples/zwrapbench.c
index 35893a4b4384..f30cad40c719 100644
--- a/zlibWrapper/examples/zwrapbench.c
+++ b/zlibWrapper/examples/zwrapbench.c
@@ -74,7 +74,7 @@ static U32 g_compressibilityDefault = 50;
 #define DEFAULT_DISPLAY_LEVEL 2
 #define DISPLAY(...)         fprintf(displayOut, __VA_ARGS__)
 #define DISPLAYLEVEL(l, ...) if (g_displayLevel>=l) { DISPLAY(__VA_ARGS__); }
-static int g_displayLevel = DEFAULT_DISPLAY_LEVEL;   /* 0 : no display;   1: errors;   2 : + result + interaction + warnings;   3 : + progression;   4 : + information */
+static unsigned g_displayLevel = DEFAULT_DISPLAY_LEVEL;   /* 0 : no display;   1: errors;   2 : + result + interaction + warnings;   3 : + progression;   4 : + information */
 static FILE* displayOut;
 
 #define DISPLAYUPDATE(l, ...) if (g_displayLevel>=l) { \
@@ -848,7 +848,7 @@ static unsigned readU32FromChar(const char** stringPtr)
 {
     unsigned result = 0;
     while ((**stringPtr >='0') && (**stringPtr <='9'))
-        result *= 10, result += **stringPtr - '0', (*stringPtr)++ ;
+        result *= 10, result += (unsigned)(**stringPtr - '0'), (*stringPtr)++ ;
     return result;
 }
 
@@ -865,24 +865,18 @@ int main(int argCount, char** argv)
     int cLevel = ZSTDCLI_CLEVEL_DEFAULT;
     int cLevelLast = 1;
     unsigned recursive = 0;
-    const char** filenameTable = (const char**)malloc(argCount * sizeof(const char*));   /* argCount >= 1 */
-    unsigned filenameIdx = 0;
+    FileNamesTable* filenames = UTIL_allocateFileNamesTable((size_t)argCount);
     const char* programName = argv[0];
     const char* dictFileName = NULL;
     char* dynNameSpace = NULL;
-#ifdef UTIL_HAS_CREATEFILELIST
-    const char** fileNamesTable = NULL;
-    char* fileNamesBuf = NULL;
-    unsigned fileNamesNb;
-#endif
 
     /* init */
-    if (filenameTable==NULL) { DISPLAY("zstd: %s \n", strerror(errno)); exit(1); }
+    if (filenames==NULL) { DISPLAY("zstd: %s \n", strerror(errno)); exit(1); }
     displayOut = stderr;
 
     /* Pick out program name from path. Don't rely on stdlib because of conflicting behavior */
     {   size_t pos;
-        for (pos = (int)strlen(programName); pos > 0; pos--) { if (programName[pos] == '/') { pos++; break; } }
+        for (pos = strlen(programName); pos > 0; pos--) { if (programName[pos] == '/') { pos++; break; } }
         programName += pos;
     }
 
@@ -930,14 +924,14 @@ int main(int argCount, char** argv)
                     case 'b':
                             /* first compression Level */
                             argument++;
-                            cLevel = readU32FromChar(&argument);
+                            cLevel = (int)readU32FromChar(&argument);
                             break;
 
                         /* range bench (benchmark only) */
                     case 'e':
                             /* last compression Level */
                             argument++;
-                            cLevelLast = readU32FromChar(&argument);
+                            cLevelLast = (int)readU32FromChar(&argument);
                             break;
 
                         /* Modify Nb Iterations (benchmark only) */
@@ -964,7 +958,7 @@ int main(int argCount, char** argv)
                         /* Pause at the end (-p) or set an additional param (-p#) (hidden option) */
                     case 'p': argument++;
                         if ((*argument>='0') && (*argument<='9')) {
-                            BMK_setAdditionalParam(readU32FromChar(&argument));
+                            BMK_setAdditionalParam((int)readU32FromChar(&argument));
                         } else
                             main_pause=1;
                         break;
@@ -984,7 +978,7 @@ int main(int argCount, char** argv)
         }
 
         /* add filename to list */
-        filenameTable[filenameIdx++] = argument;
+        UTIL_refFilename(filenames, argument);
     }
 
     /* Welcome message (if verbose) */
@@ -992,28 +986,16 @@ int main(int argCount, char** argv)
 
 #ifdef UTIL_HAS_CREATEFILELIST
     if (recursive) {
-        fileNamesTable = UTIL_createFileList(filenameTable, filenameIdx, &fileNamesBuf, &fileNamesNb, 1);
-        if (fileNamesTable) {
-            unsigned u;
-            for (u=0; u<fileNamesNb; u++) DISPLAYLEVEL(4, "%u %s\n", u, fileNamesTable[u]);
-            free((void*)filenameTable);
-            filenameTable = fileNamesTable;
-            filenameIdx = fileNamesNb;
-        }
+        UTIL_expandFNT(&filenames, 1);
     }
 #endif
 
     BMK_setNotificationLevel(g_displayLevel);
-    BMK_benchFiles(filenameTable, filenameIdx, dictFileName, cLevel, cLevelLast);
+    BMK_benchFiles(filenames->fileNames, (unsigned)filenames->tableSize, dictFileName, cLevel, cLevelLast);
 
 _end:
     if (main_pause) waitEnter();
     free(dynNameSpace);
-#ifdef UTIL_HAS_CREATEFILELIST
-    if (fileNamesTable)
-        UTIL_freeFileList(fileNamesTable, fileNamesBuf);
-    else
-#endif
-        free((void*)filenameTable);
+    UTIL_freeFileNamesTable(filenames);
     return operationResult;
 }
diff --git a/zlibWrapper/gzcompatibility.h b/zlibWrapper/gzcompatibility.h
index ea8d50c8238a..394648abbba2 100644
--- a/zlibWrapper/gzcompatibility.h
+++ b/zlibWrapper/gzcompatibility.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/zlibWrapper/zstd_zlibwrapper.c b/zlibWrapper/zstd_zlibwrapper.c
index 3fa442106f14..4d2c7be5b44c 100644
--- a/zlibWrapper/zstd_zlibwrapper.c
+++ b/zlibWrapper/zstd_zlibwrapper.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the
diff --git a/zlibWrapper/zstd_zlibwrapper.h b/zlibWrapper/zstd_zlibwrapper.h
index f828d3191b87..e791043e1d0c 100644
--- a/zlibWrapper/zstd_zlibwrapper.h
+++ b/zlibWrapper/zstd_zlibwrapper.h
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2016-present, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
+ * Copyright (c) 2016-2020, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
  * All rights reserved.
  *
  * This source code is licensed under both the BSD-style license (found in the