summaryrefslogtreecommitdiff
path: root/usr.bin/mkuzip
Commit message (Collapse)AuthorAgeFilesLines
* geom_uzip(4), mkuzip(8): Add Zstd image modeConrad Meyer2019-08-1314-131/+407
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The Zstd format bumps the CLOOP major number to 4 to avoid incompatibility with older systems. Support in geom_uzip(4) is conditional on the ZSTDIO kernel option, which is enabled in amd64 GENERIC, but not all in-tree configurations. mkuzip(8) was modified slightly to always initialize the nblocks + 1'th offset in the CLOOP file format. Previously, it was only initialized in the case where the final compressed block happened to be unaligned w.r.t. DEV_BSIZE. The "Fake" last+1 block change in r298619 means that the final compressed block's 'blen' was never correct unless the compressed uzip image happened to be BSIZE-aligned. This happened in about 1 out of every 512 cases. The zlib and lzma decompressors are probably tolerant of extra trash following the frame they were told to decode, but Zstd complains that the input size is incorrect. Correspondingly, geom_uzip(4) was modified slightly to avoid trashing the nblocks + 1'th offset when it is known to be initialized to a good value. This corrects the calculated final real cluster compressed length to match that printed by mkuzip(8). mkuzip(8) was refactored somewhat to reduce code duplication and increase ease of adding other compression formats. * Input block size validation was pulled out of individual compression init routines into main(). * Init routines now validate a user-provided compression level or select an algorithm-specific default, if none was provided. * A new interface for calculating the maximal compressed size of an incompressible input block was added for each driver. The generic code uses it to validate against MAXPHYS as well as to allocate compression result buffers in the generic code. * Algorithm selection is now driven by a table lookup, to increase ease of adding other formats in the future. mkuzip(8) gained the ability to explicitly specify a compression level with '-C'. The prior defaults -- 9 for zlib and 6 for lzma -- are maintained. The new zstd default is 9, to match zlib. Rather than select lzma or zlib with '-L' or its absense, respectively, a new argument '-A <algorithm>' is provided to select 'zlib', 'lzma', or 'zstd'. '-L' is considered deprecated, but will probably never be removed. All of the new features were documented in mkuzip.8; the page was also cleaned up slightly. Relnotes: yes Notes: svn path=/head/; revision=351005
* Minor cleanup for mkuzip(8) man page.Konstantin Belousov2019-02-191-33/+33
| | | | | | | | | | | List all single-letter options in summary. Order options alphabetically. Sponsored by: Mellanox Technologies MFC after: 3 days Notes: svn path=/head/; revision=344303
* various: general adoption of SPDX licensing ID tags.Pedro F. Giffuni2017-11-271-1/+3
| | | | | | | | | | | | | | | | | Mainly focus on files that use BSD 2-Clause license, however the tool I was using misidentified many licenses so this was mostly a manual - error prone - task. The Software Package Data Exchange (SPDX) group provides a specification to make it easier for automated tools to detect and summarize well known opensource licenses. We are gradually adopting the specification, noting that the tags are considered only advisory and do not, in any way, superceed or replace the license texts. No functional change intended. Notes: svn path=/head/; revision=326276
* DIRDEPS_BUILD: Update dependencies.Bryan Drewery2017-10-311-1/+0
| | | | | | | Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=325188
* Doh, fix some botched "fix" in r320277.Maxim Sobolev2017-06-231-1/+0
| | | | | | | | Reported by: cem MFC after: 6 weeks Notes: svn path=/head/; revision=320301
* Don't leak file descriptor in some cases.Maxim Sobolev2017-06-231-0/+4
| | | | | | | | Reported by: cem MFC after: 6 weeks Notes: svn path=/head/; revision=320277
* o Move logic that determines size of the input image into its ownMaxim Sobolev2017-06-175-29/+127
| | | | | | | | | | | | | file. That logic has grown quite significantly now; o add a special handling for the snapshot images. Those have some extra headers at the end of the image and we don't need those in the output image really. MFC after: 6 weeks Notes: svn path=/head/; revision=320048
* Don't depend on assert(3) getting evaluatedAlan Somers2017-05-101-3/+3
| | | | | | | | | | Reported by: imp MFC after: 3 weeks X-MFC-With: 318141, 318143 Sponsored by: Spectra Logic Corp Notes: svn path=/head/; revision=318144
* strcpy => strlcpyAlan Somers2017-05-101-1/+2
| | | | | | | | | | Reported by: Coverity CID: 1352771 MFC after: 3 weeks Sponsored by: Spectra Logic Corp Notes: svn path=/head/; revision=318141
* Remove spaces at EOL and sort correctly the SEE ALSO sectionBaptiste Daroussin2017-02-111-3/+3
| | | | | | | | Reported by: make manlint MFC after: 2 days Notes: svn path=/head/; revision=313667
* DIRDEPS_BUILD: Update dependencies.Bryan Drewery2016-05-041-0/+1
| | | | | | | Sponsored by: EMC / Isilon Storage Division Notes: svn path=/head/; revision=299042
* Try to make gcc builds happy again by removing a redundant declaration.Bjoern A. Zeeb2016-04-251-1/+0
| | | | Notes: svn path=/head/; revision=298577
* GC duplicate define.Maxim Sobolev2016-04-231-2/+0
| | | | Notes: svn path=/head/; revision=298505
* Improve performance in a few key areas:Maxim Sobolev2016-04-2321-192/+1037
| | | | | | | | | | | | | | | | | | | | | | | o Split the compression across several worker threads. By default, "several" matches number of CPUs, capped at 24 for sanity when running on a very big hardwares. Provide option to set that number manually; o Fix bug inherited from the mkulzma (R.I.P) which degraded already slow LZMA compression even further by calling function to release compression state after processing each block. It is neither documented as required nor actually required by the LZMA library. This caused spree of system calls to release memory and then map it again for every block. LZMA compression is more than 2x faster after this change alone; o Record time it takes to do compression and report throughput achieved. o Add simple first-level 256 entry hash table for de-dup code, so it's not becoming a bottleneck at big files. Notes: svn path=/head/; revision=298504
* In the de-duplication mode, when found matching md5 checksum also readMaxim Sobolev2016-03-132-3/+48
| | | | | | | | back block and compare actual content. Just output original block instead of back reference in the unlikely event of collision. Notes: svn path=/head/; revision=296810
* When -S is specified dump summary to stdout, not stderr, so it'sMaxim Sobolev2016-03-101-5/+10
| | | | | | | easier to capture and process it with external tools via pipe. Notes: svn path=/head/; revision=296628
* Add -S option to print out summary after compression has beenMaxim Sobolev2016-03-102-3/+12
| | | | | | | | | completed. MFC after: 2 weeks Notes: svn path=/head/; revision=296626
* DIRDEPS_BUILD: Update dependencies.Bryan Drewery2016-02-241-0/+3
| | | | | | | Sponsored by: EMC / Isilon Storage Division Notes: svn path=/head/; revision=295978
* Improve mkuzip(8) and geom_uzip(4), merge in LZMA support from mkulzma(8)Maxim Sobolev2016-02-2311-70/+717
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | and geom_uncompress(4): 1. mkuzip(8): - Proper support for eliminating all-zero blocks when compressing an image. This feature is already supported by the geom_uzip(4) module and CLOOP format in general, so it's just a matter of making mkuzip(8) match. It should be noted, however that this feature while it sounds great, results in very slight improvement in the overall compression ratio, since compressing default 16k all-zero block produces only 39 bytes compressed output block, which is 99.8% compression ratio. With typical average compression ratio of amd64 binaries and data being around 60-70% the difference between 99.8% and 100.0% is not that great further diluted by the ratio of number of zero blocks in the uncompressed image to the overall number of blocks being less than 0.5 (typically). However, this may be important from performance standpoint, so that kernel are not spinning its wheels decompressing those empty blocks every time this zero region is read. It could also be important when you create huge image mostly filled with zero blocks for testing purposes. - New feature allowing to de-duplicate output image. It turns out that if you twist CLOOP format a bit you can do that as well. And unlike zero-blocks elimination, this gives a noticeable improvement in the overall compression ratio, reducing output image by something like 3-4% on my test UFS2 3GB image consisting of full FreeBSD base system plus some of the packages (openjdk, apache etc), about 2.3GB worth of file data (800+MB compressed). The only caveat is that images created with this feature "on" would not work on older versions of FeeBSDxi kernel, hence it's turned off by default. - provide options to control both features and document them in manual page. - merge in all relevant LZMA compression support from the mkulzma(8), add new option to select between both. - switch license from ad-hoc beerware into standard 2-clause BSD. 2. geom_uzip(4): - implement support for de-duplicated images; - optimize some code paths to handle "all-zero" blocks without reading any compressed data; - beef up manual page to explain that geom_uzip(4) is not limited only to md(4) images. The compressed data can be written to the block device and accessed directly via magic of GEOM(4) and devfs(4), including to mount root fs from a compressed drive. - convert debug log code from being compiled in conditionally into being present all the time and provide two sysctls to turn it on or off. Due to intended use of the module, it can be used in environments where there may not be a luxury to put new kernel with debug code enabled. Having those options handy allows debug issues without as much problem by just having access to serial console or network shell access to a box/appliance. The resulting additional CPU cycles are just few int comparisons and branches, and those are minuscule when compared to data decompression which is the main feature of the module. - hopefully improve robustness and resiliency of the geom_uzip(4) by performing some of the data validation / range checking on the TOC entries and rejecting to attach to an image if those checks fail. - merge in all relevant LZMA decompression support from the geom_uncompress(4), enable automatically when appropriate format is indicated in the header. - move compilation work into its own worker thread so that it does not clog g_up. This allows multiple instances work in parallel utilizing smp cores. - document new knobs in the manual page. Reviewed by: adrian MFC after: 1 month Differential Revision: https://reviews.freebsd.org/D5333 Notes: svn path=/head/; revision=295943
* Add META_MODE support.Simon J. Gerraty2015-06-131-0/+19
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | Off by default, build behaves normally. WITH_META_MODE we get auto objdir creation, the ability to start build from anywhere in the tree. Still need to add real targets under targets/ to build packages. Differential Revision: D2796 Reviewed by: brooks imp Notes: svn path=/head/; revision=284345
| * dirdeps.mk now sets DEP_RELDIRSimon J. Gerraty2015-06-081-2/+0
| | | | | | | | Notes: svn path=/projects/bmake/; revision=284172
| * Merge sync of headSimon J. Gerraty2015-05-271-3/+1
| |\ | |/ |/| | | Notes: svn path=/projects/bmake/; revision=283595
| * Merge head from 7/28Simon J. Gerraty2014-08-191-1/+1
| |\ | | | | | | | | | Notes: svn path=/projects/bmake/; revision=270164
| * | Updated dependenciesSimon J. Gerraty2014-05-161-1/+0
| | | | | | | | | | | | Notes: svn path=/projects/bmake/; revision=266219
| * | Updated dependenciesSimon J. Gerraty2014-05-101-0/+2
| | | | | | | | | | | | Notes: svn path=/projects/bmake/; revision=265802
| * | Updated dependenciesSimon J. Gerraty2013-03-111-0/+1
| | | | | | | | | | | | Notes: svn path=/projects/bmake/; revision=248169
| * | Updated dependenciesSimon J. Gerraty2013-02-161-2/+0
| | | | | | | | | | | | Notes: svn path=/projects/bmake/; revision=246868
| * | Sync FreeBSD's bmake branch with Juniper's internal bmake branch.Marcel Moolenaar2012-08-221-0/+20
| | | | | | | | | | | | | | | | | | | | | Requested by: Simon Gerraty <sjg@juniper.net> Notes: svn path=/projects/bmake/; revision=239572
* | | Convert to usr.bin/ to LIBADDBaptiste Daroussin2014-11-251-3/+1
| |/ |/| | | | | | | | | | | Reduce overlinking Notes: svn path=/head/; revision=275042
* | use .Mt to mark up email addresses consistently (part3)Baptiste Daroussin2014-06-231-1/+1
|/ | | | | | | | PR: 191174 Submitted by: Franco Fichtner <franco at lastsummer.de> Notes: svn path=/head/; revision=267773
* Fixed an embedded shell script.Ruslan Ermilov2011-05-131-1/+1
| | | | | | | Reviewed by: sobomax Notes: svn path=/head/; revision=221832
* mdoc: consistently spell our email addresses <foo@FreeBSD.org>Ulrich Spörlein2010-05-191-1/+1
| | | | | | | Reviewed by: ru Notes: svn path=/head/; revision=208291
* Build usr.bin/ with WARNS=6 by default.Ed Schouten2010-01-021-1/+0
| | | | | | | Also add some missing $FreeBSD$ to keep svn happy. Notes: svn path=/head/; revision=201386
* Support character device as input file.Max Khon2007-03-061-7/+21
| | | | | | | PR: 103500 Notes: svn path=/head/; revision=167272
* Markup fixes.Ruslan Ermilov2006-09-291-1/+5
| | | | Notes: svn path=/head/; revision=162792
* A few minor corrections to the mkuzip.8 man page.Maxim Sobolev2006-03-171-25/+24
| | | | | | | | PR: 92576 Submitted by: Stefan Bethke Notes: svn path=/head/; revision=156818
* Tell the user exactly where the problem was.Pawel Jakub Dawidek2006-01-301-5/+5
| | | | Notes: svn path=/head/; revision=155074
* - check for geom_uzip module presence using kldstat -m.Max Khon2005-05-111-3/+3
| | | | | | | | | | kldstat -m finds geom_uzip module even if it is compiled in statically. - create output file with x bit set. - build mkuzip on all architectures (verified with "make universe"). - fix typo in info message. Notes: svn path=/head/; revision=146107
* Make WARNS=6 clean, which should make it compiling on amd64.Maxim Sobolev2005-05-022-4/+4
| | | | | | | Submitted by: Matteo Riondato <rionda@gufi.org> Notes: svn path=/head/; revision=145808
* Sort sections.Ruslan Ermilov2005-01-181-2/+2
| | | | Notes: svn path=/head/; revision=140420
* Added the EXIT STATUS section where appropriate.Ruslan Ermilov2005-01-171-1/+1
| | | | Notes: svn path=/head/; revision=140368
* Fix build: s/mkunzip.8/mkuzip.8/Marcel Moolenaar2004-09-121-1/+1
| | | | Notes: svn path=/head/; revision=135092
* Normalize the manpage.Ruslan Ermilov2004-09-111-20/+23
| | | | | | | Reviewed by: sobomax Notes: svn path=/head/; revision=135087
* Normalize the makefile.Ruslan Ermilov2004-09-111-2/+2
| | | | | | | Reviewed by: sobomax Notes: svn path=/head/; revision=135086
* o Print more info in the verbose mode;Maxim Sobolev2004-09-101-31/+32
| | | | | | | | | | | o use zlib(3) function which computes maximum length of the output buffer instead of rolling own version; o allow size of input file to be not multiple of cluster size by applying zero padding. Notes: svn path=/head/; revision=135058
* Clarify/extend in several places and make sure that everything matches reality.Maxim Sobolev2004-09-101-16/+32
| | | | Notes: svn path=/head/; revision=135055
* Add mkuzip(8), non-GPL utility to compress filesystem images for use withMaxim Sobolev2004-09-103-0/+354
geom_uzip module. This is based on utility I wrote some 3 years ago for a hack for md(4), which functionally was close to what geom_uzip does today. Since I don't have a time to test that it compiles/works on other arches, stick it to i386 only. Will do it later. Unlike original cloop util, this one embedds FreeBSD-compatible shell code into the generated image, not Linux one. Unfortunately severe space restriction imposed by the CLOOP format doesn't allow to put conditional code which will work both on Linux and FreeBSD. In fact it was quite a challenge to fit necessary FreeBSD code into 127 bytes. ;-) Notes: svn path=/head/; revision=135045
generated by cgit v1.2.3 (git 2.25.1) at 2024-06-13 19:14:24 +0000