# Copyright (C) 2006-2008 Junjiro Okajima # Copyright (C) 2006-2008 Tomas Matejicek, slax.org # # LICENSE follows the described ones in lzma and squashfs. # $Id: sqlzma.txt,v 1.37 2008-03-12 15:16:50 jro Exp $ LZMA patch for squashfs Junjiro Okajima o What is this? This is a patch against lzma449.tar.bz2 and squashfs3.3.tar.gz, and makes squashfs to support both of LZMA compression and ZLIB compression. The new squashfs image file (generated without '-nolzma' option) has no backward compatibility, while the patched squashfs and its tools can handle the old squashfs image generated by the unpatched squashfs-tools. This patch is developed and tested on linux-2.6.24.3. o Files - Makefile a toplevel makefile which builds everything. it is recommended to use this makefile. - sqlzma.h a global header file. this file is used from kernel and userspace utilities. - sqmagic.h defines patched squashfs magic number. - sqlzma1-XXX.patch a patch against lzma package. currently it is lzma449.tar.bz2 and the 'XXX in filename is '449'. - sqlzma2k-YYY.patch a patch against squashfs kernel source. currently it is squashfs3.3.tar.gz and the 'YYY' in filename is '3.3'. - sqlzma2u-YYY.patch a patch against squashfs utility. o Download, Extract, Patch and Build - choose the lzma version you use, and modify ./Makefile. - mkdir lzmaXXX, get SourceForge/sevenzip/lzmaXXX.tar.bz2, and extract it into lzmaXXX dir. - apply sqlzma1-XXX.patch to lzmaXXX. - get SourceForge/squashfs/squashfsYYY.tar.gz, extract, and get squashfsYYY/. - apply squashfsYYY/kernel-patches/linux-2.6.24.3/squashfsYYY-patch to your kernel source, and then also apply sqlzma2k-YYY.patch to it. - apply sqlzma2u-YYY.patch to squashfsYYY. - make them. if you have lzmaXXX and squashfsYYY in the same dir, a toplevel Makefile in this patchset builds everything for you. If you want to change their path, edit the variable definitions in ./Makefile. - It builds these things. + ${Squashfs}/kernel-patches/${KVer}/fs/squashfs/ * squashfs.ko a patched squashfs which supports LZMA uncompression. If you didn't load unlzma.ko and sqlzma.ko before compilation of squashfs.ko, the linker may produce some warnings about the functions in sqlzma.ko. But you can ignore them in safe. + ${Lzma}/C/7zip/Compress/LZMA_C/ * libunlzma.a LZMA uncompression user library for squashfs utilities. * libunlzma_r.a thread-safe one. + ${Lzma}/C/7zip/Compress/LZMA_C/kmod/ * unlzma.ko generic LZMA uncompression module for linux kernel. * sqlzma.ko LZMA uncompression kernel module for squashfs, a bridge library between squashfs and unlzma.ko. + ${Lzma}/C/7zip/Compress/LZMA_Alone/ * liblzma_r.a thread-safe LZMA compression user library for squashfs utilities. + ${Squashfs}/squashfs-tools/ * mksquashfs, unsquashfs patched squashfs utilities which support LZMA compression. - copy the modules squashfs.ko, unlzma.ko and sqlzma.ko to wherever you like. - load these modules. - create your filesystem image by mksquashfs. - mount it and enjoy. o Added options - mksquashfs -lzmadic N This option sets the dictionary size for LZMA compression. The default value is same as the block size, even if you change the block size by specifying -b (block size) option for mksquashfs. The dictionary size always follows it by default. If you want to set a different value from the block size, then use this option ALONE or AFTER -b option. In append mode, you can't change the block size, but the LZMA dictionary size since the compression is done for every and each block. The syntax is equivalent ot -b option, ie. you can use K (kilobyte) or M (megabyte). - mksquashfs -nolzma This option makes mksquashfs not to try LZMA compression. And it uses only ZLIB compression, as it was unpatched. - no options are added to unsquashfs. the patched unsquashfs automatically detects whether the given squashfs image uses LZMA compression or not. o LZMA and ZLIB There are three groups of byte stream in compression. - both of LZMA and ZLIB cannot compress. - LZMA gets better compression. - ZLIB gets better compression. This patch supports both compression methods. The patched mksquashfs tries ZLIB compression first. If the data cannot be compressed, store the data uncompressed as unpatched mksquashfs does. If it succeeds and -nolzma option is not specified, then keep the compressed data and try LZMA compression. After LZMA compression, compares the sizes of the two compressed data and discard the larger one. Honestly speaking, I do know nothing about the compression theory. This is the fact I got from using these libraries. (What is this thing called in English? a rule of thumb?) o Multi-call decompression and parallel decompression While squashfs3.1-r2 took the approach of single-call ZLIB decompression, squashfs3.2 and later takes the approach of multi-call which has an advantage of memory consumption. So I tried implementing multi-call LZMA decompression too, but I gave up. Because of memory usage. While single-call LZMA decompression doesn't require a dictionary, multi-call does. And its size can be mega-bytes (the dictionary size which I checked was 8388608). In the case of single-call LZMA decompression, it requires 64KB buffer at maximum to store the compressed squashfs block. Finally I decided to implement the parallel single-call LZMA decompression. The parallel decompression is also applied to ZLIB decompression. It also requires more memory than multi-call ZLIB decompression, but less than multi-call LZMA decompression. And it can take an advantage of speed performance, especially on SMP machine. o Backporting A user, dystryk, had backported this patch (previous version) to linux-2.6.17. His URL is, https://sourceforge.net/project/showfiles.php?group_id=116780&package_id=222970 Enjoy your smaller squashfs.