FreeBSD source tree https://cgit-dev.freebsd.org/src/atom?h=zfs-0.5.2 2010-11-11T19:40:06Z 2010-11-11T19:40:06Z Brian Behlendorf behlendorf1@llnl.gov 2010-11-11T18:22:36Z urn:sha1:e0f3df67e5174cd51284f0c26779046facfc95e0 When adding this functionality originally the options to only run specific tests (-t), or conversely skip specific tests (-s) were omitted from the usage page. This commit adds the missing documentation. 2010-11-11T19:40:06Z Brian Behlendorf behlendorf1@llnl.gov 2010-11-11T18:17:02Z urn:sha1:7dc3830c0f591ae8dfe349cf67c0c61cc8602dc8 The idea behind the '-c' flag is to cleanup everything from a previous test run which might cause the test script to fail. This should also include removing the previously loaded module. This makes it a little easier to run 'zconfig.sh -c', however remember this is a test script and it will take all of your other zpools offline for the purposes of the test. This notion has also been extended to the default 'make check' behavior. 2010-11-11T19:38:25Z Brian Behlendorf behlendorf1@llnl.gov 2010-11-11T18:00:39Z urn:sha1:cf47fad67d798f42e6d2edc8b0e9f9b43175703b Loading and unloading the zlib modules as part of the zfs.sh script has proven a little problematic for a few reasons. * First, your kernel may not need to load either zlib_inflate or zlib_deflate. This functionality may be built directly in to your kernel. It depends entirely on what your distribution decided was the right thing to do. * Second, even if you do manage to load the correct modules you may not be able to unload them. There may other consumers of the modules with a reference preventing the unload. To avoid both of these issues the test scripts have been updated to attempt to unconditionally load all modules listed in KERNEL_MODULES. If the module is successfully loaded you must have needed it. If the module can't be loaded that almost certainly means either it is built in to your kernel or is already being used by another consumer. In both cases this is not an issue and we can move on to the spl/zfs modules. Finally, by removing these kernel modules from the MODULES list we ensure they are never unloaded during 'zfs.sh -u'. This avoids the issue of the script failing because there is another consumer using the module we were not aware of. In other words the script restricts unloading modules to only the spl/zfs modules. Closes #78 2010-11-08T22:03:36Z Brian Behlendorf behlendorf1@llnl.gov 2010-11-05T19:29:58Z urn:sha1:8c3ab23f4be92a4e55f07f8c1bb467b851ed3e54 The lustre zpios-test simulates a reasonable lustre workload. It will create 128 threads, the same as a Lustre OSS, and then 4096 individual objects. Each objects is 16MiB in size and will be written/read in 1MiB from a random thread. This is fundamentally how we expect Lustre to behave for large IO intensive workloads. 2010-11-08T22:03:36Z Brian Behlendorf behlendorf1@llnl.gov 2010-11-05T18:43:20Z urn:sha1:cb39a6c6aa91531a641232e7d0c6a4d63836b0cc To streamline testing I have in the past added several custom configs to the zpool-config directory. This change reverts those custom configs and replaces them with three generic config which can do the same thing. The generic config behavior can be set by setting various environment variables when calling either the zpool-create.sh or zpios.sh scripts. For example if you wanted to create and test a single 4-disk Raid-Z2 configuration using disks [A-D]1 with dedicated ZIL and L2ARC devices you could run the following. $ ZIL="log A2" L2ARC="cache B2" RANKS=1 CHANNELS=4 LEVEL=2 \ zpool-create.sh -c zpool-raidz $ zpool status tank pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 A1 ONLINE 0 0 0 B1 ONLINE 0 0 0 C1 ONLINE 0 0 0 D1 ONLINE 0 0 0 logs A2 ONLINE 0 0 0 cache B2 ONLINE 0 0 0 errors: No known data errors 2010-10-22T19:41:57Z Ned Bass bass6@llnl.gov 2010-10-18T18:07:20Z urn:sha1:d4055aac3c4fdb6c1b5b1a7b73d20ff716e0a711 This test performs a sanity check of the zpool add and remove commands. It tests adding and removing both a cache disk and a log disk to and from a zpool. Usage of both a shorthand device path and a full path is covered. The test uses a scsi_debug device as the disk to be added and removed. This is done so that zpool will see it as a whole disk and partition it, which it does not currently done for loopback devices. We want to verify that the manipulation done to whole disks paths to hide the parition information does not break the add/remove interface. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> 2010-10-12T22:20:03Z Brian Behlendorf behlendorf1@llnl.gov 2010-09-28T23:32:12Z urn:sha1:0ee8118bd31d1c160123d0aac9c55455706d5975 Eleven new zpool configurations were added to allow testing of various failure cases. The first 5 zpool configurations leverage the 'faulty' md device type which allow us to simuluate IO errors at the block layer. The last 6 zpool configurations leverage the scsi_debug module provided by modern kernels. This device allows you to create virtual scsi devices which are backed by a ram disk. With this setup we can verify the full IO stack by injecting faults at the lowest layer. Both methods of fault injection are important to verifying the IO stack. The zfs code itself also provides a mechanism for error injection via the zinject command line tool. While we should also take advantage of this appraoch to validate the code it does not address any of the Linux integration issues which are the most concerning. For the moment we're trusting that the upstream Solaris guys are running zinject and would have caught internal zfs logic errors. Currently, there are 6 r/w test cases layered on top of the 'faulty' md devices. They include 3 writes tests for soft/transient errors, hard/permenant errors, and all writes error to the device. There are 3 matching read tests for soft/transient errors, hard/permenant errors, and fixable read error with a write. Although for this last case zfs doesn't do anything special. The seventh test case verifies zfs detects and corrects checksum errors. In this case one of the drives is extensively damaged and by dd'ing over large sections of it. We then ensure zfs logs the issue and correctly rebuilds the damage. The next test cases use the scsi_debug configuration to injects error at the bottom of the scsi stack. This ensures we find any flaws in the scsi midlayer or our usage of it. Plus it stresses the device specific retry, timeout, and error handling outside of zfs's control. The eighth test case is to verify that the system correctly handles an intermittent device timeout. Here the scsi_debug device drops 1 in N requests resulting in a retry either at the block level. The ZFS code does specify the FAILFAST option but it turns out that for this case the Linux IO stack with still retry the command. The FAILFAST logic located in scsi_noretry_cmd() does no seem to apply to the simply timeout case. It appears to be more targeted to specific device or transport errors from the lower layers. The ninth test case handles a persistent failure in which the device is removed from the system by Linux. The test verifies that the failure is detected, the device is made unavailable, and then can be successfully re-add when brought back online. Additionally, it ensures that errors and events are logged to the correct places and the no data corruption has occured due to the failure. 2010-10-12T21:55:02Z Brian Behlendorf behlendorf1@llnl.gov 2010-10-01T17:57:56Z urn:sha1:2959d94a0a53612cc1ca9ce9d17df26c3d69a513 ZFS works best when it is notified as soon as possible when a device failure occurs. This allows it to immediately start any recovery actions which may be needed. In theory Linux supports a flag which can be set on bio's called FAILFAST which provides this quick notification by disabling the retry logic in the lower scsi layers. That's the theory at least. In practice is turns out that while the flag exists you oddly have to set it with the BIO_RW_AHEAD flag. And even when it's set it you may get retries in the low level drivers decides that's the right behavior, or if you don't get the right error codes reported to the scsi midlayer. Unfortunately, without additional kernels patchs there's not much which can be done to improve this. Basically, this just means that it may take 2-3 minutes before a ZFS is notified properly that a device has failed. This can be improved and I suspect I'll be submitting patches upstream to handle this. 2010-09-17T23:24:15Z Brian Behlendorf behlendorf1@llnl.gov 2010-09-17T23:22:08Z urn:sha1:12f30129745a3da85b6bbc31e24d3738ff007d9b The scripts/zpool-layout/Makefile.in file generated by autogen.sh was accidentally omitted from the previous commit. Add it. 2010-09-17T18:02:19Z Brian Behlendorf behlendorf1@llnl.gov 2010-09-15T22:32:27Z urn:sha1:a5b4d63582005300c4526c7c15b63294e5408534 By default the zpool_layout command would always use the slot number assigned by Linux when generating the zdev.conf file. This is a reasonable default there are cases when it makes sense to remap the slot id assigned by Linux using your own custom mapping. This commit adds support to zpool_layout to provide a custom slot mapping file. The file contains in the first column the Linux slot it and in the second column the custom slot mapping. By passing this map file with '-m map' to zpool_config the mapping will be applied when generating zdev.conf. Additionally, two sample mapping have been added which reflect different ways to map the slots in the dragon drawers.