---
title: Chapter 6. Regression and Performance Testing
authors:
prev: books/developers-handbook/policies
next: books/developers-handbook/partii
description: Regression and Performance Testing
tags: ["Regression", "Performance Testing", "Testing", "Tinderbox"]
showBookMenu: true
weight: 7
path: "/books/developers-handbook/"
---

[[testing]]
= Regression and Performance Testing
:doctype: book
:toc: macro
:toclevels: 1
:icons: font
:sectnums:
:sectnumlevels: 6
:sectnumoffset: 6
:partnums:
:source-highlighter: rouge
:experimental:
:images-path: books/developers-handbook/

ifdef::env-beastie[]
ifdef::backend-html5[]
:imagesdir: ../../../../images/{images-path}
endif::[]
ifndef::book[]
include::shared/authors.adoc[]
include::shared/mirrors.adoc[]
include::shared/releases.adoc[]
include::shared/attributes/attributes-{{% lang %}}.adoc[]
include::shared/{{% lang %}}/teams.adoc[]
include::shared/{{% lang %}}/mailing-lists.adoc[]
include::shared/{{% lang %}}/urls.adoc[]
toc::[]
endif::[]
ifdef::backend-pdf,backend-epub3[]
include::../../../../../shared/asciidoctor.adoc[]
endif::[]
endif::[]

ifndef::env-beastie[]
toc::[]
include::../../../../../shared/asciidoctor.adoc[]
endif::[]

Regression tests are used to exercise a particular bit of the system to check that it works as expected, and to make sure that old bugs are not reintroduced.

The FreeBSD regression testing tools can be found in the FreeBSD source tree in the directory [.filename]#src/tools/regression#.

[[testing-micro-benchmark]]
== Micro Benchmark Checklist

This section contains hints for doing proper micro-benchmarking on FreeBSD or of FreeBSD itself.

It is not possible to use all of the suggestions below every single time, but the more used, the better the benchmark's ability to test small differences will be.

* Disable APM and any other kind of clock fiddling (ACPI ?).
* Run in single user mode. E.g., man:cron[8], and other daemons only add noise. The man:sshd[8] daemon can also cause problems. If ssh access is required during testing either disable the SSHv1 key regeneration, or kill the parent `sshd` daemon during the tests.
* Do not run man:ntpd[8].
* If man:syslog[3] events are generated, run man:syslogd[8] with an empty [.filename]#/etc/syslogd.conf#, otherwise, do not run it.
* Minimize disk-I/O, avoid it entirely if possible.
* Do not mount file systems that are not needed.
* Mount [.filename]#/#, [.filename]#/usr#, and any other file system as read-only if possible. This removes atime updates to disk (etc.) from the I/O picture.
* Reinitialize the read/write test file system with man:newfs[8] and populate it from a man:tar[1] or man:dump[8] file before every run. Unmount and mount it before starting the test. This results in a consistent file system layout. For a worldstone test this would apply to [.filename]#/usr/obj# (just reinitialize with `newfs` and mount). To get 100% reproducibility, populate the file system from a man:dd[1] file (i.e.: `dd if=myimage of=/dev/ad0s1h bs=1m`)
* Use malloc backed or preloaded man:md[4] partitions.
* Reboot between individual iterations of the test, this gives a more consistent state.
* Remove all non-essential device drivers from the kernel. For instance if USB is not needed for the test, do not put USB in the kernel. Drivers which attach often have timeouts ticking away.
* Unconfigure hardware that are not in use. Detach disks with man:atacontrol[8] and man:camcontrol[8] if the disks are not used for the test.
* Do not configure the network unless it is being tested, or wait until after the test has been performed to ship the results off to another computer.
+
If the system must be connected to a public network, watch out for spikes of broadcast traffic. Even though it is hardly noticeable, it will take up CPU cycles. Multicast has similar caveats.
* Put each file system on its own disk. This minimizes jitter from head-seek optimizations.
* Minimize output to serial or VGA consoles. Running output into files gives less jitter. (Serial consoles easily become a bottleneck.) Do not touch keyboard while the test is running, even kbd:[space] or kbd:[back-space] shows up in the numbers.
* Make sure the test is long enough, but not too long. If the test is too short, timestamping is a problem. If it is too long temperature changes and drift will affect the frequency of the quartz crystals in the computer. Rule of thumb: more than a minute, less than an hour.
* Try to keep the temperature as stable as possible around the machine. This affects both quartz crystals and disk drive algorithms. To get real stable clock, consider stabilized clock injection. E.g., get a OCXO + PLL, inject output into clock circuits instead of motherboard xtal. Contact {phk} for more information about this.
* Run the test at least 3 times but it is better to run more than 20 times both for "before" and "after" code. Try to interleave if possible (i.e.: do not run 20 times before then 20 times after), this makes it possible to spot environmental effects. Do not interleave 1:1, but 3:3, this makes it possible to spot interaction effects.
+
A good pattern is: `bababa{bbbaaa}*`.
This gives hint after the first 1+1 runs (so it is possible to stop the test if it goes entirely the wrong way), a standard deviation after the first 3+3 (gives a good indication if it is going to be worth a long run) and trending and interaction numbers later on.
* Use man:ministat[1] to see if the numbers are significant. Consider buying "Cartoon guide to statistics" ISBN: 0062731025, highly recommended, if you have forgotten or never learned about standard deviation and Student's T.
* Do not use background man:fsck[8] unless the test is a benchmark of background `fsck`. Also, disable `background_fsck` in [.filename]#/etc/rc.conf# unless the benchmark is not started at least 60+"``fsck`` runtime" seconds after the boot, as man:rc[8] wakes up and checks if `fsck` needs to run on any file systems when background `fsck` is enabled. Likewise, make sure there are no snapshots lying around unless the benchmark is a test with snapshots.
* If the benchmark show unexpected bad performance, check for things like high interrupt volume from an unexpected source. Some versions of ACPI have been reported to "misbehave" and generate excess interrupts. To help diagnose odd test results, take a few snapshots of `vmstat -i` and look for anything unusual.
* Make sure to be careful about optimization parameters for kernel and userspace, likewise debugging. It is easy to let something slip through and realize later the test was not comparing the same thing.
* Do not ever benchmark with the `WITNESS` and `INVARIANTS` kernel options enabled unless the test is interested to benchmarking those features. `WITNESS` can cause 400%+ drops in performance. Likewise, userspace man:malloc[3] parameters default differently in -CURRENT from the way they ship in production releases.

[[testing-tinderbox]]
== The FreeBSD Source Tinderbox

The source Tinderbox consists of:

* A build script, [.filename]#tinderbox#, that automates checking out a specific version of the FreeBSD source tree and building it.
* A supervisor script, [.filename]#tbmaster#, that monitors individual Tinderbox instances, logs their output, and emails failure notices.
* A CGI script named [.filename]#index.cgi# that reads a set of tbmaster logs and presents an easy-to-read HTML summary of them.
* A set of build servers that continually test the tip of the most important FreeBSD code branches.
* A webserver that keeps a complete set of Tinderbox logs and displays an up-to-date summary.

The scripts are maintained and were developed by {des}, and are now written in Perl, a move on from their original incarnation as shell scripts.
All scripts and configuration files are kept in https://www.freebsd.org/cgi/cvsweb.cgi/projects/tinderbox/[/projects/tinderbox/].

For more information about the tinderbox and tbmaster scripts at this stage, see their respective man pages: tinderbox(1) and tbmaster(1).

== The index.cgi Script

The [.filename]#index.cgi# script generates the HTML summary of tinderbox and tbmaster logs.
Although originally intended to be used as a CGI script, as indicated by its name,
this script can also be run from the command line or from a man:cron[8] job, in which case it will look for logs in the directory where the script is located.
It will automatically detect context, generating HTTP headers when it is run as a CGI script.
It conforms to XHTML standards and is styled using CSS.

The script starts in the `main()` block by attempting to verify that it is running on the official Tinderbox website.
If it is not, a page indicating it is not an official website is produced, and a URL to the official site is provided.

Next, it scans the log directory to get an inventory of configurations, branches and architectures for which log files exist, to avoid hard-coding a list into the script and potentially ending up with blank rows or columns.
This information is derived from the names of the log files matching the following pattern:

[.programlisting]
....
tinderbox-$config-$branch-$arch-$machine.{brief,full}
....

The configurations used on the official Tinderbox build servers are named for the branches they build.
For example, the `releng_8` configuration is used to build `RELENG_8` as well as all still-supported release branches.

Once all of this startup procedure has been successfully completed, `do_config()` is called for each configuration.

The `do_config()` function generates HTML for a single Tinderbox configuration.

It works by first generating a header row, then iterating over each branch build with the specified configuration,
producing a single row of results for each in the following manner:

* For each item:

** For each machine within that architecture:

*** If a brief log file exists, then:

**** Call `success()` to determine the outcome of the build.
**** Output the modification size.
**** Output the size of the brief log file with a link to the log file itself.
**** If a full log file also exists, then:

***** Output the size of the full log file with a link to the log file itself.

*** Otherwise:

**** No output.

The `success()` function mentioned above scans a brief log file for the string "tinderbox run completed" in order to determine whether the build was successful.

Configurations and branches are sorted according to their branch rank.
This is computed as follows:

* `HEAD` and `CURRENT` have rank 9999.
* `RELENG_x` has rank __``xx``__99.
* `RELENG_x_y` has rank _xxyy_.

This means that `HEAD` always ranks highest, and `RELENG` branches are ranked in numerical order, with each `STABLE` branch ranking higher than the release branches forked off of it.
For instance, for FreeBSD 8, the order from highest to lowest would be:

* `RELENG_8` (branch rank 899).
* `RELENG_8_3` (branch rank 803).
* `RELENG_8_2` (branch rank 802).
* `RELENG_8_1` (branch rank 801).
* `RELENG_8_0` (branch rank 800).

The colors that Tinderbox uses for each cell in the table are defined by CSS.
Successful builds are displayed with green text; unsuccessful builds are displayed with red text.
The color fades as time passes since the corresponding build, with every half an hour bringing the color closer to grey.

== Official Build Servers

The official Tinderbox build servers are hosted by http://www.sentex.ca[Sentex Data Communications], who also host the FreeBSD Netperf Cluster.

Three build servers currently exist:

_freebsd-current.sentex.ca_ builds:

* `HEAD` for amd64, arm, i386, i386/pc98, ia64, mips, powerpc, powerpc64, and sparc64.
* `RELENG_9` and supported 9._X_ branches for amd64, arm, i386, i386/pc98, ia64, mips, powerpc, powerpc64, and sparc64.

_freebsd-stable.sentex.ca_ builds:

* `RELENG_8` and supported 8._X_ branches for amd64, i386, i386/pc98, ia64, mips, powerpc and sparc64.

_freebsd-legacy.sentex.ca_ builds:

* `RELENG_7` and supported 7._X_ branches for amd64, i386, i386/pc98, ia64, powerpc, and sparc64.

== Official Summary Site

Summaries and logs from the official build servers are available online at http://tinderbox.FreeBSD.org[http://tinderbox.FreeBSD.org], hosted by {des} and set up as follows:

* A man:cron[8] job checks the build servers at regular intervals and downloads any new log files using man:rsync[1].
* Apache is set up to use [.filename]#index.cgi# as `DirectoryIndex`.
