summaryrefslogtreecommitdiff
path: root/docs/SourceBasedCodeCoverage.rst
diff options
context:
space:
mode:
authorDimitry Andric <dim@FreeBSD.org>2016-07-23 20:44:14 +0000
committerDimitry Andric <dim@FreeBSD.org>2016-07-23 20:44:14 +0000
commit2b6b257f4e5503a7a2675bdb8735693db769f75c (patch)
treee85e046ae7003fe3bcc8b5454cd0fa3f7407b470 /docs/SourceBasedCodeCoverage.rst
parentb4348ed0b7e90c0831b925fbee00b5f179a99796 (diff)
Notes
Diffstat (limited to 'docs/SourceBasedCodeCoverage.rst')
-rw-r--r--docs/SourceBasedCodeCoverage.rst237
1 files changed, 237 insertions, 0 deletions
diff --git a/docs/SourceBasedCodeCoverage.rst b/docs/SourceBasedCodeCoverage.rst
new file mode 100644
index 000000000000..8d0a51fd3315
--- /dev/null
+++ b/docs/SourceBasedCodeCoverage.rst
@@ -0,0 +1,237 @@
+==========================
+Source-based Code Coverage
+==========================
+
+.. contents::
+ :local:
+
+Introduction
+============
+
+This document explains how to use clang's source-based code coverage feature.
+It's called "source-based" because it operates on AST and preprocessor
+information directly. This allows it to generate very precise coverage data.
+
+Clang ships two other code coverage implementations:
+
+* :doc:`SanitizerCoverage` - A low-overhead tool meant for use alongside the
+ various sanitizers. It can provide up to edge-level coverage.
+
+* gcov - A GCC-compatible coverage implementation which operates on DebugInfo.
+
+From this point onwards "code coverage" will refer to the source-based kind.
+
+The code coverage workflow
+==========================
+
+The code coverage workflow consists of three main steps:
+
+* Compiling with coverage enabled.
+
+* Running the instrumented program.
+
+* Creating coverage reports.
+
+The next few sections work through a complete, copy-'n-paste friendly example
+based on this program:
+
+.. code-block:: cpp
+
+ % cat <<EOF > foo.cc
+ #define BAR(x) ((x) || (x))
+ template <typename T> void foo(T x) {
+ for (unsigned I = 0; I < 10; ++I) { BAR(I); }
+ }
+ int main() {
+ foo<int>(0);
+ foo<float>(0);
+ return 0;
+ }
+ EOF
+
+Compiling with coverage enabled
+===============================
+
+To compile code with coverage enabled, pass ``-fprofile-instr-generate
+-fcoverage-mapping`` to the compiler:
+
+.. code-block:: console
+
+ # Step 1: Compile with coverage enabled.
+ % clang++ -fprofile-instr-generate -fcoverage-mapping foo.cc -o foo
+
+Note that linking together code with and without coverage instrumentation is
+supported: any uninstrumented code simply won't be accounted for.
+
+Running the instrumented program
+================================
+
+The next step is to run the instrumented program. When the program exits it
+will write a **raw profile** to the path specified by the ``LLVM_PROFILE_FILE``
+environment variable. If that variable does not exist, the profile is written
+to ``default.profraw`` in the current directory of the program. If
+``LLVM_PROFILE_FILE`` contains a path to a non-existent directory, the missing
+directory structure will be created. Additionally, the following special
+**pattern strings** are rewritten:
+
+* "%p" expands out to the process ID.
+
+* "%h" expands out to the hostname of the machine running the program.
+
+* "%Nm" expands out to the instrumented binary's signature. When this pattern
+ is specified, the runtime creates a pool of N raw profiles which are used for
+ on-line profile merging. The runtime takes care of selecting a raw profile
+ from the pool, locking it, and updating it before the program exits. If N is
+ not specified (i.e the pattern is "%m"), it's assumed that ``N = 1``. N must
+ be between 1 and 9. The merge pool specifier can only occur once per filename
+ pattern.
+
+.. code-block:: console
+
+ # Step 2: Run the program.
+ % LLVM_PROFILE_FILE="foo.profraw" ./foo
+
+Creating coverage reports
+=========================
+
+Raw profiles have to be **indexed** before they can be used to generate
+coverage reports. This is done using the "merge" tool in ``llvm-profdata``, so
+named because it can combine and index profiles at the same time:
+
+.. code-block:: console
+
+ # Step 3(a): Index the raw profile.
+ % llvm-profdata merge -sparse foo.profraw -o foo.profdata
+
+There are multiple different ways to render coverage reports. One option is to
+generate a line-oriented report:
+
+.. code-block:: console
+
+ # Step 3(b): Create a line-oriented coverage report.
+ % llvm-cov show ./foo -instr-profile=foo.profdata
+
+To demangle any C++ identifiers in the output, use:
+
+.. code-block:: console
+
+ % llvm-cov show ./foo -instr-profile=foo.profdata | c++filt -n
+
+This report includes a summary view as well as dedicated sub-views for
+templated functions and their instantiations. For our example program, we get
+distinct views for ``foo<int>(...)`` and ``foo<float>(...)``. If
+``-show-line-counts-or-regions`` is enabled, ``llvm-cov`` displays sub-line
+region counts (even in macro expansions):
+
+.. code-block:: none
+
+ 20| 1|#define BAR(x) ((x) || (x))
+ ^20 ^2
+ 2| 2|template <typename T> void foo(T x) {
+ 22| 3| for (unsigned I = 0; I < 10; ++I) { BAR(I); }
+ ^22 ^20 ^20^20
+ 2| 4|}
+ ------------------
+ | void foo<int>(int):
+ | 1| 2|template <typename T> void foo(T x) {
+ | 11| 3| for (unsigned I = 0; I < 10; ++I) { BAR(I); }
+ | ^11 ^10 ^10^10
+ | 1| 4|}
+ ------------------
+ | void foo<float>(int):
+ | 1| 2|template <typename T> void foo(T x) {
+ | 11| 3| for (unsigned I = 0; I < 10; ++I) { BAR(I); }
+ | ^11 ^10 ^10^10
+ | 1| 4|}
+ ------------------
+
+It's possible to generate a file-level summary of coverage statistics (instead
+of a line-oriented report) with:
+
+.. code-block:: console
+
+ # Step 3(c): Create a coverage summary.
+ % llvm-cov report ./foo -instr-profile=foo.profdata
+ Filename Regions Miss Cover Functions Executed
+ -----------------------------------------------------------------------
+ /tmp/foo.cc 13 0 100.00% 3 100.00%
+ -----------------------------------------------------------------------
+ TOTAL 13 0 100.00% 3 100.00%
+
+A few final notes:
+
+* The ``-sparse`` flag is optional but can result in dramatically smaller
+ indexed profiles. This option should not be used if the indexed profile will
+ be reused for PGO.
+
+* Raw profiles can be discarded after they are indexed. Advanced use of the
+ profile runtime library allows an instrumented program to merge profiling
+ information directly into an existing raw profile on disk. The details are
+ out of scope.
+
+* The ``llvm-profdata`` tool can be used to merge together multiple raw or
+ indexed profiles. To combine profiling data from multiple runs of a program,
+ try e.g:
+
+ .. code-block:: console
+
+ % llvm-profdata merge -sparse foo1.profraw foo2.profdata -o foo3.profdata
+
+Format compatibility guarantees
+===============================
+
+* There are no backwards or forwards compatibility guarantees for the raw
+ profile format. Raw profiles may be dependent on the specific compiler
+ revision used to generate them. It's inadvisable to store raw profiles for
+ long periods of time.
+
+* Tools must retain **backwards** compatibility with indexed profile formats.
+ These formats are not forwards-compatible: i.e, a tool which uses format
+ version X will not be able to understand format version (X+k).
+
+* There is a third format in play: the format of the coverage mappings emitted
+ into instrumented binaries. Tools must retain **backwards** compatibility
+ with these formats. These formats are not forwards-compatible.
+
+Using the profiling runtime without static initializers
+=======================================================
+
+By default the compiler runtime uses a static initializer to determine the
+profile output path and to register a writer function. To collect profiles
+without using static initializers, do this manually:
+
+* Export a ``int __llvm_profile_runtime`` symbol from each instrumented shared
+ library and executable. When the linker finds a definition of this symbol, it
+ knows to skip loading the object which contains the profiling runtime's
+ static initializer.
+
+* Forward-declare ``void __llvm_profile_initialize_file(void)`` and call it
+ once from each instrumented executable. This function parses
+ ``LLVM_PROFILE_FILE``, sets the output path, and truncates any existing files
+ at that path. To get the same behavior without truncating existing files,
+ pass a filename pattern string to ``void __llvm_profile_set_filename(char
+ *)``. These calls can be placed anywhere so long as they precede all calls
+ to ``__llvm_profile_write_file``.
+
+* Forward-declare ``int __llvm_profile_write_file(void)`` and call it to write
+ out a profile. This function returns 0 when it succeeds, and a non-zero value
+ otherwise. Calling this function multiple times appends profile data to an
+ existing on-disk raw profile.
+
+Drawbacks and limitations
+=========================
+
+* Code coverage does not handle unpredictable changes in control flow or stack
+ unwinding in the presence of exceptions precisely. Consider the following
+ function:
+
+ .. code-block:: cpp
+
+ int f() {
+ may_throw();
+ return 0;
+ }
+
+ If the call to ``may_throw()`` propagates an exception into ``f``, the code
+ coverage tool may mark the ``return`` statement as executed even though it is
+ not. A call to ``longjmp()`` can have similar effects.