Writing Tests

This tutorial introduces the SeqAn test system. Currently, there are two kinds of tests in SeqAn: “Normal” tests written in C++ that test C++ classes and functions and “app tests” that test applications. More information is also available in Writing Tests and Writing App Tests.

Library Tests

Creating Tests Skelletons

Use the skel.py script to create a new test, in this case for the module my_module in the sandbox my_sandbox.

$ ./util/bin/skel.py test my_module sandbox/my_sandbox
Creating test in ./sandbox/my_sandbox/tests/my_module
  Target path is: ./sandbox/my_sandbox/tests/my_module

mkdir(./sandbox/my_sandbox/tests/my_module)

Configuring file.
  Source: ./util/skel/test_template/test.cpp
  Target: ./sandbox/my_sandbox/tests/my_module/test_my_module.cpp

Configuring file.
  Source: ./util/skel/test_template/test.h
  Target: ./sandbox/my_sandbox/tests/my_module/test_my_module.h

Configuring file.
  Source: ./util/skel/test_template/CMakeLists.txt
  Target: ./sandbox/my_sandbox/tests/my_module/CMakeLists.txt

The generated files are:

my_sandbox/tests/my_module/test_my_module.cpp
Contains the testsuite (the main program that calls the tests).
my_sandbox/tests/my_module/test_my_module.h
A header with tests.
my_sandbox/tests/my_module/CMakeLists.txt
The CMake file for building the test.


There always is just one .cpp file with the test suite for calling the tests. Usually, there is one test header for each library header under test. There is one CMakeLists.txt that has to be adjusted if new test headers are added.

Test Suites

Test suites are collection of tests. They are defined using SEQAN_BEGIN_TESTSUITE and SEQAN_END_TESTSUITE as shown below.

Essentially, these macros when used like SEQAN_BEGIN_TESTSUITE(test_suite_name) { /*...*/ } SEQAN_END_TESTSUITE will expand to a main function: int main() { /*initialization boilerplate*/ /*...*/ /*finalization boilerplate*/; return res; }.

#include <seqan/basic.h>
#include <seqan/file.h>

#include "test_my_module.h"

SEQAN_BEGIN_TESTSUITE(test_my_module)
{
    // Call tests.
    SEQAN_CALL_TEST(test_my_module_strings_example1);
}
SEQAN_END_TESTSUITE

The generated test .cpp file includes test_my_module which contains the actual tests (see below). The generated file only contains the test test_my_module_strings_example1 which is called using the SEQAN_CALL_TEST macro.

The macro SEQAN_CALL_TEST expands to a function call that calls the test function generated by SEQAN_DEFINE_TEST (see below). The function call will be wrapped in code that will detect if an assertion fails in the test (or the functions called by the test). Note that only SeqAn assertions can be caught, not standard C assertions using assert() from the <cassert> header.

Tests

Tests are usually grouped and each group is put into one header file. Tests are declared using SEQAN_DEFINE_TEST.

The generated file looks as follows:

#ifndef SANDBOX_MY_SANDBOX_TESTS_MY_MODULE_TEST_MY_MODULE_H_
#define SANDBOX_MY_SANDBOX_TESTS_MY_MODULE_TEST_MY_MODULE_H_

#include <seqan/basic.h>
#include <seqan/sequence.h>

// A test for strings.
SEQAN_DEFINE_TEST(test_my_module_strings_example1)
{
    using namespace seqan;

    // Define some constant test data for comparison...
    CharString const STRING1 = "test 1";
    CharString const STRING2 = "test 2";

    // Append to a string and make equality assertion on the result.
    CharString myStr = "test ";
    append(myStr, "1");
    SEQAN_ASSERT_EQ(STRING1, myStr);

    // Demonstration of other assertions.
    SEQAN_ASSERT_GT(STRING2, myStr);
    SEQAN_ASSERT_GEQ(STRING2, myStr);
    SEQAN_ASSERT_LT(myStr, STRING2);
    SEQAN_ASSERT_LEQ(STRING2, STRING2);
}

#endif  // SANDBOX_MY_SANDBOX_TESTS_MY_MODULE_TEST_MY_MODULE_H_

The macro SEQAN_DEFINE_TEST expands to a function definition. The test functions cann be called using the SEQAN_CALL_TEST macro in your test suite (see above).

Note that we do not use a global using namespace seqan but keep this local to the test functions.

CMakeLists.txt File

The relevant lines from the file my_sandbox/tests/my_module/CMakeLists.txt are as follows:

cmake_minimum_required (VERSION 2.8.2)
project (seqan_core_tests_align)
message (STATUS "Configuring my_sandbox/tests/my_module")

# Search SeqAn and select dependencies.
set (SEQAN_FIND_DEPENDENCIES NONE)
find_package (SeqAn REQUIRED)

# Add include directories.
include_directories (${SEQAN_INCLUDE_DIRS})

# Add definitions set by find_package (SeqAn).
add_definitions (${SEQAN_DEFINITIONS})

# Update the list of file names below if you add source files to your test.
add_executable (test_my_module test_my_module.cpp test_my_module.h)

# Add dependencies found by find_package (SeqAn).
target_link_libraries (test_my_module ${SEQAN_LIBRARIES})

# Add CXX flags found by find_package (SeqAn).
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${SEQAN_CXX_FLAGS}")

# Register with CTest
add_test (NAME test_test_align COMMAND $<TARGET_FILE:test_my_modulen>)

When we add a new test header test_my_module_header.h, we would change the line with add_executable() to read:

add_executable (test_my_module test_my_module.cpp test_my_module.h test_my_module_header.h)

When next building the target test_my_module via make or an IDE, CMake will run automatically. Now, the build system is aware of the new header file test_my_module_header.h and will rebuild the test binary if the file changes.

Building And Running Tests

Now, go to your build directory and re-run CMake so the Makefiles (or your IDE’s project files) include the target for the test.

$ cmake .
...
$ make test_my_module
...
$ ./sandbox/holtgrew/tests/my_module/test_my_module
TEST SUITE test_my_module
SEQAN_ENABLE_DEBUG == 1
SEQAN_ENABLE_TESTING == 1
SEQAN_ENABLE_CHECKPOINTS == 0
SEQAN_CXX_FLAGS == "SEQAN_CXX_FLAGS_NOT_SET"
test_my_module_strings_example1 OK
**************************************
 Total Check Points : 0
 Found Check Points : 0
 Lost Check Points  : 0
--------------------------------------
 Total Tests: 1
 Skipped:     0
 Errors:      0
**************************************

Tests in apps folders

It is also possible to write normal tests inside apps. One example is the extras/apps/param\_chooser/ param\_chooser app.

To add a test to your app, you have to (1) add a test .cpp file and (2) register it in the CMakeLists.txt file.

Writing the test program

The test program looks the same as above, but defines the precompiler symbol SEQAN_ENABLE_TESTING as 1. This has to happen at the top of the file (say the name of the file is test_my_app_funcs.cpp).

#undef SEQAN_ENABLE_TESTING
#define SEQAN_ENABLE_TESTING 1

#include <seqan/basic.h>
#include <seqan/file.h>

SEQAN_DEFINE_TEST(test_my_app_funcs_hello)
{
    SEQAN_FAIL("Hello, tester!");
}

SEQAN_BEGIN_TESTSUITE(test_my_app_funcs)
{
    SEQAN_CALL_TEST(test_my_app_funcs_hello);
}
SEQAN_END_TESTSUITE

Registering in CMakeLists.txt

The following lines have to be added to CMakeLists.txt.

add_executable (test_my_app_funcs test_my_app_funcs.cpp)
target_link_libraries (test_my_app_funcs ${SEQAN_LIBRARIES})
add_test (NAME test_test_my_app_funcs COMMAND <TARGET_FILE:test_my_app_funcs>)

Next Steps

The best way to learn about the testing system is to look at the existing test suites. Good examples are test_score and test_random. Note that other test suites might not be very clean since they were ported from the old test system and are not completely cleaned up yet.

Have a look at the Writing Tests for more examples.

Assertions

You can make assertions on the called functions using the SEQAN_ASSERT* macros, e.g. SEQAN_ASSERT, SEQAN_ASSERT_EQ. For each assertion macro, there is one macro that has to be passed a message and optionally parameters, e.g. SEQAN_ASSERT_MSG, SEQAN_ASSERT_EQ_MSG.

Assertions can also be used in your library and application code. When compiled in Debug mode, the assertions are enabled. They are disabled in Release or RelWithDebInfo mode.

Checks

There also is a macro called SEQAN_CHECK that creates an assertion that is enabled regardless of whether debugging is enabled. It only makes sense to use the SEQAN_CHECK macro in library or application code, not in tests. There are no variants of SEQAN_CHECK for comparisons. Also see the macro SEQAN_FAIL.

App Tests

For writing app tests, see Writing App Tests.

comments powered by Disqus