Tutorial 4 - Testing for failure and avoiding tests
Under some conditions, it is impossible to avoid tests failing. In this section, we'll see how to deal with these cases.
Skipping tests
Where a test doesn't make sense to run (for example, if the required test files aren't available, or the feature is architecture or operating system dependent), the cleanest solution is to skip the test.
Tests are skipped using the QSKIP macro. QSKIP takes two arguments - a label string that should be used to describe why the test is being skipped, and a enumerated constant that controls how much of the test is skipped. If you pass SkipSingle, and the test is data driven, then only the current row is skipped. If you pass SkipAll and the test is data driven, then all following rows are skipped. If the test is not data driven, then it doesn't matter which one is used.
You can see how QSKIP works in the example below:
Example 15. Unit test showing skipped tests
void testDate::testSkip_data()
{
QTest::addColumn<int>("val1");
QTest::addColumn<int>("val2");
QTest::newRow("1") << 1 << 1;
QTest::newRow("2") << 1 << 2;
QTest::newRow("3") << 3 << 3;
QTest::newRow("5") << 5 << 5;
QTest::newRow("4") << 4 << 5;
}
void testDate::testSkip()
{
QFETCH(int, val1);
QFETCH(int, val2);
if ( val2 == 2 )
QSKIP("Two isn't good, not doing it", SkipSingle);
if ( val1 == 5 )
QSKIP("Five! I've had enough, bailing here", SkipAll);
QCOMPARE( val1, val2 );
}
|
Example 16. Output of unit test showing skipped tests
$ ./tutorial4 testSkip -v2 ********* Start testing of testDate ********* Config: Using QTest library 4.1.0, Qt 4.1.0-snapshot-20051107 INFO : testDate::initTestCase() entering PASS : testDate::initTestCase() INFO : testDate::testSkip() entering INFO : testDate::testSkip(1) COMPARE() Loc: [tutorial4.cpp(82)] SKIP : testDate::testSkip(2) Two isn't good, not doing it Loc: [tutorial4.cpp(79)] INFO : testDate::testSkip(3) COMPARE() Loc: [tutorial4.cpp(82)] SKIP : testDate::testSkip(5) Five! I've had enough, bailing here Loc: [tutorial4.cpp(81)] PASS : testDate::testSkip() INFO : testDate::cleanupTestCase() entering PASS : testDate::cleanupTestCase() Totals: 3 passed, 0 failed, 2 skipped ********* Finished testing of testDate ********* |
From the verbose output, you can see that the test was run on the first and third rows. The second row wasn't run because of the QSKIP call with a SkipSingle argument. Similarly, the fourth and fifth rows weren't run because the fourth row triggered a QSKIP call with a SkipAll argument.
Also note that the test didn't fail, even though there were two calls to QSKIP. Conceptually, a skipped test is a test that didn't make sense to run for test validity reasons, rather than a test that is valid but will fail because of bugs or lack of features in the code being tested.
Handling expected failures
If you have valid tests, but the code that you are testing doesn't pass them, then ideally you fix the code you are testing. However sometimes that isn't possible in the time that you have available, or because of a need to avoid binary incompatible changes. In this case, it is undesirable to delete or modify the unit tests - it is better to flag the test as "expected to fail", using the QEXPECT_FAIL macro. An example of this is shown below:
Example 17. Unit test showing expected failures
void testDate::testExpectedFail()
{
QEXPECT_FAIL("", "1 is not 2, even for very large 1", Continue);
QCOMPARE( 1, 2 );
QCOMPARE( 2, 2 );
QEXPECT_FAIL("", "1 is not 2, even for very small 2", Abort);
QCOMPARE( 1, 2 );
// The next line will not be run, because we Abort on previous failure
QCOMPARE( 3, 3 );
}
|
Example 18. Output of unit test showing expected failures
$ ./tutorial4/tutorial4 testExpectedFail -v2 ********* Start testing of testDate ********* Config: Using QTest library 4.1.0, Qt 4.1.0-snapshot-20051107 INFO : testDate::initTestCase() entering PASS : testDate::initTestCase() INFO : testDate::testExpectedFail() entering INFO : testDate::testExpectedFail() Compared values are not the same Actual (1): 1 Expected (2): 2 Loc: [tutorial4.cpp(41)] XFAIL : testDate::testExpectedFail() 1 is not 2, even for very large 1 Loc: [tutorial4.cpp(41)] INFO : testDate::testExpectedFail() COMPARE() Loc: [tutorial4.cpp(42)] INFO : testDate::testExpectedFail() Compared values are not the same Actual (1): 1 Expected (2): 2 Loc: [tutorial4.cpp(45)] XFAIL : testDate::testExpectedFail() 1 is not 2, even for very small 2 Loc: [tutorial4.cpp(45)] PASS : testDate::testExpectedFail() INFO : testDate::cleanupTestCase() entering PASS : testDate::cleanupTestCase() Totals: 3 passed, 0 failed, 0 skipped ********* Finished testing of testDate ********* |
As you can see from the verbose output, we expect a failure from each time we do a QCOMPARE( 1, 2);. In the first call to QEXPECT_FAIL, we use the Continue argument, so the rest of the tests will still be run. However in the second call to QEXPECT_FAILwe use the Abort and the test bails at this point. Generally it is better to use Continue unless you have a lot of closely related tests that would each need a QEXPECT_FAIL entry.
Also note that tests that are marked as expected failures are not considered to be failures, so the test function above is considered to be a pass.
If a test that is marked to be an expected failure, and it unexpectedly passes, then that is flagged as an error, as shown below:
Example 19. Unit test showing unexpected pass
void testDate::testUnexpectedPass()
{
QEXPECT_FAIL("", "1 is not 2, even for very large 1", Continue);
QCOMPARE( 1, 1 );
QCOMPARE( 2, 2 );
QEXPECT_FAIL("", "1 is not 2, even for very small 2", Abort);
QCOMPARE( 1, 1 );
QCOMPARE( 3, 3 );
}
|
Example 20. Output of unit test showing unexpected pass
$ ./tutorial4/tutorial4 testUnexpectedPass -v2 ********* Start testing of testDate ********* Config: Using QTest library 4.1.0, Qt 4.1.0-snapshot-20051107 INFO : testDate::initTestCase() entering PASS : testDate::initTestCase() INFO : testDate::testUnexpectedPass() entering INFO : testDate::testUnexpectedPass() COMPARE() Loc: [tutorial4.cpp(53)] XPASS : testDate::testUnexpectedPass() COMPARE() Loc: [tutorial4.cpp(53)] INFO : testDate::testUnexpectedPass() COMPARE() Loc: [tutorial4.cpp(54)] INFO : testDate::testUnexpectedPass() COMPARE() Loc: [tutorial4.cpp(57)] XPASS : testDate::testUnexpectedPass() COMPARE() Loc: [tutorial4.cpp(57)] INFO : testDate::cleanupTestCase() entering PASS : testDate::cleanupTestCase() Totals: 2 passed, 2 failed, 0 skipped ********* Finished testing of testDate ********* |
The effect of unexpected passes on the running of the test is controlled by the second argument to QEXPECT_FAIL. If the argument is Continue and the test unexpectedly passes, then the rest of the test function will be run. If the argument is Abort, then the test will stop.
Checking debug messages and warnings
If you are testing border cases, you will likely run across the case where some kind of message will be produced using the qDebug or qWarning functions. Where a test produces a debug or warning message, that message will be logged in the test output (although it will still be considered a pass unless some other check fails), as shown in the example below:
Example 21. Unit test producing warning and debug messages
void testDate::testQdebug()
{
qWarning("warning");
qDebug("debug");
qCritical("critical");
}
|
Example 22. Output of unit test producing warning and debug messages
$ ./tutorial4 testQdebug ********* Start testing of testDate ********* Config: Using QTest library 4.1.0, Qt 4.1.0-snapshot-20051107 PASS : testDate::initTestCase() QWARN : testDate::testQdebug() warning QDEBUG : testDate::testQdebug() debug QSYSTEM: testDate::testQdebug() critical PASS : testDate::testQdebug() PASS : testDate::cleanupTestCase() Totals: 3 passed, 0 failed, 0 skipped ********* Finished testing of testDate ********* |
Note that while this example produces the debug and warning messages within the test function (testQdebug), those messages would normally be propagated up from the code being tested. However the source of the messages does not make any difference to how they are handled.
If your test needs include either a clean output, or verification that appropriate messages are generated, then you will probably need the QtTest::ignoreMessage function.
 | The ignoreMessage function can be used to ignore a message, however it might be clearer to think of this function as checking for the presence of an expected message. In particular, it is a test failure if you call ignoreMessage and the message is not generated. |
An example of how ignoreMessage works is shown below.
Example 23. Example of using ignoreMessage
void testDate::testValidity()
{
QTest::ignoreMessage(QtWarningMsg, "validity warning");
qWarning("validity warning");
}
void testDate::testValiditi()
{
QTest::ignoreMessage(QtWarningMsg, "validity warning");
qWarning("validiti warning");
}
|
Example 24. Output of ignoreMessage example
$ ./tutorial4 testValidity testValiditi ********* Start testing of testDate ********* Config: Using QTest library 4.1.0, Qt 4.1.0-snapshot-20051107 PASS : testDate::initTestCase() PASS : testDate::testValidity() QWARN : testDate::testValiditi() validiti warning INFO : testDate::testValiditi() Did not receive message: "validity warning" FAIL! : testDate::testValiditi() Not all expected messages were received PASS : testDate::cleanupTestCase() Totals: 3 passed, 1 failed, 0 skipped ********* Finished testing of testDate ********* |
Note that the warning message in testDate::testValidity has been "swallowed" by the the call to ignoreMessage.
By contrast, the warning message in testDate::testValiditi still causes a warning to be logged, because the ignoreMessage call does not match the text in the warning message. In addition, because a we expected a particular warning message and it wasn't received, the testDate::testValiditi test function fails.