[SOLVED] In ctest how to check for both non-zero exit code and specific string output?

Issue

I want to test that an app is both exiting with a non-zero code AND outputting a particular string. I set the properties like this:

set_tests_properties(
    myapp-test-foobar
PROPERTIES
    WILL_FAIL TRUE
    FAIL_REGULAR_EXPRESSION "^Usage: myapp"
)

But the test passes even if the app's exit code is 0. Although WILL_FAIL is necessary, otherwise the test will fail when the FRE is matched, the exit code is ignored. I searched the cmake docs for an answer to no avail.

EDIT: It turns out that ctest (I'm using v3.19.2) does not check the exit code if either PASS_REGULAR_EXPRESSION or FAIL_REGULAR_EXPRESSION is set. The pass or fail condition is met solely by the RE matching.

A nasty bug has escaped into the wild in one of my apps because of this.

EDIT: The answer suggested in comments below is for handling an app that aborts, i.e. crashes, on the error condition being tested and so is different from my case. Please see my comment below for further reasoning and solution.

Solution

The fix I described in my comment above does not work. As I recently discovered a mismatched FAIL_REGULAR_EXPRESSION will not cause the test to fail. See this question.

This is because ctest does

match FAIL_REGULAR_EXPRESSION || exit code != 0

when WILL_FAIL is set. When it is not set, ctest does

if (PASS_REGULAR_EXPRESSION)
    if (match PASS_REGULAR_EXPRESSION) pass else() fail endif()
else()
    if (exit code == 0) pass else() fail endif()
endif()

You can test for both the correct error code and error message by abusing PASS_REGULAR_EXPRESSION as in the following

add_test( NAME myapp-test-ktx2-in-exit-code
    COMMAND myapp -o foo infile.ktx2
    WORKING_DIRECTORY ${MY_WORK_DIR}
)
set_tests_properties(
    myapp-test-ktx2-in-exit-code
PROPERTIES
    WILL_FAIL TRUE
)
add_test( NAME myapp-test-ktx2-in
    COMMAND myapp -o foo input.ktx2
    WORKING_DIRECTORY ${MY_WORK_DIR}
)
set_tests_properties(
    myapp-test-ktx2-in
PROPERTIES
    PASS_REGULAR_EXPRESSION ".* is not a KTX v1 file."
)

If both tests pass we are guaranteed that we have a non-zero exit code and the error message matches PASS_REGULAR_EXPRESSION.

Answered By - msc

Answer Checked By - Robin (WPSolving Admin)