Quantcast

testing with standard output

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

testing with standard output

Henrique Almeida
 Hello,

 When running "make test", some of my unit tests require that the
result is given by a correct output, instead of (just) the correct
return value (because, I'm specifically testing the ability to do
output). Is there a way to configure CTest to report a successful
result for certain tests by comparing their outputs to a reference
output ?

 In the most general case I'd need something like:

 add_test(test source1 source2 INPUT test.input EXPECTED_OUTPUT
test.output EXPECTED_ERROR test.error)

 I'd provide test.input as stdin, test.output as expected stdout and
test.error as expected stderr.

--
 Henrique Dante de Almeida
 [hidden email]
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: testing with standard output

Bill Hoffman
Henrique Almeida wrote:

>  Hello,
>
>  When running "make test", some of my unit tests require that the
> result is given by a correct output, instead of (just) the correct
> return value (because, I'm specifically testing the ability to do
> output). Is there a way to configure CTest to report a successful
> result for certain tests by comparing their outputs to a reference
> output ?
>
>  In the most general case I'd need something like:
>
>  add_test(test source1 source2 INPUT test.input EXPECTED_OUTPUT
> test.output EXPECTED_ERROR test.error)
>
>  I'd provide test.input as stdin, test.output as expected stdout and
> test.error as expected stderr.
>


http://www.cmake.org/cmake/help/cmake2.6docs.html#section_Properties%20on%20Tests


-Bill
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: testing with standard output

Henrique Almeida
 Also, is there a way to escape a string to convert it to a literal
regular expression ?

2009/7/14 Henrique Almeida <[hidden email]>:

>  Quite good. :-) Are there equivalents for stdin and stderr ?
>
> 2009/7/14 Bill Hoffman <[hidden email]>:
>> Henrique Almeida wrote:
>>>
>>>  Hello,
>>>
>>>  When running "make test", some of my unit tests require that the
>>> result is given by a correct output, instead of (just) the correct
>>> return value (because, I'm specifically testing the ability to do
>>> output). Is there a way to configure CTest to report a successful
>>> result for certain tests by comparing their outputs to a reference
>>> output ?
>>>
>>>  In the most general case I'd need something like:
>>>
>>>  add_test(test source1 source2 INPUT test.input EXPECTED_OUTPUT
>>> test.output EXPECTED_ERROR test.error)
>>>
>>>  I'd provide test.input as stdin, test.output as expected stdout and
>>> test.error as expected stderr.
>>>
>>
>>
>> http://www.cmake.org/cmake/help/cmake2.6docs.html#section_Properties%20on%20Tests
>>
>>
>> -Bill
>>
>
>
>
> --
>  Henrique Dante de Almeida
>  [hidden email]
>



--
 Henrique Dante de Almeida
 [hidden email]
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: testing with standard output

James C. Sutherland
In reply to this post by Henrique Almeida

On Jul 14, 2009, at 6:48 AM, Henrique Almeida wrote:

> Hello,
>
> When running "make test", some of my unit tests require that the
> result is given by a correct output, instead of (just) the correct
> return value (because, I'm specifically testing the ability to do
> output). Is there a way to configure CTest to report a successful
> result for certain tests by comparing their outputs to a reference
> output ?
>
> In the most general case I'd need something like:
>
> add_test(test source1 source2 INPUT test.input EXPECTED_OUTPUT
> test.output EXPECTED_ERROR test.error)
>
> I'd provide test.input as stdin, test.output as expected stdout and
> test.error as expected stderr.
>

I asked a similar question recently.  Here is the recipe for comparing  
files in a test.  You will need to adapt this to your own needs...

1. In your CMakeLists.txt file:

#-------------------------------------------------
set( test_cmd ${CMAKE_BINARY_DIR}/util/create_expr )
set( args "--expr-name=test1 --patch-name=PatchT --field-type=FieldT" )
add_test( createExprTest_1
   ${CMAKE_COMMAND}
   -D test_cmd=${test_cmd}
   -D test_args:string=${args}
   -D output_blessed=${CMAKE_SOURCE_DIR}/util/test/test1.h
   -D output_test=${CMAKE_BINARY_DIR}/util/test1.h
   -P ${CMAKE_SOURCE_DIR}/util/run_test.cmake
)
#-------------------------------------------------


2. In the new "run_test.cmake" file:

#-------------------------------------------------
# some argument checking:
# test_cmd is the command to run with all its arguments
if( NOT test_cmd )
   message( FATAL_ERROR "Variable test_cmd not defined" )
endif( NOT test_cmd )

# output_blessed contains the name of the "blessed" output file
if( NOT output_blessed )
   message( FATAL_ERROR "Variable output_blessed not defined" )
endif( NOT output_blessed )

# output_test contains the name of the output file the test_cmd will  
produce
if( NOT output_test )
   message( FATAL_ERROR "Variable output_test not defined" )
endif( NOT output_test )

# convert the space-separated string to a list
separate_arguments( test_args )
message( ${test_args} )

execute_process(
   COMMAND ${test_cmd} ${test_args}
   COMMAND ${CMAKE_COMMAND} -E compare_files ${output_blessed} $
{output_test}
   RESULT_VARIABLE test_not_successful
)

if( test_not_successful )
   message( SEND_ERROR "${output_test} does not match $
{output_blessed}!" )
endif( test_not_successful )
#-------------------------------------------------

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: testing with standard output

Henrique Almeida
 This is quite crazy, but it may completelly solve the problem. I
wonder why there isn't a single line version of that. I'm even willing
to contribute a patch, but I'd need some guidance.

2009/7/14 James C. Sutherland <[hidden email]>:

>
>
> I asked a similar question recently.  Here is the recipe for comparing files
> in a test.  You will need to adapt this to your own needs...
>
> 1. In your CMakeLists.txt file:
>
> #-------------------------------------------------
> set( test_cmd ${CMAKE_BINARY_DIR}/util/create_expr )
> set( args "--expr-name=test1 --patch-name=PatchT --field-type=FieldT" )
> add_test( createExprTest_1
>  ${CMAKE_COMMAND}
>  -D test_cmd=${test_cmd}
>  -D test_args:string=${args}
>  -D output_blessed=${CMAKE_SOURCE_DIR}/util/test/test1.h
>  -D output_test=${CMAKE_BINARY_DIR}/util/test1.h
>  -P ${CMAKE_SOURCE_DIR}/util/run_test.cmake
> )
> #-------------------------------------------------
>
>
> 2. In the new "run_test.cmake" file:
>
> #-------------------------------------------------
> # some argument checking:
> # test_cmd is the command to run with all its arguments
> if( NOT test_cmd )
>  message( FATAL_ERROR "Variable test_cmd not defined" )
> endif( NOT test_cmd )
>
> # output_blessed contains the name of the "blessed" output file
> if( NOT output_blessed )
>  message( FATAL_ERROR "Variable output_blessed not defined" )
> endif( NOT output_blessed )
>
> # output_test contains the name of the output file the test_cmd will produce
> if( NOT output_test )
>  message( FATAL_ERROR "Variable output_test not defined" )
> endif( NOT output_test )
>
> # convert the space-separated string to a list
> separate_arguments( test_args )
> message( ${test_args} )
>
> execute_process(
>  COMMAND ${test_cmd} ${test_args}
>  COMMAND ${CMAKE_COMMAND} -E compare_files ${output_blessed} ${output_test}
>  RESULT_VARIABLE test_not_successful
> )
>
> if( test_not_successful )
>  message( SEND_ERROR "${output_test} does not match ${output_blessed}!" )
> endif( test_not_successful )
> #-------------------------------------------------
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.cmake.org/mailman/listinfo/cmake
>



--
 Henrique Dante de Almeida
 [hidden email]
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Loading...