Ctest and CxxTest

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Ctest and CxxTest

Wojciech Migda-3
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all,

I was wondering about an improvement for the way CxxTest unit tests
are coupled with CTest. Right now when CTest submits results to CDash
the tests are counted by binaries executed. In my unit tests each
binary corresponds to an entire suite (approx. 150 testcases grouped
within single CxxTest class). I'd see it very useful to have CTest
operate with CxxTest based unit tests on a testcase level. Has anyone
stumbled upon similar problem and got ideas / solutions ?

Thanks for assistance,

- -Wojciech

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKtjXN0iFl+nAyImcRAo/BAKCCjLfj8i+DjwxV3XDKO7drCSHPKQCfVl4a
MdCJI1BEpFFviP6wjN8o5JM=
=KGBE
-----END PGP SIGNATURE-----


----------------------------------------------------------------------
Kawalerka za 79 000 zl!
Sprawdz >>> http://link.interia.pl/f2348

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Ctest and CxxTest

Philip Lowman
You could split up your testcases into multiple CxxTest classes.

You could check to see if the cxxtest code generator supports preprocessor statements in generating test runners.  Then you could group your tests by #ifdef / #endif...?  Alternatively if it's not supported you may be able to modify the code generator to output multiple runners depending on the tests you want to run, e.g. cxxtestgen.pl --enable-test foo

I think splitting things up probably would be the easiest approach.

On Sun, Sep 20, 2009 at 10:01 AM, Wojciech Migda <[hidden email]> wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all,

I was wondering about an improvement for the way CxxTest unit tests
are coupled with CTest. Right now when CTest submits results to CDash
the tests are counted by binaries executed. In my unit tests each
binary corresponds to an entire suite (approx. 150 testcases grouped
within single CxxTest class). I'd see it very useful to have CTest
operate with CxxTest based unit tests on a testcase level. Has anyone
stumbled upon similar problem and got ideas / solutions ?

Thanks for assistance,

- -Wojciech

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKtjXN0iFl+nAyImcRAo/BAKCCjLfj8i+DjwxV3XDKO7drCSHPKQCfVl4a
MdCJI1BEpFFviP6wjN8o5JM=
=KGBE
-----END PGP SIGNATURE-----


----------------------------------------------------------------------
Kawalerka za 79 000 zl!
Sprawdz >>> http://link.interia.pl/f2348

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake



--
Philip Lowman

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Ctest and CxxTest

Ben Boeckel
In reply to this post by Wojciech Migda-3
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Wojciech Migda wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi all,
>
> I was wondering about an improvement for the way CxxTest unit
tests
> are coupled with CTest. Right now when CTest submits results
to CDash
> the tests are counted by binaries executed. In my unit tests
each
> binary corresponds to an entire suite (approx. 150 testcases
grouped
> within single CxxTest class). I'd see it very useful to have
CTest
> operate with CxxTest based unit tests on a testcase level. Has
anyone
> stumbled upon similar problem and got ideas / solutions ?
>
> Thanks for assistance,
>
> - -Wojciech

I use QTest for my unit tests. These have multiple tests within
each one. When one fails, I run ctest -V <test_name> to get the
details. It helps keep output down when tests are working great
and only expands when I need it (though automating the ctest -V
run is something I haven't looked into yet).

- --Ben
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iQIcBAEBAgAGBQJKtkckAAoJEKaxavVX4C1X+4cQALpfG+c5YaVaOspLE7b/ILxA
+3KIyWG34rrqpMoXzS72l+j5psDwW3GUpqkLPZdOMCLh6Z4x7Ql+OS7V6y+UuKjv
wT+2AY4tk4O4wEV4j4ISQYd1bkGDbBOzAxcbTZnrEx+3dRvvB7Dz8q6sZS875St9
rnPuPPLwCAH4IXQJ+dwdsRg/rJb2vvHGyNRY1rhQ5a/7rca0PqBi38uQQ8zZw840
KYdoi4p+i/Wvv/wX57avrDLVnXMPdFdoT7n7QMM4vk15n0eyoXk19aaTSrRvsiQP
ZbXYdgvgOlFBE2Lk9tktR/lzSXX3bGGymqFqYvfX00c06Mtofju0ZHbtEGwp8tad
2m0lSINtg/c+JdFsqp8RQaCv5xC/i285v/B6piQljnWkjBXkXP5bXncOR12qd3F1
Vgvnf71X72l93C6zeVqV0g5+e7ITSvPEWWe0ok40dP4ZAbo9lQ8/UbjTCD7J7XHJ
P8YWEG2OCiBaYp+IfTgRyt+EBYW18ig5O8HjZANgxWAG9UNbB00HSw6eEy8q/O5g
/OJOsMhMTsQIRSw0Y+5mSIkcMIGclcUDdX3Jpkq+BshOoOqEn+oAEnpkYlPi9TiF
KlewIi1QWyW8eMEu8ZJoMy8WH+N4wQPtyPlVllxmlX5MON+s82LLiWMN1aaZf/VH
//j+hhkExoZmwhtZbhPR
=kQDr
-----END PGP SIGNATURE-----


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Ctest and CxxTest

Wojciech Migda-3
In reply to this post by Philip Lowman
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Philip Lowman pisze:

> You could split up your testcases into multiple CxxTest classes.
>
> You could check to see if the cxxtest code generator supports
> preprocessor statements in generating test runners.  Then you could
> group your tests by #ifdef / #endif...?  Alternatively if it's not
> supported you may be able to modify the code generator to output
> multiple runners depending on the tests you want to run, e.g.
> cxxtestgen.pl --enable-test foo
>
> I think splitting things up probably would be the easiest approach.
>
>
> On Sun, Sep 20, 2009 at 10:01 AM, Wojciech Migda
> <[hidden email]> wrote:
>
> Hi all,
>
> I was wondering about an improvement for the way CxxTest unit tests
>  are coupled with CTest. Right now when CTest submits results to
> CDash the tests are counted by binaries executed. In my unit tests
> each binary corresponds to an entire suite (approx. 150 testcases
> grouped within single CxxTest class). I'd see it very useful to
> have CTest operate with CxxTest based unit tests on a testcase
> level. Has anyone stumbled upon similar problem and got ideas /
> solutions ?
>
> Thanks for assistance,
>
> -Wojciech
>
Indeed, splitting is the easiest approach, albeit it adds additional
effort - when a new test is added to a suite CMakeLists.txt has to be
updated accordingly.

If my idea how CTest works when it uploads results to CDash is correct
then it only looks at the return code of the executed binary and
captures its output. What if there was an option for CTest to work in
a multiple-testcases mode where the output returned by the executed
binary would be formatted in a way so as CTest would be able to parse
it and process into CDash xml files with information on all tests
executed. Just a thought, I have no idea how complicated that would
turn out to be.

I also thought about the binary itself being able to run single
testcases on the basis of command line arguments, but that would
require parallel information being stored in CMakeLists.txt, which
again add effort.

I'll see if I have more ideas.

Thanks,

- -Wojtek

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKtkz70iFl+nAyImcRAsPSAJ4yMjKSb96NZ02awttzwwu/nHZRhgCfQ2KZ
VEM63SdgrUUA4OIXGApKJd8=
=4utB
-----END PGP SIGNATURE-----


----------------------------------------------------------------------
Bezplatne konto i limit do 100 tys. Otwierasz?
http://link.interia.pl/f2342

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Ctest and CxxTest

Daniel Blezek-3
We tackled this using a "hack-ish" CMake command for Google Tests that found
all our tests in the source code, and generated CTest commands for them.
You could do a similar thing on a test suite.  Details are here:

http://www.itk.org/Wiki/Proposals:Increasing_ITK_Code_Coverage#Google_Test

-dan


On 9/20/09 10:40 AM, "Wojciech Migda" <[hidden email]> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
>
> Philip Lowman pisze:
>> You could split up your testcases into multiple CxxTest classes.
>>
>> You could check to see if the cxxtest code generator supports
>> preprocessor statements in generating test runners.  Then you could
>> group your tests by #ifdef / #endif...?  Alternatively if it's not
>> supported you may be able to modify the code generator to output
>> multiple runners depending on the tests you want to run, e.g.
>> cxxtestgen.pl --enable-test foo
>>
>> I think splitting things up probably would be the easiest approach.
>>
>>
>> On Sun, Sep 20, 2009 at 10:01 AM, Wojciech Migda
>> <[hidden email]> wrote:
>>
>> Hi all,
>>
>> I was wondering about an improvement for the way CxxTest unit tests
>>  are coupled with CTest. Right now when CTest submits results to
>> CDash the tests are counted by binaries executed. In my unit tests
>> each binary corresponds to an entire suite (approx. 150 testcases
>> grouped within single CxxTest class). I'd see it very useful to
>> have CTest operate with CxxTest based unit tests on a testcase
>> level. Has anyone stumbled upon similar problem and got ideas /
>> solutions ?
>>
>> Thanks for assistance,
>>
>> -Wojciech
>>
> Indeed, splitting is the easiest approach, albeit it adds additional
> effort - when a new test is added to a suite CMakeLists.txt has to be
> updated accordingly.
>
> If my idea how CTest works when it uploads results to CDash is correct
> then it only looks at the return code of the executed binary and
> captures its output. What if there was an option for CTest to work in
> a multiple-testcases mode where the output returned by the executed
> binary would be formatted in a way so as CTest would be able to parse
> it and process into CDash xml files with information on all tests
> executed. Just a thought, I have no idea how complicated that would
> turn out to be.
>
> I also thought about the binary itself being able to run single
> testcases on the basis of command line arguments, but that would
> require parallel information being stored in CMakeLists.txt, which
> again add effort.
>
> I'll see if I have more ideas.
>
> Thanks,
>
> - -Wojtek
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.7 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iD8DBQFKtkz70iFl+nAyImcRAsPSAJ4yMjKSb96NZ02awttzwwu/nHZRhgCfQ2KZ
> VEM63SdgrUUA4OIXGApKJd8=
> =4utB
> -----END PGP SIGNATURE-----
>
>
> ----------------------------------------------------------------------
> Bezplatne konto i limit do 100 tys. Otwierasz?
> http://link.interia.pl/f2342
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.cmake.org/mailman/listinfo/cmake

--
Daniel Blezek, PhD
Medical Imaging Informatics Innovation Center

P 127 or (77) 8 8886
T 507 538 8886
E [hidden email]

Mayo Clinic
200 First St. S.W.
Harwick SL-44
Rochester, MN 55905
mayoclinic.org


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Ctest and CxxTest

Wojciech Migda-3
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Daniel Blezek pisze:

> We tackled this using a "hack-ish" CMake command for Google Tests
> that found all our tests in the source code, and generated CTest
> commands for them. You could do a similar thing on a test suite.
> Details are here:
>
> http://www.itk.org/Wiki/Proposals:Increasing_ITK_Code_Coverage#Google_Test
>
>
> -dan
>

Thank you very much - this is pretty much the idea I started
converging to. What you've done with ITK will work 100%. I just need
to tweak the CxxTest generator script.

BR,

- -Wojciech
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKt8Rm0iFl+nAyImcRAl3yAJ9oSp7Cj5CvVfjvq9wiUQjGmnqQGgCePT/o
VdtdcPqeZgOzqBw2r+Aox5E=
=v868
-----END PGP SIGNATURE-----



----------------------------------------------------------------------
Zobacz jak mieszka Kasia Zielinska!
Kliknij >>> http://link.interia.pl/f2343

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Ctest and CxxTest

Philip Lowman
On Mon, Sep 21, 2009 at 2:22 PM, Wojciech Migda <[hidden email]> wrote:
Daniel Blezek pisze:
> We tackled this using a "hack-ish" CMake command for Google Tests
> that found all our tests in the source code, and generated CTest
> commands for them. You could do a similar thing on a test suite.
> Details are here:
>
> http://www.itk.org/Wiki/Proposals:Increasing_ITK_Code_Coverage#Google_Test

Thank you very much - this is pretty much the idea I started
converging to. What you've done with ITK will work 100%. I just need
to tweak the CxxTest generator script.

For any still following this thread, with Dan's permission I merged his macro into the FindGTest.cmake recently checked into CMake CVS.

If anyone has any patches to FindCxxTest.cmake please feel free to send them in.

--
Philip Lowman

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake