cdash/ctest question

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

cdash/ctest question

Biddiscombe, John A.
Apologies for posting a cdash question question to the cmake list…

CDash plots the run-time of tests, which is very useful indeed, but it would be even more useful if one could output a performance related ‘time’ from a test where a specific feature was being benchmarked and one could see how it changes over a period of days/weeks/months as changes to code are being made. 
The time would be specific to the tested code and not include startup and other uninteresting overheads (such as when the filesystem is misbehaving and the test starts slowly).

Does such a feature exist in ctest/cdash at all?

Thanks

JB

--

John Biddiscombe,                        email:biddisco @.at.@ cscs.ch

http://www.cscs.ch/

CSCS, Swiss National Supercomputing Centre  | Tel:  +41 (91) 610.82.07

Via Trevano 131, 6900 Lugano, Switzerland   | Fax:  +41 (91) 610.82.82



--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: cdash/ctest question

Alexander Neundorf-3
On Thursday, January 14, 2016 21:51:23 Biddiscombe, John A. wrote:

> Apologies for posting a cdash question question to the cmake list…
>
> CDash plots the run-time of tests, which is very useful indeed, but it would
> be even more useful if one could output a performance related ‘time’ from a
> test where a specific feature was being benchmarked and one could see how
> it changes over a period of days/weeks/months as changes to code are being
> made.
> The time would be specific to the tested code and not include
> startup and other uninteresting overheads (such as when the filesystem is
> misbehaving and the test starts slowly).
> Does such a feature exist in ctest/cdash at all?

yes, we do exactly that.

You have to generate <DartMeasurement> tags in the stdoutof the test
<DartMeasurement name="Simple" type="numeric/integer">42</DartMeasurement>
<DartMeasurement name="ExecutionTime"
type="numeric/double">1.23456</DartMeasurement>

and then in cdash you can configure the tests-overview page to display certain
such measurements by name, and on the tests page you can also display graphs
for such named measurements.

Alex


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: cdash/ctest question

Biddiscombe, John A.
Alex

Thank you for this information. I shall test this out right away.

Yours

JB

-----Original Message-----
From: Alexander Neundorf [mailto:[hidden email]]
Sent: 14 January 2016 23:17
To: [hidden email]
Cc: Biddiscombe, John A.
Subject: Re: [CMake] cdash/ctest question

On Thursday, January 14, 2016 21:51:23 Biddiscombe, John A. wrote:

> Apologies for posting a cdash question question to the cmake list…
>
> CDash plots the run-time of tests, which is very useful indeed, but it
> would be even more useful if one could output a performance related
> ‘time’ from a test where a specific feature was being benchmarked and
> one could see how it changes over a period of days/weeks/months as
> changes to code are being made.
> The time would be specific to the tested code and not include startup
> and other uninteresting overheads (such as when the filesystem is
> misbehaving and the test starts slowly).
> Does such a feature exist in ctest/cdash at all?

yes, we do exactly that.

You have to generate <DartMeasurement> tags in the stdoutof the test <DartMeasurement name="Simple" type="numeric/integer">42</DartMeasurement>
<DartMeasurement name="ExecutionTime"
type="numeric/double">1.23456</DartMeasurement>

and then in cdash you can configure the tests-overview page to display certain such measurements by name, and on the tests page you can also display graphs for such named measurements.

Alex


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: cdash/ctest question

Alexander Neundorf-3
Hi John,

On 2016 M01 19, Tue 07:55:27 CET Biddiscombe, John A. wrote:
> Alex
>
> Thank you for this information. I shall test this out right away.

so, did it work out ?

Alex

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake