Benchmarking with CMake

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Benchmarking with CMake

CMake mailing list

Hi!

 

Can anyone tell me if CMake (or CTest) can be customized in a way to produce decent benchmark output? An .xlsx perhaps, or something GnuPlot friendly? I have not found any examples of CTest being (ab)used in this way.

 

Is my feeling correct, that this should be something like CBench?

 

Cheers,

Máté

 


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Benchmarking with CMake

Matt McCormick-2
Hi Mate,

This blog post has an example of parsing and visualizing
CTestCostData.txt.  Although, that file was not created for that
purpose:

  http://www.kitware.com/blog/home/post/556

Cheers,
Matt

On Mon, Nov 9, 2015 at 5:21 PM, Nagy-Egri Máté Ferenc <[hidden email]> wrote:

> Hi!
>
>
>
> Can anyone tell me if CMake (or CTest) can be customized in a way to produce
> decent benchmark output? An .xlsx perhaps, or something GnuPlot friendly? I
> have not found any examples of CTest being (ab)used in this way.
>
>
>
> Is my feeling correct, that this should be something like CBench?
>
>
>
> Cheers,
>
> Máté
>
>
>
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Benchmarking with CMake

Alexander Neundorf-3
In reply to this post by CMake mailing list

On Monday, November 09, 2015 23:21:58 Nagy-Egri Máté Ferenc via CMake wrote:

> Hi!

>

> Can anyone tell me if CMake (or CTest) can be customized in a way to produce

> decent benchmark output? An .xlsx perhaps, or something GnuPlot friendly? I

> have not found any examples of CTest being (ab)used in this way.

>

> Is my feeling correct, that this should be something like CBench?

 

 

depending on what you want...

You can make your test output <DartMeasurement> tags, and cdash will create graphs from the measured numbers. I think they can also be exported to csv.

 

Alex

 


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Benchmarking with CMake

CMake mailing list
In reply to this post by CMake mailing list

Thank you all for your resonses,

 

My use case is that I have a set of executables that function as unit tests. Currently I use the return value of the .exe as a means of indicating success, though I know there is way to match std::cout against a regex for more complex tests. However, the timings of the tests are not always suitable for benchmarking, because I only want the timings of a particular part of the tests. Their initialization times are comparable to their runtimes. While I could tune a parameter so that the init phase is negligable, but then benchmarking would take for ages. I want to write benchmark executables (similar to unit tests) that write a single number to the console and nothing else. Run some command (make bench?, cbench?) which runs, these tests, fetches the sole number from the std::cout and prints those numbers as output instead of the total run time presented by ctest.

 

@Peter: I also encountered Google Benchmark. I have already seen it’s output and is perfectly fine. I am however a bit reluctant to grab another dependency that all my consumers would be advised to install. CMake is a fairly full blown suite and I was hoping to find something that could provide similar results.

 

@Matt: Seems nice. Python is not much a dependency, but VTK is less evident to be present on all machines.

 

@Alexander: I still have to look into this, because information seems a little scarce on doing it. It seems to me, that CDash has to be installed somewhere to be able visualize the data. Is there a way to generate the HTML-r for local consumption instead of submitting somewhere? (Ideally bypassing a local installation of CDash)

 

I know, I have a lot of problems. Ultimately I will choose one of the above, I was just looking for the one that is the least painful. I am very reluctant on obtaining dependencies. CMake is such a good tool, it seems strange to me that no direct benchmarking support has been added yet.

 

Any ideas?

 


Feladó: Nagy-Egri Máté Ferenc via CMake
Elküldve: 2015. november 9., hétfő 23:27
Címzett: CMake felhasználók
Tárgy: [CMake] Benchmarking with CMake

 

 

Hi!

 

Can anyone tell me if CMake (or CTest) can be customized in a way to produce decent benchmark output? An .xlsx perhaps, or something GnuPlot friendly? I have not found any examples of CTest being (ab)used in this way.

 

Is my feeling correct, that this should be something like CBench?

 

Cheers,

Máté

 

 

 


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Benchmarking with CMake

Alexander Neundorf-3
In reply to this post by CMake mailing list

On Friday, November 13, 2015 14:41:35 Nagy-Egri Máté Ferenc via CMake wrote:

> Thank you all for your resonses,

>

> My use case is that I have a set of executables that function as unit tests.

> Currently I use the return value of the .exe as a means of indicating

> success, though I know there is way to match std::cout against a regex for

> more complex tests. However, the timings of the tests are not always

> suitable for benchmarking, because I only want the timings of a particular

> part of the tests. Their initialization times are comparable to their

> runtimes. While I could tune a parameter so that the init phase is

> negligable, but then benchmarking would take for ages. I want to write

> benchmark executables (similar to unit tests) that write a single number to

> the console and nothing else. Run some command (make bench?, cbench?) which

> runs, these tests, fetches the sole number from the std::cout and prints

> those numbers as output instead of the total run time presented by ctest.

>

> @Peter: I also encountered Google Benchmark. I have already seen it’s output

> and is perfectly fine. I am however a bit reluctant to grab another

> dependency that all my consumers would be advised to install. CMake is a

> fairly full blown suite and I was hoping to find something that could

> provide similar results.

>

> @Matt: Seems nice. Python is not much a dependency, but VTK is less evident

> to be present on all machines.

>

> @Alexander: I still have to look into this, because information seems a

> little scarce on doing it. It seems to me, that CDash has to be installed

> somewhere to be able visualize the data.

 

Yes.

 

> Is there a way to generate the

> HTML-r for local consumption instead of submitting somewhere? (Ideally

> bypassing a local installation of CDash)

 

I don't think so.

 

Alex

 


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake