Split Build and Test Pipelines

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Split Build and Test Pipelines

Dustyn Blasig
Hi All,

I'm curious if anyone has had success allowing two testing paths to coexist well.

Currently, we are using CTest to run our test executables with `make test`. However, on our Jenkins system, the build machines have the whole development stack but the test machines do not. So we need a way to package up the tests in a way that the test machines can simply run some generated script to do the equivalent of what CTest would do.

My current thought is to generate OS specific scripts `make test` depends on and runs. These scripts can get installed through `make install` if ENABLE_TEST_INSTALL is set, and then the test machine can run the installed scripts. The tricky part is building up the hierarchy of scripts in such a way that the tester just needs to run "test.sh" or "test.bat" at the top-level of the install hierarchy.

Any comments and suggestions greatly appreciated!

Cheers!

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Split Build and Test Pipelines

CMake mailing list
On Fri, 2019-05-10 at 12:06 -0500, Dustyn Blasig wrote:
Hi All,

I'm curious if anyone has had success allowing two testing paths to coexist well.

Currently, we are using CTest to run our test executables with `make test`. However, on our Jenkins system, the build machines have the whole development stack but the test machines do not. So we need a way to package up the tests in a way that the test machines can simply run some generated script to do the equivalent of what CTest would do.

My current thought is to generate OS specific scripts `make test` depends on and runs. These scripts can get installed through `make install` if ENABLE_TEST_INSTALL is set, and then the test machine can run the installed scripts. The tricky part is building up the hierarchy of scripts in such a way that the tester just needs to run "test.sh" or "test.bat" at the top-level of the install hierarchy.

Any comments and suggestions greatly appreciated!

If you at least have CMake/CTest installed on the test machines (even if you don't have compilers, libraries, etc.), then you can create a CMake project that doesn't build any code, but instead runs tests against a set of external executables.

The build process for CMake itself has a mechanism to do exactly this: if you configure CMake with -DCMake_TEST_EXTERNAL_CMAKE=path/to/bin, then it will execute the test suite against the external CMake executable without building any code. You could do something similar in your own project: add an option to instead run the test suite against an set of external excutables instead of building them.

Hope that helps!

Kyle

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Split Build and Test Pipelines

Miller Henry
In reply to this post by Dustyn Blasig

We have had luck packaging all the CTestTestFile.cmake files, and using sed to adjust paths for the target location.  You need to install ctest on the test machine, along with all the tests.

 

The only paths we change are the test output location (ie google test .xml files: this might not apply to you), and the path to the build directory.  Something like the following.

 

         sed -i ‘s/$BUILD_DIR/@CTEST_TARGET_DIR@/g’ $file

 

When running we first use configure_file in cmake to substitute CTEST_TARGET_DIR with whatever path things are installed to on the test machine.

 

From: CMake <[hidden email]> On Behalf Of Dustyn Blasig
Sent: Friday, May 10, 2019 12:07 PM
To: CMake <[hidden email]>
Subject: [CMake] Split Build and Test Pipelines

 

Hi All,

 

I'm curious if anyone has had success allowing two testing paths to coexist well.

 

Currently, we are using CTest to run our test executables with `make test`. However, on our Jenkins system, the build machines have the whole development stack but the test machines do not. So we need a way to package up the tests in a way that the test machines can simply run some generated script to do the equivalent of what CTest would do.

 

My current thought is to generate OS specific scripts `make test` depends on and runs. These scripts can get installed through `make install` if ENABLE_TEST_INSTALL is set, and then the test machine can run the installed scripts. The tricky part is building up the hierarchy of scripts in such a way that the tester just needs to run "test.sh" or "test.bat" at the top-level of the install hierarchy.

 

Any comments and suggestions greatly appreciated!

 

Cheers!


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake