custom commands, targets and parallel builds

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

custom commands, targets and parallel builds

Kris Thielemans-2

Hi all

 

I’m trying to add doxygen-generated comments to my Python module build via SWIG using https://github.com/m7thon/doxy2swig. This means I need to run doxygen first, then run doxy2swig, and only then run swig. However, I’m getting reports that parallel builds are failing (sometimes).

 

My strategy is based on CMake advice and some blogs (e.g. at https://samthursfield.wordpress.com/2015/11/21/cmake-dependencies-between-targets-and-files-and-custom-commands/ ) and relies on creating custom commands to generate the files and custom targets to be able to specify dependencies on those generated files. Somewhat simplified code is below (original is at https://github.com/UCL/STIR/pull/280)

 

# command that runs doxygen

add_custom_command(

OUTPUT doxygen.stamp

DEPENDS ${doxyfile}

COMMAND ${DOXYGEN_EXECUTABLE} ${doxyfile}

COMMAND cmake -E touch doxygen.stamp

…)

# corresponding target

add_custom_target( doc ALL DEPENDS doxygen.stamp)

 

# command that runs doxy2swig

add_custom_command(

OUTPUT STIR_DOXY2SWIG.i

DEPENDS doc

COMMAND ${PYTHON_EXECUTABLE} doxy2swig.py -c index.xml STIR_DOXY2SWIG.i

)

# corresponding target

add_custom_target(doxy2swig DEPENDS STIR_DOXY2SWIG.i)

 

# now add that target to the SIWG module

add_dependencies(${SWIG_MODULE_stir_REAL_NAME} doxy2swig)

 

Any suggestions on what I’m doing wrong?

 

Many thanks!

Kris

 

PS: I note that FindDoxygen’s doxygen_add_docs merges the add_custom_command and target into one add_custom_target, see https://github.com/Kitware/CMake/blob/a44191abc489373d295ecaeb5c2eb1573c876a1a/Modules/FindDoxygen.cmake#L1104

I thought that couldn’t be done but I guess I was wrong.

PS: I also note that doxygen_add_docs doesn’t create a doxygen.stamp file to prevent re-runs though, is that not needed then?


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: custom commands, targets and parallel builds

Alan W. Irwin-2
On 2018-11-25 00:23-0000 Kris Thielemans wrote:

> Hi all
>
>
>
> I'm trying to add doxygen-generated comments to my Python module build via
> SWIG using https://github.com/m7thon/doxy2swig. This means I need to run
> doxygen first, then run doxy2swig, and only then run swig. However, I'm
> getting reports that parallel builds are failing (sometimes).
>
>
>
> My strategy is based on CMake advice and some blogs (e.g. at
> https://samthursfield.wordpress.com/2015/11/21/cmake-dependencies-between-ta
> rgets-and-files-and-custom-commands/ ) and relies on creating custom
> commands to generate the files and custom targets to be able to specify
> dependencies on those generated files. Somewhat simplified code is below
> (original is at https://github.com/UCL/STIR/pull/280)
>
>
>
> # command that runs doxygen
>
> add_custom_command(
>
> OUTPUT doxygen.stamp
>
> DEPENDS ${doxyfile}
>
> COMMAND ${DOXYGEN_EXECUTABLE} ${doxyfile}
>
> COMMAND cmake -E touch doxygen.stamp
>
> .)
>
> # corresponding target
>
> add_custom_target( doc ALL DEPENDS doxygen.stamp)
>
>
>
> # command that runs doxy2swig
>
> add_custom_command(
>
> OUTPUT STIR_DOXY2SWIG.i
>
> DEPENDS doc
>
> COMMAND ${PYTHON_EXECUTABLE} doxy2swig.py -c index.xml STIR_DOXY2SWIG.i
>
> )
>
> # corresponding target
>
> add_custom_target(doxy2swig DEPENDS STIR_DOXY2SWIG.i)
>
>
>
> # now add that target to the SIWG module
>
> add_dependencies(${SWIG_MODULE_stir_REAL_NAME} doxy2swig)
>
>
>
> Any suggestions on what I'm doing wrong?
>
>
>
> Many thanks!
>
> Kris
>
>
>
> PS: I note that FindDoxygen's doxygen_add_docs merges the add_custom_command
> and target into one add_custom_target, see
> https://github.com/Kitware/CMake/blob/a44191abc489373d295ecaeb5c2eb1573c876a
> 1a/Modules/FindDoxygen.cmake#L1104
>
> I thought that couldn't be done but I guess I was wrong.
>
> PS: I also note that doxygen_add_docs doesn't create a doxygen.stamp file to
> prevent re-runs though, is that not needed then?

Hi Kris:

To answer your last set of questions first, you can execute all
desired commands with add_custom_target, but the issue is that COMMAND
always reruns.  So your way of doing it above (with paired
add_custom_command/add_custom_target) is the recommended procedure
which should only re-run if the OUTPUT file is non-existent or older
than a dependent file.

Anyhow, I think what you have outlined above is generally correct.
For example, your "DEPENDS doc" argument to add_custom_command where
doc is a custom target argument is correct according to the
documentation at
<https://cmake.org/cmake/help/latest/command/add_custom_command.html>.
However, your simplified code above has several deviations from the
pattern I always use.  I know my pattern generally works, and I think
at least your first two deviations from it are important.

1.  I always like to specify the full pathname for all files (e.g.,
${CMAKE_CURRENT_BINARY_DIR}/doxygen.stamp and
${CMAKE_CURRENT_BINARY_DIR}/STIR_DOXY2SWIG.i.  This is true not only
for OUTPUT files and the corresponding DEPENDS but also for input
(which typically start with ${CMAKE_CURRENT_SOURCE_DIR} and output
files for COMMANDs.

2. At the same time I make sure I run all commands in
${CMAKE_CURRENT_BINARY_DIR} by specifying that as the working
directory.  So that all unmentioned files on the command line are
generated in the build tree to keep the source tree as clean as
possible.

3. I don't think it matters above but I always use the VERBATIM
attribute for COMMANDS.

I have found one of my most difficult CMake tasks is to avoid build
race conditions that can occur for parallel builds.  So your real code
is most important in that regard rather than a simplified example of
your CMake logic.  Perhaps others here would like to comment on the
most reliable way to find such race conditions.  But what I do is
process "make -j<jobs> <target_name>" output results to remove the
progress percentage marks, sort those results, and look for anything
that is repeated that shouldn't be.  But even such care is not
sufficient, and I recently got help on this list from someone who
found a race condition (several different commands using the same
temporary filename to store intermediate results) in the build system
for PLplot.  That issue had been around for years!

Finally, although I have found parallel builds to be reliable on
Linux, Arjen Markus, one of the PLplot developers, has discovered that
parallel builds are currently unreliable for all Unix-like Windows
platforms, e.g., classical MinGW/MSYS, its modern replacement
MinGW-w64/MSYS2, and Cygwin. So if some of your users are reporting
parallel build troubles on any of those platforms it is likely an
issue with the make command on those platforms rather than your build
system.

Alan
__________________________
Alan W. Irwin

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: custom commands, targets and parallel builds

Kris Thielemans-2
> From: Alan W. Irwin,  Sent: 25 November 2018 01:48

Many thanks Alan, answers and questions below

> On 2018-11-25 00:23-0000 Kris Thielemans wrote:
>
> > I'm trying to add doxygen-generated comments to my Python module build
> > via SWIG using https://github.com/m7thon/doxy2swig. This means I need
> > to run doxygen first, then run doxy2swig, and only then run swig.
> > However, I'm getting reports that parallel builds are failing
(sometimes).
> >
> >
> >
> > My strategy is based on CMake advice and some blogs (e.g. at
> >
https://samthursfield.wordpress.com/2015/11/21/cmake-dependencies-between-ta
rgets-and-files-and-custom-commands/ ) and relies on creating

> > custom commands to generate the files and custom targets to be able to
> > specify dependencies on those generated files. Somewhat simplified
> > code is below (original is at https://github.com/UCL/STIR/pull/280)
> >
> >
> >
> > # command that runs doxygen
> >
> > add_custom_command(
> >
> > OUTPUT doxygen.stamp
> >
> > DEPENDS ${doxyfile}
> >
> > COMMAND ${DOXYGEN_EXECUTABLE} ${doxyfile}
> >
> > COMMAND cmake -E touch doxygen.stamp
> >
> > .)
> >
> > # corresponding target
> >
> > add_custom_target( doc ALL DEPENDS doxygen.stamp)
> >
> >
> >
> > # command that runs doxy2swig
> >
> > add_custom_command(
> >
> > OUTPUT STIR_DOXY2SWIG.i
> >
> > DEPENDS doc
> >
> > COMMAND ${PYTHON_EXECUTABLE} doxy2swig.py -c index.xml
> > STIR_DOXY2SWIG.i
> >
> > )
> >
> > # corresponding target
> >
> > add_custom_target(doxy2swig DEPENDS STIR_DOXY2SWIG.i)
> >
> >
> >
> > # now add that target to the SIWG module
> >
> > add_dependencies(${SWIG_MODULE_stir_REAL_NAME} doxy2swig)
> >
> >
> >
> > Any suggestions on what I'm doing wrong?
> >
> >
> >
> > Many thanks!
> >
> > Kris
> >
> >
> >
> > PS: I note that FindDoxygen's doxygen_add_docs merges the
> > add_custom_command and target into one add_custom_target, see
> >
>
https://github.com/Kitware/CMake/blob/a44191abc489373d295ecaeb5c2eb153c876a1
a/Modules/FindDoxygen.cmake#L1104

> >
> > I thought that couldn't be done but I guess I was wrong.
> >
> > PS: I also note that doxygen_add_docs doesn't create a doxygen.stamp
> > file to prevent re-runs though, is that not needed then?
>
> Hi Kris:
>
> To answer your last set of questions first, you can execute all desired
> commands with add_custom_target, but the issue is that COMMAND always
> reruns.  So your way of doing it above (with paired
> add_custom_command/add_custom_target) is the recommended procedure
> which should only re-run if the OUTPUT file is non-existent or older than
a
> dependent file.
>

Ok. That makes sense to me. thanks!

> Anyhow, I think what you have outlined above is generally correct.
> For example, your "DEPENDS doc" argument to add_custom_command where
> doc is a custom target argument is correct according to the documentation
at
> <https://cmake.org/cmake/help/latest/command/add_custom_command.html
> >.
> However, your simplified code above has several deviations from the
pattern I
> always use.  I know my pattern generally works, and I think at least your
first

> two deviations from it are important.
>
> 1.  I always like to specify the full pathname for all files (e.g.,
> ${CMAKE_CURRENT_BINARY_DIR}/doxygen.stamp and
> ${CMAKE_CURRENT_BINARY_DIR}/STIR_DOXY2SWIG.i.  This is true not only for
> OUTPUT files and the corresponding DEPENDS but also for input (which
> typically start with ${CMAKE_CURRENT_SOURCE_DIR} and output files for
> COMMANDs.
>
> 2. At the same time I make sure I run all commands in
> ${CMAKE_CURRENT_BINARY_DIR} by specifying that as the working directory.
> So that all unmentioned files on the command line are generated in the
build
> tree to keep the source tree as clean as possible.
>
> 3. I don't think it matters above but I always use the VERBATIM attribute
for
> COMMANDS.
>

Thanks for these suggestions. RE 2 and 3, yes, I did the same. For the sake
of brevity I cut some of the relevant bits. Sorry! (full code at
https://github.com/UCL/STIR/pull/280)

I'm indeed not using the full pathnames. However, given the
WORKING_DIRECTORY option, all files are "local" as far as CMake concerns, it
seems to me this is not relevant, although I should give it try.

> I have found one of my most difficult CMake tasks is to avoid build race
> conditions that can occur for parallel builds.  So your real code is most
> important in that regard rather than a simplified example of your CMake
logic.
> Perhaps others here would like to comment on the most reliable way to find
> such race conditions.  But what I do is process "make -j<jobs>
<target_name>"
> output results to remove the progress percentage marks, sort those
results,
> and look for anything that is repeated that shouldn't be.  

Sounds painful :-; The person who reported the problem can currently not
provide me with a log file. He's on OSX but I don't know yet what CMake/make
versions he's using.

So instead I did some more digging in the generated makefiles. From these,
it seems all is ok (see end of email). I'm now puzzled of course.

> Finally, although I have found parallel builds to be reliable on Linux,
Arjen
> Markus, one of the PLplot developers, has discovered that parallel builds
are
> currently unreliable for all Unix-like Windows platforms, e.g., classical
> MinGW/MSYS, its modern replacement MinGW-w64/MSYS2, and Cygwin. So if
> some of your users are reporting parallel build troubles on any of those
> platforms it is likely an issue with the make command on those platforms
> rather than your build system.
>

That is pretty amazing. Who knows, maybe "make" on OSX should be added to
the list of problematic "make" implementations as well.

If I can get dig out anything else on reproducing the problem or debugging,
I'll let you know. Thanks again for your help!

Kris

PS: some checks on build files

== Visual Studio 2015 generated files (with CMake 3.11.4):

My swig module _stir has a reference "doxy2swig", which has a reference
"doc", so that seems fine. (I'm assuming that a reference means it'll check
dependencies).

== Unix Makefiles generated by CMake 3.7 and 3.13.0:
I first got confused by the fact that cross-directory dependencies in the
makefiles in each sub-dir are non-existent, but I realised that "make" is
called for each of these from the "master" makefile. Checking
CMakeFiles/Makefile2 in the root of my build, everything seems fine as this
contains dependencies:

src/swig/CMakeFiles/doxy2swig.dir/all: src/CMakeFiles/doc.dir/all
src/swig/CMakeFiles/_stir.dir/all: src/swig/CMakeFiles/doxy2swig.dir/all

(I've now learned that using "make" from in one of the subdirs is a
dangerous thing to do as it doesn't check dependencies properly, but that
makes some sense.)

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake