Concurrency issues with copy_if_different

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Concurrency issues with copy_if_different

Olivier Croquette
Hello,

in one of our projects, we use copy_if_different to copy some DLLs
required by the runtime. It's called as post-build action. The problem
is that several targets want to copy the same DLLs, and when using
parallelized builds, the different "cmake -E copy_if_different" can
conflict, leading the whole build to fail.

I see two options to fix this:

1. don't use copy_if_different, but a custom tool that can deal with the
concurrency

2. instead of adding post-build actions to different targets, fill up a
list containing all the files required, and add a single post-build
action to a single target that copies all the required files in one go

What do you think?

Cheers
Olivier


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Concurrency issues with copy_if_different

Robert Maynard
Personally would go with option number 2

On Thu, Jul 12, 2018 at 1:58 AM Olivier Croquette <[hidden email]> wrote:
Hello,

in one of our projects, we use copy_if_different to copy some DLLs
required by the runtime. It's called as post-build action. The problem
is that several targets want to copy the same DLLs, and when using
parallelized builds, the different "cmake -E copy_if_different" can
conflict, leading the whole build to fail.

I see two options to fix this:

1. don't use copy_if_different, but a custom tool that can deal with the
concurrency

2. instead of adding post-build actions to different targets, fill up a
list containing all the files required, and add a single post-build
action to a single target that copies all the required files in one go

What do you think?

Cheers
Olivier


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Concurrency issues with copy_if_different

J. Caleb Wherry
In reply to this post by Olivier Croquette
Had the same problem and went with #2

-Caleb

On Thu, Jul 12, 2018 at 1:58 AM Olivier Croquette <[hidden email]> wrote:
Hello,

in one of our projects, we use copy_if_different to copy some DLLs
required by the runtime. It's called as post-build action. The problem
is that several targets want to copy the same DLLs, and when using
parallelized builds, the different "cmake -E copy_if_different" can
conflict, leading the whole build to fail.

I see two options to fix this:

1. don't use copy_if_different, but a custom tool that can deal with the
concurrency

2. instead of adding post-build actions to different targets, fill up a
list containing all the files required, and add a single post-build
action to a single target that copies all the required files in one go

What do you think?

Cheers
Olivier


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
--
Sent from my iPhone 4s

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Concurrency issues with copy_if_different

Olivier Croquette
On 2018-7-12 15:16, J. Caleb Wherry wrote:

>> in one of our projects, we use copy_if_different to copy some DLLs
>> required by the runtime. It's called as post-build action. The problem
>> is that several targets want to copy the same DLLs, and when using
>> parallelized builds, the different "cmake -E copy_if_different" can
>> conflict, leading the whole build to fail.
>>
>> I see two options to fix this:
>>
>> 1. don't use copy_if_different, but a custom tool that can deal with the
>> concurrency
>>
>> 2. instead of adding post-build actions to different targets, fill up a
>> list containing all the files required, and add a single post-build
>> action to a single target that copies all the required files in one go
>>
>> What do you think?
> Had the same problem and went with #2

Hi Caleb,

could you give some more details how you did this? Filling up the list
is easy, but I am not sure how to define the final action that will copy
all the DLLs, and make sure it happens only once. Did you define a new
custom target for that?


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Concurrency issues with copy_if_different

J. Caleb Wherry
Our case is pretty simplified in that almost every executable uses the dependent DLLs/SOs I need copied. For C# apps it is log4net, NDesk.Options, etc. For native apps it’s fmpeg shared libraries.

So when I create the list at configure time, I immediately just use CMake to copy the files the where I want them in the build tree. This guarantees that they are copied the one time when CMake is run instead of at build time.

A few things that make this work for us:

* These libraries are checked into our repo so they aren’t changed by automatically pulling new binaries or anything like that.

* We only have a handful of these dependent 3rd party libs, like 7.

* Couldn’t think of a better way to do it.

If you did want to move it to build time instead of configure time, you’d just have to make a custom target that every other target depends on. In some systems this isn’t easy but if you write wrappers around creating targets then you can do extra book keeping when a target is create. In this case, you just add another depends on the custom target that copies the dependent libs.

-Caleb

On Sat, Dec 8, 2018 at 7:14 PM Olivier Croquette <[hidden email]> wrote:
On 2018-7-12 15:16, J. Caleb Wherry wrote:
>> in one of our projects, we use copy_if_different to copy some DLLs
>> required by the runtime. It's called as post-build action. The problem
>> is that several targets want to copy the same DLLs, and when using
>> parallelized builds, the different "cmake -E copy_if_different" can
>> conflict, leading the whole build to fail.
>>
>> I see two options to fix this:
>>
>> 1. don't use copy_if_different, but a custom tool that can deal with the
>> concurrency
>>
>> 2. instead of adding post-build actions to different targets, fill up a
>> list containing all the files required, and add a single post-build
>> action to a single target that copies all the required files in one go
>>
>> What do you think?
> Had the same problem and went with #2

Hi Caleb,

could you give some more details how you did this? Filling up the list
is easy, but I am not sure how to define the final action that will copy
all the DLLs, and make sure it happens only once. Did you define a new
custom target for that?


--
Sent from my iPhone SE

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake