ExternalProject() Question

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

ExternalProject() Question

Michael Jackson
I am intrigued by the "ExternalProject" feature of CMake 2.8. One  
question that I have after reading through the Oct 09 "Kitware Source"  
is this. If I do a "make clean" or "rebuild" are all the  
"ExternalProjects" also cleaned/rebuilt? I could make an argument both  
ways but I was curious what the default was?

   One of my projects depends on HDF5, Tiff, Expat and Boost (And Qt  
Eventually). Writing directions to get everything downloaded and built  
is getting to be a pain. Would be simpler to have all that done for  
the user.

Thanks for any comments
_________________________________________________________
Mike Jackson                  [hidden email]
BlueQuartz Software                    www.bluequartz.net
Principal Software Engineer                  Dayton, Ohio

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: ExternalProject() Question

Michael Wild

On 16. Dec, 2009, at 20:35 , Michael Jackson wrote:

> I am intrigued by the "ExternalProject" feature of CMake 2.8. One question that I have after reading through the Oct 09 "Kitware Source" is this. If I do a "make clean" or "rebuild" are all the "ExternalProjects" also cleaned/rebuilt? I could make an argument both ways but I was curious what the default was?
>
>  One of my projects depends on HDF5, Tiff, Expat and Boost (And Qt Eventually). Writing directions to get everything downloaded and built is getting to be a pain. Would be simpler to have all that done for the user.
>
> Thanks for any comments
> _________________________________________________________
> Mike Jackson                  [hidden email]
> BlueQuartz Software                    www.bluequartz.net
> Principal Software Engineer                  Dayton, Ohio
>

"make clean" behaves a bit strange for me... It seems to remove some of the stamp files, but does not actually perform a "clean" on the external project, or wipe the extracted sources. For me, this is a bit of a problem, since the PATCH_COMMAND will try to re-apply a patch, which of course will fail. Not sure how to teach CMake to not re-patch a source tree... Perhaps wrap the whole thing in a apply_patch.cmake script which creates its own stamp in the external source tree and only applies the patch if that stamp does not exist...


Michael
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: ExternalProject() Question

David Cole
On Thu, Dec 17, 2009 at 3:20 AM, Michael Wild <[hidden email]> wrote:

On 16. Dec, 2009, at 20:35 , Michael Jackson wrote:

> I am intrigued by the "ExternalProject" feature of CMake 2.8. One question that I have after reading through the Oct 09 "Kitware Source" is this. If I do a "make clean" or "rebuild" are all the "ExternalProjects" also cleaned/rebuilt? I could make an argument both ways but I was curious what the default was?
>
>  One of my projects depends on HDF5, Tiff, Expat and Boost (And Qt Eventually). Writing directions to get everything downloaded and built is getting to be a pain. Would be simpler to have all that done for the user.
>
> Thanks for any comments
> _________________________________________________________
> Mike Jackson                  [hidden email]
> BlueQuartz Software                    www.bluequartz.net
> Principal Software Engineer                  Dayton, Ohio
>

"make clean" behaves a bit strange for me... It seems to remove some of the stamp files, but does not actually perform a "clean" on the external project, or wipe the extracted sources. For me, this is a bit of a problem, since the PATCH_COMMAND will try to re-apply a patch, which of course will fail. Not sure how to teach CMake to not re-patch a source tree... Perhaps wrap the whole thing in a apply_patch.cmake script which creates its own stamp in the external source tree and only applies the patch if that stamp does not exist...


Michael Wild's observations are correct. The "make clean" of the outer project is going to wipe all the stamp files causing all of the steps to re-run again, including downloads and checkouts.

Since the patch command is empty by default and 'patch' is not readily available on Windows, the patching stuff is less well tested than the rest of the steps.

I guess I would be inclined to say that an outer/main-project "make clean" ought to default to cleaning the inner/nested/external projects as well... but it's not clear to me that there's a reasonable way to implement that quickly and easily. It's certainly worthy of a feature request and some discussion. And whatever the default behavior is, it should probably be controllable by one or more new arguments to the ExternalProject_Add function.

Let me know if you have further ideas and suggestions.


Thanks,
David Cole


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: ExternalProject() Question

Michael Wild

On 17. Dec, 2009, at 15:01 , David Cole wrote:

> On Thu, Dec 17, 2009 at 3:20 AM, Michael Wild <[hidden email]> wrote:
>
>>
>> On 16. Dec, 2009, at 20:35 , Michael Jackson wrote:
>>
>>> I am intrigued by the "ExternalProject" feature of CMake 2.8. One
>> question that I have after reading through the Oct 09 "Kitware Source" is
>> this. If I do a "make clean" or "rebuild" are all the "ExternalProjects"
>> also cleaned/rebuilt? I could make an argument both ways but I was curious
>> what the default was?
>>>
>>> One of my projects depends on HDF5, Tiff, Expat and Boost (And Qt
>> Eventually). Writing directions to get everything downloaded and built is
>> getting to be a pain. Would be simpler to have all that done for the user.
>>>
>>> Thanks for any comments
>>> _________________________________________________________
>>> Mike Jackson                  [hidden email]
>>> BlueQuartz Software                    www.bluequartz.net
>>> Principal Software Engineer                  Dayton, Ohio
>>>
>>
>> "make clean" behaves a bit strange for me... It seems to remove some of the
>> stamp files, but does not actually perform a "clean" on the external
>> project, or wipe the extracted sources. For me, this is a bit of a problem,
>> since the PATCH_COMMAND will try to re-apply a patch, which of course will
>> fail. Not sure how to teach CMake to not re-patch a source tree... Perhaps
>> wrap the whole thing in a apply_patch.cmake script which creates its own
>> stamp in the external source tree and only applies the patch if that stamp
>> does not exist...
>>
>>
> Michael Wild's observations are correct. The "make clean" of the outer
> project is going to wipe all the stamp files causing all of the steps to
> re-run again, including downloads and checkouts.
>
> Since the patch command is empty by default and 'patch' is not readily
> available on Windows, the patching stuff is less well tested than the rest
> of the steps.
>
> I guess I would be inclined to say that an outer/main-project "make clean"
> ought to default to cleaning the inner/nested/external projects as well...
> but it's not clear to me that there's a reasonable way to implement that
> quickly and easily. It's certainly worthy of a feature request and some
> discussion. And whatever the default behavior is, it should probably be
> controllable by one or more new arguments to the ExternalProject_Add
> function.
>
> Let me know if you have further ideas and suggestions.
>
>
> Thanks,
> David Cole


Also, should a "make clean" really trigger a re-download? What I would like to have is a MD5SUM option to ExternalProject_Add. Currently I'm implementing this manually with a custom sub-step, but there's no reason it shouldn't be in CMake, because it natively implements -E md5sum.

Michael
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: ExternalProject() Question

David Cole
On Thu, Dec 17, 2009 at 10:53 AM, Michael Wild <[hidden email]> wrote:

On 17. Dec, 2009, at 15:01 , David Cole wrote:

> On Thu, Dec 17, 2009 at 3:20 AM, Michael Wild <[hidden email]> wrote:
>
>>
>> On 16. Dec, 2009, at 20:35 , Michael Jackson wrote:
>>
>>> I am intrigued by the "ExternalProject" feature of CMake 2.8. One
>> question that I have after reading through the Oct 09 "Kitware Source" is
>> this. If I do a "make clean" or "rebuild" are all the "ExternalProjects"
>> also cleaned/rebuilt? I could make an argument both ways but I was curious
>> what the default was?
>>>
>>> One of my projects depends on HDF5, Tiff, Expat and Boost (And Qt
>> Eventually). Writing directions to get everything downloaded and built is
>> getting to be a pain. Would be simpler to have all that done for the user.
>>>
>>> Thanks for any comments
>>> _________________________________________________________
>>> Mike Jackson                  [hidden email]
>>> BlueQuartz Software                    www.bluequartz.net
>>> Principal Software Engineer                  Dayton, Ohio
>>>
>>
>> "make clean" behaves a bit strange for me... It seems to remove some of the
>> stamp files, but does not actually perform a "clean" on the external
>> project, or wipe the extracted sources. For me, this is a bit of a problem,
>> since the PATCH_COMMAND will try to re-apply a patch, which of course will
>> fail. Not sure how to teach CMake to not re-patch a source tree... Perhaps
>> wrap the whole thing in a apply_patch.cmake script which creates its own
>> stamp in the external source tree and only applies the patch if that stamp
>> does not exist...
>>
>>
> Michael Wild's observations are correct. The "make clean" of the outer
> project is going to wipe all the stamp files causing all of the steps to
> re-run again, including downloads and checkouts.
>
> Since the patch command is empty by default and 'patch' is not readily
> available on Windows, the patching stuff is less well tested than the rest
> of the steps.
>
> I guess I would be inclined to say that an outer/main-project "make clean"
> ought to default to cleaning the inner/nested/external projects as well...
> but it's not clear to me that there's a reasonable way to implement that
> quickly and easily. It's certainly worthy of a feature request and some
> discussion. And whatever the default behavior is, it should probably be
> controllable by one or more new arguments to the ExternalProject_Add
> function.
>
> Let me know if you have further ideas and suggestions.
>
>
> Thanks,
> David Cole


Also, should a "make clean" really trigger a re-download? What I would like to have is a MD5SUM option to ExternalProject_Add. Currently I'm implementing this manually with a custom sub-step, but there's no reason it shouldn't be in CMake, because it natively implements -E md5sum.


The MD5SUM option is a good idea that we have considered. I'm almost certain md5 (or something like it) will appear in a future implementation of ExternalProject_Add.

Ideally, I think "make clean" should destroy all created build products. And in the case of ExternalProject_Add, the downloaded source is a "build product." I understand why you would not want to re-download the source... You could put a *.tar.gz in your source tree and reference it directly there. You might not want to do that, either, though.

Thanks for the discussion.


_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: ExternalProject() Question

Michael Wild

On 17. Dec, 2009, at 17:34 , David Cole wrote:

> On Thu, Dec 17, 2009 at 10:53 AM, Michael Wild <[hidden email]> wrote:
>
>>
>> On 17. Dec, 2009, at 15:01 , David Cole wrote:
>>
>>> On Thu, Dec 17, 2009 at 3:20 AM, Michael Wild <[hidden email]> wrote:
>>>
>>>>
>>>> On 16. Dec, 2009, at 20:35 , Michael Jackson wrote:
>>>>
>>>>> I am intrigued by the "ExternalProject" feature of CMake 2.8. One
>>>> question that I have after reading through the Oct 09 "Kitware Source"
>> is
>>>> this. If I do a "make clean" or "rebuild" are all the "ExternalProjects"
>>>> also cleaned/rebuilt? I could make an argument both ways but I was
>> curious
>>>> what the default was?
>>>>>
>>>>> One of my projects depends on HDF5, Tiff, Expat and Boost (And Qt
>>>> Eventually). Writing directions to get everything downloaded and built
>> is
>>>> getting to be a pain. Would be simpler to have all that done for the
>> user.
>>>>>
>>>>> Thanks for any comments
>>>>> _________________________________________________________
>>>>> Mike Jackson                  [hidden email]
>>>>> BlueQuartz Software                    www.bluequartz.net
>>>>> Principal Software Engineer                  Dayton, Ohio
>>>>>
>>>>
>>>> "make clean" behaves a bit strange for me... It seems to remove some of
>> the
>>>> stamp files, but does not actually perform a "clean" on the external
>>>> project, or wipe the extracted sources. For me, this is a bit of a
>> problem,
>>>> since the PATCH_COMMAND will try to re-apply a patch, which of course
>> will
>>>> fail. Not sure how to teach CMake to not re-patch a source tree...
>> Perhaps
>>>> wrap the whole thing in a apply_patch.cmake script which creates its own
>>>> stamp in the external source tree and only applies the patch if that
>> stamp
>>>> does not exist...
>>>>
>>>>
>>> Michael Wild's observations are correct. The "make clean" of the outer
>>> project is going to wipe all the stamp files causing all of the steps to
>>> re-run again, including downloads and checkouts.
>>>
>>> Since the patch command is empty by default and 'patch' is not readily
>>> available on Windows, the patching stuff is less well tested than the
>> rest
>>> of the steps.
>>>
>>> I guess I would be inclined to say that an outer/main-project "make
>> clean"
>>> ought to default to cleaning the inner/nested/external projects as
>> well...
>>> but it's not clear to me that there's a reasonable way to implement that
>>> quickly and easily. It's certainly worthy of a feature request and some
>>> discussion. And whatever the default behavior is, it should probably be
>>> controllable by one or more new arguments to the ExternalProject_Add
>>> function.
>>>
>>> Let me know if you have further ideas and suggestions.
>>>
>>>
>>> Thanks,
>>> David Cole
>>
>>
>> Also, should a "make clean" really trigger a re-download? What I would like
>> to have is a MD5SUM option to ExternalProject_Add. Currently I'm
>> implementing this manually with a custom sub-step, but there's no reason it
>> shouldn't be in CMake, because it natively implements -E md5sum.
>
>
>
> The MD5SUM option is a good idea that we have considered. I'm almost certain
> md5 (or something like it) will appear in a future implementation of
> ExternalProject_Add.
>
> Ideally, I think "make clean" should destroy all created build products. And
> in the case of ExternalProject_Add, the downloaded source is a "build
> product." I understand why you would not want to re-download the source...
> You could put a *.tar.gz in your source tree and reference it directly
> there. You might not want to do that, either, though.
>
> Thanks for the discussion.

Currently I'm checking whether it is in the source tree and the MD5 matches, and otherwise I download it.

Speaking of the MD5 sum, it would be nice if it would be possible to do something like this:

if(${filename} MD5SUM_EQUAL ${md5sum})


in the CMake code, because using execute_process every time and then parsing the output is a bit tedious.

Michael
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: ExternalProject() Question

Michael Jackson
In reply to this post by David Cole

On Dec 17, 2009, at 11:34 AM, David Cole wrote:

> On Thu, Dec 17, 2009 at 10:53 AM, Michael Wild <[hidden email]>  
> wrote:
>
> On 17. Dec, 2009, at 15:01 , David Cole wrote:
>
> > On Thu, Dec 17, 2009 at 3:20 AM, Michael Wild <[hidden email]>  
> wrote:
> >
> >>
> >> On 16. Dec, 2009, at 20:35 , Michael Jackson wrote:
> >>
> >>> I am intrigued by the "ExternalProject" feature of CMake 2.8. One
> >> question that I have after reading through the Oct 09 "Kitware  
> Source" is
> >> this. If I do a "make clean" or "rebuild" are all the  
> "ExternalProjects"
> >> also cleaned/rebuilt? I could make an argument both ways but I  
> was curious
> >> what the default was?
> >>>
> >>> One of my projects depends on HDF5, Tiff, Expat and Boost (And Qt
> >> Eventually). Writing directions to get everything downloaded and  
> built is
> >> getting to be a pain. Would be simpler to have all that done for  
> the user.
> >>>
> >>> Thanks for any comments
> >>> _________________________________________________________
> >>> Mike Jackson                  [hidden email]
> >>> BlueQuartz Software                    www.bluequartz.net
> >>> Principal Software Engineer                  Dayton, Ohio
> >>>
> >>
> >> "make clean" behaves a bit strange for me... It seems to remove  
> some of the
> >> stamp files, but does not actually perform a "clean" on the  
> external
> >> project, or wipe the extracted sources. For me, this is a bit of  
> a problem,
> >> since the PATCH_COMMAND will try to re-apply a patch, which of  
> course will
> >> fail. Not sure how to teach CMake to not re-patch a source  
> tree... Perhaps
> >> wrap the whole thing in a apply_patch.cmake script which creates  
> its own
> >> stamp in the external source tree and only applies the patch if  
> that stamp
> >> does not exist...
> >>
> >>
> > Michael Wild's observations are correct. The "make clean" of the  
> outer
> > project is going to wipe all the stamp files causing all of the  
> steps to
> > re-run again, including downloads and checkouts.
> >
> > Since the patch command is empty by default and 'patch' is not  
> readily
> > available on Windows, the patching stuff is less well tested than  
> the rest
> > of the steps.
> >
> > I guess I would be inclined to say that an outer/main-project  
> "make clean"
> > ought to default to cleaning the inner/nested/external projects as  
> well...
> > but it's not clear to me that there's a reasonable way to  
> implement that
> > quickly and easily. It's certainly worthy of a feature request and  
> some
> > discussion. And whatever the default behavior is, it should  
> probably be
> > controllable by one or more new arguments to the ExternalProject_Add
> > function.
> >
> > Let me know if you have further ideas and suggestions.
> >
> >
> > Thanks,
> > David Cole
>
>
> Also, should a "make clean" really trigger a re-download? What I  
> would like to have is a MD5SUM option to ExternalProject_Add.  
> Currently I'm implementing this manually with a custom sub-step, but  
> there's no reason it shouldn't be in CMake, because it natively  
> implements -E md5sum.
>
>
> The MD5SUM option is a good idea that we have considered. I'm almost  
> certain md5 (or something like it) will appear in a future  
> implementation of ExternalProject_Add.
>
> Ideally, I think "make clean" should destroy all created build  
> products. And in the case of ExternalProject_Add, the downloaded  
> source is a "build product." I understand why you would not want to  
> re-download the source... You could put a *.tar.gz in your source  
> tree and reference it directly there. You might not want to do that,  
> either, though.
>
> Thanks for the discussion.
>

My thinking was this would be something like a "package" or "port"  
type call. I am describing in my CMakeLists.txt that my project needs  
"tiff" and to download it and compile it first before proceeding with  
the compile of my own project. In this case I am not really working on  
tiff, it is stable, so there is no point in recompiling it again. But  
there are cases where someone may be working on libTiff in conjunction  
with their project and would therefor need it cleaned/Recompiled. I  
guess you would get to the point where each step has a "YES/NO"  
property for each type of build action. Do you want source code  
cleaned? Do you want source code downloaded? ...

    Maybe I should just take the "other road" and bundle my versions  
of Tiff, Expat, HDF5, Boost inside of my project. Then it is one giant  
self contained project which for the "end user" is easy to compile.  
One Download, Once configure/compile. But when _I_ am working on it,  
every time I "clean" my project I have to rebuild EVERYTHING. Which  
sucks on slower computers. ParaView is this way and you really have to  
think twice before hitting that "clean" or "rebuild" button. Just  
saying..

Mike Jackson

_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake