Detailed graphviz graph?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Detailed graphviz graph?

Jan Wielemaker
Hi,

Debugging dependencies is not always easy.  The -graphviz option is a
nice try, but only seems to do the built-in target types.  Is there
some way to get the whole dependency graph, including custom targets
and possibly also the individual files?  Of course this can get huge.
Possibly there is some way to concentrate on anything relevant to a
particular target?

The problem I'm faced with is this:

   - I have an ordinary executable target `swipl`
   - To run, this requires a boot file `swipl.prc` that is created
     by calling `swipl -b ...`
   - I'd like to run `swipl` for creating a library index file.

So, these targets need to be built in the order above.  It turns
out that sometimes step 3 runs before 2 completes.  At least, this
happens on MacOS using cmake 3.11.2.  So far I haven't seen it on
Linux (where I use 3.10).  The definition goes like this:

add_custom_command(
     OUTPUT  swipl.prc
     COMMAND swipl -O -b ${SWIPL_BOOT_ROOT}/init.pl
     DEPENDS swipl ${SWIPL_BOOT_FILES}
)

add_custom_command(
     OUTPUT  ${PL_LIB_INDEX}
     COMMAND swipl -f none -g
"\"make_library_index('${SWIPL_LIBRARY_ROOT}')\"" -t halt
     DEPENDS swipl.prc ${PL_LIB_FILES_ALL}
)

add_custom_target(prolog_products ALL
     DEPENDS swipl.prc ${PL_LIB_INDEX}
)

The first specifies building swipl.prc, the second the index
file and the custom target ensures the default build will
create both files.  I don't see what is wrong and a visual
dependency graph might reveal this ...

        Thanks --- Jan
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Detailed graphviz graph?

Eric Noulard


Le mer. 3 oct. 2018 à 09:19, Jan Wielemaker <[hidden email]> a écrit :
Hi,

Debugging dependencies is not always easy.  The -graphviz option is a
nice try, but only seems to do the built-in target types.  Is there
some way to get the whole dependency graph, including custom targets
and possibly also the individual files? 

Not yet I guess:

 
Of course this can get huge.
Possibly there is some way to concentrate on anything relevant to a
particular target?

Besides the custom target issue. I did craft a python script which loads the dependency graph spitted out by CMake
and do various thing on it:

- search if there exist a path between two targets
- find all path between two targets
- reduce the graph (transitive closure)
etc...
 
If you use ninja Generator you may try the browse or graph 
extra tool:

This is generator specific and it may be difficult to follow your "original" CMake target in the generated ninja version.


The problem I'm faced with is this:

   - I have an ordinary executable target `swipl`
   - To run, this requires a boot file `swipl.prc` that is created
     by calling `swipl -b ...`

Hum... I don't get it.
For running? creating? target "swipl" you need to run it? 
There is a chicken & eggs problem or I misread what you said?
 
   - I'd like to run `swipl` for creating a library index file.

So, these targets need to be built in the order above.  It turns
out that sometimes step 3 runs before 2 completes.  At least, this
happens on MacOS using cmake 3.11.2.  So far I haven't seen it on
Linux (where I use 3.10).  The definition goes like this:

add_custom_command(
     OUTPUT  swipl.prc
     COMMAND swipl -O -b ${SWIPL_BOOT_ROOT}/init.pl
     DEPENDS swipl ${SWIPL_BOOT_FILES}
)

add_custom_command(
     OUTPUT  ${PL_LIB_INDEX}
     COMMAND swipl -f none -g
"\"make_library_index('${SWIPL_LIBRARY_ROOT}')\"" -t halt
     DEPENDS swipl.prc ${PL_LIB_FILES_ALL}
)

add_custom_target(prolog_products ALL
     DEPENDS swipl.prc ${PL_LIB_INDEX}
)

The first specifies building swipl.prc, the second the index
file and the custom target ensures the default build will
create both files.  I don't see what is wrong and a visual
dependency graph might reveal this ...

Are those three add_custom_xxx in the same directory?
DEPENDS for custom_xxx do not cross directory.

i.e:
``DEPENDS``
  Reference files and outputs of custom commands created with
  ``add_custom_command()`` command calls in the same directory
  (``CMakeLists.txt`` file).  They will be brought up to date when
  the target is built.


--
Eric

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Detailed graphviz graph?

Jan Wielemaker
On 10/03/2018 09:53 AM, Eric Noulard wrote:

>
>
> Le mer. 3 oct. 2018 à 09:19, Jan Wielemaker <[hidden email]
> <mailto:[hidden email]>> a écrit :
>
>     Hi,
>
>     Debugging dependencies is not always easy.  The -graphviz option is a
>     nice try, but only seems to do the built-in target types.  Is there
>     some way to get the whole dependency graph, including custom targets
>     and possibly also the individual files?
>
>
> Not yet I guess:
>
> https://gitlab.kitware.com/cmake/cmake/issues/17960
>
>     Of course this can get huge.
>     Possibly there is some way to concentrate on anything relevant to a
>     particular target?
>
>
> Besides the custom target issue. I did craft a python script which loads
> the dependency graph spitted out by CMake
> and do various thing on it:
>
> - search if there exist a path between two targets
> - find all path between two targets
> - reduce the graph (transitive closure)
> etc...
> If you use ninja Generator you may try the browse or graph
> extra tool:
> https://ninja-build.org/manual.html
>
> This is generator specific and it may be difficult to follow your
> "original" CMake target in the generated ninja version.

Good to see this coming!  Yes, I use Ninja.  In fact I have some
suspicion I didn't see this problem when still using make (-j).  As
I dive into this I'll try to verify this.   If they really differ
I guess we are either dealing with timing differences or a broken
generator.

>     The problem I'm faced with is this:
>
>         - I have an ordinary executable target `swipl`
>         - To run, this requires a boot file `swipl.prc` that is created
>           by calling `swipl -b ...`
>
>
> Hum... I don't get it.
> For running? creating? target "swipl" you need to run it?
> There is a chicken & eggs problem or I misread what you said?

Nope. Poor wording by me. The executable swipl runs fine, but to really
do what one expects from a Prolog system it requires the parts of the
system written in Prolog to be compiled to Prolog VM code. To do so,
swipl contains a C defined mini compiler that can compile a subset of
the language, just enough to compile the real compiler written mostly in
Prolog. The result is dumped in the file swipl.prc. So, given swipl and
the Prolog boot files we first need to call

        swipl -O -b boot/init.pl # simplified

and next we can call `swipl arg ...` to run normal Prolog jobs such
as creating an index of the Prolog library.

>         - I'd like to run `swipl` for creating a library index file.
>
>     So, these targets need to be built in the order above.  It turns
>     out that sometimes step 3 runs before 2 completes.  At least, this
>     happens on MacOS using cmake 3.11.2.  So far I haven't seen it on
>     Linux (where I use 3.10).  The definition goes like this:
>
>     add_custom_command(
>           OUTPUT  swipl.prc
>           COMMAND swipl -O -b ${SWIPL_BOOT_ROOT}/init.pl
>           DEPENDS swipl ${SWIPL_BOOT_FILES}
>     )
>
>     add_custom_command(
>           OUTPUT  ${PL_LIB_INDEX}
>           COMMAND swipl -f none -g
>     "\"make_library_index('${SWIPL_LIBRARY_ROOT}')\"" -t halt
>           DEPENDS swipl.prc ${PL_LIB_FILES_ALL}
>     )
>
>     add_custom_target(prolog_products ALL
>           DEPENDS swipl.prc ${PL_LIB_INDEX}
>     )
>
>     The first specifies building swipl.prc, the second the index
>     file and the custom target ensures the default build will
>     create both files.  I don't see what is wrong and a visual
>     dependency graph might reveal this ...
>
>
> Are those three add_custom_xxx in the same directory?
> DEPENDS for custom_xxx do not cross directory.

Yip. In fact, they appear in (one of the) CMakeLists.txt together as you
see above.

I'll first have a look at the Ninja analysis tools. Thanks for the
hints. I'm pretty happy with cmake/ninja. Converting a huge complicated
project as introduction to cmake/ninja poses a bit steep learning curve
though :)

        Cheers --- Jan

>
> i.e:
> ``DEPENDS``
>    Reference files and outputs of custom commands created with
>    ``add_custom_command()`` command calls in the same directory
>    (``CMakeLists.txt`` file).  They will be brought up to date when
>    the target is built.
>
>
> --
> Eric

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Detailed graphviz graph?

Jan Wielemaker
In reply to this post by Eric Noulard
On 03/10/18 09:53, Eric Noulard wrote:

>
>         - I have an ordinary executable target `swipl`
>         - To run, this requires a boot file `swipl.prc` that is created
>           by calling `swipl -b ...`
>
>
> Hum... I don't get it.
> For running? creating? target "swipl" you need to run it?
> There is a chicken & eggs problem or I misread what you said?
>
>         - I'd like to run `swipl` for creating a library index file.
>
>     So, these targets need to be built in the order above.  It turns
>     out that sometimes step 3 runs before 2 completes.  At least, this
>     happens on MacOS using cmake 3.11.2.  So far I haven't seen it on
>     Linux (where I use 3.10).  The definition goes like this:
>
>     add_custom_command(
>           OUTPUT  swipl.prc
>           COMMAND swipl -O -b ${SWIPL_BOOT_ROOT}/init.pl <http://init.pl>
>           DEPENDS swipl ${SWIPL_BOOT_FILES}
>     )
>
>     add_custom_command(
>           OUTPUT  ${PL_LIB_INDEX}
>           COMMAND swipl -f none -g
>     "\"make_library_index('${SWIPL_LIBRARY_ROOT}')\"" -t halt
>           DEPENDS swipl.prc ${PL_LIB_FILES_ALL}
>     )
>
>     add_custom_target(prolog_products ALL
>           DEPENDS swipl.prc ${PL_LIB_INDEX}
>     )
>
>     The first specifies building swipl.prc, the second the index
>     file and the custom target ensures the default build will
>     create both files.  I don't see what is wrong and a visual
>     dependency graph might reveal this ...
>
>
> Are those three add_custom_xxx in the same directory?
> DEPENDS for custom_xxx do not cross directory.

So this was not the problem. The problem was that on the Mac I had build
the system before using the autoconf/make suite, which had created
`swipl.prc` in the source tree. Now, this is the second time I've been
fooled by this:

If a custom command has a DEPENDS, this points at the
CMAKE_CURRENT_SOURCE_DIR if the file exists there and at the
CMAKE_CURRENT_BINARY_DIR otherwise.  This makes sense, but
easily leads to things that are hard to figure out ...

Thanks. In particular for pointing at Ninja's tools for figuring out
dependencies.

        Cheers --- Jan
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Detailed graphviz graph?

Eric Noulard
Le sam. 6 oct. 2018 à 18:15, Jan Wielemaker <[hidden email]> a écrit :
On 03/10/18 09:53, Eric Noulard wrote:
>
> Are those three add_custom_xxx in the same directory?
> DEPENDS for custom_xxx do not cross directory.

So this was not the problem. The problem was that on the Mac I had build
the system before using the autoconf/make suite, which had created
`swipl.prc` in the source tree. Now, this is the second time I've been
fooled by this:

Good to know you've found the issue.
 
If a custom command has a DEPENDS, this points at the
CMAKE_CURRENT_SOURCE_DIR if the file exists there and at the
CMAKE_CURRENT_BINARY_DIR otherwise.  This makes sense, but
easily leads to things that are hard to figure out ...

I've already been bitten by that kind of issue.
Now when I want to be "extra-sure" of my source/build tree state I:

- remove the build tree completely
- use the VCS at hand to clean the source tree, e.g. with git
  git -fdx being in the root source dir should remove all non versioned files (or directories) including ignore ones.
(there are more interesting feature to git-clean: https://git-scm.com/docs/git-clean)

Then you can be sure to have a "clean" starting point.
Alternatively checking out the project in a separate place works as well.

Thanks. In particular for pointing at Ninja's tools for figuring out
dependencies.

You are welcome.
 
--
Eric

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake