protobuf and imports relative to root (and --proto_path)

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

protobuf and imports relative to root (and --proto_path)

David Jobet
Hello,

I'm trying to port an existing project from premake to cmake.
I'm trying to avoid modifying the source files while doing so.

Right now, we have several libraries (read in different directories) using proto files with imports between them.
All imports are made relative to the root of the project.

e.g :
work/lib1/sublib1/a.proto
work/lib1/sublib1/b.proto

with a.proto having a link to b.proto like this "import lib1/sublib1/b.proto"

If I compile this with an unchanged FindProtobuf.cmake, I do this :

protobuf_generate_cpp(PROTO_SRCS PROTO_HRDS a.proto b.proto)
add_library(...)

the problem being that it looks like the "namespaces" generated by protoc in b.proto do not match the one used in a.proto : it does not compile.

Is there a way to make this work ?

Otherwise, I made the patch below which solves my problem. Do you think it could be included in cmake ?
it defines a new option PROTO_PATH, so the decl above becomes
protobuf_generate_cpp(PROTO_SRCS PROTO_HRDS PROTO_PATH ${CMAKE_SOURCE_DIR} a.proto b.proto)

With regards

David

$ diff share/cmake-3.10/Modules/FindProtobuf.cmake.orig share/cmake-3.10/Modules/FindProtobuf.cmake
123c123
<   cmake_parse_arguments(protobuf "" "EXPORT_MACRO;DESCRIPTORS" "" ${ARGN})
---
>   cmake_parse_arguments(protobuf "" "PROTO_PATH;EXPORT_MACRO;DESCRIPTORS" "" ${ARGN})
130a131,139
>   set(EXTRA_ARGS "")
>   set(OUT_PATH ${CMAKE_CURRENT_BINARY_DIR})
>   if(protobuf_PROTO_PATH)
>     get_filename_component(ABS_PROTO_PATH ${protobuf_PROTO_PATH} ABSOLUTE)
>     set(EXTRA_ARGS "--proto_path=${ABS_PROTO_PATH}")
>     file(RELATIVE_PATH PROTO_REL_PATH ${CMAKE_SOURCE_DIR} ${ABS_PROTO_PATH})
>     set(OUT_PATH ${CMAKE_BINARY_DIR}/${PROTO_REL_PATH})
>   endif()

198c207,208
<                "--cpp_out=${DLL_EXPORT_DECL}${CMAKE_CURRENT_BINARY_DIR}"
---
>                ${EXTRA_ARGS}
>                "--cpp_out=${DLL_EXPORT_DECL}${OUT_PATH}"

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: protobuf and imports relative to root (and --proto_path)

Alexander Neundorf
On 2018 M08 23, Thu 12:50:14 CEST David Jobet wrote:

> Hello,
>
> I'm trying to port an existing project from premake to cmake.
> I'm trying to avoid modifying the source files while doing so.
>
> Right now, we have several libraries (read in different directories) using
> proto files with imports between them.
> All imports are made relative to the root of the project.
>
> e.g :
> work/lib1/sublib1/a.proto
> work/lib1/sublib1/b.proto
>
> with a.proto having a link to b.proto like this "import
> lib1/sublib1/b.proto"
>
> If I compile this with an unchanged FindProtobuf.cmake, I do this :
>
> protobuf_generate_cpp(PROTO_SRCS PROTO_HRDS a.proto b.proto)
> add_library(...)
>
> the problem being that it looks like the "namespaces" generated by protoc
> in b.proto do not match the one used in a.proto : it does not compile.

do you mean protoc fails, or that the C++ compiler fails when compiling the
generated file ?
I'm also using proto files with imports between them and don't have problems
with finding them.
Are you setting the PROTOBUF_IMPORT_DIRS variable ?

What I do have a problem is rerunning protoc automatically on proto-files which
import proto files that have been modified.
How do you handle this ?

> Is there a way to make this work ?
>
> Otherwise, I made the patch below which solves my problem. Do you think it
> could be included in cmake ?
> it defines a new option PROTO_PATH, so the decl above becomes
> protobuf_generate_cpp(PROTO_SRCS PROTO_HRDS PROTO_PATH ${CMAKE_SOURCE_DIR}
> a.proto b.proto)
>
> With regards
>
> David
>
> $ diff share/cmake-3.10/Modules/FindProtobuf.cmake.orig
> share/cmake-3.10/Modules/FindProtobuf.cmake

can you please use diff -bup ?
This makes the patch easier to read.

Thanks
Alex

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: protobuf and imports relative to root (and --proto_path)

David Jobet
Hello, thanks for your answer.

Here's a mockup :
projectroot/lib1
projectroot/lib1/CMakeLists.txt
projectroot/lib1/a.proto
projectroot/lib1/b.proto
projectroot/lib1/test.cc

See below for the content of the files.

Here's the cmake output :
$ cmake --build build
[0/1] Re-running CMake...
-- Configuring done
-- Generating done
-- Build files have been written to: XXX/temp/build
[1/3] Building CXX object lib1/CMakeFiles/protolib1.dir/a.pb.cc.o
FAILED: lib1/CMakeFiles/protolib1.dir/a.pb.cc.o
XXX/bin/g++   -I. -isystem XXX/include -std=c++14 -Wall   -std=c++14
-MD -MT lib1/CMakeFiles/protolib1.dir/a.pb.cc.o -MF
lib1/CMakeFiles/protolib1.dir/a.pb.cc.o.d -o lib1/CMakeFiles/protoli
b1.dir/a.pb.cc.o -c lib1/a.pb.cc
lib1/a.pb.cc: In function 'void protobuf_AddDesc_a_2eproto()':
lib1/a.pb.cc:79:3: error: '::protobuf_AddDesc_lib1_2fb_2eproto' has
not been declared
  ::protobuf_AddDesc_lib1_2fb_2eproto();
  ^~
ninja: build stopped: subcommand failed.

Here's what's generated from b.proto :
$ cat build/lib1/b.pb.h | grep AddDesc
void  protobuf_AddDesc_b_2eproto();
 friend void  protobuf_AddDesc_b_2eproto();


If I remove the PROTOBUF_IMPORT_DIRS and add "PROTO_PATH
${CMAKE_SOURCE_DIR}" on protobuf_generate_cpp to use my patch (this
injects --proto_path on protoc command line), everything works.
e.g :
#set(PROTOBUF_IMPORT_DIRS ${CMAKE_SOURCE_DIR})
protobuf_generate_cpp(PROTO_SRCS PROTO_HRDS PROTO_PATH
${CMAKE_SOURCE_DIR} a.proto b.proto)

And indeed, the namespace is now correct :
$ cat build/lib1/b.pb.h | grep AddDesc
void  protobuf_AddDesc_lib1_2fb_2eproto();
 friend void  protobuf_AddDesc_lib1_2fb_2eproto();

Again, does anyone know how to make this works without the patch ?
If not, how do someone proposes a patch on cmake ? (posting to the
developper mailing list ?)

David


File contents

-----

$ cat lib1/CMakeLists.txt
project(lib1 LANGUAGES CXX C)

find_package(Protobuf REQUIRED)

set(PROTOBUF_IMPORT_DIRS ${CMAKE_SOURCE_DIR})
protobuf_generate_cpp(PROTO_SRCS PROTO_HRDS a.proto b.proto)
add_library(protolib1 STATIC ${PROTO_SRCS} ${PROTO_HRDS})
target_include_directories(protolib1 PUBLIC ${CMAKE_BINARY_DIR})
target_link_libraries(protolib1 PUBLIC protobuf::libprotobuf)

add_library(lib1 test.cc)
target_include_directories(lib1 PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})
target_link_libraries(lib1 PRIVATE protolib1)

-----

$ cat lib1/a.proto
import "lib1/b.proto";

message A
{
 optional string toto = 1;
 optional BType  type = 2;
}

-----

$ cat lib1/b.proto
message BType
{
   optional double d = 1;
}

-----

$ cat lib1/test.cc
// Protocol buffer
#include <google/protobuf/text_format.h>
#include "lib1/a.pb.h"
On Thu, Aug 23, 2018 at 9:36 PM Alexander Neundorf <[hidden email]> wrote:

>
> On 2018 M08 23, Thu 12:50:14 CEST David Jobet wrote:
> > Hello,
> >
> > I'm trying to port an existing project from premake to cmake.
> > I'm trying to avoid modifying the source files while doing so.
> >
> > Right now, we have several libraries (read in different directories) using
> > proto files with imports between them.
> > All imports are made relative to the root of the project.
> >
> > e.g :
> > work/lib1/sublib1/a.proto
> > work/lib1/sublib1/b.proto
> >
> > with a.proto having a link to b.proto like this "import
> > lib1/sublib1/b.proto"
> >
> > If I compile this with an unchanged FindProtobuf.cmake, I do this :
> >
> > protobuf_generate_cpp(PROTO_SRCS PROTO_HRDS a.proto b.proto)
> > add_library(...)
> >
> > the problem being that it looks like the "namespaces" generated by protoc
> > in b.proto do not match the one used in a.proto : it does not compile.
>
> do you mean protoc fails, or that the C++ compiler fails when compiling the
> generated file ?
> I'm also using proto files with imports between them and don't have problems
> with finding them.
> Are you setting the PROTOBUF_IMPORT_DIRS variable ?
>
> What I do have a problem is rerunning protoc automatically on proto-files which
> import proto files that have been modified.
> How do you handle this ?
>
> > Is there a way to make this work ?
> >
> > Otherwise, I made the patch below which solves my problem. Do you think it
> > could be included in cmake ?
> > it defines a new option PROTO_PATH, so the decl above becomes
> > protobuf_generate_cpp(PROTO_SRCS PROTO_HRDS PROTO_PATH ${CMAKE_SOURCE_DIR}
> > a.proto b.proto)
> >
> > With regards
> >
> > David
> >
> > $ diff share/cmake-3.10/Modules/FindProtobuf.cmake.orig
> > share/cmake-3.10/Modules/FindProtobuf.cmake
>
> can you please use diff -bup ?
> This makes the patch easier to read.
>
> Thanks
> Alex
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: protobuf and imports relative to root (and --proto_path)

David Jobet
In reply to this post by Alexander Neundorf
> What I do have a problem is rerunning protoc automatically on proto-files which
> import proto files that have been modified.
> How do you handle this ?

I just found out I don't.
Any idea on that one ?

Otherwise I'm going to create a small .proto parser (python) that will
just list the imports recursively.
I will add that as DEPENDS to the add_custom_command.

David
On Thu, Aug 23, 2018 at 9:36 PM Alexander Neundorf <[hidden email]> wrote:

>
> On 2018 M08 23, Thu 12:50:14 CEST David Jobet wrote:
> > Hello,
> >
> > I'm trying to port an existing project from premake to cmake.
> > I'm trying to avoid modifying the source files while doing so.
> >
> > Right now, we have several libraries (read in different directories) using
> > proto files with imports between them.
> > All imports are made relative to the root of the project.
> >
> > e.g :
> > work/lib1/sublib1/a.proto
> > work/lib1/sublib1/b.proto
> >
> > with a.proto having a link to b.proto like this "import
> > lib1/sublib1/b.proto"
> >
> > If I compile this with an unchanged FindProtobuf.cmake, I do this :
> >
> > protobuf_generate_cpp(PROTO_SRCS PROTO_HRDS a.proto b.proto)
> > add_library(...)
> >
> > the problem being that it looks like the "namespaces" generated by protoc
> > in b.proto do not match the one used in a.proto : it does not compile.
>
> do you mean protoc fails, or that the C++ compiler fails when compiling the
> generated file ?
> I'm also using proto files with imports between them and don't have problems
> with finding them.
> Are you setting the PROTOBUF_IMPORT_DIRS variable ?
>
> What I do have a problem is rerunning protoc automatically on proto-files which
> import proto files that have been modified.
> How do you handle this ?
>
> > Is there a way to make this work ?
> >
> > Otherwise, I made the patch below which solves my problem. Do you think it
> > could be included in cmake ?
> > it defines a new option PROTO_PATH, so the decl above becomes
> > protobuf_generate_cpp(PROTO_SRCS PROTO_HRDS PROTO_PATH ${CMAKE_SOURCE_DIR}
> > a.proto b.proto)
> >
> > With regards
> >
> > David
> >
> > $ diff share/cmake-3.10/Modules/FindProtobuf.cmake.orig
> > share/cmake-3.10/Modules/FindProtobuf.cmake
>
> can you please use diff -bup ?
> This makes the patch easier to read.
>
> Thanks
> Alex
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: protobuf and imports relative to root (and --proto_path)

Alexander Neundorf
On 2018 M09 25, Tue 16:53:27 CEST David Jobet wrote:
> > What I do have a problem is rerunning protoc automatically on proto-files
> > which import proto files that have been modified.
> > How do you handle this ?
>
> I just found out I don't.
> Any idea on that one ?

here are some suggestions:
https://cmake.org/pipermail/cmake-developers/2018-May/030664.html

Alex

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: protobuf and imports relative to root (and --proto_path)

David Jobet-2
Tx Alexander.

Unfortunately, the version of cmake that we use (2.6.0) does not have a --dependency_out arg.
So I just coded a small python parser that recursively check for imports and create a ninja depfile out of it. (injected with DEPFILE in custom command)
It looks like it works fine thanks !

David

Le 25 septembre 2018 21:15:54 GMT+01:00, Alexander Neundorf <[hidden email]> a écrit :
On 2018 M09 25, Tue 16:53:27 CEST David Jobet wrote:
What I do have a problem is rerunning protoc automatically on proto-files
which import proto files that have been modified.
How do you handle this ?

I just found out I don't.
Any idea on that one ?

here are some suggestions:
https://cmake.org/pipermail/cmake-developers/2018-May/030664.html

Alex

--
Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser ma brièveté.
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake