Protobuf - sharing

Protobuf - java package name and sharing proto file across projects

4

I have two java projects that produce and consume messages from Kafka, one project produces a message of type Ticker and another project consumes this message.

So I've created this file

syntax = "proto3";

message Ticker {
    string symbol = 1;
    float ask = 2;
    float bid = 3;
}

I know that in order to create Java objects for this message type, I use protoc

However, I have two problems.

  1. I need a way of sharing the proto file across the project (the 2 projects that I have right now and any future project that would need access to Ticker). What is the best way to do it? I've heard something about git subtrees or submodules, is it a good/reliable way?

  2. I need a way for protoc to generate the Java objects under some package, for a example, I have two projects, one is com.myorg.TickerProducer and another one is com.myorg.TickerConsumer I need protoc to generate the Java objects in the packages/namespaces com.myorg.TickerProducer.Entities and com.myorg.TickerConsumer.Entities respectively. I know that I can use option java_package inside the proto file, but I have different projects that use the same files and each want the Java objects to be generated under their namespaces. Is there a way to do it?

Thank you.

Improve this question
 

2 Answers

3

The sanest way to deal with this might be to have a separate package for these entities, which both the consumer and producer use. As long as you don't want the consumer to depend on the producer or vice-versa.

If you're using Gradle, there's a protobuf-plugin which integrates the building of the protobufs into the consumer project's code.

In general, there is a slight impedance mismatch between how protobuf's are meant to be used, due to the fact that Google has a single repository where every build unit / package can technically depend on every other build unit / package, and how they are used in the rest of the world, where projects have many repositories, and those act as a hard boundary for visibility and dependence between build units and packages.

If you can add them up to a package containing other common definitions etc. that would be best, but lacking that, a separate package, ugly as it is, might be your safest bet.

Improve this answer
 
  •  
    Thank you, I'll have a look at the plugin. 
    – areller
     Feb 15, 2018 at 12:47
  •  
    And where would i store this common package? each project has it's own git repository? should i have a separate git repository for the common proto files? how would i access them from my projects? 
    – areller
     Feb 15, 2018 at 13:01
  • 1
    @areller that's up to how you structure your project really. If you have a single repository, putting it in a common/ directory might be enough. If you have multiple, then putting it in it's own repository makes the most sense. I usually split it into packages, each one having it's own repository. It seems to be the way most of the tooling works nowadays at least. It's a bit more complex infrastructure-wise for smaller projects, but it's a boon for larger ones, since you have a very clear boundary between projects. 
    – Horia Coman
     Feb 15, 2018 at 13:16
 
2
 

A single .proto file will always generate code in the same package/namespace, so if you want to have two different layouts in the two projects, you'll need two .proto files. Frankly, though, I'd argue that it is correct to have it in the same package/namespace in both: it still is the same thing, ultimately.

Note that as long as you aren't using the "any" feature, protobuf doesn't care much about the type-name / package / namespace at runtime; so an X.Y.Z will be interchangeable with an A.B.C as long as they are actually the same fundamental shape. Even field-names are irrelevant as long as you aren't using json.

Improve this answer
 
  •  
    So you're suggesting to have the same namespace com.myorg.Entities for all Java projects? what about projects that aren't written in Java? C# for instance, I would have to have both java_package and csharp_namespace in the same proto file, I guess it would work but it makes me feel as though Protobuf was never designed to be shared across different project (let alone projects that are written in different lan gauges). Maybe my whole approach is wrong and I should just maintain a separate proto file for each project? 
    – areller
     Feb 15, 2018 at 12:42
  • 2
    @areller I'm only talking about these specific entities, not everything - but: yes; and possibly in a library project that just contains these shared types - then consume that library from both. As for between languages: that's not a problem at all and works just fine. It is expected that you would have a java_package and a csharp_namespace in the same .proto (in the case where you are using java and c#, obviously). 
    – Marc Gravell
     Feb 15, 2018 at 12:45
  •  
    It makes sense. It bothers me a bit because I'd expect the proto file to be language agnostic since the protoc already generates language specific files so it might as well accept a namespace/package. Thank you, very helpful answer. How about storing the proto files? Do you suggest to have them in a git repository and use submodules? 
    – areller
     Feb 15, 2018 at 12:58
  •  
    @areller I don't entirely disagree - in fact, for the "old" version of protobuf-net's code-gen tooling, it used to allow the namespace to be specified on the command-line. I haven't retained that for the "new" version of protobuf-net's code-gen tooling, largely to provide consistency with protoc (protobuf-net is just a separate implementation - not based on Google's codebase) 
    – Marc Gravell
     Feb 15, 2018 at 13:11 
     
     
     
    Asked 
    Modified 4 months ago
    Viewed 143 times
     
    1

    I have some input files that contain protobuf data, with a corresponding .proto definition. For the sake of a minimal example (taken from here), assume this .proto file looks like

    message Person {
    required string name = 1;
}

    According to every example I can find on the net, I have to specify my java package and class names in the .proto file itself, amending it with

    option java_package = "com.example.protobuf";
option java_outer_classname = "AddressBookProtos";

    The problem is - if I do that, I'll have to change my .proto file every time the vendor updates their definition, and of course the vendor doesn't want to add those options to their version, because a) they're not using java and b) different customers may have different requirements as to the package/class names.

    So what I'd like to do is to not have those options in the .proto file, and give them on the command line to protoc in a way similar to this:

    protoc --java_out=. --option java_package="com.example.protobuf" --option java_outer_classname="AddressBookProtos" message.proto

    but I can't find any documentation if this is possible at all or which syntax to use.

    So, is it possible to pass options to protoc in a similar way?

    Improve this question
     
    •  
      You might be able use import, i.e. create a file myAddressBook.proto containing option java_package = "com.example.protobuf"; option java_outer_classname = "AddressBookProtos"; import "vendorAddressBook.proto"; (I have not tried this) 
      – tgdavies
       Mar 26 at 6:59 
    •  
      @tgdavies: No, that's not what imports in protobuf do. 
      – Jon Skeet
       Mar 26 at 7:01
    •  
      @JonSkeet So you'd have to be using the AddressBook type in another type you defined in your own file? 
      – tgdavies
       Mar 26 at 7:03
    •  
      @tgdavies: Yes. But it wouldn't change the package or outer classname for anything in the imported proto. 
      – Jon Skeet
       Mar 26 at 7:28
    •  
      The producer of the protos should include option for the languages that their users may need. Google does this for its services. A challenge is that option applies to packages and protoc generally compiles multiple packages requiring a way to specify multiple xxx_out and xxx_opt. That said, it's reasonable to want to be able to override these with protoc and take the hit when you can't update the proto sources (file feedback) or you'll have to duplicate the proto sources and amend as you're doing 
      – DazWilkin
       Mar 26 at 17:53 
    Protobuf - Command Line Usage | Tutorialspoint
     
    Asked 
    Modified 7 months ago
    Viewed 51k times
     
    71

    I would like to include a protocol definition file in another protocol file. For example:

    // base.proto:
message P_EndPoint {
  required int32 id = 1;
  required string host = 2;
  required int32 port = 3;
}

    Then in another file:

    communication.proto:
// somehow include `base.proto'
// ...
message P_CommunicationProtocol {
  required CP_MessageType type = 1;
  optional int32 id = 2;
  optional P_EndPoint identity = 3;
  repeated P_EndPoint others = 4;
}
// ...

    (Note: developers.google.com is not available in my locale)

    Improve this question
     

    2 Answers

    71
     
    import "myproject/base.proto";

    Docs: https://developers.google.com/protocol-buffers/docs/proto#importing

    Improve this answer
     
    • 3
      Thanks. Sorry for not doing my own search. Google Developers is not accessible in my country. (I see this) 
      – sorush-r
       Jun 3, 2013 at 19:04 
    • 3
      Because of the stupid politics. US Export control laws restrict providing certain services to some countries. This really sucks. I'm a student. I'm not trying to make bombs! I just wanna finish my final project on CSP course :| 
      – sorush-r
       Jun 3, 2013 at 19:10 
    • 3
      No worries. I edited your Q so I could remove my downvote. Glad I could help. 
      – Brian Roach
       Jun 3, 2013 at 19:11
    • 11
      Oh dude :) You really don't know what are you talking about. Living here is really difficult. VPN, SOCKS, Certain IPs, Certain protocols, are blocked. Data transmission is sniffed at both ISP and governmental level. No way you can get real internet here :D 
      – sorush-r
       Jun 3, 2013 at 19:16
    • 19
      Oh wow, I didn't look at your profile until just now. Ignore that I said anything :) Ask any damn thing you want on here, and mention where you are. 
      – Brian Roach
       Jun 3, 2013 at 19:19 
     
    1
    syntax = "proto3";

import "Protos/general.proto";

    enter image description here

    Improve this answer
posted @ 2023-07-28 17:45  CharyGao  阅读(24)  评论(0)    收藏  举报