AFAIK, it's opposite: an array serialized in chunks, and they are concatenated on deserialization. Useful if you don't know how much elements you're sending, so you send them in finite chunks as the data becomes available. Client can also close connection, so you don't have to see the end of the sequence.

Oh, wait, it looks like a (possibly infinite) range rather than an array.

On 04/01/2013 02:37 PM, Kagamin wrote:
> AFAIK, it's opposite: an array serialized in chunks, and they are
> concatenated on deserialization. Useful if you don't know how much
> elements you're sending, so you send them in finite chunks as the data
> becomes available. Client can also close connection, so you don't have
> to see the end of the sequence.

https://developers.google.com/protocol-buffers/docs/encoding#optional

The "packed repeated fields" section explains it and breaks it down with an example. If the client can close like that, you probably don't want to use packed.
-Soucy

On Monday, 1 April 2013 at 18:41:57 UTC, Matt Soucy wrote:
> The "packed repeated fields" section explains it and breaks it down with an example. If the client can close like that, you probably don't want to use packed.

Why not? If you transfer a result of google search, the client will be able to peek only N first results and close or not. Though I agree it's strange that you can only transfer primitive types this way.

So as we see PB doesn't support arrays (and ranges). Hmm... that's unfortunate.

On 04/01/2013 03:11 PM, Kagamin wrote:
> On Monday, 1 April 2013 at 18:41:57 UTC, Matt Soucy wrote:
>> The "packed repeated fields" section explains it and breaks it down
>> with an example. If the client can close like that, you probably don't
>> want to use packed.
>
> Why not? If you transfer a result of google search, the client will be
> able to peek only N first results and close or not. Though I agree it's
> strange that you can only transfer primitive types this way.
It's not really strange, because of how it actually does the serialization. A message is recorded as length+serialized members. Members can happen in any order. Packed repeated messages would look like...what? How do you know when one message ends and another begins? If you try and denote it, you'd just end up with what you already have.
In your example, you'd want to send each individual result as a distinct message, so they could be read one at a time. You wouldn't want to pack, as packing is for sending a whole data set at once.

On Monday, 1 April 2013 at 17:24:05 UTC, Matt Soucy wrote:
> From what I got from the examples, Repeated fields are done roughly as following:
> auto msg = fields.map!(a=>a.serialize())().reduce!(a,b=>a~b)();
> return ((id<<3)|2) ~ msg.length.toVarint() ~ msg;
> Where msg is a ubyte[].
> -Matt Soucy

I think that would fall under some form of compression, namely compressing the ID :P

BTW, I love how easy that is to read.

On Monday, 1 April 2013 at 19:37:12 UTC, Matt Soucy wrote:
> It's not really strange, because of how it actually does the serialization. A message is recorded as length+serialized members. Members can happen in any order. Packed repeated messages would look like...what? How do you know when one message ends and another begins? If you try and denote it, you'd just end up with what you already have.

Well, messages can be just repeated, not packed. Packing is for really small elements, I guess, - namely numbers.

> In your example, you'd want to send each individual result as a distinct message, so they could be read one at a time. You wouldn't want to pack, as packing is for sending a whole data set at once.

So you suggest to send 1 message per TCP packet?

On 04/01/2013 04:38 PM, Jesse Phillips wrote:
> On Monday, 1 April 2013 at 17:24:05 UTC, Matt Soucy wrote:
>> From what I got from the examples, Repeated fields are done roughly as
>> following:
>> auto msg = fields.map!(a=>a.serialize())().reduce!(a,b=>a~b)();
>> return ((id<<3)|2) ~ msg.length.toVarint() ~ msg;
>> Where msg is a ubyte[].
>> -Matt Soucy
>
> I think that would fall under some form of compression, namely
> compressing the ID :P
>
> BTW, I love how easy that is to read.

So, I looked at the code I'm currently working on to handle these...and it's literally that, except "raw" instead of "fields". UFCS is kind of wonderful in places like this.

You're right, that does count as compression. It wouldn't be hard to number the fields during serialization, especially if you don't need to worry about the structure changing.

-Matt Soucy

On 04/01/2013 04:54 PM, Kagamin wrote:
> On Monday, 1 April 2013 at 19:37:12 UTC, Matt Soucy wrote:
>> It's not really strange, because of how it actually does the
>> serialization. A message is recorded as length+serialized members.
>> Members can happen in any order. Packed repeated messages would look
>> like...what? How do you know when one message ends and another begins?
>> If you try and denote it, you'd just end up with what you already have.
>
> Well, messages can be just repeated, not packed. Packing is for really
> small elements, I guess, - namely numbers.
>

And therefore, it supports arrays just fine (as repeated fields). Yes. That last sentence was poorly-worded, and should have said "you'd just end up with the un'packed' data with an extra header."

>> In your example, you'd want to send each individual result as a
>> distinct message, so they could be read one at a time. You wouldn't
>> want to pack, as packing is for sending a whole data set at once.
>
> So you suggest to send 1 message per TCP packet?

Unfortunately, I'm not particularly knowledgeable about networking, but that's not quite what I meant. I meant that the use case itself would result in sending individual Result messages one at a time, since packing (even if it were valid) wouldn't be useful and would require getting all of the Results at once. You would just leave off the "packed" attribute.