> I suggest to compare also against this [1].
> The author, Kenton Varda, was the primary author of Protocol Buffers version 2, which is the version that Google released open source.
>
> [1] https://capnproto.org

Will do - at some point. Writing proper benchmarks against other frameworks / encodings takes time though. That's why we have started with MessagePack, CBOR and Google Protocol Buffers.

> I suggest to compare also against this [1].
> The author, Kenton Varda, was the primary author of Protocol Buffers version 2, which is the version that Google released open source.
>
> [1] https://capnproto.org


I just had a look at Cap'n Proto. From what I can see in the encoding spec, performance of ION will be comparable.

Cap'n Proto claims to be "infinitely faster" than Google Protocol Buffers, but that is only if you do not pack the CP data - in which case it will transfer slower over the network. CP solves that using packing - but then you are back to serialization / deserialization, and the original promise of being "inifinitely faster" is gone.

Cap'n Proto also has the "problem" that its messages require an external schema. To iterate through a Cap'n Proto file / message you must already know what data is in it (the schema).

Some see this as an advantage, because it forces you to write a schema for your data structure, and you get slightly faster encoding / decoding time.

And others see this is a disadvantage because you now have to import schemas, or generate code, in order to read a serialized message. You cannot just step through it like you can with e.g. XML or JSON. I tend to be in this camp - although I am not blind to the arguments in favor of external schemas. Speed matters, but so does ease-of-use.

On a network protocol level I tend to disagree with the "distributed object" model. I know Capn Proto tries to explain why this model is not a problem with CP. However, fine grained communication between fine grained distributed objects *is* a performance killer in the long run, regardless of whether you "pipeline" requests.

ION is intended to be the message format for our IAP network protocol. IAP will be message oriented, so you can do one-way messaging, request-response, subscriptions (e.g. to a stream), pipelining, routing of messages via intermediate nodes etc.

Anyways, if you really want to use Cap'N Proto (or something else) over IAP (+ION) you can just nest a binary message inside an IAP message, and then parse it any way you like when it comes out.

On Sunday, 20 December 2015 at 17:52:40 UTC, Jakob Jenkov wrote:
> I just had a look at Cap'n Proto. From what I can see in the encoding spec, performance of ION will be comparable.

"If a disease has many treatments, it has no cure".

This is certainly true for serialization protocols.

The major advantage I see in Cap'n'Proto is the pipelining can do quite a lot to reduce round trip latency. (You don't have to google far to find rants pointing out that latency is often more important than bandwidth in determining throughput.)

I was just reading your IAP web site, when I came across "No Stateful Communication" under the heading "What is Wrong With HTTP?".

The designers of HTTP would strongly argue that is a major thing HTTP got right, and is the feature primarily responsible for it huge success.

Certainly in the realm of IoT HTTP is way too heavy.... so in that domain I would reach for
http://coap.technology/

The use case I keep challenging my colleagues with is....

So one end or the other dies. Or resets. Or fades and comes back. Or changes batteries.

This is the IoT things. It will happen, and you will be required to recover the whole end to end system automatically without manual intervention.

What is your plan?

Too often the answer is... "We don't have a plan but we will have a wheel restarting the link.... umm, then a wheel resending the stuff that was lost in the link buffers when the link went down.... and a, errrr, maybe wheel restarting Everything when we realise the other side has lost it's state about our connection.

And in practice the only wheel that works is shutting everything down and restarting everything up.

Suddenly "No stateful communication" is looking really really Good.

Coap clearly has thought these issues through.

On Sunday, 20 December 2015 at 01:16:46 UTC, Jakob Jenkov wrote:
> According to Thrift's own docs their binary encoding is not compact. For compact encoding it seems they refer to Protobuf.

There seems to be a confusion of terminology here. Thrift has a "Binary" protocol, which is not compact in the sense that it consists of the data fields more or less blitted into a message. There is also a "Compact" protocol, which is also a binary format, but employs things like variable-length integers to reduce size –  similar to Protobuf.

 — David

On Sunday, 20 December 2015 at 19:16:19 UTC, David Nadlinger wrote:
> On Sunday, 20 December 2015 at 01:16:46 UTC, Jakob Jenkov wrote:
>> According to Thrift's own docs their binary encoding is not compact. For compact encoding it seems they refer to Protobuf.
>
> There seems to be a confusion of terminology here. Thrift has a "Binary" protocol, which is not compact in the sense that it consists of the data fields more or less blitted into a message. There is also a "Compact" protocol, which is also a binary format, but employs things like variable-length integers to reduce size –  similar to Protobuf.
>
>  — David

Thanks for the clarification! I couldn't really make out from the Thrift website if they had their own compact protocol, or switched to Protobuf. But now you say that they do have their own compact protocol. Now I know that.

> The designers of HTTP would strongly argue that is a major thing HTTP got right, and is the feature primarily responsible for it huge success.

Then why is HTTP 2 moving away from it? And Web Sockets?
Clearly, having the choice between keeping state and not keeping
state is preferable to HTTP taking that choice away from you.

Lots of apps also spend quite an effort to mimic stateful communication
on top of HTTP. Sessions? Authentication tokens? Cookies? Caching
in the browser? HTML5 Local Storage?

No, HTTP did not get "stateless" right.


Your "fix-the-network" problem is definitely valid.

At this point we have mostly focused on ION - the binary object / message format for IAP.
However, we have a pretty good idea about how IAP will work on a conceptual
level.

IAP will have a set of "semantic protocols". Each semantic protocol can address
its own area of concern. File exchange, time, RPC, distributed transactions,
P2P, streaming etc.

You can also define your own semantic protocol to address exactly your specific
situation (e.g. the Byzantine Generals Problem - distributed consensus).

Everything is not yet in place - but we will get there step by step.

On Sunday, 20 December 2015 at 21:37:35 UTC, Jakob Jenkov wrote:
>> The designers of HTTP would strongly argue that is a major thing HTTP got right, and is the feature primarily responsible for it huge success.
>
> Then why is HTTP 2 moving away from it? And Web Sockets?
> Clearly, having the choice between keeping state and not keeping
> state is preferable to HTTP taking that choice away from you.
>
> Lots of apps also spend quite an effort to mimic stateful communication
> on top of HTTP. Sessions? Authentication tokens? Cookies? Caching
> in the browser? HTML5 Local Storage?
>
> No, HTTP did not get "stateless" right.

Yep, the whole stateless argument is a complete joke, it has not been true except maybe in the very beginning.  HTTP 2 is a huge step forward for this, its binary encoding, and other reasons.

> Your "fix-the-network" problem is definitely valid.
>
> At this point we have mostly focused on ION - the binary object / message format for IAP.
> However, we have a pretty good idea about how IAP will work on a conceptual
> level.
>
> IAP will have a set of "semantic protocols". Each semantic protocol can address
> its own area of concern. File exchange, time, RPC, distributed transactions,
> P2P, streaming etc.
>
> You can also define your own semantic protocol to address exactly your specific
> situation (e.g. the Byzantine Generals Problem - distributed consensus).
>
> Everything is not yet in place - but we will get there step by step.

Interesting effort, I'll check it out.

On Wednesday, 16 December 2015 at 09:47:35 UTC, Jakob Jenkov wrote:
>
> Since we are rather new to D, would anyone be interested in helping us a bit out making such a library? We can probably do the coding ourselves, but might need some tips about how to pack it nicely into a D library which can be used with Dub etc.

Be sure to look at how MsgPack is implemented in D:
https://github.com/msgpack/msgpack-d

It has a very easy interface, and is one of the better D library out there.