"Cap’n Proto is an insanely fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except faster."

This is the initial public release of my optimized port of the Java implementation of Cap'n Proto.

State:
* Passes Cap'n Proto testsuite.
* Optimized. Just a little slower than the official C++ implementation (see benchmarks on github).
* Missing RPC part of Cap'n Proto.

http://code.dlang.org/packages/capnproto-dlang
https://github.com/ThomasBrixLarsen/capnproto-dlang

On Tuesday, 18 April 2017 at 18:09:54 UTC, Thomas Brix Larsen wrote:
> "Cap’n Proto is an insanely fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except faster."
>
> This is the initial public release of my optimized port of the Java implementation of Cap'n Proto.
>
> State:
> * Passes Cap'n Proto testsuite.
> * Optimized. Just a little slower than the official C++ implementation (see benchmarks on github).
> * Missing RPC part of Cap'n Proto.
>
> http://code.dlang.org/packages/capnproto-dlang
> https://github.com/ThomasBrixLarsen/capnproto-dlang

Java?? Yikes

On Tuesday, 18 April 2017 at 18:09:54 UTC, Thomas Brix Larsen wrote:
> "Cap’n Proto is an insanely fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except faster."
>
> This is the initial public release of my optimized port of the Java implementation of Cap'n Proto.
>
> State:
> * Passes Cap'n Proto testsuite.
> * Optimized. Just a little slower than the official C++ implementation (see benchmarks on github).
> * Missing RPC part of Cap'n Proto.
>
> http://code.dlang.org/packages/capnproto-dlang
> https://github.com/ThomasBrixLarsen/capnproto-dlang

Great Job!

I'm following Cap'n Proto, and it's very very interesting...
I would love also the RPC part, maybe based on Vibe... have you any plan to implement that?

I'm really curious to try it!

---
Paolo

On 4/18/17 9:14 PM, Swoorup Joshi wrote:
> On Tuesday, 18 April 2017 at 18:09:54 UTC, Thomas Brix Larsen wrote:
>> "Cap’n Proto is an insanely fast data interchange format and
>> capability-based RPC system. Think JSON, except binary. Or think
>> Protocol Buffers, except faster."
>>
>> This is the initial public release of my optimized port of the Java
>> implementation of Cap'n Proto.
>>
>> State:
>> * Passes Cap'n Proto testsuite.
>> * Optimized. Just a little slower than the official C++ implementation
>> (see benchmarks on github).
>> * Missing RPC part of Cap'n Proto.
>>
>> http://code.dlang.org/packages/capnproto-dlang
>> https://github.com/ThomasBrixLarsen/capnproto-dlang
>
> Java?? Yikes

Risking a flamewar but what's wrong with Java?

---
Dmitry Olshansky

On Tuesday, 18 April 2017 at 20:46:33 UTC, Paolo Invernizzi wrote:
> On Tuesday, 18 April 2017 at 18:09:54 UTC, Thomas Brix Larsen wrote:
>> "Cap’n Proto is an insanely fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except faster."
>>
>> This is the initial public release of my optimized port of the Java implementation of Cap'n Proto.
>>
>> State:
>> * Passes Cap'n Proto testsuite.
>> * Optimized. Just a little slower than the official C++ implementation (see benchmarks on github).
>> * Missing RPC part of Cap'n Proto.
>>
>> http://code.dlang.org/packages/capnproto-dlang
>> https://github.com/ThomasBrixLarsen/capnproto-dlang
>
> Great Job!
>
> I'm following Cap'n Proto, and it's very very interesting...
> I would love also the RPC part, maybe based on Vibe... have you any plan to implement that?
>
> I'm really curious to try it!
>
> ---
> Paolo

Well personally I have no need for the RPC part. Though if it gets added to the Java implementation of Cap'n Proto I will probably port it.

The D implementation works well. I developed it for a hobby project. Game server in D. Client in Unity. At work we have been using the C++ Cap'n Proto for IPC for a while. Recently we have also started using the D implementation for configuration files.

On 2017-04-18 23:08, Dmitry Olshansky wrote:

> Risking a flamewar but what's wrong with Java?

I don't like any language that force me to write in a particular style or paradigm. Because all problems cannot be solved (or not in a good way) in the same way. That said, my D code is quite heavily influenced by Java.

-- 
/Jacob Carlborg

On Tuesday, 18 April 2017 at 18:09:54 UTC, Thomas Brix Larsen wrote:
> "Cap’n Proto is an insanely fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except faster."

The features below, from the capnproto.org description, interest me.  However a MappedByteBuffer would be used for the mmap feature in java.

https://capnproto.org/

mmap: Read a large Cap’n Proto file by memory-mapping it. The OS won’t even read in the parts that you don’t access.

Inter-language communication: Calling C++ code from, say, Java or Python tends to be painful or slow. With Cap’n Proto, the two languages can easily operate on the same in-memory data structure.

Inter-process communication: Multiple processes running on the same machine can share a Cap’n Proto message via shared memory. No need to pipe data through the kernel. Calling another process can be just as fast and easy as calling another thread.

This info from stackoverflow also seems to imply that MappedByteBuffer would be required for some of the capnproto features.  So, could you explain a little more about what are the capabilities of the current d library implementation, with just the ByteBuffer implemented from the java nio code?  Thanks, Jay

http://stackoverflow.com/questions/29361058/read-proto-partly-instead-of-full-parsing-in-java

'If you are willing to consider using a different protocol framework, Cap'n Proto is extremely similar in design to Protobufs but features in the ability to read only the part of the message you care about. Cap'n Proto incurs no overhead for the fields you don't examine, other than obviously the bandwidth and memory to receive the raw message bytes. If you are reading from a file, and you use memory mapping (MappedByteBuffer in Java), then only the parts of the message you actually use will be read from disk.'

On Wednesday, 19 April 2017 at 16:34:02 UTC, Jay Norwood wrote:
> [...]
>
> This info from stackoverflow also seems to imply that MappedByteBuffer would be required for some of the capnproto features.  So, could you explain a little more about what are the capabilities of the current d library implementation, with just the ByteBuffer implemented from the java nio code?  Thanks, Jay
> 
> [...]

The port of ByteBuffer just wraps a slice when reading in D.

Take a look at FileDescriptor[1]. It is a class I've added to support read/write using File from std.stdio. You can create a similar streamer using std.mmfile. I believe that this would be enough for memory mapped reading.

[1]: https://github.com/ThomasBrixLarsen/capnproto-dlang/blob/master/source/capnproto/FileDescriptor.d

On Wednesday, 19 April 2017 at 16:52:14 UTC, Thomas Brix Larsen wrote:
> Take a look at FileDescriptor[1]. It is a class I've added to support read/write using File from std.stdio. You can create a similar streamer using std.mmfile. I believe that this would be enough for memory mapped reading.
>
> [1]: https://github.com/ThomasBrixLarsen/capnproto-dlang/blob/master/source/capnproto/FileDescriptor.d

Ok, thanks. I took a look at several capnproto implementations just now, and didn't see any tests for a mmap 'feature'.  The roadmap doc below indicates it doesn't exist, and perhaps there are some details yet to be resolved to make it 'friendly' for a mmap.

https://capnproto.org/roadmap.html

mmap-friendly mutable storage format: Define a standard storage format that is friendly to mmap-based use while allowing modification. (With the current serialization format, mmap is only useful for read-only structures.) Possibly based on the ORM interface, updates only possible at the granularity of a whole ORM entry.


In java the MappedByteBuffer can be used with a RandomAccessFile channel, and then the accesses of the loaded map can be random, without requiring sequential accesses of the whole mapped file. So java nio already has some higher level classes in place that would make it easier to develop a first implementation of the mmap features.

On Wednesday, 19 April 2017 at 18:24:46 UTC, Jay Norwood wrote:
> [...]
>
> Ok, thanks. I took a look at several capnproto implementations just now, and didn't see any tests for a mmap 'feature'.  The roadmap doc below indicates it doesn't exist, and perhaps there are some details yet to be resolved to make it 'friendly' for a mmap.

But reading using random access *is* a feature of Cap'n Proto. So when reading a memory mapped Cap'n Proto file, getters will be faster if you use it in a non-sequential way.

The format of Cap'n Proto doesn't currently support *writing* to a memory mapped file.