Kris wrote:
> I've been building a multi-layer IO package for DSC (D Servlet Container)
> that's fully buffered and generates zero garbage.
> 
> The layers include:
> 
> - raw byte-array style I/O
> 
> - informal text-oriented Tokenizer I/O with text/binary conversion
> 
> - structured binary I/O (directly to/from variables) with optional endian
> flipping
> 
> - structured text-oriented input with data conversion
> 
> - structured class serialization/deserialization framework
> 
> It's quite flexible: for example, one can 'read' a line into a Token (using
> a LineTokenizer) and then map said Token into any of the other layers for
> further slicing. One can mix and match the different layers in pretty much
> any way that makes sense. There is no copying of data unless requested by
> the app, and no memory allocation other than app-allocated I/O buffers. It
> does random-access where the underlying system supports it (not on a
> socket), and is intended to support memory-mapped buffers (for *huge*
> files). It uses lookahead instead of unget... for those cases that need such
> things.
> 
> I'm writing this stuff for a high-performance server (in D), so one of the
> primary goals is very low runtime overhead (i.e. zero memory allocation). If
> anyone's interested, I'll try to get the first major I/O cut out the door
> by, say, the end of the month.
> 
> - Kris
...
>>
> 
> 
> 

That sounds extremely interesting.  Would it be portable, or platform specific?

> That sounds extremely interesting.  Would it be portable, or platform specific?

Portable.

Sounds nice! I'm definitely interested. Some comments/questions below

"Kris" <someidiot@earthlink.net> wrote in message news:c2nl32$1mm2$1@digitaldaemon.com...
> I've been building a multi-layer IO package for DSC (D Servlet Container) that's fully buffered and generates zero garbage.
>
> The layers include:
>
> - raw byte-array style I/O
>
> - informal text-oriented Tokenizer I/O with text/binary conversion

I'm not sure what informal means here or what text/binary conversion means. Does it include scanf/printf? Any replacement for std.stream needs to have that.

> - structured binary I/O (directly to/from variables) with optional endian
> flipping
>
> - structured text-oriented input with data conversion

not sure what structured means but I'm guessing the data being read has a format like "rows of numbers separated by tabs with the first column being a string" or something. Is this what you mean?

> - structured class serialization/deserialization framework

I don't know what you have in mind here but it sounds like a big chunk of work. Could it go into another module?

> It's quite flexible: for example, one can 'read' a line into a Token
(using
> a LineTokenizer) and then map said Token into any of the other layers for further slicing. One can mix and match the different layers in pretty much any way that makes sense. There is no copying of data unless requested by the app, and no memory allocation other than app-allocated I/O buffers. It does random-access where the underlying system supports it (not on a socket), and is intended to support memory-mapped buffers (for *huge* files). It uses lookahead instead of unget... for those cases that need
such
> things.
>
> I'm writing this stuff for a high-performance server (in D), so one of the
> primary goals is very low runtime overhead (i.e. zero memory allocation).
If
> anyone's interested, I'll try to get the first major I/O cut out the door by, say, the end of the month.

Below:

> I'm not sure what informal means here or what text/binary conversion means. Does it include scanf/printf? Any replacement for std.stream needs to have that.
>
Forgive the odd terminology Ben.  In this context, informal means "loosely formatted", like a command-line or word-count input, or line-input. Formal, or structured, stipulates the input data must conform exactly. Formal inputs throw an exception when the input fails to match an expectation, whereas informal ones just return eof (or an empty Token). Text/binary conversion is converting numbers to text and vice versa (what printf and scanf do). Here's a trivial example:

{
TextWriter w;
int j;

// convert to text
w << "number is : " << j << w.Newline;
}

Printf is supported via a thin adaptor:

w << Printf.format("%d %d", 10, 20) << ....


> not sure what structured means but I'm guessing the data being read has a format like "rows of numbers separated by tabs with the first column being a string" or something. Is this what you mean?
>
Pretty much, yes. In other words, structured/formal format is rigid. There's an agreed-upon layout that is reflected in the source code. That layout might be implemented as raw binary, or as delimited text (as you suggest). If it's raw binary, then there should also be an agreed Endian layout.


> I don't know what you have in mind here but it sounds like a big chunk of work. Could it go into another module?
>
If it were a Java-style reflection approach I'd quickly agree :-)  In this case it's just a couple of trivial interfaces. Each serializable class must implement a read & write method. If one were deserializing a (previously serialized) class, one would probably follow the structured/formal approach. Here's an example:

class MyClass : IReadable, IWritable
{
    private int foo;
    private float bar;

    // the IReadable interface
    void read (IReader r)
    {
           r >> foo >> bar;
    }

    // the IWritable interface
    void write (IWriter w)
    {
           w << foo << bar;
    }
}


{
    MyClass mc  = new MyClass(...);
    ....
    BinaryWriter w = new BinaryWriter(...);

    w << mc;
    ...
}

Note that there are a variety of Readers and Writers (binary, text, token, etc) and all of them can be used directly against any class implementing IReadable and/or IWritable. In other words, the implementing class doesn't know, or care, whether it's serialized in binary, text, or something else. As an aside, there's also an ISerializable interface that adds methods getGuid() and create(). This would be used when shipping class instances around a network, sans reflection (something that DSC will support for clustering).

"Kris" <someidiot@earthlink.net> wrote in message
news:c2nl32$1mm2$1@digitaldaemon.com
| I've been building a multi-layer IO package for DSC (D
| Servlet Container) that's fully buffered and generates
| zero garbage.
| ...

Do you know when you could make a release for DSC?

-----------------------
Carlos Santander Bernal

Carlos,

I would hope to release an Alpha sometime in April. Initial release will not support https, because I don't have that expertise. Any volunteers? <g>

- Kris


"Carlos Santander B." <carlos8294@msn.com> wrote in message news:c2ok8o$efd$1@digitaldaemon.com...
> "Kris" <someidiot@earthlink.net> wrote in message
> news:c2nl32$1mm2$1@digitaldaemon.com
> | I've been building a multi-layer IO package for DSC (D
> | Servlet Container) that's fully buffered and generates
> | zero garbage.
> | ...
>
> Do you know when you could make a release for DSC?
>
> -----------------------
> Carlos Santander Bernal
>
>

"Kris" <someidiot@earthlink.net> wrote in message
news:c2olf2$gn9$1@digitaldaemon.com
| Carlos,
|
| I would hope to release an Alpha sometime in April.
| Initial release will not support https, because I don't
| have that expertise. Any volunteers? <g>
|
| - Kris
|

Not here, sorry. I was just asking because right now I'm in the middle of building an e-business site but using JSP (including https, certificates, signed mail...), so I got curious to know if/how that could be done using DSC. Not that I would use it: deadline is this monday and I still have to figure out how to attach a file to a signed mail, and make Apache find the jar for creating the graphic that will be attached. So, I still have a lot of research to do.

-----------------------
Carlos Santander Bernal