I've run into a design issue surrounding ranges and am looking for advice on the best way to proceed.  To illustrate the issue, consider the Shapefile format: a 100 byte header followed by variable-length records. The tricky bit is that the header includes a field which contains the total length of the file (as measured in 16-bit words, curiously).  The header must be written first, but the total length of the file isn't known until all the records have been encoded.  When writing to a File this isn't a problem: write 100 bytes of padding, write the records, use rewind(), and write the proper header.  It's in the context of an OutputRange that I don't know how to proceed.  Consider the most flexible range type: the array.  An array is not an OutputRange, so it needs to be wrapped in something like std.array.Appender.  Ideally I could save off the initial state of the range, write a bogus header, write the records, then jump back and write the proper header.  Unfortunately, Appender is not a ForwardRange, nor does it appear that the field of OutputRanges which are also ForwardRanges has been explored.  I'm using the excellent read, write, and append functions from std.bitmanip, so write() would fit the bill if only Appender supported slicing.

My current solution is require the user to construct the ShapeWriter output range (which supports `put(Shape)`) over two separate output ranges of ubyte: one for the header and a second for the records, then delay writing to the header range until the record range is complete. This is both needlessly complex and it leaves the proper combination of the two to the user, making ShapeWriter a very leaky abstraction.

So, for particular questions:
1) Am I missing something in Phobos that would provide an OutputRange of
ubytes while also providing ForwardRange/slicing capabilities?
2) I know the relationship between streams and ranges has been discussed
at least once before; is the concept of rewinding and overwriting simply
incompatible with OutputRange in general?

Justin

On Friday, 3 January 2014 at 22:56:41 UTC, Justin Whear wrote:
> I've run into a design issue surrounding ranges and am looking for advice
> on the best way to proceed.  To illustrate the issue, consider the
> Shapefile format: a 100 byte header followed by variable-length records.
> The tricky bit is that the header includes a field which contains the
> total length of the file (as measured in 16-bit words, curiously).  The
> header must be written first, but the total length of the file isn't
> known until all the records have been encoded.  When writing to a File
> this isn't a problem: write 100 bytes of padding, write the records, use
> rewind(), and write the proper header.  It's in the context of an
> OutputRange that I don't know how to proceed.  Consider the most flexible
> range type: the array.  An array is not an OutputRange, so it needs to be
> wrapped in something like std.array.Appender.

Actaully, dynamic arrays *are* output ranges. Each "put" places the element at the front of the range, and the range is then pop'ed front.

Its' not an "expanding" output range, rather, a "fillable" output range, so not really something that would fit your need.

> Ideally I could save off
> the initial state of the range, write a bogus header, write the records,
> then jump back and write the proper header.  Unfortunately, Appender is
> not a ForwardRange, nor does it appear that the field of OutputRanges
> which are also ForwardRanges has been explored.  I'm using the excellent
> read, write, and append functions from std.bitmanip, so write() would fit
> the bill if only Appender supported slicing.
>
> My current solution is require the user to construct the ShapeWriter
> output range (which supports `put(Shape)`) over two separate output
> ranges of ubyte: one for the header and a second for the records, then
> delay writing to the header range until the record range is complete.
> This is both needlessly complex and it leaves the proper combination of
> the two to the user, making ShapeWriter a very leaky abstraction.
>
> So, for particular questions:
> 1) Am I missing something in Phobos that would provide an OutputRange of
> ubytes while also providing ForwardRange/slicing capabilities?

<pedantic>ubyte[] is such a range</pedantic>. It won't do what you want though.

> 2) I know the relationship between streams and ranges has been discussed
> at least once before; is the concept of rewinding and overwriting simply
> incompatible with OutputRange in general?
>
> Justin

I think the biggest issue is that there is actaully 0 relation between input ranges and output ranges. The two concepts are completely orthogonal.

In particular, arguably, input ranges that are also output ranges are confusing,  since rather than "growing as you add items to their tip (what you'd want)", they instead "shrink as you overwrite their front, until they are empty/full".

When using put(range, item), the "put" primitive of "range" takes precedence over "front". You could design a range that uses that, which would look a bit more like what you want, but there is nothing that exists in phobos that does this that I know of anyways.

It's a bit of a mess (IMO).

On Friday, 3 January 2014 at 23:14:31 UTC, monarch_dodra wrote:
> In particular, arguably, input ranges that are also output ranges are confusing,  since rather than "growing as you add items to their tip (what you'd want)", they instead "shrink as you overwrite their front, until they are empty/full".

To add to that, it's *kind* of like an input/output stream, that you can only write to, after you have read it...

On 2014-01-03 23:56, Justin Whear wrote:
> I've run into a design issue surrounding ranges and am looking for advice
> on the best way to proceed.  To illustrate the issue, consider the
> Shapefile format: a 100 byte header followed by variable-length records.
> The tricky bit is that the header includes a field which contains the
> total length of the file (as measured in 16-bit words, curiously).  The
> header must be written first, but the total length of the file isn't
> known until all the records have been encoded.  When writing to a File
> this isn't a problem: write 100 bytes of padding, write the records, use
> rewind(), and write the proper header.  It's in the context of an
> OutputRange that I don't know how to proceed.  Consider the most flexible
> range type: the array.  An array is not an OutputRange, so it needs to be
> wrapped in something like std.array.Appender.  Ideally I could save off
> the initial state of the range, write a bogus header, write the records,
> then jump back and write the proper header.  Unfortunately, Appender is
> not a ForwardRange, nor does it appear that the field of OutputRanges
> which are also ForwardRanges has been explored.  I'm using the excellent
> read, write, and append functions from std.bitmanip, so write() would fit
> the bill if only Appender supported slicing.

If you're satisfied with just using arrays and not a general output range you can do something similar to what you did with a file. Add the 100 extra bytes for the header to Appender, then add the contents. Extract the array used for backing in the Appender and fill the proper header.

-- 
/Jacob Carlborg

On Saturday, 4 January 2014 at 11:22:03 UTC, Jacob Carlborg wrote:
> On 2014-01-03 23:56, Justin Whear wrote:
>> I've run into a design issue surrounding ranges and am looking for advice
>> on the best way to proceed.  To illustrate the issue, consider the
>> Shapefile format: a 100 byte header followed by variable-length records.
>> The tricky bit is that the header includes a field which contains the
>> total length of the file (as measured in 16-bit words, curiously).  The
>> header must be written first, but the total length of the file isn't
>> known until all the records have been encoded.  When writing to a File
>> this isn't a problem: write 100 bytes of padding, write the records, use
>> rewind(), and write the proper header.  It's in the context of an
>> OutputRange that I don't know how to proceed.  Consider the most flexible
>> range type: the array.  An array is not an OutputRange, so it needs to be
>> wrapped in something like std.array.Appender.  Ideally I could save off
>> the initial state of the range, write a bogus header, write the records,
>> then jump back and write the proper header.  Unfortunately, Appender is
>> not a ForwardRange, nor does it appear that the field of OutputRanges
>> which are also ForwardRanges has been explored.  I'm using the excellent
>> read, write, and append functions from std.bitmanip, so write() would fit
>> the bill if only Appender supported slicing.
>
> If you're satisfied with just using arrays and not a general output range you can do something similar to what you did with a file. Add the 100 extra bytes for the header to Appender, then add the contents. Extract the array used for backing in the Appender and fill the proper header.

I guess my larger concern is providing an interface which avoids allocation.  In that case, having an overload which accepts an array for writing into and signals if it was too small could work.  It is a bit disappointing to lose the nice generic interface which would really give the user control.