On 12/10/2010 09:57 PM, Matthias Walter wrote:
> Hi all,
>
> I currently work on a parser for some file format. I wanted to use the std.stdio.ByChunk Range to read from a file and extract tokens from the chunks. Obviously it can happen that the current chunk ends before a token can be extracted, in which case I can ask for the next chunk from the Range. In order to keep the already-read part in mind, I need to dup at least the unprocessed part of the older chunk and concatenate it in front of the next part or at least write the code that works like they were concatenated. This looks like a stupid approach to me.
>
> Here is a small example:
>
> file contents: "Hello world"
> chunks: "Hello w" "orld"
>
> First I read the token "Hello" from the first chunk and maybe skip the whitespace. Then I have the "w" (which I need to move away from the buffer, because ByChunk fill overwrite it) and get "orld".
>
> My idea was to have a ByChunk-related Object, which the user can tell how much of the buffer he/she actually used, such that it can move this data to the beginning of the buffer and append the next chunk. This wouldn't need further allocations and give the user contiguous data he/she can work with.
I coded something that works like this:

foreach (ref ubyte[] data; byBuffer(file, 12))
{
  writefln("[%s]", cast(string) data);
  data = data[$-2 .. $];
}

The 2nd line in the loop tells ByBuffer that we didn't process the last two chars and would like to get them again along with newly read data. And as long as we do process something, the internal buffer does not get reallocated.

It works and respects the formal requirements of ranges. Whether it respects the intended semantics, one can discuss about. Any comments whether the above things make sense or is an evil exploit of the provided syntax sugar?

On 12/10/10 22:36, Matthias Walter wrote:
> On 12/10/2010 09:57 PM, Matthias Walter wrote:
>> Hi all,
>>
>> I currently work on a parser for some file format. I wanted to use the std.stdio.ByChunk Range to read from a file and extract tokens from the chunks. Obviously it can happen that the current chunk ends before a token can be extracted, in which case I can ask for the next chunk from the Range. In order to keep the already-read part in mind, I need to dup at least the unprocessed part of the older chunk and concatenate it in front of the next part or at least write the code that works like they were concatenated. This looks like a stupid approach to me.
>>
>> Here is a small example:
>>
>> file contents: "Hello world"
>> chunks: "Hello w" "orld"
>>
>> First I read the token "Hello" from the first chunk and maybe skip the whitespace. Then I have the "w" (which I need to move away from the buffer, because ByChunk fill overwrite it) and get "orld".
>>
>> My idea was to have a ByChunk-related Object, which the user can tell how much of the buffer he/she actually used, such that it can move this data to the beginning of the buffer and append the next chunk. This wouldn't need further allocations and give the user contiguous data he/she can work with.
> I coded something that works like this:
> 
> foreach (ref ubyte[] data; byBuffer(file, 12))
> {
>   writefln("[%s]", cast(string) data);
>   data = data[$-2 .. $];
> }
> 
> The 2nd line in the loop tells ByBuffer that we didn't process the last two chars and would like to get them again along with newly read data. And as long as we do process something, the internal buffer does not get reallocated.
> 
> It works and respects the formal requirements of ranges. Whether it respects the intended semantics, one can discuss about. Any comments whether the above things make sense or is an evil exploit of the provided syntax sugar?

I don't think it's a bad approach, but I have a suggestion.

It leaves a lot of room for abuse or misuse if you require the user code to modify the data[] array in order to send this "protect some characters" message.  I think it would be better to provide an explicit function/method that means precisely that.  Maybe return a transparent struct wrapping a view to the buffer's data, that further provides a function for doing precisely this.

foreach( data; byBuffer( file, 12 )) {
  // do things with data, decide we need to keep 2 chars
  data.save( 2 );
}

Or something like it.  With regards to this, you may want to allow the internal buffer to grow (if you aren't already) as needed.  Imagine what would otherwise happen if you needed to 'save' the entire current buffer.

-- Chris N-S

On 12/11/2010 01:00 AM, Christopher Nicholson-Sauls wrote:
> On 12/10/10 22:36, Matthias Walter wrote:
>> On 12/10/2010 09:57 PM, Matthias Walter wrote:
>>> Hi all,
>>>
>>> I currently work on a parser for some file format. I wanted to use the std.stdio.ByChunk Range to read from a file and extract tokens from the chunks. Obviously it can happen that the current chunk ends before a token can be extracted, in which case I can ask for the next chunk from the Range. In order to keep the already-read part in mind, I need to dup at least the unprocessed part of the older chunk and concatenate it in front of the next part or at least write the code that works like they were concatenated. This looks like a stupid approach to me.
>>>
>>> Here is a small example:
>>>
>>> file contents: "Hello world"
>>> chunks: "Hello w" "orld"
>>>
>>> First I read the token "Hello" from the first chunk and maybe skip the whitespace. Then I have the "w" (which I need to move away from the buffer, because ByChunk fill overwrite it) and get "orld".
>>>
>>> My idea was to have a ByChunk-related Object, which the user can tell how much of the buffer he/she actually used, such that it can move this data to the beginning of the buffer and append the next chunk. This wouldn't need further allocations and give the user contiguous data he/she can work with.
>> I coded something that works like this:
>>
>> foreach (ref ubyte[] data; byBuffer(file, 12))
>> {
>>   writefln("[%s]", cast(string) data);
>>   data = data[$-2 .. $];
>> }
>>
>> The 2nd line in the loop tells ByBuffer that we didn't process the last two chars and would like to get them again along with newly read data. And as long as we do process something, the internal buffer does not get reallocated.
>>
>> It works and respects the formal requirements of ranges. Whether it respects the intended semantics, one can discuss about. Any comments whether the above things make sense or is an evil exploit of the provided syntax sugar?
> I don't think it's a bad approach, but I have a suggestion.
>
> It leaves a lot of room for abuse or misuse if you require the user code to modify the data[] array in order to send this "protect some characters" message.  I think it would be better to provide an explicit function/method that means precisely that.  Maybe return a transparent struct wrapping a view to the buffer's data, that further provides a function for doing precisely this.
>
> foreach( data; byBuffer( file, 12 )) {
>   // do things with data, decide we need to keep 2 chars
>   data.save( 2 );
> }
>
> Or something like it.  With regards to this, you may want to allow the internal buffer to grow (if you aren't already) as needed.  Imagine what would otherwise happen if you needed to 'save' the entire current buffer.
>
> -- Chris N-S
Thank you! This is a really good idea. So I basically wrap the buffer-array and implement it such that the default behavior (without explicitely doing something) is like the ByChunk mechanism.

Matthias