phobos commit, revision 1877


user: andrei

msg:
Fixed seek and tell for large files

http://www.dsource.org/projects/phobos/changeset/1877

Works on win32wine and osx, could someone check on Linuxen?

Andrei

dsource.org wrote:
> phobos commit, revision 1877
> 
> 
> user: andrei
> 
> msg:
> Fixed seek and tell for large files
> 
> http://www.dsource.org/projects/phobos/changeset/1877
> 
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

Are you sure that the offset argument to seek() is supposed to be unsigned? How do you seek a negative offset from the end of the file or the current position?

On Mon, Aug 16, 2010 at 4:10 PM, Andrei Alexandrescu <andrei at erdani.com>wrote:

> Works on win32wine and osx, could someone check on Linuxen?
>
> Andrei
>
>
> dsource.org wrote:
>
>> phobos commit, revision 1877
>>
>>
>> user: andrei
>>
>> msg:
>> Fixed seek and tell for large files
>>
>> http://www.dsource.org/projects/phobos/changeset/1877
>>
>> _______________________________________________
>> phobos mailing list
>> phobos at puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/phobos
>>
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/phobos/attachments/20100816/1c30d98f/attachment.html>

This is wrong on two counts.

First, david is right, changing from long to ulong is a mistake.  long is correct, this supports 2^63 bytes, which is plenty larger than the current 2GB limitation, and it allows negative seeking without issue.

Second, you cannot do a tell on the file handle, because you need to take into account the buffer.  I went through this exercise with Kris in Tango, seek and tell need to account for the buffer for any semblance of normalcy.  Tango used to have a buffer that was both a read and write buffer, depending how you used it.  The problem is, seek/tell didn't know how you were using it (the code is *very* different to account for the buffer on read vs. write) :)

In response to the "why UnbufferedFile" requirement, it's for this same reason. If you pass the FILE*, you may have consumed some data from the raw handle into the buffer, and then expect the child process to read from the start of your buffer, which of course it can't.  Using UnbufferedFile makes it clear that you need to ignore the buffer.  Lars put in some methods to convert easily from File to UnbufferedFile.  I think we discussed this on the ML a while back...

-Steve



----- Original Message ----
> From: Andrei Alexandrescu <andrei at erdani.com>
> To: Discuss the phobos library for D <phobos at puremagic.com>
> Sent: Mon, August 16, 2010 4:10:05 PM
> Subject: Re: [phobos] phobos commit, revision 1877
> 
> Works on win32wine and osx, could someone check on  Linuxen?
> 
> Andrei
> 
> dsource.org wrote:
> > phobos commit, revision  1877
> > 
> > 
> > user: andrei
> > 
> > msg:
> > Fixed  seek and tell for large files
> > 
> >  http://www.dsource.org/projects/phobos/changeset/1877
> > 
> >  _______________________________________________
> > phobos mailing  list
> > phobos at puremagic.com
> >  http://lists.puremagic.com/mailman/listinfo/phobos
> _______________________________________________
> phobos  mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos
> 

I meant lseek64, not tell.

-Steve



----- Original Message ----
> From: Steve Schveighoffer <schveiguy at yahoo.com>
> Second, you cannot do a tell on the file handle,  because you need to take into
>
> account the buffer.

Got it. So we somehow need to stay in sync with the FILE* API. I managed to do so by calling flush() inside seek(), I wonder if something similar can be done in tell().

Andrei

Steve Schveighoffer wrote:
> I meant lseek64, not tell.
> 
> -Steve
> 
> 
> 
> ----- Original Message ----
>> From: Steve Schveighoffer <schveiguy at yahoo.com>
>> Second, you cannot do a tell on the file handle,  because you need to take into
>>
>> account the buffer.
> 
> 
> 
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

Flush inside seek will work for seeking absolute, but will not work for seeking relative.

I found this in the man pages of linux, not sure if it's portable:

STDIO_EXT(3)               Linux Programmer?s Manual              STDIO_EXT(3)

NAME
       __fbufsize,  __flbf,  __fpending,  __fpurge,  __freadable,  __freading,
       __fsetlocking, __fwritable, __fwriting, _flushlbf - interfaces to stdio
       FILE structure

Walter, do we have stuff like this in DMC?

-Steve



----- Original Message ----
> From: Andrei Alexandrescu <andrei at erdani.com>
> To: Discuss the phobos library for D <phobos at puremagic.com>
> Sent: Mon, August 16, 2010 6:01:41 PM
> Subject: Re: [phobos] phobos commit, revision 1877
> 
> Got it. So we somehow need to stay in sync with the FILE* API. I managed to do
>so by calling flush() inside seek(), I wonder if something similar can be done in tell().
> 
> Andrei
> 
> Steve Schveighoffer wrote:
> > I meant  lseek64, not tell.
> > 
> > -Steve
> > 
> > 
> > 
> >  ----- Original Message ----
> >> From: Steve Schveighoffer <schveiguy at yahoo.com>
> >>  Second, you cannot do a tell on the file handle,  because you need to take
>into
>
> >> account the buffer.
> > 
> > 
> >        _______________________________________________
> > phobos mailing  list
> > phobos at puremagic.com
> >  http://lists.puremagic.com/mailman/listinfo/phobos
> _______________________________________________
> phobos  mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos
> 

Some of it, but not all. I suggest just starting with the linux version and getting it right, then we'll see about porting it to windows. Note that OSX and FreeBSD often differ substantially on these sorts of APIs.

Steve Schveighoffer wrote:
> Flush inside seek will work for seeking absolute, but will not work for seeking relative.
>
> I found this in the man pages of linux, not sure if it's portable:
>
> STDIO_EXT(3)               Linux Programmer?s Manual              STDIO_EXT(3)
>
> NAME
>        __fbufsize,  __flbf,  __fpending,  __fpurge,  __freadable,  __freading,
>        __fsetlocking, __fwritable, __fwriting, _flushlbf - interfaces to stdio
>        FILE structure
>
> Walter, do we have stuff like this in DMC?
>
> -Steve
>
>
>
> ----- Original Message ----
> 
>> From: Andrei Alexandrescu <andrei at erdani.com>
>> To: Discuss the phobos library for D <phobos at puremagic.com>
>> Sent: Mon, August 16, 2010 6:01:41 PM
>> Subject: Re: [phobos] phobos commit, revision 1877
>>
>> Got it. So we somehow need to stay in sync with the FILE* API. I managed to do so by calling flush() inside seek(), I wonder if something similar can be done in tell().
>>
>> Andrei
>>
>> Steve Schveighoffer wrote:
>> 
>>> I meant  lseek64, not tell.
>>>
>>> -Steve
>>>
>>>
>>>
>>>  ----- Original Message ----
>>> 
>>>> From: Steve Schveighoffer <schveiguy at yahoo.com>
>>>>  Second, you cannot do a tell on the file handle,  because you need to take
>>>> 
>> into
>>
>> 
>>>> account the buffer.
>>>> 
>>>        _______________________________________________
>>> phobos mailing  list
>>> phobos at puremagic.com
>>>  http://lists.puremagic.com/mailman/listinfo/phobos
>>> 
>> _______________________________________________
>> phobos  mailing list
>> phobos at puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/phobos
>>
>> 
>
>
> 
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

Steve Schveighoffer wrote:
> This is wrong on two counts.
> 
> First, david is right, changing from long to ulong is a mistake.  long is correct, this supports 2^63 bytes, which is plenty larger than the current 2GB limitation, and it allows negative seeking without issue.

Good point. Fixed.

> Second, you cannot do a tell on the file handle, because you need to take into account the buffer.

My understanding is that calling fflush() before lseek should be fine. See my upcoming commit.


Andrei

Great.  I'll stress test this (i.e. test it in ways that would not be feasible to include in a unittest, such as with an actual 5GB file) tonight.  Right now I don't have time.  I'll let you know how it goes.

On 8/17/2010 3:09 AM, Andrei Alexandrescu wrote:
> Steve Schveighoffer wrote:
>> This is wrong on two counts.
>>
>> First, david is right, changing from long to ulong is a mistake. long is correct, this supports 2^63 bytes, which is plenty larger than the current 2GB limitation, and it allows negative seeking without issue.
>
> Good point. Fixed.
>
>> Second, you cannot do a tell on the file handle, because you need to take into account the buffer.
>
> My understanding is that calling fflush() before lseek should be fine. See my upcoming commit.
>
>
> Andrei
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos
>