"christopher diggins" <cdiggins@videotron.ca> wrote in message news:d150mj$2k8a$1@digitaldaemon.com...
> Revisiting the problem of inputing to the YARD parser, I realize I simply need a forward iterator for files. The STL only provides input iterators for file streams which is insufficient. So I have to roll my own. The question is whether I would be better off using stdio.h routines for the implementation or should I instead using an ifstream based implementation. Any suggestions?
>
I'm biased, I really dislike io-streams with a passion so I would say use stdio.h, but on the other hand you are creating a library for C++ user and some people might like the idea of having something based on io-streams.

Also quite a number new programmers I've meet don't even know what stdio.h
is.
Matthew would be very helpful here, you should look at the article he wrote
for windows developer back in 2003 titled "C# Performance: Comparison with
C, C++, D and Java Part 1 & 2 (It was very interesting).

Zz

"Zz" <Zz@Zz.com> wrote in message news:d158n7$2sh5$1@digitaldaemon.com...
> I don't like io-streams but wouldn't istreambuf_iterator be better than istream_iterator in the case that io-streams would be used?


Sorry, I forgot to reply to this,

istreambuf_iterator is an input iterator, which means you can't use it in multi-pass algorithms (as needed by an R-D parser). I have implemented a forward iterator for files based on fopen, fseek and fread which is available at http://www.ootl.org/ootl/ootl_file_reader.hpp.htm .

-- 
Christopher Diggins
Object Oriented Template Library (OOTL)
http://www.ootl.org

> Sorry, I forgot to reply to this,
>
> istreambuf_iterator is an input iterator, which means you can't use it in multi-pass algorithms (as needed by an R-D parser). I have implemented a forward iterator for files based on fopen, fseek and fread which is available at http://www.ootl.org/ootl/ootl_file_reader.hpp.htm .
>
Hi Chris,

It looks good but...

The only problem I personaly have with fseek and ftell is that you must represent the underlying file as a long and this will bring up issues with large files.

fsetpos and fgetpos don't have the above limitation on size but you loose random file access. While parsing you may get the current position and allways jump back to the previous point by using fsetpos.

Note: while SEEK_END is not really very portable for getting file sizes.

Zz

"Zz" <Zz@Zz.com> wrote in message news:d1lviu$2crf$1@digitaldaemon.com...
>
>> Sorry, I forgot to reply to this,
>>
>> istreambuf_iterator is an input iterator, which means you can't use it in multi-pass algorithms (as needed by an R-D parser). I have implemented a forward iterator for files based on fopen, fseek and fread which is available at http://www.ootl.org/ootl/ootl_file_reader.hpp.htm .
>>
> Hi Chris,
>
> It looks good but...
>
> The only problem I personaly have with fseek and ftell is that you must represent the underlying file as a long and this will bring up issues with large files.

Ah yes. Are there any compilers which provide 64 bit longs yet? It seems that this is inevitable soon aren't they? I believe that with Visual C++ we could at least use the _fseek64 and _ftell64 for the time being. Do any other compilers provide similar 64 bit implementations of the C runtime libraries?

What about ifstream.tellg and ifstream.seekg? How often are they implemented as 64 bits? Specifically what are your most common system compiler combinations?

> fsetpos and fgetpos don't have the above limitation on size but you loose random file access. While parsing you may get the current position and allways jump back to the previous point by using fsetpos.

Ah, I see. Perhaps I can make this work then using a combination of fsetpos and fseek.

> Note: while SEEK_END is not really very portable for getting file sizes.

Again, because of the long problem right?

Hmmm..... I have some other tricks up my sleeve. Unfortunately the large file parsing problem will have to be put on hold temporarily, I have to finish up an article and make a release of one of my other libraries.

Thanks,
-D

.. snip..
> Ah yes. Are there any compilers which provide 64 bit longs yet? It seems that this is inevitable soon aren't they? I believe that with Visual C++ we could at least use the _fseek64 and _ftell64 for the time being. Do any other compilers provide similar 64 bit implementations of the C runtime libraries?

Metrowerks does not have it.

> What about ifstream.tellg and ifstream.seekg? How often are they implemented as 64 bits? Specifically what are your most common system compiler combinations?

I maily use Metrowerks, I'm just used to it and I think it has the best
C/C++ debugger out there.
I looked at Visual Studio 2005 Beta, and I liked how it understood STL
containers.

For final compilation Intel may be used.

>> fsetpos and fgetpos don't have the above limitation on size but you loose random file access. While parsing you may get the current position and allways jump back to the previous point by using fsetpos.
>
> Ah, I see. Perhaps I can make this work then using a combination of fsetpos and fseek.

The only time I really need to jump to a specific part of the file is when
handling images, when doing regular text parsing I really don't jump to any
part of the file since at the most it will be an LL(2) parser.
Even with memory mapped files jumping to different portions can affect
performance (thats why both windows and posix have special optimizations for
sequential access).

>> Note: while SEEK_END is not really very portable for getting file sizes.

This may come as a supprise but a C89 compiler is not required to support SEEK_END for files opened with "rb".

> Again, because of the long problem right?
>
> Hmmm..... I have some other tricks up my sleeve. Unfortunately the large file parsing problem will have to be put on hold temporarily, I have to finish up an article and make a release of one of my other libraries.

I totally understand (it's better to get thing rolling and then attacking the problem later on).

Cheers,
Zz

"Zz" <Zz@Zz.com> wrote in message news:d1nkmd$10hv$1@digitaldaemon.com...
>
>>> Note: while SEEK_END is not really very portable for getting file sizes.
>
> This may come as a supprise but a C89 compiler is not required to support SEEK_END for files opened with "rb".

Thanks for telling me about that. I did not know.

>> Again, because of the long problem right?
>>
>> Hmmm..... I have some other tricks up my sleeve. Unfortunately the large file parsing problem will have to be put on hold temporarily, I have to finish up an article and make a release of one of my other libraries.
>
> I totally understand (it's better to get thing rolling and then attacking the problem later on).


Speaking of which, have you had a chance to try the latest release?

-- 
Christopher Diggins
Object Oriented Template Library (OOTL)
http://www.ootl.org

>>>
>>> Hmmm..... I have some other tricks up my sleeve. Unfortunately the large file parsing problem will have to be put on hold temporarily, I have to finish up an article and make a release of one of my other libraries.
>>
>> I totally understand (it's better to get thing rolling and then attacking the problem later on).
>
>
> Speaking of which, have you had a chance to try the latest release?
>
I'll look at it on sunday.

Zz

"Zz" <Zz@Zz.com> wrote in message news:d1vgsm$sgo$1@digitaldaemon.com...
>
>>>>
>>>> Hmmm..... I have some other tricks up my sleeve. Unfortunately the large file parsing problem will have to be put on hold temporarily, I have to finish up an article and make a release of one of my other libraries.
>>>
>>> I totally understand (it's better to get thing rolling and then attacking the problem later on).
>>
>>
>> Speaking of which, have you had a chance to try the latest release?
>>
> I'll look at it on sunday.
>
> Zz


Thanks.

I had rewritten the large file reader using ifstream routines recently and learned the hard way some new things.

At least for gcc 3.4 the following appears to be true:

- ifstream.readsome always returns 0
- ifstream.seekg(0, ios::end) fails for large files
- if you use ifstream.read at the end of the file, calls to ifstream.tellg()
fails

This really ties my hands as to what I can do.

I can probably rewrite it again to make things work using FILE* routines, but I need to know whether fread(FILE, 0) will work dependably for large files. Can you give me any hints or advice?

-- 
Christopher Diggins
Object Oriented Template Library (OOTL)
http://www.ootl.org

Chris

Have you tried using string_view + mmf yet?

If you're interested, I'd be happy to sketch out a skeleton example to show how it'd be used.

Cheers

Matthew

"christopher diggins" <cdiggins@videotron.ca> wrote in message news:d217hl$2uk3$1@digitaldaemon.com...
> "Zz" <Zz@Zz.com> wrote in message news:d1vgsm$sgo$1@digitaldaemon.com...
>>
>>>>>
>>>>> Hmmm..... I have some other tricks up my sleeve. Unfortunately the large file parsing problem will have to be put on hold temporarily, I have to finish up an article and make a release of one of my other libraries.
>>>>
>>>> I totally understand (it's better to get thing rolling and then attacking the problem later on).
>>>
>>>
>>> Speaking of which, have you had a chance to try the latest release?
>>>
>> I'll look at it on sunday.
>>
>> Zz
>
>
> Thanks.
>
> I had rewritten the large file reader using ifstream routines recently and learned the hard way some new things.
>
> At least for gcc 3.4 the following appears to be true:
>
> - ifstream.readsome always returns 0
> - ifstream.seekg(0, ios::end) fails for large files
> - if you use ifstream.read at the end of the file, calls to ifstream.tellg() fails
>
> This really ties my hands as to what I can do.
>
> I can probably rewrite it again to make things work using FILE* routines, but I need to know whether fread(FILE, 0) will work dependably for large files. Can you give me any hints or advice?
>
> -- 
> Christopher Diggins
> Object Oriented Template Library (OOTL)
> http://www.ootl.org
> 

"Matthew" <admin@stlsoft.dot.dot.dot.dot.org> wrote in message news:d2214j$rej$1@digitaldaemon.com...
> Chris
>
> Have you tried using string_view + mmf yet?
>
> If you're interested, I'd be happy to sketch out a skeleton example to show how it'd be used.

Thanks, but I haven't tried yet because Zz mentioned there is a 2(or was it 4) GIG limit on MMF's in windows. Can you either confirm or affirm this?

Do you know if fread() is reliable?

-D