In order to feed large files (e.g. the ones like Zz uses) to YARD efficiently, I am creating a class which allows files to be treated exactly like std::string. Am I reinventing the wheel here, and there is something else I should know about?

Btw, I got the Imperfect C++ book a few days ago, and I am having fun reading it. A very impressive work Matthew!

-- 
Christopher Diggins
Object Oriented Template Library (OOTL)
http://www.ootl.org

"christopher diggins" <cdiggins@videotron.ca> wrote in message news:d0vvsl$q6l$1@digitaldaemon.com...
> In order to feed large files (e.g. the ones like Zz uses) to YARD efficiently, I am creating a class which allows files to be treated exactly like std::string. Am I reinventing the wheel here, and there is something else I should know about?

Not really sure what you mean. Can you give an example of where you might use it?

FYI: both UNIXSTL and WinSTL have the path class (basic_path<char-type> template), which does a great many path-y things. I copied the Boost idea of using / and /= for path concatenation, but I didn't look any deeper into their implementation. New to STLSoft 1.8.3 beta 2 (http://www.stlsoft.org/downloads.html) is the full and proper implememtation of the canonicalise() method, which'll remove and rationalise any embedded "." and ".." within the paths.

FYI-2: The two path implementations are almost identical. I plan to refactor that somehow in the future, but haven't given it any detailed thought as yet.

> Btw, I got the Imperfect C++ book a few days ago, and I am having fun reading it. A very impressive work Matthew!

Thanks!

"Matthew" <admin@stlsoft.dot.dot.dot.dot.org> wrote in message news:d100ja$r0q$1@digitaldaemon.com...
>
> "christopher diggins" <cdiggins@videotron.ca> wrote in message news:d0vvsl$q6l$1@digitaldaemon.com...
>> In order to feed large files (e.g. the ones like Zz uses) to YARD efficiently, I am creating a class which allows files to be treated exactly like std::string. Am I reinventing the wheel here, and there is something else I should know about?
>
> Not really sure what you mean. Can you give an example of where you might use it?

You would use it in generic code. You pass to your algorithm a template parameter which is either std::string or ootl::file_as_string, and you can use the same code-base for parsing files or for parsing strings. I want to have iterators, and the ability to construct sub-strings, and the other stuff would be fun as well. Think of it as a way, to have arbitrarily large strings stored in secondary storage.

> FYI: both UNIXSTL and WinSTL have the path class (basic_path<char-type> template), which does a great many path-y things. I copied the Boost idea of using / and /= for path concatenation, but I didn't look any deeper into their implementation. New to STLSoft 1.8.3 beta 2 (http://www.stlsoft.org/downloads.html) is the full and proper implememtation of the canonicalise() method, which'll remove and rationalise any embedded "." and ".." within the paths.
>
> FYI-2: The two path implementations are almost identical. I plan to refactor that somehow in the future, but haven't given it any detailed thought as yet.

Thanks for bringing these to my attention.

-- 
Christopher Diggins
Object Oriented Template Library (OOTL)
http://www.ootl.org

"christopher diggins" <cdiggins@videotron.ca> wrote in message news:d11kmc$29kc$1@digitaldaemon.com...
> "Matthew" <admin@stlsoft.dot.dot.dot.dot.org> wrote in message news:d100ja$r0q$1@digitaldaemon.com...
>>
>> "christopher diggins" <cdiggins@videotron.ca> wrote in message news:d0vvsl$q6l$1@digitaldaemon.com...
>>> In order to feed large files (e.g. the ones like Zz uses) to YARD efficiently, I am creating a class which allows files to be treated exactly like std::string. Am I reinventing the wheel here, and there is something else I should know about?
>>
>> Not really sure what you mean. Can you give an example of where you might use it?
>
> You would use it in generic code. You pass to your algorithm a template parameter which is either std::string or ootl::file_as_string, and you can use the same code-base for parsing files or for parsing strings. I want to have iterators, and the ability to construct sub-strings, and the other stuff would be fun as well. Think of it as a way, to have arbitrarily large strings stored in secondary storage.

You'd have to give me a small example to get what you mean.

"christopher diggins" <cdiggins@videotron.ca> wrote in message news:d11kmc$29kc$1@digitaldaemon.com...
> "Matthew" <admin@stlsoft.dot.dot.dot.dot.org> wrote in message news:d100ja$r0q$1@digitaldaemon.com...
>>
>> "christopher diggins" <cdiggins@videotron.ca> wrote in message news:d0vvsl$q6l$1@digitaldaemon.com...
>>> In order to feed large files (e.g. the ones like Zz uses) to YARD efficiently, I am creating a class which allows files to be treated exactly like std::string. Am I reinventing the wheel here, and there is something else I should know about?
>>
>> Not really sure what you mean. Can you give an example of where you might use it?
>
> You would use it in generic code. You pass to your algorithm a template parameter which is either std::string or ootl::file_as_string, and you can use the same code-base for parsing files or for parsing strings. I want to have iterators, and the ability to construct sub-strings, and the other stuff would be fun as well. Think of it as a way, to have arbitrarily large strings stored in secondary storage.

(One bike ride later and) I think I understand what you mean now. It presents a string interface, but is based on a file.

Well, why not use (basic_)string_view coupled to a memory-mapped file? That should provide all that you need, I think.

Of course, a little wrapper to that might not be a bad thing. I'd suggest you try it first, and let me know if that satisfies your needs.

btw, I'm assuming that you have good grounds for not using istream_iterator??

"Matthew" <admin@stlsoft.dot.dot.dot.dot.org> wrote in message news:d12d0b$31ca$1@digitaldaemon.com...
>
> "christopher diggins" <cdiggins@videotron.ca> wrote in message news:d11kmc$29kc$1@digitaldaemon.com...
>> "Matthew" <admin@stlsoft.dot.dot.dot.dot.org> wrote in message news:d100ja$r0q$1@digitaldaemon.com...
>>>
>>> "christopher diggins" <cdiggins@videotron.ca> wrote in message news:d0vvsl$q6l$1@digitaldaemon.com...
>>>> In order to feed large files (e.g. the ones like Zz uses) to YARD efficiently, I am creating a class which allows files to be treated exactly like std::string. Am I reinventing the wheel here, and there is something else I should know about?
>>>
>>> Not really sure what you mean. Can you give an example of where you might use it?
>>
>> You would use it in generic code. You pass to your algorithm a template parameter which is either std::string or ootl::file_as_string, and you can use the same code-base for parsing files or for parsing strings. I want to have iterators, and the ability to construct sub-strings, and the other stuff would be fun as well. Think of it as a way, to have arbitrarily large strings stored in secondary storage.
>
> (One bike ride later and) I think I understand what you mean now. It presents a string interface, but is based on a file.

Yes, well put.

> Well, why not use (basic_)string_view coupled to a memory-mapped file? That should provide all that you need, I think.

I am not familiar with this technique, what is a string_view? I know the theory of memory-mapped files, but I haven't used them. What is a good api for them?

> Of course, a little wrapper to that might not be a bad thing. I'd suggest you try it first, and let me know if that satisfies your needs.

Sure.

> btw, I'm assuming that you have good grounds for not using istream_iterator??

It's not bi-directional. A recursive descent parser moves forwards and backwards a lot, and needs to be able to capture sub-strings from the input. Also didn't you and Zz say that istreams were too slow?

-- 
Christopher Diggins
Object Oriented Template Library (OOTL)
http://www.ootl.org

Revisiting the problem of inputing to the YARD parser, I realize I simply need a forward iterator for files. The STL only provides input iterators for file streams which is insufficient. So I have to roll my own. The question is whether I would be better off using stdio.h routines for the implementation or should I instead using an ifstream based implementation. Any suggestions?

Thanks,

-- 
Christopher Diggins
Object Oriented Template Library (OOTL)
http://www.ootl.org

"christopher diggins" <cdiggins@videotron.ca> wrote in message news:d14d8s$1vvb$1@digitaldaemon.com...
> "Matthew" <admin@stlsoft.dot.dot.dot.dot.org> wrote in message news:d12d0b$31ca$1@digitaldaemon.com...
>>
>> "christopher diggins" <cdiggins@videotron.ca> wrote in message news:d11kmc$29kc$1@digitaldaemon.com...
>>> "Matthew" <admin@stlsoft.dot.dot.dot.dot.org> wrote in message news:d100ja$r0q$1@digitaldaemon.com...
>>>>
>>>> "christopher diggins" <cdiggins@videotron.ca> wrote in message news:d0vvsl$q6l$1@digitaldaemon.com...
>>>>> In order to feed large files (e.g. the ones like Zz uses) to YARD
>>>>> efficiently, I am creating a class which allows files to be
>>>>> treated
>>>>> exactly like std::string. Am I reinventing the wheel here, and
>>>>> there
>>>>> is something else I should know about?
>>>>
>>>> Not really sure what you mean. Can you give an example of where you might use it?
>>>
>>> You would use it in generic code. You pass to your algorithm a
>>> template parameter which is either std::string or
>>> ootl::file_as_string, and you can use the same code-base for parsing
>>> files or for parsing strings. I want to have iterators, and the
>>> ability to construct sub-strings, and the other stuff would be fun
>>> as
>>> well. Think of it as a way, to have arbitrarily large strings stored
>>> in secondary storage.
>>
>> (One bike ride later and) I think I understand what you mean now. It presents a string interface, but is based on a file.
>
> Yes, well put.
>
>> Well, why not use (basic_)string_view coupled to a memory-mapped file? That should provide all that you need, I think.
>
> I am not familiar with this technique, what is a string_view?

(basic_)string_view is a (template) class that presents a String Model interface to the world, but on someone else's storage. It's appeared in STLSoft publicly as of 1.8.3 beta.

It is essentially a length + char ptr.

One thing I use it for is in combination with STLSoft's string_tokeniser, such that each split token does not have to have a separate block of memory for it. Naturally this makes it more efficient.

The obvious downside is that if the underlying memory goes away, then accessing the string view is not a good idea.

Also, calling c_str() is to be avoided if possible, since that will have to result in allocation in order to be able to append a nul character.

But it sounds ideal for your purposes, because you'd have that memory there, and it'd be little effort to make sure you're not taking it away prematurely.

>  I know the theory of memory-mapped files, but I haven't used them.
> What is a good api for them?

This is on the STLSoft to-do, for both UNIXSTL and WinSTL. For the moment, assuming you're developing on Windows first, just use the long handed way, calling CreateFile(), CreateFileMapping() and MapViewOfFile().

>> Of course, a little wrapper to that might not be a bad thing. I'd suggest you try it first, and let me know if that satisfies your needs.
>
> Sure.
>
>> btw, I'm assuming that you have good grounds for not using istream_iterator??
>
> It's not bi-directional.

Of course. Big Doh! to me. :-)

> A recursive descent parser moves forwards and backwards a lot, and needs to be able to capture sub-strings from the input. Also didn't you and Zz say that istreams were too slow?

Can be. :-)

"christopher diggins" <cdiggins@videotron.ca> wrote in message news:d150mj$2k8a$1@digitaldaemon.com...
> Revisiting the problem of inputing to the YARD parser, I realize I simply need a forward iterator for files. The STL only provides input iterators for file streams which is insufficient. So I have to roll my own. The question is whether I would be better off using stdio.h routines for the implementation or should I instead using an ifstream based implementation. Any suggestions?

To get a forward iterator for a file, you'd have to dup file handles, which would suggest using stdio. To be honest, it'd be a lot more effort that using a string_view, which remains my recommendation.

Also, in the general case, duping file handles to enumate Forward Iterator semantics is frought with danger, since the file might change.

.. snip..
> (One bike ride later and) I think I understand what you mean now. It presents a string interface, but is based on a file.
>
> Well, why not use (basic_)string_view coupled to a memory-mapped file? That should provide all that you need, I think.

If I recall correctly CreateFileMapping allows only a maximum of just over 2GB under windows.

> Of course, a little wrapper to that might not be a bad thing. I'd suggest you try it first, and let me know if that satisfies your needs.
>
> btw, I'm assuming that you have good grounds for not using istream_iterator??

I don't like io-streams but wouldn't istreambuf_iterator be better than istream_iterator in the case that io-streams would be used?

Zz