I've done the UNIXSTL one. It's structurally compliant with the WinSTL one, so you can do simple platform discrimination, a la.

    #if defined(unix)

    # include <unixstl/memory_mapped_file.hpp>
    namespace platform_stl
    {
        typedef ::unixstl::memory_mapped_file    memory_mapped_file;
    }

    #elif defined(WIN32)

    # include <winstl/memory_mapped_file.hpp>
    namespace platform_stl
    {
        typedef ::winstl::memory_mapped_file    memory_mapped_file;
    }

    #else
    # error Platform not recognised
    #endif /* platform */

There are lots of these structurally compliant components between the two OS-specific subprojects, and I'm starting to think about doing a PlatformSTL sub-project, to encapsulate them. (I have *no* intention of adding any platform-dependent code to the STLSoft main project, ever).

Does anyone (else) have any opinions on this? I use the namespace aliasing technique - documented in http://www.cuj.com/documents/s=9152/cujexp0404wilson/ - quite a bit in recls, so it may be time to start a real project for it.

Anyway, I'm waffling. The memory_mapped_file classes will be in STLSoft 1.8.3 beta 4 very shortly.

Cheers

Matthew


"Matthew" <admin.hat@stlsoft.dot.org> wrote in message news:d1j3ab$2qie$1@digitaldaemon.com...
> Is written and tested. It's a read-only adaptation of a more flexible (i.e. does read, write read/write) Synesis class that I've never been terribly happy with (hence it's not being included in STLSoft thus far.)
>
> Because it's read-only, it's very easy to keep it really nice and simple.
>
> I'm going to do the UNIXSTL version now, and then you'll be able to use string_view. Peachy, or what?
>
> :-)
>
> Matthew
>
> 

"Matthew" <admin.hat@stlsoft.dot.org> wrote in message news:d1j4g9$2rgf$1@digitaldaemon.com...
> I've done the UNIXSTL one. It's structurally compliant with the WinSTL one, so you can do simple platform discrimination, a la.
>
>    #if defined(unix)
>
>    # include <unixstl/memory_mapped_file.hpp>
>    namespace platform_stl
>    {
>        typedef ::unixstl::memory_mapped_file    memory_mapped_file;
>    }
>
>    #elif defined(WIN32)
>
>    # include <winstl/memory_mapped_file.hpp>
>    namespace platform_stl
>    {
>        typedef ::winstl::memory_mapped_file    memory_mapped_file;
>    }
>
>    #else
>    # error Platform not recognised
>    #endif /* platform */
>
> There are lots of these structurally compliant components between the two OS-specific subprojects, and I'm starting to think about doing a PlatformSTL sub-project, to encapsulate them. (I have *no* intention of adding any platform-dependent code to the STLSoft main project, ever).
>
> Does anyone (else) have any opinions on this? I use the namespace aliasing technique - documented in http://www.cuj.com/documents/s=9152/cujexp0404wilson/ - quite a bit in recls, so it may be time to start a real project for it.
>
> Anyway, I'm waffling. The memory_mapped_file classes will be in STLSoft 1.8.3 beta 4 very shortly.

I've done the PlatformSTL sub-project. Now one simple #includes the requisite platformstl header, e.g.

#include <platformstl/memory_mapped_file.hpp>

and use the type ::platformstl::memory_mapped_file.

I'll naturally be making all structurally compliant components between UNIXSTL and WinSTL available in this form, and it'll be included in 1.8.3 beta 4

Cheers

Matthew

"Matthew" <admin@stlsoft.dot.dot.dot.dot.org> wrote in message news:d1ik3p$2dgp$1@digitaldaemon.com...
> I'll dig into this. Remind me in a day or so if I've not responded.
>
> Naturally, as you say, an opened file, in non-shared mode, takes away the 'theoretical' uncertainty I was gassing on about. But "rb" doesn't cut it. For one, "b" is a Microsoft / Windows extension. (On UNIX, there is no \r\n, so no need for \n <=> \r\n translation.)

Thanks for pointing that out. There are a lot of facts like these that I simply don'ty know, because I have only ever done windows development my whole career.

> Also - and I may need to check my facts on this - "r", in and of itself, does not prevent another process sharing for write.

Any operating system that allowed files opened with fopen to be written to would make that API unsound. It would seem ludicrous for them to not have blocking read modes.

> But I'd have to check this again, and the interrelationship between fopen() and open() (and open()'s pmode argument).

I don't know the function open(). Isn't it a unix thing?

> But all of this is conjecture, since it's been a long time since I've had to be concerned about sharing wrt stdio (or even io), as I tend to use OS-specific file system calls when I care about it.
>
> Finally, although I've only just scanned it, I'm not seeing either any dup or multiple fopen calls, which makes me wonder how you support forward iterators. If you're seeking the same file-handle back and forth on the demands of different (concurrent) iterator instances, then there's certainly potential there for poor performance.

The iterators access memory through a buffer. Fseek/fread is only used if an iterator requests memory from a currently unallocated buffer. Buffers are freed automatically when an iterator no longer references it.

> But however well designed/implemented it is, it is very unlikely that it'll approach the performance of memory-mapped file reading. The question will be whether the gap is significant c/w other latencies within YARD. I'll try and get a mo to play with it, and let you know.

Thank you.

-- 
Christopher Diggins
Object Oriented Template Library (OOTL)
http://www.ootl.org

"Matthew" <admin.hat@stlsoft.dot.org> wrote in message news:d1j3ab$2qie$1@digitaldaemon.com...
> Is written and tested. It's a read-only adaptation of a more flexible (i.e. does read, write read/write) Synesis class that I've never been terribly happy with (hence it's not being included in STLSoft thus far.)
>
> Because it's read-only, it's very easy to keep it really nice and simple.
>
> I'm going to do the UNIXSTL version now, and then you'll be able to use string_view. Peachy, or what?
>
> :-)

Very peachy!

-- 
Christopher Diggins
Object Oriented Template Library (OOTL)
http://www.ootl.org

"Matthew" <admin.hat@stlsoft.dot.org> wrote in message news:d1j4g9$2rgf$1@digitaldaemon.com...
> I've done the UNIXSTL one. It's structurally compliant with the WinSTL one, so you can do simple platform discrimination, a la.
>
>    #if defined(unix)
>
>    # include <unixstl/memory_mapped_file.hpp>
>    namespace platform_stl
>    {
>        typedef ::unixstl::memory_mapped_file    memory_mapped_file;
>    }
>
>    #elif defined(WIN32)
>
>    # include <winstl/memory_mapped_file.hpp>
>    namespace platform_stl
>    {
>        typedef ::winstl::memory_mapped_file    memory_mapped_file;
>    }
>
>    #else
>    # error Platform not recognised
>    #endif /* platform */
>
> There are lots of these structurally compliant components between the two OS-specific subprojects, and I'm starting to think about doing a PlatformSTL sub-project, to encapsulate them. (I have *no* intention of adding any platform-dependent code to the STLSoft main project, ever).

Which brings up a good point. If the YARD includes the memory_mapped_file routines as part of the main library then it can't be put in the STLSoft main project. I think the solutation to this is that the file reading code should be placed outside of the main YARD project. The generic design of YARD input makes this an appropriate design choice I think.

 > Does anyone (else) have any opinions on this? I use the namespace
aliasing technique - documented in
> http://www.cuj.com/documents/s=9152/cujexp0404wilson/ - quite a bit in recls, so it may be time to start a real project for it.


This appears to be the poster child for namespace aliasing, and it would save a lot of code.

> Anyway, I'm waffling. The memory_mapped_file classes will be in STLSoft 1.8.3 beta 4 very shortly.

I look forward to it, thanks for responding so quickly.

-D

"Matthew" <admin@stlsoft.dot.dot.dot.dot.org> wrote in message news:d1iloo$2f6q$1@digitaldaemon.com...
> It's generally accepted to be a really bad principle to use using directives within headers. I also swim somewhat against the stream in suggesting that it's a bad idea to use them in implementation files, but that's less universally accepted.

Could you quickly summarize your position here?

> Your use of
>
>  namespace yard
>  {
>    using namespace std;
>  }
>
> in yard.hpp is causing problems for me in playing around with YARD from within Visual Studio '98 + Intel 8.0 + STLport. That's obviously one case in which the bad side of using directives crops up. I don't doubt that, if you leave it in, we'll encounter cause more.

Sweet lord, give me strength. Someday Heron will be ready, and all of this will be a thing of the past.

> Also, yard.hpp includes a *lot* of things - pretty much the fattest headers in the standard library. Are they *all* necessary for *all* parts of the library?

I really doubt they are. Header inclusion is never something I paid any detail to.

> (Note: I realise that at the moment you're wisely focusing on functionality, and will trim later. But sometimes it can be really hard to un-engineer things with such wide ramifications.)

Good point, I will try to address the problem sometime soon.

-- 
Christopher Diggins
Object Oriented Template Library (OOTL)
http://www.ootl.org

"Matthew" <admin@stlsoft.dot.dot.dot.dot.org> wrote in message news:d1ij1b$2ckn$1@digitaldaemon.com...
> Here's an initial set of results from Arturius. Despite how it looks, it's actually pretty encouraging.
>
> I've got to go out now, but will have a deeper dig in a couple of hours. Some of the things, like -EHsc missing, will be instantly resolved (by my specifying the appropriate Arturius cmd-line params.)

You should have a special congratualatory message replete with ascii art, when something does in fact compile correctly on a given compiler. Emphasize the positive ;-)

Thanks for giving it a run on Arturius.

-- 
Christopher Diggins
Object Oriented Template Library (OOTL)
http://www.ootl.org

.. snip ..
> Naturally, as you say, an opened file, in non-shared mode, takes away the 'theoretical' uncertainty I was gassing on about. But "rb" doesn't cut it. For one, "b" is a Microsoft / Windows extension. (On UNIX, there is no \r\n, so no need for \n <=> \r\n translation.)

and don't forget \r for the mac.

\r\n = Windows
\n   = Unix
\r    = Mac.

While parsing we usually open files in binary and keep track of line endinds on our own.

Zz

"Matthew" <admin.hat@stlsoft.dot.org> wrote in message news:d1j3ab$2qie$1@digitaldaemon.com...
> Is written and tested. It's a read-only adaptation of a more flexible (i.e. does read, write read/write) Synesis class that I've never been terribly happy with (hence it's not being included in STLSoft thus far.)
>
> Because it's read-only, it's very easy to keep it really nice and simple.
>
> I'm going to do the UNIXSTL version now, and then you'll be able to use string_view. Peachy, or what?
>
I really like memory mapped files but under Windows I think the maximum size is just over 2GB, the largest XML file I have is about 4.5 GB, Postscript files could run over 8GB.

On the other hand memory mapped files are very flexible, fast and preferable when you have to access the file sequentially.

As I mentioned our method is very close to the one LCC compiler and RE2C
use.
this paper covers the method used in LCC
http://www.cs.princeton.edu/software/lcc/doc/06.pdf

Zz.

"Zz" <Zz@Zz.com> wrote in message news:d1kopq$1b6c$1@digitaldaemon.com...
> .. snip ..
>> Naturally, as you say, an opened file, in non-shared mode, takes away the 'theoretical' uncertainty I was gassing on about. But "rb" doesn't cut it. For one, "b" is a Microsoft / Windows extension. (On UNIX, there is no \r\n, so no need for \n <=> \r\n translation.)
>
> and don't forget \r for the mac.
>
> \r\n = Windows
> \n   = Unix
> \r    = Mac.
>
> While parsing we usually open files in binary and keep track of line endinds on our own.

Indeed!

I tend to, out of instinct, distrust stdio translation, and tend to cater for all possible permutations of line-ending sequences. Of course, one must always write out the appropriate thing on the appropriate platform, but reading 'intelligently' has saved me much hassle in the past.