Arcane Jill wrote:
> In article <cbpp7k$1pgi$1@digitaldaemon.com>, Hauke Duden says...
> 
> 
>>You should also keep in mind that the executable is held in RAM as well, 
> 
> 
> As well as what?

As well as the data you store in explicitly allocated memory.

> The tables are directly contained in the RAM image of the executable. They are
> not duplicated or otherwise reconstructed. They are accessed in-place.

Yes, but they ARE in RAM. My point was that you don't save RAM if you put the data in the executable instead of an explicitly allocated memory block.

>>so increasing executable size to save RAM does not always give you an advantage.
> 
> 
> Curiously, you seem to be arguing in favor of my position. Had we used RLE
> decompression, THEN we'd have to worry about the "as well".

I don't think I understand what you mean. If I understood your last post correctly you didn't want to use RLE compression because disk space is cheap, but RAM is not. I am arguing that:

- executable size (=disk space) is more expensive than RAM if the file is downloaded from the internet

- RAM usage is increased only slightly (my rough estimate was 125% of the original space) but executable size is reduced significantly (down to 25%).

In an age where many programs are downloaded from the internet that is worth thinking about.

Hauke

In article <cbrvnp$1uf8$1@digitaldaemon.com>, Hauke Duden says...
>
>>>You should also keep in mind that the executable is held in RAM as well,
>> 
>> As well as what?
>
>As well as the data you store in explicitly allocated memory.

That would be zero.


>> The tables are directly contained in the RAM image of the executable. They are not duplicated or otherwise reconstructed. They are accessed in-place.
>
>Yes, but they ARE in RAM. My point was that you don't save RAM if you put the data in the executable instead of an explicitly allocated memory block.

But clearly you do. If the compressed size is X, and the uncompressed size is Y, then storing the uncompressed table in the executable costs Y bytes of RAM. Decompressing at runtime costs (X+Y) bytes of RAM, since you can't un-allocate the X. Since X is not negative, it follows that Y will always be less than (X+Y)



>I don't think I understand what you mean. If I understood your last post correctly you didn't want to use RLE compression because disk space is cheap, but RAM is not. I am arguing that:
>
>- executable size (=disk space) is more expensive than RAM if the file is downloaded from the internet

That's what zip files are for.

Besides which, I don't think my obj files actually do contain large arrays full of zeroes. Such zero blocks will all have been removed and replaced by null pointer returns. Or were you arguing that zero-blocks should be re-inserted?



>- RAM usage is increased only slightly (my rough estimate was 125% of the original space) but executable size is reduced significantly (down to 25%).

I'm not arguing with that. I'm arguing in favor of not increasing RAM usage *AT ALL*. Like, not even slightly.


>In an age where many programs are downloaded from the internet that is worth thinking about.

Zip files use much better compression than simple RLE. I say zip 'em.

Jill

Arcane Jill wrote:
> In article <cbrvnp$1uf8$1@digitaldaemon.com>, Hauke Duden says...
> 
>>>>You should also keep in mind that the executable is held in RAM as well, 
>>>
>>>As well as what?
>>
>>As well as the data you store in explicitly allocated memory.
> 
> 
> That would be zero.
> 
> 
> 
>>>The tables are directly contained in the RAM image of the executable. They are
>>>not duplicated or otherwise reconstructed. They are accessed in-place.
>>
>>Yes, but they ARE in RAM. My point was that you don't save RAM if you put the data in the executable instead of an explicitly allocated memory block.
> 
> 
> But clearly you do. If the compressed size is X, and the uncompressed size is Y,
> then storing the uncompressed table in the executable costs Y bytes of RAM.
> Decompressing at runtime costs (X+Y) bytes of RAM, since you can't un-allocate
> the X. Since X is not negative, it follows that Y will always be less than (X+Y)

In this particular case, yes, as I stated in my post. I just wanted to emphasize that moving data into statically compiled arrays (as opposed to dynamic ones) doesn't automatically reduce RAM usage.

>>I don't think I understand what you mean. If I understood your last post correctly you didn't want to use RLE compression because disk space is cheap, but RAM is not. I am arguing that:
>>
>>- executable size (=disk space) is more expensive than RAM if the file is downloaded from the internet
> 
> 
> That's what zip files are for.

What about installers and self-extractors? You do not ZIP those because they ARE the ZIP file (in a manner of speaking). I'd like to be able to write such applications in D.

Besides half of my reasons is that even if D executables could be compressed better than C++ ones, many people would still compare their size in uncompressed form. A similar thing happened to C++: C++ executables usually compress better than C ones (templates and exception handling create lots of similar code), yet C++ is often said to be the "bloat king" among languages.

I just don't want people to shun the Unicode routines because of the size difference, even if it may not have such a big impact on the end result as they might think.


> Besides which, I don't think my obj files actually do contain large arrays full
> of zeroes. Such zero blocks will all have been removed and replaced by null
> pointer returns. Or were you arguing that zero-blocks should be re-inserted?

RLE doesn't just pack zero arrays. Unicode contains lots of ranges with the same values.


>>- RAM usage is increased only slightly (my rough estimate was 125% of the original space) but executable size is reduced significantly (down to 25%).
> 
> 
> I'm not arguing with that. I'm arguing in favor of not increasing RAM usage *AT
> ALL*. Like, not even slightly.

As I said, I think 100 KB of extra RAM usage is a lot better than 400 KB of increased executable size. Especially for a garbage collected language that will always use more RAM than strictly necessary.

Hauke

"Hauke Duden" <H.NS.Duden@gmx.net> skrev i en meddelelse news:cbrvnp$1uf8> > The tables are directly contained in the RAM image of the executable. They are
> > not duplicated or otherwise reconstructed. They are accessed in-place.
> Yes, but they ARE in RAM. My point was that you don't save RAM if you put the data in the executable instead of an explicitly allocated memory block.

Are you sure about that? I would expect individual pages to be loaded on demand.

Regards,
Martin

Hi, this is really impressive! (Okay, so I'm only using isWhiteSpace and  simple folding/casing atm, I'll learn what the others do later ;-)
The unicode stuff isn't in the library in subversion AFAICS though.
(Actually, I can't get subversion to check out properly, I've been using the HTTP gateway. I tried http://svn.dsource.org/svn/projects/deimos as the location in TortoiseSVN, does that look right?)
Sam

In article <cbu5r3$2kpj$1@digitaldaemon.com>, Sam McCall says...
>
>Hi, this is really impressive! (Okay, so I'm only using isWhiteSpace and
>  simple folding/casing atm, I'll learn what the others do later ;-)
>The unicode stuff isn't in the library in subversion AFAICS though.

It is, kindof. At least the source code is there, at http://svn.dsource.org/svn/projects/deimos/trunk/etc/unicode/.

But what we really NEED is a downloadable pre-built library. That's the part that's currently missing.


>(Actually, I can't get subversion to check out properly, I've been using
>the HTTP gateway. I tried http://svn.dsource.org/svn/projects/deimos as
>the location in TortoiseSVN, does that look right?)
>Sam

Er - I hope someone else can answer that...?

Jill

Martin M. Pedersen wrote:
> "Hauke Duden" <H.NS.Duden@gmx.net> skrev i en meddelelse news:cbrvnp$1uf8> >
> The tables are directly contained in the RAM image of the executable. They
> are
> 
>>>not duplicated or otherwise reconstructed. They are accessed in-place.
>>
>>Yes, but they ARE in RAM. My point was that you don't save RAM if you
>>put the data in the executable instead of an explicitly allocated memory
>>block.
> 
> 
> Are you sure about that? I would expect individual pages to be loaded on
> demand.

I'm pretty sure about it, but not 100% sure. I have frequently observed in the past that RAM usage increases by the size of a DLL as soon as it is loaded.

Hauke

In article <cbsj6s$2s9m$1@digitaldaemon.com>, Martin M. Pedersen says...
>
>"Hauke Duden" <H.NS.Duden@gmx.net> skrev i en meddelelse news:cbrvnp$1uf8> > The tables are directly contained in the RAM image of the executable. They are
>> > not duplicated or otherwise reconstructed. They are accessed in-place.
>> Yes, but they ARE in RAM. My point was that you don't save RAM if you put the data in the executable instead of an explicitly allocated memory block.
>
>Are you sure about that? I would expect individual pages to be loaded on demand.
>
>Regards,
>Martin


Once upon a time, I did just that. I had to write some Unicode stuff for my employer a few years back, and I adopted exactly that "load on demand" approach. It is with this hindsight that I now beleive it to have been a bad idea (although with some modification, it might be a good idea in a DLL).

The basic load-on-demand approach is this: The user calls something like isUppercase(c); The library evaluates (c >> N) (for some N) to get what we might loosely call a "page" number. Then it says to itself, "Is this page cached?". If so, use the in-RAM table to look up the answer; if not, load the page from disk, decompress it into RAM, cache it so we don't have to do all that again, and THEN look up the value. Plus, you'd have to do this every time you re-ran your application, which might matter for some very small applications.

At the time, it seemed like this was quite a promising approach, but it had a huge number of drawbacks. For one thing, neither C++ nor D has any concept of resource files (essentially a Java concept), so, in order to load anything off disk, YOU FIRST HAVE TO FIND IT. This means either a DLL (per "page"? per property?), or you have to read an environment variable to tell you where to look. Requiring users to set an environment variable just to get isUppercase() working is not desirable. For another thing, the extra code you have to go through at runtime to answer the question "is it cached?" is itself a few extra cycles.

I took a different approach this time round. In this new approach, there are two important principles: (1) The souce code shall be written by robot. This protects us against Unicode itself being updated (and it is /constantly/ being updated), and it also allows for some SERIOUS optimization, because of course a robot can try many, many different optimization strategies and, using sheer brute force, pick the best. (2) Each property shall be in its own object module, so that you do not link in those properties which you do not need. This is load-on-demand in a sense, but it's COMPILE-TIME load-on-demand, which is better (in my opinion) and the split is in a different direction (property-required rather than codepoint-range).

The new approach makes complete sense when you realize just how small things get. That isUppercase() function generates a linkable object module which is a measly 3420 /bytes/ in size, and does not depend on (pull in) anything else. I mean - come on guys - just 3K! Are we REALLY saying that's too much? Plus, as a bonus, you get isUppercase() data for all characters, so no there is no bias toward any particular subrange.

And in any case, I didn't get that 3K figure by measuring the size of the generated tables, I got it by compiling an object module and typing dir at the command line to see how big it ended up. For reference, the following do-nothing functions:

#    module etc.unicode.current_version.always_true;
#
#    bool isAlwaysTrue(dchar c)
#    {
#        return true;
#    }

by the same standard comes out at 232 bytes. However, this measure overestimates, because, even if it isn't inlined, there are some linker symbols in there that will be discarded when constructing an executable. So even these small estimates are overestimates. I think you would be hard-pressed to do better than my robot.

The good news, however, is that nobody has to agree with me. (I guess it was too much to hope for that everyone would). Because, pretty soon, I'm going to make the codebuilder robot open source, once I've added a few more tweaks and got a few other things sorted out. That means that if anyone can come up with a better strategy than I, they would be perfectly welcome to take a branch of the codebuilder source tree and modify it to do something else. Then we could run various efficiency tests to compare all versions. You could do this to see if your load-on-demand-by-codepoint-range idea was feasable; Hauke could do it to see if his RLE encoding idea works out better than what I've done, and without doubt, the most efficient one is the one we'll keep (though I'm not sure how you define efficient).

In any case, I'm putting my next efforts into (a) fixing some bugs in Int, and (b) making Deimos easy to download and use. This sort of feature modification you suggest is not on my agenda in the near future, because, although there ARE a few more functions I need to add to etc.unicode., I've basically achieved what I set out to achieve, and I'm happy with it, and pretty soon I'm going to be keen to get back to my crypto stuff.

I hope that helps.

Arcane Jill

>

Arcane Jill wrote:
> In article <cbu5r3$2kpj$1@digitaldaemon.com>, Sam McCall says...
> 
>>Hi, this is really impressive! (Okay, so I'm only using isWhiteSpace and 
>> simple folding/casing atm, I'll learn what the others do later ;-)
>>The unicode stuff isn't in the library in subversion AFAICS though.
> 
> 
> It is, kindof. At least the source code is there, at
> http://svn.dsource.org/svn/projects/deimos/trunk/etc/unicode/.
> 
> But what we really NEED is a downloadable pre-built library. That's the part
> that's currently missing.
Sorry, yeah, that's what I meant.
I built one here with (from trunk/)
for /R etc %f in (*.d) dmd -c -release %f
lib -c deimos.lib age.obj
for %f in (*.obj) lib deimos.lib %f

(I'm not at all familiar with this stuff, so this may well be the Wrong Way, in particular the age.obj thing is a hack because i can't seem to get the lib tool to add an obj, creating the library if it doesn't exist).
Then I hit the problem that I couldn't use most of the functions, due to unknown symbol. Sure enough, most of the functions weren't in the library. I changed the ones I wanted to use to "public" in the source, and that worked. I'm not sure if this is the right fix.
Sam

> 
> 
> 
>>(Actually, I can't get subversion to check out properly, I've been using the HTTP gateway. I tried http://svn.dsource.org/svn/projects/deimos as the location in TortoiseSVN, does that look right?)
>>Sam
> 
> 
> Er - I hope someone else can answer that...?
> 
> Jill
> 
> 

"Arcane Jill" <Arcane_member@pathlink.com> skrev i en meddelelse news:cbu6ru$2nm4$1@digitaldaemon.com...
> >Are you sure about that? I would expect individual pages to be loaded on demand.
> Once upon a time, I did just that. I had to write some Unicode stuff for
my
> employer a few years back, and I adopted exactly that "load on demand"
approach.
> It is with this hindsight that I now beleive it to have been a bad idea (although with some modification, it might be a good idea in a DLL).

What I meant was load-on-demand implemented by the operating system. I think you have done a great job, and also in this respect made the right decision :-) When a modern operating system starts executing a program, it does not actually load the program into RAM. Instead, it sets up page tables, and uses the page-fault mechanism of the CPU to implement load-on-demand for the code segment. For example, when the very first instruction of the program is to be executed, it generates a page-fault, and it is at this time, the very first page is loaded into RAM. At least, this is my understanding. Static, constant tables like the ones in the Unicode library, would be - or can be - embraced by the same mechanism, meaning that you only pay for what you use. A decompression scheme at application level would mean that you would always pay for everything. It is better to leave that kind of stuff to the operating system, and mechanisms for doing that has existed for a long time, although they might not be universally available. NTFS has built-in support for LZW-compression, and it was a long time since "Stacker" was invented. For distribution, we have de-facto standards such as zip.

Regards,
Martin