I like to put some data into a binary
that should reside on disk even when the program
is started.

The data should be loaded in memory on request of the running part.

The aim is to get rid of zip file distributions:
- need to be extracted
- files need to be keep together

Is that possible with D?

On Wed, Mar 12, 2008 at 11:21:27PM +0000, Moritz Warning wrote:
> Is that possible with D?

I'm pretty sure this is an operating system thing. I don't think it can be done on Linux, but on Windows, you might be able to do it with resources embedded in the executable, just like you would with C programs.


-- 
Adam D. Ruppe
http://arsdnet.net

On Wed, 12 Mar 2008 19:33:29 -0400, Adam D. Ruppe wrote:

> On Wed, Mar 12, 2008 at 11:21:27PM +0000, Moritz Warning wrote:
>> Is that possible with D?
> 
> I'm pretty sure this is an operating system thing. I don't think it can be done on Linux, but on Windows, you might be able to do it with resources embedded in the executable, just like you would with C programs.

Would be a neat feature if it would be possible (on Linux).
The problem is to load only a part of the binary without the data payload.
Loading data from the binary into RAM would be less of a problem.

== Quote from Moritz Warning (moritzwarning@_nospam_web.de)'s article
> On Wed, 12 Mar 2008 19:33:29 -0400, Adam D. Ruppe wrote:
> > On Wed, Mar 12, 2008 at 11:21:27PM +0000, Moritz Warning wrote:
> >> Is that possible with D?
> >
> > I'm pretty sure this is an operating system thing. I don't think it can be done on Linux, but on Windows, you might be able to do it with resources embedded in the executable, just like you would with C programs.
> Would be a neat feature if it would be possible (on Linux).
> The problem is to load only a part of the binary without the data payload.
> Loading data from the binary into RAM would be less of a problem.

Does the full binary of an application get loaded when the app executes?  I was under the assumption that code pages were loaded as needed.


Sean

On Wed, Mar 12, 2008 at 11:59 PM, Moritz Warning <moritzwarning@_nospam_web.de> wrote:
> On Wed, 12 Mar 2008 19:33:29 -0400, Adam D. Ruppe wrote:
>
>  > On Wed, Mar 12, 2008 at 11:21:27PM +0000, Moritz Warning wrote:
>
> >> Is that possible with D?
>  >
>
> > I'm pretty sure this is an operating system thing. I don't think it can
>  > be done on Linux, but on Windows, you might be able to do it with
>  > resources embedded in the executable, just like you would with C
>  > programs.
>
>  Would be a neat feature if it would be possible (on Linux).
>  The problem is to load only a part of the binary without the data payload.
>  Loading data from the binary into RAM would be less of a problem.
>

One way is to simply concatenate the data onto the end of the binary, and write a function to open a standard stream on the file and seek to the start of the data. For example:

my.exe:
  - <PE data>
  - <ZIP file>
  - <uint32 offset to first byte of zip file>

All a function needs to do is open and read the last 4 bytes, seek to that position, and test for a zip signature. I'm pretty sure this works for both PE and ELF, it's the way the Winzip self extractor works, and some equivalent tools on Linux (I think info-zip provides something similar).

On Thu, Mar 13, 2008 at 12:44:46AM +0000, Sean Kelly wrote:
> Does the full binary of an application get loaded when the app executes? I was under the assumption that code pages were loaded as needed.
>
> Sean

Yes, it looks like you're right according to a web search. So it should be possible on Linux - using import()'s seems to me that it would be the easiest way.

How it might be done in D:

-------------

import std.stdio;
string hugeFile = import("hugefile.txt");
	// If a reader is new to D, the above loads "hugefile.txt" at compile
	// time, as if I wrote:
	// string hugeFile = "the literal contents of hugefile.txt here";

void main(){
	writefln("...");
	readln();
	writefln("%s", hugeFile);
}

-------------

And to see how it works, watch for disk activity before and after the readln. But, I'm foiled in testing the efficiency of this on the disk by the fact that when making the executable, Linux caches the whole thing in RAM, so it doesn't need to hit the disk anyway.

Note that the import used there doesn't work for absolutely huge files. It works for anything less than at least 5 MB though, and probably more, which might be enough to do the job wanted.

It's worth a try.

-- 
Adam D. Ruppe
http://arsdnet.net

Reply to Moritz,

> I like to put some data into a binary
> that should reside on disk even when the program
> is started.
> The data should be loaded in memory on request of the running part.
> 
> The aim is to get rid of zip file distributions:
> - need to be extracted
> - files need to be keep together
> Is that possible with D?
> 

I think most OS's map in the whole program but only load stuff on demand using the page swapping and virtual memeory systems.

Thanks for all the helpful replies!

BCS wrote:
> Reply to Moritz,
> 
>> I like to put some data into a binary
>> that should reside on disk even when the program
>> is started.
>> The data should be loaded in memory on request of the running part.
>>
>> The aim is to get rid of zip file distributions:
>> - need to be extracted
>> - files need to be keep together
>> Is that possible with D?
>>
> 
> I think most OS's map in the whole program but only load stuff on demand using the page swapping and virtual memeory systems.

Typically not even the whole binary file isn't even mapped, only certain segments are.  For example, the debug symbol segment is not mapped into memory.  For the parts that are, every unix-like and windows flavor in use today does demand paging of both the executables as well as any loaded libraries (be they .so or .dll as appropriate).

Later,
Brad

On DOS, Windows9x I did this very often:

* build the exe, note its final size;
* keep this exe size in some global constant;
* rebuild the exe, make sure the size is the same;
* append other data to the exe (DOS: copy /b foo.exe+my.dat final.exe)

In code you can then simply open the exe as any other file and skip to the actual data.

I'm not 100% sure whether this trick still works on XP+ or linux, but I think it should work, with the right combination of file-open/sharding rights.

L.

----- Original Message ----- From: "Moritz Warning" <moritzwarning@_nospam_web.de>
Newsgroups: digitalmars.D
Sent: Thursday, March 13, 2008 7:21 AM
Subject: start partial binary


>I like to put some data into a binary
> that should reside on disk even when the program
> is started.
>
> The data should be loaded in memory on request of the running part.
>
> The aim is to get rid of zip file distributions:
> - need to be extracted
> - files need to be keep together
>
> Is that possible with D?