Trying to speed up extracting some files that I first have to extract using the command line to files then read those in...

Not sure what is taking so long. I imagine windows caches the extraction so maybe it is pointless?

On Monday, 18 March 2019 at 21:09:55 UTC, Michelle Long wrote:
> Trying to speed up extracting some files that I first have to extract using the command line to files then read those in...
>
> Not sure what is taking so long. I imagine windows caches the extraction so maybe it is pointless?

You can speed up such things using a tmpfs or RAM disk.

On Linux you just use mount -t tmpfs ...

On Windows it is a little more complicated, there are programs like ImDisk that can create RAM drives (which can then be formatted to whatever).

If it's just one file, sometimes you can pipe it from the unpacking program (tar etc.) into the consuming program.

On Monday, 18 March 2019 at 21:14:05 UTC, Vladimir Panteleev wrote:
> On Monday, 18 March 2019 at 21:09:55 UTC, Michelle Long wrote:
>> Trying to speed up extracting some files that I first have to extract using the command line to files then read those in...
>>
>> Not sure what is taking so long. I imagine windows caches the extraction so maybe it is pointless?
>
> You can speed up such things using a tmpfs or RAM disk.
>
> On Linux you just use mount -t tmpfs ...
>
> On Windows it is a little more complicated, there are programs like ImDisk that can create RAM drives (which can then be formatted to whatever).
>
> If it's just one file, sometimes you can pipe it from the unpacking program (tar etc.) into the consuming program.

Yeah, but it seems like a lot of work simply to extract the files to memory.

On Mon, Mar 18, 2019 at 10:38:17PM +0000, Michelle Long via Digitalmars-d-learn wrote:
> On Monday, 18 March 2019 at 21:14:05 UTC, Vladimir Panteleev wrote:
> > On Monday, 18 March 2019 at 21:09:55 UTC, Michelle Long wrote:
> > > Trying to speed up extracting some files that I first have to extract using the command line to files then read those in...
> > > 
> > > Not sure what is taking so long. I imagine windows caches the extraction so maybe it is pointless?
[...]

Why not just use std.mmfile to memory-map the file into memory directly? Let the OS take care of actually paging in the file data.


T

-- 
Give me some fresh salted fish, please.

On Monday, 18 March 2019 at 23:01:27 UTC, H. S. Teoh wrote:
> On Mon, Mar 18, 2019 at 10:38:17PM +0000, Michelle Long via Digitalmars-d-learn wrote:
>> On Monday, 18 March 2019 at 21:14:05 UTC, Vladimir Panteleev wrote:
>> > On Monday, 18 March 2019 at 21:09:55 UTC, Michelle Long wrote:
>> > > Trying to speed up extracting some files that I first have to extract using the command line to files then read those in...
>> > > 
>> > > Not sure what is taking so long. I imagine windows caches the extraction so maybe it is pointless?
> [...]
>
> Why not just use std.mmfile to memory-map the file into memory directly? Let the OS take care of actually paging in the file data.
>
>
> T

The files are on disk and there is an external program that read them and converts them and then writes the converted files to disk then my program reads. Ideally the conversion program would take memory instead of disk files but it doesn't.

On Monday, 18 March 2019 at 23:40:02 UTC, Michelle Long wrote:
> On Monday, 18 March 2019 at 23:01:27 UTC, H. S. Teoh wrote:
>> On Mon, Mar 18, 2019 at 10:38:17PM +0000, Michelle Long via Digitalmars-d-learn wrote:
>>> On Monday, 18 March 2019 at 21:14:05 UTC, Vladimir Panteleev wrote:
>>> > On Monday, 18 March 2019 at 21:09:55 UTC, Michelle Long wrote:
>>> > > Trying to speed up extracting some files that I first have to extract using the command line to files then read those in...
>>> > > 
>>> > > Not sure what is taking so long. I imagine windows caches the extraction so maybe it is pointless?
>> [...]
>>
>> Why not just use std.mmfile to memory-map the file into memory directly? Let the OS take care of actually paging in the file data.
>>
>>
>> T
>
> The files are on disk and there is an external program that read them and converts them and then writes the converted files to disk then my program reads. Ideally the conversion program would take memory instead of disk files but it doesn't.

the file that was written by the first program will be in the file cache. mmap() (and the Windows equivalent of that) syscalls are at the core only giving access to the OS file cache. This means that std.mmfile is the way to go. There will be no reloading from disk if the file sizes are within reason.