Hi all,

the `import` function allows a file to be read at compile time, which opens up great opportunities for (mostly binary) file IO, where data types can be coded into files - the user doesn't need to know data types ahead of time. As specified in my old blog article: https://www.active-analytics.com/blog/reading-idx-files-in-d/.

Binary files can be very large and reading the whole file at compile time is often unnecessary.

This isn't exactly the intended use for the function, but there it is. Since compile time read capability already exists and is useful, adding capability to be able to read only a portion of the file at compile time in a given location is advantageous and has utility.

For me it's not make-or-break, it just something very useful and I think has clear use case. Please let me know if there are aspects or alternatives I am missing.

Thanks

On 2020-10-23 18:42, data pulverizer wrote:

> For me it's not make-or-break, it just something very useful and I think has clear use case. Please let me know if there are aspects or alternatives I am missing.

You could always have the build tool split up the file in multiple smaller files and read one of the smaller files with the import expression.

-- 
/Jacob Carlborg

On Sunday, 25 October 2020 at 12:16:36 UTC, Jacob Carlborg wrote:
> On 2020-10-23 18:42, data pulverizer wrote:
> You could always have the build tool split up the file in multiple smaller files and read one of the smaller files with the import expression.

Thanks. My current solution is to have a metadata file where all the type information is stored and then the data file separately.

I should have been clearer when I asked if there was something I was missing, like whether there is another way to do compile time read without reading the whole file. I think the feature I suggested is a useful one to bear in mind for future versions of D. It would save on preprocessing large files before reading them. I don't know how `import` for file reads works at the moment, and correct me if I'm wrong but it doesn't seem to me that it would be a big deal to make the change.

On Sunday, 25 October 2020 at 12:16:36 UTC, Jacob Carlborg wrote:
> On 2020-10-23 18:42, data pulverizer wrote:
>
>> For me it's not make-or-break, it just something very useful and I think has clear use case. Please let me know if there are aspects or alternatives I am missing.
>
> You could always have the build tool split up the file in multiple smaller files and read one of the smaller files with the import expression.

Which build tool are you refering to? an existing one or build one oneself to do this job?

On Friday, 23 October 2020 at 16:42:00 UTC, data pulverizer wrote:
> Hi all,
>
> the `import` function allows a file to be read at compile time, which opens up great opportunities for (mostly binary) file IO, where data types can be coded into files - the user doesn't need to know data types ahead of time. As specified in my old blog article: https://www.active-analytics.com/blog/reading-idx-files-in-d/.
>
> Binary files can be very large and reading the whole file at compile time is often unnecessary.
>
> This isn't exactly the intended use for the function, but there it is. Since compile time read capability already exists and is useful, adding capability to be able to read only a portion of the file at compile time in a given location is advantageous and has utility.
>
> For me it's not make-or-break, it just something very useful and I think has clear use case. Please let me know if there are aspects or alternatives I am missing.
>
> Thanks

That's a really good approach, I always wanted to do things like this in C# but that wasn't possible due to C#'s lack of CTFE. Could be good if we find a out a way to read part of the file and lazy-reading.Sure an external tool could do the first job, I mean if we could find a way in pure D.

On Sunday, 25 October 2020 at 16:50:09 UTC, Jack wrote:

> Which build tool are you refering to? an existing one or build one oneself to do this job?

It should work with any build tool that has hooks to execute arbitrary commands.

--
/Jacob Carlborg

On 10/23/20 12:42 PM, data pulverizer wrote:
> Hi all,
> 
> the `import` function allows a file to be read at compile time, which opens up great opportunities for (mostly binary) file IO, where data types can be coded into files - the user doesn't need to know data types ahead of time. As specified in my old blog article: https://www.active-analytics.com/blog/reading-idx-files-in-d/.
> 
> Binary files can be very large and reading the whole file at compile time is often unnecessary.
> 
> This isn't exactly the intended use for the function, but there it is. Since compile time read capability already exists and is useful, adding capability to be able to read only a portion of the file at compile time in a given location is advantageous and has utility.
> 
> For me it's not make-or-break, it just something very useful and I think has clear use case. Please let me know if there are aspects or alternatives I am missing.

I see no reason why the compiler can't do something special with:

enum slicedFile = import("foo.bin")[0 .. 100];

One thing that comes to mind -- most OSes support memory-mapped files. If import simply did a memory mapped file internally, and duplicated the string once actually used somewhere, then you can avoid full reading of the data until it's necessary.

-Steve