I have written an article targeted at people new to D on compile-time programming: https://www.active-analytics.com/blog/reading-idx-files-in-d/ and tweeted it here: https://twitter.com/chibisi/status/1296824381088440320?s=20

Comments welcome.

Thanks in advance.

On Fri, Aug 21, 2020 at 03:04:30PM +0000, data pulverizer via Digitalmars-d-announce wrote:
> I have written an article targeted at people new to D on compile-time programming: https://www.active-analytics.com/blog/reading-idx-files-in-d/
[...]

CSS leakage into text in 2nd bullet point under "Introduction": "uspadding: 0.5em;s" should be "uses".


T

-- 
Shin: (n.) A device for finding furniture in the dark.

On Friday, 21 August 2020 at 15:54:14 UTC, H. S. Teoh wrote:
> CSS leakage into text in 2nd bullet point under "Introduction": "uspadding: 0.5em;s" should be "uses".

Thanks, just fixed it.

On Fri, Aug 21, 2020 at 08:54:14AM -0700, H. S. Teoh via Digitalmars-d-announce wrote:
> On Fri, Aug 21, 2020 at 03:04:30PM +0000, data pulverizer via Digitalmars-d-announce wrote:
> > I have written an article targeted at people new to D on compile-time programming: https://www.active-analytics.com/blog/reading-idx-files-in-d/
> [...]
> 
> CSS leakage into text in 2nd bullet point under "Introduction": "uspadding: 0.5em;s" should be "uses".
[...]

Anyway, besides that typo / formatting error, this is a very interesting idea to generate type declarations at compile-time based on external files.  I'm gonna hafta "steal" this idea for my own projects. ;-)


T

-- 
Dogs have owners ... cats have staff. -- Krista Casada

On 8/21/20 8:04 AM, data pulverizer wrote:
> I have written an article targeted at people new to D on compile-time programming: https://www.active-analytics.com/blog/reading-idx-files-in-d/ and tweeted it here: https://twitter.com/chibisi/status/1296824381088440320?s=20
> 
> Comments welcome.
> 
> Thanks in advance.

Great article. (There are other typos in there; I would be happy to review your next articles. :) )

I use "import expressions" for parsing files at compile time as well but your article gave me some ideas where I may be able to parse types of fields very similar to your example.

In my case I found a limitation: I cannot "iterate a directory" and import all file contents in there (the limitation is related to a C library function not having source code so it cannot be evaluated). So, I use a build step to generate the file that contains all files in my directory. So, I first import the file list then 'static foreach' that list to import and parse contents of other files.

For the record, here is an example of code that does not work at compile time:

import std.file;

void main() {
  static foreach (entry; dirEntries(".", SpanMode.shallow)) {
  }
}

Error: `fakePureCalloc` cannot be interpreted at compile time, because it has no available source code

(I think the error message is different in dmd 2.084 where my project currently uses.)

Ali

On Fri, Aug 21, 2020 at 01:18:30PM -0700, Ali Çehreli via Digitalmars-d-announce wrote: [...]
> In my case I found a limitation: I cannot "iterate a directory" and import all file contents in there (the limitation is related to a C library function not having source code so it cannot be evaluated).

The actual limitation is that string imports do not allow reading directory contents (the C function can be replaced if such were allowed).  Generally, I don't expect directory traversal to ever be allowed at compile-time, since it opens the door to a huge can o' security worms. :-P


> So, I use a build step to generate the file that contains all files in my directory. So, I first import the file list then 'static foreach' that list to import and parse contents of other files.

Yeah, that's probably the simplest solution.


[...]
> Error: `fakePureCalloc` cannot be interpreted at compile time, because it has no available source code
> 
> (I think the error message is different in dmd 2.084 where my project
> currently uses.)
[...]

fakePureCalloc is a red herring; even if it weren't a problem, you'd eventually run into the problem that you cannot do directory traversal at compile-time.


T

-- 
Time flies like an arrow. Fruit flies like a banana.

On 8/21/20 1:33 PM, H. S. Teoh wrote:

> On Fri, Aug 21, 2020 at 01:18:30PM -0700, Ali Çehreli via Digitalmars-d-announce wrote:

> Generally, I don't expect directory traversal to ever be
> allowed at compile-time, since it opens the door to a huge can o'
> security worms. :-P

I've heard that before. So, if directory traversal is only for import expressions then it should be fine because importing two files cannot be less safe than importing one file. So, we need to let the compiler know that we will import, nothing fancy:

  // Mapping from file name -> content
  string[string] contents = static import("my/dir");

Note the clever :o) use of 'static' which means "directory" in this case. :o) Now we have all contents as an AA, ready to be parsed, mixed-in, etc. But I this is not important at all. Build step is just fine.

Ali

On Friday, 21 August 2020 at 20:33:51 UTC, H. S. Teoh wrote:
> On Fri, Aug 21, 2020 at 01:18:30PM -0700, Ali Çehreli via Digitalmars-d-announce wrote: [...]
>> In my case I found a limitation: I cannot "iterate a directory" and import all file contents in there (the limitation is related to a C library function not having source code so it cannot be evaluated).
>
> The actual limitation is that string imports do not allow reading directory contents (the C function can be replaced if such were allowed).  Generally, I don't expect directory traversal to ever be allowed at compile-time, since it opens the door to a huge can o' security worms. :-P
>

I feel like limiting CTFE just gives a false sense of security and destroys many interesting use cases. If a part of my build system will do directory traversal to build the list of files to import, what difference would it make to not have this as a single build step. The argument that somehow

    dmd -run gen_code.d | dmd -

Is more secure than just:

    dmd file.d # file.d is allowed to access the FS at CT

makes no sense to me.

See Jai for example. You can run absolutely *any* code at compile time. 5 years ago Jai's creator made a demo of running an OpenGL game at CT [1]. In the same demo he also used CTFE to validate calls to printf. He made the case that while many compilers go the route of hard-coding checks for printf style functions in the compiler, he thinks that users should be able to implement arbitrary checks in their code. And 5 years later, instead of D expanding the frontiers of what's possible via CTFE, printf checking was hard coded in the compiler [2].

[1]: https://www.youtube.com/watch?v=UTqZNujQOlA
[2]: https://github.com/dlang/dmd/pull/10812/files

I don't need say that unlimited CTFE has been a huge success for Jai. What I wish is that we can learn from this stop bringing arguments that C people would bring for D's CTFE ("My Makefile calls a Python script to generate C code and it's doing just fine, so I don't think one should be allowed to run code at compile time, as it will make the code just harder to follow").

As another example, types in Zig are first class citizens [3] and can be manipulated with CTFE just like any other value. "Type functions" in D should just be regular D functions taking types as parameters and returning types.

[3]: https://ziglang.org/documentation/master/#Introducing-the-Compile-Time-Concept

On Friday, 21 August 2020 at 15:04:30 UTC, data pulverizer wrote:
> I have written an article targeted at people new to D on compile-time programming: https://www.active-analytics.com/blog/reading-idx-files-in-d/ and tweeted it here: https://twitter.com/chibisi/status/1296824381088440320?s=20
>
> Comments welcome.
.
> Thanks in advance.
Very interesting. +1 !

"we extract that an use it to" big error! D is missing :-)

On Saturday, 22 August 2020 at 07:04:19 UTC, Petar Kirov [ZombineDev] wrote:
> On Friday, 21 August 2020 at 20:33:51 UTC, H. S. Teoh wrote:
>
> I don't need say that unlimited CTFE has been a huge success for Jai.
Never heard of that success BTW. Probably a niche success.

But that aside, do you acknowledge there are problems with allowing such read access or not?

> What I wish is that we can learn from this stop bringing arguments that C people would bring for D's CTFE

Back in the days, I used to think if D had everything under the sun it would be a sole change factor until I saw (thought through) how C (and some technically meh ones) are still gaining some adoption despite not being even close to any what we have already.

By the way, aren't we already successful? Its about time we acknowledge how successful we already are.