On 2/06/2015 8:54 p.m., ketmar wrote:
> On Mon, 01 Jun 2015 16:05:17 -0400, Steven Schveighoffer wrote:
>
>> A protocol to be used between dmd and dub (or any other package fetcher)
>> would be a good first step.
>
> that was the thing i once proposed. see, we have a powerful scripting
> language inside DMD: D! yet we never used all it's power to do something
> really exciting -- like, for example, preparing command lines for
> external package fetching tool and parsing the answers. instead of
> providing a simple module for that, it all goes to be hardcoded.
>
> i extended CTFE engine with simple file i/o functions and "system" call
> with config-defined executables (something like sudoers list) just to see
> if it will be usable. and it's surprisingly fun even with all the limits
> and without hooks for module imports and so on.
>
> this system can be extended to allow writing arbitrary subcommands (like
> git). just import the corresponding subcommand module, if any, and CTFE
> it's invocation point. bingo! the system that can be extended without
> recompiling the compiler. and user can add subcommands on per-project
> base!
>
> sadly, this seems to get no "wow!"s. :-(

I say wow.

Okay not really I know it is possible and not all the hard.
Personally I think it is a rather an awesome possibility.

Really what I want is to add the ability to add on the compiler side symbols that can be CTFE'd.
Registered at start of runtime.

Add on shared library support and wamo. You can add stuff like this pretty arbitrary :)

On Tuesday, 2 June 2015 at 08:54:46 UTC, ketmar wrote:
> On Mon, 01 Jun 2015 16:05:17 -0400, Steven Schveighoffer wrote:
>
>> A protocol to be used between dmd and dub (or any other package fetcher)
>> would be a good first step.
>
> that was the thing i once proposed. see, we have a powerful scripting
> language inside DMD: D! yet we never used all it's power to do something
> really exciting -- like, for example, preparing command lines for
> external package fetching tool and parsing the answers. instead of
> providing a simple module for that, it all goes to be hardcoded.

https://github.com/atilaneves/reggae

Atila

On Tue, 02 Jun 2015 21:12:14 +1200, Rikki Cattermole wrote:

> I say wow.
> 
> Okay not really I know it is possible and not all the hard. Personally I think it is a rather an awesome possibility.

adding some CTFE functions is not that hard at all. it's not documented, but one can look at "import()" and "pragma()" to see how they're doing their things and do the same. that's what i did, and it took me one day to write a working PoC.

i promised to make a series of articles on that for TWiD, but still didn't done that. mea maxima culpa.


> Really what I want is to add the ability to add on the compiler side
> symbols that can be CTFE'd.
> Registered at start of runtime.

i'm afraid i didn't get it. can you provide a sample?


> Add on shared library support and wamo. You can add stuff like this pretty arbitrary :)

you mean adding symbols with plugin-like system? i'm not sure that it will be easy. anyway, with well-defined CTFE API to access file system and execute commands (with restrictions, of course, so D programs will not go wild doing "rm -rf ~/" ;-) one can write modules that contains functions that acts like "filters", executing external binaries and massaging input/output.

with some hooks added (like "call this CTFE function (if it present) when module importing fails") and ability to add -I/-J pathes in that hooks (restricted to some "library root dir") one can write "CTFE rdmd with automatic package downloading".

i think i have to make a draft and PoC for that, so people can try it and see how k00l it is. ;-)

On Tue, 02 Jun 2015 09:15:12 +0000, Atila Neves wrote:

> On Tuesday, 2 June 2015 at 08:54:46 UTC, ketmar wrote:
>> On Mon, 01 Jun 2015 16:05:17 -0400, Steven Schveighoffer wrote:
>>
>>> A protocol to be used between dmd and dub (or any other package
>>> fetcher)
>>> would be a good first step.
>>
>> that was the thing i once proposed. see, we have a powerful scripting language inside DMD: D! yet we never used all it's power to do something really exciting -- like, for example, preparing command lines for external package fetching tool and parsing the answers. instead of providing a simple module for that, it all goes to be hardcoded.
> 
> https://github.com/atilaneves/reggae
> 
> Atila

hm. looks interesting, albeit a little too noisy for my taste.

On 2/06/2015 9:32 p.m., ketmar wrote:
> On Tue, 02 Jun 2015 21:12:14 +1200, Rikki Cattermole wrote:
>
>> I say wow.
>>
>> Okay not really I know it is possible and not all the hard. Personally I
>> think it is a rather an awesome possibility.
>
> adding some CTFE functions is not that hard at all. it's not documented,
> but one can look at "import()" and "pragma()" to see how they're doing
> their things and do the same. that's what i did, and it took me one day
> to write a working PoC.
>
> i promised to make a series of articles on that for TWiD, but still
> didn't done that. mea maxima culpa.
>
>
>> Really what I want is to add the ability to add on the compiler side
>> symbols that can be CTFE'd.
>> Registered at start of runtime.
>
> i'm afraid i didn't get it. can you provide a sample?
>
>
>> Add on shared library support and wamo. You can add stuff like this
>> pretty arbitrary :)
>
> you mean adding symbols with plugin-like system? i'm not sure that it
> will be easy. anyway, with well-defined CTFE API to access file system
> and execute commands (with restrictions, of course, so D programs will
> not go wild doing "rm -rf ~/" ;-) one can write modules that contains
> functions that acts like "filters", executing external binaries and
> massaging input/output.
>
> with some hooks added (like "call this CTFE function (if it present) when
> module importing fails") and ability to add -I/-J pathes in that hooks
> (restricted to some "library root dir") one can write "CTFE rdmd with
> automatic package downloading".
>
> i think i have to make a draft and PoC for that, so people can try it and
> see how k00l it is. ;-)
>

Essentially its compiler plugins that can add news types + free-functions as if it was D code provided by source code only also able to modify e.g. AST directly.
So while in of itself would not provide new language features, it could add some very nice behavior to existing ones.

So what you want with commands ext. would be done in a shared library. It have e.g. a struct added into object.d(i).

__CEIFileSystem.delete("./tmp/something");

Of course adding these plugins should be pretty explicit. Package maintainers for example shouldn't auto install any. There should be no way to auto install them.

Basically dmd-plugin-dub-bin would be an example package. That would add dub support directly into dmd.

----------
@__CEIDub.Dependency("vibe-d", ">=0.7.22")
void main() {
	// what have you
}
----------

Or:

----------
static assert(__CEIDub.dependency("vibe.d", ">=0.7.22"), "Requires vibe-d 0.7.22 or newer.");

void main() {
	// what have you
}
----------

That way it can be used for e.g. static-if and template if.

Of course I'm sure Walter would go nuts at such an idea, if it was seriously proposed.
So perhaps dmd doesn't support it. Instead a new version of dmd is also shipped (dmdext) with this enabled ;)

With the community agreeing that we should try to keep as close to dmd as possible and only reverting to dmdext + plugins to help newbies and where it just wouldn't be possible to not have an external tool. Which most likely uses multiple compilation or something else not so nice.

On Tue, 02 Jun 2015 21:51:54 +1200, Rikki Cattermole wrote:

> Essentially its compiler plugins that can add news types +
> free-functions as if it was D code provided by source code only also
> able to modify e.g. AST directly.
> So while in of itself would not provide new language features, it could
> add some very nice behavior to existing ones.
> 
> So what you want with commands ext. would be done in a shared library. It have e.g. a struct added into object.d(i).
> 
> __CEIFileSystem.delete("./tmp/something");
> 
> Of course adding these plugins should be pretty explicit. Package maintainers for example shouldn't auto install any. There should be no way to auto install them.
> 
> Basically dmd-plugin-dub-bin would be an example package. That would add dub support directly into dmd.
> 
> ----------
> @__CEIDub.Dependency("vibe-d", ">=0.7.22")
> void main() {
> 	// what have you
> }
> ----------
> 
> Or:
> 
> ----------
> static assert(__CEIDub.dependency("vibe.d", ">=0.7.22"), "Requires
> vibe-d 0.7.22 or newer.");
> 
> void main() {
> 	// what have you
> }
> ----------
> 
> That way it can be used for e.g. static-if and template if.
> 
> Of course I'm sure Walter would go nuts at such an idea, if it was
> seriously proposed.
> So perhaps dmd doesn't support it. Instead a new version of dmd is also
> shipped (dmdext) with this enabled ;)
> 
> With the community agreeing that we should try to keep as close to dmd as possible and only reverting to dmdext + plugins to help newbies and where it just wouldn't be possible to not have an external tool. Which most likely uses multiple compilation or something else not so nice.

actually, that should be doable with DDMD on non-windows systems*, i believe. DDMD -- to avoid writing plugins in other languages. ;-) and with DDMD plugins will have full access to frontend internals, including AST manipulation.

* non-windows, as dll support on windows is still not here, i believe.


the downside of this approach is that it's highly compiler-specific. i.e. one will force to have different plugins for GDC/LDC/SDC/etc. ;-) and with pure CTFE API without plugins, only with external executables, it will be compiler-agnostic, any compiler using DMDFE will get it for free. only SDC will be forced to rewrite the CTFE part, but other than that all CTFE D code will work the same.

i mean, 'cmon, we have such great built-in scripting engine, let's use it!

your samples can be built ontop of my "subcommand" proposal, for example. let's say that compiler will try to automatically do static import "DMD.subcommands.dub" when programmer writing something like `DMD.dub.Dependency("vibe-d", ">=0.7.22");`, falling back to "DMD.failure (name, args)" if there is no "dub" subcommand package.

and then subcommand can execute dub, parse it's output and do what it wants. or one can make a local override for subcommand -- without even compiling separate plugin.

that's how i see it.

On 2/06/2015 10:15 p.m., ketmar wrote:
> On Tue, 02 Jun 2015 21:51:54 +1200, Rikki Cattermole wrote:
>
>> Essentially its compiler plugins that can add news types +
>> free-functions as if it was D code provided by source code only also
>> able to modify e.g. AST directly.
>> So while in of itself would not provide new language features, it could
>> add some very nice behavior to existing ones.
>>
>> So what you want with commands ext. would be done in a shared library.
>> It have e.g. a struct added into object.d(i).
>>
>> __CEIFileSystem.delete("./tmp/something");
>>
>> Of course adding these plugins should be pretty explicit. Package
>> maintainers for example shouldn't auto install any. There should be no
>> way to auto install them.
>>
>> Basically dmd-plugin-dub-bin would be an example package. That would add
>> dub support directly into dmd.
>>
>> ----------
>> @__CEIDub.Dependency("vibe-d", ">=0.7.22")
>> void main() {
>> 	// what have you
>> }
>> ----------
>>
>> Or:
>>
>> ----------
>> static assert(__CEIDub.dependency("vibe.d", ">=0.7.22"), "Requires
>> vibe-d 0.7.22 or newer.");
>>
>> void main() {
>> 	// what have you
>> }
>> ----------
>>
>> That way it can be used for e.g. static-if and template if.
>>
>> Of course I'm sure Walter would go nuts at such an idea, if it was
>> seriously proposed.
>> So perhaps dmd doesn't support it. Instead a new version of dmd is also
>> shipped (dmdext) with this enabled ;)
>>
>> With the community agreeing that we should try to keep as close to dmd
>> as possible and only reverting to dmdext + plugins to help newbies and
>> where it just wouldn't be possible to not have an external tool. Which
>> most likely uses multiple compilation or something else not so nice.
>
> actually, that should be doable with DDMD on non-windows systems*, i
> believe. DDMD -- to avoid writing plugins in other languages. ;-) and
> with DDMD plugins will have full access to frontend internals, including
> AST manipulation.
>
> * non-windows, as dll support on windows is still not here, i believe.

I've been able to do it a little bit already with my Web server. Haven't hit its limit so far. In other words as long as exceptions aren't used and its never unloaded, should be ok. Anyway, do you really think we would get that sort of support for a couple of releases?

> the downside of this approach is that it's highly compiler-specific. i.e.
> one will force to have different plugins for GDC/LDC/SDC/etc. ;-) and
> with pure CTFE API without plugins, only with external executables, it
> will be compiler-agnostic, any compiler using DMDFE will get it for free.
> only SDC will be forced to rewrite the CTFE part, but other than that all
> CTFE D code will work the same.

Yeah it is, but it would also mean better customization of the build process!

> i mean, 'cmon, we have such great built-in scripting engine, let's use it!

Indeed lets. But can we also not go modifying the language? I'm just suggesting injecting of symbols that execute compiler side actions. Your suggesting language changes. Even if it is a new pragma.

> your samples can be built ontop of my "subcommand" proposal, for example.
> let's say that compiler will try to automatically do static import
> "DMD.subcommands.dub" when programmer writing something like
> `DMD.dub.Dependency("vibe-d", ">=0.7.22");`, falling back to "DMD.failure
> (name, args)" if there is no "dub" subcommand package.
>
> and then subcommand can execute dub, parse it's output and do what it
> wants. or one can make a local override for subcommand -- without even
> compiling separate plugin.
>
> that's how i see it.

We really need to toy with these ideas properly and implement each. Then its just a matter of time to convince the higher ups that it should be merged.

On Tuesday, 2 June 2015 at 09:33:24 UTC, ketmar wrote:
> On Tue, 02 Jun 2015 09:15:12 +0000, Atila Neves wrote:
>
>> On Tuesday, 2 June 2015 at 08:54:46 UTC, ketmar wrote:
>>> On Mon, 01 Jun 2015 16:05:17 -0400, Steven Schveighoffer wrote:
>>>
>>>> A protocol to be used between dmd and dub (or any other package
>>>> fetcher)
>>>> would be a good first step.
>>>
>>> that was the thing i once proposed. see, we have a powerful scripting
>>> language inside DMD: D! yet we never used all it's power to do
>>> something really exciting -- like, for example, preparing command lines
>>> for external package fetching tool and parsing the answers. instead of
>>> providing a simple module for that, it all goes to be hardcoded.
>> 
>> https://github.com/atilaneves/reggae
>> 
>> Atila
>
> hm. looks interesting, albeit a little too noisy for my taste.

It depends on what you want to do: the idea is to use high-level utility functions as much as possible, while at the same time offering low-level primitives that those high-level ones are based on.

If it can be made simpler, I'd really like to know.

Atila

On Tue, 02 Jun 2015 12:12:06 +0000, Atila Neves wrote:

>>> https://github.com/atilaneves/reggae
>>> 
>>> Atila
>>
>> hm. looks interesting, albeit a little too noisy for my taste.
> 
> It depends on what you want to do: the idea is to use high-level utility functions as much as possible, while at the same time offering low-level primitives that those high-level ones are based on.
> 
> If it can be made simpler, I'd really like to know.

not that i have something better to offer. it's readable and looks fine. besides, i'm really sux at design things. ;-)

On Tue, 02 Jun 2015 22:30:47 +1200, Rikki Cattermole wrote:

>> * non-windows, as dll support on windows is still not here, i believe.
> 
> I've been able to do it a little bit already with my Web server. Haven't hit its limit so far. In other words as long as exceptions aren't used and its never unloaded, should be ok. Anyway, do you really think we would get that sort of support for a couple of releases?

it seems that nobody on windows really needs that. ;-)


> Indeed lets. But can we also not go modifying the language? I'm just suggesting injecting of symbols that execute compiler side actions. Your suggesting language changes. Even if it is a new pragma.

actually, no language changes at all. CTFE is already here, and it is used in semantic analysis stage. there are no changes to the language, besides adding some hooks and defining an API. it's invisible from the user's POV and doesn't require changing of specs. now CTFE is started explicitly, by using mixins and template evaluation. with my idea it will be started implicitly when some condition is met. sure, that conditions must be documented, but it will not change the language drastically.


> We really need to toy with these ideas properly and implement each. Then its just a matter of time to convince the higher ups that it should be merged.

yes, i believe that this is the best approach. implement both and let the best solution win in a honest competition. ;-)

i don't feel very creative right now, but implementing that idea is in my TODO list, and it's not at the bottom of list. so eventually (month? two? ten years?) i'll write a PoC.