Hello everybody. My name is Zach, and I have a suggestion for the improvement of D. I've been looking at the following stalled pull request for a while now:

https://github.com/D-Programming-Language/dmd/pull/1877

...in which Walter Bright wants to introduce built-in-attribute inference for a relatively small set of functions. It seems like the most obvious thing in the world to me to desire this, and not even just for 'auto' and templated functions, but for *every* function. And there's no reason it can't be done. So long as the compiler has everything it needs to determine which attributes can be applied, there's no reason to demand anything from the programmer. Look how simple this function is:

int plusOne(int a) { return a+1; }

Let's say I later want to call it, however, from a fully attributed function:

int plusTwo(int a) pure nothrow @safe @nogc  {
  return plusOne(plusOne(a));
}

I get a compiler error. The only way to stop it is to add unnecessary visual noise to the first function. All of these attributes should be something that you *want* to add, not something that you *need*. The compiler can obviously figure out if the function throws or not. Just keep an additional internal flag for each of the attributes. When any attribute is violated, flip the bit and boom, you have your implicit function signature.

I think this is how it always should have been. It's important to remember that the above attributes have the 'covariant' property, which means they can always be called by any function without that property. Therefore no existing code will start failing to compile. Only certain things which would have *errored* before will stop. Plus new optimizations can be done.

So what's the problem? As you can read in the vehement opposition to pull 1877 above, the big fear is that function signatures will start changing willy-nilly, causing the exposed interface of the function to destabilize, which will cause linker errors or require code intended to be kept separate in large projects to be recompiled at every little change.

I find this depressing! That something so good should be ruined by something so remote as the need for separate compilation in very large projects? I mean, most projects aren't even very large. Also, because D compiles so much faster than its predecessors, is it even such a big deal to have to recompile everything?

But let's admit the point may be valid. Yes, under attribute inference, the function signatures in the exposed API will indeed find themselves changing every time one so much as adds a 'printf' or calls something that throws.

But they don't *have* to change. The compiler doesn't need to include the inferred attributes when it generates the mangled name and the .di signature, only the explicit ones. From within the program, all the opportunities for inference and optimization could be left intact, while outside programs accessing the code in precompiled form could only access the functions as explicitly indicated.

This makes no change to the language, except that it allows new things to compile. The only hitch is this: What if you want the full advantages of optimization and inference from across compilation boundaries? You'd have to add each of the covariant function attributes manually to every function you exposed. From my perspective, this is still a chore.

I suggest a new attribute, @api, which does nothing more than to tell the compiler to generate the function signature and mangle the name only with its explicit attributes, and not with its inferred ones. Inside the program, there's no reason the compiler can't continue to use inference, but with @api, the exposed interface will be stabilized, should the programmer want that. Simple.

I anticipate a couple of objections to my proposal:

The first is that we would now demand that the programmer decide whether he wants his exposed functions stabilized or not. For a large library used by different people, this choice might pose some difficulty. But it's not that bad. You just choose: do you want to improve compilation times and/or closed-source consistency by ensuring a stable interface, or do you want to speed up runtime performance without having to clutter your code? Most projects would choose the latter. @api is made available for the those who don't. The opposition to attribute inference put forth in pull 1877 is thereby appeased.

A second objection to this proposal: Another attribute? Really? Well, yeah.

But it's not a problem, I say, for these reasons:

1. This one little attribute allows you to excise gajillions of unnecessary little attributes which are currently forced on the programmer by the lack of inference, simply by appeasing the opponents of inference and allowing it to be implemented.

2. It seems like most people will be okay just recompiling projects instead of preferring to stabilize their apis. Thus, @api will only be used rarely.

3. @api forces you to add all the attributes you want exposed to the world manually. It's a candid admission that you are okay littering your code with attributes, thereby lessening the pain at having to add one more.

4. Most @api functions will come in clusters. After all, it *is* an API you are exposing, so I think it's highly likely that a single "@api:" will work in most cases.


Now, "Bombard with your gunships."

Thank you.

On Monday, 5 January 2015 at 21:15:00 UTC, Zach the Mystic wrote:
> Now, "Bombard with your gunships."
>

An alternative could be to use the already existing 'export'.

http://dlang.org/attribute.html
"Export means that any code outside the executable can access the member. Export is analogous to exporting definitions from a DLL."

On Monday, 5 January 2015 at 21:25:01 UTC, Daniel N wrote:
> An alternative could be to use the already existing 'export'.

'extern'. Yeah, something like 'extern (noinfer):'.

On Monday, 5 January 2015 at 22:00:40 UTC, Zach the Mystic wrote:
> On Monday, 5 January 2015 at 21:25:01 UTC, Daniel N wrote:
>> An alternative could be to use the already existing 'export'.
>
> 'extern'. Yeah, something like 'extern (noinfer):'.

Err, yeah, whatever works!

On 05/01/15 22:14, Zach the Mystic via Digitalmars-d wrote:
> I get a compiler error. The only way to stop it is to add unnecessary visual
> noise to the first function. All of these attributes should be something that
> you *want* to add, not something that you *need*. The compiler can obviously
> figure out if the function throws or not. Just keep an additional internal flag
> for each of the attributes. When any attribute is violated, flip the bit and
> boom, you have your implicit function signature.

Bear in mind one quite important factor -- all that alleged noise isn't simply about getting stuff to work, it's about promises that the function makes to downstream users.  You do touch on this yourself, but I think you have missed how your @api flag could go wrong.

> I suggest a new attribute, @api, which does nothing more than to tell the
> compiler to generate the function signature and mangle the name only with its
> explicit attributes, and not with its inferred ones. Inside the program, there's
> no reason the compiler can't continue to use inference, but with @api, the
> exposed interface will be stabilized, should the programmer want that. Simple.

IMHO if anything like this is to be implemented, the extra flag should be to indicate that a function is _not_ intended to be part of the API and that therefore it is OK to infer its attributes.

Here's the rationale.  Suppose that I have a bunch of functions that are all intended to be part of the public API of my project.  I accidentally forget to tag one of them with the @api attribute, so its attributes will be auto-inferred, but the function is still public, so downstream users will wind up using it.

3 months later, I realize my mistake, and add the @api attribute -- at which point downstream users' code will break if their code was relying on the unintended inferred attributes.

If on the other hand you take the assumption that attributes should by default _not_ be auto-inferred, and you accidentally forget to tag a function to auto-infer its attributes, that can be fixed without breaking downstream.

It's quite analogous in this respect to the argument about final vs. virtual by default for class methods.

On 06/01/15 00:48, Joseph Rushton Wakeling via Digitalmars-d wrote:
> IMHO if anything like this is to be implemented, the extra flag should be to
> indicate that a function is _not_ intended to be part of the API and that
> therefore it is OK to infer its attributes.

Hmm.  On thinking about this some more, it occurs to me that this might be fundamentally about protection.  If it were forbidden to auto-infer attributes for a non-templated public function, then quite a few of my objections above might go away.

On Monday, 5 January 2015 at 23:48:17 UTC, Joseph Rushton Wakeling via Digitalmars-d wrote:
> Here's the rationale.  Suppose that I have a bunch of functions that are all intended to be part of the public API of my project.  I accidentally forget to tag one of them with the @api attribute,

A more likely scenario is that your library starts small enough not to need the @api attribute, then at some point it gets really, really huge. Then in one fell swoop you decide to "@api:" your whole file so that the public interface won't change so often. I'm picking the most extreme case I can think of, in order to argue the point from a different perspective.

> so its attributes will be auto-inferred, but the function is still public, so downstream users will wind up using it.
>
> 3 months later, I realize my mistake, and add the @api attribute -- at which point downstream users' code will break if their code was relying on the unintended inferred attributes.

Attribute inference provides convenience, not guarantees. If a user was relying on the purity of a function which was never marked 'pure', it's only convenience which allows him to do it, both on the part of the user, for adding 'pure', and the library writer, for *not* adding it. Adding @api (or 'extern (noinfer)') cancels that convenience for the sake of modularity. It's a tradeoff. The problem  itself is solved either by the library writer marking the function 'pure', or the user removing 'pure' from his own function. Without @api, the problem only arises when the library writer actually does something impure, which makes perfect sense. It's @api (and D's existing default, by the way) which adds the artificiality to the process, not my suggested default.

> It's quite analogous in this respect to the argument about final vs. virtual by default for class methods.

I don't think so, because of so-called covariance. Final and virtual each have their own advantages and disadvantages, whereas inferring attributes only goes one way. There is no cost to inferring in the general case. My suggestion, (I now prefer 'extern(noinfer)'), does absolutely nothing except to restore D's existing default, for what I think are the rare cases it is needed. I could be wrong about just how rare using extern(noinfer) will actually be, but consider that phobos, for example, just doesn't need it, because it's too small a library to cause trouble if all of a sudden one of its non-templated functions becomes impure. A quick recompile, a new interface file, and now everyone's using the new thing. Even today, it's not even marked up with attributes completely, thus indicating that you never even *could* have used it for all it's worth.

Have I convinced you?

On Mon, 05 Jan 2015 21:14:58 +0000
Zach the Mystic via Digitalmars-d <digitalmars-d@puremagic.com> wrote:

> Hello everybody. My name is Zach, and I have a suggestion for the improvement of D. I've been looking at the following stalled pull request for a while now:
> 
> https://github.com/D-Programming-Language/dmd/pull/1877

heh. i did a little hack based on this patch: autoinference is turned on only for `private auto`. i also added a bunch of UDAs to control the process: `@inferattr` (can be applied to any function), `@notinferattr`, `@canthrow`, `@impure` and `@gc` (to control inference).

any explicit attribute on function will block inference too.

as druntime and phobos has no `private auto` which is not templated or without explicit attribues, it compiles fine. and for my code i have some control.

this feature can be poorly designed, but as it doesn't conflict with most of the existing code, i'm happy with it. one should be carefull with templates calling private functions with inferred attributes (it breaks linking -- for obvious reason), but it's ok for me. `private auto` is so ugly that it will rise my alarm level anyway. i mean "function returning 'auto' is suspicious".

On Monday, 5 January 2015 at 21:15:00 UTC, Zach the Mystic wrote:
> Hello everybody. My name is Zach, and I have a suggestion for the improvement of D. I've been looking at the following stalled pull request for a while now:
>
> https://github.com/D-Programming-Language/dmd/pull/1877
>
> ...in which Walter Bright wants to introduce built-in-attribute inference for a relatively small set of functions. It seems like the most obvious thing in the world to me to desire this, and not even just for 'auto' and templated functions, but for *every* function. And there's no reason it can't be done. So long as the compiler has everything it needs to determine which attributes can be applied, there's no reason to demand anything from the programmer. Look how simple this function is:
>
> int plusOne(int a) { return a+1; }
>
> Let's say I later want to call it, however, from a fully attributed function:
>
> int plusTwo(int a) pure nothrow @safe @nogc  {
>   return plusOne(plusOne(a));
> }
>
> I get a compiler error. The only way to stop it is to add unnecessary visual noise to the first function. All of these attributes should be something that you *want* to add, not something that you *need*. The compiler can obviously figure out if the function throws or not. Just keep an additional internal flag for each of the attributes. When any attribute is violated, flip the bit and boom, you have your implicit function signature.
>
> I think this is how it always should have been. It's important to remember that the above attributes have the 'covariant' property, which means they can always be called by any function without that property. Therefore no existing code will start failing to compile. Only certain things which would have *errored* before will stop. Plus new optimizations can be done.
>
> So what's the problem? As you can read in the vehement opposition to pull 1877 above, the big fear is that function signatures will start changing willy-nilly, causing the exposed interface of the function to destabilize, which will cause linker errors or require code intended to be kept separate in large projects to be recompiled at every little change.
>
> I find this depressing! That something so good should be ruined by something so remote as the need for separate compilation in very large projects? I mean, most projects aren't even very large. Also, because D compiles so much faster than its predecessors, is it even such a big deal to have to recompile everything?
>
> But let's admit the point may be valid. Yes, under attribute inference, the function signatures in the exposed API will indeed find themselves changing every time one so much as adds a 'printf' or calls something that throws.
>
> But they don't *have* to change. The compiler doesn't need to include the inferred attributes when it generates the mangled name and the .di signature, only the explicit ones. From within the program, all the opportunities for inference and optimization could be left intact, while outside programs accessing the code in precompiled form could only access the functions as explicitly indicated.
>
> This makes no change to the language, except that it allows new things to compile. The only hitch is this: What if you want the full advantages of optimization and inference from across compilation boundaries? You'd have to add each of the covariant function attributes manually to every function you exposed. From my perspective, this is still a chore.
>
> I suggest a new attribute, @api, which does nothing more than to tell the compiler to generate the function signature and mangle the name only with its explicit attributes, and not with its inferred ones. Inside the program, there's no reason the compiler can't continue to use inference, but with @api, the exposed interface will be stabilized, should the programmer want that. Simple.
>
> I anticipate a couple of objections to my proposal:
>
> The first is that we would now demand that the programmer decide whether he wants his exposed functions stabilized or not. For a large library used by different people, this choice might pose some difficulty. But it's not that bad. You just choose: do you want to improve compilation times and/or closed-source consistency by ensuring a stable interface, or do you want to speed up runtime performance without having to clutter your code? Most projects would choose the latter. @api is made available for the those who don't. The opposition to attribute inference put forth in pull 1877 is thereby appeased.
>
> A second objection to this proposal: Another attribute? Really? Well, yeah.
>
> But it's not a problem, I say, for these reasons:
>
> 1. This one little attribute allows you to excise gajillions of unnecessary little attributes which are currently forced on the programmer by the lack of inference, simply by appeasing the opponents of inference and allowing it to be implemented.
>
> 2. It seems like most people will be okay just recompiling projects instead of preferring to stabilize their apis. Thus, @api will only be used rarely.
>
> 3. @api forces you to add all the attributes you want exposed to the world manually. It's a candid admission that you are okay littering your code with attributes, thereby lessening the pain at having to add one more.
>
> 4. Most @api functions will come in clusters. After all, it *is* an API you are exposing, so I think it's highly likely that a single "@api:" will work in most cases.
>
>
> Now, "Bombard with your gunships."
>
> Thank you.

Needing a function to have a completely stable signature is the (very important) exception, not the rule and inference is awesome for everywhere else.

It's worth noting that with full inference then much less code would be explicitly annotated, which goes some way towards ameliorating the stability problem as more client code will be flexible w.r.t. changing attributes on library boundaries.

About restriction to `auto`:
`auto` is about return types, not attributes. Inference should be performed on anything that's available. *.di files should contain the inferred attributes when generated to support totally separate compilation. For more convoluted examples, tooling could help (compiler spits out json with inference results, external tool / IDE picks up that info and helps the user automatically apply it to declarations elsewhere).

On Tuesday, 6 January 2015 at 09:12:43 UTC, John Colvin wrote:
> On Monday, 5 January 2015 at 21:15:00 UTC, Zach the Mystic wrote:
>> Hello everybody. My name is Zach, and I have a suggestion for the improvement of D. I've been looking at the following stalled pull request for a while now:
>>
>> https://github.com/D-Programming-Language/dmd/pull/1877
>>
>> ...in which Walter Bright wants to introduce built-in-attribute inference for a relatively small set of functions. It seems like the most obvious thing in the world to me to desire this, and not even just for 'auto' and templated functions, but for *every* function. And there's no reason it can't be done. So long as the compiler has everything it needs to determine which attributes can be applied, there's no reason to demand anything from the programmer. Look how simple this function is:
>>
>> int plusOne(int a) { return a+1; }
>>
>> Let's say I later want to call it, however, from a fully attributed function:
>>
>> int plusTwo(int a) pure nothrow @safe @nogc  {
>>  return plusOne(plusOne(a));
>> }
>>
>> I get a compiler error. The only way to stop it is to add unnecessary visual noise to the first function. All of these attributes should be something that you *want* to add, not something that you *need*. The compiler can obviously figure out if the function throws or not. Just keep an additional internal flag for each of the attributes. When any attribute is violated, flip the bit and boom, you have your implicit function signature.
>>
>> I think this is how it always should have been. It's important to remember that the above attributes have the 'covariant' property, which means they can always be called by any function without that property. Therefore no existing code will start failing to compile. Only certain things which would have *errored* before will stop. Plus new optimizations can be done.
>>
>> So what's the problem? As you can read in the vehement opposition to pull 1877 above, the big fear is that function signatures will start changing willy-nilly, causing the exposed interface of the function to destabilize, which will cause linker errors or require code intended to be kept separate in large projects to be recompiled at every little change.
>>
>> I find this depressing! That something so good should be ruined by something so remote as the need for separate compilation in very large projects? I mean, most projects aren't even very large. Also, because D compiles so much faster than its predecessors, is it even such a big deal to have to recompile everything?
>>
>> But let's admit the point may be valid. Yes, under attribute inference, the function signatures in the exposed API will indeed find themselves changing every time one so much as adds a 'printf' or calls something that throws.
>>
>> But they don't *have* to change. The compiler doesn't need to include the inferred attributes when it generates the mangled name and the .di signature, only the explicit ones. From within the program, all the opportunities for inference and optimization could be left intact, while outside programs accessing the code in precompiled form could only access the functions as explicitly indicated.
>>
>> This makes no change to the language, except that it allows new things to compile. The only hitch is this: What if you want the full advantages of optimization and inference from across compilation boundaries? You'd have to add each of the covariant function attributes manually to every function you exposed. From my perspective, this is still a chore.
>>
>> I suggest a new attribute, @api, which does nothing more than to tell the compiler to generate the function signature and mangle the name only with its explicit attributes, and not with its inferred ones. Inside the program, there's no reason the compiler can't continue to use inference, but with @api, the exposed interface will be stabilized, should the programmer want that. Simple.
>>
>> I anticipate a couple of objections to my proposal:
>>
>> The first is that we would now demand that the programmer decide whether he wants his exposed functions stabilized or not. For a large library used by different people, this choice might pose some difficulty. But it's not that bad. You just choose: do you want to improve compilation times and/or closed-source consistency by ensuring a stable interface, or do you want to speed up runtime performance without having to clutter your code? Most projects would choose the latter. @api is made available for the those who don't. The opposition to attribute inference put forth in pull 1877 is thereby appeased.
>>
>> A second objection to this proposal: Another attribute? Really? Well, yeah.
>>
>> But it's not a problem, I say, for these reasons:
>>
>> 1. This one little attribute allows you to excise gajillions of unnecessary little attributes which are currently forced on the programmer by the lack of inference, simply by appeasing the opponents of inference and allowing it to be implemented.
>>
>> 2. It seems like most people will be okay just recompiling projects instead of preferring to stabilize their apis. Thus, @api will only be used rarely.
>>
>> 3. @api forces you to add all the attributes you want exposed to the world manually. It's a candid admission that you are okay littering your code with attributes, thereby lessening the pain at having to add one more.
>>
>> 4. Most @api functions will come in clusters. After all, it *is* an API you are exposing, so I think it's highly likely that a single "@api:" will work in most cases.
>>
>>
>> Now, "Bombard with your gunships."
>>
>> Thank you.
>
> Needing a function to have a completely stable signature is the (very important) exception, not the rule and inference is awesome for everywhere else.
>
> It's worth noting that with full inference then much less code would be explicitly annotated, which goes some way towards ameliorating the stability problem as more client code will be flexible w.r.t. changing attributes on library boundaries.
>
> About restriction to `auto`:
> `auto` is about return types, not attributes. Inference should be performed on anything that's available. *.di files should contain the inferred attributes when generated to support totally separate compilation. For more convoluted examples, tooling could help (compiler spits out json with inference results, external tool / IDE picks up that info and helps the user automatically apply it to declarations elsewhere).

tldr: I like what you're thinking, please can we have this.