I don't get it either. Any possible application for const in the form of code correctness went out the window once the invariant virus forced all strings to be invariant whether they were or not; so I still need to use dup to guarantee that data won't change underneath me, but then I need to cast it! It doesn't affect pure at all, because pure can be passed invariants which are just casted - the compiler needs to use rules which are a hell of a lot more binding than anything we can provide it to make these determinations. Now that const is not a storage class, it's actually not possible to declare function variables which should be stored in read-only memory (unless if it's encoded somewhere in the thirty or so combinations you can use), which also damages pure. It's a lot more confusing to deal with const data altogether than it used to be.

When I switched from D 1.0 to 2.0 I tried several times to port some large pieces of code over and ultimately gave up, just as everyone has given up trying to do it in C++. It's a hard task moving code through that kind of change.

I've learned to handle it but I would really like to not be fighting the compiler all the time. Is that what I'm supposed to be doing here, really?

Burton Radons wrote:
> I don't get it either. Any possible application for const in the form
> of code correctness went out the window once the invariant virus
> forced all strings to be invariant whether they were or not; so I
> still need to use dup to guarantee that data won't change underneath
> me, but then I need to cast it!

My experience is that when dup is used to guarantee that the data won't change underneath the problem is being addressed at the wrong end. The mutator of the data dups it, not the user.

> It doesn't affect pure at all,
> because pure can be passed invariants which are just casted - the
> compiler needs to use rules which are a hell of a lot more binding
> than anything we can provide it to make these determinations. Now
> that const is not a storage class, it's actually not possible to
> declare function variables which should be stored in read-only memory
> (unless if it's encoded somewhere in the thirty or so combinations
> you can use), which also damages pure. It's a lot more confusing to
> deal with const data altogether than it used to be.
> 
> When I switched from D 1.0 to 2.0 I tried several times to port some
> large pieces of code over and ultimately gave up, just as everyone
> has given up trying to do it in C++. It's a hard task moving code
> through that kind of change.
> 
> I've learned to handle it but I would really like to not be fighting
> the compiler all the time. Is that what I'm supposed to be doing
> here, really?

Can you be more specific about what was stimying you, as perhaps we can think of a solution.

Walter Bright Wrote:

> Burton Radons wrote:
> > I don't get it either. Any possible application for const in the form of code correctness went out the window once the invariant virus forced all strings to be invariant whether they were or not; so I still need to use dup to guarantee that data won't change underneath me, but then I need to cast it!
> 
> My experience is that when dup is used to guarantee that the data won't change underneath the problem is being addressed at the wrong end. The mutator of the data dups it, not the user.

I don't know the origin of the string or where it's going. I only know it's "invariant (char) []" now, because you're required to have it "invariant (char) []" because of the viral nature of const and that's how the string is defined, so that's what my API takes. It certainly doesn't mean that anyone is intentionally using the API incorrectly; everything takes "invariant (char) []", so my function doesn't deserve special treatment without stating so.

A more accurate way would be for the string type to be "const (char) []", and for functions which retain strings that can't change to take "invariant (char) []". That makes pretty good claims about the nature of the string, but it would clearly result in lots of cast management.

I think all these problems boil down to the fact that invariant tells you about the container rather than the object itself; but whether the object actually is invariant is totally independent of the container. The genius of D 1.0's const is that it makes an actual, actionable true statement about the object. That was a HUGE benefit. This tries to push it back into C territory and I don't think it works.

I don't buy that this is going to lead to any MP bonuses either. Const has turned into a form of the sufficiently complex compiler fallacy, the cure for any ail. But if you can cast const on or off, then any declaration you make about the mutability of the data is going to be wrong at some point, and compilers can't sometimes build the right code.

> > It doesn't affect pure at all,
> > because pure can be passed invariants which are just casted - the
> > compiler needs to use rules which are a hell of a lot more binding
> > than anything we can provide it to make these determinations. Now
> > that const is not a storage class, it's actually not possible to
> > declare function variables which should be stored in read-only memory
> > (unless if it's encoded somewhere in the thirty or so combinations
> > you can use), which also damages pure. It's a lot more confusing to
> > deal with const data altogether than it used to be.
> > 
> > When I switched from D 1.0 to 2.0 I tried several times to port some large pieces of code over and ultimately gave up, just as everyone has given up trying to do it in C++. It's a hard task moving code through that kind of change.
> > 
> > I've learned to handle it but I would really like to not be fighting the compiler all the time. Is that what I'm supposed to be doing here, really?
> 
> Can you be more specific about what was stimying you, as perhaps we can think of a solution.

Exactly the same thing as trying to do it in C++. You're stuck iteratively applying const to pretty much everything, and applying tons of casts to get the compiler to shut up. D does provide the mechanisms to have proper safe mutableCast/constCast/invariantCast functions (well, safe except that they make it so that neither compiler nor programmer can make any good statements about the nature of the data), but when trying to shift code around you usually just want to make it work.

I remember that one thing that really got to me was that you had a foreach term as invariant at that point if it weren't defined as inout. The thing is, that was correct and could've led you to a sweet easy compiler optimisation, but it's like const is in this balance where properly implemented it would be brutal and ugly, so we need to back everything off until it's only at an acceptable level of annoyance.

Oh, and templates. Templates that have nothing to do with any of these matters keep on having to be changed to appease the const god. Dealing with templates when you're doing really crazy stuff's bad enough without having that around.

Burton Radons wrote:
> A more accurate way would be for the string type to be "const (char)
> []", and for functions which retain strings that can't change to take
> "invariant (char) []". That makes pretty good claims about the nature
> of the string, but it would clearly result in lots of cast
> management.

I use that all the time, it's a great idiom. What cast management needs to be done? What I need to do is occasionally insert an .idup on the client side because the callee wants a copy. So that's that.

> I think all these problems boil down to the fact that invariant tells
> you about the container rather than the object itself; but whether
> the object actually is invariant is totally independent of the
> container. The genius of D 1.0's const is that it makes an actual,
> actionable true statement about the object. That was a HUGE benefit.
> This tries to push it back into C territory and I don't think it
> works.

I don't think I understand most of this, possibly because some of it is wrong. D2's immutable does offer a solid guarantee about what's going on and offers a programming model that makes it easy to write correct code without undue aliasing. So C doesn't quite enter into the picture there.

Objects in a container being invariant tell a lot about the container. That property makes the container shareable without a risk.

> I don't buy that this is going to lead to any MP bonuses either.

Wait and see.

> Const has turned into a form of the sufficiently complex compiler
> fallacy, the cure for any ail. But if you can cast const on or off,
> then any declaration you make about the mutability of the data is
> going to be wrong at some point, and compilers can't sometimes build
> the right code.

In D you will be able to break any design with a cast, unless you use the not-yet-defined D2 which disallows all risky casts. So the fact that you can cast const away is hardly changing anything.

>>> It doesn't affect pure at all, because pure can be passed
>>> invariants which are just casted - the compiler needs to use
>>> rules which are a hell of a lot more binding than anything we can
>>> provide it to make these determinations. Now that const is not a
>>> storage class, it's actually not possible to declare function
>>> variables which should be stored in read-only memory (unless if
>>> it's encoded somewhere in the thirty or so combinations you can
>>> use), which also damages pure. It's a lot more confusing to deal
>>> with const data altogether than it used to be.
>>> 
>>> When I switched from D 1.0 to 2.0 I tried several times to port
>>> some large pieces of code over and ultimately gave up, just as
>>> everyone has given up trying to do it in C++. It's a hard task
>>> moving code through that kind of change.
>>> 
>>> I've learned to handle it but I would really like to not be
>>> fighting the compiler all the time. Is that what I'm supposed to
>>> be doing here, really?
>> Can you be more specific about what was stimying you, as perhaps we
>> can think of a solution.
> 
> Exactly the same thing as trying to do it in C++. You're stuck
> iteratively applying const to pretty much everything, and applying
> tons of casts to get the compiler to shut up.

But this is misusing const. It means it didn't belong there in the first place. Const (and immutable) in D appear less frequently than in C++ because they provide superior guarantees.

> Oh, and templates. Templates that have nothing to do with any of
> these matters keep on having to be changed to appease the const god.
> Dealing with templates when you're doing really crazy stuff's bad
> enough without having that around.

Agreed, qualifiers do make templates a tad harder to define. Fortunately std.traits.Unqual (to be released) can be helpful there.


Andrei

Andrei Alexandrescu Wrote:

> Burton Radons wrote:
> > A more accurate way would be for the string type to be "const (char) []", and for functions which retain strings that can't change to take "invariant (char) []". That makes pretty good claims about the nature of the string, but it would clearly result in lots of cast management.
> 
> I use that all the time, it's a great idiom. What cast management needs to be done? What I need to do is occasionally insert an .idup on the client side because the callee wants a copy. So that's that.

So long as the object definition of string is "invariant (char) []", I can't guarantee anything about the nature of the object because you need to cast to "invariant (char) []" to be able to interface with any API.

The good side is that when I changed it to be defined as "const (char) []" only one line of code made a squeak. That gives me solid actionable information. If an API is declared as istring, then whatever you give it must not ever change. If an API is declared as string, then whatever happens in there, it won't change the data. Pretty good!

> > I think all these problems boil down to the fact that invariant tells you about the container rather than the object itself; but whether the object actually is invariant is totally independent of the container. The genius of D 1.0's const is that it makes an actual, actionable true statement about the object. That was a HUGE benefit. This tries to push it back into C territory and I don't think it works.
> 
> I don't think I understand most of this, possibly because some of it is wrong. D2's immutable does offer a solid guarantee about what's going on and offers a programming model that makes it easy to write correct code without undue aliasing. So C doesn't quite enter into the picture there.
> 
> Objects in a container being invariant tell a lot about the container. That property makes the container shareable without a risk.
> 
> > I don't buy that this is going to lead to any MP bonuses either.
> 
> Wait and see.

I don't need to wait and see when twenty years of "hinted optimisation" have had predictable results. If the programmer can set an incorrect state doing something which he's forced to do, then any compiler which uses this state as an actual description about the situation will cause problems because the compiler's opportunities to apply these optimisations will shift over the course of the development of the program. Code works, add one line, code doesn't work. Code works, try it on another machine, code doesn't work.

The only way I can see this working is if the compiler really did have a good idea about the nature of the state, at which point the programmer's statements are completely superfluous. That's not coincidentally exactly how we do these optimisations now: determine whether the state is in such a way that we can do this safely, and only then actually apply the optimisation.

That can involve automatic MP of a sort - I understand that's the way the C-based MP systems work, where you tell it that the state of the program is such and such before letting it go ahead.

Fully automatic unambiguous effective parallelisation requires that everything the programmer says is true. It's not something that can be stuffed into a language with pointers and external API calls. If you think you can do that, fine. But you haven't, and nobody else has, to my knowledge.

Burton Radons wrote:
> Andrei Alexandrescu Wrote:
> 
>> Burton Radons wrote:
>>> A more accurate way would be for the string type to be "const
>>> (char) []", and for functions which retain strings that can't
>>> change to take "invariant (char) []". That makes pretty good
>>> claims about the nature of the string, but it would clearly
>>> result in lots of cast management.
>> I use that all the time, it's a great idiom. What cast management
>> needs to be done? What I need to do is occasionally insert an .idup
>> on the client side because the callee wants a copy. So that's that.
>> 
> 
> So long as the object definition of string is "invariant (char) []",
> I can't guarantee anything about the nature of the object because you
> need to cast to "invariant (char) []" to be able to interface with
> any API.
> 
> The good side is that when I changed it to be defined as "const
> (char) []" only one line of code made a squeak. That gives me solid
> actionable information. If an API is declared as istring, then
> whatever you give it must not ever change. If an API is declared as
> string, then whatever happens in there, it won't change the data.
> Pretty good!

I have trouble following what you're saying. If what you're saying is essentially that in char[] is a better parameter definition than string for functions that don't need to escape their string argument, then yes, you are entirely right.

So what I recommend is:

void foo(in char[] s); // foo looks at s, doesn't escape it
void bar(string s); // bar needs to save s
void baz(char[] s); // baz needs to change s' contents

>>> I think all these problems boil down to the fact that invariant
>>> tells you about the container rather than the object itself; but
>>> whether the object actually is invariant is totally independent
>>> of the container. The genius of D 1.0's const is that it makes an
>>> actual, actionable true statement about the object. That was a
>>> HUGE benefit. This tries to push it back into C territory and I
>>> don't think it works.
>> I don't think I understand most of this, possibly because some of
>> it is wrong. D2's immutable does offer a solid guarantee about
>> what's going on and offers a programming model that makes it easy
>> to write correct code without undue aliasing. So C doesn't quite
>> enter into the picture there.
>> 
>> Objects in a container being invariant tell a lot about the
>> container. That property makes the container shareable without a
>> risk.
>> 
>>> I don't buy that this is going to lead to any MP bonuses either.
>> Wait and see.
> 
> I don't need to wait and see when twenty years of "hinted
> optimisation" have had predictable results. If the programmer can set
> an incorrect state doing something which he's forced to do, then any
> compiler which uses this state as an actual description about the
> situation will cause problems because the compiler's opportunities to
> apply these optimisations will shift over the course of the
> development of the program. Code works, add one line, code doesn't
> work. Code works, try it on another machine, code doesn't work.

You are completely, thoroughly losing me. I can only assume you are misunderstanding the role of const and immutable in manycore programming and build from there.

> The only way I can see this working is if the compiler really did
> have a good idea about the nature of the state, at which point the
> programmer's statements are completely superfluous. That's not
> coincidentally exactly how we do these optimisations now: determine
> whether the state is in such a way that we can do this safely, and
> only then actually apply the optimisation.
> 
> That can involve automatic MP of a sort - I understand that's the way
> the C-based MP systems work, where you tell it that the state of the
> program is such and such before letting it go ahead.
> 
> Fully automatic unambiguous effective parallelisation requires that
> everything the programmer says is true. It's not something that can
> be stuffed into a language with pointers and external API calls. If
> you think you can do that, fine. But you haven't, and nobody else
> has, to my knowledge.

I continue being lost. Words assemble in phrases, phrases assemble in sentences, sentences parse properly, but I can't understand one thing. I need to defer a response to others.


Andrei

Andrei Alexandrescu Wrote:

> In D you will be able to break any design with a cast, unless you use the not-yet-defined D2 which disallows all risky casts. So the fact that you can cast const away is hardly changing anything.

Oh, and this is an idea which is almost exactly 20 years old, from back when ANSI was developing the C standard. They had defined const but then made it so that const cannot be casted off, making the language unimplementable.

We cannot avoid these APIs which led to this unimplementability because it's a common problem: a mutable object goes through an environment where it is treated as const, but afterwards needs to continue to be mutable. For example, you might have:

  alias const (char) [] string;
  alias char [] mstring;

  // Return the first match within the string, or null if there is no match.
  string match (string text, RE expression);

  mstring text;

  auto submatch = match (text, expression);
  submatch [] = ' '; // Fails.

The only non-casting alternative is to define:

  mstring mmatch (mstring text, RE expression);

But there are two problems. One, it's the same exact function which does the same exact thing. So you're wasting the programmer's time for moving match into a template function and then overloading it twice, you're wasting the reader's time for having to learn two functions and knowing when to apply either, and you're wasting the processor's time for having to load both functions. More critically, it's not descriptive; it in fact implies that mmatch may or will modify the data within the text, when it won't, ever. So if the purpose is to make const describe the nature of the implementation of the function, it has completely failed and so can't be trusted anywhere.

Dennis Ritchie argued against const when it appeared (http://www.lysator.liu.se/c/dmr-on-noalias.html).

On Fri, 06 Mar 2009 13:30:22 -0800, Andrei Alexandrescu wrote:

> Burton Radons wrote:
>> Andrei Alexandrescu Wrote:
>> 
>>> Burton Radons wrote:
>>>> A more accurate way would be for the string type to be "const (char) []", and for functions which retain strings that can't change to take "invariant (char) []". That makes pretty good claims about the nature of the string, but it would clearly result in lots of cast management.
>>> I use that all the time, it's a great idiom. What cast management needs to be done? What I need to do is occasionally insert an .idup on the client side because the callee wants a copy. So that's that.
>>> 
>>> 
>> So long as the object definition of string is "invariant (char) []", I can't guarantee anything about the nature of the object because you need to cast to "invariant (char) []" to be able to interface with any API.
>> 
>> The good side is that when I changed it to be defined as "const (char) []" only one line of code made a squeak. That gives me solid actionable information. If an API is declared as istring, then whatever you give it must not ever change. If an API is declared as string, then whatever happens in there, it won't change the data. Pretty good!
> 
> I have trouble following what you're saying. If what you're saying is essentially that in char[] is a better parameter definition than string for functions that don't need to escape their string argument, then yes, you are entirely right.
> 
> So what I recommend is:
> 
> void foo(in char[] s); // foo looks at s, doesn't escape it
> void bar(string s); // bar needs to save s
> void baz(char[] s); // baz needs
> to change s' contents

I think what Burton is saying is by annointing immutable(char)[] as the type "string," you are essentially sending a message to developers that all strings should be immutable, and all *string parameters* should be declared immutable.  What this does is force developers who want to deal in const or mutable chars have to do lots of duplication or casting, which either makes your code dog slow, or makes your code break const.

Evidence is how (at least in previous releases) anything in Phobos that took an argument that was a utf8 string of characters used the parameter type "string", making it very difficult to use when you don't have string types.  If you want to find a substring in a string, it makes no sense that you first have to make the argument invariant.  a substring function isn't saving a pointer to that data.

I think the complaint is simply that string is defined as immutable(char) [] and therefore is promoted as *the only* string type to use.  Using other forms (such as in char[] or const(char)[] or char[]) doesn't look like the argument is a string, when the word "string" is already taken to mean something else.

-Steve

Burton Radons wrote:
> Andrei Alexandrescu Wrote:
> 
>> In D you will be able to break any design with a cast, unless you use the not-yet-defined D2 which disallows all risky casts. So the fact that you can cast const away is hardly changing anything.
> 
> Oh, and this is an idea which is almost exactly 20 years old, from back when ANSI was developing the C standard. They had defined const but then made it so that const cannot be casted off, making the language unimplementable.
> 
> We cannot avoid these APIs which led to this unimplementability because it's a common problem: a mutable object goes through an environment where it is treated as const, but afterwards needs to continue to be mutable. For example, you might have:
> 
>   alias const (char) [] string;
>   alias char [] mstring;
> 
>   // Return the first match within the string, or null if there is no match.
>   string match (string text, RE expression);
> 
>   mstring text;
> 
>   auto submatch = match (text, expression);
>   submatch [] = ' '; // Fails.
> 
> The only non-casting alternative is to define:
> 
>   mstring mmatch (mstring text, RE expression);
> 
> But there are two problems. One, it's the same exact function which does the same exact thing. So you're wasting the programmer's time for moving match into a template function and then overloading it twice, you're wasting the reader's time for having to learn two functions and knowing when to apply either, and you're wasting the processor's time for having to load both functions. More critically, it's not descriptive; it in fact implies that mmatch may or will modify the data within the text, when it won't, ever. So if the purpose is to make const describe the nature of the implementation of the function, it has completely failed and so can't be trusted anywhere.

This is a known problem for hich we will provide a solution.

Andrei

Steven Schveighoffer wrote:
> On Fri, 06 Mar 2009 13:30:22 -0800, Andrei Alexandrescu wrote:
> 
>> Burton Radons wrote:
>>> Andrei Alexandrescu Wrote:
>>>
>>>> Burton Radons wrote:
>>>>> A more accurate way would be for the string type to be "const (char)
>>>>> []", and for functions which retain strings that can't change to take
>>>>> "invariant (char) []". That makes pretty good claims about the nature
>>>>> of the string, but it would clearly result in lots of cast
>>>>> management.
>>>> I use that all the time, it's a great idiom. What cast management
>>>> needs to be done? What I need to do is occasionally insert an .idup on
>>>> the client side because the callee wants a copy. So that's that.
>>>>
>>>>
>>> So long as the object definition of string is "invariant (char) []", I
>>> can't guarantee anything about the nature of the object because you
>>> need to cast to "invariant (char) []" to be able to interface with any
>>> API.
>>>
>>> The good side is that when I changed it to be defined as "const (char)
>>> []" only one line of code made a squeak. That gives me solid actionable
>>> information. If an API is declared as istring, then whatever you give
>>> it must not ever change. If an API is declared as string, then whatever
>>> happens in there, it won't change the data. Pretty good!
>> I have trouble following what you're saying. If what you're saying is
>> essentially that in char[] is a better parameter definition than string
>> for functions that don't need to escape their string argument, then yes,
>> you are entirely right.
>>
>> So what I recommend is:
>>
>> void foo(in char[] s); // foo looks at s, doesn't escape it
>> void bar(string s); // bar needs to save s
>> void baz(char[] s); // baz needs
>> to change s' contents
> 
> I think what Burton is saying is by annointing immutable(char)[] as the type "string," you are essentially sending a message to developers that all strings should be immutable, and all *string parameters* should be declared immutable.  What this does is force developers who want to deal in const or mutable chars have to do lots of duplication or casting, which either makes your code dog slow, or makes your code break const.
> 
> Evidence is how (at least in previous releases) anything in Phobos that took an argument that was a utf8 string of characters used the parameter type "string", making it very difficult to use when you don't have string types.  If you want to find a substring in a string, it makes no sense that you first have to make the argument invariant.  a substring function isn't saving a pointer to that data.
> 
> I think the complaint is simply that string is defined as immutable(char)
> [] and therefore is promoted as *the only* string type to use.  Using other forms (such as in char[] or const(char)[] or char[]) doesn't look like the argument is a string, when the word "string" is already taken to mean something else.
> 
> -Steve

I see. Phobos is being changed to accept in char[] instead of string
wherever applicable. As far as what the default "string" ought to be,
immutable(char)[] is the safest of the three so I think it should be that.

Andrei