On Tuesday, 29 September 2020 at 13:44:55 UTC, Ola Fosheim Grøstad wrote:
> On Tuesday, 29 September 2020 at 13:38:25 UTC, bachmeier wrote:
>> That's a good example. IMO there's a difference between a language and bits and pieces that can be put together to call random libraries.
>
> Heh, not really. It is not unusual for languages to have a minimal core and default include the most commonly used parts of their standard library. The programmer can often not tell the difference.

The statement was this:

> "The most important principle in a language design is to define a small core of essential primitives. All necessary syntactic sugar lowers to core constructs. Do everything else in libraries."

It's possible I'm wrong, but it says to me that the language developers should design the language worrying only about that 'small core of essential primitives'. Maybe 'libraries' refers to the standard library only, but that's not clear to me.

On Tue, Sep 29, 2020 at 06:43:23PM +0000, bachmeier via Digitalmars-d wrote:
> On Tuesday, 29 September 2020 at 14:07:14 UTC, Andrei Alexandrescu wrote:
[...]
> > There is no doubt to Walter or myself that the way built-in AAs were done in D is a resounding failure.
> 
> If so, it's a pretty good resounding failure, because I like them and they would most likely not be available to me now if they hadn't been built into the language. [...]

Same here.  Built-in AA's were one of the big reasons that drew me to D. Now granted, the way they are implemented perhaps leaves a lot to be desired, but having built-in support for AA's IMO is a fundamental language feature.  Whether it's implemented in the compiler or the standard library is secondary, but it's not something that should be left to an external 3rd party library.

It would definitely be a detriment if D had followed the same path as C++ in having hash tables in the standard library (extremely late in the history of C++, I might add -- something I will probably hold forever against C++ :-P), but requiring a lot of manual effort to use: no default hash function for built-in types like structs or ints, need to manually declare all sorts of things just to instantiate a lousy hashtable, having inordinately long derived type names just to name something that in D is as simple as `MyStruct[string]`, and just general needless gratuitous barriers to usage.

Some level of language support is necessary to grease all these usability gears: at the very least the language must support nice, easy syntax to declare AA's -- even if the actual implementation is delegated to a library type in the standard library.


[...]
> > The way they should have been done was a templated library type AssociativeArray!(K, V) with the V[K] type syntax sugar on top. That would have allowed the implementation to improve along with the rest of the language and its library.
> 
> Does anything prevent the introduction of a new library solution? If it's good enough, it can eventually be added to the language.
[...]

A lot of effort has been put in to move the AA implementation into object.d, and the current implementation has also undergone several rounds of improvement, code quality wise and performance wise.  But there remain some magic bits that currently cannot be duplicated in "user space", so to speak, (as opposed to compiler space).

One is the handling of const/immutable on keys, which, in spite of being magically handled by the compiler, nevertheless suffers from flaws: to prevent key aliasing problems that cause inconsistencies in the AA, as of several releases ago AA keys are now implicitly `const`. However, this does not actually solve the problem -- because mutable can implicitly convert to const, so the aliasing problem still exists. It would be nice to make it immutable, but that breaks existing code in some places, and in any case switching from const to immutable isn't something that can be done as a gradual deprecation -- it's all or nothing.  But regardless of the correctness issue, the handling of const/immutable qualifiers on AA keys depends on compiler magic, and currently either isn't expressible in library code, or else is so cumbersome it's simply impractical.

Another is the magic handling of:

	Data[string][string] aa;
	aa["abc"]["def"] = Data(...);

For this, we need opIndexCreate, which we currently don't have:

	https://issues.dlang.org/show_bug.cgi?id=7753

There may be one or two other issues that I can't recall off the top of my head, but basically, we've come a long way since the original AA implementation, and now only need to remove a few more magical barriers before a fully-library AA solution is possible in druntime.

//

Of course, all of this confirms what Andrei is saying about too much magic being boiled into the compiler: it can do stuff library code cannot, and because of that, fixing problems in AAs requires digging deep into compiler innards.  Worse yet, some of the rules governing AA magic in the compiler isn't fully compatible with the rules library code is subjected to; and trying to replicate this behaviour in library code is very painful, or outright impossible.


T

-- 
Doubt is a self-fulfilling prophecy.

On 9/29/20 2:43 PM, bachmeier wrote:
> On Tuesday, 29 September 2020 at 14:07:14 UTC, Andrei Alexandrescu wrote:
>> There is no doubt to Walter or myself that the way built-in AAs were done in D is a resounding failure.
> 
> If so, it's a pretty good resounding failure, because I like them and they would most likely not be available to me now if they hadn't been built into the language. Long ago when I used Common Lisp, there was the incredible FSet library that provided functional collections kind of like Clojure. Unfortunately few people knew about them or used them, and it doesn't look like anything has been done with them in a long time: https://common-lisp.net/project/fset/ Sometimes a library solution works, but it's in general a risky proposition to rely on libraries for such a fundamental feature.
> 
> If AAs had been introduced as a library in 2012, the repo would probably have a couple hundred open issues and the last development would have been in 2017.
> 
>> The way they should have been done was a templated library type AssociativeArray!(K, V) with the V[K] type syntax sugar on top. That would have allowed the implementation to improve along with the rest of the language and its library.
> 
> Does anything prevent the introduction of a new library solution? If it's good enough, it can eventually be added to the language.
> 
> I'm not going to pretend to be a language designer. My experience as a language user is that a big set of core functionality works better than a small set plus libraries. It's possible I don't understand your proposal.

I think there's some disconnect here -- AAs as a language feature are a resounding success. They way they are implemented in the language is not. I wouldn't call it a failure, but it definitely is not as useful as a fully supported library type could be.

-Steve

On Tuesday, 29 September 2020 at 14:07:14 UTC, Andrei Alexandrescu wrote:
> As things are now, AAs don't quite behave like any other things.

The biggest difference I see is how null works, the keyword.

void foo(string[string] aa) {}

foo(null); // works.

But that's outright impossible with a library struct.

yes, I'm bringing up my wish for implicit construction again :P

On Tuesday, 29 September 2020 at 18:53:26 UTC, bachmeier wrote:
> On Tuesday, 29 September 2020 at 13:44:55 UTC, Ola Fosheim Grøstad wrote:
> It's possible I'm wrong, but it says to me that the language developers should design the language worrying only about that 'small core of essential primitives'. Maybe 'libraries' refers to the standard library only, but that's not clear to me.

Yes, it is not terribly clear what people are talking about in this thread because many different layers of a language is being lumped together. So let me try to paint up my view of these layers:

1. On the one hand you have the type system (I guess you can call that the constraints you can put on constructs) where you probably want a small set of "axioms" from which you cannot deduce contradictions (soundness). A small core set makes it possible to prove that it is consistent.

2. On the other hand you have the conceptual "quasi-theoretical" building blocks from which you can describe everything that is expressible in the language. Keeping this small is also valuable for the same reasons. It is easier to reason about and ensure correctness/soundness and build a language that is easy to grasp. Basically, this often embeds a "modelling idea". (everything should be expressible as an object, a list, a logical proposition or a mix of two core concepts etc). If you look at the Eiffel page they claim that it is a methodology supported by a language. The same was Simula and Beta in a sense, languages supporting modelling with objects. So the minimal core language is often tied to a modelling-idea.

3. Then you have the construct that is actually implemented in the compiler, which reflects the "quasi-theoretical" language, but might be done differently for implementation reasons. Though it should obey the "laws" of the quasi-theoretical language.

4. Then you have the syntax (with syntactical sugar) that allows user to express that. Since it is a massive undertaking to let libraries define new syntax it often is a lot of "syntactical sugar", basically a few words generating many constructs in the underlying "theoretical" language.

5. Then you have the language that programmers use, which basically is what programmers often call "idiomatic" styles of programming for that particular language and that includes frequently used libraries. When newbies write bad code and complain loudly the old timers will point out that they are in essence "abusing the language construct" and "should learn to speak the language before them correctly".

So, basically, a programming language design can let the programmer believe that the compiler provides a string concept, although the underlying "theoretical language" only has lists and enumerations. So it strings might look different from lists in the source, but on a "theoretical level" it isn't.

What makes this more complicated is that the "true quasi-theoretical" language may emerge over time. As an example: I think the underlying emerging core concept in C++ is vastly different from C, that is, the most important concept in C++ has turned out to be RAII in combination with exceptions. Objects with deterministic constructors and destructors. If one were to do C++ v2, then one probably should start with that and see how small the core language could be made.

In practice compilers tend to bolt on poorly thought out "built ins" that do not share the properties of the "theoretical language" and then they stand out as sore thumbs that exhibit behaviour that is perceived as bug-like by programmers.

Until recently the "aesthetics" of programming language design has typically been to find the "one true language" that never will have to change. However, languages like typescript pretty much show that is better if programmers can configure the semantics to their specific use case in a config file. Because the context keeps changing.

Maybe one can modify a language like D to do the same, so that you could get AA or even data-science libraries imported by default so it feels more like a scripting language with that config. Clearly, embedded programming and data-science programming have different basic needs. Seems reasonable that one could have seemingly builtin AA in one config geared towards scripting-like programming.

On 9/29/20 3:32 PM, Adam D. Ruppe wrote:
> On Tuesday, 29 September 2020 at 14:07:14 UTC, Andrei Alexandrescu wrote:
>> As things are now, AAs don't quite behave like any other things.
> 
> The biggest difference I see is how null works, the keyword.
> 
> void foo(string[string] aa) {}
> 
> foo(null); // works.
> 
> But that's outright impossible with a library struct.
> 
> yes, I'm bringing up my wish for implicit construction again :P

This is certainly an odd duck. I don't really want to have the language provide all-out implicit construction, but possibly a mechanism to say "I can be implicitly constructed from null" would be useful.

null is in its own category of literals, it can change into just about anything (class, pointer, array associative array). Another nice place this would be useful is std.typecons.Nullable.

-Steve

On Monday, 28 September 2020 at 02:58:13 UTC, Bruce Carneal wrote:
> I hope and believe that there is another such advance available to us in the form of type functions.  The recently discovered alternative, reify/dereify, appears to be equivalent in power but is, comparatively, baroque.

Having been following Stefan's work on type functions for some time now, and really liking the friendly syntax that it allows, I would really appreciate a more detailed breakdown of the pros and cons when compared to this approach here.

As I understand it neither requires very significant compiler changes, and both allow us to rewrite existing Phobos templates in ways that greatly improve performance.

@Andrei would you be happy to share your thoughts in more detail on all this, and how you feel we should balance out the different concerns and interests?

On 29.09.20 01:37, Stefan Koch wrote:
> On Monday, 28 September 2020 at 21:27:56 UTC, Timon Gehr wrote:
>> On 28.09.20 23:08, claptrap wrote:
>>> Instead of first class types
>>
>> (Stefan's type functions do not give you first-class types. A first-class type you could put in a runtime variable.)
> 
> That's true. At least as it stands right now.
> I _could_ give  the alias a physical form at runtime.
> (By doing the same thing that the reify template does (or rather will eventually do), that is exporting _all_ the properties of the type into an object)
> 
> However I am not actually sure what the use of this would be.
> All usage that I have seen, is in generating code, which is kindof useless if it does not happen at compile time.
> 
> Perhaps there are others?

It's not a first-class type if you can't declare a variable of that type. If this does not work, it's not first-class, it's syntax sugar for reification:

type t = int;
auto f = (t x)=>x;

On Thu, Oct 01, 2020 at 01:17:27AM +0200, Timon Gehr via Digitalmars-d wrote:
> On 29.09.20 01:37, Stefan Koch wrote:
[...]
> > That's true. At least as it stands right now.
> > I _could_ give  the alias a physical form at runtime.
> > (By doing the same thing that the reify template does (or rather
> > will eventually do), that is exporting _all_ the properties of the
> > type into an object)
> > 
> > However I am not actually sure what the use of this would be.
> > All usage that I have seen, is in generating code, which is kindof
> > useless if it does not happen at compile time.
> > 
> > Perhaps there are others?
> 
> It's not a first-class type if you can't declare a variable of that type. If this does not work, it's not first-class, it's syntax sugar for reification:
> 
> type t = int;
> auto f = (t x)=>x;

I think there's a lot of value to be had in making typeid(T) as the
reification of T.  At compile-time, we treat it specially by augmenting
it with the ability to do things that can only be done at compile-time,
such as recover T given typeid(T) (pass it into a template parameter,
construct new types out of it, etc.). At runtime, it reverts to the
current behaviour of typeid(T).

Only trouble is, according to Andrei, typeid has been written over so many times that even Walter doesn't understand how it works anymore. Meaning there are probably weird corner cases and quirky behaviours that we may not want to duplicate at compile-time. So I dunno, this seems like a roadblock to further progress.  (It really makes one wish for a gradual transition to D3 where we can correct such deeply-rooted flaws that may otherwise be uncorrectible.)


T

-- 
To provoke is to call someone stupid; to argue is to call each other stupid.

On 9/30/20 7:36 PM, H. S. Teoh wrote:
> I think there's a lot of value to be had in making typeid(T) as the
> reification of T.  At compile-time, we treat it specially by augmenting
> it with the ability to do things that can only be done at compile-time,
> such as recover T given typeid(T) (pass it into a template parameter,
> construct new types out of it, etc.). At runtime, it reverts to the
> current behaviour of typeid(T).
> 
> Only trouble is, according to Andrei, typeid has been written over so
> many times that even Walter doesn't understand how it works anymore.
> Meaning there are probably weird corner cases and quirky behaviours that
> we may not want to duplicate at compile-time. So I dunno, this seems
> like a roadblock to further progress.

Yah, that would be my favorite by far. It's a one-liner change in the definition of the language. "Since version x.xx, the result of typeid() is usable during compilation." It's also the best /kind/ of change to the language, i.e. lifts a gratuitous restriction that prevents composition of distinct features (here, ctfe and typeid).

A promising path according to Walter is to develop a parallel enhanced feature - newtypeid! :o) - and then deprecate typeid.