Hi,

In the introduction there is a section named 'Phases of Compilation'. Following the convention used in C standard, I am calling this 'Translation Phases'; translation is a more generic term too. See https://github.com/dlang/dlang.org/pull/2646/files for some initial rewording.

The recent post https://forum.dlang.org/post/xnoeazcqnysodqnjphhc@forum.dlang.org got me thinking that actually we should expand the 'Translation Phases' section to describe more formally how D code is parsed. I had a quick look at the D parsing code. It seems that following an initial parse - presumably this constructs the syntax tree - there are three deferred semantic parses. Are any details available on what actually happens in each pass? I can read the code of course but it would be helpful to get a high level picture.

Thanks and Regards
Dibyendu

On Sunday, 19 May 2019 at 10:07:37 UTC, Dibyendu Majumdar wrote:
> I had a quick look at the D parsing code. It seems that following an initial parse - presumably this constructs the syntax tree - there are three deferred semantic parses. Are any details available on what actually happens in each pass? I can read the code of course but it would be helpful to get a high level picture.
>
> Thanks and Regards
> Dibyendu

Walter Bright's DConf 2016 talk, "Spelunking D Compuler Internals", is a good resource for this:

https://www.youtube.com/watch?v=l_96Crl998E

On Sunday, 19 May 2019 at 11:39:02 UTC, Paul Backus wrote:
> Walter Bright's DConf 2016 talk, "Spelunking D Compuler Internals", is a good resource for this:
>
> https://www.youtube.com/watch?v=l_96Crl998E

Thank you - I watched it again just now. It was very helpful.

I still need to understand more about the semantic analysis phase, but I don't yet know what questions to ask.

I guess to start with one point seems to be that after the syntax analysis phase, we simply have a abstract syntax tree that corresponds one to one with the input source - is that correct? So at this stage nothing more is known about the program - only the input is in a different format that is more amenable to further analysis.

Secondly - I assume that the syntax tree "lowering" (or simplification) is done prior to the semantic analysis?

Thanks and Regards
Dibyendu

On Sunday, 19 May 2019 at 14:26:34 UTC, Dibyendu Majumdar wrote:
> On Sunday, 19 May 2019 at 11:39:02 UTC, Paul Backus wrote:
>> [...]
>
> Thank you - I watched it again just now. It was very helpful.
>
> I still need to understand more about the semantic analysis phase, but I don't yet know what questions to ask.
>
> I guess to start with one point seems to be that after the syntax analysis phase, we simply have a abstract syntax tree that corresponds one to one with the input source - is that correct? So at this stage nothing more is known about the program - only the input is in a different format that is more amenable to further analysis.
>
> Secondly - I assume that the syntax tree "lowering" (or simplification) is done prior to the semantic analysis?
>
> Thanks and Regards
> Dibyendu

Only use the code to work out what the current behaviour: The specification needs to be totally indpendant of the current implementations ('s ideology).

On Sunday, 19 May 2019 at 16:41:22 UTC, Max Haughton wrote:
> Only use the code to work out what the current behaviour: The specification needs to be totally indpendant of the current implementations ('s ideology).

Yes I understand that, but there may some some aspects implicit in the implementation that should be made more explicit in the specification. Maybe I need to come back to this topic after I have done the other sections as I don't know what I don't know.

Regards

On Sunday, 19 May 2019 at 16:52:10 UTC, Dibyendu Majumdar wrote:
> On Sunday, 19 May 2019 at 16:41:22 UTC, Max Haughton wrote:
>> Only use the code to work out what the current behaviour: The specification needs to be totally indpendant of the current implementations ('s ideology).
>
> Yes I understand that, but there may some some aspects implicit in the implementation that should be made more explicit in the specification. Maybe I need to come back to this topic after I have done the other sections as I don't know what I don't know.
>
> Regards

The current compilation model of the front end is said to be "lazy" but a eager compilation would produce the same program without changing the semantics.
I think that this is for example what Max Haughton meant.

Saying that the semantic is multi pass due to some language feature such as string mixin, template mixin, mutually dependent declarations, (etc.) would be enough. It must be just clear that a single pass, following the lexical order top to bottom, would not work at all.

On 5/19/2019 3:07 AM, Dibyendu Majumdar wrote:
> Hi,
> 
> In the introduction there is a section named 'Phases of Compilation'. Following the convention used in C standard, I am calling this 'Translation Phases'; translation is a more generic term too. See https://github.com/dlang/dlang.org/pull/2646/files for some initial rewording.
> 
> The recent post https://forum.dlang.org/post/xnoeazcqnysodqnjphhc@forum.dlang.org got me thinking that actually we should expand the 'Translation Phases' section to describe more formally how D code is parsed. I had a quick look at the D parsing code. It seems that following an initial parse - presumably this constructs the syntax tree - there are three deferred semantic parses. Are any details available on what actually happens in each pass? I can read the code of course but it would be helpful to get a high level picture.

The Translation Phases are conceptual, the compiler may not actually do them that way, it's just supposed to be "as if" they were done that way.

For example, the Digital Mars C/C++ compilers merge several of the translation phases.

On Monday, 20 May 2019 at 07:34:43 UTC, Walter Bright wrote:
> The Translation Phases are conceptual, the compiler may not actually do them that way, it's just supposed to be "as if" they were done that way.
>

Okay thank you. From the spec point of view I guess the important thing is to clarify any constraints this places on D. One constraint obviously is that the parsing into syntax tree requires arbitrary lookahead of tokens. Are there other constraints that need to be highlighted?

Regards

On 5/20/2019 4:51 AM, Dibyendu Majumdar wrote:
> On Monday, 20 May 2019 at 07:34:43 UTC, Walter Bright wrote:
>> The Translation Phases are conceptual, the compiler may not actually do them that way, it's just supposed to be "as if" they were done that way.
>>
> 
> Okay thank you. From the spec point of view I guess the important thing is to clarify any constraints this places on D. One constraint obviously is that the parsing into syntax tree requires arbitrary lookahead of tokens. Are there other constraints that need to be highlighted?

The spec isn't a tutorial in that the consequences don't necessarily need to be spelled out. The crucial thing to know is that the tokenizing is independent of parsing, and parsing is independent of semantic analysis.

I.e. the definition of a token does not change depending on what construct is being parsed, and the AST generated by the parser can be created without doing any semantic analysis (unlike C++).

These consequences fall out of the rest of the spec, hence they should be more of a clarification in the introduction. The idea is to head off attempts to add changes to D that introduce dependencies. Such proposals do crop up from time to time, for example, user-defined operator tokens.

On Sunday, 19 May 2019 at 16:52:10 UTC, Dibyendu Majumdar wrote:
> On Sunday, 19 May 2019 at 16:41:22 UTC, Max Haughton wrote:
>> Only use the code to work out what the current behaviour: The specification needs to be totally indpendant of the current implementations ('s ideology).
>
> Yes I understand that, but there may some some aspects implicit in the implementation that should be made more explicit in the specification. Maybe I need to come back to this topic after I have done the other sections as I don't know what I don't know.
>
> Regards

You're right, that is definitely an issue with the current specification. One of the really annoying issues is that dmd is understandable locally (e.g. I was able to add a little feature without looking much up) but the global structure of the code is quite disjoint e.g. lots of the analysis code is lumped together and fairly difficult to grok without watching a debugger go through it (Unless you already know)