On 22.05.2012 19:36, Roman D. Boiko wrote:
> On Tuesday, 22 May 2012 at 15:15:49 UTC, Roman D. Boiko wrote:
>> On Tuesday, 22 May 2012 at 14:56:42 UTC, Dmitry Olshansky wrote:
>>> I suggest to go with such kind of improvised notation with more
>>> examples until you fell that semantics are crystal clear.
>>
>> This is a possible, although not the only, option. Anyway I'm not
>> ready for designing the details yet. There is a lot to do before
>> that.
>
> After thinking a bit more, I decided to investigate the topic
> early, although not immediately :) It should help deciding which
> indices are needed, where are rough edges of API, etc.

Yeah I still think you tried to dismiss it too early.

In my mind it's encountered rather soon:

lex/scan -> postprocess (may be combined with lex) -> populate symbol tables (ditto  - with next step/previous step) -> parse to AST -> ...

That symbol table should contain all of the rich hierarchy of modules. That is the actual compiler is able to have a single stack of scopes that it pushes/pops as it processes code. Your DCT on the other hand should have all of local scopes (of every entity) at once.

It may be possible to simulate it with some deal of laziness, but I guess keeping the whole symbol table is the easiest and the fastest way still. Watch out for some mind-bending data structures :)

-- 
Dmitry Olshansky

On Tuesday, 22 May 2012 at 16:03:49 UTC, Dmitry Olshansky wrote:
> In my mind it's encountered rather soon:
>
> lex/scan -> postprocess (may be combined with lex) -> populate symbol tables (ditto  - with next step/previous step) -> parse to AST -> ...
> That symbol table should contain all of the rich hierarchy of modules. That is the actual compiler is able to have a single stack of scopes that it pushes/pops as it processes code. Your DCT on the other hand should have all of local scopes (of every entity) at once.
>
> It may be possible to simulate it with some deal of laziness, but I guess keeping the whole symbol table is the easiest and the fastest way still.
And likely the only feasible in D. It doesn't support lazy evaluation for immutable data structures, and immutability is necessary for most use cases.

> Watch out for some mind-bending data structures :)
Do you mean not to overcomplicate? Or use classic data structures? Or something else?

So far I think immutable red-black trees will be central in DCT architecture, as well as some others.

Le 22/05/2012 16:04, Roman D. Boiko a écrit :
> http://d-coding.com/2012/05/22/dct-use-cases.html

Nice idea but I'll not be nice.

What is needed here is design. The needs are not really news nor specific to D.

So far, you have ignored all existing project on the subject, and shown that you tend to overcomplicate design when it isn't required. I don't see here the right attitude to come up with a tool that will :
1/ Not repeat error previously done.
2/ Get resused.

On 22.05.2012 20:47, Roman D. Boiko wrote:
> On Tuesday, 22 May 2012 at 16:03:49 UTC, Dmitry Olshansky wrote:
>> In my mind it's encountered rather soon:
>>
>> lex/scan -> postprocess (may be combined with lex) -> populate symbol
>> tables (ditto - with next step/previous step) -> parse to AST -> ...
>> That symbol table should contain all of the rich hierarchy of modules.
>> That is the actual compiler is able to have a single stack of scopes
>> that it pushes/pops as it processes code. Your DCT on the other hand
>> should have all of local scopes (of every entity) at once.
>>
>> It may be possible to simulate it with some deal of laziness, but I
>> guess keeping the whole symbol table is the easiest and the fastest
>> way still.
> And likely the only feasible in D. It doesn't support lazy evaluation
> for immutable data structures, and immutability is necessary for most
> use cases.
>
>> Watch out for some mind-bending data structures :)
> Do you mean not to overcomplicate? Or use classic data structures? Or
> something else?

Prefer simple for prototype. Good old nested hash table is fine for starters.

>
> So far I think immutable red-black trees will be central in DCT
> architecture, as well as some others.

Cool ;) Though why not Tries then if the data is immutable?

-- 
Dmitry Olshansky

On Tuesday, 22 May 2012 at 14:04:18 UTC, Roman D. Boiko wrote:
> http://d-coding.com/2012/05/22/dct-use-cases.html

I'm really not sure I understand what this is. Is this use cases for a lexer or some other more high level too?. Because I don't see what projects and workspaces have to do with a lexer.

On Tuesday, 22 May 2012 at 16:55:46 UTC, Dmitry Olshansky wrote:
> On 22.05.2012 20:47, Roman D. Boiko wrote:
>> On Tuesday, 22 May 2012 at 16:03:49 UTC, Dmitry Olshansky wrote:
>>> In my mind it's encountered rather soon:
>>>
>>> lex/scan -> postprocess (may be combined with lex) -> populate symbol
>>> tables (ditto - with next step/previous step) -> parse to AST -> ...
>>> That symbol table should contain all of the rich hierarchy of modules.
>>> That is the actual compiler is able to have a single stack of scopes
>>> that it pushes/pops as it processes code. Your DCT on the other hand
>>> should have all of local scopes (of every entity) at once.
>>>
>>> It may be possible to simulate it with some deal of laziness, but I
>>> guess keeping the whole symbol table is the easiest and the fastest
>>> way still.
>> And likely the only feasible in D. It doesn't support lazy evaluation
>> for immutable data structures, and immutability is necessary for most
>> use cases.
>>
>>> Watch out for some mind-bending data structures :)
>> Do you mean not to overcomplicate? Or use classic data structures? Or
>> something else?
>
> Prefer simple for prototype. Good old nested hash table is fine for starters.
Unfortunately, hash tables don't play nicely with immutability of data - although interface could be restricted to prevent mutation, they would not be able to reuse memory space, and thus would require copying.

>>
>> So far I think immutable red-black trees will be central in DCT
>> architecture, as well as some others.
>
> Cool ;) Though why not Tries then if the data is immutable?

Those too :) For different purposes than the former.

On Tuesday, 22 May 2012 at 17:06:43 UTC, Jacob Carlborg wrote:
> On Tuesday, 22 May 2012 at 14:04:18 UTC, Roman D. Boiko wrote:
>> http://d-coding.com/2012/05/22/dct-use-cases.html
>
> I'm really not sure I understand what this is. Is this use cases for a lexer or some other more high level too?. Because I don't see what projects and workspaces have to do with a lexer.

This is a draft of use cases for all DCT libraries combined.

Scope for DCT is to provide semantic analysis, but not code generation (that may become another project some time). Information about projects, etc., is useful for e.g., analysing dependencies.

I'll improve overall structure and add some explanations + examples tomorrow. Could you elaborate on specific points which are vague?

On 2012-05-22 19:14, Roman D. Boiko wrote:

> This is a draft of use cases for all DCT libraries combined.

This seems to be mostly focused on lexing? See below for some ideas.

> Scope for DCT is to provide semantic analysis, but not code generation
> (that may become another project some time). Information about projects,
> etc., is useful for e.g., analysing dependencies.

That's a good point.

> I'll improve overall structure and add some explanations + examples
> tomorrow. Could you elaborate on specific points which are vague?

I would probably have specified some high level use cases first, like:

* IDE integration
* Refactoring tool
* Static analysis
* Compiler
* Doc generating
* Build tool

In general, use cases that can span several compile phases, i.e. lexing, parsing, semantic analysis and so on. Some of these use cases can be broken in to several new use cases at a lower level. Some examples:

IDE integration:

* Syntax highlighting
* Code completion
* Showing lex, syntax and semantic errors

Refactoring:

* Cross-referencing symbols

Build tool:

* Tracking module dependencies

Doc generating:

* Associate a declaration and its documentation

Some of these "sub" use cases are needed by several tools, then you can either repeat them or pick unique sub use cases for each high level use case.

Then you can get into more detail over lower level use cases for the different compile phases. If you have enough to write you could probably have a post about the use cases for each phase.

It seems some of your use cases are implementation details or design goals, like "Store text efficiently".

It would not be necessary to start with the high level goals, but it would be nice. The next best thing would probably be to start with the use cases compiler phase you already have started on, that is lexing, if I have understood everything correctly.

-- 
/Jacob Carlborg

On Tuesday, 22 May 2012 at 18:10:59 UTC, Jacob Carlborg wrote:
> On 2012-05-22 19:14, Roman D. Boiko wrote:
>
>> This is a draft of use cases for all DCT libraries combined.
>
> This seems to be mostly focused on lexing? See below for some ideas.
Yeah, about 50% is lexing. I pay more attention to it because lexer alone is enough for several uses. I would like to have at least some functionality used as early as possible, this would provide me great feedback.

>> Scope for DCT is to provide semantic analysis, but not code generation (that may become another project some time). Information about projects, etc., is useful for e.g., analysing dependencies.
>
> That's a good point.
>
>> I'll improve overall structure and add some explanations + examples
>> tomorrow. Could you elaborate on specific points which are vague?
>
> I would probably have specified some high level use cases first, like:
>
> * IDE integration
> * Refactoring tool
> * Static analysis
> * Compiler
> * Doc generating
> * Build tool
Thanks! I didn't think about build tool, for exapmle.

I started this way, but after your comment on my previous post that there is nothing new I reconsidered my approach and decided to start from concrete (low-lewel), then improve it according to feedback, and then split into areas (which roughly correspond to your hi-level use cases).

> In general, use cases that can span several compile phases, i.e. lexing, parsing, semantic analysis and so on. Some of these use cases can be broken in to several new use cases at a lower level. Some examples:
>
> IDE integration:
>
> * Syntax highlighting
> * Code completion
> * Showing lex, syntax and semantic errors
>
> Refactoring:
>
> * Cross-referencing symbols
>
> Build tool:
>
> * Tracking module dependencies
>
> Doc generating:
>
> * Associate a declaration and its documentation
>
> Some of these "sub" use cases are needed by several tools, then you can either repeat them or pick unique sub use cases for each high level use case.
>
> Then you can get into more detail over lower level use cases for the different compile phases. If you have enough to write you could probably have a post about the use cases for each phase.

Thanks for examples.

> It seems some of your use cases are implementation details or design goals, like "Store text efficiently".
Actually, many of those are architectural (although low-level), because they are key to achieve the project goals, and failing in this area could cause overall failure. I intend to move any non-architectural information into a separate series of posts, feel free commenting what you don't consider important for the architecture (probably don't start yet, I'm reviewing text right now).

> It would not be necessary to start with the high level goals, but it would be nice. The next best thing would probably be to start with the use cases compiler phase you already have started on, that is lexing, if I have understood everything correctly.

Yes, and even before that I'm going to document some fundamental primitives, like immutability and core data structures.

On 2012-05-22 20:33, Roman D. Boiko wrote:

> Yes, and even before that I'm going to document some fundamental
> primitives, like immutability and core data structures.

Wouldn't it be better to start with the use cases? You probably already have a fairly good idea about the use cases, but in theory the use cases could change how the data structure might look like.

-- 
/Jacob Carlborg