On Wednesday 13 October 2010 23:58:16 Jonathan M Davis wrote:
> The bugs forum is not for posting to.

Ack. I meant the bug mailing list, not forum. Oh well, it should be fairly obvious what I meant.

- Jonathan M Davis

Am 13.10.2010 21:45, schrieb dsimcha:
> I've noticed that, when passed multiple files at once, DMD is generally buggy
> in ways that I can't reproduce in small test cases.  This includes things like
> magically ignoring __gshared variables, and not being able to convert a type
> to an alias to the same type, for example, not being able to convert a size_t
> to a uint on 32-bit, or not being able to convert a float[] to an R where R is
> a template parameter instantiated as float[].
>
> I have two questions:
>
> 1.  Has anyone experienced similar things and if so are they further along
> than me at creating decent test cases?
>
> 2.  I'm used to using Code::Blocks, but want to switch to Visual D because it
> seems to work a lot better for the most part.  However, by default it seems to
> pass all project files to the compiler at once, where Code::Blocks compiles
> one file at a time.  This means I have to deal with DMD's bugginess when
> passed mutliple files at once.  Does anyone know how to make Visual D work
> Code::Blocks style, i.e. compile each file to an object file and link them
> afterwords?

Right now, I'm in exactly the same situation. However, the problems might be a bit more obvious: Sometimes symbols are missing at link time that should not and sometimes some symbols (most of the time phobos symbols) are defined twice but with exactly the same module path.

(Error: multiple definition of conv_208_199f: _D53C:\D\dmd2\windows\bin\..\..\src\phobos\std\conv.d.52012__ModuleInfoZ and conv: _D53C:\D\dmd2\windows\bin\..\..\src\phobos\std\conv.d.52012__ModuleInfoZ)

On Mac OS, using all-at-once compilation I get defective .lib files (the ar tool complains that the .o file is smaller than it reports or something similar) so I have to compile the code of all static libs together with the application code.

The problem is that it is really difficult (almost impossible) to isolate the problem in any way because sometimes appears erratically and the code base is quite large. And also I cannot share the source code for debugging.

I'll push the issue up the todo list. Maybe I'm not getting a lot of these errors because I have applied the dmd patches in the bug reports 190 and 4753...


dsimcha wrote:
> == Quote from Rainer Schuetze (r.sagitario@gmx.de)'s article
>> Could it be related to any of these?
>> http://d.puremagic.com/issues/show_bug.cgi?id=190
>> http://d.puremagic.com/issues/show_bug.cgi?id=3979
> 
> Yep, these look like at least part of the problem.  Glad someone else has already
> reduced them to sane test cases so I don't have to.  Reducing compiler bugs that
> only seem to occur in non-trivial, multi-module projects to decent test cases is a
> huge PITA.
> 
> Anyhow, the project in question is a very messy codebase because it grew very
> organically.  It's basically a haphazard collection of research prototype
> algorithms for predicting gene expression from DNA sequence, and every time I
> think of a new idea, I tend to just put it wherever I can make it fit and almost
> never bother with non-trivial refactoring.  Therefore, there are cyclic imports
> **everywhere**.
> 
>> I have that option on my todo list, but didn't implement it so far
>> because I was not aware of any problems with compiling multiple files.
>> I don't know if this is really an option for a larger project, but what
>> you can do is select "Custom Build Tool" for all files in the "Common
>> Properties" setting of the project property dialog and enter your own
>> command line and output file. But be warned: you will not get any
>> automatic dependency detection.
>> As editing the settings for a lot of files might be very annoying, you
>> might want to set it for one file and then edit the project file with a
>> text editor to copy the settings to other files.
>> Rainer
> 
> Since build process automation is by far the biggest reason why I use an IDE
> instead of a plain old editor, I'd rather just stick with Code::Blocks for now.
> Could you please bump this up the todo list, given that building multiple files
> simultaneously is buggy in ways that probably aren't going to get fixed too soon?

Here's a preliminary new version that includes an option on the general project configuration page to switch to single file compilation:

http://www.dsource.org/projects/visuald/browser/downloads/VisualD-v0.3.18rc1.exe

I've used ddmd as a medium sized project for testing. It has more than 400 files with most modules mutually importing each other. Some remarks:

- compilation speed is a lot worse, even when compared to what you would expect from a similar C/C++ project. It takes almost 8 minutes to compile ddmd on my system with single file compilation, while the standard compilation takes about 10 seconds.

- the dependency files written by dmd are so bloated that the check that selects the files to compile takes almost 20 seconds. This is because the size of each dependency file is about 700 kB, summing up to 220 MB to scan for imported files. (I had to do some optimizations and file operation caching to get the time down from several minutes).

- I consider the single compilation mode only sensible for a large collection of modules that have only limited dependencies.

- probably, compilation speed could be way better if semantic analysis would only be done on symbols that are actually directly or indirectly referred to by the modules passed on the command line. This would need a slight modification of the semantic analysis, but would also put the promise of independence of lexical order to the test - and would expose more of the problems you are currently getting.

This new version also includes some experimental debug/version statement highlighting, which has not yet been extensively tested. If it causes too many troubles, it can be disabled on the global property page Tools->Options->TextEditor->D->Colorizer.

Rainer


dsimcha wrote:
> == Quote from Rainer Schuetze (r.sagitario@gmx.de)'s article
>> Could it be related to any of these?
>> http://d.puremagic.com/issues/show_bug.cgi?id=190
>> http://d.puremagic.com/issues/show_bug.cgi?id=3979
> 
> Yep, these look like at least part of the problem.  Glad someone else has already
> reduced them to sane test cases so I don't have to.  Reducing compiler bugs that
> only seem to occur in non-trivial, multi-module projects to decent test cases is a
> huge PITA.
> 
> Anyhow, the project in question is a very messy codebase because it grew very
> organically.  It's basically a haphazard collection of research prototype
> algorithms for predicting gene expression from DNA sequence, and every time I
> think of a new idea, I tend to just put it wherever I can make it fit and almost
> never bother with non-trivial refactoring.  Therefore, there are cyclic imports
> **everywhere**.
> 
>> I have that option on my todo list, but didn't implement it so far
>> because I was not aware of any problems with compiling multiple files.
>> I don't know if this is really an option for a larger project, but what
>> you can do is select "Custom Build Tool" for all files in the "Common
>> Properties" setting of the project property dialog and enter your own
>> command line and output file. But be warned: you will not get any
>> automatic dependency detection.
>> As editing the settings for a lot of files might be very annoying, you
>> might want to set it for one file and then edit the project file with a
>> text editor to copy the settings to other files.
>> Rainer
> 
> Since build process automation is by far the biggest reason why I use an IDE
> instead of a plain old editor, I'd rather just stick with Code::Blocks for now.
> Could you please bump this up the todo list, given that building multiple files
> simultaneously is buggy in ways that probably aren't going to get fixed too soon?

Cool stuff, Rainer. Btw, I've noticed there's a new debugger installed with newer versions of VisualD (mago?). Do I need to set it up to be used in VS over the default one or is it selected by default after installation?

On 10/17/10, Rainer Schuetze <r.sagitario@gmx.de> wrote:
> Here's a preliminary new version that includes an option on the general project configuration page to switch to single file compilation:
>
> http://www.dsource.org/projects/visuald/browser/downloads/VisualD-v0.3.18rc1.exe
>
> I've used ddmd as a medium sized project for testing. It has more than 400 files with most modules mutually importing each other. Some remarks:
>
> - compilation speed is a lot worse, even when compared to what you would expect from a similar C/C++ project. It takes almost 8 minutes to compile ddmd on my system with single file compilation, while the standard compilation takes about 10 seconds.
>
> - the dependency files written by dmd are so bloated that the check that selects the files to compile takes almost 20 seconds. This is because the size of each dependency file is about 700 kB, summing up to 220 MB to scan for imported files. (I had to do some optimizations and file operation caching to get the time down from several minutes).
>
> - I consider the single compilation mode only sensible for a large collection of modules that have only limited dependencies.
>
> - probably, compilation speed could be way better if semantic analysis would only be done on symbols that are actually directly or indirectly referred to by the modules passed on the command line. This would need a slight modification of the semantic analysis, but would also put the promise of independence of lexical order to the test - and would expose more of the problems you are currently getting.
>
> This new version also includes some experimental debug/version statement highlighting, which has not yet been extensively tested. If it causes too many troubles, it can be disabled on the global property page Tools->Options->TextEditor->D->Colorizer.
>
> Rainer
>
>
> dsimcha wrote:
>> == Quote from Rainer Schuetze (r.sagitario@gmx.de)'s article
>>> Could it be related to any of these? http://d.puremagic.com/issues/show_bug.cgi?id=190 http://d.puremagic.com/issues/show_bug.cgi?id=3979
>>
>> Yep, these look like at least part of the problem.  Glad someone else has
>> already
>> reduced them to sane test cases so I don't have to.  Reducing compiler
>> bugs that
>> only seem to occur in non-trivial, multi-module projects to decent test
>> cases is a
>> huge PITA.
>>
>> Anyhow, the project in question is a very messy codebase because it grew
>> very
>> organically.  It's basically a haphazard collection of research prototype
>> algorithms for predicting gene expression from DNA sequence, and every
>> time I
>> think of a new idea, I tend to just put it wherever I can make it fit and
>> almost
>> never bother with non-trivial refactoring.  Therefore, there are cyclic
>> imports
>> **everywhere**.
>>
>>> I have that option on my todo list, but didn't implement it so far
>>> because I was not aware of any problems with compiling multiple files.
>>> I don't know if this is really an option for a larger project, but what
>>> you can do is select "Custom Build Tool" for all files in the "Common
>>> Properties" setting of the project property dialog and enter your own
>>> command line and output file. But be warned: you will not get any
>>> automatic dependency detection.
>>> As editing the settings for a lot of files might be very annoying, you
>>> might want to set it for one file and then edit the project file with a
>>> text editor to copy the settings to other files.
>>> Rainer
>>
>> Since build process automation is by far the biggest reason why I use an
>> IDE
>> instead of a plain old editor, I'd rather just stick with Code::Blocks for
>> now.
>> Could you please bump this up the todo list, given that building multiple
>> files
>> simultaneously is buggy in ways that probably aren't going to get fixed
>> too soon?
>

Awesome.  I frankly don't care about compile speeds because the projects in question are fairly small, so even with a horribly inefficient build process build times are negligible.  FWIW I use the one at a time builds in CodeBlocks, too, though I parallelize them on my quad core.

I only wanted the one at a time compilation feature as a temporary workaround for compiler bugs that will hopefully be fixed by the time people start using D2 for larger projects.  I'll try Visual D again tomorrow and provide some feedback. From my limited playing around with it, I've already noticed that the debugger works a ton better than any other D debugger I've tried, i.e. possibly good enough that I'll stop using writeln as my main debugging method.

== Quote from Rainer Schuetze (r.sagitario@gmx.de)'s article
> Here's a preliminary new version that includes an option on the general
> project configuration page to switch to single file compilation:
> http://www.dsource.org/projects/visuald/browser/downloads/VisualD-v0.3.18rc1.exe
> I've used ddmd as a medium sized project for testing. It has more than
> 400 files with most modules mutually importing each other. Some remarks:
> - compilation speed is a lot worse, even when compared to what you would
> expect from a similar C/C++ project. It takes almost 8 minutes to
> compile ddmd on my system with single file compilation, while the
> standard compilation takes about 10 seconds.
> - the dependency files written by dmd are so bloated that the check that
> selects the files to compile takes almost 20 seconds. This is because
> the size of each dependency file is about 700 kB, summing up to 220 MB
> to scan for imported files. (I had to do some optimizations and file
> operation caching to get the time down from several minutes).
> - I consider the single compilation mode only sensible for a large
> collection of modules that have only limited dependencies.
> - probably, compilation speed could be way better if semantic analysis
> would only be done on symbols that are actually directly or indirectly
> referred to by the modules passed on the command line. This would need a
> slight modification of the semantic analysis, but would also put the
> promise of independence of lexical order to the test - and would expose
> more of the problems you are currently getting.
> This new version also includes some experimental debug/version statement
> highlighting, which has not yet been extensively tested. If it causes
> too many troubles, it can be disabled on the global property page
> Tools->Options->TextEditor->D->Colorizer.
> Rainer
> dsimcha wrote:
> > == Quote from Rainer Schuetze (r.sagitario@gmx.de)'s article
> >> Could it be related to any of these? http://d.puremagic.com/issues/show_bug.cgi?id=190 http://d.puremagic.com/issues/show_bug.cgi?id=3979
> >
> > Yep, these look like at least part of the problem.  Glad someone else has already reduced them to sane test cases so I don't have to.  Reducing compiler bugs that only seem to occur in non-trivial, multi-module projects to decent test cases is a huge PITA.
> >
> > Anyhow, the project in question is a very messy codebase because it grew very organically.  It's basically a haphazard collection of research prototype algorithms for predicting gene expression from DNA sequence, and every time I think of a new idea, I tend to just put it wherever I can make it fit and almost never bother with non-trivial refactoring.  Therefore, there are cyclic imports **everywhere**.
> >
> >> I have that option on my todo list, but didn't implement it so far
> >> because I was not aware of any problems with compiling multiple files.
> >> I don't know if this is really an option for a larger project, but what
> >> you can do is select "Custom Build Tool" for all files in the "Common
> >> Properties" setting of the project property dialog and enter your own
> >> command line and output file. But be warned: you will not get any
> >> automatic dependency detection.
> >> As editing the settings for a lot of files might be very annoying, you
> >> might want to set it for one file and then edit the project file with a
> >> text editor to copy the settings to other files.
> >> Rainer
> >
> > Since build process automation is by far the biggest reason why I use an IDE instead of a plain old editor, I'd rather just stick with Code::Blocks for now. Could you please bump this up the todo list, given that building multiple files simultaneously is buggy in ways that probably aren't going to get fixed too soon?

Andrej Mitrovic wrote:
> Cool stuff, Rainer. Btw, I've noticed there's a new debugger installed
> with newer versions of VisualD (mago?). Do I need to set it up to be
> used in VS over the default one or is it selected by default after
> installation?
> 

If you did not uncheck the mago debugger when installing, you can simply select the debug engine to be used on the project property page "Debugging" (visual studio internal debugger or mago).

I've just tried it. It doesn't seem to be able to find any locals, even if I add a variable to a watchlist it can't find the symbol. I guess this is still experimental. Anyway, thanks for the info.

On 10/17/10, Rainer Schuetze <r.sagitario@gmx.de> wrote:
>
> Andrej Mitrovic wrote:
>> Cool stuff, Rainer. Btw, I've noticed there's a new debugger installed with newer versions of VisualD (mago?). Do I need to set it up to be used in VS over the default one or is it selected by default after installation?
>>
>
> If you did not uncheck the mago debugger when installing, you can simply select the debug engine to be used on the project property page "Debugging" (visual studio internal debugger or mago).
>

That's an unfortunate regression in the linker that comes with dmd 2.049. There's already an updated linker available (http://ftp.digitalmars.com/link.8.00.8.zip). See also the project page http://dsource.org/projects/mago_debugger for more info.

Andrej Mitrovic wrote:
> I've just tried it. It doesn't seem to be able to find any locals,
> even if I add a variable to a watchlist it can't find the symbol. I
> guess this is still experimental. Anyway, thanks for the info.
> 
> On 10/17/10, Rainer Schuetze <r.sagitario@gmx.de> wrote:
>> Andrej Mitrovic wrote:
>>> Cool stuff, Rainer. Btw, I've noticed there's a new debugger installed
>>> with newer versions of VisualD (mago?). Do I need to set it up to be
>>> used in VS over the default one or is it selected by default after
>>> installation?
>>>
>> If you did not uncheck the mago debugger when installing, you can simply
>> select the debug engine to be used on the project property page
>> "Debugging" (visual studio internal debugger or mago).
>>

Ah, ok. I'll have a look, thanks.

On 10/17/10, Rainer Schuetze <r.sagitario@gmx.de> wrote:
> That's an unfortunate regression in the linker that comes with dmd 2.049. There's already an updated linker available (http://ftp.digitalmars.com/link.8.00.8.zip). See also the project page http://dsource.org/projects/mago_debugger for more info.
>
> Andrej Mitrovic wrote:
>> I've just tried it. It doesn't seem to be able to find any locals, even if I add a variable to a watchlist it can't find the symbol. I guess this is still experimental. Anyway, thanks for the info.
>>
>> On 10/17/10, Rainer Schuetze <r.sagitario@gmx.de> wrote:
>>> Andrej Mitrovic wrote:
>>>> Cool stuff, Rainer. Btw, I've noticed there's a new debugger installed with newer versions of VisualD (mago?). Do I need to set it up to be used in VS over the default one or is it selected by default after installation?
>>>>
>>> If you did not uncheck the mago debugger when installing, you can simply select the debug engine to be used on the project property page "Debugging" (visual studio internal debugger or mago).
>>>
>