On 5/30/2011 9:54 PM, Benjamin Shropshire wrote:
> On 05/29/2011 04:19 PM, Brad Roberts wrote:
>> On 5/29/2011 4:07 PM, Robert Clipsham wrote:
>>> On 29 May 2011 19:13, Walter Bright<walter at digitalmars.com<mailto:walter at digitalmars.com>>  wrote:
>>>
>>>      For those doing compiler patches, I know it's a tough job, and thanks!
>>>
>>>      But please consider running the dmd test suite (and try to build/run phobos unittests) before issuing a pull
>>>      request. There are a lot of interdependencies in the compiler, and the test suite is designed to flush them out.
>>> The
>>>      cool thing is the test suite consists of cases that are already minimized! The suite also is designed to not take
>>>      that long to run.
>>>
>>>      I run it constantly when I do dev on the compiler, and I constantly get dinged by it for something I overlooked. I
>>>      consider the D test suite to be a major asset for us.
>>>
>>>      Please use it. If you've got problems using it, please post here and we'll try to help.
>>>
>>>
>>> An idea relayed from Vladimir Panteleev: It could be a good idea to integrate pull requests with the automatic test runner - github has APIs for pull requests, the test runner could pull out the commits, test them, then post a comment on the pull request.
>>>
>>> I see a few problems with this however:
>>>   - Added load for the test runner (although, they could be run with low priority and only when there aren't commits to
>>> master to test)
>>>   - Security - if you're testing all pull requests some sort of sandbox would be needed so the patch isn't running rm
>>> -rf
>>> or something more malicious
>>>   - More commits can be added to pull requests later on, it would need re-running in this case
>>>   - The test suite could be altered to make it look like the pull request is fine - although I guess anything that gets
>>> integrated will be looked over anyway, so this isn't much of an issue.
>>>
>>> -- 
>>> Robert
>>> http://octarineparrot.com/
>> All of that could be dealt with, probably, but it still doesn't fix the underlying issue: developers should be running the tests on the code they ask to be pulled themselves, BEFORE the pull request is filed.  Not doing so puts potentially buggy code in the face of others to soon.
>>
>> Automated testing could help with finding conflicts between pull requests or something ahead of it being merged.  But using it as a crutch to not test yourself isn't something I want to encourage.
> 
> Why would it need to get even as a far as a pull request? Couldn't the auto-tester be set-up to test anything that can be pulled? If nothing else, it would allow people to test on *all* platforms before they file the request.

Yes, it could.  But, if you'll allow me.. I'm not building or hosting it. :)

And it still doesn't encourage the behavior I think should be encouraged.. the developer of the change should be testing the changes.  I honestly don't get the apparent resistance to doing so that's being expressed here.

On 31 May 2011 03:34, Brad Roberts <braddr at puremagic.com> wrote:
> While I haven't developed d1 in ages, it's roughly how I experimented with it back in the day. ?I actually don't understand why you hate it so much or where you see complication. ?What's "the correct place" mean to you? ?It's all a matter of the dmd.conf file, which is easy to setup to match any configuration.

No, it isn't. The test suite insists on running ../src/dmd.
The "correct place" to me is the same arrangement as a release. I
don't see any reason for there to be a difference between a snapshot,
a beta, and a release.
In particular, I don't like the test suite having a hard-coded path
which *cannot* be used with an official release.

> Shrug. ?I'm a huge fan of test in place _then_ install.

I don't know how you can test properly without "installing"? I test lots of stuff other than the test suite.

>?But do whatever works for you. ?I'm not trying to impose
> anything on anyone, just showing how I do it and how easy it is.

Actually, you _are_ imposing. The way you've set up is, as far as I
can tell, the ONLY way it can run.
 Basically I have to do a huge hack to get it to work at all, and it's
just not worth it. The test suite is really badly behaved.

Which is why in practice I use Walter's test suite (copying the files
out of the test directory and then running them is far less painful,
and _much_ quicker).
I suspect that Walter has been doing something similar, as well.
It really seems that so far, NOBODY has been using the test suite, and
I don't think that's an accident. It's really a shame.


> On 5/30/2011 5:53 PM, Don Clugston wrote:
>> I still have absolutely no idea why you've set up the tester in such a
>> complicated way with all these hard-coded paths -- and it doesn't
>> actually work if you also develop the D1 compiler.
>> If you simply copy the build result (dmd.exe, or phobos.lib) into the
>> correct place in your dmd directory, you can just run 'dmd', no need
>> to mess with all this nonsense about putting your dmd.conf into the
>> git directory, which does not work if you also test ?D1. ?In fact,
>> sticking *anything* in any of the git directories won't work.
>> Doing it the way you've done it, is pretty useless for anything except
>> running the test suite.
>>
>>
>> 2011/5/30 Brad Roberts <braddr at puremagic.com>:
>>> On 5/30/2011 12:56 PM, Robert Clipsham wrote:
>>>> On 30 May 2011 09:21, Walter Bright <walter at digitalmars.com <mailto:walter at digitalmars.com>> wrote:
>>>>
>>>> ? ? On 5/29/2011 1:53 PM, Robert Clipsham wrote:
>>>>
>>>>
>>>> ? ? ? ? .....
>>>>
>>>> ? ? ? ? ## Running the test suite
>>>>
>>>> ? ? ? ? I actually don't know how to do this, but some instructions should go here. Also take a look at ? ? ? ? http://github.github.com/github-flavored-markdown/ for information on formatting in this file. ? ? ? ? ----
>>>>
>>>> ? ? ? ? I could clean this up and make a pull request if you're interested? ;)
>>>>
>>>>
>>>> ? ? Sure!
>>>>
>>>>
>>>> I've just spent the past 10 minutes trying to figure out how to run the test suite so I can add instructions/make a pull request and can't manage it... It's no wonder it's not getting run!
>>>>
>>>> I can get a range of different errors, I can't seem to manage to get dmd.conf set up correctly to remove all errors though. Perhaps it would be a good idea to include the relevant dmd.conf in the tests/ directory?
>>>>
>>>
>>> Lemee make this simple.. here's a simple makefile and dmd.conf file that 'just work'.
>>>
>>> The make file are more complicated than the minimal necessary because they have a bunch of targets that I use while developing from time to time. ?The default target, however, is enough to clone, build, and test all three packages.
>>>
>>>
>>> _______________________________________________
>>> dmd-internals mailing list
>>> dmd-internals at puremagic.com
>>> http://lists.puremagic.com/mailman/listinfo/dmd-internals
>>>
>> _______________________________________________
>> dmd-internals mailing list
>> dmd-internals at puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/dmd-internals
>
> _______________________________________________
> dmd-internals mailing list
> dmd-internals at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/dmd-internals
>

On 5/31/2011 12:35 AM, Don Clugston wrote:
> On 31 May 2011 03:34, Brad Roberts <braddr at puremagic.com> wrote:
>> While I haven't developed d1 in ages, it's roughly how I experimented with it back in the day.  I actually don't understand why you hate it so much or where you see complication.  What's "the correct place" mean to you?  It's all a matter of the dmd.conf file, which is easy to setup to match any configuration.
> 
> No, it isn't. The test suite insists on running ../src/dmd.
> The "correct place" to me is the same arrangement as a release. I
> don't see any reason for there to be a difference between a snapshot,
> a beta, and a release.
> In particular, I don't like the test suite having a hard-coded path
> which *cannot* be used with an official release.

If the three makefiles had a cohesive 'install' target, I might agree with you, but as there's no cohesive 'build a directory of the release artifacts' to any of the makefiles, it's a weak argument.  The path to dmd has a default value, just like every other part of the three make files.  But it's easily overridden if you want to, just like every other part of the three make files:  make DMD=/path/to/any/working/dmd  or  make DMD=dmd if you want your path to do all the heavy lifting.

Where's the problem?

>> Shrug.  I'm a huge fan of test in place _then_ install.
> 
> I don't know how you can test properly without "installing"? I test lots of stuff other than the test suite.

Not sure how to respond to that other than,  "I too test more than just the dmd test suite."  I know you're smart enough to know that with a working dmd, it really doesn't matter where it's located.  I also test with 2 or 3 fully separate build trees and have no intention of having to worry about which copy is in /usr/local/bin or whatever at any point in time.  That's just asking for mistakes when testing.

>>  But do whatever works for you.  I'm not trying to impose
>> anything on anyone, just showing how I do it and how easy it is.
> 
> Actually, you _are_ imposing. The way you've set up is, as far as I
> can tell, the ONLY way it can run.
>  Basically I have to do a huge hack to get it to work at all, and it's
> just not worth it. The test suite is really badly behaved.

See above, it's a default, a starting point, but far from so fixed in place it can't be used in any other way.

> Which is why in practice I use Walter's test suite (copying the files
> out of the test directory and then running them is far less painful,
> and _much_ quicker).
> I suspect that Walter has been doing something similar, as well.
> It really seems that so far, NOBODY has been using the test suite, and
> I don't think that's an accident. It's really a shame.

What's a shame, to me, is that you've done nothing to help improve it, just criticize.  I'd happily review pull requests to improve any part of the dmd test suite, any makefiles that are part of any of the three packages, and the auto-tester.  Don't like it?  Suggest (or preferably provide) concrete, functional, working changes.

I really really don't understand why you're projecting such a degree of anger/ire here.  It's like I've kicked your puppy or something.

Sigh,
Brad

On 5/31/11 10:02 AM, Brad Roberts wrote:
> [?] or make DMD=dmd if you want your path to do all the heavy lifting.
>
> Where's the problem?

For me, the problem actually was that specifying DMD=dmd does *not* work because it breaks the wildcard rules, as $(DMD) is specified in the dependency list there. It took me quite some time to realize that, as make only prints the following in that case:

make: *** No rule to make target `test_results/runnable/A16.d.out', needed by `run_runnable_tests'.  Stop.

Is having the DMD binary in the dependency list really necessary, or could it just be removed? In case it really is for whatever reason, some documentation for that would be nice ? apart from that, adding a short test/README might be generally helpful as well.

David

On 6/6/2011 8:35 PM, David Nadlinger wrote:
> On 5/31/11 10:02 AM, Brad Roberts wrote:
>> [?] or make DMD=dmd if you want your path to do all the heavy lifting.
>>
>> Where's the problem?
> 
> For me, the problem actually was that specifying DMD=dmd does *not* work because it breaks the wildcard rules, as $(DMD) is specified in the dependency list there. It took me quite some time to realize that, as make only prints the following in that case:
> 
> make: *** No rule to make target `test_results/runnable/A16.d.out', needed by `run_runnable_tests'.  Stop.
> 
> Is having the DMD binary in the dependency list really necessary, or could it just be removed? In case it really is for whatever reason, some documentation for that would be nice ? apart from that, adding a short test/README might be generally helpful as well.
> 
> David

It's not 100% necessary, but it's darned handy to not have to remember to make clean after changing dmd.  I'll take a look to see if I can figure out why the DMD=dmd case fails.

On 6/7/11 5:59 AM, Brad Roberts wrote:
> It's not 100% necessary, but it's darned handy to not have to remember to make clean after changing dmd.  I'll take a look to see if I can figure out why the DMD=dmd case fails.

Oh, of course ? I am so used to thinking of the compiler as an external tool that I forget to consider that. Still, a fix for DMD=dmd would be great.

I am by no means a Make expert, but I'd guess that the following happens with DMD=dmd: On searching for a rule to build the output files, it recognizes the wildcard rules, but doesn't consider them because they depend on a file ?dmd? in the current directory, which can't be satisfied.

David