Here's the task:
Given a .d source file, strip out all of the unittest {} blocks,
including everything inside them.
Strip out all comments as well.
Print out the resulting file.

Motivation: Bug reports frequently come with very large test cases.
Even ones which look small often import from Phobos.
Reducing the test case is the first step in fixing the bug, and it's frequently ~30% of the total time required. Stripping out the unit tests is the most time-consuming and error-prone part of reducing the test case.

This should be a good task if you're relatively new to D but would like to do something really useful.
-Don

On 3/20/11 1:11 AM, Don wrote:
> Here's the task:
> Given a .d source file, strip out all of the unittest {} blocks,
> including everything inside them.
> Strip out all comments as well.
> Print out the resulting file.

I realize that you asked for a very specific utility, but in several instances, http://delta.tigris.org/ worked fine for me for reducing large test cases.

Parts of it are tailored to C/C++ though, so a port/adaption for D would be a nice project as well.

David

On Saturday 19 March 2011 17:11:56 Don wrote:
> Here's the task:
> Given a .d source file, strip out all of the unittest {} blocks,
> including everything inside them.
> Strip out all comments as well.
> Print out the resulting file.
> 
> Motivation: Bug reports frequently come with very large test cases.
> Even ones which look small often import from Phobos.
> Reducing the test case is the first step in fixing the bug, and it's
> frequently ~30% of the total time required. Stripping out the unit tests
> is the most time-consuming and error-prone part of reducing the test case.
> 
> This should be a good task if you're relatively new to D but would like to do something really useful.

Unfortunately, to do that 100% correctly, you need to actually have a working D lexer (and possibly parser). You might be able to get something close enough to work in most cases, but it doesn't take all that much to throw off a basic implementation of this sort of thing if you don't lex/parse it with something which properly understands D.

- Jonathan M Davis

Jonathan M Davis wrote:
> On Saturday 19 March 2011 17:11:56 Don wrote:
>> Here's the task:
>> Given a .d source file, strip out all of the unittest {} blocks,
>> including everything inside them.
>> Strip out all comments as well.
>> Print out the resulting file.
>>
>> Motivation: Bug reports frequently come with very large test cases.
>> Even ones which look small often import from Phobos.
>> Reducing the test case is the first step in fixing the bug, and it's
>> frequently ~30% of the total time required. Stripping out the unit tests
>> is the most time-consuming and error-prone part of reducing the test case.
>>
>> This should be a good task if you're relatively new to D but would like
>> to do something really useful.
> 
> Unfortunately, to do that 100% correctly, you need to actually have a working D lexer (and possibly parser). You might be able to get something close enough to work in most cases, but it doesn't take all that much to throw off a basic implementation of this sort of thing if you don't lex/parse it with something which properly understands D.
> 
> - Jonathan M Davis

I didn't say it needs 100% accuracy. You can assume, for example, that "unittest" always occurs at the start of a line. The only other things you need to lex are {}, string literals, and comments.

BTW, the immediate motivation for this is std.datetime in Phobos. The sheer number of unittests in there is an absolute catastrophe for tracking down bugs. It makes a tool like this MANDATORY.

On Saturday 19 March 2011 18:04:57 Don wrote:
> Jonathan M Davis wrote:
> > On Saturday 19 March 2011 17:11:56 Don wrote:
> >> Here's the task:
> >> Given a .d source file, strip out all of the unittest {} blocks,
> >> including everything inside them.
> >> Strip out all comments as well.
> >> Print out the resulting file.
> >> 
> >> Motivation: Bug reports frequently come with very large test cases.
> >> Even ones which look small often import from Phobos.
> >> Reducing the test case is the first step in fixing the bug, and it's
> >> frequently ~30% of the total time required. Stripping out the unit tests
> >> is the most time-consuming and error-prone part of reducing the test
> >> case.
> >> 
> >> This should be a good task if you're relatively new to D but would like to do something really useful.
> > 
> > Unfortunately, to do that 100% correctly, you need to actually have a working D lexer (and possibly parser). You might be able to get something close enough to work in most cases, but it doesn't take all that much to throw off a basic implementation of this sort of thing if you don't lex/parse it with something which properly understands D.
> > 
> > - Jonathan M Davis
> 
> I didn't say it needs 100% accuracy. You can assume, for example, that "unittest" always occurs at the start of a line. The only other things you need to lex are {}, string literals, and comments.
> 
> BTW, the immediate motivation for this is std.datetime in Phobos. The sheer number of unittests in there is an absolute catastrophe for tracking down bugs. It makes a tool like this MANDATORY.

I tried to create a similar tool before and gave up because I couldn't make it 100% accurate and was running into problems with it. If someone wants to take a shot at it though, that's fine.

As for the unit tests in std.datetime making it hard to track down bugs, that only makes sense to me if you're trying to look at the whole thing at once and track down a compiler bug which happens _somewhere_ in the code, but you don't know where. Other than a problem like that, I don't really see how the unit tests get in the way of tracking down bugs. Is it that you need to compile in a version of std.datetime which doesn't have any unit tests compiled in but you still need to compile with -unittest for other stuff?

I _am_ working on streamlining the unit tests in std.datetime so that they take up fewer lines of code without reducing how well they cover the code, so depending on your problem with the amount of unit test code, that could help, but I expect that whatever your core problem with the number of unit tests is, that won't fix it.

- Jonathan M Davis

On 2011-03-19 20:41:09 -0400, Jonathan M Davis <jmdavisProg@gmx.com> said:

> On Saturday 19 March 2011 17:11:56 Don wrote:
>> Here's the task:
>> Given a .d source file, strip out all of the unittest {} blocks,
>> including everything inside them.
>> Strip out all comments as well.
>> Print out the resulting file.
>> 
>> Motivation: Bug reports frequently come with very large test cases.
>> Even ones which look small often import from Phobos.
>> Reducing the test case is the first step in fixing the bug, and it's
>> frequently ~30% of the total time required. Stripping out the unit tests
>> is the most time-consuming and error-prone part of reducing the test case.
>> 
>> This should be a good task if you're relatively new to D but would like
>> to do something really useful.
> 
> Unfortunately, to do that 100% correctly, you need to actually have a working D
> lexer (and possibly parser). You might be able to get something close enough to
> work in most cases, but it doesn't take all that much to throw off a basic
> implementation of this sort of thing if you don't lex/parse it with something
> which properly understands D.

Well, I made simple lexer for D strings, comments, identifiers and a few other tokens which should be up to that task. It's what I use to parse files and detect dependencies in D for Xcode. Unfortunately, it's written in Objective-C++ (but half of it is plain C)...

<https://github.com/michelf/d-for-xcode/blob/master/Sources/DXBaseLexer.h>
<https://github.com/michelf/d-for-xcode/blob/master/Sources/DXBaseLexer.mm>
<https://github.com/michelf/d-for-xcode/blob/master/Sources/DXScannerTools.h>
<https://github.com/michelf/d-for-xcode/blob/master/Sources/DXScannerTools.m>

Very short unit test:
<https://github.com/michelf/d-for-xcode/blob/master/Unit%20Tests/DXBaseLexerTest.mm>

-- 


Michel Fortin
michel.fortin@michelf.com
http://michelf.com/

Jonathan M Davis wrote:
> On Saturday 19 March 2011 18:04:57 Don wrote:
>> Jonathan M Davis wrote:
>>> On Saturday 19 March 2011 17:11:56 Don wrote:
>>>> Here's the task:
>>>> Given a .d source file, strip out all of the unittest {} blocks,
>>>> including everything inside them.
>>>> Strip out all comments as well.
>>>> Print out the resulting file.
>>>>
>>>> Motivation: Bug reports frequently come with very large test cases.
>>>> Even ones which look small often import from Phobos.
>>>> Reducing the test case is the first step in fixing the bug, and it's
>>>> frequently ~30% of the total time required. Stripping out the unit tests
>>>> is the most time-consuming and error-prone part of reducing the test
>>>> case.
>>>>
>>>> This should be a good task if you're relatively new to D but would like
>>>> to do something really useful.
>>> Unfortunately, to do that 100% correctly, you need to actually have a
>>> working D lexer (and possibly parser). You might be able to get
>>> something close enough to work in most cases, but it doesn't take all
>>> that much to throw off a basic implementation of this sort of thing if
>>> you don't lex/parse it with something which properly understands D.
>>>
>>> - Jonathan M Davis
>> I didn't say it needs 100% accuracy. You can assume, for example, that
>> "unittest" always occurs at the start of a line. The only other things
>> you need to lex are {}, string literals, and comments.
>>
>> BTW, the immediate motivation for this is std.datetime in Phobos. The
>> sheer number of unittests in there is an absolute catastrophe for
>> tracking down bugs. It makes a tool like this MANDATORY.
> 
> I tried to create a similar tool before and gave up because I couldn't make it 100% accurate and was running into problems with it. If someone wants to take a shot at it though, that's fine.
> 
> As for the unit tests in std.datetime making it hard to track down bugs, that only makes sense to me if you're trying to look at the whole thing at once and track down a compiler bug which happens _somewhere_ in the code, but you don't know where. Other than a problem like that, I don't really see how the unit tests get in the way of tracking down bugs. Is it that you need to compile in a version of std.datetime which doesn't have any unit tests compiled in but you still need to compile with -unittest for other stuff?

No. All you know there's a bug that's being triggered somewhere in Phobos (with -unittest). It's probably not in std.datetime.
But Phobos is a horrible ball of mud where everything imports everything else, and std.datetime is near the centre of that ball. What you have to do is reduce the amount of code, and especially the number of modules, as rapidly as possible; this means getting rid of imports.

To do this, you need to remove large chunks of code from the files. This is pretty simple; comment out half of the file, if it still works, then delete it. Normally this works well because typically only about a dozen lines are actually being used. After doing this about three or four times it's small enough that you can usually get rid of most of the imports.
Unittests foul this up because they use functions/classes from inside the file.

In the case of std.datetime it's even worse because the signal-to-noise ratio is so incredibly poor; it's really difficult to find the few lines of code that are actually being used by other Phobos modules.

My experience (obviously only over the last month or so) has been that if the reduction of a bug is non-obvious, more than 10% of the total time taken to fix that bug is the time taken to cut down std.datetime.

> Jonathan M Davis wrote:
> > On Saturday 19 March 2011 18:04:57 Don wrote:
> >> Jonathan M Davis wrote:
> >>> On Saturday 19 March 2011 17:11:56 Don wrote:
> >>>> Here's the task:
> >>>> Given a .d source file, strip out all of the unittest {} blocks,
> >>>> including everything inside them.
> >>>> Strip out all comments as well.
> >>>> Print out the resulting file.
> >>>> 
> >>>> Motivation: Bug reports frequently come with very large test cases.
> >>>> Even ones which look small often import from Phobos.
> >>>> Reducing the test case is the first step in fixing the bug, and it's
> >>>> frequently ~30% of the total time required. Stripping out the unit
> >>>> tests is the most time-consuming and error-prone part of reducing the
> >>>> test case.
> >>>> 
> >>>> This should be a good task if you're relatively new to D but would like to do something really useful.
> >>> 
> >>> Unfortunately, to do that 100% correctly, you need to actually have a working D lexer (and possibly parser). You might be able to get something close enough to work in most cases, but it doesn't take all that much to throw off a basic implementation of this sort of thing if you don't lex/parse it with something which properly understands D.
> >>> 
> >>> - Jonathan M Davis
> >> 
> >> I didn't say it needs 100% accuracy. You can assume, for example, that "unittest" always occurs at the start of a line. The only other things you need to lex are {}, string literals, and comments.
> >> 
> >> BTW, the immediate motivation for this is std.datetime in Phobos. The sheer number of unittests in there is an absolute catastrophe for tracking down bugs. It makes a tool like this MANDATORY.
> > 
> > I tried to create a similar tool before and gave up because I couldn't make it 100% accurate and was running into problems with it. If someone wants to take a shot at it though, that's fine.
> > 
> > As for the unit tests in std.datetime making it hard to track down bugs, that only makes sense to me if you're trying to look at the whole thing at once and track down a compiler bug which happens _somewhere_ in the code, but you don't know where. Other than a problem like that, I don't really see how the unit tests get in the way of tracking down bugs. Is it that you need to compile in a version of std.datetime which doesn't have any unit tests compiled in but you still need to compile with -unittest for other stuff?
> 
> No. All you know there's a bug that's being triggered somewhere in
> Phobos (with -unittest). It's probably not in std.datetime.
> But Phobos is a horrible ball of mud where everything imports everything
> else, and std.datetime is near the centre of that ball. What you have to
> do is reduce the amount of code, and especially the number of modules,
> as rapidly as possible; this means getting rid of imports.
> 
> To do this, you need to remove large chunks of code from the files. This is pretty simple; comment out half of the file, if it still works, then delete it. Normally this works well because typically only about a dozen lines are actually being used. After doing this about three or four times it's small enough that you can usually get rid of most of the imports. Unittests foul this up because they use functions/classes from inside the file.
> 
> In the case of std.datetime it's even worse because the signal-to-noise ratio is so incredibly poor; it's really difficult to find the few lines of code that are actually being used by other Phobos modules.
> 
> My experience (obviously only over the last month or so) has been that if the reduction of a bug is non-obvious, more than 10% of the total time taken to fix that bug is the time taken to cut down std.datetime.

Hmmm. I really don't know what could be done to fix that (other than making it easier to rip out the unittest blocks). And enough of std.datetime depends on other parts of std.datetime that trimming it down isn't (and can't be) exactly easy. In general, SysTime is the most likely type to be used, and it depends on Date, TimeOfDay, and DateTime, and all 4 of those depend on most of the free functions in the module. It's not exactly designed in a manner which allows you to cut out large chunks and still have it compile. And I don't think that it _could_ be designed that way and still have the functionality that it has.

I guess that this sort of problem is one that would pop up mainly when dealing with compiler bugs. I have a hard time seeing it popping up with your typical bug in Phobos itself. So, I guess that this is the sort of thing that you'd run into and I likely wouldn't.

I really don't know how the situation could be improved though other than making it easier to cut out the unit tests.

- Jonathan M Davis

Jonathan M Davis wrote:
>> Jonathan M Davis wrote:
>>> On Saturday 19 March 2011 18:04:57 Don wrote:
>>>> Jonathan M Davis wrote:
>>>>> On Saturday 19 March 2011 17:11:56 Don wrote:
>>>>>> Here's the task:
>>>>>> Given a .d source file, strip out all of the unittest {} blocks,
>>>>>> including everything inside them.
>>>>>> Strip out all comments as well.
>>>>>> Print out the resulting file.
>>>>>>
>>>>>> Motivation: Bug reports frequently come with very large test cases.
>>>>>> Even ones which look small often import from Phobos.
>>>>>> Reducing the test case is the first step in fixing the bug, and it's
>>>>>> frequently ~30% of the total time required. Stripping out the unit
>>>>>> tests is the most time-consuming and error-prone part of reducing the
>>>>>> test case.
>>>>>>
>>>>>> This should be a good task if you're relatively new to D but would
>>>>>> like to do something really useful.
>>>>> Unfortunately, to do that 100% correctly, you need to actually have a
>>>>> working D lexer (and possibly parser). You might be able to get
>>>>> something close enough to work in most cases, but it doesn't take all
>>>>> that much to throw off a basic implementation of this sort of thing if
>>>>> you don't lex/parse it with something which properly understands D.
>>>>>
>>>>> - Jonathan M Davis
>>>> I didn't say it needs 100% accuracy. You can assume, for example, that
>>>> "unittest" always occurs at the start of a line. The only other things
>>>> you need to lex are {}, string literals, and comments.
>>>>
>>>> BTW, the immediate motivation for this is std.datetime in Phobos. The
>>>> sheer number of unittests in there is an absolute catastrophe for
>>>> tracking down bugs. It makes a tool like this MANDATORY.
>>> I tried to create a similar tool before and gave up because I couldn't
>>> make it 100% accurate and was running into problems with it. If someone
>>> wants to take a shot at it though, that's fine.
>>>
>>> As for the unit tests in std.datetime making it hard to track down bugs,
>>> that only makes sense to me if you're trying to look at the whole thing
>>> at once and track down a compiler bug which happens _somewhere_ in the
>>> code, but you don't know where. Other than a problem like that, I don't
>>> really see how the unit tests get in the way of tracking down bugs. Is
>>> it that you need to compile in a version of std.datetime which doesn't
>>> have any unit tests compiled in but you still need to compile with
>>> -unittest for other stuff?
>> No. All you know there's a bug that's being triggered somewhere in
>> Phobos (with -unittest). It's probably not in std.datetime.
>> But Phobos is a horrible ball of mud where everything imports everything
>> else, and std.datetime is near the centre of that ball. What you have to
>> do is reduce the amount of code, and especially the number of modules,
>> as rapidly as possible; this means getting rid of imports.
>>
>> To do this, you need to remove large chunks of code from the files. This
>> is pretty simple; comment out half of the file, if it still works, then
>> delete it. Normally this works well because typically only about a dozen
>> lines are actually being used. After doing this about three or four
>> times it's small enough that you can usually get rid of most of the
>> imports. Unittests foul this up because they use functions/classes from
>> inside the file.
>>
>> In the case of std.datetime it's even worse because the signal-to-noise
>> ratio is so incredibly poor; it's really difficult to find the few lines
>> of code that are actually being used by other Phobos modules.
>>
>> My experience (obviously only over the last month or so) has been that
>> if the reduction of a bug is non-obvious, more than 10% of the total
>> time taken to fix that bug is the time taken to cut down std.datetime.
> 
> Hmmm. I really don't know what could be done to fix that (other than making it easier to rip out the unittest blocks). And enough of std.datetime depends on other parts of std.datetime that trimming it down isn't (and can't be) exactly easy. In general, SysTime is the most likely type to be used, and it depends on Date, TimeOfDay, and DateTime, and all 4 of those depend on most of the free functions in the module. It's not exactly designed in a manner which allows you to cut out large chunks and still have it compile. And I don't think that it _could_ be designed that way and still have the functionality that it has.

The problem is purely the large fraction of the module which is devoted to unit tests. That's all.

> 
> I guess that this sort of problem is one that would pop up mainly when dealing with compiler bugs. I have a hard time seeing it popping up with your typical bug in Phobos itself. So, I guess that this is the sort of thing that you'd run into and I likely wouldn't.

Yes.

> I really don't know how the situation could be improved though other than making it easier to cut out the unit tests.
> 
> - Jonathan M Davis

Hence the motivation for this utility. The problem exists in all modules, but in std.datetime it's such an obvious time-waster that I can't keep ignoring it.

> Jonathan M Davis wrote:
> >> Jonathan M Davis wrote:
> >>> On Saturday 19 March 2011 18:04:57 Don wrote:
> >>>> Jonathan M Davis wrote:
> >>>>> On Saturday 19 March 2011 17:11:56 Don wrote:
> >>>>>> Here's the task:
> >>>>>> Given a .d source file, strip out all of the unittest {} blocks,
> >>>>>> including everything inside them.
> >>>>>> Strip out all comments as well.
> >>>>>> Print out the resulting file.
> >>>>>> 
> >>>>>> Motivation: Bug reports frequently come with very large test cases.
> >>>>>> Even ones which look small often import from Phobos.
> >>>>>> Reducing the test case is the first step in fixing the bug, and it's
> >>>>>> frequently ~30% of the total time required. Stripping out the unit
> >>>>>> tests is the most time-consuming and error-prone part of reducing
> >>>>>> the test case.
> >>>>>> 
> >>>>>> This should be a good task if you're relatively new to D but would like to do something really useful.
> >>>>> 
> >>>>> Unfortunately, to do that 100% correctly, you need to actually have a working D lexer (and possibly parser). You might be able to get something close enough to work in most cases, but it doesn't take all that much to throw off a basic implementation of this sort of thing if you don't lex/parse it with something which properly understands D.
> >>>>> 
> >>>>> - Jonathan M Davis
> >>>> 
> >>>> I didn't say it needs 100% accuracy. You can assume, for example, that "unittest" always occurs at the start of a line. The only other things you need to lex are {}, string literals, and comments.
> >>>> 
> >>>> BTW, the immediate motivation for this is std.datetime in Phobos. The sheer number of unittests in there is an absolute catastrophe for tracking down bugs. It makes a tool like this MANDATORY.
> >>> 
> >>> I tried to create a similar tool before and gave up because I couldn't make it 100% accurate and was running into problems with it. If someone wants to take a shot at it though, that's fine.
> >>> 
> >>> As for the unit tests in std.datetime making it hard to track down bugs, that only makes sense to me if you're trying to look at the whole thing at once and track down a compiler bug which happens _somewhere_ in the code, but you don't know where. Other than a problem like that, I don't really see how the unit tests get in the way of tracking down bugs. Is it that you need to compile in a version of std.datetime which doesn't have any unit tests compiled in but you still need to compile with -unittest for other stuff?
> >> 
> >> No. All you know there's a bug that's being triggered somewhere in
> >> Phobos (with -unittest). It's probably not in std.datetime.
> >> But Phobos is a horrible ball of mud where everything imports everything
> >> else, and std.datetime is near the centre of that ball. What you have to
> >> do is reduce the amount of code, and especially the number of modules,
> >> as rapidly as possible; this means getting rid of imports.
> >> 
> >> To do this, you need to remove large chunks of code from the files. This is pretty simple; comment out half of the file, if it still works, then delete it. Normally this works well because typically only about a dozen lines are actually being used. After doing this about three or four times it's small enough that you can usually get rid of most of the imports. Unittests foul this up because they use functions/classes from inside the file.
> >> 
> >> In the case of std.datetime it's even worse because the signal-to-noise ratio is so incredibly poor; it's really difficult to find the few lines of code that are actually being used by other Phobos modules.
> >> 
> >> My experience (obviously only over the last month or so) has been that if the reduction of a bug is non-obvious, more than 10% of the total time taken to fix that bug is the time taken to cut down std.datetime.
> > 
> > Hmmm. I really don't know what could be done to fix that (other than making it easier to rip out the unittest blocks). And enough of std.datetime depends on other parts of std.datetime that trimming it down isn't (and can't be) exactly easy. In general, SysTime is the most likely type to be used, and it depends on Date, TimeOfDay, and DateTime, and all 4 of those depend on most of the free functions in the module. It's not exactly designed in a manner which allows you to cut out large chunks and still have it compile. And I don't think that it _could_ be designed that way and still have the functionality that it has.
> 
> The problem is purely the large fraction of the module which is devoted to unit tests. That's all.
> 
> > I guess that this sort of problem is one that would pop up mainly when dealing with compiler bugs. I have a hard time seeing it popping up with your typical bug in Phobos itself. So, I guess that this is the sort of thing that you'd run into and I likely wouldn't.
> 
> Yes.
> 
> > I really don't know how the situation could be improved though other than making it easier to cut out the unit tests.
> > 
> > - Jonathan M Davis
> 
> Hence the motivation for this utility. The problem exists in all modules, but in std.datetime it's such an obvious time-waster that I can't keep ignoring it.

Well, for the moment at least, if you remove the

version = testStdDateTime;
version = enableWindowsTest;

lines near the top of the file, then pretty much all of the unittest blocks will no longer be compiled in (there might be a couple which are still compiled in, but not many). So, that could help you until the utility that you want is done. Unfortunately, that also means that the utility will have to be smarter if it's going to work on std.datetime. While most of the version(testStdDateTime) blocks are currently _inside_ of the unittest blocks, as I've been adjusting the unit tests, I've been changing them to

version(testStdDateTime) unittest

because Andrei didn't like the extra vertical space used up by having separate blocks for the unittest and for the version. So, for instance, if the utility assumed that unittest was the first part of the line for a unittest block, it wouldn't work on std.datetime (IIRC, std.algorithm would have similar problems).

- Jonathan M Davis