phobos commit, revision 2331


user: jmdavis

msg:
Re-enabled std.datetime for Windows but disabled its unit tests for Windows.

For some reason, the std.datetime unit tests compile and pass without problems on Linux and in Wine on Linux, but on real Windows, the compiler runs out of memory when compiling them. So, for now, all of the std.datetime unit tests are versioned with testStdDateTime. On Posix, version=testdStdDateTime is used in std.datetime, so the tests will run, but that version is not set on Windows, so the unit tests don't run on Windows. So, if you want to run the std.datetime unit tests on Windows or under Wine or Linux, then compile with -version=testStdDateTime.

At the moment, all of the tests pass on both Linux and under Wine, and
the normal code does compile on proper, so I'm fairly certain that it's fine
for use on Windows, but for now, I can't enable the tests on Windows. My
best guess is that it's some sort of weird compiler bug involving system
calls that leak memory on Windows-proper but not in Wine, but I really
don't know.

http://www.dsource.org/projects/phobos/changeset/2331

paths changed:
U   trunk/phobos/std/datetime.d

On Fri, 14 Jan 2011 06:36:18 -0500, dsource.org <noreply at dsource.org> wrote:

> phobos commit, revision 2331
>
>
> user: jmdavis
>
> msg:
> Re-enabled std.datetime for Windows but disabled its unit tests for
> Windows.
>
> For some reason, the std.datetime unit tests compile and pass without problems on Linux and in Wine on Linux, but on real Windows, the compiler runs out of memory when compiling them. So, for now, all of the std.datetime unit tests are versioned with testStdDateTime. On Posix, version=testdStdDateTime is used in std.datetime, so the tests will run, but that version is not set on Windows, so the unit tests don't run on Windows. So, if you want to run the std.datetime unit tests on Windows or under Wine or Linux, then compile with -version=testStdDateTime.
>
> At the moment, all of the tests pass on both Linux and under Wine, and
> the normal code does compile on proper, so I'm fairly certain that it's
> fine
> for use on Windows, but for now, I can't enable the tests on Windows. My
> best guess is that it's some sort of weird compiler bug involving system
> calls that leak memory on Windows-proper but not in Wine, but I really
> don't know.
>
> http://www.dsource.org/projects/phobos/changeset/2331
>
> paths changed:
> U   trunk/phobos/std/datetime.d
>
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

If DMD is running out of memory at compile time, I doubt it has to do with weird compiler bugs. DMD running out of memory is usually a sign of heavy compile time string manipulation, CTFE or template usage. Remember, DMD keeps all template instantiations in memory and every time a string or array is touched, a new copy is created (bug 1382).

On Friday 14 January 2011 05:43:19 Robert Jacques wrote:
> On Fri, 14 Jan 2011 06:36:18 -0500, dsource.org <noreply at dsource.org>
> 
> wrote:
> > phobos commit, revision 2331
> > 
> > 
> > user: jmdavis
> > 
> > msg:
> > Re-enabled std.datetime for Windows but disabled its unit tests for
> > Windows.
> > 
> > For some reason, the std.datetime unit tests compile and pass without problems on Linux and in Wine on Linux, but on real Windows, the compiler runs out of memory when compiling them. So, for now, all of the std.datetime unit tests are versioned with testStdDateTime. On Posix, version=testdStdDateTime is used in std.datetime, so the tests will run, but that version is not set on Windows, so the unit tests don't run on Windows. So, if you want to run the std.datetime unit tests on Windows or under Wine or Linux, then compile with -version=testStdDateTime.
> > 
> > At the moment, all of the tests pass on both Linux and under Wine, and
> > the normal code does compile on proper, so I'm fairly certain that it's
> > fine
> > for use on Windows, but for now, I can't enable the tests on Windows. My
> > best guess is that it's some sort of weird compiler bug involving system
> > calls that leak memory on Windows-proper but not in Wine, but I really
> > don't know.
> > 
> > http://www.dsource.org/projects/phobos/changeset/2331
> > 
> > paths changed:
> > U   trunk/phobos/std/datetime.d
> > 
> > _______________________________________________
> > phobos mailing list
> > phobos at puremagic.com
> > http://lists.puremagic.com/mailman/listinfo/phobos
> 
> If DMD is running out of memory at compile time, I doubt it has to do with weird compiler bugs. DMD running out of memory is usually a sign of heavy compile time string manipulation, CTFE or template usage. Remember, DMD keeps all template instantiations in memory and every time a string or array is touched, a new copy is created (bug 1382).

Yes. But to then have it not only work on Linux but on Wine and yet not work on Windows? The only real difference between running something under Wine and Windows is that the implementation of system calls as changed.

- Jonathan M Davis

On Fri, 14 Jan 2011 11:19:34 -0500, Jonathan M Davis <jmdavisProg at gmx.com> wrote:

> On Friday 14 January 2011 05:43:19 Robert Jacques wrote:
>> On Fri, 14 Jan 2011 06:36:18 -0500, dsource.org <noreply at dsource.org>
>>
>> wrote:
>> > phobos commit, revision 2331
>> >
>> >
>> > user: jmdavis
>> >
>> > msg:
>> > Re-enabled std.datetime for Windows but disabled its unit tests for
>> > Windows.
>> >
>> > For some reason, the std.datetime unit tests compile and pass without problems on Linux and in Wine on Linux, but on real Windows, the compiler runs out of memory when compiling them. So, for now, all of
>> the
>> > std.datetime unit tests are versioned with testStdDateTime. On Posix, version=testdStdDateTime is used in std.datetime, so the tests will
>> run,
>> > but that version is not set on Windows, so the unit tests don't run on Windows. So, if you want to run the std.datetime unit tests on
>> Windows or
>> > under Wine or Linux, then compile with -version=testStdDateTime.
>> >
>> > At the moment, all of the tests pass on both Linux and under Wine, and the normal code does compile on proper, so I'm fairly certain that
>> it's
>> > fine
>> > for use on Windows, but for now, I can't enable the tests on Windows.
>> My
>> > best guess is that it's some sort of weird compiler bug involving
>> system
>> > calls that leak memory on Windows-proper but not in Wine, but I really don't know.
>> >
>> > http://www.dsource.org/projects/phobos/changeset/2331
>> >
>> > paths changed:
>> > U   trunk/phobos/std/datetime.d
>> >
>> > _______________________________________________
>> > phobos mailing list
>> > phobos at puremagic.com
>> > http://lists.puremagic.com/mailman/listinfo/phobos
>>
>> If DMD is running out of memory at compile time, I doubt it has to do
>> with
>> weird compiler bugs. DMD running out of memory is usually a sign of
>> heavy
>> compile time string manipulation, CTFE or template usage. Remember, DMD
>> keeps all template instantiations in memory and every time a string or
>> array is touched, a new copy is created (bug 1382).
>
> Yes. But to then have it not only work on Linux but on Wine and yet not
> work on
> Windows? The only real difference between running something under Wine
> and
> Windows is that the implementation of system calls as changed.

Well, is the crash due to a marginal difference in memory levels, or something major? i.e. is DMD's memory usage on Wine also very high, or is it relatively low.

On 1/14/2011 10:39 AM, Robert Jacques wrote:
> On Fri, 14 Jan 2011 11:19:34 -0500, Jonathan M Davis <jmdavisProg at gmx.com> wrote:
> 
>> On Friday 14 January 2011 05:43:19 Robert Jacques wrote:
>>> On Fri, 14 Jan 2011 06:36:18 -0500, dsource.org <noreply at dsource.org>
>>>
>>> wrote:
>>> > phobos commit, revision 2331
>>> >
>>> >
>>> > user: jmdavis
>>> >
>>> > msg:
>>> > Re-enabled std.datetime for Windows but disabled its unit tests for
>>> > Windows.
>>> >
>>> > For some reason, the std.datetime unit tests compile and pass without problems on Linux and in Wine on Linux, but on real Windows, the compiler runs out of memory when compiling them. So, for now, all of the std.datetime unit tests are versioned with testStdDateTime. On Posix, version=testdStdDateTime is used in std.datetime, so the tests will run, but that version is not set on Windows, so the unit tests don't run on Windows. So, if you want to run the std.datetime unit tests on Windows or under Wine or Linux, then compile with -version=testStdDateTime.
>>> >
>>> > At the moment, all of the tests pass on both Linux and under Wine, and
>>> > the normal code does compile on proper, so I'm fairly certain that it's
>>> > fine
>>> > for use on Windows, but for now, I can't enable the tests on Windows. My
>>> > best guess is that it's some sort of weird compiler bug involving system
>>> > calls that leak memory on Windows-proper but not in Wine, but I really
>>> > don't know.
>>> >
>>> > http://www.dsource.org/projects/phobos/changeset/2331
>>> >
>>> > paths changed:
>>> > U   trunk/phobos/std/datetime.d
>>> >
>>> > _______________________________________________
>>> > phobos mailing list
>>> > phobos at puremagic.com
>>> > http://lists.puremagic.com/mailman/listinfo/phobos
>>>
>>> If DMD is running out of memory at compile time, I doubt it has to do with weird compiler bugs. DMD running out of memory is usually a sign of heavy compile time string manipulation, CTFE or template usage. Remember, DMD keeps all template instantiations in memory and every time a string or array is touched, a new copy is created (bug 1382).
>>
>> Yes. But to then have it not only work on Linux but on Wine and yet not work on Windows? The only real difference between running something under Wine and Windows is that the implementation of system calls as changed.
> 
> Well, is the crash due to a marginal difference in memory levels, or something major? i.e. is DMD's memory usage on Wine also very high, or is it relatively low.

I haven't had time to try to help with this.. but for what it's worth, the windows test box has 2 gigs of ram and is running 32bit windows.  So, there _ought_ to be pleanty of memory.

On Friday, January 14, 2011 10:43:26 Brad Roberts wrote:
> On 1/14/2011 10:39 AM, Robert Jacques wrote:
> > On Fri, 14 Jan 2011 11:19:34 -0500, Jonathan M Davis <jmdavisProg at gmx.com>
wrote:
> >> On Friday 14 January 2011 05:43:19 Robert Jacques wrote:
> >>> On Fri, 14 Jan 2011 06:36:18 -0500, dsource.org <noreply at dsource.org>
> >>> 
> >>> wrote:
> >>> > phobos commit, revision 2331
> >>> > 
> >>> > 
> >>> > user: jmdavis
> >>> > 
> >>> > msg:
> >>> > Re-enabled std.datetime for Windows but disabled its unit tests for
> >>> > Windows.
> >>> > 
> >>> > For some reason, the std.datetime unit tests compile and pass without problems on Linux and in Wine on Linux, but on real Windows, the compiler runs out of memory when compiling them. So, for now, all of the std.datetime unit tests are versioned with testStdDateTime. On Posix, version=testdStdDateTime is used in std.datetime, so the tests will run, but that version is not set on Windows, so the unit tests don't run on Windows. So, if you want to run the std.datetime unit tests on Windows or under Wine or Linux, then compile with -version=testStdDateTime.
> >>> > 
> >>> > At the moment, all of the tests pass on both Linux and under Wine,
> >>> > and the normal code does compile on proper, so I'm fairly certain
> >>> > that it's fine
> >>> > for use on Windows, but for now, I can't enable the tests on Windows.
> >>> > My best guess is that it's some sort of weird compiler bug involving
> >>> > system calls that leak memory on Windows-proper but not in Wine, but
> >>> > I really don't know.
> >>> > 
> >>> > http://www.dsource.org/projects/phobos/changeset/2331
> >>> > 
> >>> > paths changed:
> >>> > U   trunk/phobos/std/datetime.d
> >>> > 
> >>> > _______________________________________________
> >>> > phobos mailing list
> >>> > phobos at puremagic.com
> >>> > http://lists.puremagic.com/mailman/listinfo/phobos
> >>> 
> >>> If DMD is running out of memory at compile time, I doubt it has to do with weird compiler bugs. DMD running out of memory is usually a sign of heavy compile time string manipulation, CTFE or template usage. Remember, DMD keeps all template instantiations in memory and every time a string or array is touched, a new copy is created (bug 1382).
> >> 
> >> Yes. But to then have it not only work on Linux but on Wine and yet not work on Windows? The only real difference between running something under Wine and Windows is that the implementation of system calls as changed.
> > 
> > Well, is the crash due to a marginal difference in memory levels, or something major? i.e. is DMD's memory usage on Wine also very high, or is it relatively low.
> 
> I haven't had time to try to help with this.. but for what it's worth, the windows test box has 2 gigs of ram and is running 32bit windows.  So, there _ought_ to be pleanty of memory.

IIRC, looking at the dmd process on Linux, Wine, and Windows, Linux would max out at around 100MB,  Wine would max out at around 300MB, and Windows would go past 900MB and run out of memory (the numbers aren't terribly exact since I was essentially just watching the task manager). Why it ran out of memory between 900 and 1000MB, I don't know. My box has 64-bit Windows 7 and 16GB of memory (though obviously dmd2, as a 32-bit program, couldn't use more than around 4GB). So, it looks like dmd on Windows is declaring itself as out of memory well short of the actual system limit. Regardless of that however, it's definitely using way more memory than either Linux or Wine. All around, it's very weird.

I did open bug #5454 on the issue, and maybe someone dmd-savy will have time to take a look at it. I'll keep poking at it when I have time, trying to generate a reduced test case, but I really have no idea what could be causing it. Regardless, it seems like it's either a bug in dmd or in Windows (and you'd think that it would be dmd with the bug rather than the OS, but I don't know).

- Jonathan M Davis

It could be a difference in the Windows vs. Linux memory manager.  Did anyone ever try hooking the Boehm GC into DMD?

Sent from my iPhone

On Jan 14, 2011, at 11:58 AM, Jonathan M Davis <jmdavisProg at gmx.com> wrote:

> On Friday, January 14, 2011 10:43:26 Brad Roberts wrote:
>> On 1/14/2011 10:39 AM, Robert Jacques wrote:
>>> On Fri, 14 Jan 2011 11:19:34 -0500, Jonathan M Davis <jmdavisProg at gmx.com>
> wrote:
>>>> On Friday 14 January 2011 05:43:19 Robert Jacques wrote:
>>>>> On Fri, 14 Jan 2011 06:36:18 -0500, dsource.org <noreply at dsource.org>
>>>>> 
>>>>> wrote:
>>>>>> phobos commit, revision 2331
>>>>>> 
>>>>>> 
>>>>>> user: jmdavis
>>>>>> 
>>>>>> msg:
>>>>>> Re-enabled std.datetime for Windows but disabled its unit tests for
>>>>>> Windows.
>>>>>> 
>>>>>> For some reason, the std.datetime unit tests compile and pass without problems on Linux and in Wine on Linux, but on real Windows, the compiler runs out of memory when compiling them. So, for now, all of the std.datetime unit tests are versioned with testStdDateTime. On Posix, version=testdStdDateTime is used in std.datetime, so the tests will run, but that version is not set on Windows, so the unit tests don't run on Windows. So, if you want to run the std.datetime unit tests on Windows or under Wine or Linux, then compile with -version=testStdDateTime.
>>>>>> 
>>>>>> At the moment, all of the tests pass on both Linux and under Wine,
>>>>>> and the normal code does compile on proper, so I'm fairly certain
>>>>>> that it's fine
>>>>>> for use on Windows, but for now, I can't enable the tests on Windows.
>>>>>> My best guess is that it's some sort of weird compiler bug involving
>>>>>> system calls that leak memory on Windows-proper but not in Wine, but
>>>>>> I really don't know.
>>>>>> 
>>>>>> http://www.dsource.org/projects/phobos/changeset/2331
>>>>>> 
>>>>>> paths changed:
>>>>>> U   trunk/phobos/std/datetime.d
>>>>>> 
>>>>>> _______________________________________________
>>>>>> phobos mailing list
>>>>>> phobos at puremagic.com
>>>>>> http://lists.puremagic.com/mailman/listinfo/phobos
>>>>> 
>>>>> If DMD is running out of memory at compile time, I doubt it has to do with weird compiler bugs. DMD running out of memory is usually a sign of heavy compile time string manipulation, CTFE or template usage. Remember, DMD keeps all template instantiations in memory and every time a string or array is touched, a new copy is created (bug 1382).
>>>> 
>>>> Yes. But to then have it not only work on Linux but on Wine and yet not work on Windows? The only real difference between running something under Wine and Windows is that the implementation of system calls as changed.
>>> 
>>> Well, is the crash due to a marginal difference in memory levels, or something major? i.e. is DMD's memory usage on Wine also very high, or is it relatively low.
>> 
>> I haven't had time to try to help with this.. but for what it's worth, the windows test box has 2 gigs of ram and is running 32bit windows.  So, there _ought_ to be pleanty of memory.
> 
> IIRC, looking at the dmd process on Linux, Wine, and Windows, Linux would max out at around 100MB,  Wine would max out at around 300MB, and Windows would go past 900MB and run out of memory (the numbers aren't terribly exact since I was essentially just watching the task manager). Why it ran out of memory between 900 and 1000MB, I don't know. My box has 64-bit Windows 7 and 16GB of memory (though obviously dmd2, as a 32-bit program, couldn't use more than around 4GB). So, it looks like dmd on Windows is declaring itself as out of memory well short of the actual system limit. Regardless of that however, it's definitely using way more memory than either Linux or Wine. All around, it's very weird.
> 
> I did open bug #5454 on the issue, and maybe someone dmd-savy will have time to take a look at it. I'll keep poking at it when I have time, trying to generate a reduced test case, but I really have no idea what could be causing it. Regardless, it seems like it's either a bug in dmd or in Windows (and you'd think that it would be dmd with the bug rather than the OS, but I don't know).
> 
> - Jonathan M Davis
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

On Friday 14 January 2011 12:24:06 Sean Kelly wrote:
> It could be a difference in the Windows vs. Linux memory manager.  Did anyone ever try hooking the Boehm GC into DMD?

Actually, it looks like I screwed up in my testing. Wine probably has the problem too.

When building and running the tests on Linux and Windows, I built and ran (well, tried to run on Windows) the entirety of Phobos. However, since I have yet to be able to figure out how to compile druntime with Wine, I compiled and ran only core.time and std.datetime when testing Wine. So, I just tried doing that for all three.

On Windows and Wine, dmd uses about 370MB, but does not run out of memory when just compiling core.time and std.datetime. On Linux, it uses about 270MB (so, evidentally, I did a poor job reading the memory consumption on Linux before). So, there's a definite increase on both Windows and Wine, but neither runs out of memory when just compiling core.time and std.datetime. Windows definitely runs out when compiling all of Phobos with std.datetime's unit tests enabled, but it doesn't if it's just core.time and std.datetime. Linux does fine with all of Phobos, but my guess is that Wine does just as poorly as Windows if you compile all of Phobos' unit tests (though I haven't been able to test it).

So, it doesn't look like it's Windows vs Wine problem at all, but it's still a Windows vs Linux problem.

- Jonathan M Davis

Sounds like the Windows memory manager is just a bit more prone to fragmentation, or something like that.

On Jan 14, 2011, at 1:29 PM, Jonathan M Davis wrote:

> On Friday 14 January 2011 12:24:06 Sean Kelly wrote:
>> It could be a difference in the Windows vs. Linux memory manager.  Did anyone ever try hooking the Boehm GC into DMD?
> 
> Actually, it looks like I screwed up in my testing. Wine probably has the problem too.
> 
> When building and running the tests on Linux and Windows, I built and ran (well, tried to run on Windows) the entirety of Phobos. However, since I have yet to be able to figure out how to compile druntime with Wine, I compiled and ran only core.time and std.datetime when testing Wine. So, I just tried doing that for all three.
> 
> On Windows and Wine, dmd uses about 370MB, but does not run out of memory when just compiling core.time and std.datetime. On Linux, it uses about 270MB (so, evidentally, I did a poor job reading the memory consumption on Linux before). So, there's a definite increase on both Windows and Wine, but neither runs out of memory when just compiling core.time and std.datetime. Windows definitely runs out when compiling all of Phobos with std.datetime's unit tests enabled, but it doesn't if it's just core.time and std.datetime. Linux does fine with all of Phobos, but my guess is that Wine does just as poorly as Windows if you compile all of Phobos' unit tests (though I haven't been able to test it).
> 
> So, it doesn't look like it's Windows vs Wine problem at all, but it's still a Windows vs Linux problem.
> 
> - Jonathan M Davis
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

On Friday 14 January 2011 16:10:03 Sean Kelly wrote:
> Sounds like the Windows memory manager is just a bit more prone to fragmentation, or something like that.

Well, it's definitely the case that Windows is using more memory, but I'm wondering if part of the problem is that Windows creates one program to run all of the unit tests whereas it looks like the posix makefile creates separate programs for each module (I _think_ that that's what it's doing from the behavior that I see, but I'm far from an expert at reading makefiles). So, dmd _might_ run out of memory on Linux if it were to build all of the modules at once for unit testing instead of doing them separately. However, for some reason, dmd on Windows seems to run out of memory at a little over 900MB of memory consumed whereas I've definitely seen dmd on Linux hit at least the 2 or 3 GB mark when running out of memory ( http://d.puremagic.com/issues/show_bug.cgi?id=4984 ). So, I don't know what's going on there.

Regardless, dmd obviously has issues with using up to much memory when dealing with CTFE or templates - whether on Windows or Linux -  though it does appear to have more trouble on Windows than on Linux.

- Jonathan M Davis