While writing some unit tests for Dsymbol, I noticed that Dsymbol::toPrettyChars() leaks almost everywhere.  In the simple case where a symbol has no parent, it just returns toChars(), which does not leak (at least I don't think it does).  However, whenever the symbol has a parent (or many), the returned string is composed, which requires that it is allocated dynamically (via mem.malloc()). Even though the caller owns the string, and even though it is called dozens of times, it appears that none of the callers are properly disposing of the result.  Unfortunately, it is a bit messy to do so, because you must free the string *only* if it has a parent, which is a pretty bad implementation leak, IMO.  Here is a place where std::string would have worked nicely. ;)

I suspect this has gone unnoticed because A) dmd probably has a relatively small memory footprint to begin with or B) most invokations of toPrettyChars() are during a call to error(), so the compiler is about to quit anyway.  What to do?  Leave it alone?  Try to fix it?  Note that fixing it without changing toPrettyChars() would require adding 2-3 lines of code to almost every call.

Dave


P.S.  Incidentally, this bug is one that is not easily caught with assertions (where would you place the assert that the string was freed?).  Fortunately, it is caught by unit testing; but it could also have been caught by documenting that the caller owns the string.  This is why you really want all 3 approaches to code quality.

_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On Sun, Nov 11, 2012 at 8:26 AM, David Held <dmd@wyntrmute.com> wrote:
> While writing some unit tests for Dsymbol, I noticed that Dsymbol::toPrettyChars() leaks almost everywhere.  In the simple case where a symbol has no parent, it just returns toChars(), which does not leak (at least I don't think it does).  However, whenever the symbol has a parent (or many), the returned string is composed, which requires that it is allocated dynamically (via mem.malloc()). Even though the caller owns the string, and even though it is called dozens of times, it appears that none of the callers are properly disposing of the result.  Unfortunately, it is a bit messy to do so, because you must free the string *only* if it has a parent, which is a pretty bad implementation leak, IMO.  Here is a place where std::string would have worked nicely. ;)
>
> I suspect this has gone unnoticed because A) dmd probably has a relatively
> small memory footprint to begin with or B) most invokations of
> toPrettyChars() are during a call to error(), so the compiler is about to
> quit anyway.  What to do?  Leave it alone?  Try to fix it?  Note that fixing
> it without changing toPrettyChars() would require adding 2-3 lines of code
> to almost every call.
>
> Dave
>
>
> P.S.  Incidentally, this bug is one that is not easily caught with assertions (where would you place the assert that the string was freed?). Fortunately, it is caught by unit testing; but it could also have been caught by documenting that the caller owns the string.  This is why you really want all 3 approaches to code quality.

DMD as a whole is written for garbage collection. It just so happens that it doesn't use one at the moment, which means a lot of things will leak.

Regards,
Alex
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

This isn't a bug, dmd does not free memory (with some exceptions), it assumes a garbage collector is present.

On Sun, Nov 11, 2012 at 6:26 PM, David Held <dmd@wyntrmute.com> wrote:

> While writing some unit tests for Dsymbol, I noticed that Dsymbol::toPrettyChars() leaks almost everywhere.  In the simple case where a symbol has no parent, it just returns toChars(), which does not leak (at least I don't think it does).  However, whenever the symbol has a parent (or many), the returned string is composed, which requires that it is allocated dynamically (via mem.malloc()). Even though the caller owns the string, and even though it is called dozens of times, it appears that none of the callers are properly disposing of the result.  Unfortunately, it is a bit messy to do so, because you must free the string *only* if it has a parent, which is a pretty bad implementation leak, IMO.  Here is a place where std::string would have worked nicely. ;)
>
> I suspect this has gone unnoticed because A) dmd probably has a relatively
> small memory footprint to begin with or B) most invokations of
> toPrettyChars() are during a call to error(), so the compiler is about to
> quit anyway.  What to do?  Leave it alone?  Try to fix it?  Note that
> fixing it without changing toPrettyChars() would require adding 2-3 lines
> of code to almost every call.
>
> Dave
>
>
> P.S.  Incidentally, this bug is one that is not easily caught with
> assertions (where would you place the assert that the string was freed?).
>  Fortunately, it is caught by unit testing; but it could also have been
> caught by documenting that the caller owns the string.  This is why you
> really want all 3 approaches to code quality.
>
> ______________________________**_________________
> dmd-internals mailing list
> dmd-internals@puremagic.com
> http://lists.puremagic.com/**mailman/listinfo/dmd-internals<http://lists.puremagic.com/mailman/listinfo/dmd-internals>
>

I see.  And does anyone report running out of memory due to this?  It seems like it would mostly be a problem with CTFE, but could be a problem on any large project, I suppose.

Dave


On 11/10/2012 11:34 PM, Daniel Murphy wrote:
> This isn't a bug, dmd does not free memory (with some exceptions), it assumes a garbage collector is present.
>
> On Sun, Nov 11, 2012 at 6:26 PM, David Held <dmd@wyntrmute.com <mailto:dmd@wyntrmute.com>> wrote:
>
>     While writing some unit tests for Dsymbol, I noticed that
>     Dsymbol::toPrettyChars() leaks almost everywhere.  In the simple
>     case where a symbol has no parent, it just returns toChars(),
>     which does not leak (at least I don't think it does).  However,
>     whenever the symbol has a parent (or many), the returned string is
>     composed, which requires that it is allocated dynamically (via
>     mem.malloc()). Even though the caller owns the string, and even
>     though it is called dozens of times, it appears that none of the
>     callers are properly disposing of the result.  Unfortunately, it
>     is a bit messy to do so, because you must free the string *only*
>     if it has a parent, which is a pretty bad implementation leak,
>     IMO.  Here is a place where std::string would have worked nicely. ;)
>
>     I suspect this has gone unnoticed because A) dmd probably has a
>     relatively small memory footprint to begin with or B) most
>     invokations of toPrettyChars() are during a call to error(), so
>     the compiler is about to quit anyway.  What to do?  Leave it
>     alone?  Try to fix it?  Note that fixing it without changing
>     toPrettyChars() would require adding 2-3 lines of code to almost
>     every call.
>
>     Dave
>
>
>     P.S.  Incidentally, this bug is one that is not easily caught with
>     assertions (where would you place the assert that the string was
>     freed?).  Fortunately, it is caught by unit testing; but it could
>     also have been caught by documenting that the caller owns the
>     string.  This is why you really want all 3 approaches to code quality.
>
>     _______________________________________________
>     dmd-internals mailing list
>     dmd-internals@puremagic.com <mailto:dmd-internals@puremagic.com>
>     http://lists.puremagic.com/mailman/listinfo/dmd-internals
>
>
>
>
> _______________________________________________
> dmd-internals mailing list
> dmd-internals@puremagic.com
> http://lists.puremagic.com/mailman/listinfo/dmd-internals

Yes, it's very easy to make it run out of memory with ctfe or recursive templates.

The garbage collector works as far as I know, but needs some performance tuning.

On Sun, Nov 11, 2012 at 7:06 PM, David Held <dmd@wyntrmute.com> wrote:

>  I see.  And does anyone report running out of memory due to this?  It
> seems like it would mostly be a problem with CTFE, but could be a problem
> on any large project, I suppose.
>
> Dave
>
>
>
> On 11/10/2012 11:34 PM, Daniel Murphy wrote:
>
> This isn't a bug, dmd does not free memory (with some exceptions), it assumes a garbage collector is present.
>
> On Sun, Nov 11, 2012 at 6:26 PM, David Held <dmd@wyntrmute.com> wrote:
>
>> While writing some unit tests for Dsymbol, I noticed that Dsymbol::toPrettyChars() leaks almost everywhere.  In the simple case where a symbol has no parent, it just returns toChars(), which does not leak (at least I don't think it does).  However, whenever the symbol has a parent (or many), the returned string is composed, which requires that it is allocated dynamically (via mem.malloc()). Even though the caller owns the string, and even though it is called dozens of times, it appears that none of the callers are properly disposing of the result.  Unfortunately, it is a bit messy to do so, because you must free the string *only* if it has a parent, which is a pretty bad implementation leak, IMO.  Here is a place where std::string would have worked nicely. ;)
>>
>> I suspect this has gone unnoticed because A) dmd probably has a
>> relatively small memory footprint to begin with or B) most invokations of
>> toPrettyChars() are during a call to error(), so the compiler is about to
>> quit anyway.  What to do?  Leave it alone?  Try to fix it?  Note that
>> fixing it without changing toPrettyChars() would require adding 2-3 lines
>> of code to almost every call.
>>
>> Dave
>>
>>
>> P.S.  Incidentally, this bug is one that is not easily caught with
>> assertions (where would you place the assert that the string was freed?).
>>  Fortunately, it is caught by unit testing; but it could also have been
>> caught by documenting that the caller owns the string.  This is why you
>> really want all 3 approaches to code quality.
>>
>> _______________________________________________
>> dmd-internals mailing list
>> dmd-internals@puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/dmd-internals
>>
>
>
>
> _______________________________________________
> dmd-internals mailing listdmd-internals@puremagic.comhttp://lists.puremagic.com/mailman/listinfo/dmd-internals
>
>
>
> _______________________________________________
> dmd-internals mailing list
> dmd-internals@puremagic.com
> http://lists.puremagic.com/mailman/listinfo/dmd-internals
>

Compiling the program I'm working on, dmd require more than 2Gb of memory. Looking at the memory consumption, it doesn't look like anything is freed.

I don't use a lot of CTFE. Many templates, but most of them are not recursive.

Compilation time and resource really is becoming an issue.


2012/11/11 Daniel Murphy <yebblies@gmail.com>

> Yes, it's very easy to make it run out of memory with ctfe or recursive templates.
>
> The garbage collector works as far as I know, but needs some performance tuning.
>
>
> On Sun, Nov 11, 2012 at 7:06 PM, David Held <dmd@wyntrmute.com> wrote:
>
>>  I see.  And does anyone report running out of memory due to this?  It
>> seems like it would mostly be a problem with CTFE, but could be a problem
>> on any large project, I suppose.
>>
>> Dave
>>
>>
>>
>> On 11/10/2012 11:34 PM, Daniel Murphy wrote:
>>
>> This isn't a bug, dmd does not free memory (with some exceptions), it assumes a garbage collector is present.
>>
>> On Sun, Nov 11, 2012 at 6:26 PM, David Held <dmd@wyntrmute.com> wrote:
>>
>>> While writing some unit tests for Dsymbol, I noticed that Dsymbol::toPrettyChars() leaks almost everywhere.  In the simple case where a symbol has no parent, it just returns toChars(), which does not leak (at least I don't think it does).  However, whenever the symbol has a parent (or many), the returned string is composed, which requires that it is allocated dynamically (via mem.malloc()). Even though the caller owns the string, and even though it is called dozens of times, it appears that none of the callers are properly disposing of the result.  Unfortunately, it is a bit messy to do so, because you must free the string *only* if it has a parent, which is a pretty bad implementation leak, IMO.  Here is a place where std::string would have worked nicely. ;)
>>>
>>> I suspect this has gone unnoticed because A) dmd probably has a
>>> relatively small memory footprint to begin with or B) most invokations of
>>> toPrettyChars() are during a call to error(), so the compiler is about to
>>> quit anyway.  What to do?  Leave it alone?  Try to fix it?  Note that
>>> fixing it without changing toPrettyChars() would require adding 2-3 lines
>>> of code to almost every call.
>>>
>>> Dave
>>>
>>>
>>> P.S.  Incidentally, this bug is one that is not easily caught with
>>> assertions (where would you place the assert that the string was freed?).
>>>  Fortunately, it is caught by unit testing; but it could also have been
>>> caught by documenting that the caller owns the string.  This is why you
>>> really want all 3 approaches to code quality.
>>>
>>> _______________________________________________
>>> dmd-internals mailing list
>>> dmd-internals@puremagic.com
>>> http://lists.puremagic.com/mailman/listinfo/dmd-internals
>>>
>>
>>
>>
>> _______________________________________________
>> dmd-internals mailing listdmd-internals@puremagic.comhttp://lists.puremagic.com/mailman/listinfo/dmd-internals
>>
>>
>>
>> _______________________________________________
>> dmd-internals mailing list
>> dmd-internals@puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/dmd-internals
>>
>
>
> _______________________________________________
> dmd-internals mailing list
> dmd-internals@puremagic.com
> http://lists.puremagic.com/mailman/listinfo/dmd-internals
>

On 11/10/2012 11:26 PM, David Held wrote:
> While writing some unit tests for Dsymbol, I noticed that Dsymbol::toPrettyChars() leaks almost everywhere.

It actually isn't a bug.

dmd does not bother to keep track of memory for the purpose of freeing it. It behaves as if there's a garbage collector.

There is a garbage collector for dmd, but it is left out of the build because it slows the compiler down more than it helps.

Excessive memory consumption has turned out to be a problem for the CTFE implementation, however.
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On 11/11/2012 12:06 AM, David Held wrote:
> I see.  And does anyone report running out of memory due to this?  It seems like it would mostly be a problem with CTFE, but could be a problem on any large project, I suppose.

It does cause problems on my FreeBSD machine which has only 200Mb of memory.

On 11/11/2012 12:48 PM, Walter Bright wrote:
>
> On 11/11/2012 12:06 AM, David Held wrote:
>> I see.  And does anyone report running out of memory due to this?  It seems like it would mostly be a problem with CTFE, but could be a problem on any large project, I suppose.
>
> It does cause problems on my FreeBSD machine which has only 200Mb of memory.

So what do you think about adding some smart pointers to at least recover most of the memory?

Dave

On 11 Nov 2012, at 20:58, deadal nix wrote:
> Compiling the program I'm working on, dmd require more than 2Gb of memory.
> Looking at the memory consumption, it doesn't look like anything is freed.

Yes, unfortunately this is currently the expected behavior. The GC was enabled briefly some time ago, but has since been disabled again as the incurred slowdown was perceived to be too high. In the long run, we definitely need to do something about this.

David
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals