View mode: basic / threaded / horizontal-split · Log in · Help
April 02, 2010
Memory Corruption with AAs
Has anyone else still been noticing difficult to reproduce memory corruption
issues in the presence of associative arrays with 2.042?  They seem to happen
very infrequently and non-deterministically.  I can only reproduce them in the
context of a large program.  However, they don't occur in 2.040 (the release
before the array stomping patch), and they are clearly a result of memory
corruption, as contents of arrays change from what I expect them to be to
completely random-looking values inside a loop that does a lot of memory
management and uses AAs heavily but doesn't modify the values.
April 02, 2010
Re: Memory Corruption with AAs
dsimcha wrote:
> Has anyone else still been noticing difficult to reproduce memory corruption
> issues in the presence of associative arrays with 2.042?  They seem to happen
> very infrequently and non-deterministically.  I can only reproduce them in the
> context of a large program.  However, they don't occur in 2.040 (the release
> before the array stomping patch), and they are clearly a result of memory
> corruption, as contents of arrays change from what I expect them to be to
> completely random-looking values inside a loop that does a lot of memory
> management and uses AAs heavily but doesn't modify the values.

1. is it multithreaded?
2. does your code have any dangling pointers into AAs?
April 02, 2010
Re: Memory Corruption with AAs
On Fri, 02 Apr 2010 13:15:36 -0400, dsimcha <dsimcha@yahoo.com> wrote:

> Has anyone else still been noticing difficult to reproduce memory  
> corruption
> issues in the presence of associative arrays with 2.042?  They seem to  
> happen
> very infrequently and non-deterministically.  I can only reproduce them  
> in the
> context of a large program.  However, they don't occur in 2.040 (the  
> release
> before the array stomping patch), and they are clearly a result of memory
> corruption, as contents of arrays change from what I expect them to be to
> completely random-looking values inside a loop that does a lot of memory
> management and uses AAs heavily but doesn't modify the values.

Are you using the latest trunk for druntime, or the stock 2.042 version?   
Because the AAs have been changed significantly since 2.042.

Not saying I'm blaming that, but let's make sure we are all on the same  
page.

What is the AA key/value types?

Do you use appending at all in your code?

The AA code only uses appending in one location, that is when setting the  
length of its array.

Can you narrow down what your code is doing when the corruption occurs? (I  
realize this might be impossible, but just thought I'd ask)

-Steve
April 02, 2010
Re: Memory Corruption with AAs
On Fri, 02 Apr 2010 13:15:36 -0400, dsimcha <dsimcha@yahoo.com> wrote:

> Has anyone else still been noticing difficult to reproduce memory  
> corruption
> issues in the presence of associative arrays with 2.042?  They seem to  
> happen
> very infrequently and non-deterministically.  I can only reproduce them  
> in the
> context of a large program.  However, they don't occur in 2.040 (the  
> release
> before the array stomping patch), and they are clearly a result of memory
> corruption, as contents of arrays change from what I expect them to be to
> completely random-looking values inside a loop that does a lot of memory
> management and uses AAs heavily but doesn't modify the values.

I just thought of a way to rule out or not the stomping patch, as long as  
your code does not depend on preventing array stomping.

Change your runtime's version of lifetime.d to the version before the  
append stomping patch:  
http://www.dsource.org/projects/druntime/browser/trunk/src/rt/lifetime.d?rev=251

This version should be forwards compatible with the runtime in 2.042, so  
you should be able to run your code with this runtime.  See if the errors  
still occur.

-Steve
April 02, 2010
Re: Memory Corruption with AAs
== Quote from Walter Bright (newshound1@digitalmars.com)'s article
> dsimcha wrote:
> > Has anyone else still been noticing difficult to reproduce memory corruption
> > issues in the presence of associative arrays with 2.042?  They seem to happen
> > very infrequently and non-deterministically.  I can only reproduce them in the
> > context of a large program.  However, they don't occur in 2.040 (the release
> > before the array stomping patch), and they are clearly a result of memory
> > corruption, as contents of arrays change from what I expect them to be to
> > completely random-looking values inside a loop that does a lot of memory
> > management and uses AAs heavily but doesn't modify the values.
> 1. is it multithreaded?
> 2. does your code have any dangling pointers into AAs?

The program as a whole is multithreaded, but the part where the bug occurs is an
initialization routine that is executed before any threads other than the main one
are launched.

As far as the dangling pointers question, I don't understand how there could be
dangling pointers into GC-managed memory, since if there are pointers to it, it
won't be freed.  (Ignoring dirty tricks that I'm not using in this case.)
April 02, 2010
Re: Memory Corruption with AAs
dsimcha wrote:
> == Quote from Walter Bright (newshound1@digitalmars.com)'s article
>> dsimcha wrote:
>>> Has anyone else still been noticing difficult to reproduce memory corruption
>>> issues in the presence of associative arrays with 2.042?  They seem to happen
>>> very infrequently and non-deterministically.  I can only reproduce them in the
>>> context of a large program.  However, they don't occur in 2.040 (the release
>>> before the array stomping patch), and they are clearly a result of memory
>>> corruption, as contents of arrays change from what I expect them to be to
>>> completely random-looking values inside a loop that does a lot of memory
>>> management and uses AAs heavily but doesn't modify the values.
>> 1. is it multithreaded?
>> 2. does your code have any dangling pointers into AAs?
> 
> The program as a whole is multithreaded, but the part where the bug occurs is an
> initialization routine that is executed before any threads other than the main one
> are launched.

It should be easier to find then, by removing all the main code and everything 
it calls.

> 
> As far as the dangling pointers question, I don't understand how there could be
> dangling pointers into GC-managed memory, since if there are pointers to it, it
> won't be freed.  (Ignoring dirty tricks that I'm not using in this case.)

What I meant was, do you save any pointers into the AAs, as in:

    auto p = &aa[key];

?
April 02, 2010
Re: Memory Corruption with AAs
== Quote from Walter Bright (newshound1@digitalmars.com)'s article
> dsimcha wrote:
> > == Quote from Walter Bright (newshound1@digitalmars.com)'s article
> >> dsimcha wrote:
> >>> Has anyone else still been noticing difficult to reproduce memory corruption
> >>> issues in the presence of associative arrays with 2.042?  They seem to happen
> >>> very infrequently and non-deterministically.  I can only reproduce them in the
> >>> context of a large program.  However, they don't occur in 2.040 (the release
> >>> before the array stomping patch), and they are clearly a result of memory
> >>> corruption, as contents of arrays change from what I expect them to be to
> >>> completely random-looking values inside a loop that does a lot of memory
> >>> management and uses AAs heavily but doesn't modify the values.
> >> 1. is it multithreaded?
> >> 2. does your code have any dangling pointers into AAs?
> >
> > The program as a whole is multithreaded, but the part where the bug occurs is an
> > initialization routine that is executed before any threads other than the main one
> > are launched.
> It should be easier to find then, by removing all the main code and everything
> it calls.

The code has so many dependencies (both other code from the same project and
libraries) and is such a mess (because it's a research prototype that evolved more
than it was designed and also has all kinds of speed hacks) that it would probably
be easier to try to reproduce it from scratch.  I'll try tonight because I've got
a long train ride with nothing else to do anyhow.

> >
> > As far as the dangling pointers question, I don't understand how there could be
> > dangling pointers into GC-managed memory, since if there are pointers to it, it
> > won't be freed.  (Ignoring dirty tricks that I'm not using in this case.)
> What I meant was, do you save any pointers into the AAs, as in:
>      auto p = &aa[key];
> ?

No, I definitely wasn't.  I almost never do this with any data structure other
than an array because, even if it works for now, I consider it a horrible
violation of encapsulation because you're relying on the details of how the data
structure manipulates memory.  This is also why, when I designed RandAA I didn't
see this as an issue until you pointed it out to me.
April 02, 2010
Re: Memory Corruption with AAs
dsimcha wrote:
> I almost never do this with any data structure other
> than an array because, even if it works for now, I consider it a horrible
> violation of encapsulation because you're relying on the details of how the data
> structure manipulates memory.  This is also why, when I designed RandAA I didn't
> see this as an issue until you pointed it out to me.

Andrei is working on the design of the D collection class library. After much 
thought and research, he finally came to the conclusion that a collection class 
should not allow the address of a member to be taken. I think his reasoning on 
the issue is pretty sound, and is consistent with your take on it.
April 03, 2010
Re: Memory Corruption with AAs
dsimcha wrote:
> == Quote from Walter Bright (newshound1@digitalmars.com)'s article
>> dsimcha wrote:
>>> == Quote from Walter Bright (newshound1@digitalmars.com)'s article
>>>> dsimcha wrote:
>>>>> Has anyone else still been noticing difficult to reproduce memory corruption
>>>>> issues in the presence of associative arrays with 2.042?  They seem to happen
>>>>> very infrequently and non-deterministically.  I can only reproduce them in the
>>>>> context of a large program.  However, they don't occur in 2.040 (the release
>>>>> before the array stomping patch), and they are clearly a result of memory
>>>>> corruption, as contents of arrays change from what I expect them to be to
>>>>> completely random-looking values inside a loop that does a lot of memory
>>>>> management and uses AAs heavily but doesn't modify the values.
>>>> 1. is it multithreaded?
>>>> 2. does your code have any dangling pointers into AAs?
>>> The program as a whole is multithreaded, but the part where the bug occurs is an
>>> initialization routine that is executed before any threads other than the main one
>>> are launched.
>> It should be easier to find then, by removing all the main code and everything
>> it calls.
> 
> The code has so many dependencies (both other code from the same project and
> libraries) and is such a mess (because it's a research prototype that evolved more
> than it was designed and also has all kinds of speed hacks) that it would probably
> be easier to try to reproduce it from scratch.  I'll try tonight because I've got
> a long train ride with nothing else to do anyhow.
> 
>>> As far as the dangling pointers question, I don't understand how there could be
>>> dangling pointers into GC-managed memory, since if there are pointers to it, it
>>> won't be freed.  (Ignoring dirty tricks that I'm not using in this case.)
>> What I meant was, do you save any pointers into the AAs, as in:
>>      auto p = &aa[key];
>> ?
> 
> No, I definitely wasn't.  I almost never do this with any data structure other
> than an array because, even if it works for now, I consider it a horrible
> violation of encapsulation because you're relying on the details of how the data
> structure manipulates memory.  This is also why, when I designed RandAA I didn't
> see this as an issue until you pointed it out to me.

I am having exactly the same situation. My project, which is also quite 
big, crashes randomly in 2.041 and 2.042 (without patches), but it works 
fine in 2.040. I didn't bother reporting because i thought i was doing 
something wrong (being a newbie in D).

Mine is single-threaded, no pointers in AAs.

j.
April 03, 2010
Re: Memory Corruption with AAs
Jordi wrote:
> I am having exactly the same situation.

Any chance you can reduce it to a small test case?
« First   ‹ Prev
1 2 3
Top | Discussion index | About this forum | D home