Has anyone else still been noticing difficult to reproduce memory corruption issues in the presence of associative arrays with 2.042?  They seem to happen very infrequently and non-deterministically.  I can only reproduce them in the context of a large program.  However, they don't occur in 2.040 (the release before the array stomping patch), and they are clearly a result of memory corruption, as contents of arrays change from what I expect them to be to completely random-looking values inside a loop that does a lot of memory management and uses AAs heavily but doesn't modify the values.

dsimcha wrote:
> Has anyone else still been noticing difficult to reproduce memory corruption
> issues in the presence of associative arrays with 2.042?  They seem to happen
> very infrequently and non-deterministically.  I can only reproduce them in the
> context of a large program.  However, they don't occur in 2.040 (the release
> before the array stomping patch), and they are clearly a result of memory
> corruption, as contents of arrays change from what I expect them to be to
> completely random-looking values inside a loop that does a lot of memory
> management and uses AAs heavily but doesn't modify the values.

1. is it multithreaded?
2. does your code have any dangling pointers into AAs?

On Fri, 02 Apr 2010 13:15:36 -0400, dsimcha <dsimcha@yahoo.com> wrote:

> Has anyone else still been noticing difficult to reproduce memory corruption
> issues in the presence of associative arrays with 2.042?  They seem to happen
> very infrequently and non-deterministically.  I can only reproduce them in the
> context of a large program.  However, they don't occur in 2.040 (the release
> before the array stomping patch), and they are clearly a result of memory
> corruption, as contents of arrays change from what I expect them to be to
> completely random-looking values inside a loop that does a lot of memory
> management and uses AAs heavily but doesn't modify the values.

Are you using the latest trunk for druntime, or the stock 2.042 version?  Because the AAs have been changed significantly since 2.042.

Not saying I'm blaming that, but let's make sure we are all on the same page.

What is the AA key/value types?

Do you use appending at all in your code?

The AA code only uses appending in one location, that is when setting the length of its array.

Can you narrow down what your code is doing when the corruption occurs? (I realize this might be impossible, but just thought I'd ask)

-Steve

On Fri, 02 Apr 2010 13:15:36 -0400, dsimcha <dsimcha@yahoo.com> wrote:

> Has anyone else still been noticing difficult to reproduce memory corruption
> issues in the presence of associative arrays with 2.042?  They seem to happen
> very infrequently and non-deterministically.  I can only reproduce them in the
> context of a large program.  However, they don't occur in 2.040 (the release
> before the array stomping patch), and they are clearly a result of memory
> corruption, as contents of arrays change from what I expect them to be to
> completely random-looking values inside a loop that does a lot of memory
> management and uses AAs heavily but doesn't modify the values.

I just thought of a way to rule out or not the stomping patch, as long as your code does not depend on preventing array stomping.

Change your runtime's version of lifetime.d to the version before the append stomping patch: http://www.dsource.org/projects/druntime/browser/trunk/src/rt/lifetime.d?rev=251

This version should be forwards compatible with the runtime in 2.042, so you should be able to run your code with this runtime.  See if the errors still occur.

-Steve

== Quote from Walter Bright (newshound1@digitalmars.com)'s article
> dsimcha wrote:
> > Has anyone else still been noticing difficult to reproduce memory corruption issues in the presence of associative arrays with 2.042?  They seem to happen very infrequently and non-deterministically.  I can only reproduce them in the context of a large program.  However, they don't occur in 2.040 (the release before the array stomping patch), and they are clearly a result of memory corruption, as contents of arrays change from what I expect them to be to completely random-looking values inside a loop that does a lot of memory management and uses AAs heavily but doesn't modify the values.
> 1. is it multithreaded?
> 2. does your code have any dangling pointers into AAs?

The program as a whole is multithreaded, but the part where the bug occurs is an initialization routine that is executed before any threads other than the main one are launched.

As far as the dangling pointers question, I don't understand how there could be dangling pointers into GC-managed memory, since if there are pointers to it, it won't be freed.  (Ignoring dirty tricks that I'm not using in this case.)

dsimcha wrote:
> == Quote from Walter Bright (newshound1@digitalmars.com)'s article
>> dsimcha wrote:
>>> Has anyone else still been noticing difficult to reproduce memory corruption
>>> issues in the presence of associative arrays with 2.042?  They seem to happen
>>> very infrequently and non-deterministically.  I can only reproduce them in the
>>> context of a large program.  However, they don't occur in 2.040 (the release
>>> before the array stomping patch), and they are clearly a result of memory
>>> corruption, as contents of arrays change from what I expect them to be to
>>> completely random-looking values inside a loop that does a lot of memory
>>> management and uses AAs heavily but doesn't modify the values.
>> 1. is it multithreaded?
>> 2. does your code have any dangling pointers into AAs?
> 
> The program as a whole is multithreaded, but the part where the bug occurs is an
> initialization routine that is executed before any threads other than the main one
> are launched.

It should be easier to find then, by removing all the main code and everything it calls.

> 
> As far as the dangling pointers question, I don't understand how there could be
> dangling pointers into GC-managed memory, since if there are pointers to it, it
> won't be freed.  (Ignoring dirty tricks that I'm not using in this case.)

What I meant was, do you save any pointers into the AAs, as in:

    auto p = &aa[key];

?

== Quote from Walter Bright (newshound1@digitalmars.com)'s article
> dsimcha wrote:
> > == Quote from Walter Bright (newshound1@digitalmars.com)'s article
> >> dsimcha wrote:
> >>> Has anyone else still been noticing difficult to reproduce memory corruption issues in the presence of associative arrays with 2.042?  They seem to happen very infrequently and non-deterministically.  I can only reproduce them in the context of a large program.  However, they don't occur in 2.040 (the release before the array stomping patch), and they are clearly a result of memory corruption, as contents of arrays change from what I expect them to be to completely random-looking values inside a loop that does a lot of memory management and uses AAs heavily but doesn't modify the values.
> >> 1. is it multithreaded?
> >> 2. does your code have any dangling pointers into AAs?
> >
> > The program as a whole is multithreaded, but the part where the bug occurs is an initialization routine that is executed before any threads other than the main one are launched.
> It should be easier to find then, by removing all the main code and everything it calls.

The code has so many dependencies (both other code from the same project and libraries) and is such a mess (because it's a research prototype that evolved more than it was designed and also has all kinds of speed hacks) that it would probably be easier to try to reproduce it from scratch.  I'll try tonight because I've got a long train ride with nothing else to do anyhow.

> >
> > As far as the dangling pointers question, I don't understand how there could be dangling pointers into GC-managed memory, since if there are pointers to it, it won't be freed.  (Ignoring dirty tricks that I'm not using in this case.)
> What I meant was, do you save any pointers into the AAs, as in:
>      auto p = &aa[key];
> ?

No, I definitely wasn't.  I almost never do this with any data structure other than an array because, even if it works for now, I consider it a horrible violation of encapsulation because you're relying on the details of how the data structure manipulates memory.  This is also why, when I designed RandAA I didn't see this as an issue until you pointed it out to me.

dsimcha wrote:
> I almost never do this with any data structure other
> than an array because, even if it works for now, I consider it a horrible
> violation of encapsulation because you're relying on the details of how the data
> structure manipulates memory.  This is also why, when I designed RandAA I didn't
> see this as an issue until you pointed it out to me.

Andrei is working on the design of the D collection class library. After much thought and research, he finally came to the conclusion that a collection class should not allow the address of a member to be taken. I think his reasoning on the issue is pretty sound, and is consistent with your take on it.

dsimcha wrote:
> == Quote from Walter Bright (newshound1@digitalmars.com)'s article
>> dsimcha wrote:
>>> == Quote from Walter Bright (newshound1@digitalmars.com)'s article
>>>> dsimcha wrote:
>>>>> Has anyone else still been noticing difficult to reproduce memory corruption
>>>>> issues in the presence of associative arrays with 2.042?  They seem to happen
>>>>> very infrequently and non-deterministically.  I can only reproduce them in the
>>>>> context of a large program.  However, they don't occur in 2.040 (the release
>>>>> before the array stomping patch), and they are clearly a result of memory
>>>>> corruption, as contents of arrays change from what I expect them to be to
>>>>> completely random-looking values inside a loop that does a lot of memory
>>>>> management and uses AAs heavily but doesn't modify the values.
>>>> 1. is it multithreaded?
>>>> 2. does your code have any dangling pointers into AAs?
>>> The program as a whole is multithreaded, but the part where the bug occurs is an
>>> initialization routine that is executed before any threads other than the main one
>>> are launched.
>> It should be easier to find then, by removing all the main code and everything
>> it calls.
> 
> The code has so many dependencies (both other code from the same project and
> libraries) and is such a mess (because it's a research prototype that evolved more
> than it was designed and also has all kinds of speed hacks) that it would probably
> be easier to try to reproduce it from scratch.  I'll try tonight because I've got
> a long train ride with nothing else to do anyhow.
> 
>>> As far as the dangling pointers question, I don't understand how there could be
>>> dangling pointers into GC-managed memory, since if there are pointers to it, it
>>> won't be freed.  (Ignoring dirty tricks that I'm not using in this case.)
>> What I meant was, do you save any pointers into the AAs, as in:
>>      auto p = &aa[key];
>> ?
> 
> No, I definitely wasn't.  I almost never do this with any data structure other
> than an array because, even if it works for now, I consider it a horrible
> violation of encapsulation because you're relying on the details of how the data
> structure manipulates memory.  This is also why, when I designed RandAA I didn't
> see this as an issue until you pointed it out to me.

I am having exactly the same situation. My project, which is also quite big, crashes randomly in 2.041 and 2.042 (without patches), but it works fine in 2.040. I didn't bother reporting because i thought i was doing something wrong (being a newbie in D).

Mine is single-threaded, no pointers in AAs.

j.

Jordi wrote:
> I am having exactly the same situation.

Any chance you can reduce it to a small test case?