-------- Original Message --------
Subject: Memory Corruption with AAs
Date: Fri, 2 Apr 2010 17:15:36 +0000 (UTC)
From: dsimcha <dsimcha at yahoo.com>
Organization: XXX News Server
Newsgroups: digitalmars.D

Has anyone else still been noticing difficult to reproduce memory corruption issues in the presence of associative arrays with 2.042?  They seem to happen very infrequently and non-deterministically.  I can only reproduce them in the context of a large program.  However, they don't occur in 2.040 (the release before the array stomping patch), and they are clearly a result of memory corruption, as contents of arrays change from what I expect them to be to completely random-looking values inside a loop that does a lot of memory management and uses AAs heavily but doesn't modify the values.

Haven't noticed, but then I couldn't use D in a while.

Andrei

Walter Bright wrote:
> 
> 
> -------- Original Message --------
> Subject: Memory Corruption with AAs
> Date: Fri, 2 Apr 2010 17:15:36 +0000 (UTC)
> From: dsimcha <dsimcha at yahoo.com>
> Organization: XXX News Server
> Newsgroups: digitalmars.D
> 
> Has anyone else still been noticing difficult to reproduce memory
> corruption
> issues in the presence of associative arrays with 2.042?  They seem to
> happen
> very infrequently and non-deterministically.  I can only reproduce them
> in the
> context of a large program.  However, they don't occur in 2.040 (the
> release
> before the array stomping patch), and they are clearly a result of memory
> corruption, as contents of arrays change from what I expect them to be to
> completely random-looking values inside a loop that does a lot of memory
> management and uses AAs heavily but doesn't modify the values.
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

On Fri, Apr 2, 2010 at 10:34 PM, Andrei Alexandrescu <andrei at erdani.com> wrote:
> Haven't noticed, but then I couldn't use D in a while.
>
> Andrei
>
> Walter Bright wrote:
>>
>>
>> -------- Original Message --------
>> Subject: Memory Corruption with AAs
>> Date: Fri, 2 Apr 2010 17:15:36 +0000 (UTC)
>> From: dsimcha <dsimcha at yahoo.com>
>> Organization: XXX News Server
>> Newsgroups: digitalmars.D
>>
>> Has anyone else still been noticing difficult to reproduce memory
>> corruption
>> issues in the presence of associative arrays with 2.042? ?They seem to
>> happen
>> very infrequently and non-deterministically. ?I can only reproduce them in
>> the
>> context of a large program. ?However, they don't occur in 2.040 (the
>> release
>> before the array stomping patch), and they are clearly a result of memory
>> corruption, as contents of arrays change from what I expect them to be to
>> completely random-looking values inside a loop that does a lot of memory
>> management and uses AAs heavily but doesn't modify the values.
>> _______________________________________________
>> phobos mailing list
>> phobos at puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/phobos
>
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos
>

I have tried updating ddmd to 2.042 recently and it crashed upon building druntime (deterministically). Reverted back to 2.031 for now. I'll try to test other versions to track down the regression point and report back soon. Might be just a bug, though.

Through debugging Ellery Newcomer's test case (see http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=108685), I have found another bug in the array append patch.

See if you can spot it, I literally had to stare at this code for about 40 minutes, confused by its behavior, before I finally saw the problem.  The function below searches the LRU cache for a blockinfo (base pointer, size, and attributes) containing an interior pointer:

struct BlkInfo
{
    void*  base;
    size_t size;
    uint   attr;
}

BlkInfo *__getBlkInfo(void *interior)
{
        // try to do a smart lookup, using __nextBlkIdx as the "head"
        BlkInfo *ptr = __blkcache.ptr;
        for(int i = __nextBlkIdx; i >= 0; --i)
        {
            if(ptr[i].base <= interior && (interior - ptr[i].base) < ptr.size)
                return ptr + i;
        }

        for(int i = N_CACHE_BLOCKS - 1; i > __nextBlkIdx; --i)
        {
            if(ptr[i].base <= interior && (interior - ptr[i].base) < ptr.size)
                return ptr + i;
        }
        return null; // not in cache.
}

to give you a hint, here is the erroneous behavior that puzzled me:

given a pointer 0x38bd0, the function returned a blockinfo that started at 0x38bc0 with size 16.

If you can't figure it out, here is the answer: http://www.dsource.org/projects/druntime/changeset/282

I'm waiting to hear back from Ellery that the new code fixes the problem, and I also pinged David for his issue with AAs.  If all is OK, I recommend a patch release.  I'm not sure the state of the AAs in trunk, since Walter has changed them significantly -- we could be fixing one bug and introducing many more.  Given how simple this fix is, it might be good to release just this fix in order to have a stable version that doesn't do memory stomping.

-Steve



----- Original Message ----
> From: Walter Bright <walter at digitalmars.com>
> To: Discuss the phobos library for D <phobos at puremagic.com>
> Sent: Fri, April 2, 2010 1:29:18 PM
> Subject: [phobos] Looks like more problems with the array append patches
> 
> 

-------- Original Message --------
Subject: Memory Corruption with
> AAs
Date: Fri, 2 Apr 2010 17:15:36 +0000 (UTC)
From: dsimcha <
> ymailto="mailto:dsimcha at yahoo.com" href="mailto:dsimcha at yahoo.com">dsimcha at yahoo.com>
Organization: XXX
> News Server
Newsgroups: digitalmars.D

Has anyone else still been
> noticing difficult to reproduce memory corruption
issues in the presence of
> associative arrays with 2.042?  They seem to happen
very infrequently
> and non-deterministically.  I can only reproduce them in the
context of
> a large program.  However, they don't occur in 2.040 (the release
before
> the array stomping patch), and they are clearly a result of memory
corruption, as contents of arrays change from what I expect them to be
> to
completely random-looking values inside a loop that does a lot of
> memory
management and uses AAs heavily but doesn't modify the
> values.
_______________________________________________
phobos mailing
> list

> href="mailto:phobos at puremagic.com">phobos at puremagic.com
http://lists.puremagic.com/mailman/listinfo/phobos

----- Original Message ----

> From: Steve Schveighoffer <schveiguy at yahoo.com>

> I'm waiting to hear back from Ellery that the new code fixes the problem, and I also pinged David for his issue with AAs.  If all is OK, I recommend a patch release.  I'm not sure the state of the AAs in trunk, since Walter has changed them significantly -- we could be fixing one bug and introducing many more.  Given how simple this fix is, it might be good to release just this fix in order to have a stable version that doesn't do memory stomping.

Ellery confirmed, the patch fixes his problem.  I've posted messages on the newsgroup.

Still waiting to hear from David.

-Steve

Thanks Steve for following this up so effectively.

Andrei

On 04/06/2010 02:41 PM, Steve Schveighoffer wrote:
> ----- Original Message ----
>
>> From: Steve Schveighoffer<schveiguy at yahoo.com>
>
>> I'm waiting to hear back from Ellery that the new code fixes the problem, and I also
>> pinged David for his issue with AAs.  If all is OK, I recommend a patch
>> release.  I'm not sure the state of the AAs in trunk, since Walter has
>> changed them significantly -- we could be fixing one bug and introducing many
>> more.  Given how simple this fix is, it might be good to release just this
>> fix in order to have a stable version that doesn't do memory
>> stomping.
>
> Ellery confirmed, the patch fixes his problem.  I've posted messages on the newsgroup.
>
> Still waiting to hear from David.
>
> -Steve
>
>
>
>
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

Thanks much for taking care of this. I'd also like a test case we can add to the test suite, so we can ensure it never happens again.

Steve Schveighoffer wrote:
> Through debugging Ellery Newcomer's test case (see http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=108685), I have found another bug in the array append patch.
>
> 

OK, it will take me a while to make one, but I should be able to do it.

-Steve



----- Original Message ----
> From: Walter Bright <walter at digitalmars.com>
> To: Discuss the phobos library for D <phobos at puremagic.com>
> Sent: Tue, April 6, 2010 5:55:19 PM
> Subject: Re: [phobos] Looks like more problems with the array append patches
> 
> Thanks much for taking care of this. I'd also like a test case we can
add to
> the test suite, so we can ensure it never happens again.

Steve
> Schveighoffer wrote:
> Through debugging Ellery Newcomer's test case (see
> http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=108685),
> I have found another bug in the array append patch.
>
> 
> 
_______________________________________________
phobos mailing list

> ymailto="mailto:phobos at puremagic.com" href="mailto:phobos at puremagic.com">phobos at puremagic.com
http://lists.puremagic.com/mailman/listinfo/phobos

Thank you. I want it especially because this error slipped through all the other tests that attempted to flush out memory corruption errors in the allocator and gc.

Steve Schveighoffer wrote:
> OK, it will take me a while to make one, but I should be able to do it.
>
> -Steve
>
>
>
> ----- Original Message ----
> 
>> From: Walter Bright <walter at digitalmars.com>
>> To: Discuss the phobos library for D <phobos at puremagic.com>
>> Sent: Tue, April 6, 2010 5:55:19 PM
>> Subject: Re: [phobos] Looks like more problems with the array append patches
>>
>> Thanks much for taking care of this. I'd also like a test case we can
>> 
> add to
> 
>> the test suite, so we can ensure it never happens again.
>> 
>
> 

Working on it now, it's definitely a weird one.  You have to allocate 3 different memory segments, of certain sizes, and then one memory segment has to be no-pointers, where the other isn't.  Then you have to trick the array appending code into copying the no-pointers flag to a pointers-containing block, and then get the corruption to occur.  I hope I can deterministically make it happen.

Ellery's code, which uses appending all over the place, would not deterministically reproduce it (probably 1 out of 4 runs), and that was a huge piece of code.

-Steve



----- Original Message ----
> From: Walter Bright <walter at digitalmars.com>
> To: Discuss the phobos library for D <phobos at puremagic.com>
> Sent: Tue, April 6, 2010 7:30:53 PM
> Subject: Re: [phobos] Looks like more problems with the array append patches
> 
> Thank you. I want it especially because this error slipped through all the other tests that attempted to flush out memory corruption errors in the allocator and gc.

Steve Schveighoffer wrote:
> OK, it will take me a while to make one, but I should be able to do it.
> 
> -Steve
> 
> 
> 
> 
> ----- Original Message ----
> 
> 
>> From: Walter Bright <
> href="mailto:walter at digitalmars.com">walter at digitalmars.com>
>> 
> To: Discuss the phobos library for D <
> ymailto="mailto:phobos at puremagic.com"
> href="mailto:phobos at puremagic.com">phobos at puremagic.com>
>> 
> Sent: Tue, April 6, 2010 5:55:19 PM
>> Subject: Re: [phobos] Looks like
> more problems with the array append patches
>> 
>> Thanks much
> for taking care of this. I'd also like a test case we can
> 
> add to
>> the test suite, so we can ensure it never happens
> again.
>> 
> 
> 
> 
_______________________________________________
phobos mailing list

> ymailto="mailto:phobos at puremagic.com" href="mailto:phobos at puremagic.com">phobos at puremagic.com
http://lists.puremagic.com/mailman/listinfo/phobos