Thread overview
[Issue 8583] New: AA ushort[dchar] byValue range is corrupted on x86_64
Aug 24, 2012
Dmitry Olshansky
[Issue 8583] [64 bit] AA ushort[dchar] byValue range is corrupted on x86_64
Nov 14, 2012
Don
Jan 16, 2013
Simon Harris
August 24, 2012
http://d.puremagic.com/issues/show_bug.cgi?id=8583

           Summary: AA ushort[dchar] byValue range is corrupted on x86_64
           Product: D
           Version: D2
          Platform: x86_64
        OS/Version: All
            Status: NEW
          Severity: critical
          Priority: P2
         Component: DMD
        AssignedTo: nobody@puremagic.com
        ReportedBy: dmitry.olsh@gmail.com


--- Comment #0 from Dmitry Olshansky <dmitry.olsh@gmail.com> 2012-08-24 14:00:21 PDT ---
The following sample outlines the problem:

void main(){
    ushort[dchar] simpleIndices = ['A':437];
    assert(simpleIndices['A'] == 437);
    assert(simpleIndices.byKey.front == 'A');
    assert(simpleIndices.byValue.front == 437); // this fails on x64
    //assert(simpleIndices.byValue.front == 0); //and this passes on x64 WTF??!
}

compiled with -m32 it passes
with -m64 it fails and instead the value is 0.
Looking through my corrupted data in the wild I assume it may as well be
something else then 0 too.

Tested both with latest git and vanila 2.060 on OS Linux x64 & Win32.

There is a workaround of using .values which does allocate an array of values. Still I consider it critical as corrupted value bug is usually hard to spot and narrow down.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
August 25, 2012
http://d.puremagic.com/issues/show_bug.cgi?id=8583


edmccard@verizon.net changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |edmccard@verizon.net


--- Comment #1 from edmccard@verizon.net 2012-08-24 19:40:31 PDT ---
From my tests, the problem occurs whenever keytype.sizeof <= 8, for any value type. For example, ushort[string], ushort[uint[3]] and ushort[Foo] are ok, where

  struct Foo { int x; int y; int z; }

but ushort[uint], ushort[uint[2]] and ushort[Bar] fail, where

  struct Bar { int x; int y; }

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
August 25, 2012
http://d.puremagic.com/issues/show_bug.cgi?id=8583



--- Comment #2 from edmccard@verizon.net 2012-08-24 20:05:06 PDT ---
(In reply to comment #1)
> From my tests, the problem occurs whenever keytype.sizeof <= 8, for any value type.

Spoke too soon; the problem is not that simple, or does not depend solely on the size of the key type. For example, string[uint[3]] works but ushort[uint[3]] fails, ushort[uint[4]] works, but both string[uint[5]] and ushort[uint[5]] fail. (replace uint[x] with a struct of the same size, and the same things happen).

Sorry for the noise.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
November 14, 2012
http://d.puremagic.com/issues/show_bug.cgi?id=8583


Don <clugdbug@yahoo.com.au> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |exetoc@gmail.com


--- Comment #3 from Don <clugdbug@yahoo.com.au> 2012-11-14 02:46:21 PST ---
*** Issue 7632 has been marked as a duplicate of this issue. ***

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
January 16, 2013
http://d.puremagic.com/issues/show_bug.cgi?id=8583


Simon Harris <pearfalse@googlemail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |pearfalse@googlemail.com


--- Comment #4 from Simon Harris <pearfalse@googlemail.com> 2013-01-16 07:37:10 PST ---
Something interesting I found in rt/aaA.d, line 106: aligntsize() on 64-bit compilation returns a value aligned to 16 bytes. _aaGetX uses this function on key sizes when creating new elements. So for AAs whose key is <= 8 bytes, the value is still 16 bytes further on in memory.

You can check this by repeating the same alignment trick on the addresses of byValue.front:

void testAAType(K, V)(V[K] aa)
{
    writefln("testAAType with key %s, value %s", typeid(K), typeid(V));

    for (auto v = aa.byValue; !v.empty; v.popFront()) { // emulated foreach (v
; aa.byValue)
        V* fp = &v.front();
        writef("Unaligned: <%s>", *fp);

        V* fpaligned = cast(V*) (( (cast(ptrdiff_t) fp) + 15) & ~15); // copy
what aligntsize does on x64

        version(D_LP64) {
            writefln("; Aligned: <%s>", *fp.alignto16());
        }
        else {
            writefln(" (Not testing aligned value on a 32-bit executable)");
        }
    }
}

Looking at the key sizes seems to match the pass/fail results in comment #2, with one exception: ushort[uint[3]]. Not sure why the alignment trick is needed here, given that uint[3].sizeof > 8. Maybe DMD aligns AssociativeArray!(uint[3], ushort).Slot to 4 bytes to save space, and since _aaGetX uses a completely different type set, alignment info gets lost somewhere.

Tested with DMD 2.060 and 2.061 on OS X 10.8.2.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
January 21, 2013
http://d.puremagic.com/issues/show_bug.cgi?id=8583



--- Comment #5 from github-bugzilla@puremagic.com 2013-01-21 10:28:09 PST ---
Commits pushed to master at https://github.com/D-Programming-Language/druntime

https://github.com/D-Programming-Language/druntime/commit/e9445dcbbfd7dec9275e78fba6633b689ca06c09 Unify AA value alignment to size_t.sizeof (fixes bug 8583)

Sets the alignment of values in both AA declarations (object and aaA) to size_t.sizeof bytes, which fixes byValue range iteration on x64.

https://github.com/D-Programming-Language/druntime/commit/9eb3f28e299cb0ce0b2fb22798f4086f008a8a87 Add unit tests for bug 8583; restore 16-byte aligment on x64

This adds unit tests for the alignment fix to Slot, ensuring the two declarations of an AA slot stay in sync. The 16-byte alignment rule for x64 has been restored.

https://github.com/D-Programming-Language/druntime/commit/105e12f5b933797a4cae84295a3ebe445e8ae909 Merge pull request #386 from pearfalse/master

Explicitly set AA value alignment in slot to size_t.sizeof bytes (fixes bug
8583)

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
January 21, 2013
http://d.puremagic.com/issues/show_bug.cgi?id=8583


Alex Rønne Petersen <alex@lycus.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |alex@lycus.org
         Resolution|                            |FIXED


-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
February 12, 2013
http://d.puremagic.com/issues/show_bug.cgi?id=8583



--- Comment #6 from github-bugzilla@puremagic.com 2013-02-11 23:12:16 PST ---
Commits pushed to https://github.com/D-Programming-Language/druntime

https://github.com/D-Programming-Language/druntime/commit/e9445dcbbfd7dec9275e78fba6633b689ca06c09 Unify AA value alignment to size_t.sizeof (fixes bug 8583)

https://github.com/D-Programming-Language/druntime/commit/9eb3f28e299cb0ce0b2fb22798f4086f008a8a87 Add unit tests for bug 8583; restore 16-byte aligment on x64

https://github.com/D-Programming-Language/druntime/commit/105e12f5b933797a4cae84295a3ebe445e8ae909 Merge pull request #386 from pearfalse/master

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------