Greetings

I am struggling with strange memory corruption issues with dmd-2.069.2 release.

The issue shows up only when I load a shared library created from D code from C and call some D functions from the C side. But since the program control is completely with the D code, and data structures in D get corrupted, I believe C has no role to play in the corruption. It is just that the memory layout of the executable, when the compiled D code is loaded from C, is helping in replicating the issue.

I have spent almost a week in reducing this issue to less than 100 lines of code. Now I need the developers' love and help to get this issue fixed. There is some bleak chance that I am doing something wrong while loading the D library from C code. But otherwise it looks like a DMD memory corruption issue.

Since two C, and one D files are involved in recreating the issue, I have put all the files on a github repository along with a makefile. I have been able to recreate the issue on two Ubuntu 14.04 64-bit machines.

Generally the issue seems to be with static (thread local) variables that get allocated on heap. If I create many such variables, I get data corruption in some of these variables and sometimes I get segmentation fault. In this testcase that I have reduced, contents of a dynamic array are getting corrupted. Since all the data is being accessed from only one thread, there is no chance of a multicore race condition.

To reproduce the issue, kindly clone my git repo (https://github.com/puneet/memerr.git). Change the path of the DMD installation (I have tested only with dmd-2.069.2) in the makefile and run make.

$ git clone https://github.com/puneet/memerr.git
# change DMD install path in memerr/makefile
$ make

I get an output like:

$ make
/home/puneet/local/dmd-2.069.2/linux/bin64/rdmd foo.d
Start frop from D
Successfully completed loop....
./main
Start frop from C
0 -> @�+----------------

The last line is the content of an array which is actually filled with only dashes in the code.

Kindly help. I want to make sure that I am not making a mistake before I file a bug on dlang bugzilla.

Regards
- Puneet

Am Wed, 20 Jan 2016 09:12:57 +0000
schrieb Bottled Gin <gin@bottle.com>:

> Greetings
> 
> I am struggling with strange memory corruption issues with dmd-2.069.2 release.
> 

Could be GC scanning issues. Can you try disabling the garbage collector and see if it still crashes?

On Wednesday, 20 January 2016 at 09:12:57 UTC, Bottled Gin wrote:
> Greetings
>
> I am struggling with strange memory corruption issues with dmd-2.069.2 release.
>
> [...]

As a workaround you can mark Hash class as a shared

final shared class Hash

On Wednesday, 20 January 2016 at 11:06:53 UTC, Daniel Kozak wrote:
> On Wednesday, 20 January 2016 at 09:12:57 UTC, Bottled Gin wrote:
>> Greetings
>>
>> I am struggling with strange memory corruption issues with dmd-2.069.2 release.
>>
>> [...]
>
> As a workaround you can mark Hash class as a shared
>
> final shared class Hash

But this will change behavour because it will not be in TLS anymore

On Wednesday, 20 January 2016 at 11:08:46 UTC, Daniel Kozak wrote:
> On Wednesday, 20 January 2016 at 11:06:53 UTC, Daniel Kozak wrote:
>> On Wednesday, 20 January 2016 at 09:12:57 UTC, Bottled Gin wrote:
>>> Greetings
>>>
>>> I am struggling with strange memory corruption issues with dmd-2.069.2 release.
>>>
>>> [...]
>>
>> As a workaround you can mark Hash class as a shared
>>
>> final shared class Hash
>
> But this will change behavour because it will not be in TLS anymore

GC.disable seems help

On Wednesday, 20 January 2016 at 09:12:57 UTC, Bottled Gin wrote:
> Greetings
>
> I am struggling with strange memory corruption issues with dmd-2.069.2 release.
>
> ...
> ./main
> Start frop from C
> 0 -> @�+----------------
>
> The last line is the content of an array which is actually filled with only dashes in the code.
>
> Kindly help. I want to make sure that I am not making a mistake before I file a bug on dlang bugzilla.
>
> Regards
> - Puneet

Another workaround is to use GC.addRoot for dynamic allocated data in Dynamic.proc

void proc () {
    import core.memory: GC;
    dash.length = 32;
    GC.addRoot(cast(void*)dash.ptr);
    dash[] = '-';
}

On Wednesday, 20 January 2016 at 11:49:29 UTC, Daniel Kozak wrote:
> On Wednesday, 20 January 2016 at 09:12:57 UTC, Bottled Gin wrote:
>> Greetings
>>
>> I am struggling with strange memory corruption issues with dmd-2.069.2 release.
>>
>> ...
>> ./main
>> Start frop from C
>> 0 -> @�+----------------
>>
>> The last line is the content of an array which is actually filled with only dashes in the code.
>>
>> Kindly help. I want to make sure that I am not making a mistake before I file a bug on dlang bugzilla.
>>
>> Regards
>> - Puneet
>
> Another workaround is to use GC.addRoot for dynamic allocated data in Dynamic.proc
>
> void proc () {
>     import core.memory: GC;
>     dash.length = 32;
>     GC.addRoot(cast(void*)dash.ptr);
>     dash[] = '-';
> }

And another one is hold pointer to data:

class Dynamic {
  static char[] space;
  static char[] dash;
  char* dash_ptr;
  void rehash () {
    static Hash hash ;
    hash = new Hash;
    hash.clear();
  }
  void proc () {
    import core.memory: GC;
    dash.length = 32;
    dash_ptr = dash.ptr;
    dash[] = '-';
  }
}

>> Another workaround is to use GC.addRoot for dynamic allocated data in Dynamic.proc
>>
>> void proc () {
>>     import core.memory: GC;
>>     dash.length = 32;
>>     GC.addRoot(cast(void*)dash.ptr);
>>     dash[] = '-';
>> }
>
> And another one is hold pointer to data:
>
> class Dynamic {
>   static char[] space;
>   static char[] dash;
>   char* dash_ptr;
>   void rehash () {
>     static Hash hash ;
>     hash = new Hash;
>     hash.clear();
>   }
>   void proc () {
>     import core.memory: GC;
>     dash.length = 32;
>     dash_ptr = dash.ptr;
>     dash[] = '-';
>   }
> }

Daniel, thanks for confirming the bug and for providing workaround. The second workaround (saving the pointer) will not work on my real project though. I have multiple threads and the TLS variable will have a different pointer on each thread.

Also, can you please tell me how to addRoot an assoc array to GC. It seems there is no ptr property available for assoc arrays.

Regards
- Puneet

V Wed, 20 Jan 2016 13:58:46 +0000
Bottled Gin via Digitalmars-d <digitalmars-d@puremagic.com> napsáno:

> >> Another workaround is to use GC.addRoot for dynamic allocated data in Dynamic.proc
> >>
> >> void proc () {
> >>     import core.memory: GC;
> >>     dash.length = 32;
> >>     GC.addRoot(cast(void*)dash.ptr);
> >>     dash[] = '-';
> >> }
> >
> > And another one is hold pointer to data:
> >
> > class Dynamic {
> >   static char[] space;
> >   static char[] dash;
> >   char* dash_ptr;
> >   void rehash () {
> >     static Hash hash ;
> >     hash = new Hash;
> >     hash.clear();
> >   }
> >   void proc () {
> >     import core.memory: GC;
> >     dash.length = 32;
> >     dash_ptr = dash.ptr;
> >     dash[] = '-';
> >   }
> > }
> 
> Daniel, thanks for confirming the bug and for providing workaround. The second workaround (saving the pointer) will not work on my real project though. I have multiple threads and the TLS variable will have a different pointer on each thread.
> 
> Also, can you please tell me how to addRoot an assoc array to GC. It seems there is no ptr property available for assoc arrays.
> 
> Regards
> - Puneet
> 
> 
>

You can use cast(void *)aa;

something like this:

class Dynamic {
  static char[] space;
  static char[int] dash;
  void rehash () {
    static Hash hash ;
    hash = new Hash;
    hash.clear();
  }
  void proc () {
    import core.memory: GC;
    //GC.addRoot(cast(void *)dash); // not here
    {
        dash[i] = '-';
    }
    GC.addRoot(cast(void *)dash); // must be after allocation
  }
}


Be careful because you mast use addPtr after aa is initialized (you put
something to it)

Thanks Daniel

I have added the testcase to a more obscure testcase that I had raised on Bugzilla earlier.
https://issues.dlang.org/show_bug.cgi?id=15513

I want to request developers to show some love.

Regards
- Puneet