Jump to page: 1 24  
Page
Thread overview
Garbage collector collects live objects
Dec 09, 2014
ketmar
Dec 10, 2014
ketmar
Dec 10, 2014
ketmar
Mar 25, 2015
Ali Çehreli
Dec 09, 2014
Dicebot
Dec 09, 2014
Ali Çehreli
Dec 10, 2014
ketmar
Dec 10, 2014
ketmar
December 09, 2014
Hi,

I experience very strange problem: GC somehow collects live objects.

I found it because i got segfaults. After debugging and tracing i found this is because of accessing not allocated memory.

I did the following checks:

- added to some class invariant check for access to suspicious members with assertion

assert(GC.addrOf(cast(void*)x) !is null);


where it fails DETERMINISTICALLY at some point

- printing address of allocated classes where i observe the following pattern

-> ctor
     check
     check
     check
<- dtor
     check (fails)

could anybody advice me with something? I got really frustrated by this strange behaviour which i can not fix right now.

key observations:
- it is deterministically behaviour (what gets me even more confused cause GC collections as far as i know runs from time to time)
- i do not play with pointers optimisation like hiding its in ints or floats.
- i operate with large uniformly distributed (video) data in memory where pointer like patterns may occur. but this is not the case cause (1) it brings at worst long living objects (2) input sequence constant but allocated pointers each run different.

December 09, 2014
On 12/9/14 8:54 AM, Ruslan Mullakhmetov wrote:
>
> Hi,
>
> I experience very strange problem: GC somehow collects live objects.
>
> I found it because i got segfaults. After debugging and tracing i found
> this is because of accessing not allocated memory.
>
> I did the following checks:
>
> - added to some class invariant check for access to suspicious members
> with assertion
>
> assert(GC.addrOf(cast(void*)x) !is null);
>
>
> where it fails DETERMINISTICALLY at some point
>
> - printing address of allocated classes where i observe the following
> pattern
>
> -> ctor
>       check
>       check
>       check
> <- dtor
>       check (fails)
>
> could anybody advice me with something? I got really frustrated by this
> strange behaviour which i can not fix right now.
>
> key observations:
> - it is deterministically behaviour (what gets me even more confused
> cause GC collections as far as i know runs from time to time)
> - i do not play with pointers optimisation like hiding its in ints or
> floats.
> - i operate with large uniformly distributed (video) data in memory
> where pointer like patterns may occur. but this is not the case cause
> (1) it brings at worst long living objects (2) input sequence constant
> but allocated pointers each run different.
>

A random guess, since you haven't posted any code, are you accessing GC resources inside a destructor? If so, that is not guaranteed to work. A class destructor, or a destructor of a struct that is contained inside a class, can only be used to destroy NON-GC resources.

If you want more help, you need to post some code. Something that minimally causes the issue would be good.

-Steve
December 09, 2014
On Tuesday, 9 December 2014 at 14:23:06 UTC, Steven Schveighoffer wrote:
> On 12/9/14 8:54 AM, Ruslan Mullakhmetov wrote:
>>
>> Hi,
>>
>> I experience very strange problem: GC somehow collects live objects.
>>
>> I found it because i got segfaults. After debugging and tracing i found
>> this is because of accessing not allocated memory.
>>
>> I did the following checks:
>>
>> - added to some class invariant check for access to suspicious members
>> with assertion
>>
>> assert(GC.addrOf(cast(void*)x) !is null);
>>
>>
>> where it fails DETERMINISTICALLY at some point
>>
>> - printing address of allocated classes where i observe the following
>> pattern
>>
>> -> ctor
>>      check
>>      check
>>      check
>> <- dtor
>>      check (fails)
>>
>> could anybody advice me with something? I got really frustrated by this
>> strange behaviour which i can not fix right now.
>>
>> key observations:
>> - it is deterministically behaviour (what gets me even more confused
>> cause GC collections as far as i know runs from time to time)
>> - i do not play with pointers optimisation like hiding its in ints or
>> floats.
>> - i operate with large uniformly distributed (video) data in memory
>> where pointer like patterns may occur. but this is not the case cause
>> (1) it brings at worst long living objects (2) input sequence constant
>> but allocated pointers each run different.
>>
>
> A random guess, since you haven't posted any code, are you accessing GC resources inside a destructor? If so, that is not guaranteed to work. A class destructor, or a destructor of a struct that is contained inside a class, can only be used to destroy NON-GC resources.
>
> If you want more help, you need to post some code. Something that minimally causes the issue would be good.
>
> -Steve

No, there is no accessing GC resources in dtors.

the only usage of dtor in one class is

	~this()
	{
		_file.close();
	}

where _file is of type std.file.File

i'll try to extract problem to any observable source code but all my previous attempts lead to problem being diminish.

December 09, 2014
On 12/9/14 9:52 AM, Ruslan Mullakhmetov wrote:
>
> No, there is no accessing GC resources in dtors.
>
> the only usage of dtor in one class is
>
>      ~this()
>      {
>          _file.close();
>      }
>
> where _file is of type std.file.File

That should work I think.

> i'll try to extract problem to any observable source code but all my
> previous attempts lead to problem being diminish.

Have you tried dustmite? https://github.com/CyberShadow/DustMite

-Steve
December 09, 2014
It may happen if only reference to an object is stored in memory block marked as data-only (using ubyte[] for a buffer is probably most common reason I have encountered)
December 09, 2014
On Tue, 09 Dec 2014 14:52:53 +0000
Ruslan Mullakhmetov via Digitalmars-d-learn
<digitalmars-d-learn@puremagic.com> wrote:

> On Tuesday, 9 December 2014 at 14:23:06 UTC, Steven Schveighoffer wrote:
> > On 12/9/14 8:54 AM, Ruslan Mullakhmetov wrote:
> >>
> >> Hi,
> >>
> >> I experience very strange problem: GC somehow collects live objects.
> >>
> >> I found it because i got segfaults. After debugging and
> >> tracing i found
> >> this is because of accessing not allocated memory.
> >>
> >> I did the following checks:
> >>
> >> - added to some class invariant check for access to suspicious
> >> members
> >> with assertion
> >>
> >> assert(GC.addrOf(cast(void*)x) !is null);
> >>
> >>
> >> where it fails DETERMINISTICALLY at some point
> >>
> >> - printing address of allocated classes where i observe the
> >> following
> >> pattern
> >>
> >> -> ctor
> >>      check
> >>      check
> >>      check
> >> <- dtor
> >>      check (fails)
> >>
> >> could anybody advice me with something? I got really
> >> frustrated by this
> >> strange behaviour which i can not fix right now.
> >>
> >> key observations:
> >> - it is deterministically behaviour (what gets me even more
> >> confused
> >> cause GC collections as far as i know runs from time to time)
> >> - i do not play with pointers optimisation like hiding its in
> >> ints or
> >> floats.
> >> - i operate with large uniformly distributed (video) data in
> >> memory
> >> where pointer like patterns may occur. but this is not the
> >> case cause
> >> (1) it brings at worst long living objects (2) input sequence
> >> constant
> >> but allocated pointers each run different.
> >>
> >
> > A random guess, since you haven't posted any code, are you accessing GC resources inside a destructor? If so, that is not guaranteed to work. A class destructor, or a destructor of a struct that is contained inside a class, can only be used to destroy NON-GC resources.
> >
> > If you want more help, you need to post some code. Something that minimally causes the issue would be good.
> >
> > -Steve
> 
> No, there is no accessing GC resources in dtors.
> 
> the only usage of dtor in one class is
> 
> 	~this()
> 	{
> 		_file.close();
> 	}
> 
> where _file is of type std.file.File
> 
> i'll try to extract problem to any observable source code but all my previous attempts lead to problem being diminish.

that file can be already finalized. please remember that `~this()` is more a finalizer than destructor, and it's called on *dead* object. here this means that any other object in your object (including structs) can be already finalized at the time GC decides to call your finalizer.

just avoid "destructors" unless you *really* need that. in your case simply let GC finalize your File, don't try to help GC. this is not C++ (or any other language without GC) and "destructors" aren't destructing anything at all. "destructors" must clean up the things that GC cannot (malloc()'ed memory, for example), and nothing else.


December 09, 2014
On 12/9/14 11:17 AM, ketmar via Digitalmars-d-learn wrote:
> On Tue, 09 Dec 2014 14:52:53 +0000
> Ruslan Mullakhmetov via Digitalmars-d-learn
> <digitalmars-d-learn@puremagic.com> wrote:
>
>> On Tuesday, 9 December 2014 at 14:23:06 UTC, Steven Schveighoffer
>> wrote:
>>> On 12/9/14 8:54 AM, Ruslan Mullakhmetov wrote:
>>>>
>>>> Hi,
>>>>
>>>> I experience very strange problem: GC somehow collects live
>>>> objects.
>>>>
>>>> I found it because i got segfaults. After debugging and
>>>> tracing i found
>>>> this is because of accessing not allocated memory.
>>>>
>>>> I did the following checks:
>>>>
>>>> - added to some class invariant check for access to suspicious
>>>> members
>>>> with assertion
>>>>
>>>> assert(GC.addrOf(cast(void*)x) !is null);
>>>>
>>>>
>>>> where it fails DETERMINISTICALLY at some point
>>>>
>>>> - printing address of allocated classes where i observe the
>>>> following
>>>> pattern
>>>>
>>>> -> ctor
>>>>       check
>>>>       check
>>>>       check
>>>> <- dtor
>>>>       check (fails)
>>>>
>>>> could anybody advice me with something? I got really
>>>> frustrated by this
>>>> strange behaviour which i can not fix right now.
>>>>
>>>> key observations:
>>>> - it is deterministically behaviour (what gets me even more
>>>> confused
>>>> cause GC collections as far as i know runs from time to time)
>>>> - i do not play with pointers optimisation like hiding its in
>>>> ints or
>>>> floats.
>>>> - i operate with large uniformly distributed (video) data in
>>>> memory
>>>> where pointer like patterns may occur. but this is not the
>>>> case cause
>>>> (1) it brings at worst long living objects (2) input sequence
>>>> constant
>>>> but allocated pointers each run different.
>>>>
>>>
>>> A random guess, since you haven't posted any code, are you
>>> accessing GC resources inside a destructor? If so, that is not
>>> guaranteed to work. A class destructor, or a destructor of a
>>> struct that is contained inside a class, can only be used to
>>> destroy NON-GC resources.
>>>
>>> If you want more help, you need to post some code. Something
>>> that minimally causes the issue would be good.
>>>
>>> -Steve
>>
>> No, there is no accessing GC resources in dtors.
>>
>> the only usage of dtor in one class is
>>
>> 	~this()
>> 	{
>> 		_file.close();
>> 	}
>>
>> where _file is of type std.file.File
>>
>> i'll try to extract problem to any observable source code but all
>> my previous attempts lead to problem being diminish.
>
> that file can be already finalized. please remember that `~this()` is
> more a finalizer than destructor, and it's called on *dead* object.
> here this means that any other object in your object (including
> structs) can be already finalized at the time GC decides to call your
> finalizer.

File is specially designed (although it's not perfect) to be able to close in the GC. Its ref-counted payload is placed on the C heap to allow access during finalization.

That being said, you actually don't need to write the above in the class finalizer, _file's destructor will automatically be called.

> just avoid "destructors" unless you *really* need that. in your case
> simply let GC finalize your File, don't try to help GC. this is not C++
> (or any other language without GC) and "destructors" aren't destructing
> anything at all. "destructors" must clean up the things that GC cannot
> (malloc()'ed memory, for example), and nothing else.
>

Good advice ;)

I would say other than library writers, nobody should ever write a class dtor.

-Steve
December 09, 2014
On Tuesday, 9 December 2014 at 16:53:02 UTC, Steven Schveighoffer
wrote:
> On 12/9/14 11:17 AM, ketmar via Digitalmars-d-learn wrote:
>> On Tue, 09 Dec 2014 14:52:53 +0000
>> Ruslan Mullakhmetov via Digitalmars-d-learn
>> <digitalmars-d-learn@puremagic.com> wrote:
>>
>>> On Tuesday, 9 December 2014 at 14:23:06 UTC, Steven Schveighoffer
>>> wrote:
>>>> On 12/9/14 8:54 AM, Ruslan Mullakhmetov wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I experience very strange problem: GC somehow collects live
>>>>> objects.
>>>>>
>>>>> I found it because i got segfaults. After debugging and
>>>>> tracing i found
>>>>> this is because of accessing not allocated memory.
>>>>>
>>>>> I did the following checks:
>>>>>
>>>>> - added to some class invariant check for access to suspicious
>>>>> members
>>>>> with assertion
>>>>>
>>>>> assert(GC.addrOf(cast(void*)x) !is null);
>>>>>
>>>>>
>>>>> where it fails DETERMINISTICALLY at some point
>>>>>
>>>>> - printing address of allocated classes where i observe the
>>>>> following
>>>>> pattern
>>>>>
>>>>> -> ctor
>>>>>      check
>>>>>      check
>>>>>      check
>>>>> <- dtor
>>>>>      check (fails)
>>>>>
>>>>> could anybody advice me with something? I got really
>>>>> frustrated by this
>>>>> strange behaviour which i can not fix right now.
>>>>>
>>>>> key observations:
>>>>> - it is deterministically behaviour (what gets me even more
>>>>> confused
>>>>> cause GC collections as far as i know runs from time to time)
>>>>> - i do not play with pointers optimisation like hiding its in
>>>>> ints or
>>>>> floats.
>>>>> - i operate with large uniformly distributed (video) data in
>>>>> memory
>>>>> where pointer like patterns may occur. but this is not the
>>>>> case cause
>>>>> (1) it brings at worst long living objects (2) input sequence
>>>>> constant
>>>>> but allocated pointers each run different.
>>>>>
>>>>
>>>> A random guess, since you haven't posted any code, are you
>>>> accessing GC resources inside a destructor? If so, that is not
>>>> guaranteed to work. A class destructor, or a destructor of a
>>>> struct that is contained inside a class, can only be used to
>>>> destroy NON-GC resources.
>>>>
>>>> If you want more help, you need to post some code. Something
>>>> that minimally causes the issue would be good.
>>>>
>>>> -Steve
>>>
>>> No, there is no accessing GC resources in dtors.
>>>
>>> the only usage of dtor in one class is
>>>
>>> 	~this()
>>> 	{
>>> 		_file.close();
>>> 	}
>>>
>>> where _file is of type std.file.File
>>>
>>> i'll try to extract problem to any observable source code but all
>>> my previous attempts lead to problem being diminish.
>>
>> that file can be already finalized. please remember that `~this()` is
>> more a finalizer than destructor, and it's called on *dead* object.
>> here this means that any other object in your object (including
>> structs) can be already finalized at the time GC decides to call your
>> finalizer.
>
> File is specially designed (although it's not perfect) to be able to close in the GC. Its ref-counted payload is placed on the C heap to allow access during finalization.
>
> That being said, you actually don't need to write the above in the class finalizer, _file's destructor will automatically be called.
>
>> just avoid "destructors" unless you *really* need that. in your case
>> simply let GC finalize your File, don't try to help GC. this is not C++
>> (or any other language without GC) and "destructors" aren't destructing
>> anything at all. "destructors" must clean up the things that GC cannot
>> (malloc()'ed memory, for example), and nothing else.
>>
>
> Good advice ;)
>
> I would say other than library writers, nobody should ever write a class dtor.
>
> -Steve


thanks, I got it: either C++ or D dtors are minefield =)

but i still have no clue how to overcome GC =(


December 09, 2014
On Tuesday, 9 December 2014 at 16:13:25 UTC, Dicebot wrote:
> It may happen if only reference to an object is stored in memory block marked as data-only (using ubyte[] for a buffer is probably most common reason I have encountered)

Thanks for interesting hypothesis, but that's not the issue.

innocent though collected objects are living in D array MyClass[] which are living in assoc array as value.

i checked attributes for GC block holding this array:

```
FINALIZE NO_SCAN NO_MOVE APPENDABLE NO_INTERIOR
```

I really doubting about NO_INTERIOR. can anybody confirm me that is's working with array slicing which i heavily use?


also i found that block size is quite small

<pre>
                array: [100A2FD00, 100A2F700, 100A33B80, 100A33500, 100A3FE80, 100A3F980, 100A3F400, 100A72600, 100A7DF80, 100A7DA80, 100A7D500]
		array ptr: 100A72580 root: 100A72580:128 attr: FINALIZE NO_SCAN NO_MOVE APPENDABLE NO_INTERIOR
		[100985A00] keys: [1] as: 1 au: 100A2FD00
		[100985A00] keys: [1] as: 1 au: 100A2F700
		[100985A00] keys: [1] as: 1 au: 100A33B80
</pre>

array holds 11 64bit pointers but it's block size is only 128 bytes < 11 * 64 = 704 bytes. what's wrong with this arithmetics?

December 09, 2014
On 12/9/14 12:40 PM, Ruslan Mullakhmetov wrote:
> On Tuesday, 9 December 2014 at 16:13:25 UTC, Dicebot wrote:
>> It may happen if only reference to an object is stored in memory block
>> marked as data-only (using ubyte[] for a buffer is probably most
>> common reason I have encountered)
>
> Thanks for interesting hypothesis, but that's not the issue.
>
> innocent though collected objects are living in D array MyClass[] which
> are living in assoc array as value.
>
> i checked attributes for GC block holding this array:
>
> ```
> FINALIZE NO_SCAN NO_MOVE APPENDABLE NO_INTERIOR
> ```
>

That does not sound right at all. No block should ever have both FINALIZE (reserved for objects only) and APPENDABLE (reserved for arrays only).

> I really doubting about NO_INTERIOR. can anybody confirm me that is's
> working with array slicing which i heavily use?
>
>
> also i found that block size is quite small
>
> <pre>
>                  array: [100A2FD00, 100A2F700, 100A33B80, 100A33500,
> 100A3FE80, 100A3F980, 100A3F400, 100A72600, 100A7DF80, 100A7DA80,
> 100A7D500]
>          array ptr: 100A72580 root: 100A72580:128 attr: FINALIZE NO_SCAN
> NO_MOVE APPENDABLE NO_INTERIOR
>          [100985A00] keys: [1] as: 1 au: 100A2FD00
>          [100985A00] keys: [1] as: 1 au: 100A2F700
>          [100985A00] keys: [1] as: 1 au: 100A33B80
> </pre>
>
> array holds 11 64bit pointers but it's block size is only 128 bytes < 11
> * 64 = 704 bytes. what's wrong with this arithmetics?
>

I think there is something you are missing, or something is very corrupt. Can you show the code that prints this?

-Steve
« First   ‹ Prev
1 2 3 4