View mode: basic / threaded / horizontal-split · Log in · Help
April 03, 2010
Re: Memory Corruption with AAs
Walter Bright wrote:
> Jordi wrote:
>> I am having exactly the same situation.
> 
> Any chance you can reduce it to a small test case?

Actually i am trying to add an "autotest" mode to my project to be able 
to test and benchmark different compiler versions, compilation options 
and even GC implementations in the context of a realistic application 
(in size, kind of things it does and "average" quality of code).

However i don't expect to be able to narrow this down in the short term, 
due to the relatively big amount of data involved and the randomness of 
its occurrence. Sometimes it is a crash somewhere gdb cannot provide 
info on, sometimes i get my own assert in places i am pretty sure that 
are not possible unless i get some corruption or wrong result from an AA.

And finally, i am not entirely sure yet that it is really the compiler's 
fault... if i manage to have a "send-able" case i will submit it in the 
bugtracker.

j.
April 03, 2010
Re: Memory Corruption with AAs
Jordi wrote:
> Walter Bright wrote:
>> Jordi wrote:
>>> I am having exactly the same situation.
>>
>> Any chance you can reduce it to a small test case?
> 
> Actually i am trying to add an "autotest" mode to my project to be able 
> to test and benchmark different compiler versions, compilation options 
> and even GC implementations in the context of a realistic application 
> (in size, kind of things it does and "average" quality of code).
> 
> However i don't expect to be able to narrow this down in the short term, 
> due to the relatively big amount of data involved and the randomness of 
> its occurrence. Sometimes it is a crash somewhere gdb cannot provide 
> info on, sometimes i get my own assert in places i am pretty sure that 
> are not possible unless i get some corruption or wrong result from an AA.
> 
> And finally, i am not entirely sure yet that it is really the compiler's 
> fault... if i manage to have a "send-able" case i will submit it in the 
> bugtracker.

Since it is single-threaded, it should crash in the same place in the same way 
every time. This means you can put an assert on the crashing data (even without 
gdb it can be found by inserting printf's), and slowly work it backward to where 
the data goes wrong.
April 03, 2010
Re: Memory Corruption with AAs
On Fri, 02 Apr 2010 23:01:48 -0700, Walter Bright wrote:

>> > Since it is single-threaded, it should crash in the same place in the
> same way every time. This means you can put an assert on the crashing
> data (even without gdb it can be found by inserting printf's), and
> slowly work it backward to where the data goes wrong.

According to an article I was reading that constancy does not apply with 
current Linux kernels. They deliberately randomize things to make life 
difficult for hackers.

I'll try to find it again - it's about debugging GCC itself.

Steve
April 03, 2010
Re: Memory Corruption with AAs
On Sat, 03 Apr 2010 06:24:20 +0000, Steve Teale wrote:

> On Fri, 02 Apr 2010 23:01:48 -0700, Walter Bright wrote:
>  
>>> > Since it is single-threaded, it should crash in the same place in
>>> > the
>> same way every time. This means you can put an assert on the crashing
>> data (even without gdb it can be found by inserting printf's), and
>> slowly work it backward to where the data goes wrong.
> 
> According to an article I was reading that constancy does not apply with
> current Linux kernels. They deliberately randomize things to make life
> difficult for hackers.
> 
> I'll try to find it again - it's about debugging GCC itself.
> 
> Steve

Got it - http://gcc.gnu.org/wiki/DebuggingGCC

Right at the end.
April 03, 2010
Re: Memory Corruption with AAs
Steve Teale wrote:
> On Sat, 03 Apr 2010 06:24:20 +0000, Steve Teale wrote:
> 
>> On Fri, 02 Apr 2010 23:01:48 -0700, Walter Bright wrote:
>>  
>>>>> Since it is single-threaded, it should crash in the same place in
>>>>> the
>>> same way every time. This means you can put an assert on the crashing
>>> data (even without gdb it can be found by inserting printf's), and
>>> slowly work it backward to where the data goes wrong.
>> According to an article I was reading that constancy does not apply with
>> current Linux kernels. They deliberately randomize things to make life
>> difficult for hackers.
>>
>> I'll try to find it again - it's about debugging GCC itself.
>>
>> Steve
> 
> Got it - http://gcc.gnu.org/wiki/DebuggingGCC
> 
> Right at the end.

Well, i don't know if this applies to my case, but it is definitely random:

- Compile
- Run
- it crashes after some interaction.
- Run the same again
- crashes immediately.

 I am currently trying to run it in windows to see what comes out. 
Trying to narrow it down makes it disappear in all of my attempts so far.

j.
April 03, 2010
Re: Memory Corruption with AAs
Steve Teale wrote:
> On Sat, 03 Apr 2010 06:24:20 +0000, Steve Teale wrote:
> 
>> On Fri, 02 Apr 2010 23:01:48 -0700, Walter Bright wrote:
>>  
>>>>> Since it is single-threaded, it should crash in the same place in
>>>>> the
>>> same way every time. This means you can put an assert on the crashing
>>> data (even without gdb it can be found by inserting printf's), and
>>> slowly work it backward to where the data goes wrong.
>> According to an article I was reading that constancy does not apply with
>> current Linux kernels. They deliberately randomize things to make life
>> difficult for hackers.
>>
>> I'll try to find it again - it's about debugging GCC itself.
>>
>> Steve
> 
> Got it - http://gcc.gnu.org/wiki/DebuggingGCC
> 
> Right at the end.

It links to this page which shows how to turn it off:

http://gcc.gnu.org/wiki/Randomization
April 04, 2010
Re: Memory Corruption with AAs
On 04/02/2010 03:53 PM, Walter Bright wrote:
> dsimcha wrote:
>> I almost never do this with any data structure other
>> than an array because, even if it works for now, I consider it a horrible
>> violation of encapsulation because you're relying on the details of
>> how the data
>> structure manipulates memory. This is also why, when I designed RandAA
>> I didn't
>> see this as an issue until you pointed it out to me.
>
> Andrei is working on the design of the D collection class library. After
> much thought and research, he finally came to the conclusion that a
> collection class should not allow the address of a member to be taken. I
> think his reasoning on the issue is pretty sound, and is consistent with
> your take on it.

I wouldn't call it research, but I agonized a fair amount over it. I 
think Phobos containers will all use malloc, realloc, and free for their 
own storage, while still being safe.

Andrei
April 04, 2010
Re: Memory Corruption with AAs
On 2010-04-03 23:21:48 -0400, Andrei Alexandrescu 
<SeeWebsiteForEmail@erdani.org> said:

> On 04/02/2010 03:53 PM, Walter Bright wrote:
>> dsimcha wrote:
>>> I almost never do this with any data structure other
>>> than an array because, even if it works for now, I consider it a horrible
>>> violation of encapsulation because you're relying on the details of
>>> how the data
>>> structure manipulates memory. This is also why, when I designed RandAA
>>> I didn't
>>> see this as an issue until you pointed it out to me.
>> 
>> Andrei is working on the design of the D collection class library. After
>> much thought and research, he finally came to the conclusion that a
>> collection class should not allow the address of a member to be taken. I
>> think his reasoning on the issue is pretty sound, and is consistent with
>> your take on it.
> 
> I wouldn't call it research, but I agonized a fair amount over it. I 
> think Phobos containers will all use malloc, realloc, and free for 
> their own storage, while still being safe.

I think this is a sound decision. And I'm not necessarily talking about 
using malloc, realloc, and free (even though a container capable of 
using realloc is certainly a plus), but the one about decoupling the 
container interface from any particular memory management 
implementation.

Question: if the container's memory isn't garbage-collected, how do you 
implement iterators, eh, ranges so that they are still memory-safe?

-- 
Michel Fortin
michel.fortin@michelf.com
http://michelf.com/
April 04, 2010
Re: Memory Corruption with AAs
== Quote from Michel Fortin (michel.fortin@michelf.com)'s article
> Question: if the container's memory isn't garbage-collected, how do you
> implement iterators, eh, ranges so that they are still memory-safe?

The way I'm picturing this being implemented is that a GC'd class instance exists
at the top level, and then the internal implementation-detail storage that the
class uses is implemented via malloc and free.  This storage would get freed in
the class finalizer when the instance is GC'd.  In this case all you'd need to do
is make the range hold a reference to the class instance so it wouldn't be GC'd.
April 04, 2010
Re: Memory Corruption with AAs
On 2010-04-04 09:45:36 -0400, dsimcha <dsimcha@yahoo.com> said:

> == Quote from Michel Fortin (michel.fortin@michelf.com)'s article
>> Question: if the container's memory isn't garbage-collected, how do you
>> implement iterators, eh, ranges so that they are still memory-safe?
> 
> The way I'm picturing this being implemented is that a GC'd class 
> instance exists
> at the top level, and then the internal implementation-detail storage that the
> class uses is implemented via malloc and free.  This storage would get freed in
> the class finalizer when the instance is GC'd.  In this case all you'd 
> need to do
> is make the range hold a reference to the class instance so it wouldn't 
> be GC'd.

That wouldn't work with realloc: realloc copies to a new location then 
frees the old memory if it cannot expand in place. You can't keep the 
old copy allocated.

I've been thinking of another method to ensure safety: don't allow 
expanding a container as long as there are ranges pointing to it. 
Easily implemented with a reference count. For instance, if you have a 
vector container, expanding the container would invalidate ranges. 
Instead of allowing ranges to become invalid and potentially dangerous, 
just disallow expanding the container. The range would contain a 
pointer to its upper and lower bound, and a pointer to the container to 
increment the reference count when it's copied and decrement it when 
it's destroyed.

-- 
Michel Fortin
michel.fortin@michelf.com
http://michelf.com/
1 2 3
Top | Discussion index | About this forum | D home