April 03, 2010
Walter Bright wrote:
> Jordi wrote:
>> I am having exactly the same situation.
> 
> Any chance you can reduce it to a small test case?

Actually i am trying to add an "autotest" mode to my project to be able to test and benchmark different compiler versions, compilation options and even GC implementations in the context of a realistic application (in size, kind of things it does and "average" quality of code).

However i don't expect to be able to narrow this down in the short term, due to the relatively big amount of data involved and the randomness of its occurrence. Sometimes it is a crash somewhere gdb cannot provide info on, sometimes i get my own assert in places i am pretty sure that are not possible unless i get some corruption or wrong result from an AA.

And finally, i am not entirely sure yet that it is really the compiler's fault... if i manage to have a "send-able" case i will submit it in the bugtracker.

j.
April 03, 2010
Jordi wrote:
> Walter Bright wrote:
>> Jordi wrote:
>>> I am having exactly the same situation.
>>
>> Any chance you can reduce it to a small test case?
> 
> Actually i am trying to add an "autotest" mode to my project to be able to test and benchmark different compiler versions, compilation options and even GC implementations in the context of a realistic application (in size, kind of things it does and "average" quality of code).
> 
> However i don't expect to be able to narrow this down in the short term, due to the relatively big amount of data involved and the randomness of its occurrence. Sometimes it is a crash somewhere gdb cannot provide info on, sometimes i get my own assert in places i am pretty sure that are not possible unless i get some corruption or wrong result from an AA.
> 
> And finally, i am not entirely sure yet that it is really the compiler's fault... if i manage to have a "send-able" case i will submit it in the bugtracker.

Since it is single-threaded, it should crash in the same place in the same way every time. This means you can put an assert on the crashing data (even without gdb it can be found by inserting printf's), and slowly work it backward to where the data goes wrong.
April 03, 2010
On Fri, 02 Apr 2010 23:01:48 -0700, Walter Bright wrote:

>> > Since it is single-threaded, it should crash in the same place in the
> same way every time. This means you can put an assert on the crashing data (even without gdb it can be found by inserting printf's), and slowly work it backward to where the data goes wrong.

According to an article I was reading that constancy does not apply with current Linux kernels. They deliberately randomize things to make life difficult for hackers.

I'll try to find it again - it's about debugging GCC itself.

Steve

April 03, 2010
On Sat, 03 Apr 2010 06:24:20 +0000, Steve Teale wrote:

> On Fri, 02 Apr 2010 23:01:48 -0700, Walter Bright wrote:
> 
>>> > Since it is single-threaded, it should crash in the same place in the
>> same way every time. This means you can put an assert on the crashing data (even without gdb it can be found by inserting printf's), and slowly work it backward to where the data goes wrong.
> 
> According to an article I was reading that constancy does not apply with current Linux kernels. They deliberately randomize things to make life difficult for hackers.
> 
> I'll try to find it again - it's about debugging GCC itself.
> 
> Steve

Got it - http://gcc.gnu.org/wiki/DebuggingGCC

Right at the end.
April 03, 2010
Steve Teale wrote:
> On Sat, 03 Apr 2010 06:24:20 +0000, Steve Teale wrote:
> 
>> On Fri, 02 Apr 2010 23:01:48 -0700, Walter Bright wrote:
>>  
>>>>> Since it is single-threaded, it should crash in the same place in
>>>>> the
>>> same way every time. This means you can put an assert on the crashing
>>> data (even without gdb it can be found by inserting printf's), and
>>> slowly work it backward to where the data goes wrong.
>> According to an article I was reading that constancy does not apply with
>> current Linux kernels. They deliberately randomize things to make life
>> difficult for hackers.
>>
>> I'll try to find it again - it's about debugging GCC itself.
>>
>> Steve
> 
> Got it - http://gcc.gnu.org/wiki/DebuggingGCC
> 
> Right at the end.

Well, i don't know if this applies to my case, but it is definitely random:

- Compile
- Run
- it crashes after some interaction.
- Run the same again
- crashes immediately.

 I am currently trying to run it in windows to see what comes out. Trying to narrow it down makes it disappear in all of my attempts so far.

j.
April 03, 2010
Steve Teale wrote:
> On Sat, 03 Apr 2010 06:24:20 +0000, Steve Teale wrote:
> 
>> On Fri, 02 Apr 2010 23:01:48 -0700, Walter Bright wrote:
>>  
>>>>> Since it is single-threaded, it should crash in the same place in
>>>>> the
>>> same way every time. This means you can put an assert on the crashing
>>> data (even without gdb it can be found by inserting printf's), and
>>> slowly work it backward to where the data goes wrong.
>> According to an article I was reading that constancy does not apply with
>> current Linux kernels. They deliberately randomize things to make life
>> difficult for hackers.
>>
>> I'll try to find it again - it's about debugging GCC itself.
>>
>> Steve
> 
> Got it - http://gcc.gnu.org/wiki/DebuggingGCC
> 
> Right at the end.

It links to this page which shows how to turn it off:

http://gcc.gnu.org/wiki/Randomization
April 04, 2010
On 04/02/2010 03:53 PM, Walter Bright wrote:
> dsimcha wrote:
>> I almost never do this with any data structure other
>> than an array because, even if it works for now, I consider it a horrible
>> violation of encapsulation because you're relying on the details of
>> how the data
>> structure manipulates memory. This is also why, when I designed RandAA
>> I didn't
>> see this as an issue until you pointed it out to me.
>
> Andrei is working on the design of the D collection class library. After
> much thought and research, he finally came to the conclusion that a
> collection class should not allow the address of a member to be taken. I
> think his reasoning on the issue is pretty sound, and is consistent with
> your take on it.

I wouldn't call it research, but I agonized a fair amount over it. I think Phobos containers will all use malloc, realloc, and free for their own storage, while still being safe.

Andrei
April 04, 2010
On 2010-04-03 23:21:48 -0400, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> said:

> On 04/02/2010 03:53 PM, Walter Bright wrote:
>> dsimcha wrote:
>>> I almost never do this with any data structure other
>>> than an array because, even if it works for now, I consider it a horrible
>>> violation of encapsulation because you're relying on the details of
>>> how the data
>>> structure manipulates memory. This is also why, when I designed RandAA
>>> I didn't
>>> see this as an issue until you pointed it out to me.
>> 
>> Andrei is working on the design of the D collection class library. After
>> much thought and research, he finally came to the conclusion that a
>> collection class should not allow the address of a member to be taken. I
>> think his reasoning on the issue is pretty sound, and is consistent with
>> your take on it.
> 
> I wouldn't call it research, but I agonized a fair amount over it. I think Phobos containers will all use malloc, realloc, and free for their own storage, while still being safe.

I think this is a sound decision. And I'm not necessarily talking about using malloc, realloc, and free (even though a container capable of using realloc is certainly a plus), but the one about decoupling the container interface from any particular memory management implementation.

Question: if the container's memory isn't garbage-collected, how do you implement iterators, eh, ranges so that they are still memory-safe?

-- 
Michel Fortin
michel.fortin@michelf.com
http://michelf.com/

April 04, 2010
== Quote from Michel Fortin (michel.fortin@michelf.com)'s article
> Question: if the container's memory isn't garbage-collected, how do you implement iterators, eh, ranges so that they are still memory-safe?

The way I'm picturing this being implemented is that a GC'd class instance exists at the top level, and then the internal implementation-detail storage that the class uses is implemented via malloc and free.  This storage would get freed in the class finalizer when the instance is GC'd.  In this case all you'd need to do is make the range hold a reference to the class instance so it wouldn't be GC'd.
April 04, 2010
On 2010-04-04 09:45:36 -0400, dsimcha <dsimcha@yahoo.com> said:

> == Quote from Michel Fortin (michel.fortin@michelf.com)'s article
>> Question: if the container's memory isn't garbage-collected, how do you
>> implement iterators, eh, ranges so that they are still memory-safe?
> 
> The way I'm picturing this being implemented is that a GC'd class instance exists
> at the top level, and then the internal implementation-detail storage that the
> class uses is implemented via malloc and free.  This storage would get freed in
> the class finalizer when the instance is GC'd.  In this case all you'd need to do
> is make the range hold a reference to the class instance so it wouldn't be GC'd.

That wouldn't work with realloc: realloc copies to a new location then frees the old memory if it cannot expand in place. You can't keep the old copy allocated.

I've been thinking of another method to ensure safety: don't allow expanding a container as long as there are ranges pointing to it. Easily implemented with a reference count. For instance, if you have a vector container, expanding the container would invalidate ranges. Instead of allowing ranges to become invalid and potentially dangerous, just disallow expanding the container. The range would contain a pointer to its upper and lower bound, and a pointer to the container to increment the reference count when it's copied and decrement it when it's destroyed.

-- 
Michel Fortin
michel.fortin@michelf.com
http://michelf.com/

1 2 3
Top | Discussion index | About this forum | D home