auto objectSource = new FileSource("../libs/object.d");
auto object = lex!((line, index, length) {
	import std.stdio;
	writeln("new location 2 ! ", cast(void*) objectSource);
	
	return Location(objectSource, line, index, length);
})(objectSource.content);

Running this, I see that at some point, objectSource is changed. The output is
new location 2 ! 7F494EEC1D40
new location 2 ! 7F494EEC1D40
...
new location 2 ! 7F494EEC1D40
new location 2 ! 7F494EEC2EA0

Obviously, the program segfault soon after that.

It sounds like some memory corruption occurs under the hood. What can I do to work around that bug and to help solving it ?

On Sunday, 3 March 2013 at 16:48:32 UTC, deadalnix wrote:
> auto objectSource = new FileSource("../libs/object.d");
> auto object = lex!((line, index, length) {
> 	import std.stdio;
> 	writeln("new location 2 ! ", cast(void*) objectSource);
> 	
> 	return Location(objectSource, line, index, length);
> })(objectSource.content);
>
> Running this, I see that at some point, objectSource is changed. The output is
> new location 2 ! 7F494EEC1D40
> new location 2 ! 7F494EEC1D40
> ...
> new location 2 ! 7F494EEC1D40
> new location 2 ! 7F494EEC2EA0
>
> Obviously, the program segfault soon after that.
>
> It sounds like some memory corruption occurs under the hood. What can I do to work around that bug and to help solving it ?

Is this a dmd thing, or does it affect other compilers? (don't have a laptop to test at the moment).

Regards
Iain

On Sunday, 3 March 2013 at 16:51:15 UTC, Iain Buclaw wrote:
> Is this a dmd thing, or does it affect other compilers? (don't have a laptop to test at the moment).
>

I don't know, the code is not compilable with 2.060 (or 2.061) so I'm unable to test how gdc does on that one.

On Mar 3, 2013 5:01 PM, "deadalnix" <deadalnix@gmail.com> wrote:
>
> On Sunday, 3 March 2013 at 16:51:15 UTC, Iain Buclaw wrote:
>>
>> Is this a dmd thing, or does it affect other compilers? (don't have a
laptop to test at the moment).
>>
>
> I don't know, the code is not compilable with 2.060 (or 2.061) so I'm
unable to test how gdc does on that one.

GDC has been on 2.062 frontend for at least a fortnight. I know I don't announce these things, but am still busy working on next set of refactoring.

Regards
-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';

On Sunday, 3 March 2013 at 17:03:24 UTC, Iain Buclaw wrote:
> On Mar 3, 2013 5:01 PM, "deadalnix" <deadalnix@gmail.com> wrote:
>>
>> On Sunday, 3 March 2013 at 16:51:15 UTC, Iain Buclaw wrote:
>>>
>>> Is this a dmd thing, or does it affect other compilers? (don't have a
> laptop to test at the moment).
>>>
>>
>> I don't know, the code is not compilable with 2.060 (or 2.061) so I'm
> unable to test how gdc does on that one.
>
> GDC has been on 2.062 frontend for at least a fortnight. I know I don't
> announce these things, but am still busy working on next set of
> refactoring.
>
> Regards

OK, I'll give it a try tomorrow after some happy compilation.

On Sunday, 3 March 2013 at 16:48:32 UTC, deadalnix wrote:
> auto objectSource = new FileSource("../libs/object.d");
> auto object = lex!((line, index, length) {
> 	import std.stdio;
> 	writeln("new location 2 ! ", cast(void*) objectSource);
> 	
> 	return Location(objectSource, line, index, length);
> })(objectSource.content);
>
> Running this, I see that at some point, objectSource is changed. The output is
> new location 2 ! 7F494EEC1D40
> new location 2 ! 7F494EEC1D40
> ...
> new location 2 ! 7F494EEC1D40
> new location 2 ! 7F494EEC2EA0
>
> Obviously, the program segfault soon after that.
>
> It sounds like some memory corruption occurs under the hood. What can I do to work around that bug and to help solving it ?

It is unlikely that the particular closure corrupts data since at three times the address is correct. Closure bugs are typically revealed as wrong function code, so there would no difference between addresses. Something else in scope of "objectSource" may corrupt it. You can try to pass explicitly types of closure parameters, rewrite closure as a nested function - it used to mitigate some closure bugs in the past. Without compilable source I cannot say anything more.

On Sunday, 3 March 2013 at 17:03:24 UTC, Iain Buclaw wrote:
> On Mar 3, 2013 5:01 PM, "deadalnix" <deadalnix@gmail.com> wrote:
>>
>> On Sunday, 3 March 2013 at 16:51:15 UTC, Iain Buclaw wrote:
>>>
>>> Is this a dmd thing, or does it affect other compilers? (don't have a
> laptop to test at the moment).
>>>
>>
>> I don't know, the code is not compilable with 2.060 (or 2.061) so I'm
> unable to test how gdc does on that one.
>
> GDC has been on 2.062 frontend for at least a fortnight. I know I don't
> announce these things, but am still busy working on next set of
> refactoring.
>
> Regards

Tested, the corruption don't appear with GDC. Good job !

On Sunday, 3 March 2013 at 17:27:52 UTC, Maxim Fomin wrote:
> On Sunday, 3 March 2013 at 16:48:32 UTC, deadalnix wrote:
>> auto objectSource = new FileSource("../libs/object.d");
>> auto object = lex!((line, index, length) {
>> 	import std.stdio;
>> 	writeln("new location 2 ! ", cast(void*) objectSource);
>> 	
>> 	return Location(objectSource, line, index, length);
>> })(objectSource.content);
>>
>> Running this, I see that at some point, objectSource is changed. The output is
>> new location 2 ! 7F494EEC1D40
>> new location 2 ! 7F494EEC1D40
>> ...
>> new location 2 ! 7F494EEC1D40
>> new location 2 ! 7F494EEC2EA0
>>
>> Obviously, the program segfault soon after that.
>>
>> It sounds like some memory corruption occurs under the hood. What can I do to work around that bug and to help solving it ?
>
> It is unlikely that the particular closure corrupts data since at three times the address is correct. Closure bugs are typically revealed as wrong function code, so there would no difference between addresses. Something else in scope of "objectSource" may corrupt it. You can try to pass explicitly types of closure parameters, rewrite closure as a nested function - it used to mitigate some closure bugs in the past. Without compilable source I cannot say anything more.

The problem here is that the codebase involved is rather huge. Do you have an advice to reduce the issue ?

On Monday, 4 March 2013 at 02:10:12 UTC, deadalnix wrote:
> On Sunday, 3 March 2013 at 17:27:52 UTC, Maxim Fomin wrote:
>> On Sunday, 3 March 2013 at 16:48:32 UTC, deadalnix wrote:
>>> auto objectSource = new FileSource("../libs/object.d");
>>> auto object = lex!((line, index, length) {
>>> 	import std.stdio;
>>> 	writeln("new location 2 ! ", cast(void*) objectSource);
>>> 	
>>> 	return Location(objectSource, line, index, length);
>>> })(objectSource.content);
>>>
>>> Running this, I see that at some point, objectSource is changed. The output is
>>> new location 2 ! 7F494EEC1D40
>>> new location 2 ! 7F494EEC1D40
>>> ...
>>> new location 2 ! 7F494EEC1D40
>>> new location 2 ! 7F494EEC2EA0
>>>
>>> Obviously, the program segfault soon after that.
>>>
>>> It sounds like some memory corruption occurs under the hood. What can I do to work around that bug and to help solving it ?
>>
>> It is unlikely that the particular closure corrupts data since at three times the address is correct. Closure bugs are typically revealed as wrong function code, so there would no difference between addresses. Something else in scope of "objectSource" may corrupt it. You can try to pass explicitly types of closure parameters, rewrite closure as a nested function - it used to mitigate some closure bugs in the past. Without compilable source I cannot say anything more.
>
> The problem here is that the codebase involved is rather huge. Do you have an advice to reduce the issue ?

No specific advice except for dropping everything except FileSource definition and lex template. By the way, what is objectSource.content? A tuple of line, index, length?

On Monday, 4 March 2013 at 07:10:20 UTC, Maxim Fomin wrote:
> No specific advice except for dropping everything except FileSource definition and lex template. By the way, what is objectSource.content? A tuple of line, index, length?

The problem is that lex is massive : it is a whole lexer. objectSource.content is simply a string.