On Tuesday, 12 December 2017 at 01:03:43 UTC, David Nadlinger wrote:
> Hi Diederik,
>
> On 11 Dec 2017, at 23:16, Diederik de Groot via dmd-internals wrote:
>> From the above i gather, from your response, i should be ok with using 2.068 as the intermediate stage.
>
> What might be easier than to use an intermediate stage is getting a DMD version tagged that is 2.068.2, but with the one D dependency removed. 2.068.2 is basically still entirely C++; only some auxiliary code has been converted to D to test the waters
>
> For LDC, more specifically our last C++-based version (0.17.x/ltsmaster branch), we just reverted that change. The resulting 2.068.2-based compiler can directly bootstrap current master.
>
> Best,
> David

Hi David,

Thanks for your reaction !

That's exactly what i was looking for :-) Would it be possible to get the LDC c++ based version of 2.068.2 committed to the dmd repo (something like '2.068.3' or maybe even as a new version of 'dmd-cxx'), so that i can rebase my 'dragonflybsd_v2.068.2' changes on top of that later on ? As your changes have already been proven to work, it should be easier to get it accepted.

Figuring out which branch to use during porting is making the porting process a little cumbersome.

@Iain: as dmd-cxx is not able to compile the current 'master' and '2.068.2' is, would it make sense to replace the current dmd-cxx with the LDC version c++ based of 2.068.2 ? I know you did quite a bit of work backporting several patches to dmd-cxx. I am just trying to help simplify porting of dmd to new platforms. And having only to stages would make all of this a lot easier.

Regards,

Diederik
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On 12 December 2017 at 02:17, Diederik de Groot via dmd-internals <dmd-internals@puremagic.com> wrote:
> On Tuesday, 12 December 2017 at 01:03:43 UTC, David Nadlinger wrote:
>>
>> Hi Diederik,
>>
>> On 11 Dec 2017, at 23:16, Diederik de Groot via dmd-internals wrote:
>>>
>>> From the above i gather, from your response, i should be ok with using 2.068 as the intermediate stage.
>>
>>
>> What might be easier than to use an intermediate stage is getting a DMD version tagged that is 2.068.2, but with the one D dependency removed. 2.068.2 is basically still entirely C++; only some auxiliary code has been converted to D to test the waters
>>
>> For LDC, more specifically our last C++-based version (0.17.x/ltsmaster branch), we just reverted that change. The resulting 2.068.2-based compiler can directly bootstrap current master.
>>
>> Best,
>> David
>
>
> Hi David,
>
> Thanks for your reaction !
>
> That's exactly what i was looking for :-) Would it be possible to get the LDC c++ based version of 2.068.2 committed to the dmd repo (something like '2.068.3' or maybe even as a new version of 'dmd-cxx'), so that i can rebase my 'dragonflybsd_v2.068.2' changes on top of that later on ? As your changes have already been proven to work, it should be easier to get it accepted.
>

What do you mean?  The compiler has been proven to work, and it does work, both in build dmd and gdc compilers (https://github.com/D-Programming-GDC/GDC/pull/550).

> Figuring out which branch to use during porting is making the porting process a little cumbersome.
>
> @Iain: as dmd-cxx is not able to compile the current 'master' and '2.068.2' is, would it make sense to replace the current dmd-cxx with the LDC version c++ based of 2.068.2 ? I know you did quite a bit of work backporting several patches to dmd-cxx. I am just trying to help simplify porting of dmd to new platforms. And having only to stages would make all of this a lot easier.
>

It can compile the current 'master' when backed by gdc.  If someone wants to fix dmd backend to make it work better, you're free to do that also.

Did you have a look at the last-cdmd tag?

Iain.
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On Tuesday, 12 December 2017 at 10:33:36 UTC, Iain Buclaw wrote:
>> That's exactly what i was looking for :-) Would it be possible to get the LDC c++ based version of 2.068.2 committed to the dmd repo (something like '2.068.3' or maybe even as a new version of 'dmd-cxx'), so that i can rebase my 'dragonflybsd_v2.068.2' changes on top of that later on ? As your changes have already been proven to work, it should be easier to get it accepted.

> What do you mean?  The compiler has been proven to work, and it does work, both in build dmd and gdc compilers (https://github.com/D-Programming-GDC/GDC/pull/550).

I may have formulated my intention incorrectly, sorry. I know dmd, ldc and gdc work :-) Did not want to instigate a flame war of any kind.

I was having quite a lot of issues when trying to use the dmd-cxx branch (Ran several tries on linux and my partial port to dragonflybsd) when trying to use it to compile newer dmd versions. I finally settled on the three stage method (which worked).

It's a little hard to 'fix' and 'port' at the same time, because you don't know if your porting effort is having side effects.

What i head intended to say, was: If David had a versipn dmd v2.068.2 that was still completely in c++ and able to compile 'master' that would be perfect. But reading your response and re-reading his, that was not what he meant.

Please discard my previous comment.

>> Figuring out which branch to use during porting is making the porting process a little cumbersome.
>>
>> @Iain: as dmd-cxx is not able to compile the current 'master' and '2.068.2' is, would it make sense to replace the current dmd-cxx with the LDC version c++ based of 2.068.2 ? I know you did quite a bit of work backporting several patches to dmd-cxx. I am just trying to help simplify porting of dmd to new platforms. And having only to stages would make all of this a lot easier.
>>
> It can compile the current 'master' when backed by gdc.  If someone wants to fix dmd backend to make it work better, you're free to do that also.
Once i get my port project finished and stable, i will revisit this issue and see if i can figure out what is going on in dmd-cxx.
>
> Did you have a look at the last-cdmd tag?
I did not to be honest. I will give that one a shot later today, to see if that one works.

I will have to write a little wiki article about the porting process to help others.

Thanks you for your input !

Regards,

Diederik
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On Tuesday, 12 December 2017 at 10:33:36 UTC, Iain Buclaw wrote:
> ...
>
> Did you have a look at the last-cdmd tag?

Hi Iain,

Thanks for this suggestion.

I tried the latest 'last-cdmd' tagged version first, but found that it already has a dependence on a dmd compiler (Some patches were merged, after v2.067.1 on which it was based), so i went to branch 'v2.067.1' and this one compiles cleanly and is able to compile the 'master' (v2.077.2) branch in one go.

So this gets me to a simple two stage compile, which is fantastic.

See: https://github.com/dkgroot/dragonflybsd_dmd_port.git

Thanks a lot for your help !

Regards,

Diederik
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On 2017-12-12 02:17, Diederik de Groot via dmd-internals wrote:
> @Iain: as dmd-cxx is not able to compile the current 'master' and
> '2.068.2' is, would it make sense to replace the current dmd-cxx with
> the LDC version c++ based of 2.068.2 ? I know you did quite a bit of
> work backporting several patches to dmd-cxx.

FWIW we do have Travis and auto-tester configured to run the dmd-cxx branch, see e.g.:

https://auto-tester.puremagic.com/?projectid=17 (one some platforms we can even build druntime)

It's just that Iain is the only one so far who showed interest in the dmd-cxx branch.
As you can see it's based on 2.068.2 and contains all commits until last-cdmd:

https://github.com/dlang/dmd/pull/7014
https://github.com/dlang/dmd/pull/7015

And a few backports from gdc:

https://github.com/dlang/dmd/pull/7017
https://github.com/dlang/dmd/pull/7227

There are similar dmd-cxx branches at druntime and Phobos.
Of course, work to improve the status quo is very welcome ;-)

> Regards,
> 
> Diederik
> _______________________________________________
> dmd-internals mailing list
> dmd-internals@puremagic.com
> http://lists.puremagic.com/mailman/listinfo/dmd-internals
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On Wednesday, 13 December 2017 at 05:11:38 UTC, Sebastian Wilzbach wrote:
> On 2017-12-12 02:17, Diederik de Groot via dmd-internals wrote:
>> @Iain: as dmd-cxx is not able to compile the current 'master' and
>> '2.068.2' is, would it make sense to replace the current dmd-cxx with
>> the LDC version c++ based of 2.068.2 ? I know you did quite a bit of
>> work backporting several patches to dmd-cxx.
>
> FWIW we do have Travis and auto-tester configured to run the dmd-cxx branch, see e.g.:
>
> https://auto-tester.puremagic.com/?projectid=17 (one some platforms we can even build druntime)

Because it was the only 'active' c++ based branch that was seemed to be maintained, i choose it in my first attempt to port dmd to dragonflybsd.
>
> It's just that Iain is the only one so far who showed interest in the dmd-cxx branch.
> As you can see it's based on 2.068.2 and contains all commits until last-cdmd:
>
> https://github.com/dlang/dmd/pull/7014
> https://github.com/dlang/dmd/pull/7015
>
> And a few backports from gdc:
>
> https://github.com/dlang/dmd/pull/7017
> https://github.com/dlang/dmd/pull/7227
>
> There are similar dmd-cxx branches at druntime and Phobos.
> Of course, work to improve the status quo is very welcome ;-)

I have now (mostly) finished my port, currently waiting for the original poster of the DragonFlyBSD post (http://forum.dlang.org/post/CAG2UyHqsa7tv5cVMZWOrZXo5QQus66X5fmbBixxJLooH2J=eWw@mail.gmail.com), to give it a test run), before creating a PR.

After this project is done, i will have another look at dmd-cxx to see what is causing the issues i have been seeing when compiling on linux (x86_64).
For example: '/usr/bin/ld: /data/develop/dmd/dmd-cxx/druntime/lib/libdruntime-linux64.a(config.o): relocation R_X86_64_32 against undefined symbol `__dmd_personality_v0' can not be used when making a shared object; recompile with -fPIC'
The travis build (7f7dd83) mentioned above passes cleanly, when i do the exact same steps on current version of ubuntu (artful), i get different results. This might also explain why the auto-tester is having some issues.

I think having a working reference bootstrap compiler which passes all it's unittests is something worthwhile.

Regards,

Diederik
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On Wednesday, 13 December 2017 at 12:15:07 UTC, Diederik de Groot wrote:

> After this project is done, i will have another look at dmd-cxx to see what is causing the issues i have been seeing when compiling on linux (x86_64).
> For example: '/usr/bin/ld: /data/develop/dmd/dmd-cxx/druntime/lib/libdruntime-linux64.a(config.o): relocation R_X86_64_32 against undefined symbol `__dmd_personality_v0' can not be used when making a shared object; recompile with -fPIC'

Most likely these will be related to PIC/PIE issues on hardened platforms. See: https://github.com/dlang/dmd/pull/7420. So to fix these i would have to backport some of these PR's to dmd-cxx.
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On 2017-12-13 14:26, Diederik de Groot via dmd-internals wrote:
> On Wednesday, 13 December 2017 at 12:15:07 UTC, Diederik de Groot wrote:
> 
>> After this project is done, i will have another look at dmd-cxx to see what is causing the issues i have been seeing when compiling on linux (x86_64).
>> For example: '/usr/bin/ld: /data/develop/dmd/dmd-cxx/druntime/lib/libdruntime-linux64.a(config.o): relocation R_X86_64_32 against undefined symbol `__dmd_personality_v0' can not be used when making a shared object; recompile with -fPIC'
> 
> Most likely these will be related to PIC/PIE issues on hardened
> platforms. See: https://github.com/dlang/dmd/pull/7420. So to fix
> these i would have to backport some of these PR's to dmd-cxx.

Yes, the OS hardening is rather annoying:/
Btw -fPIC also needs to be added to druntime (and Phobos, but building Phobos isn't essential for bootstrapping).
Though, I don't think that the overhead of PIE on 32-bit systems will be an issue for the intermediate bootstrap compiler and thus it doesn't need to be as complicated as for master:

https://github.com/dlang/druntime/pull/1880
https://github.com/dlang/druntime/pull/1974
https://github.com/dlang/phobos/pull/5586
https://github.com/dlang/phobos/pull/5868
https://github.com/dlang/phobos/pull/5868
https://github.com/dlang/dmd/pull/7002
https://github.com/dlang/dmd/pull/7420
https://github.com/dlang/dmd/pull/7427
https://github.com/dlang/dmd/pull/7433

So simply add -fPIC to DFLAGS or maybe even hard-code it like it's done for macOS:

https://github.com/dlang/dmd/blob/master/src/ddmd/mars.d#L465

> _______________________________________________
> dmd-internals mailing list
> dmd-internals@puremagic.com
> http://lists.puremagic.com/mailman/listinfo/dmd-internals
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On Wednesday, 13 December 2017 at 13:48:01 UTC, Sebastian Wilzbach wrote:
> Yes, the OS hardening is rather annoying:/
> Btw -fPIC also needs to be added to druntime (and Phobos, but building Phobos isn't essential for bootstrapping).
> Though, I don't think that the overhead of PIE on 32-bit systems will be an issue for the intermediate bootstrap compiler and thus it doesn't need to be as complicated as for master:
>
> https://github.com/dlang/druntime/pull/1880
> https://github.com/dlang/druntime/pull/1974
> https://github.com/dlang/phobos/pull/5586
> https://github.com/dlang/phobos/pull/5868
> https://github.com/dlang/phobos/pull/5868
> https://github.com/dlang/dmd/pull/7002
> https://github.com/dlang/dmd/pull/7420
> https://github.com/dlang/dmd/pull/7427
> https://github.com/dlang/dmd/pull/7433
>
> So simply add -fPIC to DFLAGS or maybe even hard-code it like it's done for macOS:
>
> https://github.com/dlang/dmd/blob/master/src/ddmd/mars.d#L465

Hi Sebastian,

Thanks for finding all the PR's related to this issue.

I will look into as soon as my current project is finished. Can only concentrate on one thing at a time :-)

Regards,

Diederik
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On 13 Dec 2017, at 14:09, Diederik de Groot via dmd-internals wrote:
> On Wednesday, 13 December 2017 at 13:48:01 UTC, Sebastian Wilzbach wrote:
>> […]
>> So simply add -fPIC to DFLAGS or maybe even hard-code it like it's done for macOS:
>>
>> https://github.com/dlang/dmd/blob/master/src/ddmd/mars.d#L465
>
> […]
>
> Thanks for finding all the PR's related to this issue.
>
> I will look into as soon as my current project is finished. Can only concentrate on one thing at a time :-)

Yep, just defaulting to -fPIC should be perfectly workable. LDC has been doing this since the first Ubuntu version requiring PIC was released (16.10?), and we haven't had any complaints so far. Since I'm not sure DMD has a way of negating -fPIC once it has been set, just adding it to the default config file is probably the best approach, as it leaves a backdoor open for users wanting to build non-PIC code.

Best,
David
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals