November 07, 2018
On Wed, Nov 7, 2018 at 11:55 PM Joakim via Digitalmars-d-announce <digitalmars-d-announce@puremagic.com> wrote:
>
> On Thursday, 8 November 2018 at 07:41:58 UTC, Manu wrote:
> > On Wed, Nov 7, 2018 at 10:30 PM Joakim via Digitalmars-d-announce <digitalmars-d-announce@puremagic.com> wrote:
> >>
> >> On Thursday, 8 November 2018 at 04:16:44 UTC, Manu wrote:
> >> > On Tue, Nov 6, 2018 at 10:05 AM Vladimir Panteleev via Digitalmars-d-announce <digitalmars-d-announce@puremagic.com> wrote:
> >> >> [...]
> >> >
> >> > "Indeed, a clean build of DMD itself (about 170’000 lines of D and 120’000 lines of C/C++) takes no longer than 4 seconds to build on a rather average developer machine."
> >> >
> >> > ...what!? DMD takes me... (compiling) ... 1 minute 40 seconds to build! And because DMD does all-files-at-once compilation, rather than separate compilation for each source file, whenever you change just one line in one file, you incur that entire build time, every time, because it can't just rebuild the one source file that changed. You also can't do multi-processor builds with all-in-one build strategies.
> >> >
> >> > 4 seconds? That's just untrue. D is actually kinda slow these days... In my experience it's slower than modern C++ compilers by quite a lot.
> >>
> >> It sounds like you're not using "a rather average developer machine" then, as there's no way DMD should be that slow to build on a core i5 or better:
> >>
> >> https://forum.dlang.org/post/rqukhkpxcvgiefrdcvdq@forum.dlang.org
> >
> > I'm on an i7 with 8 threads and plenty of ram... although threads are useless, since DMD only uses one ;)
>
> Running Windows XP? ;) That does sound like Windows though, as I do remember being surprised how long dmd took to build on Win7 when I tried it 8-9 years back. I still don't think the toolchain should be _that_ much slower than linux though.
>
> Btw, the extra cores are _not_ useless for the DMD backend, which has always used separate compilation, whether written in C++ or D.

No, you're right, the backend builds in 2-3 seconds.

November 08, 2018
On Thursday, 8 November 2018 at 07:54:56 UTC, Manu wrote:
> On Wed, Nov 7, 2018 at 10:30 PM Vladimir Panteleev via Digitalmars-d-announce <digitalmars-d-announce@puremagic.com> wrote:
>>
>> On Thursday, 8 November 2018 at 06:08:20 UTC, Vladimir Panteleev wrote:
>> > It was definitely about 4 seconds not too long ago, a few years at most.
>>
>> No, it's still 4 seconds.
>>
>> digger --offline --config-file=/dev/null -j auto -c local.cache=none build     7.31s user 1.51s system 203% cpu 4.340 total
>>
>> > It does seem to take more time now; I wonder why.
>>
>> If it takes longer, then it's probably because it's being built in one CPU core, or in the release build.
>
> https://youtu.be/msWuRlD3zy0

Lol, I saw that link and figured it was either some comedy video, like the Python ones Walter sometimes posts, or you were actually showing us how long it takes. Pretty funny to see the latter.

> DMD only builds with one core, since it builds altogether.

Yes, but your build time is unusually long even with one core. Are the D backend and frontend at least built in parallel to each other? It doesn't seem to be even doing that, though they're separate invocations of DMD.

> And all builds are release builds... what good is a debug build? DMD
> is unbelievably slow in debug. If it wasn't already slow enough... if
> I try and build with a debug build, it takes closer to 5 minutes.
>
> I suspect one part of the problem is that DMD used to be built with a C compiler, and now it's built with DMD... it really should be built with LDC at least?

Could be part of the problem on Windows, dunno.
November 08, 2018
On 11/7/2018 10:08 PM, Vladimir Panteleev wrote:
> It does seem to take more time now; I wonder why.

That's easy. It's because nobody has profiled it for years.

Another reason is the backend being in D now, it is being compiled with DMD rather than gcc/clang.

November 08, 2018
On 11/7/2018 11:41 PM, Manu wrote:
> I'm on an i7 with 8 threads and plenty of ram... although threads are
> useless, since DMD only uses one ;)

So does every other compiler.

To do a multicore build, you'll need to use a makefile that supports -j.

November 08, 2018
On Thu, Nov 8, 2018 at 12:10 AM Joakim via Digitalmars-d-announce <digitalmars-d-announce@puremagic.com> wrote:
>
> On Thursday, 8 November 2018 at 07:54:56 UTC, Manu wrote:
> > On Wed, Nov 7, 2018 at 10:30 PM Vladimir Panteleev via Digitalmars-d-announce <digitalmars-d-announce@puremagic.com> wrote:
> >>
> >> On Thursday, 8 November 2018 at 06:08:20 UTC, Vladimir Panteleev wrote:
> >> > It was definitely about 4 seconds not too long ago, a few years at most.
> >>
> >> No, it's still 4 seconds.
> >>
> >> digger --offline --config-file=/dev/null -j auto -c local.cache=none build     7.31s user 1.51s system 203% cpu 4.340 total
> >>
> >> > It does seem to take more time now; I wonder why.
> >>
> >> If it takes longer, then it's probably because it's being built in one CPU core, or in the release build.
> >
> > https://youtu.be/msWuRlD3zy0
>
> Lol, I saw that link and figured it was either some comedy video, like the Python ones Walter sometimes posts, or you were actually showing us how long it takes. Pretty funny to see the latter.

It's not so funny when every one-line tweak burns 2 minutes of my life away.

> > DMD only builds with one core, since it builds altogether.
>
> Yes, but your build time is unusually long even with one core. Are the D backend and frontend at least built in parallel to each other?

That doesn't matter, you can clearly see the backend built in less than 2 seconds.

> It doesn't seem to be even doing that, though they're separate invocations of DMD.

I didn't configure the build infrastructure!

> > And all builds are release builds... what good is a debug
> > build? DMD
> > is unbelievably slow in debug. If it wasn't already slow
> > enough... if
> > I try and build with a debug build, it takes closer to 5
> > minutes.
> >
> > I suspect one part of the problem is that DMD used to be built with a C compiler, and now it's built with DMD... it really should be built with LDC at least?
>
> Could be part of the problem on Windows, dunno.

Well... ffs... people need to care about this! >_<
November 08, 2018
On Thu, Nov 8, 2018 at 12:10 AM Walter Bright via Digitalmars-d-announce <digitalmars-d-announce@puremagic.com> wrote:
>
> On 11/7/2018 11:41 PM, Manu wrote:
> > I'm on an i7 with 8 threads and plenty of ram... although threads are useless, since DMD only uses one ;)
>
> So does every other compiler.
>
> To do a multicore build, you'll need to use a makefile that supports -j.

Right.
So...?

Also, MSBuild is what people use on Windows... but same applies.
November 08, 2018
On Thursday, 8 November 2018 at 08:29:28 UTC, Manu wrote:
> On Thu, Nov 8, 2018 at 12:10 AM Joakim via Digitalmars-d-announce <digitalmars-d-announce@puremagic.com> wrote:
>>
>> On Thursday, 8 November 2018 at 07:54:56 UTC, Manu wrote:
>> > On Wed, Nov 7, 2018 at 10:30 PM Vladimir Panteleev via Digitalmars-d-announce <digitalmars-d-announce@puremagic.com> wrote:
>> >>
>> >> On Thursday, 8 November 2018 at 06:08:20 UTC, Vladimir Panteleev wrote:
>> >> > It was definitely about 4 seconds not too long ago, a few years at most.
>> >>
>> >> No, it's still 4 seconds.
>> >>
>> >> digger --offline --config-file=/dev/null -j auto -c local.cache=none build     7.31s user 1.51s system 203% cpu 4.340 total
>> >>
>> >> > It does seem to take more time now; I wonder why.
>> >>
>> >> If it takes longer, then it's probably because it's being built in one CPU core, or in the release build.
>> >
>> > https://youtu.be/msWuRlD3zy0
>>
>> Lol, I saw that link and figured it was either some comedy video, like the Python ones Walter sometimes posts, or you were actually showing us how long it takes. Pretty funny to see the latter.
>
> It's not so funny when every one-line tweak burns 2 minutes of my life away.

I was laughing that you actually proved your point with direct video evidence, obviously it's sad that it takes so long.

>> > DMD only builds with one core, since it builds altogether.
>>
>> Yes, but your build time is unusually long even with one core. Are the D backend and frontend at least built in parallel to each other?
>
> That doesn't matter, you can clearly see the backend built in less than 2 seconds.

The C/C++ files in the beginning are built very fast, but the D files in the backend appear to take much longer, kicking in at 1:18 of your video and then the next compilation step starts at 1:40.

I suspect part of the problem is that your build is being done completely serially, even for separate compilation. I have no experience with VS, so I don't know why that is.

>> It doesn't seem to be even doing that, though they're separate invocations of DMD.
>
> I didn't configure the build infrastructure!

Maybe you can? I have no experience with VS, but surely it has some equivalent of ninja -j5?

>> > And all builds are release builds... what good is a debug
>> > build? DMD
>> > is unbelievably slow in debug. If it wasn't already slow
>> > enough... if
>> > I try and build with a debug build, it takes closer to 5
>> > minutes.
>> >
>> > I suspect one part of the problem is that DMD used to be built with a C compiler, and now it's built with DMD... it really should be built with LDC at least?
>>
>> Could be part of the problem on Windows, dunno.
>
> Well... ffs... people need to care about this! >_<

I agree that the official release of DMD for Windows should be faster, and we should be building it with ldc... if that's the problem.
November 08, 2018
On Thu, Nov 8, 2018 at 12:55 AM Joakim via Digitalmars-d-announce <digitalmars-d-announce@puremagic.com> wrote:
>
> On Thursday, 8 November 2018 at 08:29:28 UTC, Manu wrote:
> > On Thu, Nov 8, 2018 at 12:10 AM Joakim via Digitalmars-d-announce <digitalmars-d-announce@puremagic.com> wrote:
> >>
> >> On Thursday, 8 November 2018 at 07:54:56 UTC, Manu wrote:
> >
> > I didn't configure the build infrastructure!
>
> Maybe you can? I have no experience with VS, but surely it has some equivalent of ninja -j5?

msbuild does parallel builds quite effectively. I expect it perceives
a dependency between jobs which cause it to serialise. Maybe there's a
legit dependency, or maybe the msbuild script has a problem? Either
way, it's not acceptable.
I would log this is maximum priority bug.
(https://issues.dlang.org/show_bug.cgi?id=19377)

> >> > And all builds are release builds... what good is a debug
> >> > build? DMD
> >> > is unbelievably slow in debug. If it wasn't already slow
> >> > enough... if
> >> > I try and build with a debug build, it takes closer to 5
> >> > minutes.
> >> >
> >> > I suspect one part of the problem is that DMD used to be built with a C compiler, and now it's built with DMD... it really should be built with LDC at least?
> >>
> >> Could be part of the problem on Windows, dunno.
> >
> > Well... ffs... people need to care about this! >_<
>
> I agree that the official release of DMD for Windows should be faster, and we should be building it with ldc... if that's the problem.

I think it's a combination of problems, and primary problem being criminal negligence!
November 08, 2018
On Thursday, 8 November 2018 at 04:16:44 UTC, Manu wrote:
> On Tue, Nov 6, 2018 at 10:05 AM Vladimir Panteleev via Digitalmars-d-announce <digitalmars-d-announce@puremagic.com> wrote:
>>
>> This is a tool + article I wrote in February, but never got around to finishing / publishing until today.
>>
>> https://blog.thecybershadow.net/2018/02/07/dmdprof/
>>
>> Hopefully someone will find it useful.
>
> "Indeed, a clean build of DMD itself (about 170’000 lines of D and 120’000 lines of C/C++) takes no longer than 4 seconds to build on a rather average developer machine."
>
> ...what!? DMD takes me... (compiling) ... 1 minute 40 seconds to build! And because DMD does all-files-at-once compilation, rather than separate compilation for each source file, whenever you change just one line in one file, you incur that entire build time, every time, because it can't just rebuild the one source file that changed. You also can't do multi-processor builds with all-in-one build strategies.
>
> 4 seconds? That's just untrue. D is actually kinda slow these days... In my experience it's slower than modern C++ compilers by quite a lot.

Everything on Windows is slower. I'm 99% sure the fault lies with NTFS.

I worked on a cross platform framework back in the day, nearly all of the code was identical between Linux and Windows. On the same dual-boot machine the build was between one and two orders of magnitude slower on Windows. Again: same code, same machine, no VMs. _Anything_ that used the filesystem was significantly slower, including CMake. It got to the point that the devs that preferred to use Windows only ever used the unity build because incremental ones were too slow to bear.

I'd love to know how long it takes to build clang on your machine.
November 08, 2018
On Tuesday, 6 November 2018 at 19:01:58 UTC, H. S. Teoh wrote:
> It looks like it would be really useful one day when I try to tackle the dmd-on-lowmem-system problem again.

Based on my profiling it seems that most memory is allocated in void importAll(Scope* sc) found in attrib.d . A person with more knowledge of DMD source could create new allocator for scope data and when its no longer needed just dealloc all. My intuition says that after IR is generated we no longer need scope information.

Here is profile data for simple file.
https://imgur.com/a/ROa6JNd