December 29, 2017
On Friday, 22 December 2017 at 10:06:18 UTC, Joakim wrote:
> This one of the main strengths of D, it is what Walter focuses on, yet I have seen almost nothing on the D blog talking about this.  What brought me to emphasize this today is this recent post about how long it takes to compile the mostly-C++ Chromium web browser and the reddit discussion about it:
>
> https://lobste.rs/s/iri1te/chromium_has_compilation_time_problem
> https://www.reddit.com/r/programming/comments/7ktzog/chromium_has_a_compilation_time_problem/
>
> I'm tempted to call BS on that 6-7 hour build time, as I used to contribute to the Chromium project, and I think it used to take me 20 minutes for a release build in a FreeBSD jail on a fairly weak, 2008-vintage mini-desktop, a dual-core Celeron with 2 GBs of RAM (don't hold me to that build time, just a vague recollection, but probably in the ballpark).  Of course, the last time I built Chromium was more than 5 years ago, and a lot has likely changed since then, such as now using LTO to speed up the browser apparently, and maybe the cross-compilation toolchain for ARM is slower, though others note similar times for native x64 compilation also.
>
> That still implies a slowdown of 2-3 orders of magnitude over the last 5 years, given the much more powerful hardware he's using, which is nuts.
>
> D really needs the community to write blog posts talking about how fast it is, publicizing that there is an alternative to these glacial build times: who wants to do this?  It doesn't need to be on the D blog, could just be on your personal blog, but it is a message that really needs to be spread.

To be fair, you are basing this off C++. C++ has an obnoxiously slow build system at times and everyone knows this, have known this, and have even addressed some parts of it. Yet they still use it. I don't think build times are that big of a deal. It's a nice to have, but as long as a fairly sizable project compiles in a less than a minute, you won't stand out in a bad way. If anything, D should leverage the fast compile times to take advantage of the leeway it gives so the compiler can do a bit of extra work to give better error messages.
December 29, 2017
On Friday, December 29, 2017 22:37:56 I Love Stuffing via Digitalmars-d wrote:
> To be fair, you are basing this off C++. C++ has an obnoxiously slow build system at times and everyone knows this, have known this, and have even addressed some parts of it. Yet they still use it. I don't think build times are that big of a deal. It's a nice to have, but as long as a fairly sizable project compiles in a less than a minute, you won't stand out in a bad way.

So much about build times and how much they matter seems to be about perception. For instance, I used to work at a company whose main project took over 3 - 4 hours to do a clean build. After the project was redone in Java, the build times were about 10 minutes, and folks complained about how long they were even though they were worlds better than the C++ build times had been.

It also depends on your personality. Personally, while I like builds being near instantaneous when dealing with unit tests, if they take a few seconds, it's not a big deal, but I know folks who get seriously annoyed if they can notice much of a difference time-wise between hitting enter to run the build command and hitting enter with no command. Either way though, having build times be more than a few seconds when dealing with unit tests tends to cause serious problems even though having a build for an entire project take a few minutes usually isn't a big deal unless you're really impatient. Having the delay between making changes and testing them be very long can _really_ slow down development, and so that's where build times really matter.

- Jonathan M Davis

December 29, 2017
On Thursday, 28 December 2017 at 21:58:07 UTC, Walter Bright wrote:
> On 12/28/2017 5:22 AM, Atila Neves wrote:
>> On Thursday, 28 December 2017 at 07:21:28 UTC, Walter Bright wrote:
>>> On 12/27/2017 1:23 PM, Atila Neves wrote:
>>>> However, my experience has been that D has fast builds from scratch, but is really really slow for incremental builds.
>>> You can do fast incremental builds with D by not putting all the files on the command line. Just put one.
>> 
>> I don't build the compiler command-line myself, nor do I want to. Even then, recompiling one file isn't useful to me, what is is to recompile what's needed and run the tests. Emacs/flycheck already highlights compilation errors in just one file.
>
> I don't understand. C++ compiles files one at a time. You can do the same in D. How can this model be useful in C++ but not with D?

Because:

. As Iain pointed out independently, D modules are like C/C++ headers. All dependencies must also be recompiled. If I edit a C++ source file, only that source file has to be recompiled. I can use .di files, and they can even be generated for me (nice!) but there's no build system to do this properly. This is one of the ideas I alluded to earlier - I want to put this sort of thing in reggae. And now that dmd is a dub package I can even just use the actual compiler code. i.e. Generate the .di, if it's the same as before don't overwrite the old one to not mess with the timestamp.

. C++ doesn't have built-in unit tests. Which means a framework has to be used and since nobody wants to put their tests in #ifdef "blocks" they live in separate files. Changing the implementation doesn't require the tests to be rebuilt. Yes, I can separate them in D too, but then I lose DDoc benefits and it's actually nicer to see tests next to the code. The fact that all D files are effectively headers means editing a unittest means recompiling all modules that depend on the current one.

. Related to the previous point: compiling separately in D is actually slower overall than by package, because of all of the dependencies (I measured). Yes, I can compile by package, but I don't want to do this by hand. To the extent of my knowledge the only way to get per-package compilation by default or otherwise in the D ecosystem is reggae. But:

. Until recently, it wasn't possible to use __traits(getUnitTests) with separate compilation. I've run into other issues with when doing so (I can't remember what they were). Even with the bug fix editing a test entails compiling several modules and it's slower than just one C++ file. In C++ editing a test source file always means recompiling exactly one file.

. Speaking of dependencies, there's no easy way currently to know what they are. _If_ I want to build per package/module and only build what's needed, then I need to either hand-roll a Makefile (points gun to head) or use reggae. If there's any other alternative, I don't know about it. Even Phobos recompiles the world. That's not a good example to set.

. C++ test framework's hacky usage of global variable assignment to register tests seems to be faster to compile than D's compile-time reflection to achieve the same result. I'd have to benchmark to be sure though.

. If one has dub dependencies (and in all likelihood one has), then either one is a reggae early adopter (I keep mentioning it to highlight why I wrote it in the first place), or in the 99.9% of other users typing either `dub build` or `dub test`. There is no per-package build option. There's a per-module option, but it's single-threaded, therefore slow. Also, no dependency tracking, so let's recompile the whole dub package! In one thread. Zzzzzzzzz.

What I really want is to hit F5 in emacs, rebuild the current file _and_ all of its dependencies, cache the dependency build per file so that if they haven't change don't rebuild, link and run the unit tests. But build _only_ what's needed to avoid linker errors in order to be able to run the unit tests for _one_ file.

I spent an afternoon writing the necessary elisp to get this done and it worked! Then I tried a different project and it failed miserably. So did the 3rd one. I think I just remembered one of the separate compilation bugs I mentioned above. I know, I know, if I don't file it it won't get fixed (even if I file it I have a sneaky suspicion it'll be me fixing it anyway), but figuring out what went wrong takes time and energy and I haven't had enough of either to take this off my TODO list.

To summarise:

. Dependencies.
. There's no standard build system that isn't called dub.
. There's only one alternative build system that can do any of what's needed.
. That alternative build system hasn't implemented my .di idea yet.
. Separate compilation bugs. That nobody ever hits because dub builds everything at once.



Atila
December 29, 2017
On Fri, Dec 29, 2017 at 03:49:07PM -0700, Jonathan M Davis via Digitalmars-d wrote: [...]
> It also depends on your personality. Personally, while I like builds being near instantaneous when dealing with unit tests, if they take a few seconds, it's not a big deal, but I know folks who get seriously annoyed if they can notice much of a difference time-wise between hitting enter to run the build command and hitting enter with no command. Either way though, having build times be more than a few seconds when dealing with unit tests tends to cause serious problems even though having a build for an entire project take a few minutes usually isn't a big deal unless you're really impatient. Having the delay between making changes and testing them be very long can _really_ slow down development, and so that's where build times really matter.
[...]

If you think a few minutes is bad... I've been on the receiving end of builds that take 30-45 *minutes* for 1-line code changes (*ahem*cough*C++ header files*cough*ahem*).  It put a serious damper on my ability to make any progress with the code. It's *possible*, yes, but it's not something I'd wish on anyone.


T

-- 
MSDOS = MicroSoft's Denial Of Service
December 29, 2017
On Friday, December 29, 2017 15:51:53 H. S. Teoh via Digitalmars-d wrote:
> On Fri, Dec 29, 2017 at 03:49:07PM -0700, Jonathan M Davis via Digitalmars-d wrote: [...]
>
> > It also depends on your personality. Personally, while I like builds being near instantaneous when dealing with unit tests, if they take a few seconds, it's not a big deal, but I know folks who get seriously annoyed if they can notice much of a difference time-wise between hitting enter to run the build command and hitting enter with no command. Either way though, having build times be more than a few seconds when dealing with unit tests tends to cause serious problems even though having a build for an entire project take a few minutes usually isn't a big deal unless you're really impatient. Having the delay between making changes and testing them be very long can _really_ slow down development, and so that's where build times really matter.
>
> [...]
>
> If you think a few minutes is bad... I've been on the receiving end of builds that take 30-45 *minutes* for 1-line code changes (*ahem*cough*C++ header files*cough*ahem*).  It put a serious damper on my ability to make any progress with the code. It's *possible*, yes, but it's not something I'd wish on anyone.

Oh, certainly. But my point is that once it starts taking minutes to be able to build and run the unit tests for what you're currently doing, it becomes infeasible to edit-test-edit-etc. in any kind of reasonable cycle. It doesn't have to get anywhere near the 30 minute mark for it to be a serious problem. So, of course, 30+ minutes is an utter disaster if you're looking to test stuff as you go.

- Jonathan M Davis

December 29, 2017
On Fri, Dec 29, 2017 at 05:52:36PM -0700, Jonathan M Davis via Digitalmars-d wrote: [...]
> Oh, certainly. But my point is that once it starts taking minutes to be able to build and run the unit tests for what you're currently doing, it becomes infeasible to edit-test-edit-etc. in any kind of reasonable cycle. It doesn't have to get anywhere near the 30 minute mark for it to be a serious problem. So, of course, 30+ minutes is an utter disaster if you're looking to test stuff as you go.
[...]

Yeah... recently I've been resorting to `dmd -unittest -main module.d` on single modules for faster turnaround time, because in spite of D's famed compilation speeds, some things still make it go far too slow, e.g. if you use template-heavy / CTFE-heavy code, or there are just too many files to compile.

This has also been encouraging me to write code in a far more modular fashion than even traditional modularists would do, in the sense that where possible, even dependent modules won't be imported, but instead left to a template parameter, to be instantiated by the main program with the dependent module. Contrived example:

-----
void printSomething(alias stdio = selfImport!"std.stdio", Args...)(string fmt, Args args)
{
	stdio.writefln(fmt, args);
}

unittest
{
	struct TestHarness {
		static void writefln() { /* fake implementation here */ }
	}
	printSomething!TestHarness("%s", "blah");
}
-----

The idea is that importing std.stdio won't even happen when compiling with -unittest -main. This approach lets you completely detach a module from its dependencies, so you never have to worry about what else it depends on when compiling with -unittest -main, and also conveniently allows you to substitute complicated types with cheaper substitutes that suffice for testing, e.g., instead of writing to stdout or actually causing side-effects visible outside the program, have a substitute module (or just struct!) that redirects output or filesystem changes to a log that's verified by the unittest to be the correct sequence of operations. The substitute type doesn't even have to implement the complete API of the module/object in question; just enough of it for the unittest to do what it needs to do.

Being able to substitute a struct in place of a module inside a unittest is one of the awesome things about D that make me love it so. :-D

And a side perk is that you'll get automatic attribute inference for all such functions, if the original function would otherwise be just a non-template function. (On that note, nowadays I'm tempted to just stick empty compile-time parameters on every function I can; it enables attribute inference and also reduces executable bloat if the function is never actually called from anywhere. The only bad thing is the compiler won't run semantic on potentially broken code... but then again, if you never use it, it doesn't matter, and if it ought to matter, there should be a unittest for it anyway.)


T

-- 
"You are a very disagreeable person." "NO."
January 02, 2018
On Saturday, 30 December 2017 at 01:09:59 UTC, H. S. Teoh wrote:
> On Fri, Dec 29, 2017 at 05:52:36PM -0700, Jonathan M Davis via Digitalmars-d wrote: [...]
>> [...]
> [...]
>
> Yeah... recently I've been resorting to `dmd -unittest -main module.d` on single modules for faster turnaround time, because in spite of D's famed compilation speeds, some things still make it go far too slow, e.g. if you use template-heavy / CTFE-heavy code, or there are just too many files to compile.
>
> [...]

This is pretty interesting. Unfortunately I can see it getting messy with multiple dependent modules and/or if the function has "real" template parameters on top. I have to think about how to use your strategy in my own code.

Atila
January 02, 2018
On Tue, Jan 02, 2018 at 04:00:11PM +0000, Atila Neves via Digitalmars-d wrote:
> On Saturday, 30 December 2017 at 01:09:59 UTC, H. S. Teoh wrote:
[...]
> > Yeah... recently I've been resorting to `dmd -unittest -main module.d` on single modules for faster turnaround time, because in spite of D's famed compilation speeds, some things still make it go far too slow, e.g. if you use template-heavy / CTFE-heavy code, or there are just too many files to compile.
> > 
> > [...]
> 
> This is pretty interesting. Unfortunately I can see it getting messy with multiple dependent modules and/or if the function has "real" template parameters on top. I have to think about how to use your strategy in my own code.
[...]

Yeah, when you have dependent modules it can get pretty messy.  Which is why I've been moving towards modularising my code beyond what people would usually do, i.e., turn even symbols in dependent modules into template parameters so that you can make the module completely independent.  Here's one actual code example I'm particularly proud of:

------
/**
 * Expands '@'-directives in a range of strings.
 *
 * Returns: A range of strings with lines that begin with '@' substituted with
 * the contents of the file named by the rest of the line.
 */
auto expandFileDirectives(File = std.stdio.File, R)(R args)
    if (isInputRange!R && is(ElementType!R : const(char)[]))
{
    import std.algorithm.iteration : joiner, map;
    import std.algorithm.searching : startsWith;
    import std.range : only;
    import std.range.interfaces : InputRange, inputRangeObject;
    import std.typecons : No;

    return args.map!(arg => arg.startsWith('@') ?
                    cast(InputRange!string) inputRangeObject(
                        File(arg[1 .. $]).byLineCopy(No.keepTerminator)) :
                    cast(InputRange!string) inputRangeObject(only(arg)))
               .joiner;
}

unittest
{
    static struct FakeFile
    {
        static string[][string] files;

        string[] contents;
        this(string filename) { contents = files[filename]; }
        auto byLineCopy(KeepTerminator dummy) { return contents; }
    }

    FakeFile.files["file1"] = [
        "x=1",
        "y=2"
    ];
    FakeFile.files["file2"] = [
        "z=3"
    ];

    import std.algorithm.comparison : equal;
    assert([
        "p=abc",
        "@file1",
        "q=def",
        "@file2",
        "e=ghi"
    ].expandFileDirectives!FakeFile.equal([
        "p=abc",
        "x=1",
        "y=2",
        "q=def",
        "z=3",
        "e=ghi"
    ]));
}
------

As you can see, I made File into a template parameter so that the unittest can simulate a virtual filesystem with a struct, without ever needing to deal with the messiness of actually creating and managing temporary files on a real filesystem. This is (mostly) invisible to user code (i.e., other code that uses this function) because if you don't specify what File should be, it defaults to the usual std.stdio.File.

Note also that not all of File's API needed to be implemented; since the function only uses .byLineCopy, that's all I needed to implement in my surrogate FakeFile struct.

Not all code can be cut up in this way, though.  I've only achieved this in a few low-level modules; for other modules I'm still facing the problem of having to specify dependent modules on the command-line. But in trying to attain to this goal, it really forces you to think about structuring your code in a better way than just throwing in haphazard dependencies that may not strictly be necessary.


T

-- 
Just because you can, doesn't mean you should.
January 02, 2018
On 01/02/2018 10:13 AM, H. S. Teoh wrote:

> As you can see, I made File into a template parameter so that the
> unittest can simulate a virtual filesystem with a struct

All I'm hearing is "blah blah blog." ;)

Ali

1 2 3
Next ›   Last »