On Sunday, 14 June 2020 at 16:05:39 UTC, Andrei Alexandrescu wrote:
> What code wouldn't it work with?

It isn't so much code as it is some build paradigms.

If you are forced to do separate compilation (like if dmd is eating 30 GB of RAM...), you can't enjoy dmd -i just working. But you can still use it in individual pieces then link the result together normally, so it doesn't get in your way, it just doesn't help as much. If you do it wrong in these cases, you are liable to get duplicate symbol problems with a diamond dependency (two packages compiled separately that both include a common third package).

All manageable and -i can still help in these cases with appropriate inclusion/exclusion patterns, just -i's biggest strength is when you are doing an all-at-once build.

Ditto with non-module based dependency versions or compilation flags, since those again bring you back to separate compilation.

The other semi-tricky thing is dub's default source layout doesn't follow the convention -i expects due to having the extra source/ directory. So you need to add it with -Iproject/source each time. Or you can automatically rearrange files from a package upon downloading it (just read the `module` declaration and sort it into directories from that under a common `lib` top-level thing. This is actually what I do on my computer and why my arsd repo has most its files at top-level - if you clone the repo in your lib dir, the files are now right at `arsd/cgi.d` etc... which automatically works with the compiler. Zero extra config needed, it all just works. I get a lot of complaints about that layout but this is an objective advantage I don't think a lot of people really realize!)


I've been tempted before to write a competing package manager that does stuff like this all automatically. Imagine you just write some D and `import foo.bar;` then the compiler (or, more specifically, the helper program that works with the compiler) tells you "module foo.bar is not installed but can be found in package `footech`, install now? (y/n)". And if you made the major version part of the module names instead of git tags, that'd all magically just work too. I'd be kinda fun and very efficient thanks to dmd -i.

On Sunday, 14 June 2020 at 15:31:05 UTC, Adam D. Ruppe wrote:
> On Sunday, 14 June 2020 at 09:42:45 UTC, Walter Bright wrote:
>> I don't know what -i does exactly, or why it was added.
>
> -i is the most exciting thing that happened in the compiler from like 2015 up to 2018. Enormously useful in efficient, seamless builds, serious gamechanger in using D.
>
> Don't touch it!

+1000

That flags is the workhorse of all our builds in the company: don't touch it .... please

/Paolo

On Sunday, 14 June 2020 at 16:05:39 UTC, Andrei Alexandrescu wrote:
> On 6/14/20 11:36 AM, Adam D. Ruppe wrote:
>> On Sunday, 14 June 2020 at 10:24:33 UTC, Walter Bright wrote:
>>> It would probably be better as the implementer, as a header-only library has to be designed for it.
>> 
>> That's not true. -i just works with almost any D code. It is enormously useful exactly the way it is.
>
> What code wouldn't it work with? Would be good to document those. "How to write a header-only D library". Blog post!

The '-i' flag is the symmetric opposite of '-I'. '-I' expects either header-only libraries, or separate compilation, while '-i' expects neither. What Walter is suggesting (marking modules with 'pragma (root);' to make them header-only mainly serves the '-I' flag. The only advantage 'pragma (root);' may have to '-i' is a compile-time guarantee that modules intended to be header-only actually don't impose any link-time dependencies.

The main use case of the `-i` flag is to compile (not just import) modules. However the killer feature of '-i' is that while '-I' a is blunt tool that just adds all modules under a dir to the import path, '-i' leverages D's import system and lazily compiles only the modules that are imported starting from the root set of modules.

For example, giving the following project structure:

    project/
    ├── lib1
    │   ├── mod1.d
    │   ├── ...
    │   └── modn.d
    ├── lib2
    │   ├── mod1.d
    │   ├── ...
    │   └── modn.d
    └── main.d


Previously, in order to compile it, you could do the following:

    find lib1/ lib2/ -type f -name '*.d' | xargs dmd -betterC main.d

However, this has the disadvantage of compiling stuff that main.d doesn't need. Say main.d was this:

    extern (C) int main(int argc, const char** argv)
    {
        import lib1.mod1 : fun1a;
        return fun1a(argc, argc + 1);
    }

The only function that ever needs to be compiled is fun1a. However the command above will end up compiling the whole world (every module under lib1/).

With the '-i' flag we can compile main.d like so:

    dmd -betterC -i=lib1 -i=lib2 main.d

Now, set of modules that would be compiled equals precisely the set of modules imported from main.d - in this case only lib1/mod1.d.

The only deficiency currently is that it doesn't take advantage of selective imports - each imported module is compiled as a whole, even if you only need to compile the imported symbols. This problem is not just an implementation deficiency though, as whether something is compiled and whether it exists is somewhat conflated and has impact on D's meta programming (DbI and all that). I think the best way to address this issue is at the root - we need to define precise semantics over what lazy compilation means. I have been meaning to write a DIP that addresses this issue for a while, based on the idea of an @export attribute (orthogonal to public/protected/package/private access modifiers) and @export inference. You can see a sketch of what I have in mind in these posts:

https://forum.dlang.org/post/xbllqrpvflazfpowizwj@forum.dlang.org
https://forum.dlang.org/post/nvvgrdlucajshjngdjlj@forum.dlang.org

On Tuesday, 16 June 2020 at 07:06:16 UTC, Petar Kirov [ZombineDev] wrote:
>
> The '-i' flag is the symmetric opposite of '-I'. '-I' expects either header-only libraries, or separate compilation, while '-i' expects neither.

I meant:

The '-i' flag is the symmetric opposite of '-I'. '-I' requires either header-only libraries, or separate compilation, while '-i' *requires neither*.

On Sunday, 14 June 2020 at 15:31:05 UTC, Adam D. Ruppe wrote:
> -i is the most exciting thing that happened in the compiler from like 2015 up to 2018. Enormously useful in efficient, seamless builds, serious gamechanger in using D.
>
> Don't touch it!

+100500

On Monday, 15 June 2020 at 00:55:59 UTC, Adam D. Ruppe wrote:
>
> The other semi-tricky thing is dub's default source layout doesn't follow the convention -i expects due to having the extra source/ directory. So you need to add it with -Iproject/source each time. Or you can automatically rearrange files from a package upon downloading it (just read the `module` declaration and sort it into directories from that under a common `lib` top-level thing. This is actually what I do on my computer and why my arsd repo has most its files at top-level - if you clone the repo in your lib dir, the files are now right at `arsd/cgi.d` etc... which automatically works with the compiler. Zero extra config needed, it all just works. I get a lot of complaints about that layout but this is an objective advantage I don't think a lot of people really realize!)

On the other hand, you rely on the person checking out the repository with a specific directory name, which is IMO much worse than the dub approach. If I check it out as "arsd-v1.1.1", suddenly it breaks, unless I use two DMD flags to solve the situation (-I and -mv).
Additionally, many people want to separate their code and their diet files / config file / scripts / whatever and having the project's source file at the root make this impossible.
So yeah I'll take the dub approach over this any day of the week, I just would like dub to generate source/projectname/ and not just source/

On Tuesday, 16 June 2020 at 13:27:00 UTC, Mathias LANG wrote:
> If I check it out as "arsd-v1.1.1", suddenly it breaks, unless I use two DMD flags to solve the situation (-I and -mv).

Which is required in ALL cases by the dub approach, you just don't notice it as much because `dub build` sets all that based on the settings in the dub config file (or their conventional defaults).

But you lose something there: optional modules are compiled in by dub (unless you go through the config process to separate it), whereas with dmd -i they would not be processed at all. It is also impossible to mix versions of dependencies with the dub approach; you can rename that directory to -v1.1.1 but put in a -v1.2.1 next to it and you just have trouble (which is why dub resolves it down to just the one compatible version, or fails if it can't find one...)

Maybe one of these days I'll actually write my competing package manager and demo all this stuff (you wouldn't even have to change your layouts btw since it is easy to auto-scan packages for D modules and rearrange them as needed; my doc site does exactly this).

But since I don't use packages it isn't a big priority of mine.

On Wednesday, 17 June 2020 at 02:51:20 UTC, Adam D. Ruppe wrote:
> On Tuesday, 16 June 2020 at 13:27:00 UTC, Mathias LANG wrote:
>> If I check it out as "arsd-v1.1.1", suddenly it breaks, unless I use two DMD flags to solve the situation (-I and -mv).
>
> Which is required in ALL cases by the dub approach, you just don't notice it as much because `dub build` sets all that based on the settings in the dub config file (or their conventional defaults).

Not quite, only `-I` is required, not `-mv`, which is fair IMO, and expected from anyone familiar with C[++].

> But you lose something there: optional modules are compiled in by dub (unless you go through the config process to separate it), whereas with dmd -i they would not be processed at all.

I totally agree that the current situation on that regard is terrible, although that's more of a dub limitation than something inherent to the hierarchy.
`dub` has a concept of `mainSourceFile` and I hope one day it'll use this to drive compilation instead of selecting all files. I have a somewhat related bug report: https://github.com/dlang/dub/issues/1778

> It is also impossible to mix versions of dependencies with the dub approach; you can rename that directory to -v1.1.1 but put in a -v1.2.1 next to it and you just have trouble (which is why dub resolves it down to just the one compatible version, or fails if it can't find one...)

I don't quite see how that relates to the discussion. I was merely stating that relying on the external name of the directory was much worse than having the user pass `-Iproject_name/source` by command line.

Nonetheless, I don't see why you would ever want to mix two semver-incompatible versions of the same library. I get that it can happen due to transitive dependencies, but there's really no way to make it foolproof for non-trivial cases, and I think everyone's time and energy are best spent avoiding this situation than catering for it.

> Maybe one of these days I'll actually write my competing package manager and demo all this stuff (you wouldn't even have to change your layouts btw since it is easy to auto-scan packages for D modules and rearrange them as needed; my doc site does exactly this).


Contributions to `dub` are welcome, will be immediately available to users, and will benefit the community much more than yet another package manager (we have `dub`, `dud`, `reggae`, and probably a plethora of other tools).

On Wednesday, 17 June 2020 at 03:21:30 UTC, Mathias LANG wrote:
[...]
> Contributions to `dub` are welcome, will be immediately available to users, and will benefit the community much more than yet another package manager (we have `dub`, `dud`, `reggae`, and probably a plethora of other tools).
+1 !