> That would be great! If the per-package mode has such practical benefits, we could also simply make it the general default, for all generators that can support it. However, I just briefly looked at your benchmark results, the performance boost seems to be solely because of parallel compiler invocations. We already had issues with per-module parallel compilation and out-of-memory errors, so this optimization seems to be only suitable for certain projects or build machines. So at least with the current compiler memory requirements I'm not sure if this would be a good default build mode.

From mine and Andrei's results it seems to be a much better default.

> This is of course perfectly fine and reasonable (although I personally find the D syntax to be a bit too verbose for this task, but thats really just personal taste - oh, and JSON is awful, too ;). The problems just start to creep in as soon as public DUB packages start to depend on Reggae - then we'll possibly get another split in the ecosystem. I'd say we should generally try to focus on a single standard solution, however that looks and however the tool is split into executables or packages.

I'm working on the syntax ;)

>> As I've mentioned before, I know the kind of things I'd want to do with
>> the build system if I had a large and complicated enough project, and I
>> know I wouldn't be able to do it easily using dub alone. As I've also
>> mentioned before, building with dub is just fine for most people.
>
> Getting some of those use cases on a Wiki page or something would be great.

I'd have to go back to my old work project and read all the CMake code...

>> I don't know if fragmentation would be an issue. The packages are still
>> dub packages and I for one will use dub.json/sdl to list my dependencies
>> even if reggae is actually generating the build.
>
> Yeah that would definitely not be an issue. I just fear (maybe unnecessarily) that people might start to put packages in the registry that can *only* be built using Reggae (or some other build tool). At least for libraries that would really be bad.

I think that if a project is a dub package, then it needs to be able to be built with dub. If anything on code.dlang.org doesn't (correctly) build with `dub build` then that would be incredibly wrong.

Also, I'm not as sure as you that reggae will catch on that much :)

Atila

On 06/10/2015 08:11 AM, Sönke Ludwig wrote:
> Am 04.06.2015 um 17:47 schrieb Atila Neves:
>>
>> I don't know if fragmentation would be an issue. The packages are still
>> dub packages and I for one will use dub.json/sdl to list my dependencies
>> even if reggae is actually generating the build.
>
> Yeah that would definitely not be an issue. I just fear (maybe
> unnecessarily) that people might start to put packages in the registry
> that can *only* be built using Reggae (or some other build tool). At
> least for libraries that would really be bad.

I've found the opposite to be true.

We can *already* get packages in code.dlang.org that work with a custom buildscript, but then fail when doing "dub build". I know this because I've *already* done it by accident multiple times. And then I get bug reports of "X or Y build problem under dub".

But if we can easily tell dub via dub.json, "Hey, don't use your internal build system to build this, I've stuck the appropriate command in pre/postBuildCommands", then "dub build" will automatically just simply work. Major bonus compatibility points the more information we can pass from dub to the custom command (ex: "Which 'built type' did the user request with '--build='?" or...well, everything provided by dub describe --data).

Remember, libraries do NOT need to use the same build system to be used together. As long as there's a common interface to...

1. Trigger a build, and

2. Get the lists of import paths and output binaries (and maybe any other required flags),

...then everything's golden. This is the job of a package manager. Each library author tells the package manager:

1. "When someone asks you to build me, here's how you do it" and

2. "When someone asks you for my import paths and output binaries (etc.), this is what you tell them...".

For DUB, #1 above can be done via pre/postBuildCommands as long as we can tell it "Don't go trying to build this yourself". (I'm wondering if "targetType": "none" would help here.) And for #2, DUB can do that via my --data= pull request.

Ok, only one possible issue left: Source libraries (as opposed to binary libraries) *could* be an issue just because of the differences in collecting the list of files to be compiled. But that's solvable two ways:

1. If your lib *requires* something like RDMD-style dependency collecting, then the lib must be a binary lib.

Or much, much better yet: 2. Option in dub.json to say "when you collect this package's list of source files, do it RDMD-style". Sure, that may not be appropriate for ALL packages, but that's exactly why it'd be a per-package OPTION.

And there it is, the *existing* issues of splitting the dub ecosystem are thus eliminated, without creating new splits.

On 06/10/2015 09:06 AM, Atila Neves wrote:
>
> I think that if a project is a dub package, then it needs to be able to
> be built with dub. If anything on code.dlang.org doesn't (correctly)
> build with `dub build` then that would be incredibly wrong.
>

Right. And to summarize what I said elsewhere about that: "Building WITH dub" need NOT imply "building using dub's internal buildsystem". It only needs to mean "The CORRECT buildsystem can be trigged via 'dub build', whatever buildsystem that may be".

That's why the current situation is backwards and CAUSES splits rather than preventing them: We can always trigger dub's internal buildsystem via "dub build", but trying to get "dub build" to trigger the CORRECT one INSTEAD of built-in (whenever a different one is needed) is a world of hurt.

On 2015-06-10 14:11, Sönke Ludwig wrote:

> Getting some of those use cases on a Wiki page or something would be
> great. There have been some ideas to approach the procedural aspect from
> a different angle - using procedural DUB plugins that can be invoked
> from within the declarative build description. This would have the
> advantage that a lot of information can be gathered about a package
> without executing procedural code (with the possible accompanying
> security risks and performance issues or the efforts needed to mitigate
> those).

Orbit, the package manager I was working on, uses Ruby as the description language. It uses the more traditional way of uploading packages to the registry, basically a ZIP with with a special directory layout with every file necessary to build the package.

With this model it'a straight forward to "compile" the Ruby code to JSON or YAML when building the package locally (same thing can be done with D). On the server side, the registry only needs to read the JSON file to get the metadata of the package. When installing the package it would use the Ruby file again (or JSON description, never got that far), to be able to run pre and post actions. This part shouldn't be any less secure than the current pre/post commands.

-- 
/Jacob Carlborg