On 2015-06-07 17:27, Nick Sabalausky wrote:

> Even if that stuff isn't currently known to code.dlang.org, it'd all be
> easy for it to obtain:
>
> $ mkdir some_temp && cd some_temp
> $ git clone package_URL dir && cd dir
> $ dmd (flags dub already knows) -v | grep import
>
> Would only need to do that when it detects a new version tag (which it
> already detects).

So the registry basically needs to compile all packages? Isn't it better to "compile" and upload packages then?

-- 
/Jacob Carlborg

Am 07.06.2015 um 19:57 schrieb Jacob Carlborg:
> On 2015-06-07 17:27, Nick Sabalausky wrote:
>
>> Even if that stuff isn't currently known to code.dlang.org, it'd all be
>> easy for it to obtain:
>>
>> $ mkdir some_temp && cd some_temp
>> $ git clone package_URL dir && cd dir
>> $ dmd (flags dub already knows) -v | grep import
>>
>> Would only need to do that when it detects a new version tag (which it
>> already detects).
>
> So the registry basically needs to compile all packages? Isn't it better
> to "compile" and upload packages then?
>

Isn't just the list of *modules* needed and not the list of *imports*? The registry can easily query the list of files of a package using the GitHub API. The package description is obviously already known, too, so this would definitely be possible to do without compiling anything.

On 2015-06-08 08:04, Sönke Ludwig wrote:

> Isn't just the list of *modules* needed and not the list of *imports*?
> The registry can easily query the list of files of a package using the
> GitHub API. The package description is obviously already known, too, so
> this would definitely be possible to do without compiling anything.

There's still the issue with a module not having a matching filename.

-- 
/Jacob Carlborg

Am Mon, 1 Jun 2015 13:48:21 +1000
schrieb Manu via Digitalmars-d <digitalmars-d@puremagic.com>:

> Please declare a standard unix location for D 'includes'. Nobody
> agrees where in the filesystem D files should be.
> I use /usr/include/d2/ for my stuff […].

I would like to mention that 40 out of 62 ppl by the end of
2013 did agreed on /usr/include/dlang :
http://forum.dlang.org/thread/20131112205019.12585bbd@marco-leise

That's a few more than "nobody". Today I would include a "other" choice at least, but it wouldn't have changed the picture much.

-- 
Marco

On 9 June 2015 at 15:52, Marco Leise via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> Am Mon, 1 Jun 2015 13:48:21 +1000
> schrieb Manu via Digitalmars-d <digitalmars-d@puremagic.com>:
>
>> Please declare a standard unix location for D 'includes'. Nobody
>> agrees where in the filesystem D files should be.
>> I use /usr/include/d2/ for my stuff […].
>
> I would like to mention that 40 out of 62 ppl by the end of
> 2013 did agreed on /usr/include/dlang :
> http://forum.dlang.org/thread/20131112205019.12585bbd@marco-leise
>
> That's a few more than "nobody". Today I would include a "other" choice at least, but it wouldn't have changed the picture much.

There was no 'd2' option, which is the only one I've ever encountered. Is there a convention in use? Does anyone else bother with this? I just want someone to decide and publish it somewhere... I don't care what's chosen, but we really need to just agree on something.

I don't see how it matters as long as packaging is done properly. Developer should not ever be allowed to write to /usr anyway.

On Tuesday, 9 June 2015 at 14:09:03 UTC, Dicebot wrote:
> I don't see how it matters as long as packaging is done properly. Developer should not ever be allowed to write to /usr anyway.

+1

I don't expect to be able to rely on the location of anything under /usr except maybe /usr/bin/env and I don't see why D should be any different.

Am 08.06.2015 um 08:47 schrieb Jacob Carlborg:
> On 2015-06-08 08:04, Sönke Ludwig wrote:
>
>> Isn't just the list of *modules* needed and not the list of *imports*?
>> The registry can easily query the list of files of a package using the
>> GitHub API. The package description is obviously already known, too, so
>> this would definitely be possible to do without compiling anything.
>
> There's still the issue with a module not having a matching filename.
>

I think that will only work with a target type "sourceLibrary" anyway (since such modules always have to be on the compiler command line), so it will always be at least somewhat problematic. But there is already some code [1] to extract the module name for "dub test", so that would still be easy to do without actually invoking the compiler.

[1]: https://github.com/D-Programming-Language/dub/blob/b3ea9e9f026bd096c60b4d1de55e2ea62bd313cd/source/dub/dub.d#L777

Am 05.06.2015 um 11:56 schrieb "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm@gmx.net>":
> On Thursday, 4 June 2015 at 13:47:41 UTC, Sönke Ludwig wrote:
>> A branch *is* an exact version as far as DUB goes (just one that
>> changes over time).
>
> Well, that's the problem then.
>
>> But to make use of any commit relationships within a branch, it would
>> have to gain specific knowledge of GIT or other version management
>> systems.
>>
>> Adding support for commit hashes in dub.selections.json is still TBD,
>> but would be another alternative.
>
> I'd say it's the only consistent way.
>
>>
>> BTW, my primary concern with all of these things is retaining a
>> healthy public package eco system. I personally suspect that this is
>> much more important for the language as a whole rather than pleasing
>> every single taste. This doesn't mean that the latter isn't a
>> worthwhile goal, but to me the former has always been more important.
>> What is missing is a proposal how to solve both issues at the same time.
>
> I can understand that. But from what I've seen with rubygems, it is not
> a problem in practice. You're totally right that released packages
> shouldn't have such dependencies. A warning message in dub or a check on
> code.dlang.org is totally okay IMO.

Well, enforcing it on code.dlang.org would indeed be an option. I still think that using the version backed approach below, or VCS facilities is superior in general, though. There is not much objective reason to duplicate that functionality in the package manager AFAICS.

>> For example for conflict resolution. This is the foremost reason why
>> the *have* to be banned from dub.json. You can't really know if a
>> branch matches/is compatible with any particular version (range) or
>> another branch, so you have to either make an arbitrary guess (always
>> prefer the branch; always prefer the version; always prefer "master",
>> but a tagged version otherwise; etc.), or always issue a version
>> conflict error.
>
> Definitely always prefer the branch. The goal is to let the developer
> override the decisions made by upstream library authors.

That's what dub.selections.json is for! It applies only to the top level project, which can make such decisions. On the other hand, if you always prefer the branch during *dependency resolution*, some deeply buried dependency can simply break your build (or any upstream build) because it chose to use an incompatible branch based dependency.

>> The practical effect that was already very visible in the existing
>> ecosystem was that packages tended to separate into two clusters. One
>> cluster that used branch dependencies and another that used version
>> dependencies. Both incompatible with each other. It also meant that a
>> lot of people didn't bother to make any version tags at all, which is
>> understandable, but also a very bad influence to the stability of the
>> ecosystem. Semantic versioning, if used correctly, is a very important
>> factor in that regard.
>
> Yes, but that's a social problem. We can nudge people in the right
> direction by printing a warning message, or even by rejecting it
> completely on code.dlang.org.
>
>>
>> The compromise solution to allow branch based dependencies only in the
>> version selection file doesn't really have drawbacks, but dodges all
>> of those issues by moving the decision to the root of the dependency
>> tree.
>
> I does have obvious drawbacks: If dub.selections.json contains non-exact
> versions, different developers can get slightly different dependencies.
> dub.selections.json must fully determine the versions that are going to
> be used, otherwise it has failed its purpose.
>
>>
>>>
>>>>
>>>> You can also check out certain branches locally and use "dub
>>>> add-local" or "dub add-path" to let DUB use it instead of the publicly
>>>> registered one.
>>>
>>> This is not an option in a larger team. Everyone would have to do that
>>> manually, or you'd need to write a custom script to do what is actually
>>> the package manager's task.
>>
>> Using dub.selections.json should be the right tool then.
>
> As explained above, it isn't.

If you want commit granularity then that's a different thing. The intended way to support this is to still use version tags - DUB will then associate a branch with the latest matching version tag, so that normal version based dependencies can be used.

If that case applies, dub.selections.json will contain something like {"somedep": "1.0.0+commit.14.f23a56gb"}. Currently this information isn't used to drive GIT to fetch the right tag, but a ticket for that is open and this could be a viable middleground solution.

>> You could of course also use things like git submodules + path based
>> dependencies in this case, there is no need to force everything into
>> one schema (especially when it doesn't fit well, such as versions and
>> branches).
>
> Yes, and I'm sure I can come up with even more workarounds ;-)

It's not really a workaround. The package manager has a version based dependency handling logic that has the proper semantics for this kind of task. Trying to squeeze branches and commits into this isn't necessarily a good idea or even necessary at all. The idea here is to let you do such things and not get in the way, but why should the package manager duplicate functionality of the VCS when that can solve the task just fine? It adds additional and redundant complexity for no real benefit.

Am 04.06.2015 um 17:47 schrieb Atila Neves:
>> On a related note, it's also a pity that Reggae mixes incremental
>> build improvements (from which DUB itself could greatly profit, too -
>> just as a Ninja generator for DUB would be a nice feature) with a
>> separate, layered build description. I mean there is of course no
>> reason to not have alternative approaches for build descriptions
>> available in general, but when mixed with a public package repository,
>> it just leads to fragmentation.
>
> I can add a ninja generator to dub if you want, with the default being
> per-package compilation (since right now for dub it's one file or
> everything at once).

That would be great! If the per-package mode has such practical benefits, we could also simply make it the general default, for all generators that can support it. However, I just briefly looked at your benchmark results, the performance boost seems to be solely because of parallel compiler invocations. We already had issues with per-module parallel compilation and out-of-memory errors, so this optimization seems to be only suitable for certain projects or build machines. So at least with the current compiler memory requirements I'm not sure if this would be a good default build mode.

>
> But I wrote reggae because:
>
> 1. Declarative is preferable but imperative is needed. My (our?)
> favourite imperative language is D, so I want to specify builds in it.

This is of course perfectly fine and reasonable (although I personally find the D syntax to be a bit too verbose for this task, but thats really just personal taste - oh, and JSON is awful, too ;). The problems just start to creep in as soon as public DUB packages start to depend on Reggae - then we'll possibly get another split in the ecosystem. I'd say we should generally try to focus on a single standard solution, however that looks and however the tool is split into executables or packages.

> 2. I want a build tool that builds on dub but doesn't require it

Fair enough.

> 3. I wanted it to make it easy to link with C and C++

This really also needs to be solved within DUB (i.e. without resorting to preBuildCommands). That is going to be the next priority after getting the 1.0.0 features ready.

>
> As I've mentioned before, I know the kind of things I'd want to do with
> the build system if I had a large and complicated enough project, and I
> know I wouldn't be able to do it easily using dub alone. As I've also
> mentioned before, building with dub is just fine for most people.

Getting some of those use cases on a Wiki page or something would be great. There have been some ideas to approach the procedural aspect from a different angle - using procedural DUB plugins that can be invoked from within the declarative build description. This would have the advantage that a lot of information can be gathered about a package without executing procedural code (with the possible accompanying security risks and performance issues or the efforts needed to mitigate those).

>
> I don't know if fragmentation would be an issue. The packages are still
> dub packages and I for one will use dub.json/sdl to list my dependencies
> even if reggae is actually generating the build.

Yeah that would definitely not be an issue. I just fear (maybe unnecessarily) that people might start to put packages in the registry that can *only* be built using Reggae (or some other build tool). At least for libraries that would really be bad.