On Monday, 1 June 2015 at 11:14:43 UTC, Dicebot wrote:
>
> The way I do it right now is simply adding -I/usr/include/dlang -I/usr/include/dlang/<compiler> to dmd.conf / ldc.conf / patch GDC in a similar way.
>
> And surely if dub would ever attempt to write anything in /usr on its own I will simply throw it away from Arch packages.

This whole conversation reminds me a lot of the perl ecosystem.  How does Arch deal with CPAN?

-Wyatt

On Monday, 1 June 2015 at 15:25:13 UTC, Wyatt wrote:
> On Monday, 1 June 2015 at 11:14:43 UTC, Dicebot wrote:
>>
>> The way I do it right now is simply adding -I/usr/include/dlang -I/usr/include/dlang/<compiler> to dmd.conf / ldc.conf / patch GDC in a similar way.
>>
>> And surely if dub would ever attempt to write anything in /usr on its own I will simply throw it away from Arch packages.
>
> This whole conversation reminds me a lot of the perl ecosystem.
>  How does Arch deal with CPAN?
>
> -Wyatt

Same as pip and gems and all similar kind of crap. Bad. That is why I don't use them.

Though it is hard to invent a package management for interpreted language that is not insane. It is much better with native ones because runtime dependencies are separated from development dependencies.

On 06/01/2015 10:49 AM, Jonathan M Davis wrote:
>
> Even simply splitting dub out so that it can pull in packages without
> necessarily being able to build them would be nice (maybe adding a flag
> to the dub.json file indicating whether dub can be used as a build tool
> for the project or whether it's supposed to just act as a package
> manager for it), though that does add the problem of code.dlang.org
> becoming more confusing, since you wouldn't be able to rely on all
> projects on it being built for you by dub as part of the process of
> grabbing packages.
>

I've brought up something like that, but it was met with very strong opposition. :(

> It's a complicate problem, and dub went a route which works in the 90%
> case, but doesn't work for more complicated cases, which will make dub
> unusable for some projects - especially corporate ones. I don't know how
> fixable it is without a major redesign of dub.

Personally, I think "90%" is generous here, but I admit that's splitting hairs.

Unfortunately, "less than 100%" is a big problem in this case because those libs/projects that don't fit the "90%" effectively cannot even participate in the dub repository for others to use (at the very LEAST, they get pounded with "fails to build via dub"). So NOT using dub to build is freaking viral.

And *using* certain dub-based projects inside one which doesn't build via dub can be a major pain, as anyone who's tried to use vibe.d from a non-dub-based project should be able to attest (just *try* to figure out all the -I... and linker flags you need *and* double-check/maintain them every time you upgrade vibe.d).

So anything less than 100% effectively splits the whole damn ecosystem. NOT acceptable.

Dub's project generation feature is NOT a solution here, because you still have to manage builds dub's way, *exactly* as if you were using dub's build system, because...really...you still ARE. So all the dealbreakers for those 10+% projects still apply equally.

On the plus side though, I *do* think this is basically solvable. The most immediate way (and what I'm working towards) is to create a dub.json that feeds dub a 'dummy-src/dummy.d' to keep dub's build system happy and safely away from the real source files, and then use:

"(pre|post)-build-command-posix": "./my-real-buildscript"
"(pre|post)-build-command-win": "my-real-buildscript"

To trick dub into using a different build tool.

It'd be easily to design/publish a skeleton dub.json to do exactly that. (Kinda sucks to NEED that, but at least it'll work, and once it's in place them maybe it can be made easier. The non-buildsytem parts of dub.json really ARE very trivial, and if someone had a problem with even that trivial little bit, then I don't see how they could develop any useful software in the first place.)

I'm fairly certain the ONLY real thing still preventing all that above from actually working is that dub currently provides no (practical) way to retrieve the necessary compiler/linker args in order to actually use your dependencies (again, try using vibe.d in, say, an rdmd-based project). I say no "practical" way here because "dub describe -> parse the JSON -> extract the needed info -> convert the needed info to compiler flags" is NOT exactly practical or reasonable to expect people to do *just* to use RDMD or otherwise opt-out of dub's build-system.)

Again though, I'm working on that:

https://github.com/D-Programming-Language/dub/pull/572

That lets you directly grab any build info you need from dub. My next PR will then let you ask dub to automatically format the output for your compiler's CLI. And then one last one to provide convenience switches for the most likely configurations (like "--just-gimme-whatever-flags-I-need-to-pass-to-rdmd-so-I-can-actually-use-my-dependencies", but obviously less verbose :))

With all that, any project should be able to participate in the dub ecosystem AND opt-out of dub's build system WITHOUT segregating the ecosystem in any way.

On 06/01/2015 11:16 AM, Steven Schveighoffer wrote:
> On 5/31/15 7:01 PM, Andrei Alexandrescu wrote:
>> Let's make this part of 2.068:
>>
>> https://issues.dlang.org/show_bug.cgi?id=14636
>>
>> It's preapproved. Who would want to work on it?
>
> Just a reminder to everyone that at one point, we wanted the compiler to
> do kind of what 'dub' does -- fetch packages as dependencies for a
> project. Remember this DIP? http://wiki.dlang.org/DIP11
>

That sort of did what dub does, but badly. Among other things, it directly encouraged developers to hardcode their exact dependencies. It would have been a mess. Dub's package management side already handles that DIP's goals WAY better. The *only* issue now is trying to opt-out of dub's buildsystem side without causing problems (and I absolutely believe that to be easily solvable, see my other posts).

There is one part of that old proposal that I have come to like though, and wouldn't mind seeing: There was an idea to have DMD automatically invoke an external tool

> The problem I have with dub is it makes you as a consumer of libraries
> live within its realm. You have to make a dub file to use it, and that's
> not a small burden.
>

The dub.json file is an absolutely trivial "burden". It's a really, REALLY simple little file. Or at least, it should be...

The ONLY real burden comes when trying to teach dub how to build your project using dub's own buildsystem. If your project doesn't do everything "the dub way" then it certainly can be a bother. And I'm pushing very hard to relieve projects of that requirement.

But really, the following is NOT a burden unless extreme lazyness is in play:

{
    "name": "my-project",
    "dependencies": {
        "libblah": "~>1.7.0",
        "libfoobar": "~>2.1.0"
    }
}

Yes, a very, very, *VERY* tiny "burden".

And then to also get dub's buildsystem out of the way:

$ mkdir dummy-src-dir
# If lib:
$ touch dummy-src-dir/dummy.d
# Or if app:
$ echo "void main() {}" > dummy-src-dir/dummy.d

{
    "name": "my-project",
    "dependencies": {
        "libblah": "~>1.7.0",
        "libfoobar": "~>2.1.0"
    },

    "sourcePaths": "dummy-src-dir/",
    "pre-build-command-posix": "./buildscript",
    "pre-build-command-win": "buildscript"
}

Done. Trivial. No real burden. (Well, aside from JSON's irritating lack of comments or trailing commas.)

> I'm fairly certain the ONLY real thing still preventing all that above from actually working is that dub currently provides no (practical) way to retrieve the necessary compiler/linker args in order to actually use your dependencies (again, try using vibe.d in, say, an rdmd-based project). I say no "practical" way here because "dub describe -> parse the JSON -> extract the needed info -> convert the needed info to compiler flags" is NOT exactly practical or reasonable to expect people to do *just* to use RDMD or otherwise opt-out of dub's build-system.)

My build system already uses the output from `dub describe` to produce, well, a build system from it. I designed it with multiple backends in mind, right now the ones that I implemented are ninja and make.

If not for `dub describe` bugs (I'm working on that), you'd be able to point it at any dub project and get make/ninja files for it. Or integrate whatever you want with the existing files to compile that dub already knows about.

In fact, for any dub project that doesn't use configurations (that's the bug), you can do that _right now_.

Atila

Am Mon, 1 Jun 2015 15:26:51 +1000
schrieb Manu via Digitalmars-d <digitalmars-d@puremagic.com>:

> So, DMD/LDC/GDC know where to look to find these packages?
IIRC no.

> What
> happens if the package includes a binary lib?
> 
> That that, I still want someone to declare an official path for D 'includes' in the *nix filesystem, so D lib packages have somewhere to install...

GDC looks in /usr/include/d2. Dicebot changed that
to /usr/include/d/gdc for archlinux and there are some reasons why this
is (for now) necessary:

Sharing D sources/headers between compilers is possible, but there are some headers (druntime,phobos) which can be compiler specific. As long as these are not in a common import path it's not a problem though.

The main problem is we don't have ABI compatibility. This means we can't share the libraries between compilers. So we could make 'import foo' work but linking with -lfoo without manually adjusting linker paths is not possible.

There are some solutions, all have drawbacks:
* Have compiler specific directories for libraries
  (/usr/lib/gdc/libvibed.a, /usr/lib/dmd/libvibed.a). Only works for
  static libraries. Can only have one version of a library installed
* have per-library directories. I guess dub uses this. Can have
  multiple versions of the same library. Drawback: compiler can't know
  the correct library path.[1]
* Shared libraries should be installed in a common location (/usr/lib)
  anyway. This is only possible if we have ABI compatibility[2].
  Library versioning is limited to the standard C/C++ like versioning.

I guess the main reason why we can't have a C/C++ like out of the box experience is ABI compatibility. There'll always be some quirks as long as we don't fix this.


[1] you could actually build the directory structure logic into the compilers. Do something like dmd --library=vibe.d:0.7.23 where this argument sets up correct import paths and linking. I think .NET uses something similar (GAC). With ABI compatible compilers we might even make this work for shared libraries. Could be implemented as a small tool calling the compiler. You could also do something like this:

Create a package format: .dlib (simply a renamed .tar.xz)
Contents:
   - PACKAGEINFO //meta information (version)
   - include/*   //include files
   - doc/*       //documentation in standard format for IDEs
   - lib/dmd/libfoo.a
   - lib/gdc/libfoo.a

Write a small tool xdmd. Implement:
   //simply move to /usr/share/dlib/libfoo/version/
   //or ~/.share/dlib/libfoo/version/
   xdmd --install libfoo.dlib
   Note: when installing using a systems package manager, packages
   could be installed in the same way: pacman -S libfoo could call xdmd
   --install.

   //extract /usr/share/dlib/libfoo/0.9.3/libfoo.dlib
   //to temporary directory. Pass link and include flags and
   //rest of arguments to dmd
   xdmd --library=foo:0.9.3 test.d
   //directly use local package
   xdmd --library=libfoo-0.9.3.dlib test.d

This is essentially are very slimmed down dub. Would be a nice weekend exercise to write such a tool.

[2] or you use one compiler exclusively.

On 06/01/2015 01:43 PM, Atila Neves wrote:
>> I'm fairly certain the ONLY real thing still preventing all that above
>> from actually working is that dub currently provides no (practical)
>> way to retrieve the necessary compiler/linker args in order to
>> actually use your dependencies (again, try using vibe.d in, say, an
>> rdmd-based project). I say no "practical" way here because "dub
>> describe -> parse the JSON -> extract the needed info -> convert the
>> needed info to compiler flags" is NOT exactly practical or reasonable
>> to expect people to do *just* to use RDMD or otherwise opt-out of
>> dub's build-system.)
>
> My build system already uses the output from `dub describe` to produce,
> well, a build system from it. I designed it with multiple backends in
> mind, right now the ones that I implemented are ninja and make.
>
> If not for `dub describe` bugs (I'm working on that), you'd be able to
> point it at any dub project and get make/ninja files for it. Or
> integrate whatever you want with the existing files to compile that dub
> already knows about.
>
> In fact, for any dub project that doesn't use configurations (that's the
> bug), you can do that _right now_.
>
> Atila

That's certainly cool. I *do* think "dub describe"'s json is an awesome thing to have, exactly for those sorts of purposes.

But I do maintain that an ability to easily opt-out of dub's buildsystem *needs* to be built-in and should not *require* external tools (though external tools should certainly still be possible). And this is even more crucial if dub's to be bundled with dmd and rdmd (the latter of which dub currently does not play nicely with).

On 06/01/2015 08:34 AM, extrawurst wrote:
> On Monday, 1 June 2015 at 12:24:18 UTC, Atila Neves wrote:
>>>> dub as a build tool sux. no need to discuss that, it simply sux.
>>>
>>> why do you think it sucks ?
>>
>> dub is a good package manager, but it's not a good build system. For
>> small projects that just want to list dependencies and have it build,
>> it's sufficient. Any larger project will have needs that won't be
>> easily solved by using dub. Which is why I wrote my own.
>>
>> Atila
>
> Hi Atila,
>
> you are talking about cross language builds ?
> Other than that limitation, what do you miss ?

Unlike rdmd, dub always tries to compile all sources regardless of whether they're actually imported. This has been a constant source of problems for me, including breakage of conditional importing under various circumstances.

> But I do maintain that an ability to easily opt-out of dub's buildsystem *needs* to be built-in and should not *require* external tools (though external tools should certainly still be possible). And this is even more crucial if dub's to be bundled with dmd and rdmd (the latter of which dub currently does not play nicely with).

Maybe I can manage to get my code into dub, I don't know. I think it's extremely useful to separate concerns, which is why I started with something new instead of trying to add code to dub. Like I said before, for most people its build system is enough. When it's not, there's reggae (my tool). I hope, anyway.

The other reason is that I don't want to depend on dub. Maybe I have my priorities wrong, who knows.

Atila

On 6/1/15 1:36 PM, Nick Sabalausky wrote:
> On 06/01/2015 11:16 AM, Steven Schveighoffer wrote:
>> On 5/31/15 7:01 PM, Andrei Alexandrescu wrote:
>>> Let's make this part of 2.068:
>>>
>>> https://issues.dlang.org/show_bug.cgi?id=14636
>>>
>>> It's preapproved. Who would want to work on it?
>>
>> Just a reminder to everyone that at one point, we wanted the compiler to
>> do kind of what 'dub' does -- fetch packages as dependencies for a
>> project. Remember this DIP? http://wiki.dlang.org/DIP11
>>
>
> That sort of did what dub does, but badly. Among other things, it
> directly encouraged developers to hardcode their exact dependencies. It
> would have been a mess. Dub's package management side already handles
> that DIP's goals WAY better. The *only* issue now is trying to opt-out
> of dub's buildsystem side without causing problems (and I absolutely
> believe that to be easily solvable, see my other posts).

I'm not defending DIP11, just that the fact that it could be done without requiring an extra "dependencies" file. One thing I absolutely LOVE about git, is that when I do git status, if it sees a not-just-cloned repository, it gives me all the commands I would need to run to do all the various things. I rarely have to go to the git manual. I'd love to see something like the following:

> dmd x.d
Error: Your import of some.lib.module didn't work. Running dub --with-some-cool-arg some.lib.module would fetch the correct module from code.dlang.org
> dub --with-some-cool-arg some.lib.module
Dub : got it, fetched.
> dmd x.d
> echo 'yay, it worked! Too bad this is a fantasy :('

>> The problem I have with dub is it makes you as a consumer of libraries
>> live within its realm. You have to make a dub file to use it, and that's
>> not a small burden.
>>
>
> The dub.json file is an absolutely trivial "burden". It's a really,
> REALLY simple little file. Or at least, it should be...

Sorry, I suppose it's a small burden if you use dub frequently. I don't. Look what happens when I initialize a dub project:

{
	"name": "testdub",
	"description": "A minimal D application.",
	"copyright": "Copyright © 2015, steves",
	"authors": ["steves"],
	"dependencies": {
	}
}

Cool, I can figure out how to massage all the existing strings, except, hm... dependencies. What goes in there?

Now, it's time to get out the dub manual. annoying. Where is the comment that says "put your dependencies in here, they should look like this: ..."???

And dub uses versions. How does that work? Again no comment.

And where does the e.g. "name" get stuck into the executable? Oh, it doesn't. Why is it generated? I don't freaking know (except that it was the directory I was in when running dub init).

I'm sure dub is great for people who want to learn dub. For those who don't, it should work in a minimal way, and ESPECIALLY if we are to include it with the distribution.

-Steve