On 2015-06-02 20:54, Dicebot wrote:

> You must have though of different "development dependencies". I don't
> mean tools and stuff like that but all sorts of small static libraries
> (in native compilation world). With scripting language this unavoidably
> propagates on the end user unless you merge all deps into your sources
> inline for distribution (which no one does).

Oh, you mean like that.

-- 
/Jacob Carlborg

On Tuesday, 2 June 2015 at 18:56:22 UTC, Dicebot wrote:
> On Tuesday, 2 June 2015 at 09:15:13 UTC, Atila Neves wrote:
>> https://github.com/atilaneves/reggae
>>
>> Atila
>
> If this works up to the expectation, I will only need to write a simple code.dlang.org package fetcher and will never need to use dub again :)

Destroy away!

If it doesn't work like you like, let me know.

Atila

2015-06-01 20:03 GMT+02:00 Johannes Pfau via Digitalmars-d < digitalmars-d@puremagic.com>:

>
> The main problem is we don't have ABI compatibility. This means we can't share the libraries between compilers. So we could make 'import foo' work but linking with -lfoo without manually adjusting linker paths is not possible.
>
> There are some solutions, all have drawbacks:
> * Have compiler specific directories for libraries
>   (/usr/lib/gdc/libvibed.a, /usr/lib/dmd/libvibed.a). Only works for
>   static libraries. Can only have one version of a library installed
> * have per-library directories. I guess dub uses this. Can have
>   multiple versions of the same library. Drawback: compiler can't know
>   the correct library path.[1]
> * Shared libraries should be installed in a common location (/usr/lib)
>   anyway. This is only possible if we have ABI compatibility[2].
>   Library versioning is limited to the standard C/C++ like versioning.
>
> I guess the main reason why we can't have a C/C++ like out of the box experience is ABI compatibility. There'll always be some quirks as long as we don't fix this.
>
>
Glad to see that post !

But in addition of the ABI compatibility between compilers, we would need to have ABI compatibility between releases of the same major version if we were to walk the C[++] route, and we do not want that, for various reason, the most obvious one being that you won't be able to add an attribute like @nogc / nothrow to a free function without doing a major release.

There is good thing about C way of doing thing, but there is also some bad. For example, you can't have multiple versions of the same library installed - you don't need it -, but D definitely needs that. That also leads to major releases ending up in a different package - think qt4 / qt5 -. So, while C use SemVer for ABI compatibility, we should use SemVer for source compatibility, or we'll end up with an unmanageable mess.

On 2 June 2015 at 04:03, Johannes Pfau via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> Am Mon, 1 Jun 2015 15:26:51 +1000
> schrieb Manu via Digitalmars-d <digitalmars-d@puremagic.com>:
>
>> So, DMD/LDC/GDC know where to look to find these packages?
> IIRC no.
>
>> What
>> happens if the package includes a binary lib?
>>
>> That that, I still want someone to declare an official path for D 'includes' in the *nix filesystem, so D lib packages have somewhere to install...
>
> GDC looks in /usr/include/d2. Dicebot changed that
> to /usr/include/d/gdc for archlinux and there are some reasons why this
> is (for now) necessary:
>
> Sharing D sources/headers between compilers is possible, but there are some headers (druntime,phobos) which can be compiler specific. As long as these are not in a common import path it's not a problem though.
>
> The main problem is we don't have ABI compatibility. This means we can't share the libraries between compilers. So we could make 'import foo' work but linking with -lfoo without manually adjusting linker paths is not possible.
>
> There are some solutions, all have drawbacks:
> * Have compiler specific directories for libraries
>   (/usr/lib/gdc/libvibed.a, /usr/lib/dmd/libvibed.a). Only works for
>   static libraries. Can only have one version of a library installed
> * have per-library directories. I guess dub uses this. Can have
>   multiple versions of the same library. Drawback: compiler can't know
>   the correct library path.[1]
> * Shared libraries should be installed in a common location (/usr/lib)
>   anyway. This is only possible if we have ABI compatibility[2].
>   Library versioning is limited to the standard C/C++ like versioning.
>
> I guess the main reason why we can't have a C/C++ like out of the box experience is ABI compatibility. There'll always be some quirks as long as we don't fix this.

I'm inclined to suggest, this would be an excellent focus for 2.068... The compilers need to become aligned, and ABI issues seems like a critical issue to focus on.

On 2015-06-02 16:52, Steven Schveighoffer wrote:

> Actually, I don't. dmd *.d generally works fine.

I if it works for you, great. It doesn't work for me. And that shell globbing doesn't work on Windows.

-- 
/Jacob Carlborg

On 2015-06-02 17:01, Nick Sabalausky wrote:

> Related to this, does anyone happen to recall why rdmd uses "dmd -v" to
> get dependencies instead of "dmd -deps"? IIRC, it used to use -deps back
> at one time.

I would assumed it uses "-v" because it existed before "-deps"? I guess "-deps" was explicitly create because of the rdmd use case.

-- 
/Jacob Carlborg

On 2015-06-02 10:54, ketmar wrote:

> that was the thing i once proposed. see, we have a powerful scripting
> language inside DMD: D! yet we never used all it's power to do something
> really exciting -- like, for example, preparing command lines for
> external package fetching tool and parsing the answers. instead of
> providing a simple module for that, it all goes to be hardcoded.
>
> i extended CTFE engine with simple file i/o functions and "system" call
> with config-defined executables (something like sudoers list) just to see
> if it will be usable. and it's surprisingly fun even with all the limits
> and without hooks for module imports and so on.
>
> this system can be extended to allow writing arbitrary subcommands (like
> git). just import the corresponding subcommand module, if any, and CTFE
> it's invocation point. bingo! the system that can be extended without
> recompiling the compiler. and user can add subcommands on per-project
> base!
>
> sadly, this seems to get no "wow!"s. :-(

I think it's pretty cool. But I like the approach SDC has taken even more. Using the LLVM JIT to run CTFE code.

-- 
/Jacob Carlborg

On Wed, 03 Jun 2015 13:45:06 +0200, Jacob Carlborg wrote:

> On 2015-06-02 10:54, ketmar wrote:
> 
>> that was the thing i once proposed. see, we have a powerful scripting language inside DMD: D! yet we never used all it's power to do something really exciting -- like, for example, preparing command lines for external package fetching tool and parsing the answers. instead of providing a simple module for that, it all goes to be hardcoded.
>>
>> i extended CTFE engine with simple file i/o functions and "system" call with config-defined executables (something like sudoers list) just to see if it will be usable. and it's surprisingly fun even with all the limits and without hooks for module imports and so on.
>>
>> this system can be extended to allow writing arbitrary subcommands (like git). just import the corresponding subcommand module, if any, and CTFE it's invocation point. bingo! the system that can be extended without recompiling the compiler. and user can add subcommands on per-project base!
>>
>> sadly, this seems to get no "wow!"s. :-(
> 
> I think it's pretty cool. But I like the approach SDC has taken even more. Using the LLVM JIT to run CTFE code.

so SDC will benefit from JITting, as it's not required to modify CTFE evaluator in any way to implement my idea. just adding some more "internal functions" (like `import()`), which are nothing more than "call `interpret`, use results). porting that entry points to SDC will be trivial.

On Wednesday, 3 June 2015 at 01:23:37 UTC, Manu wrote:
> I'm inclined to suggest, this would be an excellent focus for 2.068...
> The compilers need to become aligned, and ABI issues seems like a
> critical issue to focus on.

There are much bigger fish to fry at this point. Phobos breaks ABI compatibility with every single release anyway.

 - David

Am 01.06.2015 um 16:49 schrieb Jonathan M Davis:
> On Monday, 1 June 2015 at 06:20:18 UTC, Nick Sabalausky wrote:
>> Ditto. Dub's great if you let it be your buildsystem, but as soon as
>> you want to use it as a package-manager-only it becomes an uphill
>> battle with dub fighting back every step of the way (I speak from
>> experience). That problem is worse if your project is a library that
>> you want fetchable through dub.
>
> And this is when Jacob Carlborg chimes in and says "I told you so." ;)
> His favorite complaint about dub has always been that it combined
> package management and the build tool into one.

This is actually completely irrelevant to the issue. Making the build part separate would not at all solve this. Any work on such a separate build tool (e.g. to support other languages) could instead as well go into DUB itself. Of course you could make the system completely build tool agnostic, but then you'd basically lose interoperability between packages, as each package might choose its own build tool.

>
> dub works fantastically if you want to do everything in the standard way
> without any funny stuff, but it clearly doesn't have the power of a tool
> like make, cmake, etc. As soon as you need to do anything funny - like
> include anything related to other languages in your build, or put things
> in a specific layout because of some company-specific thing, or anything
> that wasn't explicitly planned for by the folks writing dub, dub just
> won't work.

This is really not true. There is currently just one limitation w.r.t. the directory structure and that is the naming scheme of the folder to which packages are downloaded ("somepackage-1.0.0"). There are some packages that put their sources into the root of the repository and expect to be put into a folder with the name of the repository, which will then be added as an import folder. We are going to solve that.

Everything else can be done with the sourcePaths, sourceFiles, importPaths, ... settings. You basically have the same possibilities as you have when directly invoking the compiler.

Separate language support is still missing, but you always have the possibility to add generic "preBuildCommands" or "preGenerateCommands" to invoke separate tools or scripts. Of course it's desirable to get everything done with the same tool instead of relying on additional build time dependencies, but it's still *possible* to do these things.

>
> Now, to be fair, I don't know how you can be pulling in all kinds of
> stray libraries with who-knows-what for their build systems and expect
> it to work very well via a single tool (at minimum, you have the problem
> that the machine that you're trying to build them on will likely be
> missing some of those tools), but that's pretty much what we need to
> handle the general case. Without that, dub will work great for the
> normal case (which _will_ be plenty for many, many projects), but it
> won't work in the general case - and corporate folks in particular are
> probably going to have to skip out on using it because of abnormal
> requirements on their part.
>
> Even simply splitting dub out so that it can pull in packages without
> necessarily being able to build them would be nice (maybe adding a flag
> to the dub.json file indicating whether dub can be used as a build tool
> for the project or whether it's supposed to just act as a package
> manager for it), though that does add the problem of code.dlang.org
> becoming more confusing, since you wouldn't be able to rely on all
> projects on it being built for you by dub as part of the process of
> grabbing packages.

I guess using the proper targetType and preBuildCommand settings it should already be possible to invoke a separate build tool and still integrate the generated object/library files into the rest of the DUB build process. But I'd have to try that out - it would definitely be good to have something like this added to the cookbook [1].

Of course things like these always potentially degrade interoperability. DUB's build description was chosen explicitly to avoid unnecessary dependencies on a particular compiler/platform, which is where the most general build tools (especially "make") utterly fail.

>
> It's a complicate problem, and dub went a route which works in the 90%
> case, but doesn't work for more complicated cases, which will make dub
> unusable for some projects - especially corporate ones. I don't know how
> fixable it is without a major redesign of dub.

I don't see anything that wouldn't be fixable. In fact, there are really few things that are not possible now, even if not in the most comfortable way.