On Monday, 2 February 2015 at 05:23:52 UTC, Daniel Murphy wrote:
> "Vladimir Panteleev"  wrote in message news:viqwfixznbdbdwvhavuk@forum.dlang.org...
>
>> I don't use Dub
>
> You really should!  I put it off for months and months but I'm quite happy with it now.

Even if I had faith that dub was a perfectly polished piece of software, it doesn't solve any problems I have with building D programs, and in fact would make said task more complicated. Here's why.

1. rdmd

rdmd is a simple and effective way to build D programs, and I'm sad to see its use declining.

rdmd leverages two things: D's module system, and the compiler's import search path. Dub's design seems to ignore both of the above.

1a. rdmd and D's module system:

When you run `dmd -o- program.d`, the compiler will automatically read all modules imported by your program, and their imports, and so on. It does so by searching the filesystem across its search path for matches which correspond with D's module system, and only reads those files that are needed.

rdmd leverages this by collecting (and caching) the list of modules used in the program, and passing that list to the compiler. The compiler will then compile no more and no less than the exact set of modules transitively imported by the program.

In contrast, Dub's default modus operandi is to blindly send to the compiler all *.d files found in the "src" folder, whether they're actually used or not. Not only can this be slower if not all modules are always used, but it also won't work if the source code contains multiple entry points, forcing you to write complicated configuration files (i.e. do the computer's work for it).

1b. rdmd and D's search path

rdmd does not have any additional parameters to set up for where it needs to look for source files, because it relies on the compiler's search mechanism. Thus, if you can build your program with rdmd, "dmd -o- program" will succeed, and usually vice versa.

In contrast, Dub builds its own search path using its JSON configuration files, and has no equivalent of "dmd -o-".

There is no simple way to syntax-check just one file in a project when using Dub. I rely on this a lot in my workflow - I configured a syntax-check key in my editor, which I use almost as often as I save. A syntax check (dmd -o-) is much faster than a full build, as it skips parsing other parts of the project, code generation, and linking.

2. git

I've found that git's submodule feature is an excellent way to manage dependencies across D libraries, and fits really well with D's package system. For clarity to readers not too familiar with Git, I'll explain by example:

2a. Directory structure

If you have a library "fruit" with the modules "fruit.apple" and "fruit.banana", create a git repository (in a directory called "fruit") with the files apple.d and banana.d.

Now, in your project (e.g. "orchard"), add a symlink called "fruit" pointing to the repository:

~/
|- fruit/
|  |- .git/
|  |- apple.d
|  '- banana.d
'- orchard/
   |- .git/
   |- fruit -> ~/fruit/
   '- tree.d

tree.d can now import the "fruit.apple" module without any additional compiler switches. `rdmd tree` will just work. This is because the modules of the "fruit" library are placed at the repository root, and the repository's directory name matches with the library's package name, so when the compiler will look for the "fruit.apple" module, it will find it at "fruit/apple.d".

2b. Distribution

Upload the "fruit" repository to GitHub. In your "orchard" project, delete the symlink, and run:

git submodule add https://github.com/You/fruit fruit

This will clone the repository from GitHub, create a .gitmodules file, and add an entry to the git index. If you commit all these, and put "orchard" on GitHub too, someone can run the command:

git clone --recursive https://github.com/You/orchard

... to get the orchard project, and its dependencies (the fruit library).

2c. Versioning

Git stores the exact commit of each submodule in the parent repository.

So, if you make a commit in orchard/fruit/, and run `git status` in orchard/, git will show you that the "fruit" submodule has been modified.

Most importantly, this means that any breaking change in the "fruit" library will never affect existing projects which use it, since they will continue to use the registered commit which was known to work.

This gives you dependency versioning with commit granularity - you can hardly ask for something better - all without messing with configuration files.

---

The one thing I wish git had is symlink support for Windows. Windows has symlinks and directory junctions, but msysgit still doesn't support them.

For a practical instance of the above example, see Digger:

https://github.com/CyberShadow/Digger

Note the simple build instructions:

https://github.com/CyberShadow/Digger#building

2d. Git vs. Dub

Unfortunately, the above-described approach is not compatible with Dub:

- Dub packages are generally expected to have their source code in a "src" subdirectory, although you can set the source directory to "." in the configuration file.

- When cloning repositories, dub does not preserve the repository's directory name (so e.g. fruit will be cloned to ~/.dub/fruit-1.0.0/).

Somebody has created a Dub package for my library (excluding certain packages, due to point 1a above), and the other day someone complained that it doesn't work with Dscanner, because of the above issue - the module path "ae.dir.module" does not correspond to the filesystem path "ae-1.0.1/dir/module.d".

So, in order to start using Dub, I'd need to:

- restructure the directory layout of my library (breaking change)
- update all projects which use this library to use Dub instead
- give up quick syntax checking
- give up commit-granularity versioning
- begin maintaining JSON configuration files
- begin versioning libraries by hand
- install Dub on all my computers, servers, and virtual machines

No thanks.

I could invest time in improving Dub to fix or ameliorate some of the above points, but I don't see a compelling reason to. In fact, I think we should integrate rdmd into dmd - dmd clearly already knows which source files participate in compilation, as all rdmd does is basically take dmd's output and feed it back to it. This will greatly speed up compilation, too.

Change my view.

On Monday, 2 February 2015 at 08:09:39 UTC, Vladimir Panteleev wrote:
> 1a. rdmd and D's module system:
>
> When you run `dmd -o- program.d`, the compiler will automatically read all modules imported by your program, and their imports, and so on. It does so by searching the filesystem across its search path for matches which correspond with D's module system, and only reads those files that are needed.
>
> rdmd leverages this by collecting (and caching) the list of modules used in the program, and passing that list to the compiler. The compiler will then compile no more and no less than the exact set of modules transitively imported by the program.
>
> In contrast, Dub's default modus operandi is to blindly send to the compiler all *.d files found in the "src" folder, whether they're actually used or not. Not only can this be slower if not all modules are always used, but it also won't work if the source code contains multiple entry points, forcing you to write complicated configuration files (i.e. do the computer's work for it).

And this is reason why rdmd suck (in my case).
Example:

return cast(EntityManagerFactory)Object.factory("cz.dlang.ddpa.EntityManagerFactoryImpl"); // this does not work with rdmd but works ok with dub

On Monday, 2 February 2015 at 08:46:40 UTC, Daniel Kozak wrote:
> And this is reason why rdmd suck (in my case).
> Example:
>
> return cast(EntityManagerFactory)Object.factory("cz.dlang.ddpa.EntityManagerFactoryImpl");

Object.factory is a misfeature. It causes ALL classes and their virtual methods and everything that they refer, and so on, to be built into the resulting binary, regardless if they're used or not, and regardless if you use Object.factory or not.

> // this does not work with rdmd but works ok with dub

Fix: static import cz.dlang.ddpa;

On 02/02/15 09:09, Vladimir Panteleev via Digitalmars-d wrote:
> 2c. Versioning
>
> Git stores the exact commit of each submodule in the parent repository.
>
> So, if you make a commit in orchard/fruit/, and run `git status` in orchard/,
> git will show you that the "fruit" submodule has been modified.
>
> Most importantly, this means that any breaking change in the "fruit" library
> will never affect existing projects which use it, since they will continue to
> use the registered commit which was known to work.
>
> This gives you dependency versioning with commit granularity - you can hardly
> ask for something better - all without messing with configuration files.

This is both a benefit and a problem.

It's a benefit to the developer, who can specify exactly which version of the dependency they want used.  It's not necessarily good for the downstream user, because it requires that the developer actively maintain the dependencies.

Scenario: a dependency has a security hole that gets patched.  If the dub package is updated, all applications using that dub package will automatically have that update available next time they are built.  With git submodules, the developer has to recognize that update has happened and tweak the git submodule in the master project, and others downstream have to realize that has happened.

Git submodules are great for precise control of dependencies, but there are circumstances where it's preferable for upstream to be able to release new versions without them being filtered in this way.  I'd readily use git submodules in scenario where I controlled all of the code (including what was in the submodules), but not where I had external dependencies.

> 2d. Git vs. Dub
>
> Unfortunately, the above-described approach is not compatible with Dub:
>
> - Dub packages are generally expected to have their source code in a "src"
> subdirectory, although you can set the source directory to "." in the
> configuration file.

Actually, I think it's 'source'.  But either way, it's just enforcing a good layout practice, that can be reworked with the config file.  There is however some benefit to imposing a standardized project layout.

> - When cloning repositories, dub does not preserve the repository's directory
> name (so e.g. fruit will be cloned to ~/.dub/fruit-1.0.0/).

Necessary in order to maintain multiple versions of a package in order to satisfy different applications' dependencies, surely?  You have a similar situation with different versions of shared libraries installed on any UNIX system.

On Monday, 2 February 2015 at 08:58:38 UTC, Joseph Rushton Wakeling wrote:
> Scenario: a dependency has a security hole that gets patched.  If the dub package is updated, all applications using that dub package will automatically have that update available next time they are built.

Is that so? Won't a security fix entail a version bump, requiring a change in the requirements file of the parent project? Also, does Dub really check for updated versions of libraries online, every time a project is built?

>> - When cloning repositories, dub does not preserve the repository's directory
>> name (so e.g. fruit will be cloned to ~/.dub/fruit-1.0.0/).
>
> Necessary in order to maintain multiple versions of a package in order to satisfy different applications' dependencies, surely?  You have a similar situation with different versions of shared libraries installed on any UNIX system.

No, it is not necessary. The directory layout could be ~/.dub/fruit-1.0.0/fruit/...

On Monday, 2 February 2015 at 09:03:56 UTC, Vladimir Panteleev wrote:
>
> Is that so? Won't a security fix entail a version bump, requiring a change in the requirements file of the parent project? Also, does Dub really check for updated versions of libraries online, every time a project is built?
>

It does.
You have a broad range of options for specifying which version to use.
http://code.dlang.org/package-format#version-specs
If you use Semver correctly, it's a great benefit.

>>> - When cloning repositories, dub does not preserve the repository's directory
>>> name (so e.g. fruit will be cloned to ~/.dub/fruit-1.0.0/).
>>
>> Necessary in order to maintain multiple versions of a package in order to satisfy different applications' dependencies, surely?  You have a similar situation with different versions of shared libraries installed on any UNIX system.
>
> No, it is not necessary. The directory layout could be ~/.dub/fruit-1.0.0/fruit/...

What would be the advantage ? You should not deal with ~/.dub/.
Note, you can specify local packages:
"fruits": { "path": "fruits" }
Works well with git submodules.

On Monday, 2 February 2015 at 08:52:29 UTC, Vladimir Panteleev wrote:
> On Monday, 2 February 2015 at 08:46:40 UTC, Daniel Kozak wrote:
>> And this is reason why rdmd suck (in my case).
>> Example:
>>
>> return cast(EntityManagerFactory)Object.factory("cz.dlang.ddpa.EntityManagerFactoryImpl");
>
> Object.factory is a misfeature. It causes ALL classes and their virtual methods and everything that they refer, and so on, to be built into the resulting binary, regardless if they're used or not, and regardless if you use Object.factory or not.
>
>> // this does not work with rdmd but works ok with dub
>
> Fix: static import cz.dlang.ddpa;

Not ideal solution, but probbably doable. ATM I have xml file with cz.dlang.ddpa.EntityManagerFactoryImpl so maybe I can make it .d file

On Monday, 2 February 2015 at 09:25:31 UTC, Mathias LANG wrote:
> On Monday, 2 February 2015 at 09:03:56 UTC, Vladimir Panteleev wrote:
>>
>> Is that so? Won't a security fix entail a version bump, requiring a change in the requirements file of the parent project? Also, does Dub really check for updated versions of libraries online, every time a project is built?
>>
>
> It does.

It actually does a roundtrip every time you build your project???

> You have a broad range of options for specifying which version to use.
> http://code.dlang.org/package-format#version-specs
> If you use Semver correctly, it's a great benefit.

In that case, why is it becoming popular to add dub.selections.json as a versioned file to the repository?

>> No, it is not necessary. The directory layout could be ~/.dub/fruit-1.0.0/fruit/...
>
> What would be the advantage ?

The directory structure would correspond with the package structure.

As is required by the language specification:

https://github.com/Hackerpilot/DCD/issues/185

> You should not deal with ~/.dub/.

You have to, if you want to use tools that need to read your source code other than dub.

> Note, you can specify local packages:
> "fruits": { "path": "fruits" }
> Works well with git submodules.

I don't see what that would do to help. Would this tell dub that there is a package called "fruits" in a directory called "fruits"? Because the compiler already assumes that already.

On Monday, 2 February 2015 at 08:09:39 UTC, Vladimir Panteleev wrote:
> snip

Also:

- Dub installs everything in ~/ (home, which on Windows is an awful location anywho). It's a pain in the ass for browsing dependencies in your editor. If it's just a submodule you can easily view it in the source tree (e.g. just open ./submodules/).

Having it as a submodule also helps if you want to quickly try to edit a dependency in order to see if it fixes an issue. And if the fix is right you can even make a commit inside the submodule and push it to your fork of the dependency.

With all the aliases or zsh-tricks this can be as easy as a few key presses.

- By default dub emits internal stack traces. This is an insane amount of visual noise that I don't want to read.

- If I want to try a new version of a dependency, I have to change the damn .json package file instead of doing a simple 'git checkout ...'. What's worse, I have to wait 15-20 minutes for the latest tagged version of a dependency to finally show up on code.dlang.org.

I could use add-local, but it's broken[1].

- Shit breaks every N releases (where N is arbitrary). As if it's not enough that code tends to break between compiler releases, so do packages between DUB releases. Something that used to build half a year ago no longer does.

I don't recall when was the last time an RDMD or Make update has ever broken my projects.

- I'm not a fan of poorly tested software, and DUB falls into that category.

I've complained before about Git submodules being a bit broken on Windows, maybe this was fixed in the meantime. It works solidly on Linux, and I'm making the decision to transfer all my projects that use dependencies to using submodules instead.

The bottom line is, time spent on dealing with package management bugs is completely wasted developer time.

[1]: http://forum.rejectedsoftware.com/groups/rejectedsoftware.dub/thread/5280/

On Monday, 2 February 2015 at 09:03:56 UTC, Vladimir Panteleev wrote:
> On Monday, 2 February 2015 at 08:58:38 UTC, Joseph Rushton Wakeling wrote:
>> Scenario: a dependency has a security hole that gets patched.  If the dub package is updated, all applications using that dub package will automatically have that update available next time they are built.
>
> Is that so? Won't a security fix entail a version bump, requiring a change in the requirements file of the parent project? Also, does Dub really check for updated versions of libraries online, every time a project is built?

Well, as long as the requirements are expressed in the form,

    "package-name": ">=1.2.3"

then there shouldn't be any problem with a version bump from the upstream application.  Whether dub actually does check for updates I'm not sure, but it _could_.  Whereas with git submodules, you really are stuck with one version and one only.


>>> - When cloning repositories, dub does not preserve the repository's directory
>>> name (so e.g. fruit will be cloned to ~/.dub/fruit-1.0.0/).
>>
>> Necessary in order to maintain multiple versions of a package in order to satisfy different applications' dependencies, surely?  You have a similar situation with different versions of shared libraries installed on any UNIX system.
>
> No, it is not necessary. The directory layout could be ~/.dub/fruit-1.0.0/fruit/...

I don't follow your point here.  What's the meaningful difference between what dub does and what you suggest?