On Friday, 26 April 2019 at 22:56:53 UTC, Andre Pany wrote:
> On Friday, 26 April 2019 at 22:33:50 UTC, H. S. Teoh wrote:
>> On Fri, Apr 26, 2019 at 08:28:37PM +0000, H. S. Teoh via Digitalmars-d wrote: [...]
>>> According to Nick, every dub package carries its own description file (sdl/json) that encodes, among other things, its version and dependencies.  Apparently, this information resides per-package on code.dlang.org, and so every time dub has to resolve dependencies for a particular package, it has to download the description files of *every version* of the package??  Can somebody confirm whether or not this is really the case?
>> [...]
>>
>> Actually, nevermind that.  Dub already combines the descriptions into a single file.  So the bottleneck(s) must lie elsewhere...
>>
>>
>> T
>
> Also with recent Dub version a major performance optimization was done
> https://dlang.org/changelog/2.086.0.html#single-api-requests
>
> In addition Sebastian is also working on parallel dependencies download.
>
> Kind regards
> Andre

Unfortunately I'm not actively working on this. I have too many other things on my plate. I just did a quick experiment and it more than halfed the fresh build time for the tested projects (the unzipping happens in parallel too). However, the quick solution doesn't seem to work with older D versions.
In case anyone is interested:

https://github.com/dlang/dub/pull/1677

On Friday, 26 April 2019 at 22:33:50 UTC, H. S. Teoh wrote:
> On Fri, Apr 26, 2019 at 08:28:37PM +0000, H. S. Teoh via Digitalmars-d wrote: [...]
>> According to Nick, every dub package carries its own description file (sdl/json) that encodes, among other things, its version and dependencies.  Apparently, this information resides per-package on code.dlang.org, and so every time dub has to resolve dependencies for a particular package, it has to download the description files of *every version* of the package??  Can somebody confirm whether or not this is really the case?
> [...]
>
> Actually, nevermind that.  Dub already combines the descriptions into a single file.  So the bottleneck(s) must lie elsewhere...
>
>
> T

Speaking of bottlenecks... (and probably unrelated):

For dexed I have a project group that contains the 3 sdl for dmd/druntime/phobos. It's very slow to load... the IDE launches dub 3 times to convert the projects to JSON and it takes 8 to 10 seconds. But none of the sdl contains dependencies, nor configs, build types or whatever...

I have the feeling that dub does something stupid with the files since these 3 repos contains many.

On Sat, Apr 27, 2019 at 12:02:58AM +0000, Basile B. via Digitalmars-d wrote: [...]
> For dexed I have a project group that contains the 3 sdl for dmd/druntime/phobos. It's very slow to load... the IDE launches dub 3 times to convert the projects to JSON and it takes 8 to 10 seconds. But none of the sdl contains dependencies, nor configs, build types or whatever...
> 
> I have the feeling that dub does something stupid with the files since these 3 repos contains many.

Shouldn't we be able to just run a profiler on dub to figure out where it's spending those 10 seconds? (or about 3 seconds per run.)  Seems like something easy enough to identify, even if the problem itself may not be so simple to fix.  We should find out whether it's CPU-bound, I/O-bound, or network-bound, at the very least.


T

-- 
MS Windows: 64-bit rehash of 32-bit extensions and a graphical shell for a 16-bit patch to an 8-bit operating system originally coded for a 4-bit microprocessor, written by a 2-bit company that can't stand 1-bit of competition.

On Fri, Apr 26, 2019 at 05:37:35PM -0700, H. S. Teoh via Digitalmars-d wrote:
> On Sat, Apr 27, 2019 at 12:02:58AM +0000, Basile B. via Digitalmars-d wrote: [...]
> > For dexed I have a project group that contains the 3 sdl for dmd/druntime/phobos. It's very slow to load... the IDE launches dub 3 times to convert the projects to JSON and it takes 8 to 10 seconds.  But none of the sdl contains dependencies, nor configs, build types or whatever...
> > 
> > I have the feeling that dub does something stupid with the files since these 3 repos contains many.
> 
> Shouldn't we be able to just run a profiler on dub to figure out where it's spending those 10 seconds? (or about 3 seconds per run.)  Seems like something easy enough to identify, even if the problem itself may not be so simple to fix.  We should find out whether it's CPU-bound, I/O-bound, or network-bound, at the very least.
[...]

OK, so I upgraded my dub to the latest git master, and did a quick and dirty test.  Init a fresh new project with `-t vibe.d`, accept all default values (name, license, etc.), then run `time dub -v` to make the first build.

It took 2 minutes 45 seconds (!).  From a crude visual inspection, most of the time was spent downloading and compiling the dependencies, so that seems reasonable, even if unfortunate.

Subsequent runs (remove dub.selections, remove executable, rerun dub -v) took about 11 seconds or so.  Still rather long, but it's a LOT faster than the previous version of dub that I tested prior to upgrading to git master.

The previous version of dub took about 2 minutes 15 seconds for the first build of a fresh empty vibe.d project, but most of the time appeared to be spent on the "searching for versions of xyz (1 suppliers)" phase, which I presume was the per-version downloading of package descriptions.  (The fact that it took 30 seconds less than the new dub is likely caused by some cached objects from previous vibe.d builds, so that measurement shouldn't be trusted. I deleted the entire dub cache before running the test on the new version of dub.) Subsequent runs (delete dub.selections, rerun dub -v) took about 40 seconds, most of which was spent on the "searching for versions" phase.

So the new dub appeared to have eliminated a good chunk of network turnaround time, which is good news.

10 seconds is still rather long for an empty project, but from casual visual inspection of -v output, it appears to still be stuck on the "searching for versions of vibe.d" phase, which took several solid seconds.  I'm presuming this is where it's waiting on the network. Re-running dub without deleting dub.selections eliminates this step, and turnaround time drops to about 4-5 seconds.  Finally!! It's getting a lot closer to a serviceable edit-compile-test turnaround time!

So the latest version of dub is looking a lot better than before. Now, if we could lift some of its functional limitations, we might finally have an acceptably performant and functional package/build tool.

In the meantime, it would seem that we need to look into why the "searching for versions" phase takes so long.  Is it just a network-dependent thing (my network has bogonously slow DNS resolution, no thanks to my ISP), or is it something that can be fixed in dub itself?


T

-- 
VI = Visual Irritation

On Saturday, 27 April 2019 at 01:13:33 UTC, H. S. Teoh wrote:
> On Fri, Apr 26, 2019 at 05:37:35PM -0700, H. S. Teoh via Digitalmars-d wrote:
>> [...]
> [...]
>
> OK, so I upgraded my dub to the latest git master, and did a quick and dirty test.  Init a fresh new project with `-t vibe.d`, accept all default values (name, license, etc.), then run `time dub -v` to make the first build.
>
> [...]

10 seconds for a rebuild is still too much.

One quick solution is to upgrade to ld.gold - it will half your build time.

On Saturday, 27 April 2019 at 01:13:33 UTC, H. S. Teoh wrote:
> In the meantime, it would seem that we need to look into why the "searching for versions" phase takes so long.  Is it just a network-dependent thing (my network has bogonously slow DNS resolution, no thanks to my ISP), or is it something that can be fixed in dub itself?

Thanks a lot for your interest in dub and investigating this!

First off you should never delete your dub.selections.json as it locks your project dependencies (it won't be used if your project is used as a library).

Anyhow, that being said there are still a ton of things that can be done:

- The new single API request feature doesn't work for all 100% with optional dependencies (see the respective GitHub PR that introduced it for details)
- Dependencies could be checked in parallel
- The registry itself could be optimized more for caching (maybe even with a CDN proxy)
...

There are more experimental things we could try like e.g. a fully local JSON index that is only updated when needed and supports partial updates (think apt), but I believe the bigger gains in user-experience will be:

- initial fetch (important for fast CI turnaround times. First point of attack: parallelizing the fetching process)
- build times with existing dependencies (important as the default case. First points of attack: build independent dependencies in parallel, warn if ld.gold isn't the default on Linux, ...)

On 4/27/19 1:45 AM, Seb wrote:
> 
> - The new single API request feature doesn't work for all 100% with optional dependencies (see the respective GitHub PR that introduced it for details)

The API you're referring to, is it this one?

https://code.dlang.org/api/packages/[PACKAGE_NAME]/info

or something else?

On 4/26/19 4:28 PM, H. S. Teoh wrote:
> Recently, Nick & some others (including myself) created a Github project to discuss the possibility of a dub replacement that has a better architecture.

To clarify, this project is only intended to replace the package management side of dub.

It deliberately *doesn't* replace the buildsystem side of dub. It's strictly buildsystem-agnostic (and even programming language agnostic), so you can bring-your-own-buildsystem whether it be ninja, SCons, make, manual buildscripts, or indeed, even dub itself.

Also, to assuage any fears, one of the key goals is to support the existing DUB packages on code.dlang.org, with no work necessary on the part of DUB package authors. (Naturally, such packages will still be using dub as their buildsystem...or perhaps Atila's 'bud' once that's up and running.)

For anyone interested, here's the project on Github:
https://github.com/Abscissa/DPak

On Saturday, 27 April 2019 at 00:37:35 UTC, H. S. Teoh wrote:
> On Sat, Apr 27, 2019 at 12:02:58AM +0000, Basile B. via Digitalmars-d wrote: [...]
>> For dexed I have a project group that contains the 3 sdl for dmd/druntime/phobos. It's very slow to load... the IDE launches dub 3 times to convert the projects to JSON and it takes 8 to 10 seconds. But none of the sdl contains dependencies, nor configs, build types or whatever...
>> 
>> I have the feeling that dub does something stupid with the files since these 3 repos contains many.
>
> Shouldn't we be able to just run a profiler on dub to figure out where it's spending those 10 seconds? (or about 3 seconds per run.)  Seems like something easy enough to identify, even if the problem itself may not be so simple to fix.  We should find out whether it's CPU-bound, I/O-bound, or network-bound, at the very least.
>
>
> T

I've determined that the slowdown is caused by the auto-fetch option of the IDE. When disabled the group loads instantly. Actually there are dependencies in the sdl for DMD: dmd:lexer, dmd:parser, dmd:frontend. Dexed fails to determine that they are described directly in the description.