On 27/04/2022 4:41 AM, H. S. Teoh wrote:
> On Wed, Apr 27, 2022 at 03:57:24AM +1200, rikki cattermole via Digitalmars-d wrote:
>> On 27/04/2022 3:35 AM, H. S. Teoh wrote:
> [...]
>>> What I mean is this: my projects often involve a main executable,
>>> which is the primary target of the project, plus several helpers,
>>> which are either secondary targets sharing most of the same sources,
>>> or code generators that create one or more targets required to
>>> compile the main executable.  Occasionally, there may also be
>>> auxiliary targets like HTML pages, procedurally-generated images,
>>> and other resources.
> [...]
>>> As far as I know -- and if I'm wrong I'd be happy to be corrected --
>>> dub is unable to handle the above (at least not natively -- I'd have
>>> to write my own code for building the non-D parts of the build
>>> AFAIK, which defeats the purpose of having a build system in the
>>> first place).
>>
>> Pre build commands.
>>
>> For D stuff in dub something like this works fine.
>>
>> "preBuildCommands": ["dub run package:tool -- args"]
> 
> Does this mean I have to create an entire subpackage just for this
> purpose?  Or in fact, one subpackage per auxiliary target?  If so, that
> would seem needlessly cumbersome for something that, in my mind, is a
> trivial additional node in the build graph.
> 
> Also, treating these auxiliary build targets as second-class citizens
> doesn't really sit right with me. I mean, after all, it all boils down
> to "build sources S1, S2, ... into targets T1, T2, ... by running
> command(s) C1, C2, ...".  What if I decide to insert a postprocessing
> step in the middle of one of these build chains?  E.g., after creating a
> HTML file, before installing it to the staging area, I decide that I
> want to run a HTML tidying utility on it?  Does that mean I have to
> create another subpackage to represent this extra step?

No and no.

My example used dub as that is known to exist on the system, but you can use whatever you want.

Dub supports single file packages, so you the package in question could be only one D file, unrelated to the package you are building.

There are post build commands too. Its not just pre build.

>> But what you are describing is something automatic, which is not
>> currently supported.
> 
> What do you mean by "automatic"?  These targets are generally not
> automatically inferrable, i.e., I'm not expecting that if I say "build
> xyz.html" dub would magically know that in order to build HTML files it
> needs to compile a.d, b.d, c.d into abc.exe and run abc.exe on
> xyz.template in order to produce xyz.html.  Obviously these build steps
> must be explicitly stated somewhere.
> 
> But I do expect that build products generated by these steps would be
> smoothly integrated into the build, i.e., if "code.template" is
> preprocessed by some tool "helper.exe" to produce "code.d", then there
> should be a way to compile "code.d" into the main executable as well.

What I mean by automatic, is some sort of directive to tell dub to run a package that you depend upon that is executable.

> [...]
>>> - Network dependence (I'd *really* like for it *not* to depend on
>>>    internet access being available by default, only when I ask it
>>>    to).  IIRC there's some switch or option that does this, it would
>>>    be nice if there was a local setting I could toggle to make this
>>>    automatic.
>>
>> https://dub.pm/settings
>>
>> So yeah settings file already supports this.
> 
> Which setting disables network lookup by default?  Glancing at that
> page, it's not obvious which setting it is and what value I should set
> it to.

Set skipRegistry to configured or standard.

From there you will have to enable registries when you need them via cli. ``dub --skip-registry=none``

Upgrade only might be a good addition however.

On Wed, Apr 27, 2022 at 04:55:39AM +1200, rikki cattermole via Digitalmars-d wrote:
> On 27/04/2022 4:41 AM, H. S. Teoh wrote:
> > On Wed, Apr 27, 2022 at 03:57:24AM +1200, rikki cattermole via Digitalmars-d wrote:
[...]
> > > Pre build commands.
> > > 
> > > For D stuff in dub something like this works fine.
> > > 
> > > "preBuildCommands": ["dub run package:tool -- args"]
> > 
> > Does this mean I have to create an entire subpackage just for this purpose?  Or in fact, one subpackage per auxiliary target?  If so, that would seem needlessly cumbersome for something that, in my mind, is a trivial additional node in the build graph.
> > 
> > Also, treating these auxiliary build targets as second-class citizens doesn't really sit right with me. I mean, after all, it all boils down to "build sources S1, S2, ... into targets T1, T2, ... by running command(s) C1, C2, ...".  What if I decide to insert a postprocessing step in the middle of one of these build chains? E.g., after creating a HTML file, before installing it to the staging area, I decide that I want to run a HTML tidying utility on it?  Does that mean I have to create another subpackage to represent this extra step?
> 
> No and no.
> 
> My example used dub as that is known to exist on the system, but you can use whatever you want.

But doesn't that mean it will lose out on the benefits of having a build system in the first place?  E.g., if the source file for the helper utility wasn't changed, there's no need to run it again. But dub wouldn't know better, since I specified preBuildCommands it would always run it, correct?  So it would rebuild the helper utility every single time?


> Dub supports single file packages, so you the package in question could be only one D file, unrelated to the package you are building.

That's problematic.  Quite often the auxiliary utilities in my projects share a good amount of code with the main executable (which is why I don't put them in a separate workspace to begin with).  Can dub handle building multiple executables from different subsets of source files in the same source tree?


> There are post build commands too. Its not just pre build.

Are post build commands unconditionally run too?


[...]
> What I mean by automatic, is some sort of directive to tell dub to run a package that you depend upon that is executable.

I see.

But why should I need to create a separate package anyway?  The entire reason I build my helper programs from the same source tree is because they logically belong together and complement each other.  Why can't I have multiple targets per package, and why can't I use one target to build another within the same package?

It feels just like what deadalnix said: the cases dub handles can be handled with `dmd -i`, but the cases that can't be handled by `dmd -i` can't be handled by dub either.  It unfortunately only reinforces my perception that dub the package manager is good, but dub the build system sucks.  It makes me want to use dub only for package dependencies, but a different build system.


[...]
> > Which setting disables network lookup by default?  Glancing at that page, it's not obvious which setting it is and what value I should set it to.
> 
> Set skipRegistry to configured or standard.
> 
> From there you will have to enable registries when you need them via cli.  ``dub --skip-registry=none``

Thanks, this is helpful.


> Upgrade only might be a good addition however.

Doesn't dub already do this? Or am I not understanding this right?


T

-- 
What's a "hot crossed bun"? An angry rabbit.

On 27/04/2022 5:19 AM, H. S. Teoh wrote:
> On Wed, Apr 27, 2022 at 04:55:39AM +1200, rikki cattermole via Digitalmars-d wrote:
>> On 27/04/2022 4:41 AM, H. S. Teoh wrote:
>>> On Wed, Apr 27, 2022 at 03:57:24AM +1200, rikki cattermole via Digitalmars-d wrote:
> [...]
>>>> Pre build commands.
>>>>
>>>> For D stuff in dub something like this works fine.
>>>>
>>>> "preBuildCommands": ["dub run package:tool -- args"]
>>>
>>> Does this mean I have to create an entire subpackage just for this
>>> purpose?  Or in fact, one subpackage per auxiliary target?  If so,
>>> that would seem needlessly cumbersome for something that, in my
>>> mind, is a trivial additional node in the build graph.
>>>
>>> Also, treating these auxiliary build targets as second-class
>>> citizens doesn't really sit right with me. I mean, after all, it all
>>> boils down to "build sources S1, S2, ... into targets T1, T2, ... by
>>> running command(s) C1, C2, ...".  What if I decide to insert a
>>> postprocessing step in the middle of one of these build chains?
>>> E.g., after creating a HTML file, before installing it to the
>>> staging area, I decide that I want to run a HTML tidying utility on
>>> it?  Does that mean I have to create another subpackage to represent
>>> this extra step?
>>
>> No and no.
>>
>> My example used dub as that is known to exist on the system, but you
>> can use whatever you want.
> 
> But doesn't that mean it will lose out on the benefits of having a build
> system in the first place?  E.g., if the source file for the helper
> utility wasn't changed, there's no need to run it again. But dub
> wouldn't know better, since I specified preBuildCommands it would always
> run it, correct?  So it would rebuild the helper utility every single
> time?

Dub is able to cache things.

In context I don't know how much can be done prevent work, since your own tools would need to do caching as well.

>> Dub supports single file packages, so you the package in question
>> could be only one D file, unrelated to the package you are building.
> 
> That's problematic.  Quite often the auxiliary utilities in my projects
> share a good amount of code with the main executable (which is why I
> don't put them in a separate workspace to begin with).  Can dub handle
> building multiple executables from different subsets of source files in
> the same source tree?

Can be.

Single file packages just like any other package can depend upon another by path, rather than version lookup.

I do this plenty, it works fine. Quite often I swap out to a full blown package, since single file packages can't have multiple files in it (hence the name).

>> There are post build commands too. Its not just pre build.
> 
> Are post build commands unconditionally run too?

Probably.

> [...]
>> What I mean by automatic, is some sort of directive to tell dub to run
>> a package that you depend upon that is executable.
> 
> I see.
> 
> But why should I need to create a separate package anyway?  The entire
> reason I build my helper programs from the same source tree is because
> they logically belong together and complement each other.  Why can't I
> have multiple targets per package, and why can't I use one target to
> build another within the same package?

Yes this is a pretty big sticking point.

By in large this is a genuine issue that I believe needs resolving.

However in 10 years I haven't needed it, I only just recently got around to improving shared library support so it actually worked as I needed that ;)

>>> Which setting disables network lookup by default?  Glancing at that
>>> page, it's not obvious which setting it is and what value I should
>>> set it to.
>>
>> Set skipRegistry to configured or standard.
>>
>>  From there you will have to enable registries when you need them via
>> cli.  ``dub --skip-registry=none``
> 
> Thanks, this is helpful.
> 
> 
>> Upgrade only might be a good addition however.
> 
> Doesn't dub already do this? Or am I not understanding this right?

I don't know. I don't use this feature.

https://github.com/dlang/dub/issues/2244

On Wed, Apr 27, 2022 at 05:27:01AM +1200, rikki cattermole via Digitalmars-d wrote:
> 
> On 27/04/2022 5:19 AM, H. S. Teoh wrote:
[...]
> > But doesn't that mean it will lose out on the benefits of having a build system in the first place?  E.g., if the source file for the helper utility wasn't changed, there's no need to run it again. But dub wouldn't know better, since I specified preBuildCommands it would always run it, correct?  So it would rebuild the helper utility every single time?
> 
> Dub is able to cache things.
> 
> In context I don't know how much can be done prevent work, since your own tools would need to do caching as well.

Right, which brings me back to square one. Dub already knows how to cache stuff, but I can't leverage any of it because it was arbitrarily decided that dub would only work with D source files, and therefore it only caches D artifacts.  If I want to build HTML files from templates, for example, I couldn't leverage dub's caching ability; I have to roll my own and reinvent the square wheel.  If I want to build PNG files from data files, I'd also have to roll my own cache.  Eventually, I'd want to avoid reinventing my own cache for every artifact I want to build, so I'd look for a system that offers me that capability.  Such a system would obviously also be able to build D programs and cache them as artifacts, since building D programs would be a specific case of the more general task of building a target from sources and caching it. Which in turn means that dub has just become redundant: I might as well just use that other system for *all* my build needs, including building D programs, and it would let me do everything I need without needing to work with two different, overlapping systems (dub + other build system).

That's my current line of reasoning, and why I haven't seriously invested in dub.


[...]
> Single file packages just like any other package can depend upon another by path, rather than version lookup.
> 
> I do this plenty, it works fine. Quite often I swap out to a full blown package, since single file packages can't have multiple files in it (hence the name).
[...]
> > But why should I need to create a separate package anyway?  The entire reason I build my helper programs from the same source tree is because they logically belong together and complement each other. Why can't I have multiple targets per package, and why can't I use one target to build another within the same package?
> 
> Yes this is a pretty big sticking point.
> 
> By in large this is a genuine issue that I believe needs resolving.

Whether dub ends up implementing this or not, there is a larger picture I feel is being missed here.

Even if dub eventually allows building multiple executables from different subsets of the same source tree, that's still limited to only D artifacts.  It still doesn't allow the user to leverage its dependency resolution, building, and caching capabilities for non-D artifacts.  If I were to adopt dub at that point, I'd still have to roll my own dependency management, building, and caching for non-D artifacts.  Which would drive me to use a different build system that *can* offer me those capabilities. Which in turn makes dub a redundant factor in the equation.

Strictly speaking, dependency resolution isn't specific to building D programs.  There's no technical reason why it should be limited only to D.  Similarly, building a D program is merely a more specific instance of the general task of transforming a set of input files into one or more output targets by running some command (compiler in the case of D, but I seriously doubt dub actually cares what executable is being run in the place of the compiler, as long as it produces the expected outputs). Caching is also a general concept that isn't tied specifically to compiling D programs.

These general capabilities are arbitrarily restricted in dub, for whatever reason, to only building D artifacts. As a consequence, dub is unable to handle many things which, given the algorithms that it *already implements*, it ought to able to handle.  The user is unable to leverage these algorithms, which dub already uses internally, to other tasks in his project.  This imposes arbitrary limitations on dub and forces users to seek for more general solutions.

If there was a way to remove these limitations from dub so that it can be used for more general build needs, I would be happy to use it.  But as things stand, it's not meeting my build needs.


> However in 10 years I haven't needed it, I only just recently got around to improving shared library support so it actually worked as I needed that ;)

I'd be happy to contribute toward making dub more general, like I outlined above.  But my current perception is that the dub maintainers aren't interested in moving in this direction. I'd be happy to be proven wrong, though!


[...]
> > > Upgrade only might be a good addition however.
> > 
> > Doesn't dub already do this? Or am I not understanding this right?
> 
> I don't know. I don't use this feature.
> 
> https://github.com/dlang/dub/issues/2244

I think I'm missing something here. How does "upgrade only" relate to the our present discussion?


T

-- 
Windows 95 was a joke, and Windows 98 was the punchline.

On 27/04/2022 6:14 AM, H. S. Teoh wrote:
>> However in 10 years I haven't needed it, I only just recently got
>> around to improving shared library support so it actually worked as I
>> needed that ;)
> 
> I'd be happy to contribute toward making dub more general, like I
> outlined above.  But my current perception is that the dub maintainers
> aren't interested in moving in this direction. I'd be happy to be proven
> wrong, though!

Yeah, there is a strong desire in the community to make dub support other build systems. It has been known for many years that this will need to happen at some point.

It is its greatest weakness due to it not supporting other languages. Which happens to be why it also works very well for D only projects *sigh*.

Note: there really isn't any maintainers, that is the problem. Dub needs someone to step up to the plate to manage it.

If we had someone who was able to review code and merge stuff for dub without waiting months (total exaggeration) a lot of these issues might go away (man power wise).

Of course this will be needed once ImportC starts to be mature enough to use (i.e. integrated preprocessor).

> [...]
>>>> Upgrade only might be a good addition however.
>>>
>>> Doesn't dub already do this? Or am I not understanding this right?
>>
>> I don't know. I don't use this feature.
>>
>> https://github.com/dlang/dub/issues/2244
> 
> I think I'm missing something here. How does "upgrade only" relate to
> the our present discussion?


$ dub upgrade

That should be the only command that should call out to the web service by default, unless required.

This command reruns dependency resolution and things like that, which were already cached.

On Wed, Apr 27, 2022 at 06:25:40AM +1200, rikki cattermole via Digitalmars-d wrote:
> On 27/04/2022 6:14 AM, H. S. Teoh wrote:
[...]
> > I'd be happy to contribute toward making dub more general, like I outlined above.  But my current perception is that the dub maintainers aren't interested in moving in this direction. I'd be happy to be proven wrong, though!
> 
> Yeah, there is a strong desire in the community to make dub support other build systems. It has been known for many years that this will need to happen at some point.

Well, maybe now is the time to make that happen!


> It is its greatest weakness due to it not supporting other languages. Which happens to be why it also works very well for D only projects *sigh*.
> 
> Note: there really isn't any maintainers, that is the problem. Dub needs someone to step up to the plate to manage it.
> 
> If we had someone who was able to review code and merge stuff for dub
> without waiting months (total exaggeration) a lot of these issues
> might go away (man power wise).

Wait. If there are no maintainers, then how does *anything* get merged at all?!


> Of course this will be needed once ImportC starts to be mature enough to use (i.e. integrated preprocessor).

Well, so it looks like the time is ripe to break dub out of its original limitations and extend its scope to other languages. (And other build-like tasks, in general. ;-))


[...]
> > I think I'm missing something here. How does "upgrade only" relate to the our present discussion?
> 
> 
> $ dub upgrade
> 
> That should be the only command that should call out to the web service by default, unless required.
> 
> This command reruns dependency resolution and things like that, which were already cached.

Ah, I see.  That makes sense.


T

-- 
It won't be covered in the book. The source code has to be useful for something, after all. -- Larry Wall

On 27/04/2022 6:47 AM, H. S. Teoh wrote:
>> It is its greatest weakness due to it not supporting other languages.
>> Which happens to be why it also works very well for D only projects
>> *sigh*.
>>
>> Note: there really isn't any maintainers, that is the problem. Dub
>> needs someone to step up to the plate to manage it.
>>
>> If we had someone who was able to review code and merge stuff for dub
>> without waiting months (total exaggeration) a lot of these issues
>> might go away (man power wise).
> Wait. If there are no maintainers, then how does*anything*  get merged
> at all?!

Looks like WebFreak has been the main person merging stuff in the last year (and a lot of it has been mine :D).

On 27/04/2022 7:02 AM, deadalnix wrote:
> 2/ Copy 3rd party libs in your source tree and redo the build.
> 4/ Have dub build the 3rd party lib and pile on hacks to recover the results in your own build.

Something I want to see is a proper artifact repository.

For non-D that should allow us to upload binaries to it, and have dub automatically download and copy it based upon the target.

I think such a service would go a long way to give a much better experience for people.