Hi,

I have a few modules for parsing different file formats and a folder with d-scripts using these parsers to perform manipulations, extract information, ...

Until now, I just added
#!/usr/bin/env rdmd

to the d-scripts and run them that way, but I'd like to make the scripts available to my collaborators without them being required to install dmd & co, so I'll have to compile the scripts.

I usually would just write a makefile for that, but I thought I'd give DUB a go. Unfortunately, the DUB-documentation is a little thin and I cannot find a way to tell DUB
"compile all the files in the scripts folder and put the binary to the bin folder". How do I do that?

Thanks!

On Thursday, 3 September 2020 at 08:22:25 UTC, glis-glis wrote:

> I usually would just write a makefile for that, but I thought I'd give DUB a go. Unfortunately, the DUB-documentation is a little thin and I cannot find a way to tell DUB
> "compile all the files in the scripts folder and put the binary to the bin folder". How do I do that?

By default Dub will compile all files in the `source` or `src` directory and place the binary in the root directory of the project. If you want to override that you can use the following build settings `sourcePaths` or `sourceFiles` to override the source directory. To override the output directory you can use `targetPath` build setting. They are documented here [1][2].

[1] For the SDL format: https://dub.pm/package-format-sdl.html#build-settings
[2] For the JSON format: https://dub.pm/package-format-json.html#build-settings

--
/Jacob Carlborg

On Thursday, 3 September 2020 at 09:36:53 UTC, Jacob Carlborg wrote:
> On Thursday, 3 September 2020 at 08:22:25 UTC, glis-glis wrote:
>
>> I usually would just write a makefile for that, but I thought I'd give DUB a go. Unfortunately, the DUB-documentation is a little thin and I cannot find a way to tell DUB
>> "compile all the files in the scripts folder and put the binary to the bin folder". How do I do that?
>
> By default Dub will compile all files in the `source` or `src` directory and place the binary in the root directory of the project. If you want to override that you can use the following build settings `sourcePaths` or `sourceFiles` to override the source directory. To override the output directory you can use `targetPath` build setting. They are documented here [1][2].
>
> [1] For the SDL format: https://dub.pm/package-format-sdl.html#build-settings
> [2] For the JSON format: https://dub.pm/package-format-json.html#build-settings
>
> --
> /Jacob Carlborg

Yes I already tried that, but I get the error

Error: only one main allowed. Previously found main at src/scripts/copy.d

Looks like DUB doesn't like multiple binaries?

On 2020-09-03 14:41, glis-glis wrote:

> Yes I already tried that, but I get the error
> 
> Error: only one main allowed. Previously found main at src/scripts/copy.d
> 
> Looks like DUB doesn't like multiple binaries?

Oh, multiple binaries, I missed that. You can try to add multiple configurations [1]. Or if you have executables depending on only one source file, you can use single-file packages [2].

Here are some real-world examples [3][4].

[1] https://dub.pm/package-format-sdl.html#configurations
[2] https://dub.pm/advanced_usage.html
[3] https://github.com/jacob-carlborg/dstep/blob/master/dub.json#L26
[4] https://github.com/d-widget-toolkit/dwt/blob/master/org.eclipse.swt.snippets/src/org/eclipse/swt/snippets/Snippet10.d

-- 
/Jacob Carlborg

On Thursday, 3 September 2020 at 14:34:48 UTC, Jacob Carlborg wrote:
> Oh, multiple binaries, I missed that. You can try to add multiple configurations [1]. Or if you have executables depending on only one source file, you can use single-file packages [2].

Thanks, but this still means I would have to write an install-script running

`dub build --single`

on each script, right?
I looked at tsv-utils [1] which seems to be a similar use-case as mine, and they declare each tool as a subpackage. The main package runs a d-file called `dub_build.d` which compiles all subpackages. Fells like an overkill to me, I'll probably just stick to a makefile.


[1] https://github.com/eBay/tsv-utils/blob/master/docs/AboutTheCode.md#building-and-makefile

On 9/4/20 10:27 AM, glis-glis wrote:
> On Thursday, 3 September 2020 at 14:34:48 UTC, Jacob Carlborg wrote:
>> Oh, multiple binaries, I missed that. You can try to add multiple configurations [1]. Or if you have executables depending on only one source file, you can use single-file packages [2].
> 
> Thanks, but this still means I would have to write an install-script running
> 
> `dub build --single`
> 
> on each script, right?
> I looked at tsv-utils [1] which seems to be a similar use-case as mine, and they declare each tool as a subpackage. The main package runs a d-file called `dub_build.d` which compiles all subpackages. Fells like an overkill to me, I'll probably just stick to a makefile.
> 
> 
> [1] https://github.com/eBay/tsv-utils/blob/master/docs/AboutTheCode.md#building-and-makefile 
> 

I don't know is it suitable for your use case but I do the following:
1. add a recipe in every tool files like:
```
drug@drug: ~/utils$ cat ./tools/count.d
#!/usr/bin/env dub
/+ dub.sdl:
	name        "count"
	targetType  "executable"
	targetPath  "../bin"
	targetName  "count"

	dependency "abcdLibrary" version="*" path="path/to/abcdLibrary"
+/

int main()
{
	...
}
```
2. place all tool files into one directory `tools`
3. build them using `dub build --single tools/count.d`
4. Now I have all binary in `bin` directory, all sources in `tools` directory and have no subpackages at all

On Thursday, 3 September 2020 at 08:22:25 UTC, glis-glis wrote:
> Hi,
>
> I have a few modules for parsing different file formats and a folder with d-scripts using these parsers to perform manipulations, extract information, ...
>
> Until now, I just added
> #!/usr/bin/env rdmd
>
> to the d-scripts and run them that way, but I'd like to make the scripts available to my collaborators without them being required to install dmd & co, so I'll have to compile the scripts.
>
> I usually would just write a makefile for that, but I thought I'd give DUB a go. Unfortunately, the DUB-documentation is a little thin and I cannot find a way to tell DUB
> "compile all the files in the scripts folder and put the binary to the bin folder". How do I do that?
>
> Thanks!

Option 1: (recommended for multiple single file binaries)

use preBuildCommands to invoke dub or dmd for each of your scripts you need

Compatibility tip: use the special $DUB and $DC variables, see https://dub.pm/package-format-json.html#environment-variables

If built often, consider using rdmd for easy caching or make this a sub-package which you depend on which will therefore only be rebuilt in case of changes to that sub-path folder. Add a glob of "extraDependencyFiles" matching your script files to make the package rebuild on script changes.

Option 2: (recommended for multiple larger binaries)

use sub-packages and make your root package targetType: none to build all child packages, see https://dub.pm/package-format-json.html#sub-packages

root dub.sdl:
name "test"
targetType "none"
dependency ":a" version="*"
dependency ":b" version="*"
subPackage "a"
subPackage "b"

a/dub.sdl:
name "a"
targetType "executable"

b/dub.sdl:
name "b"
targetType "executable"

$ dub build

Result:
a/test_a.exe
b/test_b.exe

Note: this only works if your package is targetType none, use targetName and targetPath to customize how the exe is called and where it's placed

On Friday, 4 September 2020 at 07:27:33 UTC, glis-glis wrote:
> On Thursday, 3 September 2020 at 14:34:48 UTC, Jacob Carlborg wrote:
>> Oh, multiple binaries, I missed that. You can try to add multiple configurations [1]. Or if you have executables depending on only one source file, you can use single-file packages [2].
>
> Thanks, but this still means I would have to write an install-script running
>
> `dub build --single`
>
> on each script, right?
> I looked at tsv-utils [1] which seems to be a similar use-case as mine, and they declare each tool as a subpackage. The main package runs a d-file called `dub_build.d` which compiles all subpackages. Fells like an overkill to me, I'll probably just stick to a makefile.
>
>
> [1] https://github.com/eBay/tsv-utils/blob/master/docs/AboutTheCode.md#building-and-makefile

The `dub_build.d` is so that people can use `$ dub fetch` to download and build the tools with `$ dub run`, from code.dlang.org. dub fetch/run is the typical dub sequence. But it's awkward. And it geared toward users that have a D compiler plus dub already installed. For building your own binaries you might as well use `make`. However, if you decide to add your tools to the public dub package registry you might consider the technique.

My understanding is that the dub developers recognize that multiple binaries are inconvenient at present and have ideas on improvements. Having a few more concrete use cases might help nail down the requirements.

The tsv-utils directory layout may be worth a look. It's been pretty successful for multiple binaries in a single repo with some shared code. (Different folks made suggestions leading to this structure.) It works for both make and dub, and works well with other tools, like dlpdocs (Adam Ruppe's doc generator). The tsv-utils `make` setup is quite messy at this point, you can probably do quite a bit better.

--Jon

Thanks everybody!

My folder structure looks like this:
.
├── bin
├── dub.sdl
├── src
│   ├── biophysics
│   │   └── ...
│   └── tools
│   │   └── ...
└── test
    ├── ...

Following the advice of drug, I moved the "tools" folder from src to the parent directory, added a dependency to "biophysics" with path ".." and build the binaries with  "dub build --single tools/..."

This works fine, however, now I have a problem when editing with Emacs: Flycheck gives errors as it cannot find my package anymore and company autocomplete does not work anymore neither.
By the way, I have the same problem when I clone tsv-utils and open the source file with Emacs. I have to admit I am using emacs' d-mode as a black-box since there seems to be no documentation, and I am not very fluent in elisp...

I tried keeping the tools folder in src and added the following line to dub.sdl:
`excludedSourceFiles "./src/tools/*.d"`
however, this does not seem to make any difference.

A solution that works is putting dub.sdl into src, adding
`sourcePaths "biophysics"`,
however, "dub build --single" is only working in the src-folder and not the parent folder...

On Tuesday, 8 September 2020 at 08:41:24 UTC, glis-glis wrote:
>
> Following the advice of drug, I moved the "tools" folder from src to the parent directory, added a dependency to "biophysics" with path ".." and build the binaries with  "dub build --single tools/..."
>
> This works fine, however, now I have a problem when editing with Emacs: Flycheck gives errors as it cannot find my package anymore and company autocomplete does not work anymore neither.
> By the way, I have the same problem when I clone tsv-utils and open the source file with Emacs. I have to admit I am using emacs' d-mode as a black-box since there seems to be no documentation, and I am not very fluent in elisp...


To follow up, I came up with the following folder structure:

.
├── bin
│   └── ...
├── dub.sdl
├── install
├── lib
│   └── ...
├── src
│   └── biophysics
│       └── ...
├── test
│   └── ...
└── tools
    ├── biophysics -> ../src/biophysics
    └── ...

the symbolic link did the trick for emacs Flycheck and Company. The install script simply reads

#!/usr/bin/env bash
for f in tools/*.d; do
	dub build --build=release --single $f
done