DSSS, the D Shared Software System, is a tool to ease the building, installation, configuration and acquisition of D software.

Yet another mainly-bug-fix release. Here's the changelogs for 0.68-0.69:

0.69 from 0.68:
        - Added --test option to test built libraries.
        - Rebuild: Fixed an intermittent segfault on Windows.
        - Added --doc-binaries option to generate documentation for binary
          builds.
        - Fixed DSSS_Light build.

0.68 from 0.67:
        - Rebuild: -of<name> now converts / to \ on Windows (see ticket #85).
        - `dsss uninstall` now removes (empty) directories as well as files
          (see ticket #84).
        - Rebuild: Merged DMD 1.018.
        - Rebuild: Added proper documentation for -Dq (see ticket #90).
        - /etc will now be preferred to /usr/etc when installing to the prefix
          /usr.
        - Rebuild: Added -no-export-dynamic flag (see ticket #91).


As of 0.69, I will no longer be delivering binaries of rebuild alone - I have decided that doing so is backstabbing my own strategy. Instead, I'm providing a "light" binary of DSSS with no net support on Windows, where there are necessary prerequisite binaries for net support. It's only 300K bigger than the binary release of rebuild.

As per usual, more information and downloads are available at http://www.dsource.org/projects/dsss/

 - Gregor Richards

Gregor Richards wrote:

> 
> As of 0.69, I will no longer be delivering binaries of rebuild alone - I have decided that doing so is backstabbing my own strategy. Instead, I'm providing a "light" binary of DSSS with no net support on Windows, where there are necessary prerequisite binaries for net support. It's only 300K bigger than the binary release of rebuild.

Awww! I really have no interest in DSSS or any sort of DSSS-lite. I think Rebuild rocks, though. Having the binary available for download has been a great convenience. I wish you'd reconsider!

On 8/4/07, Mike Parker <aldacron71@yahoo.com> wrote:
> Awww! I really have no interest in DSSS or any sort of DSSS-lite. I think Rebuild rocks, though. Having the binary available for download has been a great convenience. I wish you'd reconsider!

DSSS comes with rebuild. OK, so you have DSSS installed too, but that doesn't mean you have to actually use it :-)

-- 
Anders

Assuming you don't lose any functionality, I think it's a great idea to merge it with dsss. Personally, will use it a lot more as time goes on. As an example, I use php all the time, and having pear/pecl didn't hurt php at all, but when I decided to try out xdebug or memcache bindings, it took like 5 minutes of my time. good stuff.

Anders Bergh wrote:
> On 8/4/07, Mike Parker <aldacron71@yahoo.com> wrote:
>> Awww! I really have no interest in DSSS or any sort of DSSS-lite. I think Rebuild rocks, though. Having the binary available for download has been a great convenience. I wish you'd reconsider!
> 
> DSSS comes with rebuild. OK, so you have DSSS installed too, but that doesn't mean you have to actually use it :-)
> 

Mike Parker wrote:
> Gregor Richards wrote:
> 
>>
>> As of 0.69, I will no longer be delivering binaries of rebuild alone - I have decided that doing so is backstabbing my own strategy. Instead, I'm providing a "light" binary of DSSS with no net support on Windows, where there are necessary prerequisite binaries for net support. It's only 300K bigger than the binary release of rebuild.
> 
> Awww! I really have no interest in DSSS or any sort of DSSS-lite. I think Rebuild rocks, though. Having the binary available for download has been a great convenience. I wish you'd reconsider!

I wrote rebuild as a utility for use by DSSS. That was its primary purpose in existence. The fact that it can be useful in isolation is irrelevant.

Statements like this annoy me to no end. I create this tool that integrates and standardizes all sorts of things and makes the whole situation of using 3rd party libraries infinitely easier with D, and everybody goes, "Hey, look, it works with a build utility that works well. Let's use that and not the tool itself. I have no interest because I don't know what it does and I'm happy with copying all my libraries into the same directory as the modules for my binary in a gigantic mess of unmaintainable code. Oh, and including obnoxious and platform-specific build scripts that use rebuild rather than DSSS, a tool for making easy and cross-platform build scripts, is brilliant."

This puts me into a chicken-and-egg scenario. DSSS is most useful when many available D libraries support it, but the developers for many of the D libraries continue to maintain incompatible and difficult-to-use build scripts (or, worse yet, encourage duplication), which degrades its usefulness. When I started seeing build scripts that actually used rebuild, that's when I decided I'm not going to distribute rebuild in isolation.

I don't consider people who use rebuild in isolation to be valid consumers of my code, and I'm not going to actively support that configuration. Yes, it will still work (rebuild isn't going away any time soon, it is indeed a necessary component of DSSS), but I'm not actually going to create bundles which /encourage/ people to not use the software that rebuild was built to support. Honestly, I have no idea why I was ever doing that in the first place.

</rant>

 - Gregor Richards

Reply to Gregor,

> Statements like this annoy me to no end. I create this tool that
> integrates and standardizes all sorts of things and makes the whole
> situation of using 3rd party libraries infinitely easier with D, and
> everybody goes, "Hey, look, it works with a build utility that works
> well. Let's use that and not the tool itself. I have no interest
> because I don't know what it does and I'm happy with copying all my
> libraries into the same directory as the modules for my binary in a
> gigantic mess of unmaintainable code. Oh, and including obnoxious and
> platform-specific build scripts that use rebuild rather than DSSS, a
> tool for making easy and cross-platform build scripts, is brilliant."
> 
> This puts me into a chicken-and-egg scenario. DSSS is most useful when
> many available D libraries support it, but the developers for many of
> the D libraries continue to maintain incompatible and difficult-to-use
> build scripts (or, worse yet, encourage duplication), which degrades
> its usefulness. When I started seeing build scripts that actually used
> rebuild, that's when I decided I'm not going to distribute rebuild in
> isolation.
> 
> </rant>
> 
> - Gregor Richards
> 

I hear you; I'd love to use DSSS as described. However a few issues arise

1. My primary dev systems are not connected to the internet (Not your issue)
2. The documentation is slim
3. Setting things up to use it isn't easy
4. I want to do a lot more than just build apps and libs

For #2, I've tried it several times and can't seem to make it work. A lot of this comes from the fact that I'm not totally clear on what how it's supposed to work. A page with a how-to would be REALY nice. I'd want one starting with "download this" and going though things like "put filse ot build in line 27 of dsss.conf", "look at c:\foobar for you libs" and "set setting baz to the directory where you want stuff to show up". Ideally each step would include a section on what is happening and such. That would also cover a lot of #3

as to #4, here is a list of things that my build scripts do that I haven't heard DSSS can
--run dimple to build import graphs
--sort pragma(msg,"") output into memo, notice, error, diagnostic and other categories (this is done with about 6 passes through grep/grep -v/sort)
--generate a list of functions selectively imported from std.* (find, grep, sed, sort)
--build list of asserts without messages
--build list of debug sections without debug identifiers
--build a command file that will turn on all debugging
--build full docs twice (.html + index and .tex + aggregator file)
--build test version of app
--log most of this to a time stamped file

All of this is done with a build script that runs some stuff in the background, ignores errors from some things, generates a return condition for others. It is all a bash script that makes extensive use of the "()" and "&" structures and a bunch f other stuff.

It is also totally un-portable. If dsss could duplicate all of that and be portable I would write you a testimonial that will make you look like a saint (ok I exaggerate a little).

BCS wrote:
> I hear you; I'd love to use DSSS as described. However a few issues arise
> 
> 1. My primary dev systems are not connected to the internet (Not your issue)
I fail to see what this has to do with anything.

> 2. The documentation is slim
I'm not good at documentation. There's really nothing I can do about that lack in my own person. Most documentation requests I've received, however, are for documentation that already exists.

> 3. Setting things up to use it isn't easy
Extract. Add to PATH. Make two-line dsss.conf file (for most stuff). You're right, complicated.

> 4. I want to do a lot more than just build apps and libs
> 
> For #2, I've tried it several times and can't seem to make it work. A lot of this comes from the fact that I'm not totally clear on what how it's supposed to work. A page with a how-to would be REALY nice.

http://www.dsource.org/projects/dsss/wiki/DSSSByExample ... I guess what you want is sort of a walkthrough?

http://www.codu.org/dsss_tutorials/ ?


> I'd want one starting with "download this" and going though things like "put filse ot build in line 27 of dsss.conf", "look at c:\foobar for you libs" and "set setting baz to the directory where you want stuff to show up".

Egad, you must think DSSS is incredibly complex >_O. Of all the dsss.conf files I've written, MAYBE one is 27 lines long - most are about two. You rarely have to list all of the files at all, and you never have to dig around somewhere to find libs.


> Ideally each step would include a section on what is happening and such. That would also cover a lot of #3
> 
> as to #4, here is a list of things that my build scripts do that I haven't heard DSSS can

First let me point out that DSSS has hooks (preinstall/postinstall, prebuild/postbuild, etc) to run any command at any point during the build.


> --run dimple to build import graphs

... why is this part of your build?


> --sort pragma(msg,"") output into memo, notice, error, diagnostic and other categories (this is done with about 6 passes through grep/grep -v/sort)

... why is this part of your build?


> --generate a list of functions selectively imported from std.* (find, grep, sed, sort)

... why is this part of your build?


> --build list of asserts without messages

... why is this part of your build?


> --build list of debug sections without debug identifiers

... why is this part of your build?


> --build a command file that will turn on all debugging

What do you mean by "command file" in this context?


> --build full docs twice (.html + index and .tex + aggregator file)

DSSS can build DDoc documentation automatically. Other forms of docs can be built with a postbuild hook.


> --build test version of app

Not sure precisely what you mean by this - with a specific version flag to have testing stuff enabled, or with debug/unittests/etc enabled?


> --log most of this to a time stamped file
> 
> All of this is done with a build script that runs some stuff in the background, ignores errors from some things, generates a return condition for others. It is all a bash script that makes extensive use of the "()" and "&" structures and a bunch f other stuff.

It sounds like the vast majority of your build script has nothing whatsoever to do with building, so having it in your build script is a bit ludicrous. It ought to be relegated to some sort of analysis script. Then, your (now-DSSS'd) build script would be platform-compatible and your analysis script, though not portable, wouldn't need to be.


> 
> It is also totally un-portable. If dsss could duplicate all of that and be portable I would write you a testimonial that will make you look like a saint (ok I exaggerate a little).
> 
> 

 - Gregor Richards

BCS Wrote:

> as to #4, here is a list of things that my build scripts do that I haven't
> heard DSSS can
> --run dimple to build import graphs
> --sort pragma(msg,"") output into memo, notice, error, diagnostic and other
> categories (this is done with about 6 passes through grep/grep -v/sort)
> --generate a list of functions selectively imported from std.* (find, grep,
> sed, sort)
> --build list of asserts without messages
> --build list of debug sections without debug identifiers
> --build a command file that will turn on all debugging
> --build full docs twice (.html + index and .tex + aggregator file)
> --build test version of app
> --log most of this to a time stamped file
> 
> All of this is done with a build script that runs some stuff in the background, ignores errors from some things, generates a return condition for others. It is all a bash script that makes extensive use of the "()" and "&" structures and a bunch f other stuff.
> 
> It is also totally un-portable. If dsss could duplicate all of that and be portable I would write you a testimonial that will make you look like a saint (ok I exaggerate a little).
> 
> 

I once wrote a big Perl script (5 files!) which did everything from generating code based on template files (I'm sure not as big a deal with D's insane metaprogramming capabilities, but a huge deal when generating repetitive code or code that needs to be synchronized across a lot of files for web programming - the script would generate stylesheets, JavaScript and SVGs so that with a change of a single file, the entire color scheme of the site could be changed) to running the unit tests, pushing the nightly build out and generating reports (I know CruiseControl and stuff can do that, but this way is more customizable). It was a lot of work (8-10 hours of mucking about with Perl's syntactical oddities), but the end result was well worth it, since I had a portable and highly customizable build system.

There's simply no way to get this sort of power with a tool focused simply on building. Even ant, with all its many extensions and its rather simple API makes larger projects that require more than just compilation a nightmare. At work, we have a set of eight Ant scripts averaging 10k, not to mention the 15+ taskdefs we have lying around... and still the push to the test environments is done by a set of Perl, Ruby, Transact-SQL and bash scripts strung together oddly.

DSSS looks like a great tool, and the net feature of it, which bears similarities to CPAN, etc., looks like a great way to get dependencies. But as far as building goes, I feel that build-focused tools should be better integrated with scripting languages for larger projects. If DSSS had a Perl interface, it's be perfect for me. As it is, when I get to working on a D project, I plan to invoke DSSS from within the script to do the actual build and let the script handle the things surrounding it.

I'm not actually working on a D project right now, so maybe DSSS is a "learn as you try" thing, but the docs seem REALLY spotty, so that's the other thing that's keeping me from embracing it.

But thanks, Gregor, for all your hard work!

Reply to Gregor,

> BCS wrote:
> 
>> I hear you; I'd love to use DSSS as described. However a few issues
>> arise
>> 
>> 1. My primary dev systems are not connected to the internet (Not your
>> issue)
>> 
> I fail to see what this has to do with anything.
> 

As for using dsss as a build tool for my stuff; nothing
As for using it with 3rd party stuff; "dsss net" is useless on that systems and the only thing you could do about it is add something that would let me cache stuff on a thumb drive or something. That is a rare enough situation that I don't expect you to fix it.

>> 2. The documentation is slim
>> 
> I'm not good at documentation. There's really nothing I can do about
> that lack in my own person. Most documentation requests I've received,
> however, are for documentation that already exists.
> 
>> 3. Setting things up to use it isn't easy
>> 
> Extract. Add to PATH. Make two-line dsss.conf file (for most stuff).
> You're right, complicated.
> 

Well I hadn't tried it lately but last time I tried dsss, this didn't work:

download
extract
add to path
dsss net ...  ;gives error of some sort

>> 4. I want to do a lot more than just build apps and libs
>> 
>> For #2, I've tried it several times and can't seem to make it work. A
>> lot of this comes from the fact that I'm not totally clear on what
>> how it's supposed to work. A page with a how-to would be REALY nice.
>> 
> http://www.dsource.org/projects/dsss/wiki/DSSSByExample ... I guess
> what you want is sort of a walkthrough?
> 
> http://www.codu.org/dsss_tutorials/ ?
> 

downloading now...

>> I'd want one starting with "download this" and going though things
>> like "put filse ot build in line 27 of dsss.conf", "look at c:\foobar
>> for you libs" and "set setting baz to the directory where you want
>> stuff to show up".
>> 
> Egad, you must think DSSS is incredibly complex >_O. Of all the
> dsss.conf files I've written, MAYBE one is 27 lines long - most are
> about two. You rarely have to list all of the files at all, and you
> never have to dig around somewhere to find libs.
> 


Ahh... I was looking at that part more from the user standpoint. Maybe I'm strange but I really don't like using systems that I don't know what they are doing.

OTOH if dsss is going to rely take off, a user should be able to be using it 5min after first hearing that it exists. *Everything* they need to set up should be listed on the download page. If they forget to set it up, the error should tell them how to set it up and what it is.

I'm no half wit and every time I try dsss I get fed up with it before I figure it out (mostly because I can get everything but the net dependencies in shell scripts and that is easy enough for me that I get fed up really quick).

>> Ideally each step would include a section on what is happening and
>> such. That would also cover a lot of #3
>> 
>> as to #4, here is a list of things that my build scripts do that I
>> haven't heard DSSS can
>> 
> First let me point out that DSSS has hooks (preinstall/postinstall,
> prebuild/postbuild, etc) to run any command at any point during the
> build.
> 
>> --run dimple to build import graphs
>> 
> ... why is this part of your build?

it's part of building the docs

> 
>> --sort pragma(msg,"") output into memo, notice, error, diagnostic and
>> other categories (this is done with about 6 passes through grep/grep
>> -v/sort)
>> 
> ... why is this part of your build?

I get about 200 lines of output per build, lots of this fall into the TODO list, some of it is there to tell me what template are getting built, some of it is notes.

I want this output so tuning it off is not an option. But to use it I need to filter it to make stuff easier to find.

> 
>> --generate a list of functions selectively imported from std.* (find,
>> grep, sed, sort)
>> 
> ... why is this part of your build?

I want to known when I use new function (I'm trying to avoid gratuitous 3rd party dependencies)

> 
>> --build list of asserts without messages
>> 
> ... why is this part of your build?

I want to never use then

> 
>> --build list of debug sections without debug identifiers
>> 
> ... why is this part of your build?

I want to never use then (turn on all debug and I get ~1MB output per run)

> 
>> --build a command file that will turn on all debugging
>> 
> What do you mean by "command file" in this context?
> 

dmd @cmdfile

<file>
-debug=DebugFunctionFoo
-debug=DebugFunctionBar
...
</file>

>> --build full docs twice (.html + index and .tex + aggregator file)
>>
> DSSS can build DDoc documentation automatically. Other forms of docs
> can be built with a postbuild hook.
> 

Would I have to call back into dsss? It would be nice to be able to say "also run a build with these args". This would also get the test version issue from below

>> --build test version of app
>> 
> Not sure precisely what you mean by this - with a specific version
> flag to have testing stuff enabled, or with debug/unittests/etc
> enabled?

some of each (and I want 2 executables every time, e.i. you can't build only one version)

> 
>> --log most of this to a time stamped file
>> 
>> All of this is done with a build script that runs some stuff in the
>> background, ignores errors from some things, generates a return
>> condition for others. It is all a bash script that makes extensive
>> use of the "()" and "&" structures and a bunch f other stuff.
>> 

The reason for most of this is that I want to run the analysis etc. *every time*. I want to know when things break before the build's finished running (even when it's just forgetting a coding standard). I want the doc's to be up to date as of the last build *every time*.

This all boils down to "one step build" (I'd have the unittests and regression tests in there but they take to long to run). When I start having distros, I'll have the rpm/zip/tar.gz generation as part of that script (or DSSS if it can)

> It sounds like the vast majority of your build script has nothing
> whatsoever to do with building, so having it in your build script is a
> bit ludicrous. It ought to be relegated to some sort of analysis
> script. Then, your (now-DSSS'd) build script would be
> platform-compatible and your analysis script, though not portable,
> wouldn't need to be.

If I don't need the analysis then the build step is:

build -v1 -full main.d

and dsss give me next to nothing. I want the "building the executable" step to be that simple. If it starts getting more complicated, I'll work hard to get it back to that. So dsss (for me from a developers standpoint) would end up being a fix for an error on my part.

Where dsss has the most to offer me is in the "other stuff" area.

Robert Fraser wrote:
>
> There's simply no way to get this sort of power with a tool focused simply on building. Even ant, with all its many extensions and its rather simple API makes larger projects that require more than just compilation a nightmare. At work, we have a set of eight Ant scripts averaging 10k, not to mention the 15+ taskdefs we have lying around... and still the push to the test environments is done by a set of Perl, Ruby, Transact-SQL and bash scripts strung together oddly.

True enough.  One of the larger projects I have worked on builds tools in one part of the build process (including a compiler) and uses the tools later in the build process.  These tools in turn may generate code and other data used even further on in the same build.  Then things are published, tested, etc.  However, this is beyond what I'd expect of DSSS.

> DSSS looks like a great tool, and the net feature of it, which bears similarities to CPAN, etc., looks like a great way to get dependencies. But as far as building goes, I feel that build-focused tools should be better integrated with scripting languages for larger projects. If DSSS had a Perl interface, it's be perfect for me. As it is, when I get to working on a D project, I plan to invoke DSSS from within the script to do the actual build and let the script handle the things surrounding it.

Good point.  I guess solid scripting support would go a long way towards supporting custom build options.  Perhaps MiniD would be useful here.


Sean