DSSS seems to be getting a lot of flak recently because it's not particularly happy being used with multiple versions of DMD, GDC, etc all layered together.

This is hardly fair to DSSS. Detecting such situations is an arbitrary-complexity problem, and DSSS can never really know what has been changed under it. It can make some reasonable guesses, but since IMHO those would usually been wrong, I've erred on the side of ignorance. That is, if you're going to change DSSS' underlying compilation environment, it anticipates that you will consider the consequences and `dsss distclean` first.

I don't know whether this situation will improve. To be honest, I hardly consider the fact that DSSS is not psychic a bug. As I already said, it's not something that can be fixed magically, it's actually quite complex. If you're going to be forcing DSSS to jump through multiple environments, just clean up after yourself.

Do a `dsss distclean` before building with a different compiler.

 - Gregor Richards

Gregor Richards wrote:

> DSSS seems to be getting a lot of flak recently because it's not particularly happy being used with multiple versions of DMD, GDC, etc all layered together.
> 
> This is hardly fair to DSSS. Detecting such situations is an arbitrary-complexity problem, and DSSS can never really know what has been changed under it. It can make some reasonable guesses, but since IMHO those would usually been wrong, I've erred on the side of ignorance. That is, if you're going to change DSSS' underlying compilation environment, it anticipates that you will consider the consequences and `dsss distclean` first.

I don't think distclean should never be a solution. Wouldn't it be better if there was a way to specify separate locations for binaries for the different compiler/stdlib combinations in the dsss.conf file? (gdc-phobos-bin,dmd-tango-bin,etc...)

Personally i consider the need to remove object files before a build a bug in the build system and try to write makefiles (or other commandfiles) to not require any make clean or similar.

> I don't know whether this situation will improve. To be honest, I hardly consider the fact that DSSS is not psychic a bug. As I already said, it's not something that can be fixed magically, it's actually quite complex. If you're going to be forcing DSSS to jump through multiple environments, just clean up after yourself.
> 
> Do a `dsss distclean` before building with a different compiler.
> 
>   - Gregor Richards

ps. while we already are talking dsss deficiencies will it be possible to run your own repositories (think of how git works) and specify in the dsss.conf that this lib is from here and that lib from there etc ... ?

Johan Granberg wrote:
> Gregor Richards wrote:
> 
>> DSSS seems to be getting a lot of flak recently because it's not
>> particularly happy being used with multiple versions of DMD, GDC, etc
>> all layered together.
>>
>> This is hardly fair to DSSS. Detecting such situations is an
>> arbitrary-complexity problem, and DSSS can never really know what has
>> been changed under it. It can make some reasonable guesses, but since
>> IMHO those would usually been wrong, I've erred on the side of
>> ignorance. That is, if you're going to change DSSS' underlying
>> compilation environment, it anticipates that you will consider the
>> consequences and `dsss distclean` first.
> 
> I don't think distclean should [n]ever be a solution.

Neither do I. But detecting environment changes is not always possible. It would be easy enough to detect changed flags at build time and then clean up old things before building, but that's a nonsolution after installing it. DSSS was not designed to rectify all the problems with combining DMD and GDC, Phobos and Tango.

> Wouldn't it be better if
> there was a way to specify separate locations for binaries for the
> different compiler/stdlib combinations in the dsss.conf file?
> (gdc-phobos-bin,dmd-tango-bin,etc...)

No, that would not be better. That would mean that one person could write a dsss.conf file for their library that would work with building on multiple compilers, and the next person would write one that didn't. Builds should not be this inconsistent. Besides that, it would make dsss.conf files unduly complex. Besides that, it builds flexibility limits into DSSS, making it assume that there will only ever be two compilers.

> 
> Personally i consider the need to remove object files before a build a bug
> in the build system  and try to write makefiles (or other commandfiles) to
> not require any make clean or similar.

Unless your makefiles also solve the halting problem, they cannot detect all such scenarios. Try making, then doing `make CC=incompatible_c_compiler`. The autotools "solve" this very much like DSSS does - if you want to suddenly use a different compiler, you have to clean up and start again. Try reconfiguring with a different compiler and then just typing `make` - if you're really lucky, it'll clean up for you. If you're not, it'll fail miserably. This only works for autotools because it has a configure "lock-in" phase that DSSS does not have - it can detect most environmental changes because it locks you into an environment.

As I've said over and over again: Detecting when the environment has changed in an incompatible way is an arbitrary-complexity problem.

> 
>> I don't know whether this situation will improve. To be honest, I hardly
>> consider the fact that DSSS is not psychic a bug. As I already said,
>> it's not something that can be fixed magically, it's actually quite
>> complex. If you're going to be forcing DSSS to jump through multiple
>> environments, just clean up after yourself.
>>
>> Do a `dsss distclean` before building with a different compiler.
>>
>>   - Gregor Richards
> 
> ps. while we already are talking dsss deficiencies will it be possible to
> run your own repositories (think of how git works) and specify in the
> dsss.conf that this lib is from here and that lib from there etc ... ?

The dsss.conf would be a wholly-inappropriate place to specify this. As per having multiple upstream sources, that's a possibility that will enter into it as I add the variety of changes to the source system that are coming in with DSource integration.

 - Gregor Richards

Gregor Richards wrote:

> Johan Granberg wrote:
>> Gregor Richards wrote:
>> 
>>> DSSS seems to be getting a lot of flak recently because it's not particularly happy being used with multiple versions of DMD, GDC, etc all layered together.
>>>
>>> This is hardly fair to DSSS. Detecting such situations is an arbitrary-complexity problem, and DSSS can never really know what has been changed under it. It can make some reasonable guesses, but since IMHO those would usually been wrong, I've erred on the side of ignorance. That is, if you're going to change DSSS' underlying compilation environment, it anticipates that you will consider the consequences and `dsss distclean` first.
>> 
>> I don't think distclean should [n]ever be a solution.
> 
> Neither do I. But detecting environment changes is not always possible. It would be easy enough to detect changed flags at build time and then clean up old things before building, but that's a nonsolution after installing it. DSSS was not designed to rectify all the problems with combining DMD and GDC, Phobos and Tango.
> 
>> Wouldn't it be better if
>> there was a way to specify separate locations for binaries for the
>> different compiler/stdlib combinations in the dsss.conf file?
>> (gdc-phobos-bin,dmd-tango-bin,etc...)
> 
> No, that would not be better. That would mean that one person could write a dsss.conf file for their library that would work with building on multiple compilers, and the next person would write one that didn't. Builds should not be this inconsistent. Besides that, it would make dsss.conf files unduly complex. Besides that, it builds flexibility limits into DSSS, making it assume that there will only ever be two compilers.
> 
>> 
>> Personally i consider the need to remove object files before a build a
>> bug
>> in the build system  and try to write makefiles (or other commandfiles)
>> to not require any make clean or similar.
> 
> Unless your makefiles also solve the halting problem, they cannot detect all such scenarios. Try making, then doing `make CC=incompatible_c_compiler`. The autotools "solve" this very much like DSSS does - if you want to suddenly use a different compiler, you have to clean up and start again. Try reconfiguring with a different compiler and then just typing `make` - if you're really lucky, it'll clean up for you. If you're not, it'll fail miserably. This only works for autotools because it has a configure "lock-in" phase that DSSS does not have - it can detect most environmental changes because it locks you into an environment.
> 
> As I've said over and over again: Detecting when the environment has changed in an incompatible way is an arbitrary-complexity problem.

In what way? if the binaries lives in a project subdirectory named after the compiler and standard library like cc-stdlib-bin and dsss always uses this to place binaries. That way when the compiler or standard library was swapped dsss would use a different directory and no conflicts would occur.

It is possible that I missunderstand the issue but in that case whats wrong with the above system?

>> 
>>> I don't know whether this situation will improve. To be honest, I hardly consider the fact that DSSS is not psychic a bug. As I already said, it's not something that can be fixed magically, it's actually quite complex. If you're going to be forcing DSSS to jump through multiple environments, just clean up after yourself.
>>>
>>> Do a `dsss distclean` before building with a different compiler.
>>>
>>>   - Gregor Richards
>> 
>> ps. while we already are talking dsss deficiencies will it be possible to run your own repositories (think of how git works) and specify in the dsss.conf that this lib is from here and that lib from there etc ... ?
> 
> The dsss.conf would be a wholly-inappropriate place to specify this. As per having multiple upstream sources, that's a possibility that will enter into it as I add the variety of changes to the source system that are coming in with DSource integration.
> 
>   - Gregor Richards

Where else if not dsss.conf? The way I envisioned it was a line per hos used
something like this.
with host1.addres1.com
with host2.addres2.org
possibly with specifiers for which library to get from where.

Personally this is an extremely important feature to have. It would mean that I don't need to trust some external central point of failure to host my project.

Gregor Richards schreef:
> DSSS seems to be getting a lot of flak recently because it's not particularly happy being used with multiple versions of DMD, GDC, etc all layered together.

I see this as a more complex problem than just a problem with DSSS. But maybe I'm making assumptions here that are incorrect, so feel free to correct me.

If I'm not mistaken, the (new) goal with standard libraries is to have them live peacefully next to each other with a minimal layer that provides the necessary language support (dynamic arrays, garbage collection, etc.).

Furthermore, D has a documented ABI, and I assume that this is supposed
to be a standard ABI (at some point). This means that libraries compiled
using different compilers will be compatible. (like C)

Lots of this is still far away (if at all planned), but this means that the only problem left for DSSS is to simply select a compiler. And that's an option that is already available from what I've gathered.

The two problems mentioned really have been D annoyances all along. It seems to me that the flak you were getting was just another rant about these two problems, the _real_ problems. DSSS ended up unjustly mingled in with these rants.

You are already walking the right path with DSSS, in my opinion.

- -- Stéphan

Johan Granberg wrote:
> Gregor Richards wrote:
>> Unless your makefiles also solve the halting problem, they cannot detect
>> all such scenarios. Try making, then doing `make
>> CC=incompatible_c_compiler`. The autotools "solve" this very much like
>> DSSS does - if you want to suddenly use a different compiler, you have
>> to clean up and start again. Try reconfiguring with a different compiler
>> and then just typing `make` - if you're really lucky, it'll clean up for
>> you. If you're not, it'll fail miserably. This only works for autotools
>> because it has a configure "lock-in" phase that DSSS does not have - it
>> can detect most environmental changes because it locks you into an
>> environment.
>>
>> As I've said over and over again: Detecting when the environment has
>> changed in an incompatible way is an arbitrary-complexity problem.
> 
> In what way? if the binaries lives in a project subdirectory named after the
> compiler and standard library like cc-stdlib-bin and dsss always uses this
> to place binaries. That way when the compiler or standard library was
> swapped dsss would use a different directory and no conflicts would occur.

Well, first, DSSS doesn't really know what standard library is being used, since it's not that easy to detect whether Tango is installed. It should always be configured to know this, but is at least as often configured improperly as properly, since it can't detect that automatically.

I suppose I could reliably get which compiler is being used straight out of the configuration, and put all my object files into the appropriate place per that. That's a pretty good strategy actually. http://www.dsource.org/projects/dsss/ticket/128

However, that still doesn't solve anything once you've installed. People want to install everything into DSSS' default prefix, which means intermingling DMD-compiled and GDC-compiled libraries and the .di files associated with them.

I'm starting to mentally formulate a solution for the .di file problem. I don't know whether it will work, it needs more thought. My mindset is actually shown perfectly by Stéphan's post in this thread - I don't want DSSS to depend on things being incompatible. I'll take a few steps to make things happy together, but I don't want DSSS to start failing when/if things work together properly.

> 
> It is possible that I missunderstand the issue but in that case whats wrong
> with the above system?
> 
>>>> I don't know whether this situation will improve. To be honest, I hardly
>>>> consider the fact that DSSS is not psychic a bug. As I already said,
>>>> it's not something that can be fixed magically, it's actually quite
>>>> complex. If you're going to be forcing DSSS to jump through multiple
>>>> environments, just clean up after yourself.
>>>>
>>>> Do a `dsss distclean` before building with a different compiler.
>>>>
>>>>   - Gregor Richards
>>> ps. while we already are talking dsss deficiencies will it be possible to
>>> run your own repositories (think of how git works) and specify in the
>>> dsss.conf that this lib is from here and that lib from there etc ... ?
>> The dsss.conf would be a wholly-inappropriate place to specify this. As
>> per having multiple upstream sources, that's a possibility that will
>> enter into it as I add the variety of changes to the source system that
>> are coming in with DSource integration.
>>
>>   - Gregor Richards
> 
> Where else if not dsss.conf? The way I envisioned it was a line per hos used
> something like this.
> with host1.addres1.com
> with host2.addres2.org
> possibly with specifiers for which library to get from where.
> 
> Personally this is an extremely important feature to have. It would mean
> that I don't need to trust some external central point of failure to host
> my project.

I have avoided encoding dependencies into dsss.conf at all. When the video of my presentation at dconf comes out, watch it - my ideal is that all forms of dependencies are encoded into imports. This keeps dsss.conf files simple and unilateral.

Also, you don't need to (nor even can you) host it on some DSSS-specific host. DSSS' net system just has a list of upstream sources. It needs this "central point of failure" because that's where it aggregates information about module-library relationships. That aggregation is why you can do `dsss net deps` without specifying library dependencies - they're specified by your imports.

So, the solution would be to just have an alternate source list. This is like yum, apt-get, etc (except that those usually can't live separately from the actual source).

 - Gregor Richards

Reply to Gregor,

> Also, you don't need to (nor even can you) host it on some
> DSSS-specific host. DSSS' net system just has a list of upstream
> sources. It needs this "central point of failure" because that's where
> it aggregates information about module-library relationships. That
> aggregation is why you can do `dsss net deps` without specifying
> library dependencies - they're specified by your imports.
> 
> So, the solution would be to just have an alternate source list. This
> is like yum, apt-get, etc (except that those usually can't live
> separately from the actual source).

So what would it take to let the user play with the list of servers that 'dsss net ...' starts looking at?

> 
> - Gregor Richards
> 

BCS wrote:
> Reply to Gregor,
> 
>> Also, you don't need to (nor even can you) host it on some
>> DSSS-specific host. DSSS' net system just has a list of upstream
>> sources. It needs this "central point of failure" because that's where
>> it aggregates information about module-library relationships. That
>> aggregation is why you can do `dsss net deps` without specifying
>> library dependencies - they're specified by your imports.
>>
>> So, the solution would be to just have an alternate source list. This
>> is like yum, apt-get, etc (except that those usually can't live
>> separately from the actual source).
> 
> So what would it take to let the user play with the list of servers that 'dsss net ...' starts looking at?
> 
>>
>> - Gregor Richards
>>
> 
> 

I'm not sure which of two questions you're asking here (which isn't surprising since the net architecture is a bit complicated), so I'll just answer both:


So what would it take to let the user play with the default list of sources from which `dsss net` will download libraries?

Just register at http://dsss.codu.org/ and tell me what project you need access to.


So what would it take to let the user use a different list of sources from which `dsss net` will download libraries?

Currently you can only specify one at a time, but the --source option exists for this purpose.

 - Gregor Richards

Reply to Gregor,

> I'm not sure which of two questions you're asking here (which isn't
> surprising since the net architecture is a bit complicated), so I'll
> just answer both:
> 

this question

> 
> So what would it take to let the user use a different list of sources
> from which `dsss net` will download libraries?
> 
> Currently you can only specify one at a time, but the --source option
> exists for this purpose.
>

Cool, what would it take to set up my own list server? If it is easy, It would let me play with DSSS locally. (My main dev system has, by design, no net access)

> - Gregor Richards
> 

BCS wrote:
> Reply to Gregor,
> 
>> I'm not sure which of two questions you're asking here (which isn't
>> surprising since the net architecture is a bit complicated), so I'll
>> just answer both:
>>
> 
> this question
> 
>>
>> So what would it take to let the user use a different list of sources
>> from which `dsss net` will download libraries?
>>
>> Currently you can only specify one at a time, but the --source option
>> exists for this purpose.
>>
> 
> Cool, what would it take to set up my own list server? If it is easy, It would let me play with DSSS locally. (My main dev system has, by design, no net access)
> 
>> - Gregor Richards
>>
> 
> 

In docs/README.technical, read the section on 'net installation'. It may or may not be sufficient (nobody has ever really used it as a reference :) ), so suggestions for improvements would be appreciated.

 - Gregor Richards