On 4/13/18 5:57 PM, Jonathan M Davis wrote:
> On Friday, April 13, 2018 16:15:21 Steven Schveighoffer via Digitalmars-d

>> I don't know if the compiler can determine if a version statement
>> affects the layout, I suppose it could, but it would have to compile
>> both with and without the version to see. It's probably an intractable
>> problem.
> 
> Also, does it really matter? If there's a mismatch, then you'll get a linker
> error, so it's not like you're going to get subtle bugs out of the deal or
> anything like that. I don't see why detection is an issue here.

Well, for layout changes, there is no linker error. It's just one version of the code thinks the layout is one way, and another version thinks it's another way. This is definitely bad, and causes memory corruption errors.

But I don't think it's a problem we can "solve" exactly.

-Steve

On Friday, 13 April 2018 at 22:29:25 UTC, Steven Schveighoffer wrote:
> On 4/13/18 5:57 PM, Jonathan M Davis wrote:
>> On Friday, April 13, 2018 16:15:21 Steven Schveighoffer via Digitalmars-d
>
>>> I don't know if the compiler can determine if a version statement
>>> affects the layout, I suppose it could, but it would have to compile
>>> both with and without the version to see. It's probably an intractable
>>> problem.
>> 
>> Also, does it really matter? If there's a mismatch, then you'll get a linker
>> error, so it's not like you're going to get subtle bugs out of the deal or
>> anything like that. I don't see why detection is an issue here.
>
> Well, for layout changes, there is no linker error. It's just one version of the code thinks the layout is one way, and another version thinks it's another way. This is definitely bad, and causes memory corruption errors.
>
> But I don't think it's a problem we can "solve" exactly.
>
> -Steve

@JonathanDavis, the original post goes through an example where you won't get a compile-time or link-time error...it results in a very bad runtime stack stomp.

@Steven You're just addressing the example I gave and not thinking of all the other ways version (or other compiler flags) could change things.  For example, you could have version code inside a template that changes mangling because it is no longer inferred to be pure/safe/whatever.

The point is, this is a solvable problem.  All we need to do is save the compiler configuration (i.e. versions/special flags that affects compilation) used when compiling a library and use that information when we are interpreting the module's source as as an "pre-compiled import". Interpreting a module with a different version than was compiled can create any error you can possibly come up with and could manifest at any time (i.e. compile-time, link time, runtime).

The jist is that if we don't solve this, then it's up to the applications to use the same versions that were used to compile all their pre-compiled D libraries...and if they don't...all bets are off.  They could run into any error at any time and the compiler/type system can't help them.

On 4/13/18 7:00 PM, Jonathan Marler wrote:

> @Steven You're just addressing the example I gave and not thinking of all the other ways version (or other compiler flags) could change things.  For example, you could have version code inside a template that changes mangling because it is no longer inferred to be pure/safe/whatever.

Yeah, but that results in a linker error. Your solution results in a linker error as well. Either way, you need to adjust your build. Trying to make the liner spit out a nice error is an exercise in futility.

> The point is, this is a solvable problem.  All we need to do is save the compiler configuration (i.e. versions/special flags that affects compilation) used when compiling a library and use that information when we are interpreting the module's source as as an "pre-compiled import". Interpreting a module with a different version than was compiled can create any error you can possibly come up with and could manifest at any time (i.e. compile-time, link time, runtime).

consider:

int libraryFunction(int x)
{
    version(UseSpecializedMethod)
    {
       // do it the specialized way
       ...
    }
    else
    {
       // do it the slow way
       ...
    }
}

Do we need to penalize user code that doesn't define the library-special version UseSpecializedMethod? Making code not link because it didn't define library specific implementation details the same as the library isn't going to help.

> The jist is that if we don't solve this, then it's up to the applications to use the same versions that were used to compile all their pre-compiled D libraries...and if they don't...all bets are off.  They could run into any error at any time and the compiler/type system can't help them.

For versions, it only makes a difference if the versions affect the public API (and in a silent way). I'm fine with linker errors to diagnose these.

Note: it's really bad form to make a library who has public API changes when you define different versions. It's why I'm trying to eliminate all of those cases for version(unittest) from phobos.

For compiler features, if you get different symbols from the exact same code (in other words, ALL code involved is exactly the same), then it may be useful to embed such a compilation linker mechanism to give a somewhat better linker error. For example, with dip1000, if a library function using dip1000 adds an attribute that normally wouldn't be added, you could include such a symbol, and then the linker failure would show that symbol missing (and hopefully clue in the user).

But even this has drawbacks -- what if you never call that function? Now you are having a linker error where there normally wouldn't be.

But there are actually a couple real ways to solve this, and they aren't simple. One is to invent our own linker/object format that allows embedding the stuff we want. Then you don't import source, you import the object (this is similar to Java).

The second is to actually spit out a specialized import file that puts the right attributes/definitions on the public API (and should include any implementation that needs to be inlined or templated).

-Steve

On Fri, Apr 13, 2018 at 11:00:20PM +0000, Jonathan Marler via Digitalmars-d wrote: [...]
> @JonathanDavis, the original post goes through an example where you won't get a compile-time or link-time error...it results in a very bad runtime stack stomp.

To put things in perspective, this is essentially the same problem in C/C++ as compiling your program with one version of header files, but linking against a different version of the shared library.  Well, this isn't restricted to C/C++, but affects basically anything that uses the OS's dynamic linker.  It's essentially an ABI change that wasn't properly reflected in the API, thus causing problems at runtime.

The whole thing about sonames and shared library versioning is essentially to solve this problem.  But even then, it's not a complete solution (e.g., I can still compile against the wrong version of a header file, and get a struct definition of the wrong size vs. the one expected by the linked shared library).

Basically, it boils down to, "don't make your build system do this".


[...]
> The point is, this is a solvable problem.  All we need to do is save the compiler configuration (i.e. versions/special flags that affects compilation) used when compiling a library and use that information when we are interpreting the module's source as as an "pre-compiled import".  Interpreting a module with a different version than was compiled can create any error you can possibly come up with and could manifest at any time (i.e.  compile-time, link time, runtime).

The problem with this "solution" is that it breaks valid use cases.  For example, a shared library can have multiple versions, e.g., one compiled with debugging symbols, another with optimization flags, but as long as the ABI remains unchanged, it *should* be valid to link the program against these different versions of the library.

One example where you really don't want to insist on identical compiler flags is if you have a plugin system where plugins are 3rd party supplied, compiled against a specific ABI.  It seems impractically heavy-handed to ask all your 3rd party plugin writers to recompile their plugins just because you changed a compile flag in your application that, ultimately, doesn't even change the ABI anyway.


> The jist is that if we don't solve this, then it's up to the applications to use the same versions that were used to compile all their pre-compiled D libraries...and if they don't...all bets are off. They could run into any error at any time and the compiler/type system can't help them.

Linking objects compiled with different flags, in general, is not recommended, but in the cases where you *do* want to do that, it's essential that you *should* be able to choose to do so, without running into the red tape of the compiler playing nanny and stopping you from doing something that "might" "possibly" be dangerous.


T

-- 
Help a man when he is in trouble and he will remember you when he is in trouble again.

On Friday, 13 April 2018 at 23:36:46 UTC, H. S. Teoh wrote:
> On Fri, Apr 13, 2018 at 11:00:20PM +0000, Jonathan Marler via Digitalmars-d wrote: [...]
>> @JonathanDavis, the original post goes through an example where you won't get a compile-time or link-time error...it results in a very bad runtime stack stomp.
>
> To put things in perspective, this is essentially the same problem in C/C++ as compiling your program with one version of header files, but linking against a different version of the shared library.  Well, this isn't restricted to C/C++, but affects basically anything that uses the OS's dynamic linker.  It's essentially an ABI change that wasn't properly reflected in the API, thus causing problems at runtime.
>
> The whole thing about sonames and shared library versioning is essentially to solve this problem.  But even then, it's not a complete solution (e.g., I can still compile against the wrong version of a header file, and get a struct definition of the wrong size vs. the one expected by the linked shared library).
>
> Basically, it boils down to, "don't make your build system do this".
>
>
> [...]
>> The point is, this is a solvable problem.  All we need to do is save the compiler configuration (i.e. versions/special flags that affects compilation) used when compiling a library and use that information when we are interpreting the module's source as as an "pre-compiled import".  Interpreting a module with a different version than was compiled can create any error you can possibly come up with and could manifest at any time (i.e.  compile-time, link time, runtime).
>
> The problem with this "solution" is that it breaks valid use cases.  For example, a shared library can have multiple versions, e.g., one compiled with debugging symbols, another with optimization flags, but as long as the ABI remains unchanged, it *should* be valid to link the program against these different versions of the library.
>
> One example where you really don't want to insist on identical compiler flags is if you have a plugin system where plugins are 3rd party supplied, compiled against a specific ABI.  It seems impractically heavy-handed to ask all your 3rd party plugin writers to recompile their plugins just because you changed a compile flag in your application that, ultimately, doesn't even change the ABI anyway.

You've missed part of the solution. The solution doesn't require you to compile with the same flags, what it does it takes the flags that were used to compile the modules you're linking to and interprets their "import source code" the same way it was interpreted when it was compiled.

If the precompiled module was compiled with the debug version, the `version(debug)` blocks will be enabled in the imported module source code whether or not you are compiling your application with debug enabled.  This guarantees that the source is an accurate representation of the precompiled library you'll be linking to later.

By the way...you're right that C/C++ suffer from the same problems with header files :)