On Tuesday, 17 May 2016 at 14:59:45 UTC, Ola Fosheim Grøstad wrote:
> There are lots of algorithms that will break if you randomly switch precision of different expressions.

There is nothing "random" about increasing precision till the end, it follows a well-defined rule.

> Heck, nearly all of the SIMD optimizations on differentials that I use will break if one lane is computed with a different precision than the other lanes. I don't give a rats ass about increased precision. I WANT THE DAMN LANES TO BE IN THE SAME PHASE (or close to it)!! Phase-locking is much more important than value accuracy.

So you're planning on running phase-locking code partially in CTFE and runtime and it's somehow very sensitive to precision?  If your "phase-locking" depends on producing bit-exact results with floating-point, you're doing it wrong.

>> Now, people can always make mistakes in their implementation and unwittingly depend on lower precision somehow, but that _should_ fail.
>
> People WILL make mistakes, but if you cannot control precision then you cannot:
>
> 1. create a reference implementation to compare with
>
> 2. unit test floating point code in a reliable way
>
> 3. test for convergence/divergence in feedback loops (which can have _disastrous_ results and could literally ruin your speakers/hearing in the case of audio).

If any of this depends on comparing bit-exact floating-point results, you're doing it wrong.

>> None of this is controversial to me: you shouldn't be comparing floating-point numbers with anything other than approxEqual,
>
> I don't agree.
>
> 1. Comparing typed constants for equality should be unproblematic. In D that is broken.

If the constant is calculated rather than a literal, you should be checking using approxEqual.

> 2. Testing for zero is a necessity when doing division.

If the variable being tested is calculated, you should be using approxEqual.

> 3. Comparing for equality is the same as subtraction followed by testing for zero.

So what?  You should be using approxEqual.

> So, the rule is: you shouldn't compare at all unless you know the error bounds, but that depends on WHY you are comparing.

No, you should always use error bounds.  Sometimes you can get away with checking bit-exact equality, say for constants that you defined yourself with no calculation, but it's never a good practice.

> However, with constants/sentinels and some methods you do know... Also, with some input you do know that the algorithm WILL fail for certain values at a _GIVEN_ precision. Testing for equality for those values makes a lot of sense, until some a**hole decides to randomly "improve" precision where it was typed to something specific and known.
>
> Take this:
>
> f(x) = 1/(2-x)
>
> Should I not be able to test for the exact value "2" here?

It would make more sense to figure out what the max value of f(x) is you're trying to avoid, say 1e6, and then check for approxEqual(x, 2, 2e-6).  That would make much more sense than only avoiding 2, when an x that is arbitrarily close to 2 can also blow up f(x).

> I don't see why "1.3" typed to a given precision should be different. You want to force me to a more than 3x more expensive test just to satisfy some useless FP semantic that does not provide any real world benefits whatsoever?

Oh, it's real world alright, you should be avoiding more than just 2 in your example above.

>> increasing precision should never bother your algorithm, and a higher-precision, common soft-float for CTFE will help cross-compiling and you'll never notice the speed hit.
>
> Randomly increasing precision is never a good idea. Yes, having different precision in different code paths can ruin the quality of both rendering, data analysis and break algorithms.
>
> Having infinite precision untyped real that may downgrade to say 64 bits mantissa is acceptable. Or in the case of Go, a 256 bit mantissa. That's a different story.
>
> Having single precision floats that randomly are turned into arbitrary precision floats is not acceptable. Not at all.

Simply repeating the word "random" over and over again does not make it so.

On 5/17/2016 5:22 PM, H. S. Teoh via Digitalmars-d wrote:
> Wasn't Manu's original complaint that, given a particular piece of FP
> code that uses floats, evaluating that code at compile-time may produce
> different results than evaluating it at runtime, because (as you're
> proposing) the compiler will use higher precision than specified for
> intermediate results?  Of course, the compile-time answer is arguably
> "more correct" because it has less roundoff error, but the point here is
> not how accurate that answer is, but that it *doesn't match the runtime
> results*. This mismatch, from what I understand, is what causes the
> graphical glitches that Manu was referring to.

Except this happens with C/C++ too, and in fact Manu wasn't using D.

> According to your prescription, then, the runtime code should be "fixed"
> to use higher precision, so that it will also produce the same, "more
> correct" answer.  But unfortunately, that's not workable because of the
> performance implications. At the end of the day, nobody cares whether a
> game draws a polygon with the most precise coordinates, what people *do*
> care is that there's a mismatch between the "more correct" and "less
> correct" rendering of the polygon (produced, respectively, from CTFE and
> from runtime) that causes a visually noticeable glitch. It *looks*
> wrong, no matter how much you may argue that it's "more correct". You
> are probably right scientifically, but in a game, people are concerned
> about what they see, not whether polygon coordinates have less roundoff
> error at CTFE vs. at runtime.

That wasn't my prescription. My prescription was either changing the algorithm so it was not sensitive to exact bits-in-last-place, or to use roundToFloat() and roundToDouble() functions.

On Wednesday, 18 May 2016 at 03:01:14 UTC, Joakim wrote:
> There is nothing "random" about increasing precision till the end, it follows a well-defined rule.

Can you please quote that well-defined rule?

It is indeed random, or arbitrary (which is the same thing):

if(x<0){
  // DMD choose 64 bit mantissa
  const float y = ...
  ...

} else {
  // DMD choose 24 bit mantissa
  float y = ...
  ...
}

How is this not arbitrary?

If x is the amplitude then a flaw like this can cause a DC offset to accumulate and you end up with reduced precision, not improved precision. A DC filter at the end does not help on this precision loss.

> So you're planning on running phase-locking code partially in CTFE and runtime and it's somehow very sensitive to precision?  If your "phase-locking" depends on producing bit-exact results with floating-point, you're doing it wrong.

I am not doing anything wrong. D is doing it wrong. If you add different deltas then you will get drift. So, no improved precision in calculating some deltas is not improving the accuracy. It makes it worse.

> If any of this depends on comparing bit-exact floating-point results, you're doing it wrong.

It depends on the unit tests running with the exact same precision as the production code.

Fast floating point code depends on the specifics of the hardware. A system level language should not introduce a different kind of bias that isn't present in the hardware!

D is doing it wrong because it makes it is thereby forcing programmers to use algorithms that are 10-100x slower to get reliable results.

That is _wrong_.

> If the constant is calculated rather than a literal, you should be checking using approxEqual.

No. 1+2+3+4 is exact on all floating point units I know of.

>> 3. Comparing for equality is the same as subtraction followed by testing for zero.
>
> So what?  You should be using approxEqual.

No.

>> So, the rule is: you shouldn't compare at all unless you know the error bounds, but that depends on WHY you are comparing.
>
> No, you should always use error bounds.  Sometimes you can get away with checking bit-exact equality, say for constants that you defined yourself with no calculation, but it's never a good practice.

It is if you know WHY you are doing a test.

Btw, have you ever tried to prove error bounds for an iterative method?
You actually think most people prove them to be correct?

Or perhaps almost all of them just pick a number out of thin air which they think will work out and rely on testing their code?

Well, the latter is no better than checking for exact equality. And I can assure you that the vast majority of programmers do not prove error bounds with the level of rigour it takes to get it correct.

The reality is that it is common practice to write code that seems to work. But that does not make it correct. However, making it correct is way too time consuming and often not worth the trouble. So people rely on testing. Floating point code is no different.

But with D semantics you cannot rely on testing. That's bad, because most people write incorrect code. Whether they are experts or not. (it is only matter of difference in frequency)

>> f(x) = 1/(2-x)
>>
>> Should I not be able to test for the exact value "2" here?
>
> It would make more sense to figure out what the max value of f(x) is you're trying to avoid, say 1e6, and then check for approxEqual(x, 2, 2e-6).  That would make much more sense than only avoiding 2, when an x that is arbitrarily close to 2 can also blow up f(x).

I am trying to avoid an exception, not a value.

> Oh, it's real world alright, you should be avoiding more than just 2 in your example above.

Which number would that be?

> Simply repeating the word "random" over and over again does not make it so.

That's right. It is DMD that makes it so, not my words.

However, in order to reject what other say, you have to make an argument. And in this case we have:

1. A system level programming language that claims to excel at floating point.
2. Hardware platforms with specific behaviour.

Unfortunately 2 is true, but not 1.

D is not matching up to the minimum requirements for people wanting to write fast and reliable floating point code for a given platform.

According to the D spec, the compiler could schedule typed single precision floating point calculations to two completely different floating point units (and yes, there are platforms that provide multiple incompatible floating point units with widely differing characteristics).

That is random.
And so is "float" behaving differently than "const float".

On Wednesday, 18 May 2016 at 05:40:57 UTC, Walter Bright wrote:
> That wasn't my prescription. My prescription was either changing the algorithm so it was not sensitive to exact bits-in-last-place, or to use roundToFloat() and roundToDouble() functions.

With Manu's example, that would have been a good old fashioned matrix multiply to transform a polygon vertex from local space to screen space, with whatever other values were required for the render effect. The problem there being that the hardware itself only calculated 24 bits of precision while dealing with 32 bit values. Such a solution was not an option.

Gaming hardware has gotten a lot less cheap and nasty. But Manu brought it up because it is conceptually the same problem as 32/64 bit run time values vs 80 bit compile time values. Every solution offered here either comes down to "rewrite your code" or "increase code complexity", neither of which is often an option (changing the code in Manu's example required a seven+ hour compile time each iteration of the code; and being a very hot piece of code, it needed to be as simple as possible to maintain speed). Unlike the hardware, game programming has not gotten less cheap nor nasty. We will cheat our way to the fastest performing code using whatever trick we can find that doesn't cause the whole thing to come crashing down. The standard way for creating float values at compile time is to calculate them manually at the correct precision and put a #define in with that value. Being unable to specify/override compile time precision means that the solution is to declare enums in the exact same manner, and might result in more maintenance work if someone decides they want to switch from float to double etc. for their value.

On 5/17/2016 11:15 PM, Ethan Watson wrote:
> With Manu's example, that would have been a good old fashioned matrix multiply
> to transform a polygon vertex from local space to screen space, with whatever
> other values were required for the render effect. The problem there being that
> the hardware itself only calculated 24 bits of precision while dealing with 32
> bit values. Such a solution was not an option.

I don't understand the 24 vs 32 bit value thing. There is no 32 bit mantissa floating point type. Floats have 24 bit mantissas, doubles 52.

> Gaming hardware has gotten a lot less cheap and nasty. But Manu brought it up
> because it is conceptually the same problem as 32/64 bit run time values vs 80
> bit compile time values. Every solution offered here either comes down to
> "rewrite your code" or "increase code complexity", neither of which is often an
> option (changing the code in Manu's example required a seven+ hour compile time
> each iteration of the code; and being a very hot piece of code, it needed to be
> as simple as possible to maintain speed). Unlike the hardware, game programming
> has not gotten less cheap nor nasty. We will cheat our way to the fastest
> performing code using whatever trick we can find that doesn't cause the whole
> thing to come crashing down. The standard way for creating float values at
> compile time is to calculate them manually at the correct precision and put a
> #define in with that value. Being unable to specify/override compile time
> precision means that the solution is to declare enums in the exact same manner,
> and might result in more maintenance work if someone decides they want to switch
> from float to double etc. for their value.

I do not understand why the compile time version cannot use roundToFloat() in places where it matters. And if the hardware was using a different precision than float/double, which appears to have been the case, the code would have to be written to account for that anyway.

In any case, the problem Manu was having was with C++. The precision of calculations is implementation defined in C++, and does vary all over the place depending on compiler brands, compiler versions, compiler switches, and exactly how the code is laid out. There can also be differences in how the FP hardware works on the compiler host machine and the target machine.

My proposal would make the behavior more consistent than C++, not less.

Lastly, it is hard to make suggestions on how to deal with the problem without seeing the actual offending code. There may very well be something else going on, or some simple adjustment that can be made.

One way that *will* make the results exactly the same as on the target hardware is to actually run the code on the target hardware, save the results to a file, and incorporate that file in the build.

On Wednesday, 18 May 2016 at 05:49:16 UTC, Ola Fosheim Grøstad wrote:
> On Wednesday, 18 May 2016 at 03:01:14 UTC, Joakim wrote:
>> There is nothing "random" about increasing precision till the end, it follows a well-defined rule.
>
> Can you please quote that well-defined rule?

It appears to be "the compiler carries everything internally to 80 bit precision, even if they are typed as some other precision."
http://forum.dlang.org/post/nh59nt$1097$1@digitalmars.com

> It is indeed random, or arbitrary (which is the same thing):

No, they're not the same thing: rules can be arbitrarily set yet consistent over time, whereas random usually means both arbitrary and inconsistent over time.

> if(x<0){
>   // DMD choose 64 bit mantissa
>   const float y = ...
>   ...
>
> } else {
>   // DMD choose 24 bit mantissa
>   float y = ...
>   ...
> }
>
> How is this not arbitrary?

I believe that means any calculation used to compute y at compile-time will be done in 80-bit or larger reals, then rounded to a const float for run-time, so your code comments would be wrong.

> If x is the amplitude then a flaw like this can cause a DC offset to accumulate and you end up with reduced precision, not improved precision. A DC filter at the end does not help on this precision loss.

I don't understand why you're using const for one block and not the other, seems like a contrived example.  If the precision of such constants matters so much, I'd be careful to use the same const float everywhere.

>> So you're planning on running phase-locking code partially in CTFE and runtime and it's somehow very sensitive to precision?
>>  If your "phase-locking" depends on producing bit-exact results with floating-point, you're doing it wrong.
>
> I am not doing anything wrong. D is doing it wrong. If you add different deltas then you will get drift. So, no improved precision in calculating some deltas is not improving the accuracy. It makes it worse.

If matching such small deltas matters so much, I wouldn't be using floating-point in the first place.

>> If any of this depends on comparing bit-exact floating-point results, you're doing it wrong.
>
> It depends on the unit tests running with the exact same precision as the production code.

What makes you think they don't?

> Fast floating point code depends on the specifics of the hardware. A system level language should not introduce a different kind of bias that isn't present in the hardware!

He's doing this to take advantage of the hardware, not the opposite!

> D is doing it wrong because it makes it is thereby forcing programmers to use algorithms that are 10-100x slower to get reliable results.
>
> That is _wrong_.

If programmers want to run their code 10-100x slower to get reliably inaccurate results, that is their problem.

>> If the constant is calculated rather than a literal, you should be checking using approxEqual.
>
> No. 1+2+3+4 is exact on all floating point units I know of.

If you're so convinced it's exact for a few cases, then check exact equality there.  For most calculation, you should be using approxEqual.

> Btw, have you ever tried to prove error bounds for an iterative method?
> You actually think most people prove them to be correct?
>
> Or perhaps almost all of them just pick a number out of thin air which they think will work out and rely on testing their code?

No, I have never done so, and I'm well aware that most just pull the error bound they use out of thin air.

> Well, the latter is no better than checking for exact equality. And I can assure you that the vast majority of programmers do not prove error bounds with the level of rigour it takes to get it correct.

Even an unproven error bound is better than "exact" equality, which technically is really assuming that the error bound is smaller than the highest precision you can specify the number.  Since the real error bound is always larger than that, almost any error bound you pick will tend to be closer to the real error bound, or at least usually bigger and therefore more realistic, than checking for exact equality.

> The reality is that it is common practice to write code that seems to work. But that does not make it correct. However, making it correct is way too time consuming and often not worth the trouble. So people rely on testing. Floating point code is no different.
>
> But with D semantics you cannot rely on testing. That's bad, because most people write incorrect code. Whether they are experts or not. (it is only matter of difference in frequency)

You can still test with approxEqual, so I don't understand why you think that's not testing.  std.math uses feqrel, approxEqual, and equalsDigit and other such lower-precision checks extensively in its tests.

>>> f(x) = 1/(2-x)
>>>
>>> Should I not be able to test for the exact value "2" here?
>>
>> It would make more sense to figure out what the max value of f(x) is you're trying to avoid, say 1e6, and then check for approxEqual(x, 2, 2e-6).  That would make much more sense than only avoiding 2, when an x that is arbitrarily close to 2 can also blow up f(x).
>
> I am trying to avoid an exception, not a value.

That is the problem.  In the real world, all such formulas are approximations that only apply over a certain range.  If you were cranking that out by hand and some other calculation gave you an x really close to 2, say 2.0000035, you'd go back and check your math, as f(x) would blow up and give you unrealistically large numbers for the rest of your calculations.

The computer doesn't know that, so it will just plug that x in and keep cranking, till you get nonsense data out the end, if you don't tell it to check that x isn't too close to 2 and not just 2.

You have a wrong mental model that the math formulas are the "real world," and that the computer is mucking it up.  The truth is that the computer, with its finite maximums and bounded precision, better models _the measurements we make to estimate the real world_ than any math ever written.

>> Oh, it's real world alright, you should be avoiding more than just 2 in your example above.
>
> Which number would that be?

I told you, any numbers too close to 2.

>> Simply repeating the word "random" over and over again does not make it so.
>
> That's right. It is DMD that makes it so, not my words.
>
> However, in order to reject what other say, you have to make an argument. And in this case we have:
>
> 1. A system level programming language that claims to excel at floating point.
> 2. Hardware platforms with specific behaviour.
>
> Unfortunately 2 is true, but not 1.
>
> D is not matching up to the minimum requirements for people wanting to write fast and reliable floating point code for a given platform.

On the contrary, it is done because 80-bit is faster and more precise, whereas your notion of reliable depends on an incorrect notion that repeated bit-exact results are better.

> According to the D spec, the compiler could schedule typed single precision floating point calculations to two completely different floating point units (and yes, there are platforms that provide multiple incompatible floating point units with widely differing characteristics).

You noted that you don't care that the C++ spec says similar things, so I don't see why you care so much about the D spec now.  As for that scenario, nobody has suggested it.

> That is random.

It may be arbitrary, but it is not random unless it's inconsistently done.

> And so is "float" behaving differently than "const float".

I don't believe it does.

On Tuesday, 17 May 2016 at 20:50:06 UTC, Walter Bright wrote:
> On 5/16/2016 7:47 AM, Max Samukha wrote:
>> On Monday, 16 May 2016 at 14:21:34 UTC, Ola Fosheim Grøstad wrote:
>>
>>> C++17 is getting hex literals for floating point for a reason: accurate bit
>>> level representation.
>>
>> D has had hex FP literals for ages.
>
>
> Since the first version, if I recall correctly. Of course, C++ had the idea first!
>
> (Actually, the idea came from the NCEG (Numerical C Extensions Group) work in the early 90's, which was largely abandoned. C++ has been very slow to adapt to the needs of numerics programmers.)

The argument is not that C++ did anything first. The argument is that bit level precision for float is becoming the norm. Both in standards, languages and hardware. And that this makes it easier to write faster and more accurate code.

Please note that I don't complain about D providing 80 bit floats. Just don't apply it when I request 32 bit floats.

Some file formats have fields that are in the 80 bit intel format, so having access to it is a good thing.

On Wednesday, 18 May 2016 at 06:57:58 UTC, Walter Bright wrote:
> I don't understand the 24 vs 32 bit value thing. There is no 32 bit mantissa floating point type. Floats have 24 bit mantissas, doubles 52.

Not in the standards, no. But older gaming hardware was never known to be standards-conformant.

As it turns out, the original hardware manuals can be found on the internet.

https://www.dropbox.com/s/rsgx6xmpkf2zzz8/VU_Users_Manual.pdf

Relevant info copied from page 27:

Calculation
* A 24-bit calculation including hidden bits is performed, and the result is truncated.  The rounding-off operation in IEEE 754 is performed in the 0 direction, so the values for the least significant bit may vary.

> In any case, the problem Manu was having was with C++.

VU code was all assembly, I don't believe there was a C/C++ compiler for it.

> My proposal would make the behavior more consistent than C++, not less.

This is why I ask for a compiler option to make it consistent with the C++ floating point architecture I select. Making it better than C++ is great for use cases where you're not inter-opping with C++ extensively.

Although I do believe saying C++ is just clouding things here. Language doesn't matter, it's the runtime code using a different floating point instruction set/precision to compile time code that's the problem. See the SSE vs x87 comparisons posted in this thread for a concrete example. Same C++ code, different instruction sets and precisions.

Regardless. Making extra build steps with pre-calculated values or whatever is of course a workable solution, but it also raises the barrier of entry. You can't just, for example, select a compiler option in your IDE and have it just work. You have to go out of your way to make it work the way you want it to. And if there's one thing you can count on, it's end users being lazy.

On 5/18/2016 12:56 AM, Ethan Watson wrote:
> > In any case, the problem Manu was having was with C++.
> VU code was all assembly, I don't believe there was a C/C++ compiler for it.

The constant folding part was where, then?


> This is why I ask for a compiler option to make it consistent with the C++
> floating point architecture I select. Making it better than C++ is great for use
> cases where you're not inter-opping with C++ extensively.

Trying to make D behave exactly like various C++ compilers do, with all their semi-documented behavior and semi-documented switches that affect constant folding behavior, is a hopeless task.

I doubt various C++ compilers are this compatible, even if they follow the same ABI.

You're also asking for a mode where the compiler for one machine is supposed to behave like hand-coded assembler for another machine with a different instruction set. I doubt any compiler is going to deliver that, unless you create a customized compiler specifically for that.


> Although I do believe saying C++ is just clouding things here. Language doesn't
> matter, it's the runtime code using a different floating point instruction
> set/precision to compile time code that's the problem. See the SSE vs x87
> comparisons posted in this thread for a concrete example. Same C++ code,
> different instruction sets and precisions.
>
> Regardless. Making extra build steps with pre-calculated values or whatever is
> of course a workable solution, but it also raises the barrier of entry. You
> can't just, for example, select a compiler option in your IDE and have it just
> work. You have to go out of your way to make it work the way you want it to. And
> if there's one thing you can count on, it's end users being lazy.

From your description, there would have been a problem *regardless* of the precision used for constant folding. This is because the target hardware used truncation, and constant folding (in all compilers I know of) use rounding, and no compiler I know of allows controlling the rounding mode in constant folding.

On Wednesday, 18 May 2016 at 08:21:18 UTC, Walter Bright wrote:
> The constant folding part was where, then?

Probably on the EE, executed with different precision (32-bit) and passed over to the VU via one of its registers. Manu spent more time with that code than I did and can probably give exact details. But pasting the code, given it's proprietary code from a 15 year old engine, will be difficult at best considering the code is likely on a backup tape somewhere.

> You're also asking for a mode where the compiler for one machine is supposed to behave like hand-coded assembler for another machine with a different instruction set.

Actually, I'm asking for something exactly like the arch option for MSVC/-mfmath option for GCC/etc, and have it respect that for CTFE.