Hi,

Can I ask for feedback on what people expect an optional/maybe type to have, and how to behave. There have been a number of discussions on the forums about it so far, and attempted PRs in phobos. And I currently maintain one that I've been using in-house very happily (backend server with vibe-d). I want to nail it down further and get to a version 1.0.0 so feedback and comments would be highly appreciated.

Optional semantics:
===================

So first, the semantics of Optional!T (these may or may not already be implemented):
* Is at least a forward range
* Equality with itself, T, and sentinel-type "none", which checks if optional is empty or not
* Assignment to T, `none`, or Optional!T
* Includes type constructors some(x) and no!T
* Forwards operators to T if T implements those operators
  E.g. some(1)++ and no!int++ both work
  E.g. If T has an opCall(...), then some(x)(...) and no!T()(...) both work
* Null pointers are valid objects, so some!(int*)(null) is non empty
* Null classses and interfaces are empty, so some!Class(null) is empty

Optional utilities:
===================

Two utilities are included. Optional chaining and a match function. Optional chaining allows you to go through an object's hierarchy and get an optional back at the end:

oc(obj).property.function()

If obj is none, then the end result is no!T where T is the return type of function. Else it has the actual value if the chain goes all the way though. This will also work if property is in turn an Optional itself. Secondly, the chaining function is also provided for NullableT and reference type.

The match function takes two handlers (lambda aliases) and calls the correct one based on if the Optional!T is empty or not.

orElse semantics:
=================

orElse will either get the front of a range or the range itself depending on the alternative value and also works on reference types. So:

Range!int range1, range2;
range1.orElse(range2) // returns range1 if range1 is not empty, else range2
range1.orElse(8) // returns front or range1 if non empty, else 8.

In essence, it also coalesces. I was on the fence on this, but it's turning out (again in our project) to be very convenient. I'm considering doing this for primitive types like int, float, or anything that can be cast to bool (while taking things like NaN in to account for e.g.)

So orElse works on a number of types and these are each types's semantics:

* If T is a reference type, val1.orElse(val2) will return val1 if (val1 !is null), else val2
* If T is a range, the example above shows the semantics
* If T is an Optional!U then it is treated separately in order to work with const Optional!U. The value returned by orElse is determined by the alternative value. If it is a U then optional.front will be returned, if it is an Optional!U then an optional will be returned. So the same semantics of range.
* If T is Nullable!U, then isNull is checked. If it is false and alternative value is a U, then nullable.get is returned. If T is another Nullable, then the return type is a Nullable.

I may have forgotten some stuff but I think that's everything.

Thanks in advance!

PS: I realize an obvious first comment is "what about Nullable!T". A number of reasons: 1) it doesn't have a range interface 2) the reference semantics are not desirable for me and writing generic code is awkward (https://github.com/aliak00/optional#what-about-stdtypeconsnullable), 3) alias this, although deprecated, breaks it completely, 4) maintaining a dub package feels much more productive.

On Saturday, 27 July 2019 at 13:17:32 UTC, aliak wrote:
> Hi,
>
> Can I ask for feedback on what people expect an optional/maybe type to have, and how to behave. There have been a number of discussions on the forums about it so far, and attempted PRs in phobos. And I currently maintain one that I've been using in-house very happily (backend server with vibe-d). I want to nail it down further and get to a version 1.0.0 so feedback and comments would be highly appreciated.

I think it is an awesome library and I an happy someone is spearheading this. Before this I used Optional, but it ain't as nice.

> Optional utilities:
> ===================
>
> Two utilities are included. Optional chaining and a match function. Optional chaining allows you to go through an object's hierarchy and get an optional back at the end:
>
> oc(obj).property.function()

In Scala you have to map constantly. This is way much better.

> orElse semantics:
> =================
>
> orElse will either get the front of a range or the range itself depending on the alternative value and also works on reference types.

Nice addition.

On Saturday, 27 July 2019 at 13:17:32 UTC, aliak wrote:
> Hi,
>
> Can I ask for feedback on what people expect an optional/maybe type to have, and how to behave. There have been a number of discussions on the forums about it so far, and attempted PRs in phobos.

Would be nice to have functionality similar to Optional from java 8+, such as orElseThrow, orElseGet, ifPresent etc. Also would be nice if it worked perfectly with immutable/const data, and when Optional itself is immutable/const. Another nice feature would be conversion from immutable to const optional, and from immutable/const optional to mutable Optional with const payload through probably copy constructor and opCast methods.

Best regards,
Alexandru

On Saturday, 27 July 2019 at 13:17:32 UTC, aliak wrote:
> * Null pointers are valid objects, so some!(int*)(null) is non empty
> * Null classses and interfaces are empty, so some!Class(null) is empty

I think it would be better to avoid weird special cases like this. The optional package already includes a NotNull type, so users can write `Optional!(NotNull!Class)` to opt into your proposed semantics explicitly.

> orElse semantics:
> =================
>
> [...]
>
> So orElse works on a number of types and these are each types's semantics:
>
> * If T is a reference type, val1.orElse(val2) will return val1 if (val1 !is null), else val2
> * If T is a range, the example above shows the semantics
> * If T is an Optional!U then it is treated separately in order to work with const Optional!U. The value returned by orElse is determined by the alternative value. If it is a U then optional.front will be returned, if it is an Optional!U then an optional will be returned. So the same semantics of range.
> * If T is Nullable!U, then isNull is checked. If it is false and alternative value is a U, then nullable.get is returned. If T is another Nullable, then the return type is a Nullable.

My first impression, upon reading this, is that I'm going to have to refer to the documentation every time I use `orElse` in order to be sure I'm doing it right. To me, this is a smell: if it takes me this long to describe what ought to be a very simple utility function, I can tell I've made it too complicated.

As far as concrete advice: the overloads for reference types and ranges seem completely unrelated to Optional, so it seems like an odd choice to include them in the "optional" package. The overloads for Optional and Nullable make sense to include (though the Nullable one really *ought* to be in Phobos), but special-casing them to wrap the return value is too "magic" for my tastes. Much better to just let people write `some(val1.orElse(val2))` if that's what they want--it's only a few more characters, and it makes it obvious at a glance what the code is doing.

On Saturday, 27 July 2019 at 14:46:55 UTC, Alexandru Ermicioi wrote:
> On Saturday, 27 July 2019 at 13:17:32 UTC, aliak wrote:
>> Hi,
>>
>> Can I ask for feedback on what people expect an optional/maybe type to have, and how to behave. There have been a number of discussions on the forums about it so far, and attempted PRs in phobos.
>
> Would be nice to have functionality similar to Optional from java 8+, such as orElseThrow, orElseGet, ifPresent etc. Also

A lot of the java stuff is covered with std.algorithm and orElse

auto a = some(3);
auto gotValue = a.orElse(7); // orElseGet
a.each!(a => writeln(a)); // ifPresent

orElseThrow on the other hand would have to be added separately. I use that quite often as well but I can't work it in to the provided orElse because "throw blah" is not an expression. I have another experimental thing where that's available though -> https://aliak00.github.io/ddash/ddash/utils/orelse/orElseThrow.html (but that library is not the most stable for now)

One other way would be to add an expression throw:

off top of me head:

auto throw_(T : Throwable)(lazy T e) { throw e(); }

Then this would work:

a.orElse(throw_(new Exception("")));

> would be nice if it worked perfectly with immutable/const data, and when Optional itself is immutable/const. Another nice

You can currently wrap cost/immutable data. I try to test to ensure things work with qualified optionals [0], but it's tricky with the const system in D, and with how inout behaves with wrapper objects [1, 2]. The other problem here is that const ranges are not ranges. So maybe a decay function on an optional type?

const a = some(3);
a.decay.map...

[0]: https://github.com/aliak00/optional/blob/de223a7d63d346386b0d02d119cf4bdd79366299/tests/optional.d#L11
[1]: https://issues.dlang.org/show_bug.cgi?id=19126
[2]: https://issues.dlang.org/show_bug.cgi?id=19125

> feature would be conversion from immutable to const optional, and from immutable/const optional to mutable Optional with const payload through probably copy constructor and opCast methods.

Hmm. Yes. The latter can probably be handled with the "decay-like" function. Isn't the former handled by D anyway? immutable is implicitly convertible to const?

immutable a = some(3);
const Optional!int b = a;

>
> Best regards,
> Alexandru

Thanks for the feedback!

Am 27.07.19 um 15:17 schrieb aliak:
> Hi,
> 
> Can I ask for feedback on what people expect an optional/maybe type to have, and how to behave. There have been a number of discussions on the forums about it so far, and attempted PRs in phobos. And I currently maintain one that I've been using in-house very happily (backend server with vibe-d). I want to nail it down further and get to a version 1.0.0 so feedback and comments would be highly appreciated.
> 
> Optional semantics:
> ===================
> 
> So first, the semantics of Optional!T (these may or may not already be
> implemented):
> * Is at least a forward range
> * Equality with itself, T, and sentinel-type "none", which checks if
> optional is empty or not
> * Assignment to T, `none`, or Optional!T
> * Includes type constructors some(x) and no!T
> * Forwards operators to T if T implements those operators
>   E.g. some(1)++ and no!int++ both work
>   E.g. If T has an opCall(...), then some(x)(...) and no!T()(...) both work
> * Null pointers are valid objects, so some!(int*)(null) is non empty
> * Null classses and interfaces are empty, so some!Class(null) is empty
> 
> Optional utilities:
> ===================
> 
> Two utilities are included. Optional chaining and a match function. Optional chaining allows you to go through an object's hierarchy and get an optional back at the end:
> 
> oc(obj).property.function()
> 
> If obj is none, then the end result is no!T where T is the return type of function. Else it has the actual value if the chain goes all the way though. This will also work if property is in turn an Optional itself. Secondly, the chaining function is also provided for NullableT and reference type.
> 
> The match function takes two handlers (lambda aliases) and calls the correct one based on if the Optional!T is empty or not.
> 
> orElse semantics:
> =================
> 
> orElse will either get the front of a range or the range itself depending on the alternative value and also works on reference types. So:
> 
> Range!int range1, range2;
> range1.orElse(range2) // returns range1 if range1 is not empty, else range2
> range1.orElse(8) // returns front or range1 if non empty, else 8.
> 
> In essence, it also coalesces. I was on the fence on this, but it's turning out (again in our project) to be very convenient. I'm considering doing this for primitive types like int, float, or anything that can be cast to bool (while taking things like NaN in to account for e.g.)
> 
> So orElse works on a number of types and these are each types's semantics:
> 
> * If T is a reference type, val1.orElse(val2) will return val1 if (val1
> !is null), else val2
> * If T is a range, the example above shows the semantics
> * If T is an Optional!U then it is treated separately in order to work
> with const Optional!U. The value returned by orElse is determined by the
> alternative value. If it is a U then optional.front will be returned, if
> it is an Optional!U then an optional will be returned. So the same
> semantics of range.
> * If T is Nullable!U, then isNull is checked. If it is false and
> alternative value is a U, then nullable.get is returned. If T is another
> Nullable, then the return type is a Nullable.
> 
> I may have forgotten some stuff but I think that's everything.
> 
> Thanks in advance!
> 
> PS: I realize an obvious first comment is "what about Nullable!T". A number of reasons: 1) it doesn't have a range interface 2) the reference semantics are not desirable for me and writing generic code is awkward (https://github.com/aliak00/optional#what-about-stdtypeconsnullable), 3) alias this, although deprecated, breaks it completely, 4) maintaining a dub package feels much more productive.

orElse currently has an overload that takes a callable that returns a
fallback value as template parameter. I think something like this is
needed, but it might be better to give it a different name (in Java,
Scala etc. this is called orElseGet). Other convenience utilities might
be nice, e.g. ifPresent, ifPresentOrElse, orElseThrow etc. (take Java's
Optional or Scala's Option as reference). These are all very easy to
implement and I have already done so several times when using your library.

I also think that your current orElse semantics are a bit too complicated: There is simply too much stuff (with slighty differing behavior) in that one single function (I know, it is several overloads, but the user does not care). E.g. I think the overload that takes another range (or Optional or Nullable) should be a seperate function because it actually has different semantics. All overloads should have the same general behavior, in my opinion. In Java, this function is called "or".

Otherwise, I actually quite like the semantics of your library (and I use it regularly). Thanks for your work!

On Saturday, 27 July 2019 at 15:07:56 UTC, Paul Backus wrote:
> On Saturday, 27 July 2019 at 13:17:32 UTC, aliak wrote:
>> * Null pointers are valid objects, so some!(int*)(null) is non empty
>> * Null classses and interfaces are empty, so some!Class(null) is empty
>
> I think it would be better to avoid weird special cases like this. The optional package already includes a NotNull type, so users can write `Optional!(NotNull!Class)` to opt into your proposed semantics explicitly.

Egad! I just (i.e. today/yesterday) removed NotNull as I could not figure out how to guarantee NotNull semantics without it just being awkward to use. I'm not sure it's generally used either. But that's based on what I've been using myself, and issues and prs in the repo - so not a very wide view.

But maybe you're right in making it the same. I'm a bit weary because null classes inside optional values has bitten me and colleagues multiple times in Swift, Scala, and Kotlin over the past few years. Swift got rid of the issue in version 2 or something. Scala and Kotlin still have it because of necessary Java interop.

>
>> orElse semantics:
>> =================
>>
>> [...]
>>
>> So orElse works on a number of types and these are each types's semantics:
>>
>> * If T is a reference type, val1.orElse(val2) will return val1 if (val1 !is null), else val2
>> * If T is a range, the example above shows the semantics
>> * If T is an Optional!U then it is treated separately in order to work with const Optional!U. The value returned by orElse is determined by the alternative value. If it is a U then optional.front will be returned, if it is an Optional!U then an optional will be returned. So the same semantics of range.
>> * If T is Nullable!U, then isNull is checked. If it is false and alternative value is a U, then nullable.get is returned. If T is another Nullable, then the return type is a Nullable.
>
> My first impression, upon reading this, is that I'm going to have to refer to the documentation every time I use `orElse` in order to be sure I'm doing it right. To me, this is a smell: if it takes me this long to describe what ought to be a very simple utility function, I can tell I've made it too complicated.

Very fair point. What if the description was simplified?

The thing is I want it to "just work" based on the call site. So if a user does a.orElse(literal) on a container type, it's pretty obvious what they want. If a user does a.orElse(container), it's again (I think) obvious what they want. But essentially those are the only two cases.  And since reference types are not containers, then only the second case applies. Does that make it simpler?

So make coalescing separate from orElse-ing is basically the gist of this right?

>
> As far as concrete advice: the overloads for reference types and ranges seem completely unrelated to Optional, so it seems like an odd choice to include them in the "optional" package. The overloads for Optional and Nullable make sense to include (though the Nullable one really *ought* to be in Phobos), but special-casing them to wrap the return value is too "magic" for

This is true. They are kinda unrelated and just there for convenience because I didn't know where else to get the functionality from. I actually played around with having the orelse in a completely separate package, and providing a hook:

auto ref orElse(alias elseValue, T)(auto ref T value) {
  static if (hasMember!(T, "hookOrElse") {
    return value.hookOrElse!elseValue;
  }
  ...
}

And then implementing the hook in the optional package. But this bit me: https://forum.dlang.org/thread/lddwnnzlktszwspldxqu@forum.dlang.org

> my tastes. Much better to just let people write `some(val1.orElse(val2))` if that's what they want--it's only a few more characters, and it makes it obvious at a glance what the code is doing.

Sorry, what was the `some(val1.orElse(val2))`? Did you mean that instead of range1.orElse(range2) or range1.orElse(elementOfRange)?

And thanks for the feedback!

Am 27.07.19 um 18:26 schrieb aliak:
> 
> A lot of the java stuff is covered with std.algorithm and orElse
> 
> auto a = some(3);
> auto gotValue = a.orElse(7); // orElseGet
> a.each!(a => writeln(a)); // ifPresent

I still think it might be valuable to have separate names for these things. As mentioned in my other answer, I think that orElse is overloaded too much. And while sometimes it is really nice to think of an optional as a range, it is not always straight forwards and in that case using something like "each" is a bit awkward. It has the benefit of working in combination with other range algorithms though, so maybe we should implement "ifPresent" etc. to also work in ranges (in the same way you did for "orElse", i.e. only using the first element). Another utility that would be very nice is ifPresentOrElse. It is very similar to match, but does never return anything and for that reason, the two template parameters do not need to have the same return type:

```
int x = 0;

void foo(ref int value)
{
    value = -1;
}

some(1).match!(
    value => x += value,
    () => foo(x) // won't compile because the return types differ
);
```

The solution is to make the return types the same, but this prevents us from using the error notation for lambdas, which is kind of annoying:

```
int x = 0;

void foo(ref int value)
{
    value = -1;
}

some(1).match!(
    (value) {x += value;},
    () => foo(x) // won't compile because the return types differ
);
```

ifPresentOrElse would basically do this for us:

```
void ifPresentOrElse(alias fun1, alias fun2, T)(inout auto ref
Optional!T opt)
{
	opt.match!(
    		(value) { fun1(value); },
    		() { fun2(); }
	);
}

int x = 0;

void foo(ref int value)
{
    value = -1;
}

some(1).ifPresentOrElse!(
    value => x += value,
    () => foo(x) // everything fine now :)
);
```

On Saturday, 27 July 2019 at 15:07:56 UTC, Paul Backus wrote:
> On Saturday, 27 July 2019 at 13:17:32 UTC, aliak wrote:
>> * Null pointers are valid objects, so some!(int*)(null) is non empty
>> * Null classses and interfaces are empty, so some!Class(null) is empty
>
> I think it would be better to avoid weird special cases like this. The optional package already includes a NotNull type, so users can write `Optional!(NotNull!Class)` to opt into your proposed semantics explicitly.

It's surprising that pointers and references are special-cased differently, right.

But I suggest that null is the same as empty, both for pointers and for references. I.e., the original post's proposal should change its pointer semantics, some!(int*)(null) == no!(int*).

What was the original reason to treat pointers and class references differently?

Reason to treat null as empty, always: We want at most one layer of nullability/emptiness, and usually no nullability at all, because we want to avoid nesting of "if it exists, do stuff, otherwise do nothing". All possible nullability/emptiness should be handled at once.

This usual case (at most one layer of nullability/emptiness) should be quickest to write. If Optional!Class introduced emptiness different from nullability, then Optional!(NotNull!Class) would be standard usage because people want at most one layer of emptiness, but then standard usage would not be succinct.

I accept that Optional!(NotNull!Class) is more explicit, but this is only because of D's lack of non-nullable references, which is the problem in the first place. NotNull is an abstraction inversion: It builds a non-nullable type (the less complex type because it allows fewer values and is crash-free to call methods) from a nullable type (which instead should be a sum type of the nonexisting non-nullable type and empty).

I already use Optional!Class for single layer of nullability, and Class for non-null by convention. 95 % of the time, I want non-nullable. Exported methods, then, need in/out contracts to check for non-nullability. These contracts should ideally be inserted by the compiler on seeing a language-builtin non-nullable reference.

-- Simon

Am 27.07.19 um 18:58 schrieb Johannes Loher:
> The solution is to make the return types the same, but this prevents us from using the error notation for lambdas, which is kind of annoying:

This should read: "[...] prevents us from using the _arrow_ notation [...]".