On Thursday, 18 May 2017 at 08:24:18 UTC, Walter Bright wrote:
> On 5/17/2017 10:07 PM, Patrick Schluter wrote:
>> D requires afaict at least a 32 bit system
>
> Yes.
>

You've said this some times before but never explained why there's such a limitation? I've actually used GDC to run code on 8bit AVR as well as 16bit MSP430 controllers.

The only thing I can think of is 'far pointer' support, but the times have changed in this regard as well:

TI implements 16bit or 20bit pointers for their 16 bit MSP architecture, but they never mix pointers: [1]

> The problem with a "medium" model, or any model where size_t and sizeof(void *)
> are not the same, is that they technically violate the ISO C standard. GCC has
> minimal support for such models, and having done some in the past, I recommend against it.

AVR for a long time only allowed access to high memory using special functions, no compiler support [2]. Nowadays GCC supports named address spaces [3] but I think we could implement this completely in library code: Basically using a type wrapper template should work. The only difficulty is making it work with volatile_load and if we can't find a better solution we'll need a new intrinsic data_load!(Flags = volatile, addrSpace = addrspace(foo),...)(address).


Then there's the small additional 'problem' that slices will be more expensive on these architectures: If you already need 2 registers to fit a pointer and 2 for size_t a slice will need 4 registers. So there may be some performance penalty but OTOH these RISC machines usually have more general purpose registers available than X86.

[1] https://e2e.ti.com/support/development_tools/compiler/f/343/t/451127
[2] http://www.nongnu.org/avr-libc/user-manual/pgmspace.html
[3] https://gcc.gnu.org/onlinedocs/gcc/Named-Address-Spaces.html

-- Johannes

On 5/18/17 12:40 AM, H. S. Teoh via Digitalmars-d wrote:
> On Wed, May 17, 2017 at 08:58:31PM -0400, Steven Schveighoffer via Digitalmars-d wrote:
>> On 5/17/17 8:27 PM, H. S. Teoh via Digitalmars-d wrote:
> [...]
>> What will cause a shift is a continuous business loss.
>>
>> If business A and B are competing in the same space, and business A
>> has a larger market share, but experiences a customer data breach.
>> Business B consumes many of A's customers, takes over the market, and
>> it turns out that the reason B wasn't affected was that they used a
>> memory-safe language.
>>
>> The business cases like this will continue to pile up until it will be
>> considered ignorant to use a non-memory safe language. It will be even
>> more obvious when companies like B are much smaller and less funded
>> than companies like A, but can still overtake them because of the
>> advantage.
>
> This is a possible scenario, but I don't see it being particularly
> likely, because in terms of data breaches, memory safety is only part of
> the equation. Other factors will also come into play in determining the
> overall susceptibility of a system. Things like secure coding practices,
> and by that I include more than just memory safety, such as resource
> management, proper use of cryptographic technology, privilege
> separation, access control, data sanitation, etc..  In spite of C's
> flaws, it *is* still possible to create a relatively secure system in C.
> It's harder, no argument about that, but possible.  It depends on how
> the company implements secure coding practices (or not).  In a memory
> safe language you can still make blunders that allow breaches like SQL
> injection in spite of memory safety.

Of course. But what business people would see is a huge company like facebook being marginalized by a small startup, and having the analysts say "well, it's mostly because they used Rust/D". The game would be over at that point, regardless of the technical details of the "true" root cause.

Note: I just use facebook as an example of a company that is so large and pervasive that everyone thinks they are unkillable, I don't really think the strawman scenario above is likely. Remember the old saying, "Nobody ever got fired for picking IBM"? How relevant is that today?

> Nevertheless, there is still an ongoing stream of exploits and security
> incidents in the web programming world largely driven by supposedly
> memory-safe languages like Java or Javascript. (Well, there is that
> disaster called PHP that's still prevalent on the web, maybe that
> accounts for some percentage of these exploits. But that's mostly in the
> implementation of PHP rather than the language itself, since AFAIK it
> doesn't let you manipulate memory directly in an unsafe way like C does.
> But it does let you do a lot of other stupid things, security-wise, that
> will still pose problems even though it's technically memory-safe.
> That's why I said, memory safety only goes so far -- you need a lot more
> than that to stand in the face of today's security threats.)

Speaking of "memory safe" languages like PHP whose implementation is not necessarily memory safe, there is a danger here also in how D is moving towards memory safety. We still allow unsafe operations inside @safe code, using @trusted. This is a necessary evil, but it's so very important that the base libraries (druntime and phobos) keep this to a minimum, and that we review those @trusted blocks to death.

-Steve

On Thu, May 18, 2017 at 08:12:18AM -0400, Steven Schveighoffer via Digitalmars-d wrote: [...]
> Of course. But what business people would see is a huge company like facebook being marginalized by a small startup, and having the analysts say "well, it's mostly because they used Rust/D". The game would be over at that point, regardless of the technical details of the "true" root cause.

But how likely is it for the analysts to say "it's because they used Rust/D instead of C"?


> Note: I just use facebook as an example of a company that is so large and pervasive that everyone thinks they are unkillable, I don't really think the strawman scenario above is likely. Remember the old saying, "Nobody ever got fired for picking IBM"? How relevant is that today?

Yeah, probably the shift away from C will be gradual, rather than overnight.


[...]
> Speaking of "memory safe" languages like PHP whose implementation is not necessarily memory safe, there is a danger here also in how D is moving towards memory safety. We still allow unsafe operations inside @safe code, using @trusted. This is a necessary evil, but it's so very important that the base libraries (druntime and phobos) keep this to a minimum, and that we review those @trusted blocks to death.
[...]

Yes, and that is why it's a grave concern that Phobos has (or used to have) giant blocks of code under the heading `@trusted:`. Even entire functions marked @trusted are a concern, to me, if the function is more than 5-10 lines long.

In the long run, I fear that if there are too many @trusted blocks in a given codebase (not necessarily Phobos), it will become too onerous to review, and could lead to hidden exploits that are overlooked by reviewers.  I don't know how to solve this conundrum.


T

-- 
"Hi." "'Lo."

On Thursday, 18 May 2017 at 17:53:52 UTC, H. S. Teoh wrote:

> In the long run, I fear that if there are too many @trusted blocks in a given codebase (not necessarily Phobos), it will become too onerous to review, and could lead to hidden exploits that are overlooked by reviewers.  I don't know how to solve this conundrum.

Simple. You reject such codebase from the get-go ;)

On 5/18/2017 10:53 AM, H. S. Teoh via Digitalmars-d wrote:
> Yes, and that is why it's a grave concern that Phobos has (or used to
> have) giant blocks of code under the heading `@trusted:`. Even entire
> functions marked @trusted are a concern, to me, if the function is more
> than 5-10 lines long.

Please pick one and submit a PR to fix it!

On Thursday, 18 May 2017 at 12:12:18 UTC, Steven Schveighoffer wrote:
> [...]
>
> We still allow unsafe operations inside @safe code, using @trusted. This is a necessary evil, but it's so very important that the base libraries (druntime and phobos) keep this to a minimum, and that we review those @trusted blocks to death.

That and we need to make sure it is understood by everyone using third party @safe code that it is *not* a "I don't have to audit this code" free card. It merely reduced the amount of code you need to review to what is marked as @trusted (with regards to memory safety); as long as you don't *know* whether some third party code is @safe or @trusted, you (as the programmer) have to assume it is @trusted and that means you have to extend trust to the author and cannot assume any of the @safe guarantees for that code.

On Thursday, 18 May 2017 at 18:15:28 UTC, Stanislav Blinov wrote:
> On Thursday, 18 May 2017 at 17:53:52 UTC, H. S. Teoh wrote:
>
>> In the long run, I fear that if there are too many @trusted blocks in a given codebase (not necessarily Phobos), it will become too onerous to review, and could lead to hidden exploits that are overlooked by reviewers.  I don't know how to solve this conundrum.
>
> Simple. You reject such codebase from the get-go ;)

To be honest, I don't think you *can* solve this problem (rejecting such a codebase is a workaround that may or may not work, depending on the use case and what the codebase as to do; there are valid reasons for why the majority of a codebase may need to be @trusted, such as OS abstractions). As long as we build software on top of operating systems with APIs that may or may not be unsafe we *need* such an unsafe layer and any codebase that heavily interacts with the OS will be littered with @trusted. All you can do is educate people to spot when @trusted is actually necessary and when something could genuinely be written @safe without @trusted and educate them to choose the latter when and if possible.

On 5/19/17 5:12 AM, Moritz Maxeiner wrote:
> On Thursday, 18 May 2017 at 12:12:18 UTC, Steven Schveighoffer wrote:
>> [...]
>>
>> We still allow unsafe operations inside @safe code, using @trusted.
>> This is a necessary evil, but it's so very important that the base
>> libraries (druntime and phobos) keep this to a minimum, and that we
>> review those @trusted blocks to death.
>
> That and we need to make sure it is understood by everyone using third
> party @safe code that it is *not* a "I don't have to audit this code"
> free card. It merely reduced the amount of code you need to review to
> what is marked as @trusted (with regards to memory safety); as long as
> you don't *know* whether some third party code is @safe or @trusted, you
> (as the programmer) have to assume it is @trusted and that means you
> have to extend trust to the author and cannot assume any of the @safe
> guarantees for that code.

What we need are 2 things:

1. @trusted blocks need to be rock-solid in Phobos and Druntime. And as rare as possible. This provides a foundation to build completely @safe libraries. It's like atomics -- they are hugely important and very easy to get wrong. Leave the actual implementation to the pros. We should be the pros on phobos/druntime safety.

2. @trusted blocks in any project need to be considered red flags. You should not need to audit @safe code. What you need to do is audit @trusted code when it interacts with @safe code. If you can prove that in *all cases* the @safe code is still @safe even with the included @trusted blocks, then you don't have to audit @safe code that calls that "tainted" function.

If we get into "@safe really means @trusted" territory, we have lost.

-Steve

On Friday, 19 May 2017 at 11:53:57 UTC, Steven Schveighoffer wrote:
> On 5/19/17 5:12 AM, Moritz Maxeiner wrote:
>> On Thursday, 18 May 2017 at 12:12:18 UTC, Steven Schveighoffer wrote:
>>> [...]
>>>
>>> We still allow unsafe operations inside @safe code, using @trusted.
>>> This is a necessary evil, but it's so very important that the base
>>> libraries (druntime and phobos) keep this to a minimum, and that we
>>> review those @trusted blocks to death.
>>
>> That and we need to make sure it is understood by everyone using third
>> party @safe code that it is *not* a "I don't have to audit this code"
>> free card. It merely reduced the amount of code you need to review to
>> what is marked as @trusted (with regards to memory safety); as long as
>> you don't *know* whether some third party code is @safe or @trusted, you
>> (as the programmer) have to assume it is @trusted and that means you
>> have to extend trust to the author and cannot assume any of the @safe
>> guarantees for that code.
>
> What we need are 2 things:
>
> 1. @trusted blocks need to be rock-solid in Phobos and Druntime. And as rare as possible.

Agreed 100%.

> This provides a foundation to build completely @safe libraries.

Agreed if you mean libraries being marked completely as @safe (which I assume).
Disagreed if you mean libraries that are proven to never corrupt memory (not possible with unsafe operating system).

> It's like atomics -- they are hugely important and very easy to get wrong.

Sure.

> Leave the actual implementation to the pros.

If you mean the act of implementing: Yes, agreed.
If you mean the entire mind space of the implementation, aka you (the programmer) receives a "get out of audit this" free card, because "professionals" wrote it: No. You are *always* responsible for verifying *all* third party @trusted code *yourself*.

> We should be the pros on phobos/druntime safety.

Agreed.

>
> 2. @trusted blocks in any project need to be considered red flags. You should not need to audit @safe code.

Yes you do, because it can call into @trusted like this:

---
void foo(int[] bar) @safe
{
  () @trusted {
    // Exploitable code here
  }();
}
---

You *must* audit third party @safe code for such hidden @trusted code (e.g. grep recursively through such third party code for @trusted and verify).

> What you need to do is audit @trusted code when it interacts with @safe code.

You need to audit *all* @trusted code, because you don't necessarily control who calls it.

> If you can prove that in *all cases* the @safe code is still @safe even with the included @trusted blocks, then you don't have to audit @safe code that calls that "tainted" function.

s/prove/promise/
But yes, that is precisely what I wrote in the above with regards to the reduction of what you have to audit.

>
> If we get into "@safe really means @trusted" territory, we have lost.

For code that you write yourself, @safe means @safe, of course. For code other people write and you want to call, it being marked @safe does really mean @trusted as long as you yourself have not looked inside it and verified there either is no hidden @trusted, or verified *yourself* that the hidden @trusted is memory safe.
I consider any other behaviour to be negligent to the degree of "you don't actually care about memory safety at all".

On 5/19/17 9:46 AM, Moritz Maxeiner wrote:
> On Friday, 19 May 2017 at 11:53:57 UTC, Steven Schveighoffer wrote:

>> This provides a foundation to build completely @safe libraries.
>
> Agreed if you mean libraries being marked completely as @safe (which I
> assume).
> Disagreed if you mean libraries that are proven to never corrupt memory
> (not possible with unsafe operating system).

I mean libraries which only contain @safe and @system calls.

i.e.:

$ grep -R '@trusted' libsafe | wc -l
0

>> 2. @trusted blocks in any project need to be considered red flags. You
>> should not need to audit @safe code.
>
> Yes you do, because it can call into @trusted like this:
>
> ---
> void foo(int[] bar) @safe
> {
>   () @trusted {
>     // Exploitable code here
>   }();
> }
> ---
>
> You *must* audit third party @safe code for such hidden @trusted code
> (e.g. grep recursively through such third party code for @trusted and
> verify).

This is what I mean by auditing @trusted when it interacts with @safe code.

Using your example, if we confirm that no matter how you call foo, the @trusted block cannot break memory safety, then foo becomes a verified @safe function. Then any @safe function that calls foo can be considered @safe without auditing.

This is actually a necessity, because templates can infer safety, so you may not even know the call needs to be audited. The most dangerous thing I think is to have @trusted blocks which use templated types.

A real example recently was a PR that added @safe to a function, and made the following call:

() @trusted { return CreateDirectoryW(pathname.tempCStringW(), null); }

Where pathname was a range. The trusted block is to allow the call to CreateDirectoryW, but inadvertently, you are trusting all the range functions inside pathname, whatever that type is!

The correct solution is:

auto cstr = pathname.tempCStringW();
() @trusted { return CreateDirectoryW(cstr, null); }

So yes, if the third party has @trusted code, you need to audit it. But once you have audited the *block* of trusted code, and how it interacts within its function, you can consider the calling @safe functions actually safe without auditing.

>> If we get into "@safe really means @trusted" territory, we have lost.
>
> For code that you write yourself, @safe means @safe, of course. For code
> other people write and you want to call, it being marked @safe does
> really mean @trusted as long as you yourself have not looked inside it
> and verified there either is no hidden @trusted, or verified *yourself*
> that the hidden @trusted is memory safe.
> I consider any other behaviour to be negligent to the degree of "you
> don't actually care about memory safety at all".

I think there will be a good market for separating libraries between @trusted-containing libraries, and only @safe-containing libraries. This will make the auditing more focused, and more shareable. I don't expect people to use Phobos and audit all the @trusted blocks personally. If "D is memory safe" means "D is memory safe ONLY if you verify all of the standard library personally", we still have lost.

-Steve