I've moved this to another thread to allay complaints.

> "Vladimir Panteleev" <vladimir@thecybershadow.net>
> On Friday, 12 April 2013 at 10:14:35 UTC, Regan Heath wrote:
>> All true.  However, complexity can and should be packaged insuch a way as to localise, and this localised complex codeshould be tested to death and maintained by someone whounderstands it.  It should be bracketed by sufficient commentsand warnings about how/why it does what it does.  The resultingpackaged complexity, with it's associated cost can be re-usedmany times over for all the benefit it gives.
>>
>> Stack allocating the environment variables need not be alocalised improvement but could be a standard library functionwhich can be reused, for example.
>
> Performant abstractions. Like the much-awaited allocator design?
> Either way, they can only diminuate, not remove the costs.

non localised improvements have a fixed cost and ever increasing benefit - is the point I was making here.  Agreed, you cannot remove the cost, and in fact well written reusable code often carries a slightly higher cost by it's very nature.

>>> Would you still say that the above costs are worth thenearly-intangible gain?
>>
>> "nearly-intangible" is wrong. Library code is code which isused by (hopefully) millions of people, writing millions ofapplications, running for millions of hours, on millions ofsystems, creating thousands of processes, etc..  In short, alittle effort now pays massive dividends over it's lifetime.
>>
>> So, yes, IMO the costs shown above are worth the resultinggains.
>>
>> D is constantly being compared to other languages on the basisof performance, so it's clearly an important aspect of D'ssuccess.
>>
>> Library code needs to work first time and work well or peoplewill roll their own wasting time, energy and in many casesgetting some aspect of it just plain wrong.
>
> But once again, you speak in vague terms.

The initial point was a vague one, not a specific one.  Manu wasn't attempting to block std.process, he had a general concern which I share.

> Consider the following hypothetical decisions and outcomes:
>
> 1. std.process is left at is. One user is angry / turned awaybecause it performs 0.1% slower than it can be.

It very much matters *who* that 1 user is.  And, the count may be higher, and we might never "hear" from these people as they find other solutions.  We're lucky that some people who try D and have issues tell us about them, they may be 5% of the total for all we know.

> 2. std.process is rewritten to minimize allocations. Codecomplexity goes up, new improvements are challenging to add; bugspop up and go unfixed for a while because fewer programmers arequalified or willing to commit the effort of making correctfixes. More people are angry / turned away from D because itsstandard library is buggy.
>
> Of course, the above is an exaggerated illustration.

Indeed, you've downplayed point #1 and exaggerated point #2.  In reality the suggested improvements would add only very minor complexity and prevent none of the current crop of contributors from working with/on std.process.

> But would optimizing all code left and right really make more D usershappier?

Yes, as well as the users of their applications.  True, none of them will even realise they could have been less happy, so none of them will realise the effort that went into it, but all of them will be better off.

> There's also the question of priorities. Would you rather thaneffort is spent on optimizing std.process (and dealing with allthe fallout from any such optimizations), or working on somethingthat is acutely missing and hurting D?

Add the missing items, without a doubt - which is why no-one is suggesting blocking std.process over this issue.

>> D is a systems programming language, there is hope that it willpenetrate a wide range of systems and environments - sure inmany cases a little bit of memory use or performance loss isunimportant, but for many it will be the decisive factor whichmakes D unusable there.
>
> This is surely an exaggeration.

Why?

There exist platforms and environments where memory and performance are concerns, if the D standard library code is not "careful" in it's use of both then it will be less suitable than C (for example) and so D will not penetrate those platforms.

Manu is using D for games development on modern high-end gaming PCs and he is still concerned with memory and performance.

So, there's 2 very different cases where memory and performance are still a concern, and .. if they become too much of a concern another solution will have to be sought.. and that's bad news for D.

> D does not attempt to please everyone out there who is choosing aprogramming language for their next project. There is no suchlanguage, nor can one exist. One has to accept that D has anumber of goals, none of which are absolute, but merely pointtowards a certain, but not overly specific, point in themultidimensional matrix of trade-offs. D never was aboutachieving maximum performance in all possible cases.

All true, but performance is one of D's top draw cards:

<quote>The D programming language. Modern convenience. Modeling power. Native **efficiency**.</quote> (**emphasis mine**)

So, it behoves us to make sure the standard library keeps that in mind.

R

On Friday, 12 April 2013 at 11:37:14 UTC, Regan Heath wrote:
> I've moved this to another thread to allay complaints.

Thanks!

I completely agree that if code can be made more performant without a significant increase in complexity, then we should do so.  While it is mostly (but not entirely) irrelevant in the context of std.process, it is a problem that should be tackled in Phobos as a whole.  Several things could/should be done:

It would be nice to have some sugar on top of alloca(), the use of which is usually considered bad practice.  Someone (bearophile?) once suggested static arrays whose length is determined at runtime, which would be a great addition to the language:

    void foo(int n)
    {
        int[n] myArr;
        ...
    }

Furthermore, we need to get the allocator design in place.  In SciD, I use David Simcha's region allocator to allocate temporary workspace, and it works really well.  The only times I use 'new' is when I need persistent memory (e.g. for a return value) and the user-supplied buffer is too small.  Phobos would greatly benefit from doing this too.

Finally, an example from the new std.process which got some heavy criticism in the other thread:

    envz[pos++] = (var~'='~val~'\0').ptr;

I have been operating under the assumption that the compiler is smart enough to make the above a single allocation.  If it isn't, I would consider it a compiler issue.

That said, I am aware that std.process could be improved in some places.

Lars

On Friday, 12 April 2013 at 12:30:09 UTC, Lars T. Kyllingstad wrote:
> Finally, an example from the new std.process which got some heavy criticism in the other thread:
>
>     envz[pos++] = (var~'='~val~'\0').ptr;
>
> I have been operating under the assumption that the compiler is smart enough to make the above a single allocation.  If it isn't, I would consider it a compiler issue.

Multiple chained array concatenations are performed at once, using the _d_arraycatnT function in Druntime.

On Friday, 12 April 2013 at 12:43:57 UTC, Vladimir Panteleev wrote:
> On Friday, 12 April 2013 at 12:30:09 UTC, Lars T. Kyllingstad wrote:
>> Finally, an example from the new std.process which got some heavy criticism in the other thread:
>>
>>    envz[pos++] = (var~'='~val~'\0').ptr;
>>
>> I have been operating under the assumption that the compiler is smart enough to make the above a single allocation.  If it isn't, I would consider it a compiler issue.
>
> Multiple chained array concatenations are performed at once, using the _d_arraycatnT function in Druntime.

Good to know.

Lars

On Friday, 12 April 2013 at 11:37:14 UTC, Regan Heath wrote:
> The initial point was a vague one, not a specific one.  Manu wasn't attempting to block std.process, he had a general concern which I share.

OK, but so far my interpretation and replies were mostly in the context of std.process - this module being an example where performance improvements would have a very small real-life benefit. I agree that (generally speaking) improving the performance of the code in std.algorithm/array/range would be worth the effort and complexity.

> It very much matters *who* that 1 user is.  And, the count may be higher, and we might never "hear" from these people as they find other solutions.  We're lucky that some people who try D and have issues tell us about them, they may be 5% of the total for all we know.

The same applies to the other side of the argument. A buggy standard library probably leaves a worse impression than a slow standard library...

> In reality the suggested improvements would add only very minor complexity and prevent none of the current crop of contributors from working with/on std.process.

Well, how do you qualify the amount of optimization that is appropriate?

For example, the code in std.process would be even faster, if it was completely written in assembler. I hope we'll agree than in practice, this would be absurd. Now, what set of well-defined arguments would conclude that rewriting it in assembler is pointless, but optimizing memory allocations is not? All three versions of std.process would perform as well as far as the end-user can perceive.

> Yes, as well as the users of their applications.  True, none of them will even realise they could have been less happy, so none of them will realise the effort that went into it, but all of them will be better off.

Absolutely - if you ignore the costs. 100%-correct faster code is always better than 100%-correct slower code, but the costs are the counter-argument.

> Add the missing items, without a doubt - which is why no-one is suggesting blocking std.process over this issue.

Blocking is one thing, but asking for faster code where it doesn't really matter - when there are areas where D could be improved at much higher gain per effort - is another.

>>> D is a systems programming language, there is hope that it
> Why?
>
> There exist platforms and environments where memory and performance are concerns, if the D standard library code is not "careful" in it's use of both then it will be less suitable than C (for example) and so D will not penetrate those platforms.

OK, but once again - how does that line up with the purpose of std.process? I can see how std.algorithm can be useful in low-spec embedded/gaming systems, but std.process?

> Manu is using D for games development on modern high-end gaming PCs and he is still concerned with memory and performance.

In Manu's case, every bit of performance counts in the code that runs in tight loops, e.g. for every game frame. However, does that include std.process?

> All true, but performance is one of D's top draw cards:
>
> <quote>The D programming language. Modern convenience. Modeling power. Native **efficiency**.</quote> (**emphasis mine**)
>
> So, it behoves us to make sure the standard library keeps that in mind.

Again, I don't (generally) disagree for the general case, however I think it pays to mind the context and perspective. When the context is std.process and the perspective is the relative cost of process creation, it seems like quite a pointless argument.

On Friday, 12 April 2013 at 07:04:23 UTC, Manu wrote:
>
> string[string] is used in the main API to receive environment variables;
> perhaps kinda convenient, but it's impossible to supply environment
> variables with loads of allocations.

Environment variables are a mapping of strings to strings.  The natural way to express such a mapping in D is with a string[string].  It shouldn't be necessary to allocate an AA literal, though.


> toStringz is used liberally; alternatively, alloca() could allocate the
> c-string's on the stack and zero terminate them there, passing a pointer to
> the stack string to the OS functions.

It is kind of hard to use alloca() in a safe manner in D, because DMD will happily inline functions that use it.  The following program will overflow the stack if compiled with -inline:

void doStuff()
{
    auto p = alloca(100);
}

void main()
{
    foreach (i; 0 .. 1_000_000) doStuff();
}

This is of course fixable, but until that happens, I would consider alloca() a no-go for Phobos.

On 12 April 2013 22:30, Lars T. Kyllingstad <public@kyllingen.net> wrote:

> On Friday, 12 April 2013 at 11:37:14 UTC, Regan Heath wrote:
>
>> I've moved this to another thread to allay complaints.
>>
>
> Thanks!
>
> I completely agree that if code can be made more performant without a significant increase in complexity, then we should do so.  While it is mostly (but not entirely) irrelevant in the context of std.process, it is a problem that should be tackled in Phobos as a whole.  Several things could/should be done:
>
> It would be nice to have some sugar on top of alloca(), the use of which is usually considered bad practice.  Someone (bearophile?) once suggested static arrays whose length is determined at runtime, which would be a great addition to the language:
>
>     void foo(int n)
>     {
>         int[n] myArr;
>         ...
>     }
>

That's beautiful!

Furthermore, we need to get the allocator design in place.  In SciD, I use
> David Simcha's region allocator to allocate temporary workspace, and it
> works really well.  The only times I use 'new' is when I need persistent
> memory (e.g. for a return value) and the user-supplied buffer is too small.
>  Phobos would greatly benefit from doing this too.
>
> Finally, an example from the new std.process which got some heavy criticism in the other thread:
>
>     envz[pos++] = (var~'='~val~'\0').ptr;
>
> I have been operating under the assumption that the compiler is smart enough to make the above a single allocation.  If it isn't, I would consider it a compiler issue.
>

Does it? I've not seen the compiler do that, although I'd like to think it should be possible. 1 allocation is better than 3 I guess, however, I wonder if that code could be restructured to use the stack aswell. alloca() is really underrated! I can't imagine why people don't like it. Perhaps some more helpers built around it might encourage its use? It does feel a little bit 'raw', like malloc(). It implies some annoying casts.

On 12 April 2013 23:08, Vladimir Panteleev <vladimir@thecybershadow.net>wrote:

> On Friday, 12 April 2013 at 11:37:14 UTC, Regan Heath wrote:
>
>> It very much matters *who* that 1 user is.  And, the count may be higher,
>> and we might never "hear" from these people as they find other solutions.
>>  We're lucky that some people who try D and have issues tell us about them,
>> they may be 5% of the total for all we know.
>>
>
> The same applies to the other side of the argument. A buggy standard library probably leaves a worse impression than a slow standard library...


If allocating a string on the stack makes it buggy, then there is something
really wrong. It should be no less convenient if appropriate helpers are
available.
With consideration to the string[string] argument, surely instances like
that can be reconsidered? How is string[] going to produce more bugs than
string[string]?
You're being paranoid, or sensationalising the effect of simple
optimisation.

In reality the suggested improvements would add only very minor complexity
>> and prevent none of the current crop of contributors from working with/on std.process.
>>
>
> Well, how do you qualify the amount of optimization that is appropriate?
>

As much is convenient without causing you to start obscuring your code?
That's my personal rule.
But I make it a habit to consider efficiency when designing code, I never
retrofit it. I tend to choose designs that are both simple and efficient at
the start.

For example, the code in std.process would be even faster, if it was
> completely written in assembler. I hope we'll agree than in practice, this would be absurd. Now, what set of well-defined arguments would conclude that rewriting it in assembler is pointless, but optimizing memory allocations is not? All three versions of std.process would perform as well as far as the end-user can perceive.


Actually, it would probably be slower if hand-written in assembler. And again, speed is not my concern here, it's inconsiderate the allocation policy.

 Yes, as well as the users of their applications.  True, none of them will
>> even realise they could have been less happy, so none of them will realise the effort that went into it, but all of them will be better off.
>>
>
> Absolutely - if you ignore the costs. 100%-correct faster code is always better than 100%-correct slower code, but the costs are the counter-argument.


Can you describe the 'costs'?

 Add the missing items, without a doubt - which is why no-one is suggesting
>> blocking std.process over this issue.
>>
>
> Blocking is one thing, but asking for faster code where it doesn't really matter - when there are areas where D could be improved at much higher gain per effort - is another.
>

I'm asking for code that doesn't needlessly allocate, as a policy/habit in phobos.

 D is a systems programming language, there is hope that it
>>>>
>>> Why?
>>
>>
>> There exist platforms and environments where memory and performance are concerns, if the D standard library code is not "careful" in it's use of both then it will be less suitable than C (for example) and so D will not penetrate those platforms.
>>
>
> OK, but once again - how does that line up with the purpose of std.process? I can see how std.algorithm can be useful in low-spec embedded/gaming systems, but std.process?
>
>
>  Manu is using D for games development on modern high-end gaming PCs and
>> he is still concerned with memory and performance.
>>
>
> In Manu's case, every bit of performance counts in the code that runs in tight loops, e.g. for every game frame. However, does that include std.process?


I'm interested in eliminating allocations. It's just another function that can't be called in a no-gc area. If it used the stack for its temporaries, no problem.

 All true, but performance is one of D's top draw cards:
>>
>> <quote>The D programming language. Modern convenience. Modeling power. Native **efficiency**.</quote> (**emphasis mine**)
>>
>> So, it behoves us to make sure the standard library keeps that in mind.
>>
>
> Again, I don't (generally) disagree for the general case, however I think it pays to mind the context and perspective. When the context is std.process and the perspective is the relative cost of process creation, it seems like quite a pointless argument.
>

It was the first module that appeared for consideration since the recent
discussions about irresponsible GC usage. The argument applies to
everything considered for acceptance into phobos.
I'd like to see it applied as a systematic consideration in the future,
irrespective of the module being considered. Avoiding allocation for
temporaries shouldn't be hard, if some tools are missing, then that is
something that needs further discussion I guess.

On 12 April 2013 23:21, Lars T. Kyllingstad <public@kyllingen.net> wrote:

> On Friday, 12 April 2013 at 07:04:23 UTC, Manu wrote:
>
>>
>> string[string] is used in the main API to receive environment variables; perhaps kinda convenient, but it's impossible to supply environment variables with loads of allocations.
>>
>
> Environment variables are a mapping of strings to strings.  The natural way to express such a mapping in D is with a string[string].  It shouldn't be necessary to allocate an AA literal, though.
>

That's a good point, do AA's support literals that don't allocate? You can't even produce an array literal without it needlessly allocating.

 toStringz is used liberally; alternatively, alloca() could allocate the
>> c-string's on the stack and zero terminate them there, passing a pointer
>> to
>> the stack string to the OS functions.
>>
>
> It is kind of hard to use alloca() in a safe manner in D, because DMD will happily inline functions that use it.  The following program will overflow the stack if compiled with -inline:
>
> void doStuff()
> {
>     auto p = alloca(100);
> }
>
> void main()
> {
>     foreach (i; 0 .. 1_000_000) doStuff();
> }
>
> This is of course fixable, but until that happens, I would consider
> alloca() a no-go for Phobos.
>

Very good point. This is a problem.
Hmmm...

On Friday, 12 April 2013 at 13:39:38 UTC, Manu wrote:
> If allocating a string on the stack makes it buggy, then there is something
> really wrong. It should be no less convenient if appropriate helpers are
> available.

Please see my reply to your other post.

> With consideration to the string[string] argument, surely instances like
> that can be reconsidered? How is string[] going to produce more bugs than
> string[string]?

env ~= "FOO=BAR";

This will probably not do what you want if there was already a line starting with "FOO=" in env.

An array of strings is a less direct representation of the environment than a string map. Certain common operations, such as finding the value of a variable, or setting / overwriting a variable, become more difficult.

> You're being paranoid, or sensationalising the effect of simple
> optimisation.

Strong words...

> And
> again, speed is not my concern here, it's inconsiderate the allocation
> policy.

> I'm interested in eliminating allocations. It's just another function that
> can't be called in a no-gc area. If it used the stack for its temporaries,
> no problem.

Why allocations, specifically, if not for the performance costs of allocation and garbage collection?

> Can you describe the 'costs'?

See my previous posts in today's discussions.

> As much is convenient without causing you to start obscuring your code?
> That's my personal rule.
> But I make it a habit to consider efficiency when designing code, I never
> retrofit it. I tend to choose designs that are both simple and efficient at
> the start.

OK, so if I understand you correctly: you would like Phobos to adopt a policy of avoiding heap allocations whenever possible, and this argument applies to std.process not because doing so would result in a tangible improvement of its performance or other metric, but for the purpose of being consistent across Phobos. Assuming that the language can provide or allow implementing suitably safe abstractions for doing so without complicating the code much, I think that's a goal worth looking forward, and we have been doing so for some time (hence the pending allocator design).