On Friday, 12 April 2013 at 04:46:52 UTC, Jesse Phillips wrote:
> It is that time, If you would like to see the proposed std.process include into Phobos please vote yes. If one condition must be met specify under what condition, otherwise vote no.
>
> std.process by Lars Kyllingstad and Steven Schveighoffer is a suggested improvement to the existing std.process and is a major change to the API. The original API remains but these will be going through deprecation.

Yes.

A compiler warning like for deprecation should added for "experimental" IMO rather than use exp. vs std.

On 2013-04-12 09:04, Manu wrote:

> I've said before, I sadly have to avoid phobos like the plague. Some
> modules (like this one) that provide fundamental functionality - not
> just helper functions - can't be avoided. Requirements for those should
> be extra strict in my opinion.

Have you tried Tango? For most functions that do some array or string processing you have the option of passing your own buffer. If the buffer is too small, the function will allocate, otherwise it won't.

Docs: http://www.dsource.org/projects/tango/docs/current/
Tango D2: https://github.com/SiegeLord/Tango-D2

-- 
/Jacob Carlborg

On Friday, 12 April 2013 at 10:14:35 UTC, Regan Heath wrote:
> All true.  However, complexity can and should be packaged in such a way as to localise, and this localised complex code should be tested to death and maintained by someone who understands it.  It should be bracketed by sufficient comments and warnings about how/why it does what it does.  The resulting packaged complexity, with it's associated cost can be re-used many times over for all the benefit it gives.
>
> Stack allocating the environment variables need not be a localised improvement but could be a standard library function which can be reused, for example.

Performant abstractions. Like the much-awaited allocator design?

Either way, they can only diminuate, not remove the costs.

>> Would you still say that the above costs are worth the nearly-intangible gain?
>
> "nearly-intangible" is wrong. Library code is code which is used by (hopefully) millions of people, writing millions of applications, running for millions of hours, on millions of systems, creating thousands of processes, etc..  In short, a little effort now pays massive dividends over it's lifetime.
>
> So, yes, IMO the costs shown above are worth the resulting gains.
>
> D is constantly being compared to other languages on the basis of performance, so it's clearly an important aspect of D's success.
>
> Library code needs to work first time and work well or people will roll their own wasting time, energy and in many cases getting some aspect of it just plain wrong.

But once again, you speak in vague terms.

Consider the following hypothetical decisions and outcomes:

1. std.process is left at is. One user is angry / turned away because it performs 0.1% slower than it can be.

2. std.process is rewritten to minimize allocations. Code complexity goes up, new improvements are challenging to add; bugs pop up and go unfixed for a while because fewer programmers are qualified or willing to commit the effort of making correct fixes. More people are angry / turned away from D because its standard library is buggy.

Of course, the above is an exaggerated illustration. But would optimizing all code left and right really make more D users happier?

There's also the question of priorities. Would you rather than effort is spent on optimizing std.process (and dealing with all the fallout from any such optimizations), or working on something that is acutely missing and hurting D?

> D is a systems programming language, there is hope that it will penetrate a wide range of systems and environments - sure in many cases a little bit of memory use or performance loss is unimportant, but for many it will be the decisive factor which makes D unusable there.

This is surely an exaggeration.

D does not attempt to please everyone out there who is choosing a programming language for their next project. There is no such language, nor can one exist. One has to accept that D has a number of goals, none of which are absolute, but merely point towards a certain, but not overly specific, point in the multidimensional matrix of trade-offs. D never was about achieving maximum performance in all possible cases.

12-Apr-2013 11:04, Manu пишет:
> On 12 April 2013 14:46, Jesse Phillips <Jessekphillips+d@gmail.com
> <mailto:Jessekphillips+d@gmail.com>> wrote:
>
>     It is that time, If you would like to see the proposed std.process
>     include into Phobos please vote yes. If one condition must be met
>     specify under what condition, otherwise vote no.
>
>
> I didn't think I had much of an opinion on std.process, but I just gave
> it a quick once over and noticed, relating to our recent discussions
> about flagrant GC usage throughout phobos, that this is riddled with the
> exact sorts of issues I've been talking about:
>
> string[string] is used in the main API to receive environment variables;
> perhaps kinda convenient, but it's impossible to supply environment
> variables with loads of allocations.
>
> toStringz is used liberally; alternatively, alloca() could allocate the
> c-string's on the stack and zero terminate them there, passing a pointer
> to the stack string to the OS functions.
>

Most of this is unsolvable until there is a STANDARD interface/policy/hooks for allocators in Phobos. All work to get around the cases like that at the moment is just a waste of time or specific hacks for particular bottlenecks.

I did a fair amount of these in std.regex to make matching GC-free and almost heap-free - it preallocates on the first call. It's an example of a necessary hack and it's not pretty. More then that I'm sure I'd have to re-write the whole memory allocation code once we (finally!) have he standard way to do it.

TL;DR - until using allocators is easy and straight-forward (and standard) there would be no end of GC-only hegeomny 'cause it's easier/simpler. Not much to discuss ;)

> String concatenation is rampant! Look at this code to parse the env
> variables (which are already an AA):
>
> foreach (var, val; childEnv)
>
> envz[pos++] = (var~'='~val~'\0').ptr;
>

Horrible...
But let's focus on the API parts. The internals could be fixed with a pull request or two. Freaking out because of a GC happy code doesn't help those waiting for *any* sane process management module in std.


-- 
Dmitry Olshansky

12-Apr-2013 12:26, Manu пишет:
> On 12 April 2013 18:18, Vladimir Panteleev <vladimir@thecybershadow.net
> <mailto:vladimir@thecybershadow.net>> wrote:
>
>     On Friday, 12 April 2013 at 07:53:04 UTC, Manu wrote:
>
>         Which OS are we talking about?
>         What OS runs on an a Nintendo Wii? There's only 24mb of system
>         memory in
>         that machine, can we afford to allocate it frivolously?
>
>
>     I wasn't aware that writing games for the Wii involves creating
>     processes in the thousands.
>
>
> Again, I'm just suggesting possibilities, and trying to illustrate that
> it's a STANDARD library, you can never predict where users will want to
> use it.

Very true.

-- 
Dmitry Olshansky

On 12 April 2013 18:34, Johannes Pfau <nospam@example.com> wrote:

> Am Fri, 12 Apr 2013 17:04:08 +1000
> schrieb Manu <turkeyman@gmail.com>:
>
> >
> > I've said before, I sadly have to avoid phobos like the plague. Some modules (like this one) that provide fundamental functionality - not just helper functions - can't be avoided. Requirements for those should be extra strict in my opinion.
> >
>
> Most (GC) allocations could be fixed without breaking the API. I can
> see 2 places where the API forces _GC_ allocations:
> * string[string]
> * thrown Exceptions are allocated with the GC
>
> There is no simple solution for these. Maybe we'll have a hashtable in the standard library at some point which will allow to use custom allocators. Then std.process could use these. Exceptions require some thinking. You can of course allocate them using any allocator, but freeing them is difficult...
>

Exceptions are exceptional, I don't think that's an issue.
the string[string] in the api is guaranteed garbage, and will never be
changed. What's the advantage of using an AA in this case?

If you want to get rid of all (not only GC) allocations, there's another
> issue: As D strings are not zero terminated we'll always have allocations passing those to C code.


They are being passed through to a C function that doesn't retain the pointer; they can be allocated on the stack.

Maybe we should have a cstring
> type in phobos which would just be a string which is guaranteed to be zero terminated.
>

Maybe, although the point here is to wrap up and hide the C API, so that's not useful here. A helper in phobos for allocating a (c)string on the stack may be useful...

On 12 April 2013 18:34, Johannes Pfau <nospam@example.com> wrote:

> Am Fri, 12 Apr 2013 17:04:08 +1000
> schrieb Manu <turkeyman@gmail.com>:
>
> >
> > I've said before, I sadly have to avoid phobos like the plague. Some modules (like this one) that provide fundamental functionality - not just helper functions - can't be avoided. Requirements for those should be extra strict in my opinion.
> >
>
> Most (GC) allocations could be fixed without breaking the API. I can
> see 2 places where the API forces _GC_ allocations:
> * string[string]
> * thrown Exceptions are allocated with the GC
>
> There is no simple solution for these. Maybe we'll have a hashtable in the standard library at some point which will allow to use custom allocators. Then std.process could use these. Exceptions require some thinking. You can of course allocate them using any allocator, but freeing them is difficult...
>

Exceptions are exceptional, I don't think that's an issue.
the string[string] in the api is guaranteed garbage, and will never be
changed. What's the advantage of using an AA in this case?

If you want to get rid of all (not only GC) allocations, there's another
> issue: As D strings are not zero terminated we'll always have allocations passing those to C code.


They are being passed through to a C function that doesn't retain the pointer; they can be allocated on the stack.

Maybe we should have a cstring
> type in phobos which would just be a string which is guaranteed to be zero terminated.
>

Maybe, although the point here is to wrap up and hide the C API, so that's not useful here. A helper in phobos for allocating a (c)string on the stack may be useful...

This is good feedback and all, but it would be good to have had this placed in the official review thread.

But it is too late now and I would not want to break the discussion up.

On Friday, 12 April 2013 at 06:25:10 UTC, Manu wrote:
> I see this pattern where something is designed, discussed, and then voted
> into phobos. At this time the design looks good on paper, but there is very
> little practical experience using the library.
> The problem then is, once accepted, people start using it, and at some
> point some issues are found, or ideas for improvement are made based on
> user experience, but the module can no longer be touched due to the general
> phobia of making breaking changes...

I think this needs to happen prior to the formal review/voting. I would say it should be a precursor to starting the official review, however this would raise the bar too high for things like Jacob's Serialization library; he has a working library, but it isn't ready for Phobos and it would be silly to require the translation prior to approving it for Phobos.

How we choose to add to the exp module would need some consideration.

On 4/12/13, Manu <turkeyman@gmail.com> wrote:
> string[string] is used in the main API to receive environment variables; perhaps kinda convenient, but it's impossible to supply environment variables with loads of allocations.

I think the functions which take the environment as an associative array can be changed without breaking any user code. E.g.:

Pid spawnProcess(Env)(in char[][] args,
                 File stdin = std.stdio.stdin,
                 File stdout = std.stdio.stdout,
                 File stderr = std.stdio.stderr,

                 /* was const string[string] env = null */
                 const Env env = Env.init,

                 Config config = Config.none)
    if (isSomeAssociativeArray!Env)  // introduce constraint
{
}

Then the user can provide their custom stack-based type which implements opIn_r.