I think you are right that phobos should avoid unnecessary allocations, if it does not affect maintainability/readability of code.

I would suggest, instead of writing your own std.process, that you simply provide a pull request to the now to be approved one, this way you can prove your point that it is easy by already improving a part of the standard library.

If your pull gets accepted, std.process could be referenced from a wiki page "how to minimize GC usage" for phobos contributions or something.

Best regards,
Robert

On Friday, 12 April 2013 at 13:00:36 UTC, Robert wrote:
> I would suggest, instead of writing your own std.process, that you
> simply provide a pull request to the now to be approved one, this way
> you can prove your point that it is easy by already improving a part of
> the standard library.

Actually, Manu (or anyone) could do this for *any* part of Phobos.  And considering that std.process is probably the module that would benefit the least from it, maybe he should start somewhere else?  std.string, perhaps, or std.algorithm?

On Friday, 12 April 2013 at 13:09:19 UTC, Lars T. Kyllingstad wrote:
> On Friday, 12 April 2013 at 13:00:36 UTC, Robert wrote:
>> I would suggest, instead of writing your own std.process, that you
>> simply provide a pull request to the now to be approved one, this way
>> you can prove your point that it is easy by already improving a part of
>> the standard library.
>
> Actually, Manu (or anyone) could do this for *any* part of Phobos.  And considering that std.process is probably the module that would benefit the least from it, maybe he should start somewhere else?  std.string, perhaps, or std.algorithm?

Or probably druntime. This rise again and again - until standard global allocators are not here it is all a premature and useless optimization. It makes no sense to try to workaround a fundamental issue. That will be a lot of effort for no real gain.

Allocators are a blocker for almost any memory-related issue in both druntime and phobos.

12-Apr-2013 17:20, Dicebot пишет:
> On Friday, 12 April 2013 at 13:09:19 UTC, Lars T. Kyllingstad wrote:
>> On Friday, 12 April 2013 at 13:00:36 UTC, Robert wrote:
>>> I would suggest, instead of writing your own std.process, that you
>>> simply provide a pull request to the now to be approved one, this way
>>> you can prove your point that it is easy by already improving a part of
>>> the standard library.
>>
>> Actually, Manu (or anyone) could do this for *any* part of Phobos.
>> And considering that std.process is probably the module that would
>> benefit the least from it, maybe he should start somewhere else?
>> std.string, perhaps, or std.algorithm?
>
> Or probably druntime. This rise again and again - until standard global
> allocators are not here it is all a premature and useless optimization.
> It makes no sense to try to workaround a fundamental issue. That will be
> a lot of effort for no real gain.
>
> Allocators are a blocker for almost any memory-related issue in both
> druntime and phobos.

Same here - trying to get out of your way to avoid GC right now is both painful and useless in the long run. More then that it's limited as you can't have the default allocation scheme that fits all nor use some scheme that would require change of API (so that later it's broken again once allocators come).

-- 
Dmitry Olshansky

On 12 April 2013 23:09, Lars T. Kyllingstad <public@kyllingen.net> wrote:

> On Friday, 12 April 2013 at 13:00:36 UTC, Robert wrote:
>
>> I would suggest, instead of writing your own std.process, that you simply provide a pull request to the now to be approved one, this way you can prove your point that it is easy by already improving a part of the standard library.
>>
>
> Actually, Manu (or anyone) could do this for *any* part of Phobos.  And considering that std.process is probably the module that would benefit the least from it, maybe he should start somewhere else?  std.string, perhaps, or std.algorithm?
>

I think I'll have a go at this after dconf. For now, I need to stop distracting myself from my lecture preparation! ;)

On Fri, 12 Apr 2013 14:25:22 +0100, Dmitry Olshansky <dmitry.olsh@gmail.com> wrote:

> 12-Apr-2013 17:20, Dicebot пишет:
>> On Friday, 12 April 2013 at 13:09:19 UTC, Lars T. Kyllingstad wrote:
>>> On Friday, 12 April 2013 at 13:00:36 UTC, Robert wrote:
>>>> I would suggest, instead of writing your own std.process, that you
>>>> simply provide a pull request to the now to be approved one, this way
>>>> you can prove your point that it is easy by already improving a part of
>>>> the standard library.
>>>
>>> Actually, Manu (or anyone) could do this for *any* part of Phobos.
>>> And considering that std.process is probably the module that would
>>> benefit the least from it, maybe he should start somewhere else?
>>> std.string, perhaps, or std.algorithm?
>>
>> Or probably druntime. This rise again and again - until standard global
>> allocators are not here it is all a premature and useless optimization.
>> It makes no sense to try to workaround a fundamental issue. That will be
>> a lot of effort for no real gain.
>>
>> Allocators are a blocker for almost any memory-related issue in both
>> druntime and phobos.
>
> Same here - trying to get out of your way to avoid GC right now is both painful and useless in the long run. More then that it's limited as you can't have the default allocation scheme that fits all nor use some scheme that would require change of API (so that later it's broken again once allocators come).

My impression is that there is some low hanging fruit here.  My impression (pls correct me if I'm wrong) is that Manu is not trying to avoid the GC, but to avoid allocations (and presumably collections) at the wrong moment in time.

So...

If the GC were to have a hook function for allocation and for free, and if when these were in use it would not itself trigger collection.

Then...

Manu could supply hook functions, use the alloc function to supply pre-allocated memory, and trigger collection as/when convenient.

For collection itself some upper bound time would also be desireable.

These ideas are nowhere near as complex as a full blown allocator proposal/idea.

R

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

On 13 April 2013 00:01, Regan Heath <regan@netmail.co.nz> wrote:

> On Fri, 12 Apr 2013 14:25:22 +0100, Dmitry Olshansky < dmitry.olsh@gmail.com> wrote:
>
>  12-Apr-2013 17:20, Dicebot пишет:
>>
>>> On Friday, 12 April 2013 at 13:09:19 UTC, Lars T. Kyllingstad wrote:
>>>
>>>> On Friday, 12 April 2013 at 13:00:36 UTC, Robert wrote:
>>>>
>>>>> I would suggest, instead of writing your own std.process, that you simply provide a pull request to the now to be approved one, this way you can prove your point that it is easy by already improving a part of the standard library.
>>>>>
>>>>
>>>> Actually, Manu (or anyone) could do this for *any* part of Phobos. And considering that std.process is probably the module that would benefit the least from it, maybe he should start somewhere else? std.string, perhaps, or std.algorithm?
>>>>
>>>
>>> Or probably druntime. This rise again and again - until standard global allocators are not here it is all a premature and useless optimization. It makes no sense to try to workaround a fundamental issue. That will be a lot of effort for no real gain.
>>>
>>> Allocators are a blocker for almost any memory-related issue in both druntime and phobos.
>>>
>>
>> Same here - trying to get out of your way to avoid GC right now is both painful and useless in the long run. More then that it's limited as you can't have the default allocation scheme that fits all nor use some scheme that would require change of API (so that later it's broken again once allocators come).
>>
>
> My impression is that there is some low hanging fruit here.  My impression (pls correct me if I'm wrong) is that Manu is not trying to avoid the GC, but to avoid allocations (and presumably collections) at the wrong moment in time.
>
> So...
>
> If the GC were to have a hook function for allocation and for free, and if when these were in use it would not itself trigger collection.
>
> Then...
>
> Manu could supply hook functions, use the alloc function to supply pre-allocated memory, and trigger collection as/when convenient.
>

Believe it or not, I'm not actually a fan of over-complexity. And I'm
focusing here on totally unnecessary allocations.
Isn't using the stack where applicable just a whole low easier? That's what
it's there for.

There are of course plenty of functions that do allocate by design, and in many cases, allocators would assist with those. But I'm not arguing for that right now, since allocators still seem to be a way off.

For collection itself some upper bound time would also be desireable.
>

Absolutely. Is it possible for collection to run like a coroutine? Can it run a fragmented collection process, each time picking up where it left off?

These ideas are nowhere near as complex as a full blown allocator
> proposal/idea.

12-Apr-2013 18:12, Manu пишет:
> On 13 April 2013 00:01, Regan Heath <regan@netmail.co.nz
> <mailto:regan@netmail.co.nz>> wrote:
>
>     So...
>
>     If the GC were to have a hook function for allocation and for free,
>     and if when these were in use it would not itself trigger collection.
>
>     Then...
>
>     Manu could supply hook functions, use the alloc function to supply
>     pre-allocated memory, and trigger collection as/when convenient.
>
>
> Believe it or not, I'm not actually a fan of over-complexity. And I'm
> focusing here on totally unnecessary allocations.
> Isn't using the stack where applicable just a whole low easier? That's
> what it's there for.

'cause nobody can tell you how big it is. This knowledge is only available to end user and there is still no easy way to "tell" that to the library. The end result is utterly useless as library can't reliably use stack space.

In the end if one library thinks it's fine to burn say 32K of stack with alloca but then it calls into another one that burns another chunk with alloca and so on untill happily stack overflowing (sometimes, on some systems at the right time!). Call graphs to calculate this is only available for the final app.

Maybe just adding a separate thread-local growable stack for data would work - at least it wouldn't depend on sheer luck and particular OS settings.

-- 
Dmitry Olshansky

On 13 April 2013 00:18, Dmitry Olshansky <dmitry.olsh@gmail.com> wrote:

> 12-Apr-2013 18:12, Manu пишет:
>
>> On 13 April 2013 00:01, Regan Heath <regan@netmail.co.nz <mailto:regan@netmail.co.nz>> wrote:
>>
>>     So...
>>
>>     If the GC were to have a hook function for allocation and for free,
>>     and if when these were in use it would not itself trigger collection.
>>
>>     Then...
>>
>>     Manu could supply hook functions, use the alloc function to supply
>>     pre-allocated memory, and trigger collection as/when convenient.
>>
>>
>> Believe it or not, I'm not actually a fan of over-complexity. And I'm
>> focusing here on totally unnecessary allocations.
>> Isn't using the stack where applicable just a whole low easier? That's
>> what it's there for.
>>
>
> 'cause nobody can tell you how big it is. This knowledge is only available to end user and there is still no easy way to "tell" that to the library. The end result is utterly useless as library can't reliably use stack space.
>

Filenames, strings, etc have a fairly predictable size. Allocate slightly
above that, and fallback to the heap upon overflow.
I think it's safe to assume the stack is 'big enough' if you apply some
common sense.
It measures in the megabytes on PC's, I'm used to working on machines with
~32k stack, I use it aggressively, and I don't tend to run out.

In the end if one library thinks it's fine to burn say 32K of stack with
> alloca but then it calls into another one that burns another chunk with alloca and so on untill happily stack overflowing (sometimes, on some systems at the right time!). Call graphs to calculate this is only available for the final app.
>
> Maybe just adding a separate thread-local growable stack for data would work - at least it wouldn't depend on sheer luck and particular OS settings.


If you're saying the stack is a limited resource, therefore it's unsafe to
use it, then you might as well argue that calling any function is an unsafe
process.
Some common sense is required. I wouldn't personally burn more than 1k in a
single function, unless I knew it was close to a leaf by design (which many
library calls are).
90% of what we're dealing with here are strings, and they tend to measure
in the 10s of bytes.

12-Apr-2013 18:40, Manu пишет:
> On 13 April 2013 00:18, Dmitry Olshansky <dmitry.olsh@gmail.com
> <mailto:dmitry.olsh@gmail.com>> wrote:
>         Believe it or not, I'm not actually a fan of over-complexity.
>         And I'm
>         focusing here on totally unnecessary allocations.
>         Isn't using the stack where applicable just a whole low easier?
>         That's
>         what it's there for.
>
>
>     'cause nobody can tell you how big it is. This knowledge is only
>     available to end user and there is still no easy way to "tell" that
>     to the library. The end result is utterly useless as library can't
>     reliably use stack space.
>
>
> Filenames, strings, etc have a fairly predictable size. Allocate
> slightly above that, and fallback to the heap upon overflow.
> I think it's safe to assume the stack is 'big enough' if you apply some
> common sense.

Up to 32K on WinNT. Hence the 32K reference.

> It measures in the megabytes on PC's, I'm used to working on machines
> with ~32k stack, I use it aggressively, and I don't tend to run out.

Just a moment ago you were arguing for some quite different platform :)

Regardless - think fibers on say servers. These get no more then around 64K of stack. In fact, current D fibers have something like 16K or 32K.
There even was report that writeln triggered stack overflow with these, so the size was extended a bit.

In general simple non-GUI threads on Windows get 64K by default (IRC).

>
>     In the end if one library thinks it's fine to burn say 32K of stack
>     with alloca but then it calls into another one that burns another
>     chunk with alloca and so on untill happily stack overflowing
>     (sometimes, on some systems at the right time!). Call graphs to
>     calculate this is only available for the final app.
>
>     Maybe just adding a separate thread-local growable stack for data
>     would work - at least it wouldn't depend on sheer luck and
>     particular OS settings.
>
>
> If you're saying the stack is a limited resource, therefore it's unsafe
> to use it, then you might as well argue that calling any function is an
> unsafe process.

s/limited/unpredictably limited/

"calling any function is an  unsafe process" - indeed in principle you don't know how much of stack these use unless you measure them or analyze otherwise.

It's awful that on 32 bit system you can't expect stack to be arbitrarily long (as much as you'd use of it) due to threads quickly using up all of virtual memory. On 64-bit something to that effect is achievable.

> Some common sense is required.

Exactly except that in the library there is no knowledge to get a measure of "common sense". It can't tell how somebody intends to use it, especially the standard library.

> I wouldn't personally burn more than 1k
> in a single function, unless I knew it was close to a leaf by design
> (which many library calls are).

And you trust that nobody will build a ton of wrappers on top of your function (even close to leaf one)? Come on, "they" all do it.

> 90% of what we're dealing with here are strings, and they tend to
> measure in the 10s of bytes.
???
Not at all, paths could easily get longer the 256, much to legacy apps chagrin.

-- 
Dmitry Olshansky