View mode: basic / threaded / horizontal-split · Log in · Help
February 19, 2012
Everything on the Stack
Thanks to a reddit thread I stumbled over D and thought I should give
it a chance. I'm currently working on project where hard RT
requirements matter. In most places the GC is not suitable. I tried to
port some core functionaly of an existing application which is
currently written in C/C++. I tried to convert some things 1:1 to D,
well aware that a better approach may exist.

I think that only "struct" and "scope class" are suitable to create
objects. structs have the simple advantage that I don't have to type
"scope" in my whole program (can't alias it e.g.), but have the
disadvantage that I can't use an interface nor inheritance.

The application should have a dictionary which contains values of many
different objects. For this example, all objects should support a toFoo
method. To make it generic I've to rule out a struct, or to write
everything in a more functional style (a everythingToFoo method handles
all different types).

interface Entry {
	Foo toFoo();
}

scope class EntryNumeric(T) : Entry {
	T value = T.init;

	Foo toFoo() {
		...
	}
}

alias Entry!(int) EntryInt;
alias Entry!(float) EntryFloat;

setup(Entry[] entries) {
	scope auto e1 = new EntryInt();
	Foo = e1.toFoo();

	scope auto e2 = new EntryInt();
	Foo = e2.toFoo();

	entries[0] = e1;
	entries[1] = e2;
}

init() {
	Entry entries[2];
	setup(entries);
	entries[0].toFoo();
	entries[1].toFoo();
}

I wonder why I can compile the above piece, because e1/e2 shouldn't be
allowed to leave the scope. It obviously crashes.

The next thing I tried to solve is some kind of a templated Queue
class, which can store anything. In C++ it's easy to write a container
class that only works on the stack and uses some uninitialized bytes of
memory and placement new to copy construct objects into the Queue. With
"scope class" this seems to fail because it seems to be impossible to
return a class if it's type uses "scope" class. Missing copy
construction one could probably solve manually, by just carefully
create an own core class library (or just juse memcpy).

struct Queue(T, int N) {
}

scope class MyClass {
}

Queue!(MyClass) m;

Another problem with my whole, everything on the stack idea, is that D
limits my arrays to an internal limit (any paramter to change it?).
It's easy to create a stack overflow anyway, so why a limit? That
static arrays are stored in the .data section make things even more
complicated. That the new core.memory.GC somehow misses a function to
analyze if there is any hidden malloc in any used function would be
usefull as well, because it's not always obvious which construct had
what effect (forgetting scope on a delegate etc...)

Thanks for any insight, tips, hints, ...
February 20, 2012
Re: Everything on the Stack
> 	scope auto e1 = new EntryInt();
> 	Foo = e1.toFoo();

You don't need auto there.
And scope is deprecated, use std.typecons' scoped
February 20, 2012
Re: Everything on the Stack
Janthinidae:

> Thanks to a reddit thread I stumbled over D and thought I should give
> it a chance.

Welcome here then. Your post is quite interesting, your questions are complex.


> I'm currently working on project where hard RT
> requirements matter. In most places the GC is not suitable.

Please, show me the benchmarks you have seen or written, that rule out the D GC for your purposes. There's always the possibility that you have written bad benchmarks.


> I tried to
> port some core functionaly of an existing application which is
> currently written in C/C++. I tried to convert some things 1:1 to D,
> well aware that a better approach may exist.

It looks like an interesting project for D, also useful to spot possible problems in D itself or its Phobos.


> I think that only "struct" and "scope class" are suitable to create
> objects. structs have the simple advantage that I don't have to type
> "scope" in my whole program (can't alias it e.g.), but have the
> disadvantage that I can't use an interface nor inheritance.

scope, used for your purpose, is deprecated (another usage of scope, for function arguments, to avoid them to exit the function scope, or to avoid to pass heap-allocated closures to functions, is still usable).
As Trass3r has said, there is a replacement for scoped classes (but it has some problems still).


> I wonder why I can compile the above piece, because e1/e2 shouldn't be
> allowed to leave the scope. It obviously crashes.

scope was deprecated also because it was broken. The library defined "scoped" is similarly broken, but better to have a broken library than a broken language feature :-)


> The next thing I tried to solve is some kind of a templated Queue
> class, which can store anything. In C++ it's easy to write a container
> class that only works on the stack and uses some uninitialized bytes of
> memory and placement new to copy construct objects into the Queue. With
> "scope class" this seems to fail because it seems to be impossible to
> return a class if it's type uses "scope" class. Missing copy
> construction one could probably solve manually, by just carefully
> create an own core class library (or just juse memcpy).

Generally what's possible in C++ is possible in D too, even if maybe in a different way. I think all you say here is doable in D too, with different means.


> Another problem with my whole, everything on the stack idea, is that D
> limits my arrays to an internal limit (any paramter to change it?).
> It's easy to create a stack overflow anyway, so why a limit?

Walter says the limit is present because linkers have such limits and different linkers have different limits. So I think to allow people to compile (link) D code with different linkers, there is a built-in limit that is probably smaller than the limit of most linkers.
Walter also says that past a certain size, it's not good to allocate an array on the stack, and it's better to allocate it on the heap. If you really want the stack there is alloca(), but I think allocating a single chunk of memory from the D-GC or C heaps at the start of the program, doesn't damage the hard RT qualities of your code.


> That static arrays are stored in the .data section make things
> even more complicated.

I am not expert on this, sorry. What problems is this giving you?


> That the new core.memory.GC somehow misses a function to
> analyze if there is any hidden malloc in any used function would be
> usefull as well, because it's not always obvious which construct had
> what effect (forgetting scope on a delegate etc...)

I have an enhancement request related to this, that usually people appreciate:
http://d.puremagic.com/issues/show_bug.cgi?id=5070
I think there is a similar enhancement request for generic allocations. Plus I have suggested a @noheap annotation, but it was not appreciated by Don.
I suggest you to ask for this feature in the main D newsgroup. The more people asks for something similar, the more probable/sooner that it will be implemented.

Bye,
bearophile
February 21, 2012
Re: Everything on the Stack
scope/scoped isn't broken, they're just not safe.  It's better to have an 
unsafe library feature than an unsafe language feature.
February 21, 2012
Re: Everything on the Stack
On 02/21/2012 11:27 AM, Daniel Murphy wrote:
> scope/scoped isn't broken, they're just not safe.  It's better to have an
> unsafe library feature than an unsafe language feature.
>
>

scope is broken because it is not enforced by the means of 
flow-analysis. As a result, it is not safe. Copying it to the library 
and temporarily disabling 'scope' for classes is a good move however, 
because this means we will be able to fix it at an arbitrary point in 
the future without additional code breakage.
February 21, 2012
Re: Everything on the Stack
On 21/02/12 12:12, Timon Gehr wrote:
> On 02/21/2012 11:27 AM, Daniel Murphy wrote:
>> scope/scoped isn't broken, they're just not safe. It's better to have an
>> unsafe library feature than an unsafe language feature.
>>
>>
>
> scope is broken because it is not enforced by the means of
> flow-analysis. As a result, it is not safe. Copying it to the library
> and temporarily disabling 'scope' for classes is a good move however,
> because this means we will be able to fix it at an arbitrary point in
> the future without additional code breakage.

Does the library solution actually work the same as the language solution?
February 21, 2012
Re: Everything on the Stack
Don Clugston:

> Does the library solution actually work the same as the language solution?

Some time ago scoped worked worse than the built-in scope:
http://d.puremagic.com/issues/show_bug.cgi?id=5115

But I don't know how well scoped works now, I have never used it any more.

Bye,
bearophile
February 22, 2012
Re: Everything on the Stack
First thanks for you long response,

> > I'm currently working on project where hard RT
> > requirements matter. In most places the GC is not suitable.
> 
> Please, show me the benchmarks you have seen or written, that rule out the D GC for your purposes. There's always the possibility that you have written bad benchmarks.

There is no benchmark, it's just that the software has to have a
worstcase latency to some relevant below 1 ms. If the GC kicks in the
wrong time shit happens. With not using malloc at all or the GC I just
have some kind of proof that this will never be a problem.

> > I think that only "struct" and "scope class" are suitable to create
> > objects. structs have the simple advantage that I don't have to type
> > "scope" in my whole program (can't alias it e.g.), but have the
> > disadvantage that I can't use an interface nor inheritance.
> 
> scope, used for your purpose, is deprecated (another usage of scope, for function arguments, to avoid them to exit the function scope, or to avoid to pass heap-allocated closures to functions, is still usable).
> As Trass3r has said, there is a replacement for scoped classes (but it has some problems still).

I looked at this class and just have seen that it internally uses a
uint8 array to store the data. This means executable size gets larger
and larger like for any other array.

> scope was deprecated also because it was broken. The library defined "scoped" is similarly broken, but better to have a broken library than a broken language feature :-)

Ok didn't knew that it is deprecated.

> > The next thing I tried to solve is some kind of a templated Queue
> > class, which can store anything. In C++ it's easy to write a container
> > class that only works on the stack and uses some uninitialized bytes of
> > memory and placement new to copy construct objects into the Queue. With
> > "scope class" this seems to fail because it seems to be impossible to
> > return a class if it's type uses "scope" class. Missing copy
> > construction one could probably solve manually, by just carefully
> > create an own core class library (or just juse memcpy).
> 
> Generally what's possible in C++ is possible in D too, even if maybe in a different way. I think all you say here is doable in D too, with different means.

Yes both are Turing complete :) But after all I hope that D makes
things easier and helps me to spot errors of certain aspects of my
program.

> > Another problem with my whole, everything on the stack idea, is that D
> > limits my arrays to an internal limit (any paramter to change it?).
> > It's easy to create a stack overflow anyway, so why a limit?
> 
> Walter says the limit is present because linkers have such limits and different linkers have different limits. So I think to allow people to compile (link) D code with different linkers, there is a built-in limit that is probably smaller than the limit of most linkers.
> Walter also says that past a certain size, it's not good to allocate an array on the stack, and it's better to allocate it on the heap. If you really want the stack there is alloca(), but I think allocating a single chunk of memory from the D-GC or C heaps at the start of the program, doesn't damage the hard RT qualities of your code.

Ok I understand it, but if some linkers fail, then isn't this the
linkgers problem? I still think something like an option would be nice
until the compiler can ask the linker (which never may happen). Even if
we would say D has the correct approach here, then one can ask why it's
not possible to write

int[10000000] t10 = void;

but possible to write

int[1000000] t0 = void;
...
int[1000000] t0 = void;

But the problem for me here is that all static arrays are in the .data
section (that's why I mentioned it) and make the executable that big.
If I declare an array with "Type[N] = void" I somehow expect, that it
is not initialized and that there is no data allocated in the .data
section.

Trying everything with alloca is maybe a better idea (and rewriting the
Scope template).

> > That the new core.memory.GC somehow misses a function to
> > analyze if there is any hidden malloc in any used function would be
> > usefull as well, because it's not always obvious which construct had
> > what effect (forgetting scope on a delegate etc...)
> 
> I have an enhancement request related to this, that usually people appreciate:
> http://d.puremagic.com/issues/show_bug.cgi?id=5070
> I think there is a similar enhancement request for generic allocations. Plus I have suggested a @noheap annotation, but it was not appreciated by Don.
> I suggest you to ask for this feature in the main D newsgroup. The more people asks for something similar, the more probable/sooner that it will be implemented.

Ok I'll do that. Thanks once more.
February 23, 2012
Re: Everything on the Stack
Janthinidae:

> There is no benchmark, it's just that the software has to have a
> worstcase latency to some relevant below 1 ms. If the GC kicks in the
> wrong time shit happens. With not using malloc at all or the GC I just
> have some kind of proof that this will never be a problem.
> ...
> Trying everything with alloca is maybe a better idea (and rewriting the
> Scope template).

The GC gets active when you allocate memory. So if you allocate some GC-heap memory at the start of your program, and then you never allocate some more of it, the GC will be dormant. So it's not required to allocate everything on the stack.


> Yes both are Turing complete :)

I meant to add "with similar efficiency of C++ code too" :-)


> Ok I understand it, but if some linkers fail, then isn't this the
> linkgers problem?

I don't know, but I presume lot of people lump compiler and linker in their mind.


> then one can ask why it's
> not possible to write
> 
> int[10000000] t10 = void;
> 
> but possible to write
> 
> int[1000000] t0 = void;
> ...
> int[1000000] t0 = void;

I don't know. I guess the total amount of memory and the largest chunk of memory are two different parameters for a linker.


> But the problem for me here is that all static arrays are in the .data
> section (that's why I mentioned it) and make the executable that big.
> If I declare an array with "Type[N] = void" I somehow expect, that it
> is not initialized and that there is no data allocated in the .data
> section.

Try this too:

__gshared int[1_000_000] array;
void main() {}


> Ok I'll do that. Thanks once more.

Generally Walter listens to efficiency concerns. You are welcome.

Bye,
bearophile
Top | Discussion index | About this forum | D home