I wrote a little program that given some number it generates a list of different combinations (represented by a ubyte array), so in the end my function with name GenerateCombinations(int x) returns a ubyte[][] (list of arrays of ubytes).

Now the problem is, the quantity of combinations generated are pow(2, `x`), thus, giving 20 it returns me a list of 1_048_576 arrays of ubyte, every array has length of `x` so, in the end this function allocates (seeing the Windows Task Manager) 42 MB of data after input `20` the first time.

When the list is ready it prints the quantity of combinations given (combinations.length), after that I added a foreach(ubyte[] a; combinations) calling destroy to every array and then destroy(combinations), and before to re-input a new number to generate a new combination list I call GC.Collect() and GC.minimize().

My question is if I'm doing good calling destroy to those arrays, because I don't see any change in the quantity of memory the program is using: In the start it uses 1496 KB, after input `20` it now have 43000 KB (approx.) and if I input again 20 then it sums up to 90000 KB.

In the end I don't see some kind of real memory free. I'm doing something wrong?
That's the reason I'm asking for your advice.

PD: All arrays are constructed with the `new` expression.
PD2: Don't ask me why I don't better do an `int count` instead of constructing arrays, this little program is going to be part of a more huge application and I need the combination list.

Regards. JV

On Wednesday, 6 April 2016 at 19:54:32 UTC, Jonathan Villa wrote:
> I wrote a little program that given some number it generates a list of different combinations (represented by a ubyte array), so in the end my function with name GenerateCombinations(int x) returns a ubyte[][] (list of arrays of ubytes).
>

Sample code.

void main()
{
	while(true)
	{
		write("Alternatives quantity: ");
		string value = chomp(readln());
		
		if (value == "x")
			break;
		
		int x = to!int(value);
		
		ubyte[][] combs = GenerateCombinations(x);
		writefln("There are %d combinations.", combs.length);
		
		foreach(ubyte[] a; combs)
			destroy(a);
		destroy(combs);
		
		writeln();
		
		GC.collect();
		GC.minimize();
	}
	
	return;
}

On Wednesday, 6 April 2016 at 19:54:32 UTC, Jonathan Villa wrote:
> I wrote a little program that given some number it generates a

TL;DR: My program generates a very large `ubyte[][]`, and after I call destroy and GC.collect() and GC.minimize(), the memory occuping doesn't seems to decrease.

On Wednesday, 6 April 2016 at 19:54:32 UTC, Jonathan Villa wrote:
> I wrote a little program that given some number it generates a list of different combinations (represented by a ubyte array), so in the end my function with name GenerateCombinations(int x) returns a ubyte[][] (list of arrays of ubytes).
>
> ...

Why not make a range instead? No need to reserve memory for the entire array if you can compute the elements as-needed.

If you really want an array, std.experimental.allocator will let you manually allocate/release objects.

On Wednesday, 6 April 2016 at 21:33:14 UTC, Alex Parrill wrote:
> On Wednesday, 6 April 2016 at 19:54:32 UTC, Jonathan Villa wrote:
>> I wrote a little program that given some number it generates a ...
>
> Why not make a range instead? No need to reserve memory for the entire array if you can compute the elements as-needed.
>
> If you really want an array, std.experimental.allocator will let you manually allocate/release objects.

I'm going to try use a custom allocator for tomorrow.

My general idea is first to get the predicted quantity of combinations so I can divide and parallelize them.

What do you think can be the problem with the lack of deallocation?

Regards.

On Wednesday, 6 April 2016 at 23:14:10 UTC, Jonathan Villa wrote:
> On Wednesday, 6 April 2016 at 21:33:14 UTC, Alex Parrill wrote:
>
> My general idea is first to get the predicted quantity of combinations

Seems like you already know; your OP says you have 2^n combinations.

> so I can divide and parallelize them.

std.parallelism.parallel can do this for you, and works on ranges.

http://dlang.org/phobos/std_parallelism.html#.parallel

> What do you think can be the problem with the lack of deallocation?

Don't know exactly. destroy just runs the destructors, it doesn't free any memory. Since the arrays are still in scope, it might not be able to free them.

On Wednesday, 6 April 2016 at 20:30:33 UTC, Jonathan Villa wrote:
> On Wednesday, 6 April 2016 at 19:54:32 UTC, Jonathan Villa wrote:
>> I wrote a little program that given some number it generates a
>
> TL;DR: My program generates a very large `ubyte[][]`, and after I call destroy and GC.collect() and GC.minimize(), the memory occuping doesn't seems to decrease.

Anything change if you wrap your code like:


while ...
{
...
   {
      ubyte[][] ...
      ...
   }

   GC.collect();
   GC.minimieze();
}

?

On Thursday, 7 April 2016 at 10:02:05 UTC, Andrea Fontana wrote:
> On Wednesday, 6 April 2016 at 20:30:33 UTC, Jonathan Villa wrote:
>> On Wednesday, 6 April 2016 at 19:54:32 UTC, Jonathan Villa wrote:
>>> I wrote a little program that given some number it generates a
>>
>> TL;DR: My program generates a very large `ubyte[][]`, and after I call destroy and GC.collect() and GC.minimize(), the memory occuping doesn't seems to decrease.
>
> Anything change if you wrap your code like:
>
>
> while ...
> {
> ...
>    {
>       ubyte[][] ...
>       ...
>    }
>
>    GC.collect();
>    GC.minimieze();
> }
>
> ?

Or maybe you could try to do:

combs = null;

before collecting.

On Thursday, 7 April 2016 at 10:05:02 UTC, Andrea Fontana wrote:
> On Thursday, 7 April 2016 at 10:02:05 UTC, Andrea Fontana wrote:
>> On Wednesday, 6 April 2016 at 20:30:33 UTC, Jonathan Villa wrote:
>>
>> Anything change if you wrap your code like:
>>
>>
>> while ...
>> {
>> ...
>>    {
>>       ubyte[][] ...
>>       ...
>>    }
>>
>>    GC.collect();
>>    GC.minimieze();
>> }
>>
>> ?
>
> Or maybe you could try to do:
>
> combs = null;
>
> before collecting.


Yep, I enclosed the ubyte[][] allocation in a new scope, like our friend Alex suggested, but nothing seems to change. I'm going to try combs = null...

Big news, I've tested my code using Debian x64 8.2 and it really frees memory! even without adding `combs = null` (I'm going to add it anyway just in case).

So, looks like is some kind of Windows plataform problem. Next time I boot on Windows I'm going to add the null assignment and hope for results.

Regards!

On 4/6/16 3:54 PM, Jonathan Villa wrote:
> I wrote a little program that given some number it generates a list of
> different combinations (represented by a ubyte array), so in the end my
> function with name GenerateCombinations(int x) returns a ubyte[][] (list
> of arrays of ubytes).
>
> Now the problem is, the quantity of combinations generated are pow(2,
> `x`), thus, giving 20 it returns me a list of 1_048_576 arrays of ubyte,
> every array has length of `x` so, in the end this function allocates
> (seeing the Windows Task Manager) 42 MB of data after input `20` the
> first time.
>
> When the list is ready it prints the quantity of combinations given
> (combinations.length), after that I added a foreach(ubyte[] a;
> combinations) calling destroy to every array and then
> destroy(combinations), and before to re-input a new number to generate a
> new combination list I call GC.Collect() and GC.minimize().
>
> My question is if I'm doing good calling destroy to those arrays,
> because I don't see any change in the quantity of memory the program is
> using: In the start it uses 1496 KB, after input `20` it now have 43000
> KB (approx.) and if I input again 20 then it sums up to 90000 KB.
>
> In the end I don't see some kind of real memory free. I'm doing
> something wrong?
> That's the reason I'm asking for your advice.
>
> PD: All arrays are constructed with the `new` expression.
> PD2: Don't ask me why I don't better do an `int count` instead of
> constructing arrays, this little program is going to be part of a more
> huge application and I need the combination list.
>
> Regards. JV

You are likely running into the GC being conservative. You are also possibly running into an issue where you are expecting the compiler to do something with the stack where it may do something else. A classic question that is asked on D forums all the time is why when you null some variable and then call GC collect, the memory isn't collected. The answer is because quite possibly the value is still in a register, and the registers must be scanned too.

As for conservatism, if you generate a 1-million element array of pointers (each element has a pointer in it), then you have some random stack variable that "happens" to point into that giant array (more likely in 32-bit systems), then both the array, and all it's pointing at gets saved.

Note that destroying an array is equivalent to setting it to null. It doesn't write any data to the array itself.

Ironically, your example of this:

foreach(ubyte[] a; combs) destroy(a);

does absolutely nothing. Because all you are doing is setting a to null, and a is a local variable.

The only thing that could make a difference is destroying combs, which is equivalent to setting combs to null. But like I said, it's possible there is still a pointer in registers, or that some other stack variable points at your huge array.

-Steve