Jump to page: 1 2
Thread overview
The app hanging after reach 1750MB of RAM
Apr 18, 2017
Suliman
Apr 18, 2017
Suliman
Apr 18, 2017
Stanislav Blinov
Apr 18, 2017
Stanislav Blinov
Apr 18, 2017
Suliman
Apr 19, 2017
Suliman
Apr 19, 2017
Suliman
Apr 19, 2017
Stanislav Blinov
Apr 24, 2017
Suliman
Apr 19, 2017
crimaniak
Apr 19, 2017
Suliman
April 18, 2017
I am writing app that extract data from DB to array of structures.

	void getSingleTrackInfo()
	{
		
		foreach(item; getTablesGPSSensorList)
		{
			ResultRange result = mysqlconnection.query(sqlquery);
			auto MySQLPointsLonLat = result.array;
			
			carGPSPoint cargpspoint; // create struct
			carGPSPoint [] cargpspoints; // create array of structures

			foreach(i, point;MySQLPointsLonLat)
			{
				cargpspoint.id = point[0].coerce!ulong;
				cargpspoint.recordDate = point[1].coerce!string;
				cargpspoint.velocity = point[2].coerce!double;
				cargpspoint.lat = point[3].coerce!string;
				cargpspoint.lon = point[4].coerce!string;
				cargpspoints ~=cargpspoint;
					
			}

		}
	}


I expected that on every step:
`foreach(item; getTablesGPSSensorList)` structures `carGPSPoint` will be recreated and memory will be free. But App after starting begin eat memory, and do it till reach 1750 of RAM. After it it's not crush, but simply stop other processing.

1. By the code on every iterate `carGPSPoint` should be recreated and memory should be free, why this does not happen?
2. How to free my memory?

April 18, 2017
Also I can't understand why app take so much memory? I checked array of structures size with this code:

auto mymem = cargpspoints.length * typeof(cargpspoints[0]).sizeof;			
writeln(mymem);

And it's print: 16963440
it's about 16MB...

What is takes all other memory?
April 18, 2017
On Tuesday, 18 April 2017 at 13:28:57 UTC, Suliman wrote:
> Also I can't understand why app take so much memory? I checked array of structures size with this code:
>
> auto mymem = cargpspoints.length * typeof(cargpspoints[0]).sizeof;			
> writeln(mymem);
>
> And it's print: 16963440
> it's about 16MB...
>
> What is takes all other memory?

1. You're measuring it wrong. Array length is already measured in terms of type size. But remember that every call result.array will allocate, and very call coerce!string will also allocate.
2. Since you're iterating over every result once, there's no point in converting it to an array first.
3. Consider using the Row.toStruct() method for conversion.
4. Consider using std.array.appender instead of ~=.

void getSingleTrackInfo()
{

    foreach(item; getTablesGPSSensorList)
    {
        ResultRange result = mysqlconnection.query(sqlquery);

        carGPSPoint cargpspoint; // create struct
        auto arr = appender!(carGPSPoint[]); // create array of structures

        foreach(row; result)
        {
            arr ~= row.toStruct(cargpspoint);
        }

        // arr.data should hold the array of structures
    }
}

April 18, 2017
On Tuesday, 18 April 2017 at 14:09:28 UTC, Stanislav Blinov wrote:

>         foreach(row; result)
>         {
>             arr ~= row.toStruct(cargpspoint);
>         }

Sorry, this should be

foreach(row; result)
{
    row.toStruct(cargpspoint);
    arr ~= cargpspoint;
}
April 18, 2017
On Tuesday, 18 April 2017 at 14:15:59 UTC, Stanislav Blinov wrote:
> On Tuesday, 18 April 2017 at 14:09:28 UTC, Stanislav Blinov wrote:
>
>>         foreach(row; result)
>>         {
>>             arr ~= row.toStruct(cargpspoint);
>>         }
>
> Sorry, this should be
>
> foreach(row; result)
> {
>     row.toStruct(cargpspoint);
>     arr ~= cargpspoint;
> }

Thanks I will try! But why in my code memory do not released? Do you have any idea?
April 19, 2017
>> auto mymem = cargpspoints.length * typeof(cargpspoints[0]).sizeof;			
>> writeln(mymem);
>>
>> And it's print: 16963440
>> it's about 16MB...
>>
>> What is takes all other memory?
>
> 1. You're measuring it wrong. Array length is already measured in terms of type size.

So should I do:
cargpspoints.length * cargpspoints[0].sizeof ?

Btw, `cargpspoints.length * typeof(cargpspoints[0]).sizeof` and `cargpspoints.length * cargpspoints[0].sizeof` show same result.
April 19, 2017
I have added GC.stat https://dlang.org/library/core/memory/gc.stats.html here the result:

freeSize: 49698640  | usedSize: 170502320
freeSize: 41174592  | usedSize: 217823680
freeSize: 53868576  | usedSize: 247072736
freeSize: 86494800  | usedSize: 307769776
freeSize: 58176640  | usedSize: 499665792
freeSize: 148233232 | usedSize: 534389744
freeSize: 148141376 | usedSize: 534481600
freeSize: 108467312 | usedSize: 641264528
freeSize: 55118432  | usedSize: 694613408
freeSize: 80579472  | usedSize: 803370096
freeSize: 78444480  | usedSize: 1006831680
freeSize: 291629360 | usedSize: 860755664
freeSize: 242912736 | usedSize: 976581152
freeSize: 241673232 | usedSize: 977820656
freeSize: 168092160 | usedSize: 1118510592
freeSize: 128405616 | usedSize: 1426632592
freeSize: 73146272  | usedSize: 1616109664
freeSize: 17962320  | usedSize: 1671293616
freeSize: 20342912  | usedSize: 1736021888

on the last value app is hanging.


April 19, 2017
On Tuesday, 18 April 2017 at 11:43:24 UTC, Suliman wrote:
> I am writing app that extract data from DB to array of structures.
>
> 	void getSingleTrackInfo()
> 	{
> 		
> 		foreach(item; getTablesGPSSensorList)
> 		{
> 			ResultRange result = mysqlconnection.query(sqlquery);
> 			auto MySQLPointsLonLat = result.array;
Is ResultRange closing query when exhausted? Did you try to call .close() on it after work?
April 19, 2017
On Wednesday, 19 April 2017 at 15:18:32 UTC, crimaniak wrote:
> On Tuesday, 18 April 2017 at 11:43:24 UTC, Suliman wrote:
>> I am writing app that extract data from DB to array of structures.
>>
>> 	void getSingleTrackInfo()
>> 	{
>> 		
>> 		foreach(item; getTablesGPSSensorList)
>> 		{
>> 			ResultRange result = mysqlconnection.query(sqlquery);
>> 			auto MySQLPointsLonLat = result.array;
> Is ResultRange closing query when exhausted? Did you try to call .close() on it after work?

Yes I tried:

ResultRange result = mysqlconnection.query(sqlquery);
auto MySQLPointsLonLat = result.array;
result.close();

Did not help :(
April 19, 2017
On Wednesday, 19 April 2017 at 07:28:32 UTC, Suliman wrote:

>> 1. You're measuring it wrong. Array length is already measured in terms of type size.
>
> So should I do:
> cargpspoints.length * cargpspoints[0].sizeof ?

No. .sizeof is the statically known size of a type, it can't take into account dynamically allocated memory.
cargpspoint.length * cargpspoints[0].sizeof will tell you the estimate size of the array, in bytes.
But each of the elements also has strings - dynamic arrays that are allocated elsewhere, their length is not included in that calculation.
So you could iterate over the array and sum up lengths of all strings to get an estimate.
Even then, that's just that: an estimate. Actual amount of memory allocated for a dynamic array T[] *may* be greater than length * T.sizeof. The only way to know that is to query the allocator used (in this case, GC).

« First   ‹ Prev
1 2