Thread overview
Assertion Error
Sep 12
vino
Sep 13
Vino.B
Sep 13
Vino.B
1 day ago
Vino.B
September 12
Hi All,

I have a small piece of code which executes perfectly 8 out of 10 times, very rarely it throws an assertion error, so is there a way to find which line of code is causing this error.

From,

Vino.B
September 12
On Tuesday, 12 September 2017 at 19:44:19 UTC, vino wrote:
> Hi All,
>
> I have a small piece of code which executes perfectly 8 out of 10 times, very rarely it throws an assertion error, so is there a way to find which line of code is causing this error.

You should be getting the line number as part of the crash, like here:

--- test.d ---
void main(string[] args)
{
	assert(args.length > 1);
}
--------------

-----------------
$ dmd -run test.d

core.exception.AssertError@test.d(3): Assertion failure
[Stack trace]
-----------------

If you don't what are the steps to reproduce?
September 13
On Tuesday, 12 September 2017 at 21:01:26 UTC, Moritz Maxeiner wrote:
> On Tuesday, 12 September 2017 at 19:44:19 UTC, vino wrote:
>> Hi All,
>>
>> I have a small piece of code which executes perfectly 8 out of 10 times, very rarely it throws an assertion error, so is there a way to find which line of code is causing this error.
>
> You should be getting the line number as part of the crash, like here:
>
> --- test.d ---
> void main(string[] args)
> {
> 	assert(args.length > 1);
> }
> --------------
>
> -----------------
> $ dmd -run test.d
>
> core.exception.AssertError@test.d(3): Assertion failure
> [Stack trace]
> -----------------
>
> If you don't what are the steps to reproduce?

Hi Max,

 I tried to run the code for at least 80+ time the code ran without any issue, will let you know in case if I hit the same issue in feature, Below is the piece of code, plese do let me know if you find any issue with the below code.

Program Code:
import core.stdc.stdlib: exit;
import std.algorithm: all, among, filter, map, setDifference, sort, uniq, each, joiner;
import std.array: appender, join;
import std.container.array;
import std.conv: to;
import std.datetime.systime: Clock, days, SysTime;
import std.file: SpanMode, dirEntries, exists, isFile, mkdir, remove, rmdirRecurse;
import std.getopt;
import std.parallelism: parallel, task, taskPool;
import std.path: absolutePath, baseName, dirName, isValidFilename, isValidPath, globMatch;
import std.range: empty, zip,  chain, chunks;
import std.stdio: File, writefln, writeln;
import std.string: chomp, chop, isNumeric, split, strip;
import std.typecons: tuple, Tuple;
import std.uni: isAlpha, toLower, isWhite;
import std.conv;

auto coSizeDirList (string FFs, int SizeDir) {
	int subdirTotal;
	int subdirTotalGB;
	Array!string Subdir;
	Array!string Subsize;
	Array!string Result;
	auto dFiles = Array!string ((dirEntries(FFs, SpanMode.shallow).filter!(a => a.isDir))[].map!(a => a.name));
    foreach (d; dFiles[]) {
				auto SdFiles = Array!ulong(dirEntries(d, SpanMode.depth).map!(a => a.size));
				foreach(f; SdFiles[]) { subdirTotal += f; }
				subdirTotalGB = (subdirTotal/1024/1024);
				if (subdirTotalGB > SizeDir) { Result ~= d; Result ~= to!string(subdirTotalGB); }
				subdirTotal = 0;
		    }
			return Result;
}

void ptSizeDirList (Array!string SizeDirlst, int SizeDir) {
 alias DirSizeList = typeof(coSizeDirList(string.init, int.init));
 auto MSresult = taskPool.workerLocalStorage!DirSizeList();
 foreach (string Fs; parallel(SizeDirlst[0 .. $], 1)) {
			auto FFs = Fs.strip;
			auto MSizeDirList = task(&coSizeDirList, FFs, SizeDir);
			MSizeDirList.executeInNewThread();
			auto MSizeDirListData = MSizeDirList.workForce;
			MSresult.get ~= MSizeDirListData;
			}
		foreach(i; MSresult.toRange)
		chain(i[]).chunks(2).each!(e => writefln!"%-60s %s"(e[0], e[1]));
		
}

void main () {
auto SizeDirlst = Array!string( "C:\\Temp\\TEST2\\BACKUP", "C:\\Temp\\TEST2\\EXPORT", "C:\\Temp\\TEST2\\PROD_TEAM", "C:\\Temp\\TEST2\\TEAM", "C:\\Temp\\TEST3\\BACKUP", "C:\\Temp\\TEST3\\EXPORT" );
int SizeDir = 10;
ptSizeDirList(SizeDirlst, SizeDir);
}

From,
Vino.B

September 13
On Wednesday, 13 September 2017 at 07:39:46 UTC, Vino.B wrote:
> On Tuesday, 12 September 2017 at 21:01:26 UTC, Moritz Maxeiner wrote:
>> On Tuesday, 12 September 2017 at 19:44:19 UTC, vino wrote:
>>> Hi All,
>>>
>>> I have a small piece of code which executes perfectly 8 out of 10 times, very rarely it throws an assertion error, so is there a way to find which line of code is causing this error.
>>
>> You should be getting the line number as part of the crash, like here:
>>
>> --- test.d ---
>> void main(string[] args)
>> {
>> 	assert(args.length > 1);
>> }
>> --------------
>>
>> -----------------
>> $ dmd -run test.d
>>
>> core.exception.AssertError@test.d(3): Assertion failure
>> [Stack trace]
>> -----------------
>>
>> If you don't what are the steps to reproduce?
>
> Hi Max,
>
>  I tried to run the code for at least 80+ time the code ran without any issue, will let you know in case if I hit the same issue in feature, Below is the piece of code, plese do let me know if you find any issue with the below code.
>
> Program Code:
> [...]
>  foreach (string Fs; parallel(SizeDirlst[0 .. $], 1))
>  {
> 			auto FFs = Fs.strip;
> 			auto MSizeDirList = task(&coSizeDirList, FFs, SizeDir);
> 			MSizeDirList.executeInNewThread();
> 			auto MSizeDirListData = MSizeDirList.workForce;
> 			MSresult.get ~= MSizeDirListData;
>  }

From reading I don't see anything that I would expect to assert, but I am wondering why you first parallelize your work with a thread pool (`parallel(...)`) and then inside each (implicitly created) task (that is already being serviced by a thread in the thread pool) you create another task, have it executed in a new thread, and make the thread pool thread wait for that thread to complete servicing that new task.
This should yield the same result, but without the overhead of spawning additional threads:

---
foreach (string Fs; parallel(SizeDirlst[0 .. $], 1))
{
  MSresult.get ~= coSizeDirList(Fs.strip, SizeDir);
}
---
September 13
On Wednesday, 13 September 2017 at 11:03:38 UTC, Moritz Maxeiner wrote:
> On Wednesday, 13 September 2017 at 07:39:46 UTC, Vino.B wrote:
>> On Tuesday, 12 September 2017 at 21:01:26 UTC, Moritz Maxeiner wrote:
>>> On Tuesday, 12 September 2017 at 19:44:19 UTC, vino wrote:
>>>> Hi All,
>>>>
>>>> I have a small piece of code which executes perfectly 8 out of 10 times, very rarely it throws an assertion error, so is there a way to find which line of code is causing this error.
>>>
>>> You should be getting the line number as part of the crash, like here:
>>>
>>> --- test.d ---
>>> void main(string[] args)
>>> {
>>> 	assert(args.length > 1);
>>> }
>>> --------------
>>>
>>> -----------------
>>> $ dmd -run test.d
>>>
>>> core.exception.AssertError@test.d(3): Assertion failure
>>> [Stack trace]
>>> -----------------
>>>
>>> If you don't what are the steps to reproduce?
>>
>> Hi Max,
>>
>>  I tried to run the code for at least 80+ time the code ran without any issue, will let you know in case if I hit the same issue in feature, Below is the piece of code, plese do let me know if you find any issue with the below code.
>>
>> Program Code:
>> [...]
>>  foreach (string Fs; parallel(SizeDirlst[0 .. $], 1))
>>  {
>> 			auto FFs = Fs.strip;
>> 			auto MSizeDirList = task(&coSizeDirList, FFs, SizeDir);
>> 			MSizeDirList.executeInNewThread();
>> 			auto MSizeDirListData = MSizeDirList.workForce;
>> 			MSresult.get ~= MSizeDirListData;
>>  }
>
> From reading I don't see anything that I would expect to assert, but I am wondering why you first parallelize your work with a thread pool (`parallel(...)`) and then inside each (implicitly created) task (that is already being serviced by a thread in the thread pool) you create another task, have it executed in a new thread, and make the thread pool thread wait for that thread to complete servicing that new task.
> This should yield the same result, but without the overhead of spawning additional threads:
>
> ---
> foreach (string Fs; parallel(SizeDirlst[0 .. $], 1))
> {
>   MSresult.get ~= coSizeDirList(Fs.strip, SizeDir);
> }
> ---

Hi Max,

 Below is the explanation of the above code.

The Fs that is passed to the function ptSizeDirList is any array of 10 -15 file system (NetApp Filers)from 15 different servers mounted on a single server (Network share).

The function ptSizeDirList pass each of the FS to the function coSizeDirList in parallel thread to find the folder size under each of these Fs,

Each of this FS has around 1000+ folders. so the requirement is that we need the size of each of the folders under each of these 15 Fs along with the folder name in less than an hour or two as this script is about to be schedule to run once every 3 hours, and at present we are able to achieve the same in 10 mins / fs

Each Fs is of about 10-15 TB's

Hence this code was written.
September 13
On Wednesday, 13 September 2017 at 15:12:57 UTC, Vino.B wrote:
> On Wednesday, 13 September 2017 at 11:03:38 UTC, Moritz Maxeiner wrote:
>> On Wednesday, 13 September 2017 at 07:39:46 UTC, Vino.B wrote:
>>>
>>> Hi Max,
>>>
>>>  [...]
>>>
>>> Program Code:
>>> [...]
>>>  foreach (string Fs; parallel(SizeDirlst[0 .. $], 1))
>>>  {
>>> 			auto FFs = Fs.strip;
>>> 			auto MSizeDirList = task(&coSizeDirList, FFs, SizeDir);
>>> 			MSizeDirList.executeInNewThread();
>>> 			auto MSizeDirListData = MSizeDirList.workForce;
>>> 			MSresult.get ~= MSizeDirListData;
>>>  }
>>
>> [...]
>>
>> ---
>> foreach (string Fs; parallel(SizeDirlst[0 .. $], 1))
>> {
>>   MSresult.get ~= coSizeDirList(Fs.strip, SizeDir);
>> }
>> ---
>
> Hi Max,

It's Moritz, not Max. ;)

>
>  Below is the explanation of the above code.
>
> [...]

AFAICT that's a reason why you want parallelization of coSizeDirList, but not why you need to spawn another thread inside of an *already parallelelized" task. Try my shortened parallel foreach loop vs your longer one and monitor system load (threads, memory, etc).
1 day ago
On Wednesday, 13 September 2017 at 15:27:30 UTC, Moritz Maxeiner wrote:
> On Wednesday, 13 September 2017 at 15:12:57 UTC, Vino.B wrote:
>> On Wednesday, 13 September 2017 at 11:03:38 UTC, Moritz Maxeiner wrote:
>>> On Wednesday, 13 September 2017 at 07:39:46 UTC, Vino.B wrote:
>>>>  [...]
>>>
>>> [...]
>>>
>>> ---
>>> foreach (string Fs; parallel(SizeDirlst[0 .. $], 1))
>>> {
>>>   MSresult.get ~= coSizeDirList(Fs.strip, SizeDir);
>>> }
>>> ---
>>
>> Hi Max,
>
> It's Moritz, not Max. ;)
>
>>
>>  Below is the explanation of the above code.
>>
>> [...]
>
> AFAICT that's a reason why you want parallelization of coSizeDirList, but not why you need to spawn another thread inside of an *already parallelelized" task. Try my shortened parallel foreach loop vs your longer one and monitor system load (threads, memory, etc).

Hi Moritz,

 Thank you very much, it was very helpful and time saving and fast.

From,
Vino.B