Hi,


I have just started to learn D. Its a great language. I am trying to achieve the following but I am not sure is it possible or should be done at all:

I want to have one array where I will store like 100000  objects.

But I want to use 4 threads where each thread will create 25000 objects and store them in array above mentioned. And all 4 threads should be working in parallel because I have 4 core processor for example. I do not care in which order objects are created nor objects should be aware of one another. I just need them stored in array.

Can threading help in creating many objects at once?

Note that I am beginner at working with threads so any help is welcome :)


Thanks

On 05/05/2014 10:14 AM, Caslav Sabani wrote:

> I want to have one array where I will store like 100000  objects.
>
> But I want to use 4 threads where each thread will create 25000 objects
> and store them in array above mentioned.

1) If it has to be a single array, meaning that all of the objects are in consecutive memory, you can create the array and give four slices of it to the four tasks.

To do that, you can either create a proper D array filled with objects with .init values; or you can allocate any type of memory and create objects in place there.

2) If it doesn't have to a single array, you can have the four tasks create four separate arrays. You can then use them as a single range by std.range.chain. This option allows you to have a single array as well.

I would like to give examples of those methods later. Gotta go now... :)

> And all 4 threads should be working in parallel because I have 4 core
> processor for example. I do not care in which order objects are created nor
> objects should be aware of one another. I just need them stored in array.
> Can threading help in creating many objects at once?  Note that I am
> beginner at working with threads so any help is welcome :) Thanks

I recommend looking at std.parallelism first and then std.concurrency. Here are two chapters that may be helpful:

  http://ddili.org/ders/d.en/parallelism.html

  http://ddili.org/ders/d.en/concurrency.html

Ali

On 05/05/2014 10:25 AM, Ali Çehreli wrote:

> On 05/05/2014 10:14 AM, Caslav Sabani wrote:
>
>  > I want to have one array where I will store like 100000  objects.
>  >
>  > But I want to use 4 threads where each thread will create 25000 objects
>  > and store them in array above mentioned.
>
> 1) If it has to be a single array, meaning that all of the objects are
> in consecutive memory, you can create the array and give four slices of
> it to the four tasks.
>
> To do that, you can either create a proper D array filled with objects
> with .init values; or you can allocate any type of memory and create
> objects in place there.

Here is an example:

import std.stdio;
import std.parallelism;
import core.thread;
import std.conv;

enum elementCount = 8;
size_t elementPerThread;

static this ()
{
    assert((elementCount % totalCPUs) == 0,
           "Cannot distribute tasks to cores evenly");

    elementPerThread = elementCount / totalCPUs;
}

void main()
{
    auto arr = new int[](elementCount);

    foreach (i; 0 .. totalCPUs) {
        const beg = i * elementPerThread;
        const end = beg + elementPerThread;
        arr[beg .. end] = i.to!int;
    }

    thread_joinAll();    // (I don't think this is necessary with std.parallelism)

    writeln(arr);    // [ 0, 0, 1, 1, 2, 2, 3, 3 ]
}

> 2) If it doesn't have to a single array, you can have the four tasks
> create four separate arrays. You can then use them as a single range by
> std.range.chain.

That is a lie. :) chain would work but it had to know the number of total cores at compile time. Instead, joiner or join can be used:

import std.stdio;
import std.parallelism;
import core.thread;

enum elementCount = 8;
size_t elementPerThread;

static this ()
{
    assert((elementCount % totalCPUs) == 0,
           "Cannot distribute tasks to cores evenly");

    elementPerThread = elementCount / totalCPUs;
}

void main()
{
    auto arr = new int[][](totalCPUs);

    foreach (i; 0 .. totalCPUs) {
        foreach (e; 0 .. elementPerThread) {
            arr[i] ~= i;
        }
    }

    thread_joinAll();    // (I don't think this is necessary with std.parallelism)

    writeln(arr);             // [[0, 0], [1, 1], [2, 2], [3, 3]]

    import std.range;
    writeln(arr.joiner);      // [ 0, 0, 1, 1, 2, 2, 3, 3 ]

    import std.algorithm;
    auto arr2 = arr.joiner.array;
    static assert(is (typeof(arr2) == int[]));
    writeln(arr2);           // [ 0, 0, 1, 1, 2, 2, 3, 3 ]

    auto arr3 = arr.join;
    static assert(is (typeof(arr3) == int[]));
    writeln(arr3);           // [ 0, 0, 1, 1, 2, 2, 3, 3 ]
}

> This option allows you to have a single array as well.

arr2 and arr3 above are examples of that.

Ali

Hi Ali,


Thanks for your reply. But I am struggling to understand from your example where is the code that creates or spawns new thread.


How do you create new thread and fill array with instantiated objects in that thread?



Thanks

On 05/05/2014 01:38 PM, Caslav Sabani wrote:

> I am struggling to understand from your example where is the code that
> creates or spawns new thread.

The .parallel in the foreach loop makes the body of the loop be executed in parallel.

> How do you create new thread and fill array with instantiated objects in
> that thread?

It is automatic in that example but you can created thread explicitly by std.concurrency or core.thread as well.

Ali

On Monday, 5 May 2014 at 17:14:54 UTC, Caslav Sabani wrote:
> Hi,
>
>
> I have just started to learn D. Its a great language. I am trying to achieve the following but I am not sure is it possible or should be done at all:
>
> I want to have one array where I will store like 100000  objects.
>
> But I want to use 4 threads where each thread will create 25000 objects and store them in array above mentioned. And all 4 threads should be working in parallel because I have 4 core processor for example. I do not care in which order objects are created nor objects should be aware of one another. I just need them stored in array.
>
> Can threading help in creating many objects at once?
>
> Note that I am beginner at working with threads so any help is welcome :)
>
>
> Thanks

I could be wrong here, but I think that the GC actually blocks when creating objects, and thus multiple threads creating instances would not provide a significant speedup, possibly even a slowdown.

You'd want to benchmark this to be certain it helps.

On 05/05/2014 02:38 PM, Kapps wrote:

> I think that the GC actually blocks when
> creating objects, and thus multiple threads creating instances would not
> provide a significant speedup, possibly even a slowdown.

Wow! That is the case. :)

> You'd want to benchmark this to be certain it helps.

I did:

import std.range;
import std.parallelism;

class C
{}

void foo()
{
    auto c = new C;
}

void main(string[] args)
{
    enum totalElements = 10_000_000;

    if (args.length > 1) {
        foreach (i; iota(totalElements).parallel) {
            foo();
        }

    } else {
        foreach (i; iota(totalElements)) {
            foo();
        }
    }
}

Typical run on my system for "-O -noboundscheck -inline":

$ time ./deneme parallel

real	0m4.236s
user	0m4.325s
sys	0m9.795s

$ time ./deneme

real	0m0.753s
user	0m0.748s
sys	0m0.003s

Ali

Hi all,


Thanks for your reply. So basically using threads in D for creating multiple instances of class is actually slower.


But what does exactly means that Garbage Collector blocks? What does it blocks and in which way?



Thanks

On 05/05/2014 04:32 PM, Caslav Sabani wrote:

> So basically using threads in D for creating multiple instances of class is
> actually slower.

Not at all! That statement can be true only in certain programs. :)

Ali

On Tuesday, 6 May 2014 at 03:26:52 UTC, Ali Çehreli wrote:
> On 05/05/2014 04:32 PM, Caslav Sabani wrote:
>
> > So basically using threads in D for creating multiple
> instances of class is
> > actually slower.
>
> Not at all! That statement can be true only in certain programs. :)
>
> Ali


But what does exactly means that Garbage Collector blocks? What
does it blocks and in which way?


And can I use threads to create multiple instance faster or that is just not possible?



Thanks