Hi,

I'm writing a picture browser program and I'd like to ask for a bit of assistance.

My app is basically a Win32 window with an OpenGL surface. Currently I'm able to display a lot of pictures in it. The decompression of the bitmaps take place at the beginning of the program.

I want to make this more user friendly, so at first I only draw placeholders for the pictures while the actual loading and decompression is happening on worker threads.

I used std.concurrency.spawn. The parameters are the filename, and an index. After the decompression I let the main thread know about there is a bitmap it should upload to a texture later. I do it using synchronize, it works without error and was really easy to implements in D.

My problem is that when I launch 1000 spawn(), the work overload on the main thread is so heavy that the 60FPS timer which is refreshing the windows never can be called. Obviously spawn() is not the right choice for this.

In my previous projects I addressed this by creating NumberOfProcessorThreads threads,and locked them on specific ProcessorThreads by using ProcessAffinity, so the main thread has enough time to generate fluid user interface while the other worker threads used 100%CPU for the fastest processing.

I also remember that with OPENMP I was also able to lock workers onto specific ProcessorThreads, it also worked fine.

But how can I do this elegantly in D?

Thanks in advance!


(Also I'm not sure how spaw()-n is working exactly. Is it creates a new process for every spawn() call, or is it assigns the task to one of the N worker processes? 1000 CreateProcess() api calls would be not so nice. I just wanna know it in order to avoid it.)

I'm no threading expert but 1000 - if those indeed are real threads (like, preemptive OS threads) - is wayyyy too many.

My understanding is that there's no benefit to creating more than the number of hyperthreads your CPU supports (minus your main thread?). So you'd want a work queue of all the available work/images, and then some reasonable number of threads (4-12 depending on core count / hyperthreading) taking work as they need it from that queue.

You should probably be looking at std.parallelism (TaskPool etc) for this. Perhaps somebody can provide a more detailed how-to ...

On Wednesday, 3 April 2019 at 04:09:45 UTC, DanielG wrote:
> My understanding is that there's no benefit to creating more than the number of hyperthreads your CPU

Yes.

I'm just searching for a higher abstraction to manage this easily in D. I've already did it years ago on Delphi. (-> Manage N worker threads, and only put light stress on the thread which is running on the same core as the main thread.) Later it was easier to make it in OpenMP, and now I'm just looking for an even prettier way in DLang.

Also it's good to know what happens inside. I've checked Resource Monitor, and as you said, the thread count reached above 1000 in the first 3 seconds of running the program.
But after that something unusual happens: While my overall CPU usage was still high, the number of threads dropped down to 15, and the remaining 20 seconds of processing was don like this.

But still, while all the processing, the main thread got no cpu% at all, it was able to render only the very first frame, the one that displayed the empty 'loading in progress' images. After 23 sec I was able to see all the 1000 pictures.

>You should probably be looking at std.parallelism (TaskPool etc) for this.

Thank you for mentioning, this std.parallelism.Task.executeOnNewThread will be the next test. Is that behaving the same way like 'spawn'? From its name, I don't think so, but I'll find out.

On Wednesday, 3 April 2019 at 02:59:51 UTC, realhet wrote:
> Hi,
>
> I'm writing a picture browser program and I'd like to ask for a bit of assistance.
>
> [...]

Hello,

maybe the following links can be useful:

http://ddili.org/ders/d.en/parallelism.html
http://ddili.org/ders/d.en/fibers.html