View mode: basic / threaded / horizontal-split · Log in · Help
July 14, 2010
Getting # Physical CPUs
When I do parallel programming in D, it's nice to have a sane default for the
number of threads.  The ideal number is the number of cores that can see the
current address space.  core.cpuid provides coresPerCPU, but it doesn't appear
to provide nPhysialCPUs or something similar, which would allow the total
number of physical CPUs in the system to be determined.  Is there any easy way
to determine this in D2?
July 14, 2010
Re: Getting # Physical CPUs
dsimcha wrote:
> When I do parallel programming in D, it's nice to have a sane default for the
> number of threads.  The ideal number is the number of cores that can see the
> current address space.  core.cpuid provides coresPerCPU, but it doesn't appear
> to provide nPhysialCPUs or something similar, which would allow the total
> number of physical CPUs in the system to be determined.  Is there any easy way
> to determine this in D2?

coresPerCPU is a misnomer. It's actually the total number of cores. Will 
fix.
July 14, 2010
Re: Getting # Physical CPUs
== Quote from Don (nospam@nospam.com)'s article
> dsimcha wrote:
> > When I do parallel programming in D, it's nice to have a sane default for the
> > number of threads.  The ideal number is the number of cores that can see the
> > current address space.  core.cpuid provides coresPerCPU, but it doesn't appear
> > to provide nPhysialCPUs or something similar, which would allow the total
> > number of physical CPUs in the system to be determined.  Is there any easy way
> > to determine this in D2?
> coresPerCPU is a misnomer. It's actually the total number of cores. Will
> fix.

Ok, then maybe this is a bug and belongs in bugzilla.  Here's a little  test
program I wrote:

import core.cpuid, std.stdio;

void main() {
   writeln("Cores:  ", coresPerCPU, "  Threads:  ", threadsPerCPU);
}

This prints the following on the machine in question:

Cores:  1  Threads:  2

Here's a cat/proc/cpuinfo on the same machine:

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 15
model           : 4
model name      :                Intel(R) Xeon(TM) MP CPU 3.66GHz
stepping        : 1
cpu MHz         : 3657.825
cache size      : 1024 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor
ds_cpl est tm2 cid cx16 xtpr
bogomips        : 7321.79
clflush size    : 64
cache_alignment : 128
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 15
model           : 4
model name      :                Intel(R) Xeon(TM) MP CPU 3.66GHz
stepping        : 1
cpu MHz         : 3657.825
cache size      : 1024 KB
physical id     : 4
siblings        : 2
core id         : 4
cpu cores       : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor
ds_cpl est tm2 cid cx16 xtpr
bogomips        : 7314.82
clflush size    : 64
cache_alignment : 128
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 15
model           : 4
model name      :                Intel(R) Xeon(TM) MP CPU 3.66GHz
stepping        : 1
cpu MHz         : 3657.825
cache size      : 1024 KB
physical id     : 3
siblings        : 2
core id         : 3
cpu cores       : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor
ds_cpl est tm2 cid cx16 xtpr
bogomips        : 7315.15
clflush size    : 64
cache_alignment : 128
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 15
model           : 4
model name      :                Intel(R) Xeon(TM) MP CPU 3.66GHz
stepping        : 1
cpu MHz         : 3657.825
cache size      : 1024 KB
physical id     : 7
siblings        : 2
core id         : 7
cpu cores       : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor
ds_cpl est tm2 cid cx16 xtpr
bogomips        : 7315.01
clflush size    : 64
cache_alignment : 128
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 4
vendor_id       : GenuineIntel
cpu family      : 15
model           : 4
model name      :                Intel(R) Xeon(TM) MP CPU 3.66GHz
stepping        : 1
cpu MHz         : 3657.825
cache size      : 1024 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor
ds_cpl est tm2 cid cx16 xtpr
bogomips        : 7315.01
clflush size    : 64
cache_alignment : 128
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 5
vendor_id       : GenuineIntel
cpu family      : 15
model           : 4
model name      :                Intel(R) Xeon(TM) MP CPU 3.66GHz
stepping        : 1
cpu MHz         : 3657.825
cache size      : 1024 KB
physical id     : 4
siblings        : 2
core id         : 4
cpu cores       : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor
ds_cpl est tm2 cid cx16 xtpr
bogomips        : 7315.07
clflush size    : 64
cache_alignment : 128
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 6
vendor_id       : GenuineIntel
cpu family      : 15
model           : 4
model name      :                Intel(R) Xeon(TM) MP CPU 3.66GHz
stepping        : 1
cpu MHz         : 3657.825
cache size      : 1024 KB
physical id     : 3
siblings        : 2
core id         : 3
cpu cores       : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor
ds_cpl est tm2 cid cx16 xtpr
bogomips        : 7315.13
clflush size    : 64
cache_alignment : 128
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 7
vendor_id       : GenuineIntel
cpu family      : 15
model           : 4
model name      :                Intel(R) Xeon(TM) MP CPU 3.66GHz
stepping        : 1
cpu MHz         : 3657.825
cache size      : 1024 KB
physical id     : 7
siblings        : 2
core id         : 7
cpu cores       : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor
ds_cpl est tm2 cid cx16 xtpr
bogomips        : 7315.03
clflush size    : 64
cache_alignment : 128
address sizes   : 40 bits physical, 48 bits virtual
power management:
July 14, 2010
Re: Getting # Physical CPUs
dsimcha Wrote:

> When I do parallel programming in D, it's nice to have a sane default for the
> number of threads.  The ideal number is the number of cores that can see the
> current address space.  core.cpuid provides coresPerCPU, but it doesn't appear
> to provide nPhysialCPUs or something similar, which would allow the total
> number of physical CPUs in the system to be determined.  Is there any easy way
> to determine this in D2?

dsimcha,

This is a relatively difficult problem in general to do portably due to hardware differences, topology differences, changes to hardware, OS variations.  Even the pthreads library doesn't reliably implement it in a portable manner.

I came to the conclusion that the people most motivated to keep up to date on a portable CPU core topology are the national supercomputing labs.  INRIA and various US labs came up with "Portable Hardware Locality" library.  It gives you *everything* you need to discover the number of CPU sockets, memory architecture, number of cores per socket, control cpu affinity etc. 

The HWLoc C libraries are written by the open-mpi project here: http://www.open-mpi.org/projects/hwloc/

I created an interface to their C library called HWLoc.d and I haven't published it anywhere.  I just needed it for my multi-core Dendrite system.  Give me an email you want it sent to and it's yours.  I could post it here, but it would be a lot of noise in the discussion.  I'll probably post it on dsource later.

My license on all of my Dendrite code is zlib (which means you can do almost anything except claim that you wrote the original) and the open-mpi C libraries are new BSD, so have at it.

You'll obviously have to download and install the hwloc c libraries also.

Here's some output from "hwhello.d" on a 2 socket system:
*** Objects at level 0
CPU Set: 00000003
Index 0: System(1002MB)
*** Objects at level 1
CPU Set: 00000001
Index 0: Socket#0
CPU Set: 00000002
Index 1: Socket#2
*** Objects at level 2
CPU Set: 00000001
Index 0: L2(2048KB)
CPU Set: 00000002
Index 1: L2(2048KB)
*** Objects at level 3
CPU Set: 00000001
Index 0: L1(16KB)
CPU Set: 00000002
Index 1: L1(16KB)
*** Objects at level 4
CPU Set: 00000001
Index 0: Core#0
CPU Set: 00000002
Index 1: Core#0
*** Objects at level 5
CPU Set: 00000001
Index 0: P#0
CPU Set: 00000002
Index 1: P#1
*** Printing overall tree
System(1002MB)
 Socket#0
   L2(2048KB)
     L1(16KB)
       Core#0
         P#0
 Socket#2
   L2(2048KB)
     L1(16KB)
       Core#0
         P#1
*** 2 socket(s)
depth is now: 4
number objects at this depth: 2
ob 0: 00000001
ob 0: 00000002
Last CPU Set: 00000002
Singleified CPU Set: 00000002
burning CPU

... and here's the code snippet from hwloc.d that generates the last part of that output:
   /* Get last level. */
   obj = hwloc_get_obj_by_depth(topology, depth,
                  hwloc_get_nbobjs_by_depth(topology, depth) - 1);
   if (obj) {
       /* Get a copy of its cpuset that we may modify. */
       cpuset = hwloc_cpuset_dup(obj.cpuset);

       // convert it to a string
       nchars = hwloc_cpuset_snprintf(string.ptr, string.sizeof, cpuset);
       Stdout("Last CPU Set: ")(string[0..nchars]).newline;

       /* Get only one logical processor (in case the core is
          SMT/hyperthreaded). */
       hwloc_cpuset_singlify(cpuset);

       nchars = hwloc_cpuset_snprintf(string.ptr, string.sizeof, cpuset);
       Stdout("Singleified CPU Set: ")(string[0..nchars]).newline;

       /* And try to bind ourself there. */
       if (hwloc_set_cpubind(topology, cpuset, 0)) {
           char* str;
           nchars = hwloc_cpuset_asprintf(&str, obj.cpuset);
           Stdout.formatln("Couldn't bind to cpuset {}", str[0..nchars]);
           free(str);
       }

       Stdout("burning CPU").newline;
       /* do something we can see in htop */
       for (j=0; j<100000000; j++) f+=f/2;


       /* Free our cpuset copy */
       hwloc_cpuset_free(cpuset);
   }

.... finally here's the output from the 'lstopo' utility in the hwloc distro that displays the topology visually (mine is a 4 socket, hexacore NUMA machine.  Use a fixed font to view it).
&#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488;
&#9474; System(4096MB)                                                 &#9474;
&#9474;                                                                &#9474;
&#9474; &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488; &#9474;
&#9474; &#9474; &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488; &#9474; &#9474;
&#9474; &#9474; &#9474; Node#0(1024MB)                                         &#9474; &#9474; &#9474;
&#9474; &#9474; &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496; &#9474; &#9474;
&#9474; &#9474;                                                            &#9474; &#9474;
&#9474; &#9474; &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488;  &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488;  &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488;  &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488;  &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488;  &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488; &#9474; &#9474;
&#9474; &#9474; &#9474; P#0  &#9474;  &#9474; P#1  &#9474;  &#9474; P#2  &#9474;  &#9474; P#3  &#9474;  &#9474; P#4  &#9474;  &#9474; P#5  &#9474; &#9474; &#9474;
&#9474; &#9474; &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496;  &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496;  &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496;  &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496;  &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496;  &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496; &#9474; &#9474;
&#9474; &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496; &#9474;
&#9474;                                                                &#9474;
&#9474; &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488; &#9474;
&#9474; &#9474; &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488; &#9474; &#9474;
&#9474; &#9474; &#9474; Node#1(1024MB)                                         &#9474; &#9474; &#9474;
&#9474; &#9474; &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496; &#9474; &#9474;
&#9474; &#9474;                                                            &#9474; &#9474;
&#9474; &#9474; &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488;  &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488;  &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488;  &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488;  &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488;  &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488; &#9474; &#9474;
&#9474; &#9474; &#9474; P#6  &#9474;  &#9474; P#7  &#9474;  &#9474; P#8  &#9474;  &#9474; P#9  &#9474;  &#9474; P#10 &#9474;  &#9474; P#11 &#9474; &#9474; &#9474;
&#9474; &#9474; &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496;  &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496;  &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496;  &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496;  &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496;  &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496; &#9474; &#9474;
&#9474; &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496; &#9474;
&#9474;                                                                &#9474;
&#9474; &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488; &#9474;
&#9474; &#9474; &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488; &#9474; &#9474;
&#9474; &#9474; &#9474; Node#2(1024MB)                                         &#9474; &#9474; &#9474;
&#9474; &#9474; &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496; &#9474; &#9474;
&#9474; &#9474;                                                            &#9474; &#9474;
&#9474; &#9474; &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488;  &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488;  &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488;  &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488;  &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488;  &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488; &#9474; &#9474;
&#9474; &#9474; &#9474; P#12 &#9474;  &#9474; P#13 &#9474;  &#9474; P#14 &#9474;  &#9474; P#15 &#9474;  &#9474; P#16 &#9474;  &#9474; P#17 &#9474; &#9474; &#9474;
&#9474; &#9474; &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496;  &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496;  &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496;  &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496;  &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496;  &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496; &#9474; &#9474;
&#9474; &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496; &#9474;
&#9474;                                                                &#9474;
&#9474; &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488; &#9474;
&#9474; &#9474; &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488; &#9474; &#9474;
&#9474; &#9474; &#9474; Node#3(1024MB)                                         &#9474; &#9474; &#9474;
&#9474; &#9474; &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496; &#9474; &#9474;
&#9474; &#9474;                                                            &#9474; &#9474;
&#9474; &#9474; &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488;  &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488;  &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488;  &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488;  &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488;  &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488; &#9474; &#9474;
&#9474; &#9474; &#9474; P#18 &#9474;  &#9474; P#19 &#9474;  &#9474; P#20 &#9474;  &#9474; P#21 &#9474;  &#9474; P#22 &#9474;  &#9474; P#23 &#9474; &#9474; &#9474;
&#9474; &#9474; &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496;  &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496;  &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496;  &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496;  &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496;  &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496; &#9474; &#9474;
&#9474; &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496; &#9474;
&#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496;
draco% 

Have fun and let me know if you need anything...


eris
July 14, 2010
Re: Getting # Physical CPUs
== Quote from eris (jvburnes@gmail.com)'s article
> This is a relatively difficult problem in general to do portably due to hardware
differences, topology differences, changes to hardware, OS variations.  Even the
pthreads library doesn't reliably implement it in a portable manner.
> I came to the conclusion that the people most motivated to keep up to date on a
portable CPU core topology are the national supercomputing labs.  INRIA and
various US labs came up with "Portable Hardware Locality" library.  It gives you
*everything* you need to discover the number of CPU sockets, memory architecture,
number of cores per socket, control cpu affinity etc.
> The HWLoc C libraries are written by the open-mpi project here:
http://www.open-mpi.org/projects/hwloc/

I appreciate the help, but honestly, if detecting this properly requires adding
dependencies to my projects, I'm happier with the simple workaround of having a
manual command line switch to specify the number of CPUs.  The projects in
question are internal research projects, not things that are going to be released
on the computer-illiterate masses.  It would be nice to not have to manually
specify such a parameter on every run, but not nice enough to be worth introducing
a dependency.
July 14, 2010
Re: Getting # Physical CPUs
dsimcha wrote:
> == Quote from Don (nospam@nospam.com)'s article
>> dsimcha wrote:
>>> When I do parallel programming in D, it's nice to have a sane default for the
>>> number of threads.  The ideal number is the number of cores that can see the
>>> current address space.  core.cpuid provides coresPerCPU, but it doesn't appear
>>> to provide nPhysialCPUs or something similar, which would allow the total
>>> number of physical CPUs in the system to be determined.  Is there any easy way
>>> to determine this in D2?
>> coresPerCPU is a misnomer. It's actually the total number of cores. Will
>> fix.
> 
> Ok, then maybe this is a bug and belongs in bugzilla.  Here's a little  test
> program I wrote:

[snip]
Thanks, that's definitely a bug. The code in core.cpuid has not been 
tested on the most recent CPUs (Intel added a totally new method) and 
their documentation is quite convoluted. It's hard to get it right 
without an actual machine.
July 14, 2010
Re: Getting # Physical CPUs
== Quote from Don (nospam@nospam.com)'s article
> [snip]
> Thanks, that's definitely a bug. The code in core.cpuid has not been
> tested on the most recent CPUs (Intel added a totally new method) and
> their documentation is quite convoluted. It's hard to get it right
> without an actual machine.

Bug 4462.  http://d.puremagic.com/issues/show_bug.cgi?id=4462
July 15, 2010
Re: Getting # Physical CPUs
dsimcha wrote:
> == Quote from Don (nospam@nospam.com)'s article
>> [snip]
>> Thanks, that's definitely a bug. The code in core.cpuid has not been
>> tested on the most recent CPUs (Intel added a totally new method) and
>> their documentation is quite convoluted. It's hard to get it right
>> without an actual machine.
> 
> Bug 4462.  http://d.puremagic.com/issues/show_bug.cgi?id=4462

Please check if the latest druntime commit fixes this.
July 15, 2010
Re: Getting # Physical CPUs
== Quote from Don (nospam@nospam.com)'s article
> dsimcha wrote:
> > == Quote from Don (nospam@nospam.com)'s article
> >> [snip]
> >> Thanks, that's definitely a bug. The code in core.cpuid has not been
> >> tested on the most recent CPUs (Intel added a totally new method) and
> >> their documentation is quite convoluted. It's hard to get it right
> >> without an actual machine.
> >
> > Bug 4462.  http://d.puremagic.com/issues/show_bug.cgi?id=4462
> Please check if the latest druntime commit fixes this.

Thanks.  Unfortunately I can't test this because the Linux build script for Phobos
is broken on my machine in some inscrutable way.  Frankly, my success rate at
building stuff from other people's make files is well under 50%.  Make is just a
horrible technology that needs to die a horrible death.  We should be eating our
own dogfood and using rdmd for build scripts.

Here's the error message I'm getting.  I know basically nothing about make except
that it's a build system and that it almost never works, so I can't even begin to
debug this.  Here's the error message I've been getting, on a freshly unpacked
2.047 directory on some ancient  Linux distro that my sys admin insists on using:

$ make -flinux.mak
make --no-print-directory -f  OS=posix BUILD=release
make[1]: OS=posix: No such file or directory
make[1]: *** No rule to make target `OS=posix'.  Stop.
make: *** [release] Error 2

And here's the error I get when I try on a different machine w/ a more modern
distro (this one is probably due to lack of 64 bit libs):

$ make -flinux.mak
make --no-print-directory -f linux.mak OS=posix BUILD=release
cc -c  -m32 -O3 etc/c/zlib/adler32.c -ogenerated/posix/release/etc/c/zlib/adler32.o
cc -c  -m32 -O3 etc/c/zlib/compress.c -ogenerated/posix/release/etc/c/zlib/compress.o
cc -c  -m32 -O3 etc/c/zlib/crc32.c -ogenerated/posix/release/etc/c/zlib/crc32.o
In file included from /usr/include/features.h:378,
                from /usr/include/string.h:26,
                from etc/c/zlib/zutil.h:23,
                from etc/c/zlib/crc32.c:29:
/usr/include/gnu/stubs.h:7:27: error: gnu/stubs-32.h: No such file or directory
make[1]: *** [generated/posix/release/etc/c/zlib/crc32.o] Error 1
make: *** [release] Error 2
July 15, 2010
Re: Getting # Physical CPUs
dsimcha wrote:
> == Quote from Don (nospam@nospam.com)'s article
>> dsimcha wrote:
>>> == Quote from Don (nospam@nospam.com)'s article
>>>> [snip]
>>>> Thanks, that's definitely a bug. The code in core.cpuid has not been
>>>> tested on the most recent CPUs (Intel added a totally new method) and
>>>> their documentation is quite convoluted. It's hard to get it right
>>>> without an actual machine.
>>> Bug 4462.  http://d.puremagic.com/issues/show_bug.cgi?id=4462
>> Please check if the latest druntime commit fixes this.
> 
> Thanks.  Unfortunately I can't test this because the Linux build script for Phobos
> is broken on my machine in some inscrutable way.  Frankly, my success rate at
> building stuff from other people's make files is well under 50%.  Make is just a
> horrible technology that needs to die a horrible death.  We should be eating our
> own dogfood and using rdmd for build scripts.

I agree.
In this case, core.cpuid is completely stand-alone. So you could just 
copy it into another directory and change the module statement.
« First   ‹ Prev
1 2 3
Top | Discussion index | About this forum | D home