View mode: basic / threaded / horizontal-split · Log in · Help
July 15, 2010
Re: Getting # Physical CPUs
== Quote from Don (nospam@nospam.com)'s article
> dsimcha wrote:
> > == Quote from Don (nospam@nospam.com)'s article
> >> dsimcha wrote:
> >>> == Quote from Don (nospam@nospam.com)'s article
> >>>> [snip]
> >>>> Thanks, that's definitely a bug. The code in core.cpuid has not been
> >>>> tested on the most recent CPUs (Intel added a totally new method) and
> >>>> their documentation is quite convoluted. It's hard to get it right
> >>>> without an actual machine.
> >>> Bug 4462.  http://d.puremagic.com/issues/show_bug.cgi?id=4462
> >> Please check if the latest druntime commit fixes this.
> >
> > Thanks.  Unfortunately I can't test this because the Linux build script for Phobos
> > is broken on my machine in some inscrutable way.  Frankly, my success rate at
> > building stuff from other people's make files is well under 50%.  Make is just a
> > horrible technology that needs to die a horrible death.  We should be eating our
> > own dogfood and using rdmd for build scripts.
> I agree.
> In this case, core.cpuid is completely stand-alone. So you could just
> copy it into another directory and change the module statement.

Great idea.  Unfortunately that still doesn't fix it.  I get different wrong
information on some machines, but it's still wrong, and the specific example I
posted to Bugzilla hasn't changed at all.
July 15, 2010
Re: Getting # Physical CPUs
As a side note, why is there both a std.cpuid and a core.cpuid?

Does std use core? In that case, why not import std.cpuid?
July 15, 2010
Re: Getting # Physical CPUs
Philippe Sigaud wrote:
> As a side note, why is there both a std.cpuid and a core.cpuid?
> 
> Does std use core? 

No.

In that case, why not import std.cpuid?
> 
> 
std.cpuid is deprecated.
July 15, 2010
Re: Getting # Physical CPUs
dsimcha wrote:
> Here's the error message I'm getting.  I know basically nothing about make except
> that it's a build system and that it almost never works, so I can't even begin to
> debug this.  Here's the error message I've been getting, on a freshly unpacked
> 2.047 directory on some ancient  Linux distro that my sys admin insists on using:
> 
> $ make -flinux.mak
> make --no-print-directory -f  OS=posix BUILD=release
> make[1]: OS=posix: No such file or directory
> make[1]: *** No rule to make target `OS=posix'.  Stop.
> make: *** [release] Error 2

The "OS=posix" sets the macro OS to the value posix, it does not set the target. 
This has been a feature of make since at least the 1980's, earlier than Linux 
even existed. So I'm astonished you're seeing this error.


> And here's the error I get when I try on a different machine w/ a more modern
> distro (this one is probably due to lack of 64 bit libs):
> 
> $ make -flinux.mak
> make --no-print-directory -f linux.mak OS=posix BUILD=release
> cc -c  -m32 -O3 etc/c/zlib/adler32.c -ogenerated/posix/release/etc/c/zlib/adler32.o
> cc -c  -m32 -O3 etc/c/zlib/compress.c -ogenerated/posix/release/etc/c/zlib/compress.o
> cc -c  -m32 -O3 etc/c/zlib/crc32.c -ogenerated/posix/release/etc/c/zlib/crc32.o
> In file included from /usr/include/features.h:378,
>                  from /usr/include/string.h:26,
>                  from etc/c/zlib/zutil.h:23,
>                  from etc/c/zlib/crc32.c:29:
> /usr/include/gnu/stubs.h:7:27: error: gnu/stubs-32.h: No such file or directory
> make[1]: *** [generated/posix/release/etc/c/zlib/crc32.o] Error 1
> make: *** [release] Error 2

This is most likely because you have not gotten the 32 bit dev system installed 
on your 64 bit system (it usually is not installed by the default linux 
install). Which one do you have?

BTW, using a script rather than make wouldn't have helped you with the second issue.
July 15, 2010
Re: Getting # Physical CPUs
== Quote from Walter Bright (newshound2@digitalmars.com)'s article
> > And here's the error I get when I try on a different machine w/ a more modern
> > distro (this one is probably due to lack of 64 bit libs):
> >
> > $ make -flinux.mak
> > make --no-print-directory -f linux.mak OS=posix BUILD=release
> > cc -c  -m32 -O3 etc/c/zlib/adler32.c
-ogenerated/posix/release/etc/c/zlib/adler32.o
> > cc -c  -m32 -O3 etc/c/zlib/compress.c
-ogenerated/posix/release/etc/c/zlib/compress.o
> > cc -c  -m32 -O3 etc/c/zlib/crc32.c -ogenerated/posix/release/etc/c/zlib/crc32.o
> > In file included from /usr/include/features.h:378,
> >                  from /usr/include/string.h:26,
> >                  from etc/c/zlib/zutil.h:23,
> >                  from etc/c/zlib/crc32.c:29:
> > /usr/include/gnu/stubs.h:7:27: error: gnu/stubs-32.h: No such file or directory
> > make[1]: *** [generated/posix/release/etc/c/zlib/crc32.o] Error 1
> > make: *** [release] Error 2
> This is most likely because you have not gotten the 32 bit dev system installed
> on your 64 bit system (it usually is not installed by the default linux
> install). Which one do you have?
> BTW, using a script rather than make wouldn't have helped you with the second issue.

Yeah, in my original most I meant to say it was probably due to lack of 32-bit
libs, which I don't have root privileges to install on the machine in question.
It can see the same file systems via NFS as some machines that do have 32-bit libs
installed, so my usual kludge is to always use a machine w/ 32-bit libs for
building.  Even so, I only figured this out b/c I was previously aware of the
problem.  The point isn't that make failed here, it's that it failed with an
absolutely inscrutable error message.  I wouldn't have even been able to begin
guessing what was wrong if I didn't already know about the 32-bit lib issue.
July 15, 2010
Re: Getting # Physical CPUs
Walter Bright, el 15 de julio a las 11:40 me escribiste:
> dsimcha wrote:
> >Here's the error message I'm getting.  I know basically nothing about make except
> >that it's a build system and that it almost never works, so I can't even begin to
> >debug this.  Here's the error message I've been getting, on a freshly unpacked
> >2.047 directory on some ancient  Linux distro that my sys admin insists on using:
> >
> >$ make -flinux.mak
> >make --no-print-directory -f  OS=posix BUILD=release
> >make[1]: OS=posix: No such file or directory
> >make[1]: *** No rule to make target `OS=posix'.  Stop.
> >make: *** [release] Error 2
> 
> The "OS=posix" sets the macro OS to the value posix, it does not set
> the target. This has been a feature of make since at least the
> 1980's, earlier than Linux even existed. So I'm astonished you're
> seeing this error.

This is even standard POSIX:
http://www.opengroup.org/onlinepubs/009695399/utilities/make.html
(see the OPERANDS section)

-- 
Leandro Lucarella (AKA luca)                     http://llucax.com.ar/
----------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------
- Que hac├ęs, ratita?
- Espero un ratito...
July 15, 2010
Re: Getting # Physical CPUs
Walter Bright Wrote:

> > 
> > $ make -flinux.mak
> > make --no-print-directory -f  OS=posix BUILD=release
> > make[1]: OS=posix: No such file or directory
> > make[1]: *** No rule to make target `OS=posix'.  Stop.
> > make: *** [release] Error 2
> 
> The "OS=posix" sets the macro OS to the value posix, it does not set the target. 
> This has been a feature of make since at least the 1980's, earlier than Linux 
> even existed. So I'm astonished you're seeing this error.

Looks to me like a macro isn't being set. In the first output line it has the argument '-f  ' there are two spaces after requesting to look in a file. So the file name it sees is OS=posix
July 15, 2010
Re: Getting # Physical CPUs
On 07/14/2010 08:55 PM, dsimcha wrote:
> == Quote from eris (jvburnes@gmail.com)'s article
>> This is a relatively difficult problem in general to do portably due to hardware
> differences, topology differences, changes to hardware, OS variations.  Even the
> pthreads library doesn't reliably implement it in a portable manner.
>> I came to the conclusion that the people most motivated to keep up to date on a
> portable CPU core topology are the national supercomputing labs.  INRIA and
> various US labs came up with "Portable Hardware Locality" library.  It gives you
> *everything* you need to discover the number of CPU sockets, memory architecture,
> number of cores per socket, control cpu affinity etc.
>> The HWLoc C libraries are written by the open-mpi project here:
> http://www.open-mpi.org/projects/hwloc/
>
> I appreciate the help, but honestly, if detecting this properly requires adding
> dependencies to my projects, I'm happier with the simple workaround of having a
> manual command line switch to specify the number of CPUs.  The projects in
> question are internal research projects, not things that are going to be released
> on the computer-illiterate masses.  It would be nice to not have to manually
> specify such a parameter on every run, but not nice enough to be worth introducing
> a dependency.

I can't imagine how this would not be a required part of the core library.

For a language that claims to be thread savvy, knowing the number of 
cpus and the number of cores, is simply obligatory homework.

An extra point: the code that identifies them, should not ever assume 
that all cores are identical. Nor that they have identical access to 
machine resources.

The day that someone invents the 'unequal cores paradigm', where cores 
of dissimilar power are included in the same computer, should not expose 
us with our pants down.

(A case in point, at bootup, the Linux core already enumerates and 
evaluates each found core individually.)
July 16, 2010
Re: Getting # Physical CPUs
Georg Wrede wrote:
> On 07/14/2010 08:55 PM, dsimcha wrote:
>> == Quote from eris (jvburnes@gmail.com)'s article
>>> This is a relatively difficult problem in general to do portably due 
>>> to hardware
>> differences, topology differences, changes to hardware, OS 
>> variations.  Even the
>> pthreads library doesn't reliably implement it in a portable manner.
>>> I came to the conclusion that the people most motivated to keep up to 
>>> date on a
>> portable CPU core topology are the national supercomputing labs.  
>> INRIA and
>> various US labs came up with "Portable Hardware Locality" library.  It 
>> gives you
>> *everything* you need to discover the number of CPU sockets, memory 
>> architecture,
>> number of cores per socket, control cpu affinity etc.
>>> The HWLoc C libraries are written by the open-mpi project here:
>> http://www.open-mpi.org/projects/hwloc/
>>
>> I appreciate the help, but honestly, if detecting this properly 
>> requires adding
>> dependencies to my projects, I'm happier with the simple workaround of 
>> having a
>> manual command line switch to specify the number of CPUs.  The 
>> projects in
>> question are internal research projects, not things that are going to 
>> be released
>> on the computer-illiterate masses.  It would be nice to not have to 
>> manually
>> specify such a parameter on every run, but not nice enough to be worth 
>> introducing
>> a dependency.
> 
> I can't imagine how this would not be a required part of the core library.
> 
> For a language that claims to be thread savvy, knowing the number of 
> cpus and the number of cores, is simply obligatory homework.
> 
> An extra point: the code that identifies them, should not ever assume 
> that all cores are identical. Nor that they have identical access to 
> machine resources.

> The day that someone invents the 'unequal cores paradigm', where cores 
> of dissimilar power are included in the same computer, should not expose 
> us with our pants down.

It really depends on what the purpose is. If you want to determine the 
precise core topology, the available information is heavily 
OS-dependent. Note that there's potentially a large difference between 
the number of cores in the machine, versus the number of cores which the 
OS makes available to your app. Generally the second number is the one 
which matters.

> (A case in point, at bootup, the Linux core already enumerates and 
> evaluates each found core individually.)

Of course it does. It's trivial when you're an OS and have unrestricted 
access to the machine. An app is severely limited to what it can get 
from the OS.

Currently core.cpuid doesn't make any OS calls at all.
I think std.cpuid should be replaced with a new module std.sysinfo, 
which determines more features (such as available RAM).
July 16, 2010
Re: Getting # Physical CPUs
On Thu, 15 Jul 2010 14:40:57 -0400, Walter Bright  
<newshound2@digitalmars.com> wrote:

> dsimcha wrote:
>> Here's the error message I'm getting.  I know basically nothing about  
>> make except
>> that it's a build system and that it almost never works, so I can't  
>> even begin to
>> debug this.  Here's the error message I've been getting, on a freshly  
>> unpacked
>> 2.047 directory on some ancient  Linux distro that my sys admin insists  
>> on using:
>>  $ make -flinux.mak
>> make --no-print-directory -f  OS=posix BUILD=release
>> make[1]: OS=posix: No such file or directory
>> make[1]: *** No rule to make target `OS=posix'.  Stop.
>> make: *** [release] Error 2
>
> The "OS=posix" sets the macro OS to the value posix, it does not set the  
> target. This has been a feature of make since at least the 1980's,  
> earlier than Linux even existed. So I'm astonished you're seeing this  
> error.

The issue is the -f is passed a blank argument.  Probably in linux.mak,  
it's calling something like this:

make --no-print-directory -f $(SUB_MAKE_FILE) OS=$(OS) BUILD=release

or something like that, and SUB_MAKE_FILE isn't being set for some reason.

I think make is interpreting that OS=posix as a makefile to use.

-Steve
1 2 3
Top | Discussion index | About this forum | D home