On 3/19/2015 9:59 AM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= <ola.fosheim.grostad+dlang@gmail.com>" wrote:
> On Thursday, 19 March 2015 at 00:42:51 UTC, weaselcat wrote:
>> On Wednesday, 18 March 2015 at 12:59:17 UTC, bearophile wrote:
>>> High level constructs in D are often slower than low-level code, so in some
>>> cases you don't want to use them.
>>
>> I actually found that LDC does an _amazing_ job of optimizing high level
>> constructs and converting "low level" code to higher level functional code
>> resulted in minor speedups in a lot of cases.
>>
>>
>> (Other performance benefits include the algorithm primitives being extensively
>> optimized in phobos.)
>
> If the code/compiler generates suboptimal code in the first place then
> improvements can be somewhat random. But if you write code with good cache
> locality, filling the pipeline properly then there is no alternative to going
> low level.
>
> Btw, take a look at this:
> http://stackoverflow.com/questions/28922323/improving-line-wise-i-o-operations-in-d
>
> That's really bad marketing...

Sigh. The Python version:
-----------
import sys

if __name__ == "__main__":
    if (len(sys.argv) < 2):
        sys.exit()
    infile = open(sys.argv[1])
    linect = 0
    for line in infile:
        linect += 1
    print "There are %d lines" % linect
----------
does not allocate memory. The splitLines() version:
----------
import std.stdio;
import std.string;
import std.file;

int main(string[] args)
{
  if (args.length < 2) {
    return 1;
  }
  auto c = cast(string) read(args[1]);
  auto l = splitLines(c);
  writeln("There are ", l.length, " lines.");
  return 0;
}
---------
allocates memory for all the lines and an array of them. No wonder it's slow! The allocations are slow, and filling them all when they are cold-cached - AWFUL! (It also uselessly and maddeningly auto-decodes, a misfeature of Phobos if there ever was one.)

http://dlang.org/phobos/std_string.html#.splitLines

On 3/19/2015 10:44 AM, Joakim wrote:
> One underused resource seems to be all the examples bearophile has put on
> Rosetta Code:
>
> http://rosettacode.org/wiki/Category:D
>
> If he, Adam, or some other proficient D user were to do a weekly series breaking
> down each of those 733 examples one at a time- what idioms were used, why
> certain variations were more efficient- that could go a long way to introduce
> the language and its idioms to beginners.  It would provide enough examples for
> 14 years of such a weekly series, by which time D3 will be getting started!

I didn't know about rosettacode. Thanks! It also might be a great resource for better examples to use in the Phobos documentation.

On Friday, 20 March 2015 at 05:17:11 UTC, Walter Bright wrote:
> On 3/19/2015 9:59 AM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= <ola.fosheim.grostad+dlang@gmail.com>" wrote:
>> On Thursday, 19 March 2015 at 00:42:51 UTC, weaselcat wrote:
>>> On Wednesday, 18 March 2015 at 12:59:17 UTC, bearophile wrote:
>>>> High level constructs in D are often slower than low-level code, so in some
>>>> cases you don't want to use them.
>>>
>>> I actually found that LDC does an _amazing_ job of optimizing high level
>>> constructs and converting "low level" code to higher level functional code
>>> resulted in minor speedups in a lot of cases.
>>>
>>>
>>> (Other performance benefits include the algorithm primitives being extensively
>>> optimized in phobos.)
>>
>> If the code/compiler generates suboptimal code in the first place then
>> improvements can be somewhat random. But if you write code with good cache
>> locality, filling the pipeline properly then there is no alternative to going
>> low level.
>>
>> Btw, take a look at this:
>> http://stackoverflow.com/questions/28922323/improving-line-wise-i-o-operations-in-d
>>
>> That's really bad marketing...
>
> Sigh. The Python version:
> -----------
> import sys
>
> if __name__ == "__main__":
>     if (len(sys.argv) < 2):
>         sys.exit()
>     infile = open(sys.argv[1])
>     linect = 0
>     for line in infile:
>         linect += 1
>     print "There are %d lines" % linect
> ----------
> does not allocate memory. The splitLines() version:
> [...] cutted

Of course it does allocate memory.

Python's "for...in" makes use of iterators and generators, already there you have some allocations going around.

Not only that, there is one string being allocated for each line in the file being read, even if it isn't used.

Not counting the whole minor allocations going on.



--
Paulo

> One underused resource seems to be all the examples bearophile has put on Rosetta Code:
>
> http://rosettacode.org/wiki/Category:D

Indeed, I found myself frequently looking though these examples. Thank you bearophile!

> If he, Adam, or some other proficient D user were to do a weekly series breaking down each of those 733 examples one at a time- what idioms were used, why certain variations were more efficient- that could go a long way to introduce the language and its idioms to beginners.

+1000

On Friday, 20 March 2015 at 00:28:24 UTC, Almighty Bob wrote:
> Maybe I'm a bit long in the tooth but for something like cvsReader I want an array of records, there's isnt anything else that would ever make any sense.

Just use std.array.array on it. You get your array, people who see the benefit of alternative approaches get what they want.

On 3/20/2015 12:42 AM, Paulo Pinto wrote:
> On Friday, 20 March 2015 at 05:17:11 UTC, Walter Bright wrote:
>> Sigh. The Python version:
>> -----------
>> import sys
>>
>> if __name__ == "__main__":
>>     if (len(sys.argv) < 2):
>>         sys.exit()
>>     infile = open(sys.argv[1])
>>     linect = 0
>>     for line in infile:
>>         linect += 1
>>     print "There are %d lines" % linect
>> ----------
>> does not allocate memory. The splitLines() version:
>> [...] cutted
>
> Of course it does allocate memory.
>
> Python's "for...in" makes use of iterators and generators, already there you
> have some allocations going around.
>
> Not only that, there is one string being allocated for each line in the file
> being read, even if it isn't used.

Since 'line' is never referred to again after constructed, even a simple optimizer could elide it.

It would be easy to test - accumulate the lines in an array, and check the times.

On Thursday, 19 March 2015 at 17:40:46 UTC, weaselcat wrote:
> On Thursday, 19 March 2015 at 16:59:36 UTC, Ola Fosheim Grøstad

>> If the code/compiler generates suboptimal code in the first place then improvements can be somewhat random. But if you write code with good cache locality, filling the pipeline properly then  there is no alternative to going low level.
>>
>> Btw, take a look at this:
>> http://stackoverflow.com/questions/28922323/improving-line-wise-i-o-operations-in-d
>>
>> That's really bad marketing...

[…]

> ldc -O5 -inline -release -boundscheck=off wc.d
> time ./wc enwiki-latest-pages-articles1.xml-p000000010p000010000
> There are 1245473 lines.
> ./wc enwiki-latest-pages-articles1.xml-p000000010p000010000  0.04s user 0.08s system 98% cpu 0.125 total
>
>
> ahem
>>> I actually found that LDC does an _amazing_ job of optimizing high level constructs and converting "low level" code to

I think you missed my point here. A simple loop like this should result in similar execution times for all AoT languages with decent library support and optimisation because it ought to be bandwidth limited, not CPU limited. The issue here is that when decent programmers end up asking for help on stackoverflow it exposes underlying library/compiler/language issues that needs addressing on way or another.

When it comes to more complicated algorithms then it usually impossible for a high level construct to match a low level one for several reasons, a major one being that you need to tailor the algorithm to OS/caches/ALU/CPU. Pure high level code cannot easily be reduced to high quality iterative algorithms that match the hardware.

High level constructs may be  cleaner if done right, and sometimes saves programmer time, but it will never be as fast on the standard CPU architectures we have today. The hardware favours carefully planned iterative, imperative approaches. That was true before SIMD and caching, and it is even more true now.

Try to do some signal processing and you'll quickly learn that high level code is out of the question if you want high performance.

On Friday, 20 March 2015 at 10:50:44 UTC, Walter Bright wrote:
> On 3/20/2015 12:42 AM, Paulo Pinto wrote:
>> On Friday, 20 March 2015 at 05:17:11 UTC, Walter Bright wrote:
>>> Sigh. The Python version:
>>> -----------
>>> import sys
>>>
>>> if __name__ == "__main__":
>>>    if (len(sys.argv) < 2):
>>>        sys.exit()
>>>    infile = open(sys.argv[1])
>>>    linect = 0
>>>    for line in infile:
>>>        linect += 1
>>>    print "There are %d lines" % linect
>>> ----------
>>> does not allocate memory. The splitLines() version:
>>> [...] cutted
>>
>> Of course it does allocate memory.
>>
>> Python's "for...in" makes use of iterators and generators, already there you
>> have some allocations going around.
>>
>> Not only that, there is one string being allocated for each line in the file
>> being read, even if it isn't used.
>
> Since 'line' is never referred to again after constructed, even a simple optimizer could elide it.
>
> It would be easy to test - accumulate the lines in an array, and check the times.

Which the default Python implementation doesn't have, hence my comment.

Also even if it did have one, it cannot elide it as it cannot guarantee the semantics of the generators/iterators side effects will be kept.

--
Paulo

On 3/19/15 10:22 PM, Walter Bright wrote:
> On 3/19/2015 10:44 AM, Joakim wrote:
>> One underused resource seems to be all the examples bearophile has put on
>> Rosetta Code:
>>
>> http://rosettacode.org/wiki/Category:D
>>
>> If he, Adam, or some other proficient D user were to do a weekly
>> series breaking
>> down each of those 733 examples one at a time- what idioms were used, why
>> certain variations were more efficient- that could go a long way to
>> introduce
>> the language and its idioms to beginners.  It would provide enough
>> examples for
>> 14 years of such a weekly series, by which time D3 will be getting
>> started!
>
> I didn't know about rosettacode. Thanks! It also might be a great
> resource for better examples to use in the Phobos documentation.

Someone who knows about copyright/licensing would probably need to check that it's okay if we plan to use them verbatim. If we can't then it might be worth linking to the above page from somewhere on dlang.org.

On Friday, 20 March 2015 at 15:12:44 UTC, David Gileadi wrote:
> On 3/19/15 10:22 PM, Walter Bright wrote:
>> On 3/19/2015 10:44 AM, Joakim wrote:
>>> One underused resource seems to be all the examples bearophile has put on
>>> Rosetta Code:
>>>
>>> http://rosettacode.org/wiki/Category:D
>>>
>>> If he, Adam, or some other proficient D user were to do a weekly
>>> series breaking
>>> down each of those 733 examples one at a time- what idioms were used, why
>>> certain variations were more efficient- that could go a long way to
>>> introduce
>>> the language and its idioms to beginners.  It would provide enough
>>> examples for
>>> 14 years of such a weekly series, by which time D3 will be getting
>>> started!
>>
>> I didn't know about rosettacode. Thanks! It also might be a great
>> resource for better examples to use in the Phobos documentation.
>
> Someone who knows about copyright/licensing would probably need to check that it's okay if we plan to use them verbatim. If we can't then it might be worth linking to the above page from somewhere on dlang.org.


All of the content on rosettacode appears to be licensed under GNU FDL, I believe it would just have to be released under the GNU FDL or a similar copyleft license that fulfills the GNU FDL.