In article <an8chi$2qp4$1@digitaldaemon.com>, Mark Evans says...
>Chris,
>
>If you focus attention on numerical routines like convolution then you can probably isolate cases where negative indices cause a real performance hit.  So what.  My experience teaches that coding correctly and quickly is vastly more important.  Performance issues always take a back seat.  I say to you about extreme performance just what you say to me about negative indexing: it's only helpful to a small minority who should be hand-coding anyway.
>
>Mark

One of the problems I have with "ease of programming" over performance is that if I have a language that provides performance, I can make inline functions to provide functionality.  But if my language forces the functionality on me at the expense of performance, I cannot get the performance back without resorting to a lower level language.

I have had the misfortune to do graphics programming and manipulation in VB. While the array checks in that language did make it much easier to debug my program, turning them off made a 200%-300% difference in runtime.  When the application is a special effects video editing tool that has to process hundreds of thousands of images, it is impossible to say "but it was so much easier to write the slow version" and get any kind of user sympathy.  (And for what it is worth, with the various array checks turned off, the compiled VB program did end up very close to as fast as a compiled C app would be -- both languages use the same back end compiler and optimizer, and the "pure" math and memory access I was using was indistinguishable between the languages)

Don't get me wrong -- I like functionality in languages.  I like having stuff built-in that makes my life easier.  That's why I seconded the vote for a "decoration" syntax that would mark certain array accesses as being wrapped (reflected?) or ringed, without imposing that penalty on unadorned arrays.  I also think that my alternate syntax, with wrap() and ring() functions built into the array system is not horrible.  If such functions are built in, the compiler knows what they do and can apply any special optimizations the implementor can conceive of inside those accesses.  It also provides the benefit of flagging those uses to the human who has to read and maintain the code.  As a long time C/C++/Fortan/VB/Ada/etc programmer, my first reaction to "array[-2]" is that I'm about to witness a memory corruption.  If, instead, I saw "array.wrap(-2)" or "array:wrap[-2]", I would know that something was up and would go find out what was happening.

So, in short, I like negative indices, I just prefer to have either:

- a special syntax -- "array:wrap[-2]"
- a builtin inline routine that looks like a member function of an array
"array.wrap(-2)"
- a standard inline routine in Phobos that provides the functionality
"wrap(array, -2)"

Any of these solutions will provide the functionality you want, with much less visual ugliness than seeing a ?: operator at the access point and (by being built in) without having five different names for it from five different authors.  It also preserves my ability to write fast code when I need to, without having to resort to "extern (C)" or inline assembly.

Mac

I believe that forcing *all* array indexing to be slower just so certain routines can be written a little more easily is not the way to go.

Make a different accessor for negative/positive indexing than normal positive indexing.

If you blatantly disregard performance issues up front you will likely find yourself up against a brick wall later trying to optimize algorithms to make up for a general overall slowness, akin to what you get using VB or Java or some interpreted language.  At some point they Just Can't Be Made To Run ANY Faster.

We don't want to put a speed limit on D.  It should be competitive performance-wise with C/C++ or it won't be used where they are used.  As a systems programming language, slow array accesses are entirely unacceptable.

As someone who deals with performance issues every day, performance issues don't always take a back seat.  In realtime apps (games etc) performance issues are often crucial.  You can't ignore them, and you can't say "well we'll go back and optimize it later" because your product will get canned if the publisher doesn't see good progress toward 60 fps.  In fact they often want it running fast well before it's time to ship.

Sure you shouldn't prematurely optimize, but you also shouldn't go slowing down the entire language on a whim either.  Not all apps need negative indexing, but all apps would suffer the performance loss if it were made the default.

Sean

"Mark Evans" <Mark_member@pathlink.com> wrote in message news:an8chi$2qp4$1@digitaldaemon.com...
> Chris,
>
> This whole area of array manipulation is one of the huge, gaping holes in
C/C++
> that any replacement language should strive to fix.  I categorically
disagree
> that array (and by extension, string) manipulation is some kind of obscure
> backwater.
>
> If you focus attention on numerical routines like convolution then you can probably isolate cases where negative indices cause a real performance
hit.  So
> what.  My experience teaches that coding correctly and quickly is vastly
more
> important.  Performance issues always take a back seat.  I say to you
about
> extreme performance just what you say to me about negative indexing: it's
only
> helpful to a small minority who should be hand-coding anyway.
>
> I've written literally dozens of convolution routines (1D and 2D) using C
and
> also using script languages.  The first thing I did was to construct a
"negative
> index" system to facilitate the convolution algorithms.  In some languages
that
> was not necessary because it was already built-in.  Guess which languages
I
> preferred?  I would have been in heaven if my C compiler had supported
negative
> indices.
>
> I've also used negative indexing extensively in Icon string processing and
it
> is a real blessing.  (Consider right-to-left languages, among other
things.)  I
> swear, one of these days I am going to port Icon to C, just so I can have
its
> string processing power in C.
>
> I am beginning to wonder how D will become "a better C++" if nobody wants non-C++ features in the language!
>
> Mark

> What if this "symmetric" indexing is done through a separate property? The [] operator remains the same, and you invent a new property such as:
>
> char a = str.at(i);

I second this. Just three more characters than a normal array access.

> This new property would do symmetric indexing, and ... well ... upper
bounds
> checking?

I kinda thought the "ring" property was neat too, but I don't know how that would fit into this scheme (would "at" do that kind of access? probably not...). I do think that this is great; C++'s vector does this same thing, its [] operator is direct access and its .at() function is exactly like this. It's a clear bonus for D because C++ would be using the same syntax, and Walter seems concerned about attracting & transitioning those users to D, so those users would be immediately comfortable with this syntax.

> I think this combination would result in maximum runtime performance.
> If we don't have the opportunity to build own dynamic array types (lack of
> [] operator overload),
> then the built-in array type should be feature-rich.
Since the basic array type cannot be overloaded (and that's kinda good in a way for consistency, but bad for extending it), I think that is why everyone is concerned about the built in array type(s) being just right; it has to be "good enough" for 99.5% of situations, and I don't see a STL for D coming along (for that 0.5% of code) unless D gets really popular.

Now a question: according to the "Memory Model" document, dynamic arrays have their first dword pointing to the start of the data, so (for example) an array located in memory at 0xF0000000 would have as its first four bytes 0xF0000008. When doing an array access why can't it be *assumed* that the array data starts eight bytes into the array. So say you coded "aNumber = myIntegers[5];", then the value at (pointer to array) + (element number * type scalar + 8 bytes) would be accessed. Then the unused first four bytes could be used for other things. I know there's probably a good reason for the current setup, I just don't know what it is, or I could be missing something entirely.

float[4] myvec;
myvec.at[-1] = 3; myvec.at[-2] = 2; myvec.at[-3] = 1; myvec.at[-4] = 0;
printf("%f%f%f%f\n",myvec[0],myvec[1],myvec[2],myvec.at[3]);

prints 0123

I think it would be cool to be able to override a local [] operator for any container type (any class that has an [] operator itself) for any index type, implement it, and call it with normal array index syntax with values of the right types.

I know this isn't D but it's not C++ either:

int& float[]::operator[int x]  // reverses any float[] arrays in this scope.
{
    return super[super.size() - 1 - x];
}

void foo()
{
    float[2] x = {0,1};
    print x[0];   // writes 1
    print x[1];   // writes 0
}

Fun huh?  Reversing arrays is no good obviously but you could do a hash map or something.

With templates that could end up being really powerful.  A suite of data connector tools, when taken to extremes.  Everything from straight pipes to compressors to translators to reindexers to databases to RPC to whatever.

Just think what your compiler could do if it had access to a teensy compiler at runtime.  It could choose from several algorithms depending on how efficient they are at handling recent data.  It could choose between random access and searches and brute force linear traversal with varying sizes in mind (i.e. exploit SIMD to unroll loops).  Every once in a while it tries new algorithms in different spots in the program and sees which ones are handling the data flow most efficiently on the target platform.  .net could become this kind of backend.  I suppose gcc might could do this.  Alot of that could be done statically.

Sean

"Les Baker" <REMOVETHISlesbaker@innova.netREMOVETHIS> wrote in message news:anaoor$2ejr$1@digitaldaemon.com...
> > What if this "symmetric" indexing is done through a separate property? The [] operator remains the same, and you invent a new property such as:
> >
> > char a = str.at(i);
>
> I second this. Just three more characters than a normal array access.
>
> > This new property would do symmetric indexing, and ... well ... upper
> bounds
> > checking?
>
> I kinda thought the "ring" property was neat too, but I don't know how
that
> would fit into this scheme (would "at" do that kind of access? probably not...). I do think that this is great; C++'s vector does this same thing, its [] operator is direct access and its .at() function is exactly like this. It's a clear bonus for D because C++ would be using the same syntax, and Walter seems concerned about attracting & transitioning those users to D, so those users would be immediately comfortable with this syntax.
>
> > I think this combination would result in maximum runtime performance. If we don't have the opportunity to build own dynamic array types (lack
of
> > [] operator overload),
> > then the built-in array type should be feature-rich.
> Since the basic array type cannot be overloaded (and that's kinda good in
a
> way for consistency, but bad for extending it), I think that is why
everyone
> is concerned about the built in array type(s) being just right; it has to
be
> "good enough" for 99.5% of situations, and I don't see a STL for D coming along (for that 0.5% of code) unless D gets really popular.
>
> Now a question: according to the "Memory Model" document, dynamic arrays have their first dword pointing to the start of the data, so (for example) an array located in memory at 0xF0000000 would have as its first four
bytes
> 0xF0000008. When doing an array access why can't it be *assumed* that the array data starts eight bytes into the array. So say you coded "aNumber = myIntegers[5];", then the value at (pointer to array) + (element number * type scalar + 8 bytes) would be accessed. Then the unused first four bytes could be used for other things. I know there's probably a good reason for the current setup, I just don't know what it is, or I could be missing something entirely.
>
>

"Mac Reiter" <Mac_member@pathlink.com> wrote in message news:an9mu2$17at$1@digitaldaemon.com...
> In article <an59h5$2ptq$1@digitaldaemon.com>, Mike Wynn says...
> >I would prefer a syntax like
> >array:wrap[idx]  (-ve is from end, +ve as usual : bounds checked)
> >and
> >array:ring[idx]  (idx % array.length so never throws an exception)
> >
> >Mike.
>
> And a hearty second for that!  The performance problem with allowing
negative
> indices on unadorned arrays is that you take the performance hit on every
array
> index, even for arrays that you know are never going to be negatively
indexed.
> I really like the wrap/ring idea, because it lets normal arrays be
accessed at
> full speed, and special case arrays be treated as special case arrays.
>
> Dunno about the syntax.  Works well enough for me, but I could also see "building it in" to arrays, kind of like reverse():
>
> array.wrap()[idx]
> array.ring()[idx]
>
> or
> array.wrap(idx)
> array.ring(idx)

Fine. This fits better into the property syntax. And we could also have:

array.checked(idx)

Which would not do wrapping, but bounds checking in both directions.

Sandor

I will buy into the idea of an extra property for fancy indexing.  That sounds good.  I myself was considering a new D type ("buffer"?) but it makes more sense to add array properties.

I understand where Walter is coming from.  It is true that C++ has an underserved rap sheet on the performance issue.

My own view of the language world is a little different from Walter's.  I see no need for a new language built around screaming performance as its (a) number one design priority, (b) main user attraction, and (c) chief evaluation criterion. We already have C, C++, and assembly for that, and D can link to them.

I see a need for a compiled language midway between C++ and script languages.  I would consider it fine if D were on a par with C++ in most performance categories, while higher in some and lower in others.  To my mind there is no requirement to beat C++ in every single category.  I do see Mac's point that it is better not to be forced into lower performance if a choice can be provided. Neither should I be forced into higher performance if it means algorithms take 5-10 times longer to write and debug.

Having said that, no one has yet convinced me that this negative index trick is a real killer anyway, but think the proposed compromises are on the right track. In Icon for example, I can easily write code that performs just as fast as equivalent C, negative indices and all.  The difference is the time it takes to write the program correctly.  Icon is a breeze and C is a pain.

I agree with Walter that #pragmas and compiler switches are ugly. It's just that I view this whole performance orientation as servicing such a narrow audience that it almost merits such ugliness.  D will perform well enough for most work -- better than all (interpreted) script languages and practically par with C. The language acceptance factor will revolve around: what it is good for? how easy is it to use? what can it do that C++ can't? can it get the job done faster? and so on.  Performance will matter too, but it is not going to be the show-stopper.  D needs much, much more than that to win a place for itself.

It might be worth considering how many languages have fallen by the wayside and what caused them to do so.  Icon is an example -- a very beautiful language with no market presence.  One sees an ugly creature like Visual Basic which, for all its performance problems, has a huge market presence and almost outstrips C++ itself.  So there is more to language acceptance than performance or even elegance.  In the VB case the answer is "easy to use" and that factor is greatly enhanced by the sort of features we are debating here.

Mark

"Mark Evans" <Mark_member@pathlink.com> wrote in message news:and4mr$27aq$1@digitaldaemon.com...
> My own view of the language world is a little different from Walter's.  I
see no
> need for a new language built around screaming performance as its (a)
number one
> design priority, (b) main user attraction, and (c) chief evaluation
criterion.
> We already have C, C++, and assembly for that, and D can link to them.

Performance is a high priority, but for D it does not trump everything else. For example, in D all functions are virtual, which will cost some very slight performance compared with an expertly crafted C++ program, but for the relief from bugs and for real programs, it's worth it. D also loses some performance by not allowing class objects on the stack, but the huge gains are worth it.

> I see a need for a compiled language midway between C++ and script
languages.  I
> would consider it fine if D were on a par with C++ in most performance categories, while higher in some and lower in others.  To my mind there is
no
> requirement to beat C++ in every single category.  I do see Mac's point
that it
> is better not to be forced into lower performance if a choice can be
provided.
> Neither should I be forced into higher performance if it means algorithms
take
> 5-10 times longer to write and debug.

D does the dhrystone benchmark faster than C <g>.


> I agree with Walter that #pragmas and compiler switches are ugly. It's
just that
> I view this whole performance orientation as servicing such a narrow
audience
> that it almost merits such ugliness.  D will perform well enough for most
work
> -- better than all (interpreted) script languages and practically par with
C.
> The language acceptance factor will revolve around: what it is good for?
how
> easy is it to use? what can it do that C++ can't? can it get the job done faster? and so on.  Performance will matter too, but it is not going to be
the
> show-stopper.  D needs much, much more than that to win a place for
itself.

I believe D has that. I've gone well past the point where I keep finding myself frustrated when doing a bit of C++, because it's much easier to get it done, right, and tested in D.

> It might be worth considering how many languages have fallen by the
wayside and
> what caused them to do so.  Icon is an example -- a very beautiful
language with
> no market presence.

I have no idea why Icon failed. It's a remarkable language. I have the manual for it now and the implementation book <g>.

>  One sees an ugly creature like Visual Basic which, for all
> its performance problems, has a huge market presence and almost outstrips
C++
> itself.  So there is more to language acceptance than performance or even elegance.  In the VB case the answer is "easy to use" and that factor is
greatly
> enhanced by the sort of features we are debating here.

Basic is a fine language if your program is less than 24 lines long. I don't see it as a competitor for C++ or D, it serves a completely different need. Even if VB vanished tomiorrow, VB programmers won't switch to C++. They'll switch to javascript, Perl, Python or something similar.

"Mac Reiter" <Mac_member@pathlink.com> wrote in message news:an9o61$18j0$1@digitaldaemon.com...
> And since arrays know their length at all times, D's str.length is
'length'
> times faster than C's strlen(str), which had to do a linear traversal
looking
> for the NULL.  That makes the D version not just a syntactic difference,
but a
> very real performance improvement.

Yup. That's one of the reasons why D programs can run faster than C.

"Walter" <walter@digitalmars.com> ha scritto nel messaggio news:ane4kc$5fv$1@digitaldaemon.com...
>[...]
>
> Basic is a fine language if your program is less than 24 lines long. I
don't
> see it as a competitor for C++ or D, it serves a completely different
need.
> Even if VB vanished tomiorrow, VB programmers won't switch to C++. They'll switch to javascript, Perl, Python or something similar.

I think Delphi has a point here: simple, fast, powerful and portable. Do you think D will have a little slice of them?

Ciao

>Performance is a high priority, but for D it does not trump everything else. For example, in D all functions are virtual

Good, good.

>> I see a need for a compiled language midway between C++ and script
>
>D does the dhrystone benchmark faster than C <g>.
>

Yes, but does it take less time to *write* the benchmark in D?  That is the kind of question performance benchmarks can never answer.

>I believe D has that. I've gone well past the point where I keep finding myself frustrated when doing a bit of C++, because it's much easier to get it done, right, and tested in D.

D as it stands is better than C++, but is still much too close to C and much too far from script langugaes.

I haven't used D enough to make fair statements.  I have used negative indexing enough to make opinionated statements about that :-).


>I have no idea why Icon failed. It's a remarkable language. I have the manual for it now and the implementation book <g>.

The point was that D might suffer a similar fate.  I have some guesses about Icon.  Lack of open-source development (until very recently), lack of marketing (comparison/contrast articles), lack of serious GUI tools, bad luck.

>
>Basic is a fine language if your program is less than 24 lines long. I don't see it as a competitor for C++ or D, it serves a completely different need. Even if VB vanished tomiorrow, VB programmers won't switch to C++. They'll switch to javascript, Perl, Python or something similar.
>

You missed my point completely here.  The point was that Visual Basic fails catastrophically on all benchmark performance tests, yet conquered the market. The fact that VB users would switch to Python, not C++, only proves the point more emphatically:  performance is not the trump suit.  It does not prove that they serve different needs at all.

People are doing a lot more with VB than writing 24-line programs.  Serious, real-world programs are written with VB all day long, and people make good money doing it.  That is the kind of outcome I would like for D.  All of these VB programs could instead be written in C++.

My point was that performance is only one factor in market acceptance, and VB proves it is not necessarily a major one.  Ease of use is much higher on the list.  VB and C++ do not really serve different needs; VB edged out C++ in several problem domains because it is easier to use.  I'd also point out that Microsoft (no dummy at marketing) has recognized the success of VB-like tools and come up with C# and .NET, which brings C++ closer to VB in the language sense.

So when a proposed feature only costs 5-10%, but yields 3x algorithm design gains, I say that it is worth it.  If I want extreme performance, and have all day long to tweak and tune, then I will extern-C the thing and suffer the consequences.

Nice chatting Walter...and D is still fabulous,

Mark