Currently, in the array runtime code, all appends and length settings copy the BlkAttr attributes from the existing block to the new block.  This is important if you set block attributes other than the default, or you are appending to a block which is currently typed differently than what it was when allocated.

However, the one function which does *not* do this is dup.  dup uses the typeinfo to determine the block attributes, including the NO_SCAN flag.

I think it should copy the attributes like all the other functions do.  However, there is one caveat to this.  Currently dup doesn't look up the block attributes of the existing data, because it doesn't have to.  So changing dup to copy the attributes would slow down dup a bit because in addition to allocating a new block, it must look up the flags for the old block.

What do you guys think?

-Steve

One more thing -- concatenation suffers from the same problem.  This one might be much harder to swallow since you may have to do N flag lookups.

-Steve



----- Original Message ----
> From: Steve Schveighoffer <schveiguy at yahoo.com>
> To: d-runtime <d-runtime at puremagic.com>
> Sent: Wed, August 4, 2010 8:32:38 AM
> Subject: [D-runtime] dup attributes?
> 
> Currently, in the array runtime code, all appends and length settings copy the

> BlkAttr attributes from the existing block to the new block.  This is
>important
>
> if you set block attributes other than the default, or you are  appending to a

> block which is currently typed differently than what it was  when allocated.
> 
> However, the one function which does *not* do this is  dup.  dup uses the typeinfo to determine the block attributes,  including the NO_SCAN flag.
> 
> I think it should copy the attributes like  all the other functions do.
>However,
>
> there is one caveat to  this.  Currently dup doesn't look up the block
>attributes
>
> of the  existing data, because it doesn't have to.  So changing dup to copy the
>
> attributes would slow down dup a bit because in addition to allocating a new block, it must look up the flags for the old block.
> 
> What do you guys  think?
> 
> -Steve
> 
> 
> 
> 
> _______________________________________________
> D-runtime mailing  list
> D-runtime at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/d-runtime
> 

Yuck.  Seems like dup should be consistent with the other functions, though it could go either way.

On Aug 4, 2010, at 5:32 AM, Steve Schveighoffer wrote:

> Currently, in the array runtime code, all appends and length settings copy the BlkAttr attributes from the existing block to the new block.  This is important if you set block attributes other than the default, or you are appending to a block which is currently typed differently than what it was when allocated.
> 
> However, the one function which does *not* do this is dup.  dup uses the typeinfo to determine the block attributes, including the NO_SCAN flag.
> 
> I think it should copy the attributes like all the other functions do.  However, there is one caveat to this.  Currently dup doesn't look up the block attributes of the existing data, because it doesn't have to.  So changing dup to copy the attributes would slow down dup a bit because in addition to allocating a new block, it must look up the flags for the old block.
> 
> What do you guys think?
> 
> -Steve
> 
> 
> 
> 
> _______________________________________________
> D-runtime mailing list
> D-runtime at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/d-runtime

Concatenation doesn't preserve bit flags?  Like if I do this:

mystr ~= "a" ~ "b";

You're saying that it's possible for the bits set on mystr to be lost?

On Aug 4, 2010, at 10:19 AM, Steve Schveighoffer wrote:

> One more thing -- concatenation suffers from the same problem.  This one might be much harder to swallow since you may have to do N flag lookups.
> 
> -Steve
> 
> 
> 
> ----- Original Message ----
>> From: Steve Schveighoffer <schveiguy at yahoo.com>
>> To: d-runtime <d-runtime at puremagic.com>
>> Sent: Wed, August 4, 2010 8:32:38 AM
>> Subject: [D-runtime] dup attributes?
>> 
>> Currently, in the array runtime code, all appends and length settings copy the
> 
>> BlkAttr attributes from the existing block to the new block.  This is important
>> 
>> if you set block attributes other than the default, or you are  appending to a
> 
>> block which is currently typed differently than what it was  when allocated.
>> 
>> However, the one function which does *not* do this is  dup.  dup uses the typeinfo to determine the block attributes,  including the NO_SCAN flag.
>> 
>> I think it should copy the attributes like  all the other functions do. However,
>> 
>> there is one caveat to  this.  Currently dup doesn't look up the block attributes
>> 
>> of the  existing data, because it doesn't have to.  So changing dup to copy the
>> 
>> attributes would slow down dup a bit because in addition to allocating a new block, it must look up the flags for the old block.
>> 
>> What do you guys  think?
>> 
>> -Steve
>> 
>> 
>> 
>> 
>> _______________________________________________
>> D-runtime mailing  list
>> D-runtime at puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/d-runtime
>> 
> 
> 
> 
> _______________________________________________
> D-runtime mailing list
> D-runtime at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/d-runtime

On 6-ago-10, at 01:15, Sean Kelly wrote:

> Concatenation doesn't preserve bit flags?  Like if I do this:
>
> mystr ~= "a" ~ "b";
>
> You're saying that it's possible for the bits set on mystr to be lost?
>
> On Aug 4, 2010, at 10:19 AM, Steve Schveighoffer wrote:
>
>> One more thing -- concatenation suffers from the same problem.
>> This one might
>> be much harder to swallow since you may have to do N flag lookups.

well it is not so clear to me what the flags of a~b should be if the
flags of a & b happens to be different.
I think that probably always using the default type flags from the
typeinfo for the result is the way to go.

Append on the other end, a~=b should preserve the flags of a.
If the space is correctly reserved the realloc part should just be
called log(N) times, so the cost is ok IMHO.

Dup should likewise preserve the flags.

Fawzi

Well, no, not exactly like that.

The issue is, if you have two void[]'s, and each one has flags associated with them, the flags of the existing blocks are disregarded.  The flags of the void[] type are used instead.

So something like this:

int[] x = new int[20];

void[] y = x; // y is still the same block, so the NO_SCAN bit is set

y ~= y; // appending preserves bit, so NO_SCAN bit remains set

y = y ~ y; // concatenation ignores the bits of the arguments, so the type is used to determine bits.  y now has the NO_SCAN bit cleared.

y = x; // reset

y = y.dup; // dup ignores current flags, the type is used instead.

-Steve



----- Original Message ----
> From: Sean Kelly <sean at invisibleduck.org>
> To: D's runtime library developers list <d-runtime at puremagic.com>
> Sent: Thu, August 5, 2010 7:15:19 PM
> Subject: Re: [D-runtime] dup attributes?
> 
> Concatenation doesn't preserve bit flags?  Like if I do this:
> 
> mystr  ~= "a" ~ "b";
> 
> You're saying that it's possible for the bits set on mystr  to be lost?
> 
> On Aug 4, 2010, at 10:19 AM, Steve Schveighoffer  wrote:
> 
> > One more thing -- concatenation suffers from the same  problem.  This one
>might
>
> > be much harder to swallow since you may  have to do N flag lookups.
> > 
> > -Steve
> > 
> > 
> > 
> > ----- Original Message ----
> >> From: Steve Schveighoffer  <schveiguy at yahoo.com>
> >> To:  d-runtime <d-runtime at puremagic.com>
> >>  Sent: Wed, August 4, 2010 8:32:38 AM
> >> Subject: [D-runtime] dup  attributes?
> >> 
> >> Currently, in the array runtime code, all  appends and length settings copy
>the
>
> > 
> >> BlkAttr attributes  from the existing block to the new block.  This is
> >>  important
> >> 
> >> if you set block attributes other than the  default, or you are  appending
>to a
>
> > 
> >> block which is  currently typed differently than what it was  when
>allocated.
> >> 
> >> However, the one function which does *not* do this is   dup.  dup uses the typeinfo to determine the block  attributes,  including the NO_SCAN flag.
> >> 
> >> I think  it should copy the attributes like  all the other functions do. However,
> >> 
> >> there is one caveat to   this.  Currently dup doesn't look up the block attributes
> >> 
> >> of the  existing data, because it doesn't have  to.  So changing dup to copy
>the
>
> >> 
> >> attributes would  slow down dup a bit because in addition to allocating a
>new
>
> >> block,  it must look up the flags for the old block.
> >> 
> >> What do  you guys  think?
> >> 
> >> -Steve
> >> 
> >> 
> >> 
> >> 
> >>  _______________________________________________
> >> D-runtime  mailing  list
> >> D-runtime at puremagic.com
> >>  http://lists.puremagic.com/mailman/listinfo/d-runtime
> >> 
> > 
> > 
> > 
> >  _______________________________________________
> > D-runtime mailing  list
> > D-runtime at puremagic.com
> > http://lists.puremagic.com/mailman/listinfo/d-runtime
> 
> _______________________________________________
> D-runtime  mailing list
> D-runtime at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/d-runtime
> 

----- Original Message ----
> From: Fawzi Mohamed <fawzi at gmx.ch>
> 
> well it is not so clear to me what the  flags of a~b should be if the flags of
>a & b happens to be different.
> I  think that probably always using the default type flags from the typeinfo
>for  the result is the way to go.

What about an intelligent choice?  For example, the NO_SCAN bits should be and-ed.  That's what I was thinking.  But of course, this means N flag lookups, so there is a performance concern.

> Append on the other end, a~=b should  preserve the flags of a.
> If the space is correctly reserved the realloc part  should just be called
>log(N) times, so the cost is ok IMHO.

Append already is this way.

> Dup should  likewise preserve the  flags.

OK, good, I agree there.  The cost is relatively constant.

-Steve

On 6-ago-10, at 13:09, Steve Schveighoffer wrote:

> ----- Original Message ----
>> From: Fawzi Mohamed <fawzi at gmx.ch>
>>
>> well it is not so clear to me what the  flags of a~b should be if
>> the flags of
>> a & b happens to be different.
>> I  think that probably always using the default type flags from the
>> typeinfo
>> for  the result is the way to go.
>
> What about an intelligent choice?  For example, the NO_SCAN bits
> should be
> and-ed.  That's what I was thinking.  But of course, this means N
> flag lookups,
> so there is a performance concern.

I dislike "intelligent choices" if they have a cost and makes
dependencies less local (as here),
unless it is really what one would expect.
In this case if I use a global void[], of ubyte[] buffer to do
concatenation of arbitrary types of the results change,
and it is unclear how to extend the behaviour wrt. exact scanning
should be.
It seems a source of hard to track bugs...

The cost should not be really too much, but I prefer that the target
type "decides" the correct flags.
Using templates one can make it so that the target type is the correct
one.
It is faster and, IMHO, logically as well founded as the alternatives,
thus I would keep the current behaviour (It is also less work :).

----- Original Message ----
> From: Fawzi Mohamed <fawzi at gmx.ch>
> To: D's runtime library developers list <d-runtime at puremagic.com>
> Sent: Fri, August 6, 2010 7:35:57 AM
> Subject: Re: [D-runtime] dup attributes?
> 
> On 6-ago-10, at 13:09, Steve Schveighoffer wrote:
> 
> > ----- Original  Message ----
> >> From: Fawzi Mohamed <fawzi at gmx.ch>
> >> 
> >> well it  is not so clear to me what the  flags of a~b should be if the flags
>of
> >> a & b happens to be different.
> >> I  think that  probably always using the default type flags from the
>typeinfo
> >>  for  the result is the way to go.
> > 
> > What about an  intelligent choice?  For example, the NO_SCAN bits should be
> >  and-ed.  That's what I was thinking.  But of course, this means N flag
>lookups,
> > so there is a performance concern.
> 
> I dislike  "intelligent choices" if they have a cost and makes dependencies
>less local (as  here),
> unless it is really what one would expect.
> In this case if I use a  global void[], of ubyte[] buffer to do concatenation
>of arbitrary types of the  results change,
> and it is unclear how to extend the behaviour wrt. exact  scanning should be. It seems a source of hard to track bugs...
> 
> The  cost should not be really too much, but I prefer that the target type
>"decides"  the correct flags.
> Using templates one can make it so that the target type is  the correct one. It is faster and, IMHO, logically as well founded as the  alternatives, thus I would keep the current behaviour (It is also less work  :).

Well, let me get to the root of the problem then :)

The issue is that phobos deals with void[] for things like I/O, which to me seems like the correct type.  It allows you to pass in any array type without casting, and it says "I don't know what type this is".  One of these functions is std.file.read() which reads the entire file into an array and returns that array as a void[].  Given that all the C functions use void *, it seems like the right call.  But problems come in because void[] has the NO_SCAN bit cleared, but this clearly has no pointers in it (the source is a file!).  So what to do?

std.file.read gets around this by calling gc_malloc directly.  However, it was rightfully pointed out that if one simply dups the array, or uses concatenation with it, it magically loses the NO_SCAN bit.  See the bug report here: http://d.puremagic.com/issues/show_bug.cgi?id=4572

OK, after writing all this, I'm going to change my mind :P  I just realized that the only time, and I'm pretty sure this is the *only* time, we have to worry about this is when you're concatenating two or more void[]'s.  This corner case seems highly unlikely. Append doesn't have the problem, and if you have casted your void[]'s to something else, then why not use that new type's bits?  I think you are right, concatenation should stay as is.

I guess concatenation is not an issue, and we agree that dup should replicate the flags.

OK, I'll go about making the change for dup.

-Steve

On 6-ago-10, at 14:03, Steve Schveighoffer wrote:

> ----- Original Message ----
>> From: Fawzi Mohamed <fawzi at gmx.ch>
>> To: D's runtime library developers list <d-runtime at puremagic.com>
>> Sent: Fri, August 6, 2010 7:35:57 AM
>> Subject: Re: [D-runtime] dup attributes?
>>
>> On 6-ago-10, at 13:09, Steve Schveighoffer wrote:
>>
>>> ----- Original  Message ----
>>>> From: Fawzi Mohamed <fawzi at gmx.ch>
>>>>
>>>> well it  is not so clear to me what the  flags of a~b should be if the flags
>> of
>>>> a & b happens to be different.
>>>> I  think that  probably always using the default type flags from
>>>> the
>> typeinfo
>>>> for  the result is the way to go.
>>>
>>> What about an  intelligent choice?  For example, the NO_SCAN bits
>>> should be
>>> and-ed.  That's what I was thinking.  But of course, this means N
>>> flag
>> lookups,
>>> so there is a performance concern.
>>
>> I dislike  "intelligent choices" if they have a cost and makes
>> dependencies
>> less local (as  here),
>> unless it is really what one would expect.
>> In this case if I use a  global void[], of ubyte[] buffer to do
>> concatenation
>> of arbitrary types of the  results change,
>> and it is unclear how to extend the behaviour wrt. exact  scanning
>> should be.
>> It seems a source of hard to track bugs...
>>
>> The  cost should not be really too much, but I prefer that the
>> target type
>> "decides"  the correct flags.
>> Using templates one can make it so that the target type is  the
>> correct one.
>> It is faster and, IMHO, logically as well founded as the
>> alternatives,
>> thus I would keep the current behaviour (It is also less work  :).
>
> Well, let me get to the root of the problem then :)
>
> The issue is that phobos deals with void[] for things like I/O,
> which to me
> seems like the correct type.  It allows you to pass in any array
> type without
> casting, and it says "I don't know what type this is".  One of these
> functions
> is std.file.read() which reads the entire file into an array and
> returns that
> array as a void[].  Given that all the C functions use void *, it
> seems like the
> right call.  But problems come in because void[] has the NO_SCAN bit
> cleared,
> but this clearly has no pointers in it (the source is a file!).  So
> what to do?

>
> std.file.read gets around this by calling gc_malloc directly.
> However, it was
> rightfully pointed out that if one simply dups the array, or uses
> concatenation
> with it, it magically loses the NO_SCAN bit.  See the bug report here:
> http://d.puremagic.com/issues/show_bug.cgi?id=4572

also tango has this problem, and I had discussed about it with kris
exactly because
I was worried about the NO_SCAN bits would be wrong, and so I found
ubyte[] a better
choice.
But it turns out that (at least in tango) void[] is almost only used
to pass in a buffer where one then
either reads or writes, and never resized, so it was almost never a
problem.
So the fact that one can cast to void[] any array easily it makes it a
better choice than ubyte[].

Still load and get methods that load a whole file have this problem,
and should allocate their
buffers as ubyte[] (something read by a file should *not* contain
pointers).

Not sure if the return type of those methods should be ubyte[] after
all (I lean a bit on yes
because calling those methods you will need to cast the void[] away,
not cast to void[]
  as all other cases), but for sure the internal allocation should be
of an ubyte[].

> OK, after writing all this, I'm going to change my mind :P  I just
> realized that
> the only time, and I'm pretty sure this is the *only* time, we have
> to worry
> about this is when you're concatenating two or more void[]'s.  This
> corner case
> seems highly unlikely. Append doesn't have the problem, and if you
> have casted
> your void[]'s to something else, then why not use that new type's
> bits?  I think
> you are right, concatenation should stay as is.
>
> I guess concatenation is not an issue, and we agree that dup should
> replicate
> the flags.
>
> OK, I'll go about making the change for dup.
ok great, it seems that sometime it is possible to reach a conclusion
to a discussion quickly,
somehow refreshing :)

ciao
Fawzi