18-Jan-2013 01:49, Era Scarecrow пишет:
> On Thursday, 17 January 2013 at 19:36:34 UTC, Dmitry Olshansky wrote:
>> I'm thinking that a opSlice of stack-allocated must be @system and a
>> heap allocated can be @safe.
>
>   That's just a small part of the problem. With the new design, 90% of
> it can be safe; Just the actual slice buffer when you request it (with
> Fixed storage) would be @system,and slices (having a pointer)

Slices should have a pointer and length so they can check range (in debug builds), alternatively just a pointer to storage that nows the range.

 But
> @safe code can't call @system code which means that the current design
> (convert to slices before operations) it all has to all be @system.
>

I think slice ops could be made @safe if the primitives for bulk transfer and opIndex in the Storage are @trusted.

If we just mark all of opSlice on Fixed-sized one as @system then it should be good enough.In other words (I'm lost in terminology):

FixedBitArray!(1024) fixed;
auto slice = fixed[0..100]; //this is @system
slice[10..20] = slice[20..30]; //this is @safe
foreach(bool x; slice[0..10])  //this is @safe
{... }

The idea is to make code responsible for the "root" of unsafety to be @system everything else then can be @safe or @trusted(where compiler can't see it).

>   Guess once it's debugged the entire thing can be @trusted and only
> certain things can be marked @system instead.

I think that the most of slice ops can be @safe/@trusted. It's the process of taking a slice out of stack-based container that is not @safe. The one taking a slice out of GC-based one is @safe.

-- 
Dmitry Olshansky

On Friday, 18 January 2013 at 16:42:04 UTC, Dmitry Olshansky wrote:
> Slices should have a pointer and length so they can check range (in debug builds), alternatively just a pointer to storage that nows the range.

 So write an alternate opIndex...

> I think slice ops could be made @safe if the primitives for bulk transfer and opIndex in the Storage are @trusted.
>
> If we just mark all of opSlice on Fixed-sized one as @system then it should be good enough. In other words (I'm lost in terminology):
>

<snip>

> The idea is to make code responsible for the "root" of unsafety to be @system everything else then can be @safe or @trusted(where compiler can't see it).

 Hmmm okay... Somehow this seemed so much more complex than it's put here.

> I think that the most of slice ops can be @safe/@trusted. It's the process of taking a slice out of stack-based container that is not @safe. The one taking a slice out of GC-based one is @safe.

 And depending on if it is fixed or dynamic will determine if the slice is @system or @safe... Makes sense. That sets me straight.


 So for the slice/range, what all should it support? I've tried to have it support (or foward) opBinary, opBinaryAssign, comparing, slicing, copying, as well as inputRange, bidirectional, opIndex & opIndex assign, opDollar and length. If it should support a much smaller sub-set I'll need to know; Could greatly affect how it is written.

19-Jan-2013 11:14, Era Scarecrow пишет:
> On Friday, 18 January 2013 at 16:42:04 UTC, Dmitry Olshansky wrote:
>> Slices should have a pointer and length so they can check range (in
>> debug builds), alternatively just a pointer to storage that nows the
>> range.
>
>   So write an alternate opIndex...
>
>> I think slice ops could be made @safe if the primitives for bulk
>> transfer and opIndex in the Storage are @trusted.
>>
>> If we just mark all of opSlice on Fixed-sized one as @system then it
>> should be good enough. In other words (I'm lost in terminology):
>>
>
> <snip>
>
>> The idea is to make code responsible for the "root" of unsafety to be
>> @system everything else then can be @safe or @trusted(where compiler
>> can't see it).
>
>   Hmmm okay... Somehow this seemed so much more complex than it's put here.
>
>> I think that the most of slice ops can be @safe/@trusted. It's the
>> process of taking a slice out of stack-based container that is not
>> @safe. The one taking a slice out of GC-based one is @safe.
>
>   And depending on if it is fixed or dynamic will determine if the slice
> is @system or @safe... Makes sense. That sets me straight.
>
>
>   So for the slice/range, what all should it support? I've tried to have
> it support (or foward) opBinary, opBinaryAssign, comparing, slicing,
> copying, as well as inputRange, bidirectional, opIndex & opIndex assign,
> opDollar and length. If it should support a much smaller sub-set I'll
> need to know; Could greatly affect how it is written.


opBinaryAssign --> opOpAssign. opSlice/opSlice assign.
In any case it seems to me that (I think you already did this trick before) you can reuse a lot of code by making an operator string a template parameter (for functions that implement  =, |=, &=, ^=, and therefore ^, |, &) and use string mixins.

Other then this - don't forget the '~' _unary_ operator as it makes a lot of sense for bit-vectors. Now since slices don't support '~' concatenation it won't look so bad. Worst case:

auto ba = BitArray(...);
auopt slice = ba[...];
ba ~= ~slice;

And it's not much of problem.
-- 
Dmitry Olshansky

On Saturday, 19 January 2013 at 09:07:03 UTC, Dmitry Olshansky wrote:
> opBinaryAssign --> opOpAssign. opSlice/opSlice assign. In any case it seems to me that (I think you already did this trick before) you can reuse a lot of code by making an operator string a template parameter (for functions that implement  =, |=, &=, ^=, and therefore ^, |, &) and use string mixins.

 Yes I did when I could; The slice versions can only forwards them to the bitarray one.

> Other then this - don't forget the '~' _unary_ operator as it makes a lot of sense for bit-vectors. Now since slices don't support '~' concatenation it won't look so bad. Worst case:
>
> auto ba = BitArray(...);
> auopt slice = ba[...];
> ba ~= ~slice;

 Only works if you create a new version and append it; And all opBinary operations make duplicates due to their nature. So the above couldn't work if slice is a separate type, and instead:

  auto slice = ba[...];   //slice type of BitArraySlice
  auto bc ~= ~slice;      //ba/bc type of BitArray

  ba = (ba ~ (~slice))[]; //type of BitArraySlice


> And it's not much of problem.

 Hmmm.. Well as mentioned I don't think a separate slice/range type is the best design or as practical; I have it half working but one problem I'm really having is trying to make sense of the errors after I converted to use a pointer. It's confusing since it was compiling before and is an annoyance more-so now; And suddenly can't deduce the type when it should be able to, and if I explicitly call it with the type it doesn't help any.


[quote]
Error: template bitmanip.BitArray!(Fixed!(1024)).BitArray.opEquals(alias SL)(const SL rhs) if (hasMember!(SL, "isBitArray") || hasMember!(SL, "isBitArraySlice")) forward reference to template opEquals(alias SL)(const SL rhs) if (hasMember!(SL, "isBitArray") || hasMember!(SL, "isBitArraySlice"))
Error: template bitmanip.BitArray!(Fixed!(1024)).BitArray.opEquals cannot deduce template function from argument types !()(BitArraySlice!(BitArray!(Fixed!(1024)), true),const(ulong),const(ulong))

Error: cannot implicitly convert expression ((*this.slice).toString(this.start, this.end)) of type string to string
[/quote]

[code]
  //isBitArray and isBitArraySlice are enums in the structs
  //almost all functions templated to handle fixed/dynamic
  //types as compatible. Plus constness for range(s).

  //auto ref would be nice/preferred in a lot in these functions..
  //Otherwise copying is required unless i do more duplication/code forwarding

  //if opSlice is @system, then almost all these are @system, or @trusted

  //slice forwarding
  bool opEquals(SL)(const SL rhs) const
  if (hasMember!(SL, "isBitArray") || hasMember!(SL, "isBitArraySlice"))
  {
    //opSlice always returns BitArraySlice
    return opEquals(rhs[], 0, length);
  }

  bool opEquals(SL)(const SL rhs, ulong sliceStart, ulong sliceEnd) const
  if (hasMember!(SL, "isBitArraySlice"))
  {
    if ((sliceEnd-sliceStart) != rhs.length)
      return false;

      return opCmp(rhs, sliceStart, sliceEnd) == 0;
  }

  //slice forwarding
  int opCmp(SL)(const SL rhs) const
  if (hasMember!(SL, "isBitArray"))
  {
    //opSlice always returns BitArraySlice
    return opCmp(rhs[], 0, length);
  }

  int opCmp(SL)(const SL rhs, ulong sliceStart, ulong sliceEnd) const
  if (hasMember!(SL, "isBitArraySlice"));
[/code]



 A small consideration was coming to mind in cases where storage management if you only needed to prepend a few bits or append just after for a few bits but don't need to do a full reallocation. In that case perhaps a before/after allocated memory would serve nice.

  size_t beforeArray, afterArray;
  int beforeUsed, afterUsed; //course would likely be bitfield or ubyte

 Now if we use that it comes to mind that although it wouldn't be perfect storage for small compact arrays (as was the original complaint(s)), you could however rely on them just as much as the main array and the first size_t bits (*2 if you prepend/append to the array a lot. It would also be useful to append past the end of a sliced area without tampering with data outside it's access area until it's forced to resize.


 Fixed BitArraySizes seem far more wanted where you don't want to do extra memory allocation, so using reference counting wouldn't work, but with dynamic arrays you could. So if you did all the slices to a particular bit array could always point to the proper memory.

 If you did do that, prepending to an array would be a nightmare unless you could offset it with a virtual offset to represent the actual beginning. So...

  //start in the middle
  struct STORE {
    //virtualOffset instead?
    ulong startOffset = ulong.max / 2;
    size_t[] store;
  }

  //in slice/Bitarray, hope that's right
  RefCounted!(STORE) store;
  ulong start, end;

 With these three you can safely do dynamic slices while all slices point to the same data; in cases where there's only one reference store, start, end & startOffset can freely be changed.

 Assume you slice. So..

  BitArray ba = BitArray(100);
  auto sl = ba[10 .. 20];

  //assume startOffset says say 1024, start, 1024 & end = 1124.
  //slice holds start 1034, end 1044.

 Now if we have it prepend a whole lot.

  foreach(i; 0 .. 100)
    ba = true ~ ba;

 It had to be resized, so if it gave it say 512 more bits then...
  //now startOffset = 512.

  slice[0] = true; //start-startOffset = actual bit offset in store[]
  assert(ba[110]); //was set by the slice

 Full reference could be preserved, although if you prepend to a slice vs the main array then either you overlap data or need some temporary storage until duping/reallocating is required, or just dup/allocate new memory. Or have
a separate end/before arrays which won't force reallocation but could make for interesting overlapping as they wouldn't all always point to the same thing.


 The interesting problems that crop up again.


 Another thought, is what if we give an ID to fixed arrays so they know if they are pointing to proper data or junk; This would likely mostly be for the known fixed array slices. So..

  size_t ID; //in both slice and fixed array.
  size_t[1024] fixedSize;

 Now add a variant to ensure the ID's match on each and every access. If the ID gets corrupted the data is corrupted too, although not a guarantee it's pretty darn close if the ID is random.


 But as you mentioned making it @system then we could have it silently convert the BitArray from fixed to dynamic; And make the slice @system where normally it's @safe (although immediately duping the memory location after it's using the dynamic form would be safe).


 Reminds me... Should initialization/blit dup or reference/preserve the memory? I'd think it would point to the old memory unless explicitly told to (or going from const to non-const versions).

(Moved from: What have I missed?)

 If this is the wrong place to ask these questions I apologize, getting back into this takes some work.


 So I guess I need to ask: Should I try to resume work on the BitManip library? (So far it seems none of my code has been integrated to phobos)

 Assuming I do, should I try to fix lots of small bugs and make smaller pulls or should I try to do several at once? When I re-wrote the BitArray I personally think it is an overall improvement in many ways, and being a complete re-write you can't just do bug #xxxx and then bug #xxxx and then bug... etc etc.

 Also to ask, how many people tried out the rewrite I proposed, and do they think it was actually an improvement for ease of use, speed, fewer bugs/issues, etc?

On Thu, Aug 07, 2014 at 01:10:06AM +0000, Era Scarecrow via Digitalmars-d-learn wrote:
> (Moved from: What have I missed?)
> 
>  If this is the wrong place to ask these questions I apologize,
>  getting back into this takes some work.

Since this is about contributing to Phobos, probably a better place to ask is on the main D forum.


>  So I guess I need to ask: Should I try to resume work on the BitManip
>  library? (So far it seems none of my code has been integrated to
>  phobos)

Do you have a pull request? Which one is it?


>  Assuming I do, should I try to fix lots of small bugs and make
>  smaller pulls or should I try to do several at once? When I re-wrote
>  the BitArray I personally think it is an overall improvement in many
>  ways, and being a complete re-write you can't just do bug #xxxx and
>  then bug #xxxx and then bug... etc etc.
> 
>  Also to ask, how many people tried out the rewrite I proposed, and do
>  they think it was actually an improvement for ease of use, speed,
>  fewer bugs/issues, etc?

There have been a few fixes to std.bitmanip recently, including a current PR for adding attributes to some of the functions. Are these your PRs, or are you proposing something totally new? If it's something totally new, where can we get the code?


T

-- 
"Maybe" is a strange word.  When mom or dad says it it means "yes", but when my big brothers say it it means "no"! -- PJ jr.

On Thursday, 7 August 2014 at 01:51:46 UTC, H. S. Teoh via Digitalmars-d-learn wrote:
> Since this is about contributing to Phobos, probably a better place to ask is on the main D forum.

 Yeah posted in my 'what have i missed?' as well...

> Do you have a pull request? Which one is it?

https://github.com/rtcvb32/phobos/pull/1

From the looks of things it's 4 commits that are merged into a single request i think...


> There have been a few fixes to std.bitmanip recently, including a current PR for adding attributes to some of the functions. Are these your PRs, or are you proposing something totally new? If it's something totally new, where can we get the code?

 Glancing at the current code, none of my stuff got in. I do a large number of fixes, so likely a lot of those changes would get thrown away or integrated...

 Knock yourself out... the pull/1 above holds most of the commits and diff/changes to note of.

 https://github.com/rtcvb32/phobos/blob/master/std/bitmanip.d

On Thursday, 7 August 2014 at 02:04:13 UTC, Era Scarecrow wrote:
> On Thursday, 7 August 2014 at 01:51:46 UTC, H. S. Teoh via Digitalmars-d-learn wrote:
>> Do you have a pull request? Which one is it?
>
> https://github.com/rtcvb32/phobos/pull/1
>
> From the looks of things it's 4 commits that are merged into a single request i think...

 Looks like that pull is just the bitfields... but the actual bitmanip one i listed has the whole source.

On Thu, Aug 07, 2014 at 02:04:12AM +0000, Era Scarecrow via Digitalmars-d-learn wrote:
> On Thursday, 7 August 2014 at 01:51:46 UTC, H. S. Teoh via Digitalmars-d-learn wrote:
> >Since this is about contributing to Phobos, probably a better place to ask is on the main D forum.
> 
>  Yeah posted in my 'what have i missed?' as well...
> 
> >Do you have a pull request? Which one is it?
> 
> https://github.com/rtcvb32/phobos/pull/1
> 
> From the looks of things it's 4 commits that are merged into a single request i think...

Hold on a sec... that's a pull for your own fork of Phobos. You need to submit a pull to the main Phobos repo in order to get it reviewed and merged. :-)


> >There have been a few fixes to std.bitmanip recently, including a current PR for adding attributes to some of the functions. Are these your PRs, or are you proposing something totally new? If it's something totally new, where can we get the code?
> 
> Glancing at the current code, none of my stuff got in. I do a large number of fixes, so likely a lot of those changes would get thrown away or integrated...
[...]

Well, no wonder, your pull was submitted against your local fork, not to the main Phobos repo, so nobody knew about it (except Dmitry, apparently). It would help to submit pulls against the main Phobos repo. :-)

Also, looks like you made major changes to the code... if it's a complete rewrite, I'd say submit it as a single pull. If it's a collection of many fixes, it's probably better to submit separate pulls so that the easy fixes will get in first while we work out the wrinkles on the more complicated fixes.


T

-- 
Любишь кататься - люби и саночки возить.

On Thursday, 7 August 2014 at 02:12:20 UTC, H. S. Teoh via Digitalmars-d-learn wrote:
> Hold on a sec... that's a pull for your own fork of Phobos. You need to submit a pull to the main Phobos repo in order to get it reviewed and merged. :-)


> Well, no wonder, your pull was submitted against your local fork, not to the main Phobos repo, so nobody knew about it (except Dmitry, apparently). It would help to submit pulls against the main  Phobos repo.
> :-)

> Also, looks like you made major changes to the code... if it's a complete rewrite, I'd say submit it as a single pull. If it's a collection of many fixes, it's probably better to submit separate pulls so that the easy fixes will get in first while we work out the  wrinkles on the more complicated fixes.

 I did submit it against the original Phobos, and the auto tester picked it up. For something like 3-4 weeks it tried and kept failing over and over again because it couldn't merge it.

 The reason? It was something like 13 unresolved newlines that didn't match up... or something that I have no idea how I could fix because whitespace is invisible.

 Later Walter or Andrei (I forget who) complained when they looked at it and wanted me to break it apart into smaller more indivdual bug fixes (since it's hard to tell what i changed as a single blob/replacement) but as a complete re-write and I don't know if I COULD do that...

 Trust me. I tried. It failed. Eventually I just pulled the request and gave up at that time... Right about the time of the Dconf 2013 I got burned out and got depressed (Probably major changes in meds).