I have an array of structs eg

struct PriceBar
{
  DateTime date;
  double open;
  double high;
  double low;
  double close;
}

(which fields are present in this particular struct will depend on template arguments).

what is the best way to turn these at compile time into a struct of arrays? eg

struct PriceBars
{
  DateTime[] dates;
  double[] opens;
  double[] highs;
  double[] lows;
  double[] closes;
}

I can use FieldTypeTuple and FieldNameTuple, but I am a bit lost as to how without static foreach to loop through these in order to generate a mixin to declare the new type.  I can turn it into a string, but what is the better option?

Thanks.

On Friday, 3 July 2015 at 10:52:03 UTC, Laeeth Isharc wrote:
> I have an array of structs eg
>
> struct PriceBar
> {
>   DateTime date;
>   double open;
>   double high;
>   double low;
>   double close;
> }
>
> (which fields are present in this particular struct will depend on template arguments).
>
> what is the best way to turn these at compile time into a struct of arrays? eg
>
> struct PriceBars
> {
>   DateTime[] dates;
>   double[] opens;
>   double[] highs;
>   double[] lows;
>   double[] closes;
> }
>
> I can use FieldTypeTuple and FieldNameTuple, but I am a bit lost as to how without static foreach to loop through these in order to generate a mixin to declare the new type.  I can turn it into a string, but what is the better option?
>
> Thanks.

see https://github.com/economicmodeling/soa

> see https://github.com/economicmodeling/soa

Thanks v much for this.

Figured out in the meantime one solution here (probably quite ugly):
https://gist.github.com/Laeeth/9174498f9b79dc90ac18

On Friday, 3 July 2015 at 10:52:03 UTC, Laeeth Isharc wrote:
> I have an array of structs eg
>
> struct PriceBar
> {
>   DateTime date;
>   double open;
>   double high;
>   double low;
>   double close;
> }
>
> (which fields are present in this particular struct will depend on template arguments).
>
> what is the best way to turn these at compile time into a struct of arrays? eg
>
> struct PriceBars
> {
>   DateTime[] dates;
>   double[] opens;
>   double[] highs;
>   double[] lows;
>   double[] closes;
> }
>
> I can use FieldTypeTuple and FieldNameTuple, but I am a bit lost as to how without static foreach to loop through these in order to generate a mixin to declare the new type.  I can turn it into a string, but what is the better option?
>
> Thanks.

Not necessarily the best solution to the problem, but I just wanted to show this because I think it's kinda cool:

https://gist.github.com/John-Colvin/103d3c064ad6cb4cf435#file-transformmembers-d

>
> Not necessarily the best solution to the problem, but I just wanted to show this because I think it's kinda cool:
>
> https://gist.github.com/John-Colvin/103d3c064ad6cb4cf435#file-transformmembers-d


Very impressive - now I have to try to understand how it works...!

On 07/03/15 12:52, Laeeth Isharc via Digitalmars-d-learn wrote:
> I have an array of structs eg
> 
> struct PriceBar
> {
>   DateTime date;
>   double open;
>   double high;
>   double low;
>   double close;
> }
> 
> (which fields are present in this particular struct will depend on template arguments).
> 
> what is the best way to turn these at compile time into a struct of arrays? eg
> 
> struct PriceBars
> {
>   DateTime[] dates;
>   double[] opens;
>   double[] highs;
>   double[] lows;
>   double[] closes;
> }
> 
> I can use FieldTypeTuple and FieldNameTuple, but I am a bit lost as to how without static foreach to loop through these in order to generate a mixin to declare the new type.  I can turn it into a string, but what is the better option?

The simplest solution is something like:

   template SOA(Struct, size_t LENGTH) {
      struct SOA  {
         enum MEMBERNAME(size_t N) = __traits(identifier, Struct.tupleof[N]);

         static __gentypes() {
            string ret;
            foreach (I, TYPE; typeof(Struct.tupleof))
               ret ~= "align(16) typeof(Struct.tupleof["~I.stringof~"])["~LENGTH.stringof~"] "
                       ~ MEMBERNAME!I ~ ";";
            return ret;
         }
         mixin(__gentypes());
      }
   }

   alias PriceBars = SOA!(PriceBar, 8);

which you'll have to adjust to your requirements (eg drop the '"~LENGTH.stringof~"' part to get a struct-of-slices, which is what your example above shows).

artur

>> I can use FieldTypeTuple and FieldNameTuple, but I am a bit lost as to how without static foreach to loop through these in order to generate a mixin to declare the new type.  I can turn it into a string, but what is the better option?
>
> The simplest solution is something like:
>
>    template SOA(Struct, size_t LENGTH) {
>       struct SOA  {
>          enum MEMBERNAME(size_t N) = __traits(identifier, Struct.tupleof[N]);
>
>          static __gentypes() {
>             string ret;
>             foreach (I, TYPE; typeof(Struct.tupleof))
>                ret ~= "align(16) typeof(Struct.tupleof["~I.stringof~"])["~LENGTH.stringof~"] "
>                        ~ MEMBERNAME!I ~ ";";
>             return ret;
>          }
>          mixin(__gentypes());
>       }
>    }
>
>    alias PriceBars = SOA!(PriceBar, 8);
>
> which you'll have to adjust to your requirements (eg drop the '"~LENGTH.stringof~"' part to get a struct-of-slices, which is what your example above shows).
>
> artur



Thanks to Artur, John Colvin, and Babu for the very helpful responses.  This should be up on the wiki, and ideally someone should write a little tutorial text talking the reader through how this works and why choices were made a certain way.  I will stick these up on the wiki with attribution if there are no objections.  Something is better than perfect in this situation.

There is a big gap between understanding what language terms do and knowing in practice how to use them for metaprogramming if you don't previously come from a C++ background.  I originally learnt C with parameters in K&R style, and the world has changed quite a lot since those days - both the relative slowness of memory, and CTFE/metaprogramming are quite new concepts to me (Forth doesn't count).  It's very easy to be quickly productive in D for most things (I found it easier than Python, which I learnt previously) but there is a bigger leap in learning templates+CTFE - not because it's intrinsically hard, but it's a different way of thinking that is unfamiliar.  Probably I should read more source code - the harder way is often easier in the longer term.

In case it's of interest my own reason for wanting to do this transformation was not cache efficiency, but just to be able to easily get the data into a form I can access from PydMagic and PyD, since for some chart applications it's easier to have the data in this form, and I would rather do all transformations in D and stick to Python only for pure charting.

I appreciate the thoughtful responses.


Laeeth.

Posted short write-up here.  Please make it better...

http://wiki.dlang.org/Transforming_slice_of_structs_into_struct_of_slices

On Sunday, 5 July 2015 at 00:07:59 UTC, Laeeth Isharc wrote:
> Posted short write-up here.  Please make it better...
>
> http://wiki.dlang.org/Transforming_slice_of_structs_into_struct_of_slices


In John Colvin's solution, should

    alias TransformMembers(alias TypeTransform, alias NameTransform, T) = Tuple!(
        RoundRobin!(Pack!(staticMap!(TypeTransform, FieldTypeTuple!PriceBar)),
                    Pack!(staticMap!(NameTransform, FieldNameTuple!PriceBar))));

read

    alias TransformMembers(alias TypeTransform, alias NameTransform, T) = Tuple!(
        RoundRobin!(Pack!(staticMap!(TypeTransform, FieldTypeTuple!T)),
                    Pack!(staticMap!(NameTransform, FieldNameTuple!T))));

???

On Monday, 6 July 2015 at 17:35:22 UTC, Vlad Levenfeld wrote:
> On Sunday, 5 July 2015 at 00:07:59 UTC, Laeeth Isharc wrote:
>> Posted short write-up here.  Please make it better...
>>
>> http://wiki.dlang.org/Transforming_slice_of_structs_into_struct_of_slices
>
>
> In John Colvin's solution, should
>
>     alias TransformMembers(alias TypeTransform, alias NameTransform, T) = Tuple!(
>         RoundRobin!(Pack!(staticMap!(TypeTransform, FieldTypeTuple!PriceBar)),
>                     Pack!(staticMap!(NameTransform, FieldNameTuple!PriceBar))));
>
> read
>
>     alias TransformMembers(alias TypeTransform, alias NameTransform, T) = Tuple!(
>         RoundRobin!(Pack!(staticMap!(TypeTransform, FieldTypeTuple!T)),
>                     Pack!(staticMap!(NameTransform, FieldNameTuple!T))));
>
> ???

Yikes, yes, of course!