On Monday, 14 September 2020 at 19:31:04 UTC, Steven Schveighoffer wrote:
> On Monday, 14 September 2020 at 18:30:16 UTC, mw wrote:
>> On Monday, 14 September 2020 at 18:20:02 UTC, Steven Schveighoffer wrote:
>>>    // this would be easier with string interpolation!
>>> , D has far superior string manipulation capabilities.
>>
>> Basically, your solution is: let's just use strings! (forget about of all the other fancy mechanism I've listed in point (1)).
>>  Yes, that's right: source code should just be strings.
>>
>> From that point of view, C-preprocessor is conceptually at a high level.
>
> This is hilariously backwards. The c preprocessor is literally just pasting strings around.

When we manually type code, we are literally just pasting strings around.

C preprocessor automates some parts of that job.

... and sometimes can even do very fancy things:

"this entry _is_ a Turing machine implemented in the preprocessor"

http://www.ioccc.org/2001/herrmann1.hint

On Mon, Sep 14, 2020 at 07:42:58PM +0000, mw via Digitalmars-d wrote: [...]
> "this entry _is_ a Turing machine implemented in the preprocessor"
> 
> http://www.ioccc.org/2001/herrmann1.hint

Yawn.  Lambda calculus also implements a Turing machine.  So does BF. So what?  I wouldn't wish to write *anything* in lambda calculus.  Or BF.  Lots of things are Turing-complete; that says nothing about their usefulness.  (Though if you wish to write your program in Lambda calculus or BF, I have no objections.  Go right ahead while I stand over here and cheer you on -- at the distance of a 10-foot pole.  :-P)

Besides, 2001/herrmann1 is too verbose.  A much less painful version of Turing completeness is this one:

	https://www.ioccc.org/1988/spinellis.c

:-D

(Or, if you wish, `cat | dmd -run -`, which doesn't even need you to
create an input file. :-D)


T

-- 
I think Debian's doing something wrong, `apt-get install pesticide', doesn't seem to remove the bugs on my system! -- Mike Dresser

On 9/14/20 3:42 PM, mw wrote:
> On Monday, 14 September 2020 at 19:31:04 UTC, Steven Schveighoffer wrote:
>> On Monday, 14 September 2020 at 18:30:16 UTC, mw wrote:
>>> On Monday, 14 September 2020 at 18:20:02 UTC, Steven Schveighoffer wrote:
>>>>    // this would be easier with string interpolation!
>>>> , D has far superior string manipulation capabilities.
>>>
>>> Basically, your solution is: let's just use strings! (forget about of all the other fancy mechanism I've listed in point (1)).
>>>  Yes, that's right: source code should just be strings.
>>>
>>> From that point of view, C-preprocessor is conceptually at a high level.
>>
>> This is hilariously backwards. The c preprocessor is literally just pasting strings around.
> 
> When we manually type code, we are literally just pasting strings around.
> 
> C preprocessor automates some parts of that job.
> 
> .... and sometimes can even do very fancy things:

"Well the answer is simply that the preprocessor is _not_ Turing complete,..."

> 
> "[But] this entry _is_ a Turing machine implemented in the preprocessor"

"so of course, it has to be preprocessed several times"

D is actually Turing complete. And you only need to run it once.

I appreciate your enthusiasm and experience report. But my time is limited, so I have to respectfully bow out of this conversation. Good luck with your programming journey!

-Steve

On Monday, 14 September 2020 at 20:03:09 UTC, H. S. Teoh wrote:
> does BF. So what?  I wouldn't wish to write *anything* in lambda calculus.  Or BF.  Lots of things are Turing-complete; that says nothing about their usefulness.  (Though if you wish

The point I want to make is: all the supposed high level fancy D mechanisms are too complex to use (or may not even made to work at all, I'm not sure).

The safe bet so far is: go back to raw string manipulation, and *keep* the generated code, to be read by human (for inspection) and compiled by the compiler in the 2nd pass.

As to the tool for the such string manipulation: it can be:
-- C preprocessor,
-- Python script, or
-- D string interpolation (can I save the generated code?).

On Monday, 14 September 2020 at 20:18:32 UTC, Steven Schveighoffer wrote:
> I appreciate your enthusiasm and experience report. But my time is limited, so I have to respectfully bow out of this conversation. Good luck with your programming journey!

Thank you for show me that go back to string manipulation is more straight-forward in D than those supposed high level fancy stuff.

On Mon, Sep 14, 2020 at 08:25:36PM +0000, mw via Digitalmars-d wrote: [...]
> The point I want to make is: all the supposed high level fancy D mechanisms are too complex to use (or may not even made to work at all, I'm not sure).
> 
> The safe bet so far is: go back to raw string manipulation, and *keep* the generated code, to be read by human (for inspection) and compiled by the compiler in the 2nd pass.
[...]

It makes me that feel you're not using the right tool for the right job. What exactly are your requirements, and what exactly are you trying to accomplish?  If you're thinking of your "meta-programming" in terms of string manipulations, then what's the point of trying to redress it with "high level fancy D mechanisms"?  Just generate the string and mixin. Job done.

Your original post in D.learn consisted of a bunch of exercises with no clear explanation of what the goal is.  You mentioned that you're trying to wrap a library, but I didn't see any concrete list of C functions that you're trying to interface with, and your Github links are dead. Instead of assuming a certain way of doing things and complaining when it doesn't do what you thought it should do, I wish you'd post the list of C functions you're trying to wrap, and then we can discuss what are the options for wrapping them, instead of this finger-pointing exercise that, fun as it may be, leads to no constructive result.


T

-- 
An elephant: A mouse built to government specifications. -- Robert Heinlein

On Monday, 14 September 2020 at 22:20:15 UTC, H. S. Teoh wrote:
> with, and your Github links are dead. Instead of assuming a

Sorry, I changed the dir structure, but the git repo root always stay the same:

https://github.com/mingwugmail/talibd


Maybe you didn't follow the discussion very closely, but I think both Paul and Steven knows what I'm trying to do.


Never mind, let's start from fresh again, here is the *challenge*:

in this generated file:

https://github.com/mingwugmail/talibd/blob/master/source/talibd/talib_func.d

Notice the similarity, but different parameters of these 3 functions:

bool TA_MA(double[] inData , double[] outMA , int MA_optInTimePeriod=default_MA_optInTimePeriod, TA_MAType optInMAType=default_optInMAType) { ... }

bool TA_RSI(double[] inData , double[] outRSI , int RSI_optInTimePeriod=default_RSI_optInTimePeriod) { ... }

bool TA_MACD(double[] inData , double[] outMACD, double[] outMACDSignal, double[] outMACDHist , int optInFastPeriod=default_optInFastPeriod, int optInSlowPeriod=default_optInSlowPeriod, int optInSignalPeriod=default_optInSignalPeriod) { ... }

https://github.com/mingwugmail/talibd/blob/master/source/talibd/talib_func.d#L51
https://github.com/mingwugmail/talibd/blob/master/source/talibd/talib_func.d#L79
https://github.com/mingwugmail/talibd/blob/master/source/talibd/talib_func.d#L104


These are the target functions we want to generate.

Now the goal: write a single function template to generate these 3 functions in D, better not to use the very raw string interpolation :-)

On Mon, Sep 14, 2020 at 11:17:09PM +0000, mw via Digitalmars-d wrote: [...]
> https://github.com/mingwugmail/talibd

Yes, I managed to find this link.


[...]
> bool TA_MA(double[] inData , double[] outMA , int MA_optInTimePeriod=default_MA_optInTimePeriod, TA_MAType optInMAType=default_optInMAType) { ... }
> 
> bool TA_RSI(double[] inData , double[] outRSI , int RSI_optInTimePeriod=default_RSI_optInTimePeriod) { ... }
> 
> bool TA_MACD(double[] inData , double[] outMACD, double[] outMACDSignal, double[] outMACDHist , int optInFastPeriod=default_optInFastPeriod, int optInSlowPeriod=default_optInSlowPeriod, int optInSignalPeriod=default_optInSignalPeriod) { ... }
[...]
> These are the target functions we want to generate.

If you already have these declarations, what's the purpose of "generating" them?  Just curious if there is a reason for this, or this is just some arbitrary decision?


> Now the goal: write a single function template to generate these 3 functions in D, better not to use the very raw string interpolation :-)
[...]

Honestly, if what you want is to generate separate function declarations, then why *not* just use string interpolation?  That's essentially what you're trying to do, after all: you have a list of function names, and some scheme for generating the parameter lists.  So, generate the parameter list as a string, then mixin.  Mission accomplished.  I don't understand this phobia of string mixins -- they were included in D precisely for this sort of use case!

Or is the point to unify everything into a single function call? If so, based on the description on this page:

	https://www.ta-lib.org/d_api/d_api.html#Output%20Size

which describes a specific parameter structure with inter-relations between them, I'd say, the way I'd do it is to slurp the entire argument list into a single variadic parameter, and handle the breakdown of the parameter groups manually (beats trying to coax the compiler to do something it isn't expecting to be doing).

Furthermore, based on the description in the above-linked page, there are double/float overloads you wish to support, which can be handled by another compile-time parameter.

So I'd start with something like this:

	bool TA(string funcName, Float, Args...)(Args args)
		if (is(Float == float) || is(Float == double))
	{
		// to be filled in, see below
		...
	}

You'd call it something like this:

	auto result = TA!"TA_MA"( /* arguments go here */ );

Slurping the entire argument list as a general blob of arguments gives us the flexibility to do whatever we wish with it, like optional parameters in the middle of the list, relationships between the numbers of subgroups of parameters, and what-not.  Sounds like what we have here, so that's how I'd do it.

So now, what do with do with the arguments we received? According to the above linked page, we have a list of start/end indices, which I assume should come in pairs? (Correct me if I'm wrong.)  At first I thought the number of pairs must match the number of input arrays, but apparently I'm wrong, based on your examples above?  Anyway, for now I'm just going to assume the number of pairs must match the number of inputs.  So then it would look something like this:

	bool TA(string funcName, Float, Args...)(Args args)
		if (is(Float == float) || is(Float == double))
	{
		enum nPairs = NumPairs!Args;
		enum arraysStart = 2*nPairs;

		// This part is optional, it's just to type-check that
		// the caller passed the right number of arguments
		static foreach (i; 0 .. nArrays)
		{
			static assert(is(Args[arraysStart + i] == Float[]),
				/* Optional: add nice error message */);
		}

		// To get at the array contents: write:
		//	args[arraysStart + i]
		// where i is the index.

		... // more to follow
	}

	// Helper template
	template NumPairs(Args...) {
		static if (Args.length <= 1)
			enum NumPairs = 0;
		else static if (!is(Args[0] == int) || !is(Args[0] == int))
			enum NumPairs = 0;
		else
			enum NumPairs = 2 + NumPairs(Args[2..$]);
	}

Next, you have a bunch of optional parameters. I'm not sure exactly what determines what these parameters are or how, but let's assume there's some template that, given the name of the target function, tells us (1) the names of these parameters and (2) their types.  So we have:

	bool TA(string funcName, Float, Args...)(Args args)
		if (is(Float == float) || is(Float == double))
	{
		enum nPairs = NumPairs!Args;
		enum nArrays = nPairs;
		enum arraysStart = 2*nPairs;

		// This part is optional, it's just to type-check that
		// the caller passed the right number of arguments
		static foreach (i; 0 .. nArrays)
		{
			static assert(is(Args[arraysStart + i] == Float[]),
				/* Optional: add nice error message */);
		}

		// To get at the array contents: write:
		//	args[arraysStart + i]
		// where i is the index.

		enum optionsStart = arraysStart + nArrays;
		enum nOptions = NumberOfOptions!funcName;

		// Optional: type check
		alias OptTypes = OptionTypes!funcName;
		static foreach (i, T; OptTypes)
		{
			static assert(is(Args[optionsStart + i] == OptTypes[i]),
				/* Optional: nice error message here */);
		}

		// To get at the options, write:
		//	args[optionsStart + i]
		// where i is the index.

		... // more to follow
	}

Next, we have two fixed output parameters, which follow the optional parameters. So something like this:

	bool TA(string funcName, Float, Args...)(Args args)
		if (is(Float == float) || is(Float == double))
	{
		enum nPairs = NumPairs!Args;
		enum nArrays = nPairs;
		enum arraysStart = 2*nPairs;

		// This part is optional, it's just to type-check that
		// the caller passed the right number of arguments
		static foreach (i; 0 .. nArrays)
		{
			static assert(is(Args[arraysStart + i] == Float[]),
				/* Optional: add nice error message */);
		}

		// To get at the array contents: write:
		//	args[arraysStart + i]
		// where i is the index.

		enum optionsStart = arraysStart + nArrays;
		enum nOptions = NumberOfOptions!funcName;

		// Optional: type check
		alias OptTypes = OptionTypes!funcName;
		static foreach (i, T; OptTypes)
		{
			static assert(is(Args[optionsStart + i] == OptTypes[i]),
				/* Optional: nice error message here */);
		}

		// To get at the options, write:
		//	args[optionsStart + i]
		// where i is the index.

		// Output parameters
		enum outStart = optionsStart + nOptions;
		int* outBegIdx = args[outStart];
		int* outNbElement = args[outStart + 1];

		... // more to follow
	}

Finally, we have the list of output arrays, which presumably must match the number of input arrays. So:

	bool TA(string funcName, Float, Args...)(Args args)
		if (is(Float == float) || is(Float == double))
	{
		enum nPairs = NumPairs!Args;
		enum nArrays = nPairs;
		enum arraysStart = 2*nPairs;

		// This part is optional, it's just to type-check that
		// the caller passed the right number of arguments
		static foreach (i; 0 .. nArrays)
		{
			static assert(is(Args[arraysStart + i] == Float[]),
				/* Optional: add nice error message */);
		}

		// To get at the array contents: write:
		//	args[arraysStart + i]
		// where i is the index.

		enum optionsStart = arraysStart + nArrays;
		enum nOptions = NumberOfOptions!funcName;

		// Optional: type check
		alias OptTypes = OptionTypes!funcName;
		static foreach (i, T; OptTypes)
		{
			static assert(is(Args[optionsStart + i] == OptTypes[i]),
				/* Optional: nice error message here */);
		}

		// To get at the options, write:
		//	args[optionsStart + i]
		// where i is the index.

		// Output parameters
		enum outStart = optionsStart + nOptions;
		int* outBegIdx = args[outStart];
		int* outNbElement = args[outStart + 1];

		// Output arrays
		enum outArraysStart = outStart + 2;

		// Optional type check
		static foreach (i; 0 .. nArrays)
		{
			static assert(is(Args[outArraysStart + i] == Float[]),
				/* Optional: add nice error message */);
		}

		// To get at the output arrays, write:
		//	args[outArraysStart + i]
		// where i is the index.

		... // more to follow
	}

Now we're ready to actually implement the function.  Presumably we're not just handing a bunch of stuff over to a C function -- because for that, a string mixin would've sufficed instead of this entire charade. The whole point of even doing any of the above is so that you can write D code that takes advantage of type information in Args and generic iteration over (subsets of) it.

Since I've no idea what these functions are supposed to be doing, I'm just going to assume there's some template function called Impl that, given some one set of start/end indices, an input array, and an output array, will do whatever it is that needs to be done. So:

	bool TA(string funcName, Float, Args...)(Args args)
		if (is(Float == float) || is(Float == double))
	{
		enum nPairs = NumPairs!Args;
		enum nArrays = nPairs;
		enum arraysStart = 2*nPairs;

		// This part is optional, it's just to type-check that
		// the caller passed the right number of arguments
		static foreach (i; 0 .. nArrays)
		{
			static assert(is(Args[arraysStart + i] == Float[]),
				/* Optional: add nice error message */);
		}

		// To get at the array contents: write:
		//	args[arraysStart + i]
		// where i is the index.

		enum optionsStart = arraysStart + nArrays;
		enum nOptions = NumberOfOptions!funcName;

		// Optional: type check
		alias OptTypes = OptionTypes!funcName;
		static foreach (i, T; OptTypes)
		{
			static assert(is(Args[optionsStart + i] == OptTypes[i]),
				/* Optional: nice error message here */);
		}

		// To get at the options, write:
		//	args[optionsStart + i]
		// where i is the index.

		// Output parameters
		enum outStart = optionsStart + nOptions;
		int* outBegIdx = args[outStart];
		int* outNbElement = args[outStart + 1];

		// Output arrays
		enum outArraysStart = outStart + 2;

		// Optional type check
		static foreach (i; 0 .. nArrays)
		{
			static assert(is(Args[outArraysStart + i] == Float[]),
				/* Optional: add nice error message */);
		}

		// To get at the output arrays, write:
		//	args[outArraysStart + i]
		// where i is the index.

		// Finally, do the actual computation:
		bool result;
		foreach (i; 0 .. nArrays)
		{
			Impl!funcName(
				args[2*i .. 2*i+2],	// start/end indices
				args[arraysStart + i],	// input array
				args[optionsStart .. optionsStart + nOptions], // optional arguments
				outBegIdx, outNbElement,
				args[outArraysStart + i], // output arrays
			);

			result = ...; // do whatever's needed here
		}

		return result;
	}

The implementation of `Impl` is left as an exercise for the reader. ;-) So there we have it.

Now, if this whole charade is *really* just to call some C functions in the most pointlessly convoluted way, then you'd just replace the last part of this function with something like this:

	bool TA(string funcName, Float, Args...)(Args args)
		if (is(Float == float) || is(Float == double))
	{
		...

		// See? I told you, you should have just used a string
		// mixin from the get-go instead of this pointless
		// charade.  Of course, the charade does win you the
		// ability to type-check these things, but so does
		// mixing in the generated string declarations.
		return mixin(funcName ~ "(" ~
			"args[2*i .. 2*i+2], "~
			"args[arraysStart + i],	"~
			"args[optionsStart .. optionsStart + nOptions], "~
			"outBegIdx, outNbElement, "~
			"args[outArraysStart + i]);");
	}

Well OK, I omitted something, that is, mapping the D arrays to C pointers.  You can just map them to their respective .ptr values with staticMap if you wish.  Or just:

	iota(0, nArrays).map!(i => format("&args[arraysStart + %d].ptr", i)).joiner.array

Which again, shows that the whole thing is kinda pointless if all you want to do with it at the end is to call some C function.  The metaprogramming machinery is really for when you want to write actual D code where you can access the args[] array at compile-time and do useful stuff with it.  After all, this is *D* metaprogramming we're talking about; the code needs to be in D in order to benefit from it.  If all you want is to generate a bunch of C declarations, the C processor does its job very well, and so do string mixins.

Use the right tool for the right job.


T

-- 
Why did the mathematician reinvent the square wheel?  Because he wanted to drive smoothly over an inverted catenary road.

On Tuesday, 15 September 2020 at 00:41:12 UTC, H. S. Teoh wrote:
>
> If you already have these declarations, what's the purpose of "generating" them?  Just curious if there is a reason for this, or this is just some arbitrary decision?

The is the same usage pattern to call hundreds of such TA_xxx functions, i.e.

-- check the input params
-- calculate TA_xxx_lookback
-- call the raw C func
-- zero out the lookback area
-- return status


> I don't understand this phobia of string mixins -- they were included in D

I'm not phobia to string mixins; I'm saying that because Steven provide one such solution already (see summary 3 below), and pure string mixins solution is no different from C's macros solution (or use a Python script to output strings as generated D code).

> I'd say, the way I'd do it is to slurp the entire argument list into a single variadic parameter, and handle the breakdown of the parameter groups manually

I saw you method.

That's what I want to avoid, as the discussion with Paul earlier, because ...

> So now, what do with do with the arguments we received? According to the above linked page, we have a list of start/end indices, which I assume should come in pairs? (Correct me if I'm wrong.)  At first I thought the number of pairs must match the number of input arrays, but apparently I'm wrong, based on your examples above?

to use a single variadic parameter and *then* break them down, you need extra assumptions on the different groups of actual parameters.

(1) we do NOT have so much assumptions on these groups of params, neither on each individual type, or the total number of them in each of the groups.

(2) even if we do have some assumptions, this kind of breakdown (a kind of simple parsing) code is fragile to maintain.

That's why I also asked Paul, is there any D Marker type I can use: i.e. split the single variadic parameter based on some Marker type.


> 	bool TA(string funcName, Float, Args...)(Args args)
> 		if (is(Float == float) || is(Float == double))
> 	{
> 		enum nPairs = NumPairs!Args;
> 		enum arraysStart = 2*nPairs;

As said, we don't have such assumptions to calculate the arraysStart index; let's just assume we have some other ways to calculate, this breakdown code is going to be more complex than you are showing in your solution.


So far, I've got 4 solutions:

1) my C-macros based solution,
https://github.com/mingwugmail/talibd/blob/master/source/talibd/talib_func.h#L139

#define MA_INS  int MA_optInTimePeriod, TA_MAType optInMAType
#define MA_OUTS outMA
DECL_TA_FUNC(TA_MA, MA_INS, MA_OUTS, (MA_optInTimePeriod-1))

Note: the input parameters are *logically* grouped, no need the breakdown parsing logic.


2) Paul's solution based on the same logical grouping idea as mine, but use D-template:
https://forum.dlang.org/post/nrziyhvdgkqvarlccmmf@forum.dlang.org

which I quoted to started this thread in general.


3) Steven's solution based on D string mixin
https://forum.dlang.org/post/rjoc7h$234g$1@digitalmars.com


4) your solution based on breakdown single variadic parameter.
https://forum.dlang.org/post/mailman.5794.1600130478.31109.digitalmars-d@puremagic.com


I'm wondering if Paul, Steven, and you have time each to summit a proper working PR, and we ask the forum users to vote which one is most readable, and maintainable; and which one is most fragile? :-)

On Tue, Sep 15, 2020 at 01:44:25AM +0000, mw via Digitalmars-d wrote:
> On Tuesday, 15 September 2020 at 00:41:12 UTC, H. S. Teoh wrote:
> > 
> > If you already have these declarations, what's the purpose of "generating" them?  Just curious if there is a reason for this, or this is just some arbitrary decision?
> 
> The is the same usage pattern to call hundreds of such TA_xxx functions, i.e.
> 
> -- check the input params
> -- calculate TA_xxx_lookback
> -- call the raw C func
> -- zero out the lookback area
> -- return status

Sounds like the code example I wrote would work.  It's not *that* complicated, really, it's just grouping the input arguments based on some criterion.

If that's not desirable to you for whatever reason, then just write a string function to generate the code and mix it in.  Or better yet -- and this is actually my preferred approach if I were to do hundreds of such TA_xxx functions -- write a D program that parses some input file (perhaps even the C header itself, if it follows a simple to parse format) and generates a .d file.

As I said, the heavy metaprogramming machinery in D is really intended more for D code that would benefit from compile-time type information, manipulation of type lists, and CTFE computations.  It's not really intended to generate arbitrary token strings.  Past a certain point, IMO it's no longer worth the trouble to try to do it all in one compile pass; just write a helper program to parse the input data and generate D code into a .d file, and import that.  You'll have the benefit of faster compilation (only need to generate the .d file once, assuming the C library isn't constantly changing), and being able to read the actual generated code in case something goes wrong and you need to debug it. Debugging a deeply-nested mixin string buried deep in multiple layers of templates is Not Fun(tm), and unless you have some special needs that require that, I don't recommend it.  It's also faster for a D program to read data and do whatever transformations you fancy on the string representation, than to do all of that in CTFE/templates at compile-time anyway.  Plus, you get to even unittest your generation code to ensure there are no regressions -- which may be a LOT harder to do deep inside nested templates and mixins.

In my own projects, I often write little helper programs to do exactly this sort of transformation.  In fact, in one project I even used the C preprocessor to transform a C header file into D code. :-D (It had a specific format with specific macros that can be redefined to emit D code instead of C -- and it so happened to be exactly what I needed.) With a proper build system, none of this is anything special, just a routine day's work. (Off-topic rant: this is why I can't abide dub. It wants me to jump through hoops to do something I can do in 3 lines with SCons. No thanks.)  Like I said, use the right tools for the right job. If heavy metaprogramming machinery isn't cutting it for you, then use a string function to generate code. Or use the C preprocessor. Whatever gets your job done with a minimum of fuss.


[...]
> I'm wondering if Paul, Steven, and you have time each to summit a proper working PR, and we ask the forum users to vote which one is most readable, and maintainable; and which one is most fragile? :-)

Sorry, I have very limited free time, and I've already spent enough time to elaborate on a solution that, really, ought to be obvious to anyone who has spent some effort to learn and use D's templating system.  I'm really not inclined to spend any more time on this, sorry.  But now you have at least 4 different approaches to weigh to find one that works best for you.  Try them out and see how they work out in practice, and let us know about it.  Who knows, maybe you might even come across something worthy of a DIP that will improve the state of D's metaprogramming capabilities. ;-)


T

-- 
I am a consultant. My job is to make your job redundant. -- Mr Tom