Still trying to get the com automation code working. This is a general issue with COM programming as I do not have the experience to solve the problem.

There are two files at:

http://s000.tinyupload.com/index.php?file_id=33201858563271816000

Gen.d is the automatically generated COM wrapper in D for photoshop.

main.d is the test code.

You'll need windows and photoshop to make use of the code.

The idea is simple. Set the RGBColor foreground color using one of the COM methods in cSolidColor. We can set the color using Red, Green, or Blue and doubles and it works because one can pass a simple type without any marshaling.

But if one wants to set all three using the RGBColor inteface, it requires more work; and this is where I'm stuck at.

What I have done is create a simple router that brings all the work in to main.d.

The first method is `void RGB(icRGBColor, cSolidColor)` and it simply attempts to invoke PS's SolidColor's RGB method to have it set the colors.

When we call this method, it passes our interface that we want photoshop to use as to get at the colors and set them.

The second method(s) is basically in icRGBColor. This class is suppose to represent the callback interface. Photoshop is suppose to call these methods to get the colors and such.


The idea is that we pass our icRGBColor instance to invoke which informs PS that we want to set the color using the interface(rather than say a double). PS then calls the appropriate methods on the interface to do what it needs to do(e.g., Red() to get the red value to set).

At least I believe that is how it goes.

When we call the invoke function eventually the QueryInterface is called, which is good because this means that PS(or COM) must have called it and this is probably what it is suppose to do.

the problem is, that the code crashes and I can't get it past that obstacle. What I'd like to do is get to the point where we can get PS to set the colors using that interface(icRGBColor). Right now I just have dummy returns and have it print to the console when the method is called, which is good enough for getting the kinks worked out.

Anyone that basically has more of a clue than I do mind giving it a look over and see if they can figure out how to get it to work. It's relatively simple. Only main.d really needs to be messed with and that's just about 50 lines of code(and only a few are actually relevant(icRGBColor.QueryInterface, RGB() and a few of the lines at the end of main).

Once this is figured out I should have a near complete automation of COM in D. This allows one to simply run the tool on the .idl file and it will generate a D file with all the machinery to use the COM back end.



Thanks.

On Monday, 24 April 2017 at 00:55:45 UTC, Nierjerson wrote:
> Still trying to get the com automation code working. This is a

Please, use ZIP for archive.

On Monday, 24 April 2017 at 17:31:09 UTC, MGW wrote:
> On Monday, 24 April 2017 at 00:55:45 UTC, Nierjerson wrote:
>> Still trying to get the com automation code working. This is a
>
> Please, use ZIP for archive.

http://s000.tinyupload.com/index.php?file_id=67286353487198133918

On Monday, 24 April 2017 at 00:55:45 UTC, Nierjerson wrote:
> Still trying to get the com automation code working. This is a general issue with COM programming as I do not have the experience to solve the problem.
>
> [...]

I tried looking at this because I just did some COM work even if most of it in C++. There's a _lot_ of code and I was instantly lost, even with your explanations.

I'd forget about Photoshop for now, just write a simple COM client. That'd also ease in getting other people to help because having Photoshop is a high barrier for entry.

Can you get this to work in C++?

On Tuesday, 25 April 2017 at 10:03:30 UTC, Atila Neves wrote:
> On Monday, 24 April 2017 at 00:55:45 UTC, Nierjerson wrote:
>> Still trying to get the com automation code working. This is a general issue with COM programming as I do not have the experience to solve the problem.
>>
>> [...]
>
> I tried looking at this because I just did some COM work even if most of it in C++. There's a _lot_ of code and I was instantly lost, even with your explanations.

There isn't a lot of code! All the code in Gen is irrelevant except for a few functions... even then, you don't ever really need to mess with it except to see a few of the definitions.

99.9% of the code is irrelevant, I could have removed most of it and it would function the same. The reason why I left it in is for completeness and because many of the different classes reference other classes so having one sorta requires having all the rest(even though they are never used, it is to avoid missing symbols, etc, which can be fixed but would require some editing and would prevent the code from working).

Only three classes are used is used in Gen.d: cApplication, cSolidColor, and cRGBColor.

One only really needs to look at main.d. I've put all the code necessary in that file. One only needs to look at Gen.d to get the general idea and at that, only those specific classes.




> I'd forget about Photoshop for now, just write a simple COM client. That'd also ease in getting other people to help because having Photoshop is a high barrier for entry.

Photoshop is the main app I'm trying to get to work so this is appropriate for me and it is also the only real way I have to test COM.



> Can you get this to work in C++?

The code generation probably could be tailored to work in C++ without too much issue. I'm not going to do it though! Too much work as it is and I first need to get this to work. I don't see any real difference between D and C++(if I'm going to use C++ for anything I'm going to use D instead). I've already had to rewrite the engine because D's CTFE could not handle the load. So this has turned out to be quite a bit of work.


---

The main issue has nothing to do with PS though(I imagine it is a general COM programming task).

The server(PS in this case) gives me an interface that has functions like RGB(cRGBColor rgb); (Actually a _RGBColor com interface which I wrap with a cRGBColor interface to provide the ability to handle the marshaling and such behind the scenes),


So I pass it a cRGBcolor interface, which actually can't be used because photoshop has no clue about that type.

Instead, I construct a dummy interface that inherits from IDispatch which I'll pass to server instead. I'll use it as a wrapper to hook up everything I need.

The problem is, when I do this, the app crashes. I do not know why. What I do know is that it seems to be calling QueryInterface and so it seems like it might actually be getting to the server and the server is calling the QueryInterface method. If that is the case, then the passing of the interface is working but something else is failing.

I'm just not sure though as I don't know enough about COM to determine what exactly is going on(after all, it's all happening on the server "behind closed doors").



....

	auto rgb2 = new icRGBColor();
	rgb2.iDispatch = rgb2;
	//auto rgb = cast(icRGBColor)&rgb2;
	//dd.RGB(cast(cRGBColor)(cast(void*)rgb));
	//dd.RGB(cast(icRGBColor)(cast(void*)rgb));


	cSolidColor.RGBSet = &RGB;
	auto rgb = new icRGBColor();
	//auto rgb = cast(icRGBColor)(cast(void*)dd.RGB());
	dd.RGB(rgb); // <--- This is where the work begins! Tries to get photoshop to set the RGB color through the rgb object


    return 0;
}

// This is the invoker.Only useful for debugging(brought out from Gen.d here for convenience
// The invoke line is what is important. When it is called, photoshop will be given the icRGBColor object and will do things with it(if it works, that is... seems to and seems to call QueryInterface but doesn't get any further).
void RGB(icRGBColor ic, cSolidColor s)
{
	import main;
	EXCEPINFO exception;
	uint argErr = 0;
	auto iidNULL = IID_NULL;
	auto RT = new SafeVariantPtr();
	VARIANT[1] paramVars;
	DISPPARAMS params = {rgvarg: paramVars.ptr, cArgs: 1, cNamedArgs: 0};
	auto ID = s.COMMethodIDs[`RGB`];
	paramVars[0].punkVal = ic; paramVars[0].vt = VARENUM.VT_DISPATCH; scope(exit) VariantClear(&paramVars[0]);
	auto res = s.iDispatch.Invoke(cast(int)ID, &iidNULL, 0, DISPATCH_PROPERTYPUT, &params, cast(VARIANT*)RT, &exception, &argErr);
	assert(res == S_OK, `Could not invoke COM Function cSolidColor.RGB. Error `~to!string(res, 16));

}



...

The icRGBColor interface that follows is a simple callback interface. All the methods simply print their function name when called, this helps to know when they are called.

When one runs the program they should see a few calls to QueryInterface. Either photoshop is calling this method on purpose or by accident.(calling something else but the vtable is wrong)


Hope that helps some. I only need some advice, guidance on how things are suppose to work so I don't wait a lot of time going down a blind alley.

On Tuesday, 25 April 2017 at 18:39:56 UTC, Nierjerson wrote:
> void RGB(icRGBColor ic, cSolidColor s)
> {
> 	import main;
> 	EXCEPINFO exception;
> 	uint argErr = 0;
> 	auto iidNULL = IID_NULL;
> 	auto RT = new SafeVariantPtr();
> 	VARIANT[1] paramVars;
> 	DISPPARAMS params = {rgvarg: paramVars.ptr, cArgs: 1, cNamedArgs: 0};
> 	auto ID = s.COMMethodIDs[`RGB`];
> 	paramVars[0].punkVal = ic; paramVars[0].vt = VARENUM.VT_DISPATCH; scope(exit) VariantClear(&paramVars[0]);
> 	auto res = s.iDispatch.Invoke(cast(int)ID, &iidNULL, 0, DISPATCH_PROPERTYPUT, &params, cast(VARIANT*)RT, &exception, &argErr);
> 	assert(res == S_OK, `Could not invoke COM Function cSolidColor.RGB. Error `~to!string(res, 16));
>
> }

When you use DISPATCH_PROPERTYPUT you need to set cNamedArgs and rgdispidNamedArgs like so:

  int dispidNamed = DISPID_PROPERTYPUT;
  params.cNamedArgs = 1;
  params.rgdispidNamedArgs = &dispidNamed;

You should also call AddRef on any COM objects you add to your paramVars array.

On Wednesday, 26 April 2017 at 15:30:37 UTC, John Chapman wrote:
> On Tuesday, 25 April 2017 at 18:39:56 UTC, Nierjerson wrote:
>> [...]
>
> When you use DISPATCH_PROPERTYPUT you need to set cNamedArgs and rgdispidNamedArgs like so:
>
>   int dispidNamed = DISPID_PROPERTYPUT;
>   params.cNamedArgs = 1;
>   params.rgdispidNamedArgs = &dispidNamed;
>
> You should also call AddRef on any COM objects you add to your paramVars array.


Did you try this? I tried and same issue. Are you sure the above is required? I'm not using any "named" args so not sure why it should matter?

On Wednesday, 26 April 2017 at 23:04:53 UTC, Nierjerson wrote:
> On Wednesday, 26 April 2017 at 15:30:37 UTC, John Chapman wrote:
>> On Tuesday, 25 April 2017 at 18:39:56 UTC, Nierjerson wrote:
>>> [...]
>>
>> When you use DISPATCH_PROPERTYPUT you need to set cNamedArgs and rgdispidNamedArgs like so:
>>
>>   int dispidNamed = DISPID_PROPERTYPUT;
>>   params.cNamedArgs = 1;
>>   params.rgdispidNamedArgs = &dispidNamed;
>>
>> You should also call AddRef on any COM objects you add to your paramVars array.
>
>
> Did you try this? I tried and same issue. Are you sure the above is required? I'm not using any "named" args so not sure why it should matter?

From the documentation (https://msdn.microsoft.com/en-us/library/windows/desktop/ms221479(v=vs.85).aspx):

"When you use IDispatch::Invoke() with DISPATCH_PROPERTYPUT or DISPATCH_PROPERTYPUTREF, you have to specially initialize the cNamedArgs and rgdispidNamedArgs elements of your DISPPARAMS structure"

Thought it might help.

On Thursday, 27 April 2017 at 07:51:26 UTC, John Chapman wrote:
> On Wednesday, 26 April 2017 at 23:04:53 UTC, Nierjerson wrote:
>> On Wednesday, 26 April 2017 at 15:30:37 UTC, John Chapman wrote:
>>> On Tuesday, 25 April 2017 at 18:39:56 UTC, Nierjerson wrote:
>>>> [...]
>>>
>>> When you use DISPATCH_PROPERTYPUT you need to set cNamedArgs and rgdispidNamedArgs like so:
>>>
>>>   int dispidNamed = DISPID_PROPERTYPUT;
>>>   params.cNamedArgs = 1;
>>>   params.rgdispidNamedArgs = &dispidNamed;
>>>
>>> You should also call AddRef on any COM objects you add to your paramVars array.
>>
>>
>> Did you try this? I tried and same issue. Are you sure the above is required? I'm not using any "named" args so not sure why it should matter?
>
> From the documentation (https://msdn.microsoft.com/en-us/library/windows/desktop/ms221479(v=vs.85).aspx):
>
> "When you use IDispatch::Invoke() with DISPATCH_PROPERTYPUT or DISPATCH_PROPERTYPUTREF, you have to specially initialize the cNamedArgs and rgdispidNamedArgs elements of your DISPPARAMS structure"
>
> Thought it might help.

Ok, thanks. I will try again, Maybe I did something wrong. It seems though that it isn't necessary, at least in my specific test case since the same behavior was seen(although it might have been a future bug).

I think the main issue though, is that I really don't know what is going on when I invoke the PS function. It seems to call the server method that takes the interface and then the server does it's "magic"(which is calling my QueryInterface) but how the implemented QueryInterface is suppose to respond is beyond me... I've tried some based stuff but nothing seem to work. The good news is that it is doing something(calling QueryInterface) which means that the server is at work.

Any more ideas?  I think the issue currently is is the QueryInterface(it is simply not doing what it is suppose to). I'll probably have to look at some other implementations to see what is going on.

On Thursday, 27 April 2017 at 20:20:23 UTC, Nierjerson wrote:
>
> I think the main issue though, is that I really don't know what is going on when I invoke the PS function. It seems to call the server method that takes the interface and then the server does it's "magic"(which is calling my QueryInterface) but how the implemented QueryInterface is suppose to respond is beyond me... I've tried some based stuff but nothing seem to work. The good news is that it is doing something(calling QueryInterface) which means that the server is at work.
>
> Any more ideas?  I think the issue currently is is the QueryInterface(it is simply not doing what it is suppose to). I'll probably have to look at some other implementations to see what is going on.

QueryInterface is COM's version of opCast. It asks if you support the interface represented by an IID (riid). If you don't, then you return E_NOINTERFACE. If you do, then you point the result (pvObject) to yourself and return S_OK. Here's a basic implementation:

extern(Windows)
HRESULT QueryInterface(IID* riid, void** pvObject) {
  if (pvObject is null) return E_POINTER;
  *pvObject = null;

  if (*riid == IID_IUnknown) *pvObject = cast(void*)cast(IUnknown)this;
  else if (*riid == IID_IDispatch) *pvObject = cast(void*)cast(IDispatch)this;
  // and so on for all interfaces we support

  if (*pvObject is null) return E_NOINTERFACE;
  (cast(IUnknown)this).AddRef();
  return S_OK;
}

AddRef/Release perform the COM object's reference counting, so you should implement them too.

However, I don't understand why your icRBCColor class both implements and encapsulates IDispatch - the generated version cRGBColor from Gen.d just encapsulates it. Why do you need to instantiate an instance of icRGBColor? Can't you just use the rgb1 object you got from the dd.RGB() getter, assign the colour values to its Red, Blue, Green properties as needed, then call the dd.RGB(rgb1) setter? Does that not work?