Jump to page: 1 2
Thread overview
D struct with String type accessing from Python
Dec 26, 2022
alchemypy
Dec 26, 2022
cc
Dec 26, 2022
Ali Çehreli
Dec 27, 2022
cc
Dec 27, 2022
Ali Çehreli
Dec 27, 2022
cc
Dec 27, 2022
Ali Çehreli
Dec 28, 2022
Ali Çehreli
Dec 28, 2022
Siarhei Siamashka
Dec 29, 2022
cc
Dec 28, 2022
alchemypy
December 26, 2022

Hi Everyone, I am new to Dlang and i am trying to consume .so (Dlang based dynamic library) from Python.

My D program has something like below,

struct gph
{
	string x;
}

struct myStruct
{
	pragma(mangle, "Print_gph")
	void Print_gph(gph g)
	{
		stderr.writeln("here: gph ");
		stderr.writeln(g.x);
	
	}
}

i have created a .so for the same and my Python code looks as follows:

from ctypes import *

class gph(Structure):
	_fields_ = (
		('x_p', c_char_p),
		('x_len', c_size_t)
	)
	def __init__(self, x):
		self.x_p = x
		self.x_len = len(x)


so_path = "./test.so"
lib = CDLL(so_path)

sample_struct = gph(b'gph')
lib.SubOcc_gph(c_void_p(),sample_struct)

When i executed i got bellow error,

here: gph
src/rt/dwarfeh.d:330: uncaught exception reached top of stack
This might happen if you're missing a top level catch in your fiber or signal handler
std.exception.ErrnoException@/usr/include/dmd/phobos/std/stdio.d(3170): Enforcement failed (Bad address)
Aborted

Can you please help me what went wrong here ?

December 26, 2022

On Monday, 26 December 2022 at 03:05:33 UTC, alchemypy wrote:

>

Can you please help me what went wrong here ?

I tried to do something similar in C# once. Internally, I believe D stores the size of a slice before the pointer.

Not sure if this will help, I abandoned this idea after playing with it for a while. I think I had trouble getting the D compiler to send one of its own strings/slices to an extern(C) function. I believe it worked fine if I sent the string result AS a struct that separated out the size and pointer, but failed for unexplained reasons if I just tried to send the basic string/immutable(char)[] as-is.

// C#
		[StructLayout(LayoutKind.Sequential, Size=16, Pack=1)]
		public struct DString {
			public ulong length;
			public IntPtr ptr;
			public string str {
				get {
					byte[] b = new byte[length];
					for (int i = 0; i < (int)length; i++) {
						b[i] = Marshal.ReadByte(ptr, i);
					}
					return Encoding.UTF8.GetString(b);
				}
			}
		}
		[DllImport("mydll.dll")]
		private static extern void DDLL_testString(out DString dd);
		public static string testString() {
			DString d;
			DDLL_testString(out d);
			return d.str;
		}
// D DLL
static struct DString {
	size_t length;
	immutable(char)* ptr;
	this(string str) {
		length = str.length;
		ptr = str.ptr;
	}
	void opAssign(string str) {
		length = str.length;
		ptr = str.ptr;
	}
}
//extern(C) export DString testString() {
extern(C) export void testString(out string ret) {
	string str = "hello王".idup;
	//return DString(str);
	ret = str;
}
December 26, 2022
On 12/25/22 20:41, cc wrote:
> D stores the size of a slice before the pointer.

And that's in the ABI spec:

  https://dlang.org/spec/abi.html#arrays

Another important topic is object lifetimes because Python and D garbage collectors not necessarily know about objects created by the other side.

In any case, I have a presentation "Exposing a D Library to Python Through a C API", which touches up on these topics:

  https://www.youtube.com/watch?v=FNL-CPX4EuM

(Note: Yes, that was covid hair. :) )

Ali

December 27, 2022

On Monday, 26 December 2022 at 16:44:37 UTC, Ali Çehreli wrote:

>

In any case, I have a presentation "Exposing a D Library to Python Through a C API", which touches up on these topics:

I found the problem I was running into before. Why does returning a struct, or sending a D string via an out parameter, work here, but simply returning a D string does not?

D DLL

static struct DString {
	size_t length;
	immutable(char)* ptr;
	this(string str) {
		length = str.length;
		ptr = str.ptr;
	}
	void opAssign(string str) {
		length = str.length;
		ptr = str.ptr;
	}
}

static this() {
	cache = null; // DLL memory error if not initialized here
}
string[string] cache;

export string MyDLL_testStringReturn() {
	string s = "hello";
	cache.require(s);
	writefln("[D] pass: %s", s);
	return s;
}
export DString MyDLL_testStringReturnStruct() {
	string s = "hello";
	cache.require(s);
	auto dstr = DString(s);
	writefln("[D] pass: %s", dstr);
	return dstr;
}
export void MyDLL_testStringOut(out string str) {
	string s = "hello";
	cache.require(s);
	str = s;
	writefln("[D] pass: %s", str);
}

C# Main

[StructLayout(LayoutKind.Sequential, Size=16, Pack=1)]
public struct DString {
	public ulong length;
	public IntPtr ptr;
	public string str {
		get {
			byte[] b = new byte[length];
			for (int i = 0; i < (int)length; i++) {
				b[i] = Marshal.ReadByte(ptr, i);
			}
			return Encoding.UTF8.GetString(b);
		}
	}
}

[DllImport("mydll.dll")]
private static extern DString MyDLL_testStringReturn();
[DllImport("mydll.dll")]
private static extern DString MyDLL_testStringReturnStruct();
[DllImport("mydll.dll")]
private static extern void MyDLL_testStringOut(out DString dd);

public static void testString() {
	System.Console.WriteLine("Test [RETURN D string]");
	DString d1 = MyDLL_testStringReturn();
	System.Console.WriteLine("ok");
	System.Console.WriteLine("len: {0}", d1.length);
	System.Console.WriteLine("d1: {0}", d1.str);

	System.Console.WriteLine("Test [RETURN D struct]");
	DString d2 = MyDLL_testStringReturnStruct();
	System.Console.WriteLine("ok");
	System.Console.WriteLine("len: {0}", d2.length);
	System.Console.WriteLine("d2: {0}", d2.str);

	System.Console.WriteLine("Test [OUT D string]");
	DString d3;
	MyDLL_testStringOut(out d3);
	System.Console.WriteLine("ok");
	System.Console.WriteLine("len: {0}", d3.length);
	System.Console.WriteLine("d3: {0}", d3.str);
}

Output

Test [RETURN D string]
[D] pass: hello
ok
len: 0         !! Should be 5!
d1:            !! Should be hello!
Test [RETURN D struct]
[D] pass: DString(5, 7FFA497BCDD0)
ok
len: 5
d2: hello
Test [OUT D string]
[D] pass: hello
ok
len: 5
d3: hello
December 27, 2022
On 12/26/22 22:42, cc wrote:

> Why does returning a
> struct, or sending a D string via an out parameter, work here, but
> simply returning a D string does not?

I don't know C# to be sure but it must be ABI related. The way D passes returned objects must be different from what C# expects. (There is no extern(C#) to rely on here. ;) )

> static struct DString {
>      size_t length;

[...]

> [StructLayout(LayoutKind.Sequential, Size=16, Pack=1)]
> public struct DString {
>      public ulong length;

[...]

Apparently that works in your case but D's size_t will be 32 bits on a 32-bit build. I checked: C#'s ulong is always 64 bits.

Ali

December 27, 2022

On Tuesday, 27 December 2022 at 17:04:03 UTC, Ali Çehreli wrote:

>

I don't know C# to be sure but it must be ABI related. The way D passes returned objects must be different from what C# expects. (There is no extern(C#) to rely on here. ;) )

No need for extern(C#), extern(C) should be sufficient. However, how does D pass a D slice as a return value under extern(C)? Does it just return nothing at all? Returning a struct that is the same size as a D slice and memcpy'd from one works.

Here it is under C instead:

C

#include <stdio.h>
#pragma comment(lib, "mydll.lib")
struct DString {
	unsigned __int64 len;
	char *ptr;
};
extern __declspec(dllimport) struct DString __cdecl MyDLL_testStringReturn(void);

int main(int argc, char** argv) {
	printf("[CMain] START\n");
	struct DString d = MyDLL_testStringReturn();
	printf("[CMain] len: %I64u\n", d.len);
	printf("[CMain] str: ");
	for (int i = 0; i < d.len; i++) {
		printf("%c", d.ptr[i]);
	}
	printf("\n");
	printf("[CMain] END\n");
}

This doesn't work (D):

export auto MyDLL_testStringReturn() {
	string s = "hello";
	return s;
}
Segmentation fault

But this does (D):

export auto MyDLL_testStringReturn() {
	import core.stdc.string : memcpy;
	string s = "hello";
	static struct N {
		ubyte[string.sizeof] b;
	}
	static assert(N.sizeof == string.sizeof);
	N n;
	memcpy(&n, &s, string.sizeof);
	return n;
}
[CMain] START
[CMain] len: 5
[CMain] str: hello
[CMain] END
December 27, 2022
On 12/27/22 10:31, cc wrote:

> No need for extern(C#), extern(C) should be sufficient.

Sure but you are not using extern(C) for your D example. (?)

I find quotes like this: "C defines no ABI. In fact, it bends over backwards to avoid defining an ABI."

At least we have this guarantee: "D is designed to fit comfortably with a C compiler for the target system.":

  https://dlang.org/spec/interfaceToC.html#structs_and_unions

> This doesn't work (D):
> ```d
> export auto MyDLL_testStringReturn() {
>      string s = "hello";
>      return s;
> }

But that's not extern(C).

There must be clues on the following page but it doesn't show anything special for returning arrays nor it specifies whether an array is the same as a struct with 2 members. (We know it is equivalent but it doesn't say it is the same as a struct definition.)

  https://dlang.org/spec/abi.html#return_value

Ali

December 27, 2022
On 12/27/22 11:02, Ali Çehreli wrote:

> whether an array is the same as a struct with 2 members. (We
> know it is equivalent but it doesn't say it is the same as a
> struct definition.)

I've just checked. No, D arrays are not returned as a struct that has two members.

Source code and the output of -vasm compilation:

$ tail -n 17 deneme.d
struct S {
    size_t length;
    char * ptr;
}

S returns_S() {
    return S();
}

string returns_string() {
    return "hello world";
}

void main() {
    auto a = returns_S();
    auto b = returns_string();
}

$ dmd -vasm deneme.d
_D6deneme9returns_SFZSQu1S:
0000:   55                       push      RBP
0001:   48 8B EC                 mov       RBP,RSP
0004:   48 83 EC 10              sub       RSP,010h
0008:   48 C7 45 F0 00 00 00 00  mov       qword ptr -010h[RBP],0
0010:   48 C7 45 F8 00 00 00 00  mov       qword ptr -8[RBP],0
0018:   48 8B 55 F8              mov       RDX,-8[RBP]
001c:   48 8B 45 F0              mov       RAX,-010h[RBP]
0020:   C9                       leave
0021:   C3                       ret
_D6deneme14returns_stringFZAya:
0000:   48 8D 15 FC FF FF FF     lea       RDX,[0FFFFFFFCh][RIP]
0007:   B8 0B 00 00 00           mov       EAX,0Bh
000c:   C3                       ret
_Dmain:
0000:   55                       push      RBP
0001:   48 8B EC                 mov       RBP,RSP
0004:   E8 00 00 00 00           call      L0
0009:   E8 00 00 00 00           call      L0
000e:   31 C0                    xor       EAX,EAX
0010:   5D                       pop       RBP
0011:   C3                       ret
main:
0000:   55                       push      RBP
0001:   48 8B EC                 mov       RBP,RSP
0004:   48 8B 15 FC FF FF FF     mov       RDX,[0FFFFFFFCh][RIP]
000b:   E8 00 00 00 00           call      L0
0010:   5D                       pop       RBP
0011:   C3                       ret

A string is passed by registers while a struct object is copied to stack. For strings, EAX carries the length because 0Bh is 11, the length of "hello world".

As expected, compiling with -O changes how the struct object is returned; now by registers like a string is:

_D6deneme9returns_SFZSQu1S:
0000:   31 C0                    xor       EAX,EAX
0002:   31 D2                    xor       EDX,EDX
0004:   C3                       ret

Ali

December 28, 2022
On Wednesday, 28 December 2022 at 00:15:42 UTC, Ali Çehreli wrote:
> A string is passed by registers while a struct object is copied to stack.

It looks like they both are returned in the RDX:RAX pair in all cases with and without optimizations. I don't see any difference at all.
December 28, 2022

On Tuesday, 27 December 2022 at 18:31:34 UTC, cc wrote:

>

On Tuesday, 27 December 2022 at 17:04:03 UTC, Ali Çehreli wrote:

>

I don't know C# to be sure but it must be ABI related. The way D passes returned objects must be different from what C# expects. (There is no extern(C#) to rely on here. ;) )

No need for extern(C#), extern(C) should be sufficient. However, how does D pass a D slice as a return value under extern(C)? Does it just return nothing at all? Returning a struct that is the same size as a D slice and memcpy'd from one works.

I may be wrong (Please ignore this if i am wrong) , This example will return String from D to C#. however the actual issue is (the challenge i am facing is) i am not able to call a D function which has Struct parameter with String property.

I am trying to see how i can call Print_gph function which has Struct parameter with String property.

struct gph
{
	string x;
}

struct myStruct
{
	pragma(mangle, "Print_gph")
	void Print_gph(gph g)
	{
		stderr.writeln("here: gph ");
		stderr.writeln(g.x);
	
	}
}
« First   ‹ Prev
1 2