I'm sorry for opening such a topic; I've heard it's not liked a lot, but I think it might be necessary.

I'm not asking for a 'volatile' keyword, but rather to find out what the right thing to use is.
After reading a few different threads related to microcontrollers, I started wondering how to program the following in D:

1: System level drivers, which writes directly to hardware registers (any architecture, PC/i386 [Linux, Windows, others], Atari ST, IBM BladeCenter, etc.)
2: Interrupts that needs to share variables.

1) is what we basically need on microcontrollers. If it's possible to write a driver in D, which has no problems with accessing hardware, then it should be possible to do it for any microcontroller as well.

2) shared variables could be used for interrupts, but what happens if the TLS is disabled; will shared variables work ? -Interrupts are not threads.

Regarding (1), because marking a variable 'shared' is not enough (it allows instructions to be moved around), Johannes already made a volatileLoad and volatileStore, which will be usable for microcontrollers, though for convenience, it requires writing additional code.
-But this solution ... I do not know if it would work, when writing a driver for Debian running on a i586 platform or PowerMac G3 for instance.

If variable 'alice' and variable 'bob' are both shared, and reading from 'bob', then writing to 'alice'; would instructions be moved around, so reading from 'bob' could actually occur after writing to 'alice' ?

Imagine that in the microcontroller world, there's a *bunch* of hardware registers.
The best thing a microcontroller knows, is to read/write hardware registers; often only a few bits at a time - but definitely also 8-, 16- and 32-bit values.

Thus you will most likely not be able to find a single piece of source-code for a microcontroller, where it does not access hardware. This means: Hardware is used often, and reading/writing hardware registers should not take up a lot of space in the source code.


For those, who are unfamiliar with such code, here's a silly example in C on how it might look.
The following code changes the clock-frequency of the microcontroller, so it runs at 100 MHz:

---8<-----8<-----8<-----
uint32_t setupCoreClock()
{
	/* By default, we run at 100MHz, however,
	 * our PLL clock frequency is 300 MHz.
	 * We'd like to run 400MHz! */

	LPC_SC->SCS = SCS_MainOscillatorEnable;	/* enable Main Oscillator */
	if(LPC_SC->SCS & SCS_MainOscillatorEnable)
	{
		while(0 == (LPC_SC->SCS & SCS_MainOscillatorStatus)){}
	}

	LPC_SC->CCLKCFG = CCLK_Divider - 1;

	LPC_SC->PCLKSEL0 = 0;
	LPC_SC->PCLKSEL1 = 0;

	LPC_SC->CLKSRCSEL = PLL0_ClockSource;

	LPC_SC->PLL0CFG = PLL0_Configuration;
	LPC_SC->PLL0FEED = 0xaa;
	LPC_SC->PLL0FEED = 0x55;

	LPC_SC->PLL0CON = PLL_Enable;
	LPC_SC->PLL0FEED = 0xaa;
	LPC_SC->PLL0FEED = 0x55;
	while(!(LPC_SC->PLL0STAT & PLL0_Lock)){}	/* Wait for PLOCK0 */

	LPC_SC->PLL0CON = PLL_Enable | PLL_Connect;
	LPC_SC->PLL0FEED = 0xaa;
	LPC_SC->PLL0FEED = 0x55;
	while((LPC_SC->PLL0STAT & (PLL0_EnabledFlag | PLL0_ConnectedFlag)) != (PLL0_EnabledFlag | PLL0_ConnectedFlag)){}	/* Wait until connected */

	return(F_CCLK);
}
--->8----->8----->8-----

Everything you see above, which starts with 'LPC_' are hardware register accesses.
Not counting the curly braces, nearly every line of the code access hardware registers.

Things like LPC_SC are pointers to structures, which contains a load of volatile keywords; some prohibit reading, some prohibit writing, some allow both, some allow neither.

Most code for ARM Cortex-M looks like the above, because the silicon vendors just modifies the stock (template) source code, which they receive from ARM. Thus there's some convenience, but not a lot. Bitfields are not provided by ARM, so it propagates down to the vendor, which do not supply bitfields either. However, byte/halfword/word access (8/16/32-bit) are sometimes provided through anonymous unions, which is very helpful.

So back to what originally triggered this post; this is what the question is actually about:
If writing a driver for <platform X>, how is reading/writing hardware registers usually done in D ?

Am Thu, 07 May 2015 16:04:55 +0000
schrieb "Jens Bauer" <doctor@who.no>:

> I'm sorry for opening such a topic; I've heard it's not liked a lot, but I think it might be necessary.
> 
> I'm not asking for a 'volatile' keyword, but rather to find out
> what the right thing to use is.
> After reading a few different threads related to
> microcontrollers, I started wondering how to program the
> following in D:
> 
> 1: System level drivers, which writes directly to hardware
> registers (any architecture, PC/i386 [Linux, Windows, others],
> Atari ST, IBM BladeCenter, etc.)
> 2: Interrupts that needs to share variables.
> 
> 1) is what we basically need on microcontrollers. If it's possible to write a driver in D, which has no problems with accessing hardware, then it should be possible to do it for any microcontroller as well.
> 
> 2) shared variables could be used for interrupts, but what happens if the TLS is disabled; will shared variables work ? -Interrupts are not threads.
> 
> Regarding (1), because marking a variable 'shared' is not enough (it allows instructions to be moved around), Johannes already made a volatileLoad and volatileStore, which will be usable for microcontrollers, though for convenience, it requires writing additional code.

1)
Actually these were proposed for DMD some time before I've implemented
the GDC part ;-) Shared should be avoided for volatile data for
now. It might make sense if you want to synchronize accesses from
threads to a volatile memory location, but then things might get
complicated.

2)
This can be done with volatileLoad/Store as well. Just access a global
variable using volatileLoad/Store. This can also be used nicely in a
wrapper. If you want to enforce atomic access to volatile variables
(bigger than the word size) you might need shared(Volatile!T). Again
things might get complicated ;-)


> If variable 'alice' and variable 'bob' are both shared, and reading from 'bob', then writing to 'alice'; would instructions be moved around, so reading from 'bob' could actually occur after writing to 'alice' ?

Not sure about shared (I don't think anybody knows what exactly shared
is supposed to do) but if you use volatileLoad/Store
these instructions won't be moved around.

On 7 May 2015 at 18:04, Jens Bauer via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> I'm sorry for opening such a topic; I've heard it's not liked a lot, but I think it might be necessary.
>
> I'm not asking for a 'volatile' keyword, but rather to find out what the
> right thing to use is.
> After reading a few different threads related to microcontrollers, I started
> wondering how to program the following in D:
>
> 1: System level drivers, which writes directly to hardware registers (any
> architecture, PC/i386 [Linux, Windows, others], Atari ST, IBM BladeCenter,
> etc.)
> 2: Interrupts that needs to share variables.
>
> 1) is what we basically need on microcontrollers. If it's possible to write a driver in D, which has no problems with accessing hardware, then it should be possible to do it for any microcontroller as well.
>
> 2) shared variables could be used for interrupts, but what happens if the TLS is disabled; will shared variables work ? -Interrupts are not threads.
>
> Regarding (1), because marking a variable 'shared' is not enough (it allows
> instructions to be moved around), Johannes already made a volatileLoad and
> volatileStore, which will be usable for microcontrollers, though for
> convenience, it requires writing additional code.
> -But this solution ... I do not know if it would work, when writing a driver
> for Debian running on a i586 platform or PowerMac G3 for instance.
>
> If variable 'alice' and variable 'bob' are both shared, and reading from 'bob', then writing to 'alice'; would instructions be moved around, so reading from 'bob' could actually occur after writing to 'alice' ?
>

Yes.  Take this example:

---
shared int alice, bob;
void foo()
{
    alice = bob + 1;
    bob = 0;
}
---

gdc without optimisations produces:
---
; load bob into %eax
movl    _D7reorder3bobOi(%rip), %eax
; + 1
addl    $1, %eax
; store to alice
movl    %eax, _D7reorder5aliceOi(%rip)
; bob = 0
movl    $0, _D7reorder3bobOi(%rip)
---

gdc with optimisations produces:
---
; load bob into eax
movl    _D7reorder3bobOi(%rip), %eax
; bob = 0
movl    $0, _D7reorder3bobOi(%rip)
; + 1
addl    $1, %eax
; store to alice
movl    %eax, _D7reorder5aliceOi(%rip)
---

Now, this does not change the behaviour of the program if we consider that bob and alice do not have any side effects.  However on a micro-controller dealing with direct memory I/O ...


Iain.

On 7 May 2015 at 20:18, Johannes Pfau via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>> If variable 'alice' and variable 'bob' are both shared, and reading from 'bob', then writing to 'alice'; would instructions be moved around, so reading from 'bob' could actually occur after writing to 'alice' ?
>
> Not sure about shared (I don't think anybody knows what exactly shared
> is supposed to do) but if you use volatileLoad/Store
> these instructions won't be moved around.

I've change my mind over the years, so who knows whether or not I'll change again, but this is the current definition I give to it.

Shared data on an ABI level is exactly the same as __gshared, but that's where the similarities end.  On a semantic level, shared comes with a transitive, and statically type-checked guarantee that either taking it's address or passing it's reference around stays within the bounds of it's given qualifier, so everything it's address passes through must also be shared.  To make it an even stronger type, non-shared data can not be implicitly promoted to shared.

Of course, explicit casts allow you to demote and promote the 'shared' qualifier as much as you like, circumventing all guarantee.

When used properly though, it's properties make it a prime candidate for the foundation of libraries/programs that centre around the use of atomics.  However, shared is not to be confused with a thread-safe atomic type.  Thread-safety is left to the end-user on such matters of access.

Regards
Iain.

On 7 May 2015 at 23:39, Iain Buclaw <ibuclaw@gdcproject.org> wrote:
> When used properly though, it's properties make it a prime candidate for the foundation of libraries/programs that centre around the use of atomics.  However, shared is not to be confused with a thread-safe atomic type.  Thread-safety is left to the end-user on such matters of access.
>

That last sentence should also include a mention of memory ordering too being left to the end-user.

On Thursday, 7 May 2015 at 16:04:56 UTC, Jens Bauer wrote:

> So back to what originally triggered this post; this is what the question is actually about:
> If writing a driver for <platform X>, how is reading/writing hardware registers usually done in D ?

Here's my pre 2.067 code for `volatile` semantics

@inline T volatileLoad(T)(T* a)
{
    asm { "" ::: "memory"; };
    return *cast(shared T*)a;
}

@inline void volatileStore(T)(T* a, in T v)
{
    asm { "" ::: "memory"; };
    *cast(shared T*)a = v;
}

As Iain showed, shared does not provide any order guarantees, so that's why I added the asm { "" ::: "memory"; };  memory barrier.

This code, however, is still incorrect because it uses a bug in GDC as a feature.  Iain explained that bug here:  https://youtu.be/o5m0m_ZG9e8?t=3141

There's also a post here that shows a volatileLoad/Store implementation using memory barriers without shared (http://forum.dlang.org/post/501A6E01.7010809@gmail.com), but when I tried it with my cross-compiler, I got an ICE.  Furthermore, I don't even understand it.  I don't understand exactly what the +g and +m constraints do, and I hate using code I don't understand.

Once 2.067 is properly merged and implemented, none of this will be necessary anyway as volatileLoad/Store intrinsics were introduced.  That being said, you will still have to add some boilerplate around volatileLoad/Store to make the syntax tolerable.  Johannes has a nice implementation of Volatile!T for that purpose here (http://dpaste.dzfl.pl/dd7fa4c3d42b).  I have my own CTFE Template implementation here (https://github.com/JinShil/stm32f42_discovery_demo/blob/master/source/stm32f42/mmio.d).

In terms of shared, I think the following thread is quite telling:  http://forum.dlang.org/post/lruc3n$at1$1@digitalmars.com.  Based on that, I wouldn't trust shared for anything, and I think it should just be ignored altogether.  See also http://p0nce.github.io/d-idioms/#The-truth-about-shared

I'm not sure what to do yet for synchronized/atomic access.  If I had to do in now, I'd probably just stick with C-like techniques.

Mike

On Thursday, 7 May 2015 at 21:42:08 UTC, Iain Buclaw wrote:
> On 7 May 2015 at 23:39, Iain Buclaw <ibuclaw@gdcproject.org> wrote:
>> When used properly though, it's properties make it a prime candidate
>> for the foundation of libraries/programs that centre around the use of
>> atomics.  However, shared is not to be confused with a thread-safe
>> atomic type.  Thread-safety is left to the end-user on such matters of
>> access.
>>
>
> That last sentence should also include a mention of memory ordering
> too being left to the end-user.

Thank you for explaining this; it does make sense. :)

-So 'shared' is not to be confused with 'atomic'; its behaviour seems closer to C's "extern", do I understand this correctly ?

One of the reasons I asked the question, is because no doubt D will need to write to hardware on any kind of platform, and I'd like it to be the same on both microcontrollers and a Personal Computer (not limited to a specific kind of processor).

Another thing, slightly related, is 'atomic'. C's implementation allows you to have a function, which reads/modifies/writes a value atomically. This is all implemented 'outside' the C language.
Would it make sense to mark a variable 'atomic', or would it be rather crazy, because if it's extern(C) anyway, C wouldn't access it atomically ?

On Thursday, 7 May 2015 at 16:04:56 UTC, Jens Bauer wrote:
> Regarding (1), because marking a variable 'shared' is not enough (it allows instructions to be moved around), Johannes already made a volatileLoad and volatileStore, which will be usable for microcontrollers, though for convenience, it requires writing additional code.
> -But this solution ... I do not know if it would work, when writing a driver for Debian running on a i586 platform or PowerMac G3 for instance.

System calls on sufficiently smart processors like x86 use C-like ABI good practices: registers and buffers to pass data instead of global variables, because multithreaded programs will have race condition on accessing the global variables. See read(2) syscall as an example of such API http://man7.org/linux/man-pages/man2/read.2.html

On Thursday, 7 May 2015 at 18:18:02 UTC, Johannes Pfau wrote:
> Not sure about shared (I don't think anybody knows what exactly shared is supposed to do)

Shared is supposed to prevent the programmer from accidentally putting unshared data in a shared context. Expectedly people wanted it to be a silver bullet for concurrency, instead std.concurrency provides high-level concurrency safety.

On Saturday, 9 May 2015 at 12:16:58 UTC, Kagamin wrote:
> On Thursday, 7 May 2015 at 16:04:56 UTC, Jens Bauer wrote:
>> Regarding (1), because marking a variable 'shared' is not enough (it allows instructions to be moved around), Johannes already made a volatileLoad and volatileStore, which will be usable for microcontrollers, though for convenience, it requires writing additional code.
>> -But this solution ... I do not know if it would work, when writing a driver for Debian running on a i586 platform or PowerMac G3 for instance.
>
> System calls on sufficiently smart processors like x86 use C-like ABI good practices: registers and buffers to pass data instead of global variables, because multithreaded programs will have race condition on accessing the global variables. See read(2) syscall as an example of such API http://man7.org/linux/man-pages/man2/read.2.html

To make my question a little clearer: This part is not about RAM locations, but I/O-memory locations AKA. hardware addresses.
On some systems, such as the 68xxx based Atari and Amiga, peripherals are accessed by reading and writing to memory locations; those locations are addresses belonging to hardware; eg. "hardware space". This depends on the CPU.
As an example, the Atari, the address space was usually 0x..ffxxxx, where the first two dots were "don't care", as the 68000 was only 24-bit; later, 0x00fxxxxx was mirrored to 0xfffxxxxx, for backwards compatibility.
Thus the address space between 0x..f00000 and 0x..ffffff was not ordinary RAM, but I/O-space.
On Z80, for instance, it's common to use the IN and OUT instructions to access hardware, so normally you wouldn't use precious memory locations for peripherals on such systems.

... "System calls" will need to access the peripherals in some way, in order to send data to for instance a printer or harddisk. If the way it's done is using a memory location, then it's necessary to tell the compiler that this is not ordinary memory, but I/O-memory AKA hardware address space.

> On Thursday, 7 May 2015 at 18:18:02 UTC, Johannes Pfau wrote:
>> Not sure about shared (I don't think anybody knows what exactly shared is supposed to do)
>
> Shared is supposed to prevent the programmer from accidentally putting unshared data in a shared context. Expectedly people wanted it to be a silver bullet for concurrency, instead std.concurrency provides high-level concurrency safety.

In other words, it's the oposite of 'static' ?
-If so, then that makes the purpose much clearer to me, and it absolutely makes sense. :)
... Like the "export <symbol>" or "xdef <symbol>" directives in assembly language.

On Saturday, 9 May 2015 at 16:59:35 UTC, Jens Bauer wrote:
> ... "System calls" will need to access the peripherals in some way, in order to send data to for instance a printer or harddisk. If the way it's done is using a memory location, then it's necessary to tell the compiler that this is not ordinary memory, but I/O-memory AKA hardware address space.

Userland code still uses system calls and not global variables, whatever is expressed in read(2) signature tells the compiler enough to pass data via buffer.

>> Shared is supposed to prevent the programmer from accidentally putting unshared data in a shared context. Expectedly people wanted it to be a silver bullet for concurrency, instead std.concurrency provides high-level concurrency safety.
>
> In other words, it's the oposite of 'static' ?

Whether data is shared or not is not tied to its storage class, that's why its shared nature is expressed in its type and storage class can be anything; for the same reason shared type qualifier is transitive.