Thread overview
Creating a pointer/slice to a specific-size buffer? (Handing out a page/frame from a memory manager)
Jan 13, 2023
Gavin Ray
Jan 13, 2023
Ali Çehreli
Jan 13, 2023
Gavin Ray
Jan 13, 2023
Gavin Ray
Jan 13, 2023
Ali Çehreli
Jan 13, 2023
Ali Çehreli
Jan 13, 2023
H. S. Teoh
Jan 13, 2023
Gavin Ray
Jan 13, 2023
Gavin Ray
January 13, 2023

Suppose that you have a memory manager, or arena-like class, which contains a buffer used to store memory.

And you want to hand out chunks of this memory to other parts of your program. These chunks should all be PAGE_SIZE.

You might have something like:

enum PAGE_SIZE = 4096;
enum BUF_POOL_NUM_PAGES = 1024;

class BufferPool
{
	align(PAGE_SIZE) ubyte[PAGE_SIZE * BUF_POOL_NUM_PAGES] data;

	this()
	{
		mmap(&data, data.sizeof,
                     PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
	}
}

Now there should be a function, like get_page(), that returns a PAGE_SIZE pointer of ubyte. But I can only figure out how to return a ubyte[]:

ubyte[] get_page(frame_idx_t frame_idx)
{
	return data[frame_idx * PAGE_SIZE .. (frame_idx + 1) * PAGE_SIZE];
}

I am curious if you can return something like ubyte[PAGE_SIZE]* or ref ubyte[PAGE_SIZE]?

Thank you =)

January 13, 2023
On 1/13/23 06:49, Gavin Ray wrote:

> I am curious if you can return something like `ubyte[PAGE_SIZE]*` or
> `ref ubyte[PAGE_SIZE]`?

A simple cast seems to work:

enum PAGE_SIZE = 4096;
enum BUF_POOL_NUM_PAGES = 1024;
alias frame_idx_t = size_t;

ubyte[10_000] data;

ubyte[PAGE_SIZE]* get_page(frame_idx_t frame_idx)
{
    auto ptr = data.ptr + frame_idx * PAGE_SIZE;
    return cast(ubyte[PAGE_SIZE]*)ptr;
}

void main() {
}

Ali

January 13, 2023

I probably should have mentioned, the equivalent in C++ is the below:

#include <cstddef>
#include <span>
#include <sys/mman.h>

static constexpr size_t PAGE_SIZE          = 4096;
static constexpr size_t BUF_POOL_NUM_PAGES = 1024;

class BufferPool
{
  private:
    alignas(PAGE_SIZE) std::byte data[BUF_POOL_NUM_PAGES * PAGE_SIZE];

  public:
    BufferPool()
    {
        mmap(data, BUF_POOL_NUM_PAGES * PAGE_SIZE,
             PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    }

    // Return a fat-pointer of PAGE_SIZE bytes over the page, containing { T* ptr, len }
    std::span<std::byte, PAGE_SIZE> get_page(size_t page_num)
    {
        return std::span<std::byte, PAGE_SIZE>(data + page_num * PAGE_SIZE, PAGE_SIZE);
    }
};
January 13, 2023

On Friday, 13 January 2023 at 14:57:40 UTC, Ali Çehreli wrote:

>

On 1/13/23 06:49, Gavin Ray wrote:

>

I am curious if you can return something like
ubyte[PAGE_SIZE]* or
ref ubyte[PAGE_SIZE]?

A simple cast seems to work:

enum PAGE_SIZE = 4096;
enum BUF_POOL_NUM_PAGES = 1024;
alias frame_idx_t = size_t;

ubyte[10_000] data;

ubyte[PAGE_SIZE]* get_page(frame_idx_t frame_idx)
{
auto ptr = data.ptr + frame_idx * PAGE_SIZE;
return cast(ubyte[PAGE_SIZE]*)ptr;
}

void main() {
}

Ali

Ah, much thanks Ali!

This is "valid" D I hope?

I searched Github for "ubyte[4096]*" and I found two uses of it, in the xtrix repo which seems to be an Operating System written in D:

screenshot-of-ubyte-sized-ptr-on-github

January 13, 2023

Maybe it would be better to wrap the slice in a new class with an invariant?

Because what I want to do is:

  1. Ensure that the length of the underlying referenced/pointed-to data is PAGE_SIZE
  2. Benefit from the features of D that I can

The bounds-checking of slices saves a lot of headaches during development.

So maybe doing something like this might be better?

struct Frame
{
	ubyte[] data;

	invariant
	{
		assert(data.length == PAGE_SIZE);
	}
}

class BufferPool
{
	align(PAGE_SIZE) ubyte[PAGE_SIZE * BUF_POOL_NUM_PAGES] data;
	Frame[BUF_POOL_NUM_PAGES] frames;

	this()
	{
		// mmap
		foreach (i; 0 .. BUF_POOL_NUM_PAGES)
		{
			frame_idx_t frame_idx = cast(frame_idx_t) i;
			frames[frame_idx].data = data[frame_idx * PAGE_SIZE .. (frame_idx + 1) * PAGE_SIZE];
		}
	}
}
January 13, 2023
On 1/13/23 07:07, Gavin Ray wrote:

> This is "valid" D I hope?

Yes because static arrays are just elements side-by-side in memory. You can cast any piece of memory to a static array provided the length and alignment are correct.

However, such a cast is not allowed in @safe code.

Ali

January 13, 2023
On 1/13/23 07:22, Gavin Ray wrote:
> Maybe it would be better to wrap the slice in a new class with an
> invariant?

Possibly but please check before using because I think 'invariant' requires presence of member functions:

  https://dlang.org/spec/struct.html#Invariant

> Because what I want to do is:
>
> 1. Ensure that the length of the underlying referenced/pointed-to data
> is `PAGE_SIZE`

My first thought was why not use a slice anyway?

Worth noting that static arrays are value types. Also, they all have different types from each other and have the potential to cause template bloat.

> class BufferPool

Off-topic, most D programmers use struct unless they need class.

>      Frame[BUF_POOL_NUM_PAGES] frames;

Makes sense to me.

Ali

January 13, 2023
On Fri, Jan 13, 2023 at 08:31:17AM -0800, Ali Çehreli via Digitalmars-d-learn wrote:
> On 1/13/23 07:07, Gavin Ray wrote:
> 
> > This is "valid" D I hope?
> 
> Yes because static arrays are just elements side-by-side in memory. You can cast any piece of memory to a static array provided the length and alignment are correct.
[...]

Or to be more precise, cast the memory to a *pointer* to a static array of the right size.  Static arrays are by-value types; passing around the raw array will cause the array to be copied every time, which is probably not what is intended.


T

-- 
"You are a very disagreeable person." "NO."
January 13, 2023

On Friday, 13 January 2023 at 19:16:17 UTC, H. S. Teoh wrote:

>

On Fri, Jan 13, 2023 at 08:31:17AM -0800, Ali Çehreli via Digitalmars-d-learn wrote:

>

On 1/13/23 07:07, Gavin Ray wrote:

>

This is "valid" D I hope?

Yes because static arrays are just elements side-by-side in memory. You can cast any piece of memory to a static array provided the length and alignment are correct.
[...]

Or to be more precise, cast the memory to a pointer to a static array of the right size. Static arrays are by-value types; passing around the raw array will cause the array to be copied every time, which is probably not what is intended.

T

Thanks Teoh and Ali, I wound up passing a pointer to a static array.

If anyone is curious, here's the full WIP implementation -- it's for a toy database (Postgres-like) I'm writing for a hobby and learning project:

https://ldc.godbolt.org/z/Kvh7dv96c