Harry Vennik wrote:
>> What about I/O? mmap the file from a service for block devices? shared
>> memory for the buffer for stream? I have thought this would be an
>> interesting way to go.
> 
> Indeed, might be something like the shared memory buffer. Such would allow very good performance.
> There may be some process isolation problems / security risks. But I think those will be solvable.

Put the security in stetting up the shared buffers and that should do it I'd think.
Read access lets you map it in in a read only page, read/write... etc.

One thought I had on this issue is that it allows much of the IO to run without system calls/context switches if the stream access and whatnot is all run as part of the standard system lib. Plus, while you can mess with your stream meta data, it only hoses you.

BCS wrote:
> Put the security in stetting up the shared buffers and that should do
> it I'd think.
> Read access lets you map it in in a read only page, read/write... etc.
>
> One thought I had on this issue is that it allows much of the IO to run without system calls/context switches if the stream access and whatnot is all run as part of the standard system lib. Plus, while you can mess with your stream meta data, it only hoses you.

I have been thinking that way too, but it is difficult to see how much it really
prevents context switches. It may just obscure them by having them occur only in
response to certain events (which may occur regularly), instead of having the
predictable context switch for every read/write/whatever you do.
Memory-mapping the file will prevent this, but is costly for bigger files, so this
needs some thought. Anyway, if a way is found to have a reliable shared-buffer
implementation that will not compensate its lower number of context switches by
having more of another unwanted side-effect (e.g. excessive memory usage), than
that would certainly be the way to go.

Harry

Harry Vennik wrote:
> BCS wrote:
> 
>>Put the security in stetting up the shared buffers and that should do
>>it I'd think.
>>Read access lets you map it in in a read only page, read/write... etc.
>>
>>One thought I had on this issue is that it allows much of the IO to
>>run without system calls/context switches if the stream access and
>>whatnot is all run as part of the standard system lib. Plus, while you
>>can mess with your stream meta data, it only hoses you.
> 
> 
> I have been thinking that way too, but it is difficult to see how much it really
> prevents context switches. It may just obscure them by having them occur only in
> response to certain events (which may occur regularly), instead of having the
> predictable context switch for every read/write/whatever you do.
> Memory-mapping the file will prevent this, but is costly for bigger files, so this
> needs some thought. Anyway, if a way is found to have a reliable shared-buffer
> implementation that will not compensate its lower number of context switches by
> having more of another unwanted side-effect (e.g. excessive memory usage), than
> that would certainly be the way to go.
> 
> Harry

The appropriate snide response here is "memory is cheap". But I've never bought into that one. If the IO system allowed arbitrary mapping of memory pages to disk blocks:

sysCallMapPage2Block(int fd, size_t block, void* page, uint count = 1);

this might not be fairly fast as far as the OS side goes.

if(fd.good && fd.size >= block &&
	user.pageCount + count < suer.pageLimit)
	MapPage();

The other side would be totally up to the stdlib ("not your problem" :-)

This would have some other cool effects like use this under malloc (fd == -1 indicates blank pages)

>Harry Vennik wrote:
>> BCS wrote:
>>
>>>Put the security in stetting up the shared buffers and that should do
>>>it I'd think.
>>>Read access lets you map it in in a read only page, read/write... etc.
>>>
>>>One thought I had on this issue is that it allows much of the IO to run without system calls/context switches if the stream access and whatnot is all run as part of the standard system lib. Plus, while you can mess with your stream meta data, it only hoses you.
>>
>>
>> I have been thinking that way too, but it is difficult to see how much it really
>> prevents context switches. It may just obscure them by having them occur only in
>> response to certain events (which may occur regularly), instead of having the
>> predictable context switch for every read/write/whatever you do.
>> Memory-mapping the file will prevent this, but is costly for bigger files, so this
>> needs some thought. Anyway, if a way is found to have a reliable shared-buffer
>> implementation that will not compensate its lower number of context switches by
>> having more of another unwanted side-effect (e.g. excessive memory usage), than
>> that would certainly be the way to go.
>>
>> Harry
>
> The appropriate snide response here is "memory is cheap". But I've never bought into that one. If the IO system allowed arbitrary mapping of memory pages to disk blocks:
>
> sysCallMapPage2Block(int fd, size_t block, void* page, uint count = 1);
>
> this might not be fairly fast as far as the OS side goes.
>
> if(fd.good && fd.size >= block &&
>  user.pageCount + count < suer.pageLimit)
>  MapPage();
>
> The other side would be totally up to the stdlib ("not your problem" :-)
>
> This would have some other cool effects like use this under malloc
> (fd == -1 indicates blank pages)

No, this won't fit in the concept. I think of it like this (Only mentioning the relevant steps.):
- A process does an IPC call to the filesystem service to map some data from a file into memory.
- The fs service reads the data from disk and puts it into newly allocated memory page(s).
- The fs service announces to the kernel which pages should be mapped into the virtual address space
of the calling process, and returns the pointer.
- The kernel maps the shareable memory into the virtual address space of the calling process, and
translates the returned pointer to reflect the correct address in the calling process' address space.

I think that's the best we can do. It is secure (at least at first sight), and it should save quite a lot of syscalls, except when reading/writing a large file randomly.

Harry

Walter Bright wrote:
> Harry Vennik wrote:
>> In general, most mentioned already is the question 'why C instead of D for the microkernel?'. The answer is that there is no OOP involved in the microkernel at all, and the added value of D would probably be nothing there.
>
> I'd turn the question around and ask what advantage there is to writing in C?

Hmmm, You are undoubtedly asking a very legitimate question. And I don't know the answer now, but I
am not yet sure there really won't be a good reason in the end. If there appears to be none, it will be D to
keep things consistent.

Harry

Harry Vennik wrote:
> 
> No, this won't fit in the concept. I think of it like this (Only mentioning the relevant steps.):
> - A process does an IPC call to the filesystem service to map some data from a file into memory.
> - The fs service reads the data from disk and puts it into newly allocated memory page(s).
> - The fs service announces to the kernel which pages should be mapped into the virtual address space
> of the calling process, and returns the pointer.
> - The kernel maps the shareable memory into the virtual address space of the calling process, and
> translates the returned pointer to reflect the correct address in the calling process' address space.
> 
> I think that's the best we can do. It is secure (at least at first sight), and it should save quite a lot of
> syscalls, except when reading/writing a large file randomly.
> 
> Harry

This is quite a bit different from what I was thinking of. I'll have to think on it.

Could it handle the "lazy-load" type of mmap? The kind where nothing is loaded when you map in the file, instead that all waits for page faults using the same device as the virtual memory system. Maybe the file system process could just do a memory page to disk block mapping for the other process.

Harry Vennik wrote:
> Is there anyone here who is interested in working with me to get that project going, and help by
> contributing in the development of a specification and/or in implementation of those (including PoC
> coding)?

Just in case nobody has mentioned it: a small and elegant UNIX kernel to use as a source of inspiration is Minix 3 (http://www.minix3.org/).

I've seen Tanenbaum talking about it, and Minix 3 is a fresh new approach to OS design featuring high modularity: the kernel itself boasts only 4000 lines of code. It would be interesting to see how that translates in D.

Andrei

Andrei Alexandrescu (See Website For Email) wrote:
> Harry Vennik wrote:
> 
>> Is there anyone here who is interested in working with me to get that project going, and help by
>> contributing in the development of a specification and/or in implementation of those (including PoC
>> coding)?
> 
> 
> Just in case nobody has mentioned it: a small and elegant UNIX kernel to use as a source of inspiration is Minix 3 (http://www.minix3.org/).
> 
> I've seen Tanenbaum talking about it, and Minix 3 is a fresh new approach to OS design featuring high modularity: the kernel itself boasts only 4000 lines of code. It would be interesting to see how that translates in D.
> 
> Andrei

That looks really interesting ... thanks for the link

On Thu, 04 Jan 2007 17:36:26 -0800, Andrei Alexandrescu (See Website For
Email) wrote:

> Harry Vennik wrote:
>> Is there anyone here who is interested in working with me to get that project going, and help by contributing in the development of a specification and/or in implementation of those (including PoC coding)?
> 
> Just in case nobody has mentioned it: a small and elegant UNIX kernel to use as a source of inspiration is Minix 3 (http://www.minix3.org/).
> 
> I've seen Tanenbaum talking about it, and Minix 3 is a fresh new approach to OS design featuring high modularity: the kernel itself boasts only 4000 lines of code. It would be interesting to see how that translates in D.
> 
> Andrei

Very interesting!

I'll download the svn source and check it out. :)

-JJR

Hi all,

I announced this project just a little more than one year ago, but then I had to stop it right away because of some personal problems. Recently I picked it up again, but to really gain any progress, I really need help from one or more real D gurus, as there is one very essential problem I cannot solve myself.

I'd like to build my OS on top of the OKL4 microkernel from Open Kernel Labs. That microkernel is written in C and C++, but in my OS, everything running on top of it should be in D. (Although it might be that I use some existing servers at first, and replace them by D-implementations later.)

Now I'd like to have automated code generation for IPC, without using IDL. I.e. i want to be able to simply code a D interface and then have some way to expose that interface to be called through IPC, without the need to write the actual IPC code over and over again for each interface. Ideally the IPC code would be compatible to code generated from IDL through magpie, as such would allow D-clients to call C-servers and vice versa, this is not a requirement however.

Currently I feel like this is the most difficult part to code of the whole OS (maybe just because I do have quite some knowledge about OSes, but almost nothing about compilers and code generation...)

Anyone interested to help me to get this done?

Regards,

Harry

== Quote from John Reimer (terminal.node@gmail.com)'s article
> On Thu, 04 Jan 2007 17:36:26 -0800, Andrei Alexandrescu (See Website For
> Email) wrote:
> > Harry Vennik wrote:
> >> Is there anyone here who is interested in working with me to get that project going, and help by contributing in the development of a specification and/or in implementation of those (including
PoC
> >> coding)?
> >
> > Just in case nobody has mentioned it: a small and elegant UNIX kernel to use as a source of inspiration is Minix 3 (http://www.minix3.org/).
> >
> > I've seen Tanenbaum talking about it, and Minix 3 is a fresh new approach to OS design featuring high modularity: the kernel itself boasts only 4000 lines of code. It would be interesting to see how that translates in D.
> >
> > Andrei
> Very interesting!
> I'll download the svn source and check it out. :)
> -JJR