Dan wrote:
> I went to freenode and nobody was in the #D channel.  : (

That's weird.  There are normally like 40 people in there.


Sean

Alexander Panek wrote:
> [...]
> It might look like a "hackish" (yes, I call anything that's not
> properly designed and implemented - with pedantic attention for each
> detail - hackish..no offense intended, that just applies to my own
> code) kernel would has its benefits in the long-term, but I think a
> proper research to see what features of D can ease the pain of
> operating system development will gain more knowledge and will
> hopefully help me invent A Good Thing (tm).
> [...]

If you guys really are young college students and can afford to sink time into something like this, I say go for it 100%.  It really doesn't matter whether it amounts to anything, because you will learn a ton from it and it will be a fun project.  That's just the kind of thing to do at your age.  At my age, you have to do things that matter (read: pay the bills), which is to say that you simply cannot afford to do something like write an OS kernel, and I really envy you.  If you're really good, you can make a name for yourself with it, even if the kernel is a commercial failure, which is worth plenty to your career.

That being said, I think it would be *particularly* clever of you to use your start-from-scratch kernel to explore new ideas in multi-processing and massive multithreading, but this would imply having decent multi-CPU hardware to play with.  Most of the OS support we have is decades old technology hacked together to keep up with the hardware, but maybe you have an opportunity to do it right from the start.  That is, to create a kernel where multiprocessing is as fundamental as memory management or process scheduling.  We try to write lock-free algorithms using just one special assembly instruction, and look how far we have to bend over backwards to make it work.  If an MP-friendly kernel gave us more primitives than CAS, because it was designed that way from the start (meaning, the process scheduler was hacked up in a new way), I suspect that you could make MP a lot easier on users.  Just some stuff to think about...

Dave

David B. Held wrote:
> That being said, I think it would be *particularly* clever of you to use your start-from-scratch kernel to explore new ideas in multi-processing and massive multithreading, but this would imply having decent multi-CPU hardware to play with.

I'd also like to see better operating system support for garbage collection - either making gc an operating system service, or providing hooks to the virtual memory subsystem so the gc can tell when pages are dirty.

On Thu, 19 Apr 2007 01:07:22 -0700
Walter Bright <newshound1@digitalmars.com> wrote:

> David B. Held wrote:
> > That being said, I think it would be *particularly* clever of you to use your start-from-scratch kernel to explore new ideas in multi-processing and massive multithreading, but this would imply having decent multi-CPU hardware to play with.
> 
> I'd also like to see better operating system support for garbage collection - either making gc an operating system service, or providing hooks to the virtual memory subsystem so the gc can tell when pages are dirty.

Well, something along the lines of that is what I was thinking of. I'm not so sure how to make the GC an operating system service, though..?

On Wed, 18 Apr 2007 22:33:18 -0700
"David B. Held" <dheld@codelogicconsulting.com> wrote:

> Alexander Panek wrote:
> > [...]
> > It might look like a "hackish" (yes, I call anything that's not
> > properly designed and implemented - with pedantic attention for each
> > detail - hackish..no offense intended, that just applies to my own
> > code) kernel would has its benefits in the long-term, but I think a
> > proper research to see what features of D can ease the pain of
> > operating system development will gain more knowledge and will
> > hopefully help me invent A Good Thing (tm).
> > [...]
> 
> If you guys really are young college students and can afford to sink time into something like this, I say go for it 100%.  It really doesn't matter whether it amounts to anything, because you will learn a ton from it and it will be a fun project.  That's just the kind of thing to do at your age.  At my age, you have to do things that matter (read: pay the bills), which is to say that you simply cannot afford to do something like write an OS kernel, and I really envy you.  If you're really good, you can make a name for yourself with it, even if the kernel is a commercial failure, which is worth plenty to your career.
> 
> That being said, I think it would be *particularly* clever of you to use your start-from-scratch kernel to explore new ideas in multi-processing and massive multithreading, but this would imply having decent multi-CPU hardware to play with.  Most of the OS support we have is decades old technology hacked together to keep up with the hardware, but maybe you have an opportunity to do it right from the start.  That is, to create a kernel where multiprocessing is as fundamental as memory management or process scheduling.  We try to write lock-free algorithms using just one special assembly instruction, and look how far we have to bend over backwards to make it work.  If an MP-friendly kernel gave us more primitives than CAS, because it was designed that way from the start (meaning, the process scheduler was hacked up in a new way), I suspect that you could make MP a lot easier on users.  Just some stuff to think about...
> 
> Dave

Now those are some ideas and encouragement. Thanks! :)

Alexander Panek wrote:
> On Thu, 19 Apr 2007 01:07:22 -0700
> Walter Bright <newshound1@digitalmars.com> wrote:
> 
>> David B. Held wrote:
>>> That being said, I think it would be *particularly* clever of you
>>> to use your start-from-scratch kernel to explore new ideas in
>>> multi-processing and massive multithreading, but this would imply
>>> having decent multi-CPU hardware to play with.
>> I'd also like to see better operating system support for garbage collection - either making gc an operating system service, or
>> providing hooks to the virtual memory subsystem so the gc can tell
>> when pages are dirty.
> 
> Well, something along the lines of that is what I was thinking of. I'm
> not so sure how to make the GC an operating system service, though..?

http://www-cs.canisius.edu/~hertzm/bc.html has a nice idea on support the OS could give to garbage collection: they modified Linux to notify the process when it was about to get some of its memory swapped out. Then the process could perform a quick GC (focusing on freeing up pages of memory) and tell the OS which pages to unmap or swap out. The GC freeing pages means the GC can tell the OS memory manager that certain pages can be thrown out without saving them to disk first (no disk writes is faster than disk writes :) ).
If it can't avoid swapping pages out, the GC checks the page about to be swapped out for pointers and marks the pointed-to objects as referenced by a swapped-out page, then tells the OS to put that page back at the head of the swap-out queue (since it has just been touched by the GC, it was removed from that queue). In this way, their GC also avoids touching swapped-out pages except in rare cases, so most GC cycles shouldn't cause a single page fault.
Pretty cool stuff.

About making it a system service: I'm not sure how much help that would be, though I guess it could cut down on privilege level switches (caused by warnings about swapping and/or system calls to nominate pages for swapping) if the kernel could perform something like the above GC itself.

Alexander Panek Wrote:
> Now those are some ideas and encouragement. Thanks! :)

Indeed.

Lately a big thing in kernel development world is to move everything possible out of the kernel.  Why?

The kernel is supposed to provide one thing and one thing only: securely allowing multiple programs to run.

To that end, the best kernel is the smallest one that is proveably secure and can efficiently use other programs to do hardware management stuff (read:drivers)

So when I think of a kernel, I write an exokernel that controls the system by means of controlling the:

1) memory
2) ports
3) thread/process switcher attached to timer interrupt

If you control memory, you control what agents can be loaded where and when.  If an agent cannot get memory for a code block, they obviously cannot run it.

Just my view.  As I said, I'm interested in helping.  Unfortunately, I've been pulling overtime the last few so I haven't had a chance to try getting on freenode.net again.

David B. Held wrote:
> 
> That being said, I think it would be *particularly* clever of you to use your start-from-scratch kernel to explore new ideas in multi-processing and massive multithreading, but this would imply having decent multi-CPU hardware to play with.  Most of the OS support we have is decades old technology hacked together to keep up with the hardware, but maybe you have an opportunity to do it right from the start.  That is, to create a kernel where multiprocessing is as fundamental as memory management or process scheduling.  We try to write lock-free algorithms using just one special assembly instruction, and look how far we have to bend over backwards to make it work.  If an MP-friendly kernel gave us more primitives than CAS, because it was designed that way from the start (meaning, the process scheduler was hacked up in a new way), I suspect that you could make MP a lot easier on users.  Just some stuff to think about...

Great idea :-)  BeOS had very good multithreaded performance, but it didn't offer a new way to write multithreaded code.  If such a beast existed, I'd definitely be interested in playing with it.


Sean

Walter Bright wrote:
> David B. Held wrote:
>> That being said, I think it would be *particularly* clever of you to use your start-from-scratch kernel to explore new ideas in multi-processing and massive multithreading, but this would imply having decent multi-CPU hardware to play with.
> 
> I'd also like to see better operating system support for garbage collection - either making gc an operating system service, or providing hooks to the virtual memory subsystem so the gc can tell when pages are dirty.

Same here.  That the OS is aware of context switches and such should allow for some clever methods of garbage collection.  I'm actually a bit surprised that MS didn't do anything with GC support for Vista, given their push with CLI.


Sean

Frits van Bommel wrote:
> Alexander Panek wrote:
>> On Thu, 19 Apr 2007 01:07:22 -0700
>> Walter Bright <newshound1@digitalmars.com> wrote:
>>
>>> David B. Held wrote:
>>>> That being said, I think it would be *particularly* clever of you
>>>> to use your start-from-scratch kernel to explore new ideas in
>>>> multi-processing and massive multithreading, but this would imply
>>>> having decent multi-CPU hardware to play with.
>>> I'd also like to see better operating system support for garbage collection - either making gc an operating system service, or
>>> providing hooks to the virtual memory subsystem so the gc can tell
>>> when pages are dirty.
>>
>> Well, something along the lines of that is what I was thinking of. I'm
>> not so sure how to make the GC an operating system service, though..?
> 
> http://www-cs.canisius.edu/~hertzm/bc.html has a nice idea on support the OS could give to garbage collection: they modified Linux to notify the process when it was about to get some of its memory swapped out. Then the process could perform a quick GC (focusing on freeing up pages of memory) and tell the OS which pages to unmap or swap out. The GC freeing pages means the GC can tell the OS memory manager that certain pages can be thrown out without saving them to disk first (no disk writes is faster than disk writes :) ).

I can think of at least two papers on this topic for anyone interested.  One is "Reducing Garbage Collector Cache Missed" by Hans Boehm, and the other is "Garbage Collection Without Paging" by Hertz, Feng, and Berger.  Both should be available with a bit of googling.


Sean