Dan wrote:
> Sean Kelly Wrote:
>> Lionello Lunesu wrote:
>>> I don't care if there address changes, just as long as there's no data being copied physically. I mean, a block of continuous memory might already be all-over-the-place in physical memory. That's what I want: a VirtualRealloc that returns the address of a new virtual memory block, but which is using the same physical memory.. 'Free' array resizes, surely this is worth something!
>> Hm, assuming there is nothing else of the page(s) but the data to move, it should be possible to do so without explicit copying.  I have no idea is VirtualAlloc can do this, but it's definitely possible.
> 
> Oh my!
> 
> On the x86, this sort of implementation would slaughter your performance, as you'd get massive numbers of cache and page misses.
> 
> You could theoretically implement a system on the x86 that would virtualize memory locations arbitrarily, but.... ewww...

Yeah.  I was thinking that the way to do this would be in the virtual memory manager.  When the pages are moved the page table is updated, etc.


Sean

Sean Kelly wrote:
> Walter Bright wrote:
>> David B. Held wrote:
>>> That being said, I think it would be *particularly* clever of you to use your start-from-scratch kernel to explore new ideas in multi-processing and massive multithreading, but this would imply having decent multi-CPU hardware to play with.
>>
>> I'd also like to see better operating system support for garbage collection - either making gc an operating system service, or providing hooks to the virtual memory subsystem so the gc can tell when pages are dirty.
> 
> Same here.  That the OS is aware of context switches and such should allow for some clever methods of garbage collection.  I'm actually a bit surprised that MS didn't do anything with GC support for Vista, given their push with CLI.
> 

Not only that, but I've never even read/heard of any plans to add a GC to any main-stream OS's.

Seems to me there's got to be a reason for this (not having an OS implement a GC). I wonder what that would be, since now-a-days catering to application development is such a vital thing for any commercial OS?

> 
> Sean

Dave Wrote:
> Seems to me there's got to be a reason for this (not having an OS implement a GC). I wonder what that would be, since now-a-days catering to application development is such a vital thing for any commercial OS?

Well, people are developing them.  The reason they aren't included by default is a mystery to me, then most OS developers have left the entire process this gigantic undocumented undescribable mess that explains why most new OS starts just give up.  I think it also explains why currently widely-used OS's still don't implement any additional features.

It's difficult, and they don't have to.

That said, I've been architecting a ring -1 program that multiplexes hardware resources without any drivers, HAL or OS underneath it; taking the exokernel principle to the farthest extreme.

My exokernel isn't even a kernel, it's just a gatekeeper.  I was trying to think of a good name for it:

Maat, Aker, St. Peter, Gatekeeper, Guardian, BlackSphere, or Garmies were the first that came to mind.

Dan wrote:
> Dave Wrote:
>> Seems to me there's got to be a reason for this (not having an OS implement a GC). I wonder what that would be, since now-a-days catering to application development is such a vital thing for any commercial OS?
> 
> Well, people are developing them.  The reason they aren't included by default is a mystery to me, then most OS developers have left the entire process this gigantic undocumented undescribable mess that explains why most new OS starts just give up.  I think it also explains why currently widely-used OS's still don't implement any additional features.
> 
> It's difficult, and they don't have to.
> 
> That said, I've been architecting a ring -1 program that multiplexes hardware resources without any drivers, HAL or OS underneath it; taking the exokernel principle to the farthest extreme.
> 
> My exokernel isn't even a kernel, it's just a gatekeeper.  I was trying to think of a good name for it:
> 
> Maat, Aker, St. Peter, Gatekeeper, Guardian, BlackSphere, or Garmies were the first that came to mind.

Going the route of those first couple ideas (naming after psychopomps) there's the possibilities Yinepu/Anpu and Wepwawet (which literally means he who opens the way). Actually I think that's a perfect theme for selecting a name for something like that. Sounds like an interesting project indeed.

-- Chris Nicholson-Sauls

Sean Kelly Wrote:

> Dan wrote:
> > Sean Kelly Wrote:
> >> Lionello Lunesu wrote:
> >>> I don't care if there address changes, just as long as there's no data being copied physically. I mean, a block of continuous memory might already be all-over-the-place in physical memory. That's what I want: a VirtualRealloc that returns the address of a new virtual memory block, but which is using the same physical memory.. 'Free' array resizes, surely this is worth something!
> >> Hm, assuming there is nothing else of the page(s) but the data to move, it should be possible to do so without explicit copying.  I have no idea is VirtualAlloc can do this, but it's definitely possible.
> > 
> > Oh my!
> > 
> > On the x86, this sort of implementation would slaughter your performance, as you'd get massive numbers of cache and page misses.
> > 
> > You could theoretically implement a system on the x86 that would virtualize memory locations arbitrarily, but.... ewww...
> 
> Yeah.  I was thinking that the way to do this would be in the virtual memory manager.  When the pages are moved the page table is updated, etc.
> 
> 
> Sean

That works Sean, if your arrays are only ever split if they're bigger than a page; in which case letting the OS split them and 'handle it' is the best way.  You *can* just declare a 64kb array and it'll handle it well enough.

If you're talking about arbitrarily splitting arrays whenever they happen to collide, and hoping the paging system will handle it, you'll be grabbing 4kb segments for each misplaced segment.  What's almost as bad, is that you'll have to have additional code tracking what segments belong to what virtual array.

Chris Nicholson-Sauls Wrote:
> Dan wrote:
> > Maat, Aker, St. Peter, Gatekeeper, Guardian, BlackSphere, or Garmies were the first that came to mind.
> 
> Going the route of those first couple ideas (naming after psychopomps) there's the possibilities Yinepu/Anpu and Wepwawet (which literally means he who opens the way). Actually I think that's a perfect theme for selecting a name for something like that. Sounds like an interesting project indeed.
> 
> -- Chris Nicholson-Sauls

I had to look psychopomps up, thanks Chris, I learned something new today.  I initially took 'he who opens the way' in a buddhist sense, which would be cool, but Wepwawet was actually more scout-for-the-military sort of God.

Check the mythology for Nandi, that's not too bad either.

So, if I named it after Mr Pete, they catholics would probably hang me.
If I name it 'Garmies', the Mormons will.
If I name it 'Maat' I would probably get away with it;

Maat is the Egyptian god of justice, who's feather would be weighed against one's heart upon death; and who ancient judges and even gods would seek approval.

It's also unique enough that I don't think there'll be a name collision.  So Maat it will be?

Now to keep working on Walnut.  : p

Dan wrote:
> Chris Nicholson-Sauls Wrote:
>> Dan wrote:
>>> Maat, Aker, St. Peter, Gatekeeper, Guardian, BlackSphere, or Garmies were the first that came to mind.
>> Going the route of those first couple ideas (naming after psychopomps) there's the possibilities Yinepu/Anpu and Wepwawet (which literally means he who opens the way). Actually I think that's a perfect theme for selecting a name for something like that. Sounds like an interesting project indeed.
>>
>> -- Chris Nicholson-Sauls
> 
> I had to look psychopomps up, thanks Chris, I learned something new today.  I initially took 'he who opens the way' in a buddhist sense, which would be cool, but Wepwawet was actually more scout-for-the-military sort of God.
> 
> Check the mythology for Nandi, that's not too bad either.
> 
> So, if I named it after Mr Pete, they catholics would probably hang me.
> If I name it 'Garmies', the Mormons will.
> If I name it 'Maat' I would probably get away with it;
> 
> Maat is the Egyptian god of justice, who's feather would be weighed against one's heart upon death; and who ancient judges and even gods would seek approval.
> 
> It's also unique enough that I don't think there'll be a name collision.  So Maat it will be?
> 
> Now to keep working on Walnut.  : p

I just don't like anything else with a feather for its obvious logo.  I've got a side project named Wren that intends to use that... ;)  Then again, Apache beat us both to it, so who cares.

Okay, shutting up now.  (Du'a Netjer.)

-- Chris Nicholson-Sauls

"Dan" <murpsoft@hotmail.com> wrote in message news:f0b16i$2k5p$1@digitalmars.com...
> Sean Kelly Wrote:
>> Lionello Lunesu wrote:
>> > I don't care if there address changes, just as long as there's no data
>> > being
>> > copied physically. I mean, a block of continuous memory might already
>> > be
>> > all-over-the-place in physical memory. That's what I want: a
>> > VirtualRealloc
>> > that returns the address of a new virtual memory block, but which is
>> > using
>> > the same physical memory.. 'Free' array resizes, surely this is worth
>> > something!
>>
>> Hm, assuming there is nothing else of the page(s) but the data to move, it should be possible to do so without explicit copying.  I have no idea is VirtualAlloc can do this, but it's definitely possible.
>
> Oh my!
>
> On the x86, this sort of implementation would slaughter your performance, as you'd get massive numbers of cache and page misses.

Is a cache using virtual addresses or physical addresses?

> You could theoretically implement a system on the x86 that would virtualize memory locations arbitrarily, but.... ewww...
>

Isn't this what's happening already?

L.

Dan wrote:
> Sean Kelly Wrote:
> 
>> Dan wrote:
>>> Sean Kelly Wrote:
>>>> Lionello Lunesu wrote:
>>>>> I don't care if there address changes, just as long as there's no data being copied physically. I mean, a block of continuous memory might already be all-over-the-place in physical memory. That's what I want: a VirtualRealloc that returns the address of a new virtual memory block, but which is using the same physical memory.. 'Free' array resizes, surely this is worth something!
>>>> Hm, assuming there is nothing else of the page(s) but the data to move, it should be possible to do so without explicit copying.  I have no idea is VirtualAlloc can do this, but it's definitely possible.
>>> Oh my!
>>>
>>> On the x86, this sort of implementation would slaughter your performance, as you'd get massive numbers of cache and page misses.
>>>
>>> You could theoretically implement a system on the x86 that would virtualize memory locations arbitrarily, but.... ewww...
>> Yeah.  I was thinking that the way to do this would be in the virtual memory manager.  When the pages are moved the page table is updated, etc.
>>
>>
>> Sean
> 
> That works Sean, if your arrays are only ever split if they're bigger than a page; in which case letting the OS split them and 'handle it' is the best way.  You *can* just declare a 64kb array and it'll handle it well enough.
> 
> If you're talking about arbitrarily splitting arrays whenever they happen to collide, and hoping the paging system will handle it, you'll be grabbing 4kb segments for each misplaced segment.  What's almost as bad, is that you'll have to have additional code tracking what segments belong to what virtual array.

This is basically how the DMD garbage collector already works, so it could be done almost "for free" if there were VMM calls to handle it.


Sean

Lionello Lunesu Wrote:

> 
> "Dan" <murpsoft@hotmail.com> wrote in message news:f0b16i$2k5p$1@digitalmars.com...
> > Sean Kelly Wrote:
> >> Lionello Lunesu wrote:
> >> > I don't care if there address changes, just as long as there's no data
> >> > being
> >> > copied physically. I mean, a block of continuous memory might already
> >> > be
> >> > all-over-the-place in physical memory. That's what I want: a
> >> > VirtualRealloc
> >> > that returns the address of a new virtual memory block, but which is
> >> > using
> >> > the same physical memory.. 'Free' array resizes, surely this is worth
> >> > something!
> >>
> >> Hm, assuming there is nothing else of the page(s) but the data to move, it should be possible to do so without explicit copying.  I have no idea is VirtualAlloc can do this, but it's definitely possible.
> >
> > Oh my!
> >
> > On the x86, this sort of implementation would slaughter your performance, as you'd get massive numbers of cache and page misses.
> 
> Is a cache using virtual addresses or physical addresses?

The cache loads in _blocks_ of physical memory, which can be specified using a virtual address or a physical address regardless; "virtual memory" just means you're using a translated address.

> > You could theoretically implement a system on the x86 that would virtualize memory locations arbitrarily, but.... ewww...
> >
> 
> Isn't this what's happening already?

Not arbitrarily sized ones.  If you have a buffer like so:

[slice0,slice1,slice2] = virtual address table [minemineminemine][7Mb][minemineminemine][24b][minemine] = data buffer

Iterate through your virtual array, and you're going to get a hit on the virtual address table, then on slice0, then on slice1 which will also load the 24b segment and probably slice2 for you.

So you get 3 requests to memory, which if they're not already in the cache, will cause a cache miss.  If they're not in memory (swap), they'll cause a page miss.

If you force that memory to stay in one place, it'll all load in 1 shot instead of 3.

If you're curious, a cache miss costs several dozen cycles; while a page miss will cost you thousands.

Additionally, your arbitrary virtual address table takes up alot of space if someone has alot of small dynamic arrays.  Worst case, your virtual table can be double the size of the data itself.

Sincerely,
Dan