hashmap per thread is not an option. The cache may be a few GBs of data, there is no way we can duplicate that data per thread.

Not to mention the start up time when we have to warm up the cache.

On Thursday, 14 November 2013 at 21:31:52 UTC, Jacek Furmankiewicz wrote:
> hashmap per thread is not an option. The cache may be a few GBs of data, there is no way we can duplicate that data per thread.
>
> Not to mention the start up time when we have to warm up the cache.

How often do you change the data? Probably, you should use `immutable` variables.

Jacek Furmankiewicz:

> hashmap per thread is not an option. The cache may be a few GBs of data, there is no way we can duplicate that data per thread.

But is the D garbage collector able to manage efficiently enough associative arrays of few gigabytes? You are not dealing with a GC nearly as efficient as the JavaVM one.

Bye,
bearophile

On Thursday, 14 November 2013 at 21:36:46 UTC, ilya-stromberg wrote:
> On Thursday, 14 November 2013 at 21:31:52 UTC, Jacek Furmankiewicz wrote:
> How often do you change the data? Probably, you should use `immutable` variables.

Customer specific. It may change once a year. It may change multiple times per second for a while, then nothing again for weeks.

Others may do mass loads of business rules, hence do mass changes every few hours.

Next to impossible to predict.

On Thursday, 14 November 2013 at 21:39:53 UTC, bearophile wrote:
> Jacek Furmankiewicz:
>
>> hashmap per thread is not an option. The cache may be a few GBs of data, there is no way we can duplicate that data per thread.
>
> But is the D garbage collector able to manage efficiently enough associative arrays of few gigabytes? You are not dealing with a GC nearly as efficient as the JavaVM one.
>

Well, these are the types of questions I have as a Java veteran
who is having a first look at D after the recent Facebook announcement.

By now I have a decent idea of where most of the new languages (Go has same issues, for the most part) come up short when compared to Java's very mature SDK, so that is usually where I start probing first.

Sorry :-(

Jacek Furmankiewicz:

> Well, these are the types of questions I have as a Java veteran
> who is having a first look at D after the recent Facebook announcement.
>
> By now I have a decent idea of where most of the new languages (Go has same issues, for the most part) come up short when compared to Java's very mature SDK, so that is usually where I start probing first.
>
> Sorry :-(

The development of the Java language, its GC, Oracle JVM, standard library (and its IDEs, etc) have received tons of money, time, and hours of work, so it's not strange Java is "better" than D.

On the other hand different languages are fitter for different purposes. I like D a lot, but programmers should choose languages wisely, and Java is a wiser choice for several commercial purposes. If you rewrite Minecraft from Java to D I suspect you produce a game that's faster and with a shorter source code, while keeping most of its programmer-friendly nature and its coding safety, despite the current limits of the D GC.

If you want to use D try to find niches where it could be useful and fit. I am using D where it's better than equivalent Java code. Today Python is used a lot, but in many cases it's not replacing equivalent Java code.

Probably you can replace some Java code with Scala code.

Bye,
bearophile

True.

While looking a D, I am just trying to focus on the parts which I know would be a showstopper for us on day one...and this particular issue is it.

I do like D a lot as well from what I've seen so far.

Regarding the GC, I've seen some slides on DConf about other garbage collectors available. Is there any resource/tutorial that shows how you can swap out the default GC for those alternative implementations?

On 11/14/2013 01:36 PM, ilya-stromberg wrote:
> On Thursday, 14 November 2013 at 21:31:52 UTC, Jacek Furmankiewicz wrote:
>> hashmap per thread is not an option. The cache may be a few GBs of data, there is no way we can duplicate that data per thread.
>>
>> Not to mention the start up time when we have to warm up the cache.
>
> How often do you change the data? Probably, you should use `immutable` variables.
>
Immutable variables are nice when they can be used.  Often, however, they can't.

I think that for the "concurrent hashmap" the best answer is probably to run the map in a thread, with message passing access whether for read or write.  And I wouldn't be surprised if that's how Java's concurrent hashmap is implemented under the covers. (OTOH, I haven't ever debugged such a setup.  Someone who has may have a better answer.)

-- 
Charles Hixson

On Thursday, 14 November 2013 at 22:12:10 UTC, Jacek Furmankiewicz wrote:
> On Thursday, 14 November 2013 at 21:36:46 UTC, ilya-stromberg wrote:
>> On Thursday, 14 November 2013 at 21:31:52 UTC, Jacek Furmankiewicz wrote:
>> How often do you change the data? Probably, you should use `immutable` variables.
>
> Customer specific. It may change once a year. It may change multiple times per second for a while, then nothing again for weeks.
>
> Others may do mass loads of business rules, hence do mass changes every few hours.
>
> Next to impossible to predict.

You can use `immutable` variables. It allows you to share the data without any synchronization. Like this:

class MyData
{
   int data1;
   string data2;

   //creates new object
   this(int data1, string data2)
   {
      this.data1 = data1;
      this.data2 = data2;
   }

   //modify the data
   immutable(MyData) editData(int i) const
   {
      //copy this object - we can't change immutable variables
      MyData dataCopy = new MyData(this.data1, this.data2)

      //modify the data copy
      dataCopy.data1 += i;

      //assume that `dataCopy` is immutable
      return cast(immutable(MyData)) dataCopy;
   }
}

shared myMap;

//map implementation
synchronized class MyMap
{
   HashMap!(int, immutable(MyData)) map;

   void foo()
   {
      map[1] = new immutable MyData(1, "data");
   }

   void bar()
   {
      map[1] = map[1].editData(5);
   }
}

//init map
shared static this()
{
   myMap = new MyMap();
}

void main()
{
   myMap.foo();
   myMap.bar();
}

15-Nov-2013 03:35, Charles Hixson пишет:
> On 11/14/2013 01:36 PM, ilya-stromberg wrote:
>> On Thursday, 14 November 2013 at 21:31:52 UTC, Jacek Furmankiewicz wrote:
>>> hashmap per thread is not an option. The cache may be a few GBs of
>>> data, there is no way we can duplicate that data per thread.
>>>
>>> Not to mention the start up time when we have to warm up the cache.
>>
>> How often do you change the data? Probably, you should use `immutable`
>> variables.
>>
> Immutable variables are nice when they can be used.  Often, however,
> they can't.
>
> I think that for the "concurrent hashmap" the best answer is probably to
> run the map in a thread, with message passing access whether for read or
> write.

Would be slow unless batched. At least in D message passing involves locking/unlocking a queue of messages. Sending back and forth you get 2 lock-wait-unlock and correspondingly context switches.

> And I wouldn't be surprised if that's how Java's concurrent
> hashmap is implemented under the covers. (OTOH, I haven't ever debugged
> such a setup.  Someone who has may have a better answer.)

As stated in Oracle's documentation somewhere it's implemented with fine grained locking (a lock per bucket of a hash map), some operations still lock the whole map. Rehashing still locks the whole thing I bet.

-- 
Dmitry Olshansky