On Wednesday, 14 October 2015 at 15:49:20 UTC, David DeWitt wrote:
> I agree but the quora question ask why it is popular despite being slow and this is the reason.  If you are doing tasks that are computationally expensive in Python then yes it will be slow but Python is popular largely because of their web frameworks and support.

If it is slow then one can configure the load balancer to give more resources to that URL, or even run it on an insanely fast compute node with lots of memory for caching/memoing.

Or just implement that single high frequency URL handler in a different language if it scaling up resources become costly.

> You pick the right tool for the right job maybe D and maybe Python and this doesn't mean your results will be slow.

Yes, you pick the right tool for the job, and in a web context that means picking a tool that has infrastructure support and makes it possible to cover future unexpected needs in a cost efficient and timely manner where you offload as much maintenance costs as possible to the infrastructure maintainers rather than the application maintainers.

I use AppEngine and Python is by far the most pleasant choice when choosing between Java, Go, Php and Python. For many very good reasons. But you can use whatever language you want for any particular URL so using Python is not a lock-in solution.

Using Rust or D for web developement do imply lockin on many dimensions (infrastructure, libraries, people, maintenance) compared to Python, Java and Ruby.

On Wednesday, 14 October 2015 at 05:42:12 UTC, Ola Fosheim Grøstad wrote:
> On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc wrote:
>> https://www.quora.com/Why-is-Python-so-popular-despite-being-so-slow
>> Andrei suggested posting more widely.
>
> That's flaimbait:
>
> «Many really popular websites use Python. But why is that? Doesn't it affect the performance of the website?»
>
> No. Really popular websites use pre-generated content / front end caches / CDNs or wait for network traffic from distributed databases.

really popular portals, news sites? yes. really popular websites? nope. like booking.com, airbnb.com, reddit.com are popular websites that have many parts which have to be dynamic and responsive as hell and they cannot use caching, pre-generated content, etc.

using python affect the performance of your website. if you were to use ruby or php your web app would be slower than it's python version. and python version would be slower than go or d version.

On Wednesday, 14 October 2015 at 18:37:40 UTC, Mengu wrote:
> websites? nope. like booking.com, airbnb.com, reddit.com are popular websites that have many parts which have to be dynamic and responsive as hell and they cannot use caching, pre-generated content, etc.

They can if they know what they are doing. E.g. Reddit can push indexes to their CDN (Cloudflare?) for commonly requested topics.

There are many many layers and options for caching or delaying recomputation (e.g. eventual consistency strategies).

> using python affect the performance of your website.

No. You can do a lot of computation in Python in 10ms. Using Python over C++/Go affects the number of instances, not the performance.

You don't handle one request at the time, so if you use Python you handle fewer concurrent requests than in Go. That is the only difference unless you are doing something real time (like a game server).

On Wednesday, 14 October 2015 at 18:55:28 UTC, Ola Fosheim Grøstad wrote:
> On Wednesday, 14 October 2015 at 18:37:40 UTC, Mengu wrote:
>> websites? nope. like booking.com, airbnb.com, reddit.com are popular websites that have many parts which have to be dynamic and responsive as hell and they cannot use caching, pre-generated content, etc.
>
> They can if they know what they are doing. E.g. Reddit can push indexes to their CDN (Cloudflare?) for commonly requested topics.


And that is exactly what reddit does:

«
cache-control:max-age=0, must-revalidate
cf-cache-status:HIT
cf-ray:23558623ce9b231e-FRA
»

As you can see my request for /r/programming at reddit.com found a hit in their CDN cache Cloudflare...

CDNs may allow explicit preloading and removal of outdated resources from caches (contrary to HTTP).

On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc wrote:
> https://www.quora.com/Why-is-Python-so-popular-despite-being-so-slow
> Andrei suggested posting more widely.

I am coming at D by way of R, C++, Python etc. so I speak as a statistician who is interested in data science applications.

It's about programmer time. You have to weight the time it takes you to do the task in each programming language, if you are doing statistical analysis now, R and Python come out streaks ahead.

The scope roughly speaking is Research -> Deployment. R and Python sit on the research side, and Python/JVM technologies sit on the deployment side (broadly speaking). The question is where does D sit? What should D's data science strategy be?

To sit on the deployment side, D needs to grow it's big data/noSQL infrastructure for a start, then hook into a whole ecosystem of analytic tools in an easy and straightforward manner. This will take a lot of work!

I believe it is easier and more effective to start on the research side. D will need:

1. A data table structure like R's data.frame or data.table. This is a dynamic data structure that represents a table that can have lots of operations applied to it. It is the data structure that separates R from most programming languages. It is what pandas tries to emulate. This includes text file and database i/o from mySQL and ODBC for a start.

2. Formula class : the ability to talk about statistical models using formulas e.g. y ~ x1 + x2 + x3 etc and then use these formulas to generate model matrices for input into statistical algorithms.

3. Solid interface to a big data database, that allows a D data table <-> database easily

4. Functional programming: especially around data table and array structures. R's apply(), lapply(), tapply(), plyr and now data.table(,, by = list()) provides powerful tools for data manipulation.

5. A factor data type:for categorical variables. This is easy to implement! This ties into the creation of model matrices.

6. Nullable types makes talking about missing data more straightforward and gives you the opportunity to code them into a set value in your analysis. D is streaks ahead of Python here, but this is built into R at a basic level.

If D can get points 1, 2, 3 many people would be all over D because it is a fantastic programming language and is wicked fast.

On Wednesday, 14 October 2015 at 22:11:56 UTC, data pulverizer wrote:
> On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc wrote:
>> https://www.quora.com/Why-is-Python-so-popular-despite-being-so-slow
>> Andrei suggested posting more widely.
>
>
> I believe it is easier and more effective to start on the research side. D will need: [snip]

Great list, but tons of work!

On Thursday, 15 October 2015 at 02:20:42 UTC, jmh530 wrote:
> On Wednesday, 14 October 2015 at 22:11:56 UTC, data pulverizer wrote:
>> On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc wrote:
>>> https://www.quora.com/Why-is-Python-so-popular-despite-being-so-slow
>>> Andrei suggested posting more widely.
>>
>>
>> I believe it is easier and more effective to start on the research side. D will need: [snip]
>
> Great list, but tons of work!

A journey of a thousand miles ...

I tried to start creating a data table type object by investigating variantArray: http://forum.dlang.org/thread/hhzavwrkbrkjzfohczyq@forum.dlang.org but hit the snag that D is a static programming language and may not allow the kind of behaviour you need for creating the same kind of behaviour you need in data table - like objects.

I envisage such an object as being composed of arrays of vectors where each vector represents a column in a table as in R - easier for model matrix creation. Some people believe that you should work with arrays of tuple rows - which may be more big data friendly. I am not overly wedded to either approach.

Anyway it seems I have hit an inherent limitation in the language. Correct me if I am wrong. The data frame needs to have dynamic behaviour bind rows and columns and return parts of itself as a data table etc and since D is a static language we cannot do this.

On Thu, 2015-10-15 at 06:48 +0000, data pulverizer via Digitalmars-d- learn wrote:
> 
[…]
> A journey of a thousand miles ...

Exactly.

> I tried to start creating a data table type object by
> investigating variantArray:
> http://forum.dlang.org/thread/hhzavwrkbrkjzfohczyq@forum.dlang.org
>  but hit the snag that D is a static programming language and may not
> allow the kind of behaviour you need for creating the same kind of
> behaviour you need in data table - like objects.
> 
> I envisage such an object as being composed of arrays of vectors where each vector represents a column in a table as in R - easier for model matrix creation. Some people believe that you should work with arrays of tuple rows - which may be more big data friendly. I am not overly wedded to either approach.
> 
> Anyway it seems I have hit an inherent limitation in the language. Correct me if I am wrong. The data frame needs to have dynamic behaviour bind rows and columns and return parts of itself as a data table etc and since D is a static language we cannot do this.

Just because D doesn't have this now doesn't mean it cannot. C doesn't have such capability but R and Python do even though R and CPython are just C codes.

Pandas data structures rely on the NumPy n-dimensional array implementation, it is not beyond the bounds of possibility that that data structure could be realized as a D module.

Is R's data.table written in R or in C? In either case, it is not beyond the bounds of possibility that that data structure could be realized as a D module.

The core issue is to have a seriously efficient n-dimensional array that is amenable to data parallelism and is extensible. As far as I am aware currently (I will investigate more) the NumPy array is a good native code array, but has some issues with data parallelism and Pandas has to do quite a lot of work to get the extensibility. I wonder how the R data.table works.

I have this nagging feeling that like NumPy, data.table seems a lot better than it could be. From small experiments D is (and also Chapel is even more) hugely faster than Python/NumPy at things Python people think NumPy is brilliant for. Expectations of Python programmers are set by the scale of Python performance, so NumPy seems brilliant. Compared to the scale set by D and Chapel, NumPy is very disappointing. I bet the same is true of R (I have never really used R).

This is therefore an opportunity for D to step in. However it is a journey of a thousand miles to get something production worthy. Python/NumPy/Pandas have had a very large number of programmer hours expended on them.  Doing this poorly as a D modules is likely worse than not doing it at all.

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On Wednesday, 14 October 2015 at 18:37:40 UTC, Mengu wrote:
> On Wednesday, 14 October 2015 at 05:42:12 UTC, Ola Fosheim Grøstad wrote:
>> On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc wrote:
>>> https://www.quora.com/Why-is-Python-so-popular-despite-being-so-slow
>>> Andrei suggested posting more widely.
>>
>> That's flaimbait:
>>
>> «Many really popular websites use Python. But why is that? Doesn't it affect the performance of the website?»
>>
>> No. Really popular websites use pre-generated content / front end caches / CDNs or wait for network traffic from distributed databases.
>
> really popular portals, news sites? yes. really popular websites? nope. like booking.com, airbnb.com, reddit.com are popular websites that have many parts which have to be dynamic and responsive as hell and they cannot use caching, pre-generated content, etc.
>
> using python affect the performance of your website. if you were to use ruby or php your web app would be slower than it's python version. and python version would be slower than go or d version.

Yep. This occurred to me too. Sorry Ola, but I think you don't know how sausages are made. Do you really think that all the websites out there are performance tuned by network programming specialists? You'd be surprised!

On Thursday, 15 October 2015 at 07:57:51 UTC, Russel Winder wrote:
> lot better than it could be. From small experiments D is (and also Chapel is even more) hugely faster than Python/NumPy at things Python people think NumPy is brilliant for. Expectations

Have you had a chance to look at PyOpenCL and PYCUDA?