Beautiful newsgroup reader, makes a forum obselute but I am against making features only available on the web interface. Maybe smileys and code tags but no further then that.

On 12/8/11 3:30 PM, Vladimir Panteleev wrote:
> On Thursday, 8 December 2011 at 20:15:38 UTC, Andrei Alexandrescu wrote:
>> - Would you consider merging your codebase with
>> d-programming-language.org? That way we'd have a simple deployment
>> target and simpler unified maintenance.
>
> I'm not against moving the project to the D-Programming-Language GitHub
> organization, but I'm not sure what would be achieved by merging the two
> git repositories. Git's "submodule" feature may be more helpful here.

Essentially I'm looking at achieving a streamlined process that would allow me to type "make rsync" and deploy the whole thing. See https://github.com/D-Programming-Language/d-programming-language.org/blob/master/linux.mak. The process of generating an up-to-date site should be 100% automated.

> One thing that needs to be mentioned is that the project's scope is not
> solely the web interface. "DFeed" actually started out as an IRC bot,
> which among other sources, included the newsgroup as an event source for
> IRC notifications. I've refactored the code to use a publish/subscribe
> pattern, so IRC and the local message database (used by the web
> interface) are two of the consumers. A third one will be the front-page
> news widget I was planning to have a go at.

Sounds cool. Perhaps we'll get to modularize things nicely in the future.

>> - I'm thinking something with iframes may work for the above issue,
>> but cross-domain iframes are limited. Could we serve your content off
>> d-p-l.org?
>
> Yes. By far the best method would be subdomains (e.g.
> forum.d-programming-language.org or discussion.d-p-l.org).

That should be doable but I should say I have a mild preference for directories because every subdomain operation would need a roundtrip to the website admin.

>> * This may be a good time to retire a few unused forums such as dtl.,
>> debugger, dwt, ide, and merge their contents with digitalmars.D.
>
> I'm not sure if merging groups is possible on the NNTP level. Doing it
> in DFeed would involve lots of hacks.

Oh, I was thinking of merging at file/archive level on the server, which would involve no work on your part. But let's not spread ourselves too thin.

>> * The style file should be loaded from d-p-l.org such that changes in
>> style will be automatically picked up by the forums page.
>
> I edited the main stylesheet to avoid yet another few !important CSS
> overrides. I think it's best to handle this in the same way as the HTML
> template (automatic or semi-automatic patching).

Anything that's 100% automated.

One thing I remembered from my list of nits - would be great to have votes like on reddit... but first I'd say let's go head to toe through one release of this software. I'll be looking forward to your integration ideas.


Andrei

On Friday, 9 December 2011 at 09:17:11 UTC, Andrei Alexandrescu wrote:
> That should be doable but I should say I have a mild preference for directories because every subdomain operation would need a roundtrip to the website admin.

We need to decide on one URL structure and stick to it. Breaking links is bad.

A few people commented on the app's fast speed, so I'd hate to introduce an artificial bottleneck there.

Note that subdomains are tied to the DNS system, not web hosting. For example, I see that d-p-l.org was registered by Brad and is hosted on PureMagic's name servers, so he would be the person to contact regarding subdomains on d-p-l.org. d-programming-language.org is hosted on ns0/1.digitaldaemon.com and is registered to Walter.

You can make subdomains a CNAME (like an alias). That way, if I move servers, you won't need to bother whoever is responsible for the DNS records.

There are also some security considerations with attachments. We need to either have a separate subdomain for attachments, or forbid viewing them in the browser (which would mean that viewing non-whitelisted attachments, including HTML parts, would need to force a download). Gmail uses googleusercontent.com for attachments, for example.

> Essentially I'm looking at achieving a streamlined process that would allow me to type "make rsync" and deploy the whole thing. See https://github.com/D-Programming-Language/d-programming-language.org/blob/master/linux.mak. The process of generating an up-to-date site should be 100% automated.

How about this: the makefile pings an URL on my server, which runs a script that downloads the files from http://d-programming-language.org/, patches and installs them?

A completely different approach would be to host the program on DigitalMars servers, but I'm worried that maintenance will become very difficult without access to it.

> Oh, I was thinking of merging at file/archive level on the server, which would involve no work on your part. But let's not spread ourselves too thin.

Yeah, that's what I meant. NNTP clients make assumptions regarding pre-existing server state which I don't see a way to maintain while merging group content.

> One thing I remembered from my list of nits - would be great to have votes like on reddit... but first I'd say let's go head to toe through one release of this software. I'll be looking forward to your integration ideas.

This is a complicated issue - mainly due to the possibility of abuse. Reddit has complicated heuristic algorithms for detecting misbehavior and balancing votes, built up from years of experience. Even though their code is open, it's probably not worth the effort. We could make little attempt to police votes, but then they would lose meaning as soon as someone bored enough comes along.

On Friday, 9 December 2011 at 09:17:11 UTC, Andrei Alexandrescu wrote:
> On 12/8/11 3:30 PM, Vladimir Panteleev wrote:
>> On Thursday, 8 December 2011 at 20:15:38 UTC, Andrei Alexandrescu wrote:
> That should be doable but I should say I have a mild preference for directories because every subdomain operation would need a roundtrip to the website admin.

I just realized I missed another simple option: an entirely new domain, for example dpl-forum.net. I can take care of the registration and setup if you like the idea.

On 12/9/11 7:58 CST, Vladimir Panteleev wrote:
> On Friday, 9 December 2011 at 09:17:11 UTC, Andrei Alexandrescu wrote:
>> That should be doable but I should say I have a mild preference for
>> directories because every subdomain operation would need a roundtrip
>> to the website admin.
>
> We need to decide on one URL structure and stick to it. Breaking links
> is bad.

Great.

> A few people commented on the app's fast speed, so I'd hate to introduce
> an artificial bottleneck there.

It's news to me that opting for subdomains vs. directories introduces significant differences in speed. Why?

> Note that subdomains are tied to the DNS system, not web hosting. For
> example, I see that d-p-l.org was registered by Brad and is hosted on
> PureMagic's name servers, so he would be the person to contact regarding
> subdomains on d-p-l.org. d-programming-language.org is hosted on
> ns0/1.digitaldaemon.com and is registered to Walter.

There's also dlang.org. So in order to introduce new domains or do domain operations I need to talk to:

1. Jan Knepper who manages d-programming-language.org

2. Brad Roberts who manages d-p-l.org

3. Oscar Brynolf who owns dlang.org (and whom I don't know)

I can do this, but it's slow, suboptimal, and involves dependencies out of my control.

> You can make subdomains a CNAME (like an alias). That way, if I move
> servers, you won't need to bother whoever is responsible for the DNS
> records.
>
> There are also some security considerations with attachments. We need to
> either have a separate subdomain for attachments, or forbid viewing them
> in the browser (which would mean that viewing non-whitelisted
> attachments, including HTML parts, would need to force a download).
> Gmail uses googleusercontent.com for attachments, for example.

Interesting. I don't understand the security issues involved and why they can't be addressed by the application, but you're the doc. Perhaps we could hardcode only one domain there instead of allowing attachments.xxx.org for three domains.

>> Essentially I'm looking at achieving a streamlined process that would
>> allow me to type "make rsync" and deploy the whole thing. See
>> https://github.com/D-Programming-Language/d-programming-language.org/blob/master/linux.mak.
>> The process of generating an up-to-date site should be 100% automated.
>
> How about this: the makefile pings an URL on my server, which runs a
> script that downloads the files from http://d-programming-language.org/,
> patches and installs them?

Sounds good. So, say I change doc.ddoc and/or style.css. What is the flow through which those will be reflected on your end? What would the URLs for posts look like?

> A completely different approach would be to host the program on
> DigitalMars servers, but I'm worried that maintenance will become very
> difficult without access to it.

I understand. We should find a way that allows you to play with the code and us to play with the other aspects of the website to keep it consistent.


Thanks,

Andrei

On Friday, 9 December 2011 at 15:15:05 UTC, Andrei Alexandrescu wrote:
> Interesting. I don't understand the security issues involved and why they can't be addressed by the application, but you're the doc.

Separate domains are a defense-in-depth kind of thing. The browser
puts up walls between domains for scripting, framing, cookies, locally
stored data, authentication headers, and so on.

While the app can filter this stuff, the separate domain helps be
sure that a bug in the app doesn't lead to a security hole in deployment.


And, of course, if you *want* to allow user scripting, the separate domain
is a must so put at least one hard wall in there.

On Friday, 9 December 2011 at 15:15:05 UTC, Andrei Alexandrescu wrote:
> It's news to me that opting for subdomains vs. directories introduces significant differences in speed. Why?

A domain or subdomain simply points at the server to which the request should go - so, once the (sub)domain name is resolved to an IP address, the requests go directly to the application.

Assuming I continue running the program on my server, the only way to have the program appear under a subdirectory is to make the main website work as a proxy for all requests going to URLs in that "directory". This means that all data has to go to through the main website's server.

> There's also dlang.org. So in order to introduce new domains or do domain operations I need to talk to:
>
> 1. Jan Knepper who manages d-programming-language.org
>
> 2. Brad Roberts who manages d-p-l.org
>
> 3. Oscar Brynolf who owns dlang.org (and whom I don't know)
>
> I can do this, but it's slow, suboptimal, and involves dependencies out of my control.

So you'd like to create subdomains for all domains that lead to the website? I don't see why the necessity, other than to help the occasional person who doesn't remember which domain name we use for the forum subdomain. Note that if you're interested in aliases, they should be redirects, not mirrors (like d-p-l.org and unlike dlang.org).

> Interesting. I don't understand the security issues involved and why they can't be addressed by the application, but you're the doc. Perhaps we could hardcode only one domain there instead of allowing attachments.xxx.org for three domains.

The short version is that HTML attachments may contain malicious JavaScript code, which will have access to all forum cookies. For example, it would be possible to write a worm which would repost itself whenever anyone opened the attachment.

> Sounds good. So, say I change doc.ddoc and/or style.css. What is the flow through which those will be reflected on your end?

I'm thinking about a script that will download the files from the main website, apply whatever patches are necessary, and makes them live. I'm going to have a look at reducing the number of changes to minimum. Perhaps with a few changes on d-p-l.org's side, we can avoid any patching entirely.

> What would the URLs for posts look like?

url = site-root "/post/" stripped-message-id

site-root is currently "http://dfeed.kimsufi.thecybershadow.net/discussion".

stripped-message-id is the Message ID as it appears in the message headers, stripped of the terminal angle brackets, and with special characters percent-encoded using usual URL escape rules.

So, for example with the new domain idea, the URL to your post would be:

http://dpl-forum.net/post/jbt8lp$1v37$1@digitalmars.com

On 12/9/11 9:39 AM, Vladimir Panteleev wrote:
> On Friday, 9 December 2011 at 15:15:05 UTC, Andrei Alexandrescu wrote:
>> It's news to me that opting for subdomains vs. directories introduces
>> significant differences in speed. Why?
>
> A domain or subdomain simply points at the server to which the request
> should go - so, once the (sub)domain name is resolved to an IP address,
> the requests go directly to the application.
>
> Assuming I continue running the program on my server, the only way to
> have the program appear under a subdirectory is to make the main website
> work as a proxy for all requests going to URLs in that "directory". This
> means that all data has to go to through the main website's server.

I see, thanks.

>> There's also dlang.org. So in order to introduce new domains or do
>> domain operations I need to talk to:
>>
>> 1. Jan Knepper who manages d-programming-language.org
>>
>> 2. Brad Roberts who manages d-p-l.org
>>
>> 3. Oscar Brynolf who owns dlang.org (and whom I don't know)
>>
>> I can do this, but it's slow, suboptimal, and involves dependencies
>> out of my control.
>
> So you'd like to create subdomains for all domains that lead to the
> website? I don't see why the necessity, other than to help the
> occasional person who doesn't remember which domain name we use for the
> forum subdomain. Note that if you're interested in aliases, they should
> be redirects, not mirrors (like d-p-l.org and unlike dlang.org).

Great. So we'd only have one domain a la forum.d-programming-language.org.

>> Interesting. I don't understand the security issues involved and why
>> they can't be addressed by the application, but you're the doc.
>> Perhaps we could hardcode only one domain there instead of allowing
>> attachments.xxx.org for three domains.
>
> The short version is that HTML attachments may contain malicious
> JavaScript code, which will have access to all forum cookies. For
> example, it would be possible to write a worm which would repost itself
> whenever anyone opened the attachment.

Interesting, thanks.

>> Sounds good. So, say I change doc.ddoc and/or style.css. What is the
>> flow through which those will be reflected on your end?
>
> I'm thinking about a script that will download the files from the main
> website, apply whatever patches are necessary, and makes them live. I'm
> going to have a look at reducing the number of changes to minimum.
> Perhaps with a few changes on d-p-l.org's side, we can avoid any
> patching entirely.
>
>> What would the URLs for posts look like?
>
> url = site-root "/post/" stripped-message-id
>
> site-root is currently
> "http://dfeed.kimsufi.thecybershadow.net/discussion".
>
> stripped-message-id is the Message ID as it appears in the message
> headers, stripped of the terminal angle brackets, and with special
> characters percent-encoded using usual URL escape rules.
>
> So, for example with the new domain idea, the URL to your post would be:
>
> http://dpl-forum.net/post/jbt8lp$1v37$1@digitalmars.com

I think the domain dfeed.kimsufi.thecybershadow.net should be completely invisible to people who browse the news, google, etc. When googling for posts one should /only/ find forum.d-programming-language.org/xxx URLs. I hope you don't take this the wrong way, but (a) I think having posts under that master domain looks integrated and professional, (b) if you later get hit by a bus or simply move on to other interests the domain doesn't go dark. The second point also raises questions such as who owns the user database etc. Long-term I don't think it's reasonable (and I don't think you want!) to keep user account information on your own server for all installations of dfeed.

On the other hand, I don't want this "invisibility" to affect negatively letting others know about your product. You should put a notice in e.g. the footer a la "Page generated by dfeed", with a link to the product. I suggest you package your code in ways that make it easy to deploy to any other website.

This all leads to the idea that at best you should package dfeed as a classic "product" that you offer via github as an independent project, with releases, updates, bug fixes, and the such. Whenever you update the product you notify your clients that they can update their installations through a classic process (running a script etc). The disadvantage is that clients would run the binaries which means more deployment issues and the such. Also, given that dfeed does quite a few more things than just bridging with NNTP you'd need to do some serious work on packaging.

Please advise on how to best proceed in the short and long term. I think in the short term we can run off your domain (with the visible URLs using top domain d-programming-language.org), and migrate later to a classic product installation configuration that will preserve the URLs.


Andrei

On 12/9/11 4:15 PM, Andrei Alexandrescu wrote:
> There's also dlang.org. So in order to introduce new domains or do
> domain operations I need to talk to:
>
> 1. Jan Knepper who manages d-programming-language.org
>
> 2. Brad Roberts who manages d-p-l.org
>
> 3. Oscar Brynolf who owns dlang.org (and whom I don't know)
>
> I can do this, but it's slow, suboptimal, and involves dependencies out
> of my control.

This strikes me as overly complicated. Why aren't the alias domains just 301-forwarding all HTTP requests, including subdomains (wildcard), to the canonical one?

Also, the amount of time (resp. work) required for updating a domain record/zone file should really be far smaller than the time it takes for the changes to propagate to all the name servers out there. If not, there is something wrong (or the TTL way too low).

David

On 12/9/11 9:06 AM, Vladimir Panteleev wrote:
> On Friday, 9 December 2011 at 09:17:11 UTC, Andrei Alexandrescu wrote:
>> On 12/8/11 3:30 PM, Vladimir Panteleev wrote:
>>> On Thursday, 8 December 2011 at 20:15:38 UTC, Andrei Alexandrescu wrote:
>> That should be doable but I should say I have a mild preference for
>> directories because every subdomain operation would need a roundtrip
>> to the website admin.
>
> I just realized I missed another simple option: an entirely new domain,
> for example dpl-forum.net. I can take care of the registration and setup
> if you like the idea.

At this point I'd say we have too many domains, not too few.

Andrei