ЖЖ-логия: опять проблемы

[avva]

Комменты не работают, ага. И вообще все страницы, кончающиеся на .bml. А всё остальное работает вроде.

Всё, без запасного администратора где-нибудь в европейских часовых зонах нельзя больше жить. Постараюсь изо всех сил Брада в этом убедить, когда он проснётся.

P.S. Прогноз такой: комменты не будут работать примерно следующие 7 часов. Потом заработает всё.

Comments

[avva.livejournal.com]

Hmmm.

Can you present more clearly your vision of what a distributed platform means, in context of LJ?

Let's try to keep separate two different applications of distribution, so as to avoid confusion (as I've been trying to do in our philosophical discussions, and I apologise, BTW, for neglecting them somewhat during the last few days ;)):

1. Distribution to lighten the load off servers.
2. Distribution to enable very different kinds of LJ-code-based entities to interoperate.

What's different is the intent: in the first case the intent is to allow the main LJ site to grow, in the second it's to allow all kinds of LJ-code-based sites (not affiliated with lj.com, just using the open-source LJ code) to interoperate, use each other's users in friends views, etc.

The second kind is what Brad and I were talking about. You seem to focus on the first kind, I think: complete distribution and independence of LJ servers from each other is necessary to survival of LJ, in your opinion.

It may be true, but I don't see that as self-evident, currently. As you probably know, LJ *is* employing a lot of distribution currently, distributing database access over many servers and web access over many more servers. It's true that this distribution is not complete, i.e. there's a single database and there's a master machine holding all of it. But the master machine is not a bottleneck and historically hasn't been; almost all the problems LJ faced were *not* due to traffic or access bottleneck on the master machine. They were usually over replication issues, or web slave failures, etc.; stuff that can fail just as well if you have completely independent servers (since they still will have to replicate/share information, and web servers will still fail sometimes or be overloaded).

So, can you be more specific about what you see as necessary, in light of the above?

[jsn.livejournal.com]

I apologise, BTW, for neglecting them somewhat during the last few days ;)

Tew, nevermind :) That's always very interesting but rarely deadly urgent.

1. Distribution to lighten the load off servers.
2. Distribution to enable very different kinds of LJ-code-based entities to interoperate.
...
You seem to focus on the first kind, I think: complete distribution and independence of LJ servers from each other is necessary to survival of LJ, in your opinion.

That's right, that's what i think.

It may be true, but I don't see that as self-evident, currently. As you probably know, LJ *is* employing a lot of distribution currently, distributing database access over many servers and web access over many more servers.

Yeah, i know that.

But the master machine is not a bottleneck and historically hasn't been; almost all the problems LJ faced were *not* due to traffic or access bottleneck on the master machine. They were usually over replication issues, or web slave failures, etc.; stuff that can fail just as well if you have completely independent servers (since they still will have to replicate/share information, and web servers will still fail
sometimes or be overloaded).

You already have probably tens of boxes running web frontends, right? If the things are going to develop the way they currently do, master machine will eventually become a bottleneck. The same tendency about traffic and everything. I mean, on the long enought time scale every singularity (like master server) will become a bottle neck.
Actually, i think that webslave failures also mean a bottleneck of some special kind: there probably *too many* webslaves, and thus they are harder to control perfectly. If, in other hand, we could have a bunch of relatively independent serving locations, under different authorities and stuff, each of these (smaller) locations would be much easier to control and maintain.
Probably i also should mention another kind of bottleneck situation: like a power failure or even planned maintenance procedures.

[jsn.livejournal.com]

BTW, could you please provide me with a description of the current technical LJ problems? I read lj_dev and lj_maintenance for a while, but probably failed to extract much of internals. Obvious guess would be:
1. Slave mysql servers are unstable under the heavy load of sql requests ("select"s).
2. Frontend webservers run out of cpu || ram || maxproc || file descriptors while parsing sql replies and processing templates.