Re: Message Pushing and TCP Transport

[Dave Crocker <dcrocker@brandenburg.com>]

At 09:18 AM 8/23/2001, Daniel Manley wrote:
>>And you are placing a load on the server that the registry operator 
>>should not appreciate very much.
>
>Ok, let's say push replaces poll.  The registrars will still have to ping 
>the registry to keep persistent connections alive.

You seemed to have missed my point, that such pinging is not a good 
thing.   If the server wants to be shutting down inactive connections, then 
the folks running the server are not going to appreciate your circumventing 
their management policies.  They will then add mechanisms for disabling the 
ping, in whatever form it is commonly achieved.

So, you want to believe that polling will happen no matter what.  I am 
trying to point out the negative side of such an activity.


>Thinking also of a possible server design:  a multi-threaded or -processed 
>server is working for many clients.

yup.  that's why the pinging creates an aggregate overhead that server 
operators will not appreciate.


>clientA issues a transfer for an object and places a message on an 
>internal queue for clientB.

Huh?  ClientA places a message on a queue for ClientB.  What ARE you 
talking about?


>any of the threads working for clientB has to read the the queue when it 
>has time (i.e. "poll" the queue) and then pushes the message to the 
>registrar.

You have a strang model of server and thread implementation.  The idea that 
it requires internal polling, no matter what, is simply wrong.


>... somewhere along the line, a process or thread will have to poll 
>something to see if there's anything to do.

wrong.


>>Actually, no.  Pushing has no wasted effort, unlike polling.  Hence the 
>>overhead of push is only incurred when there is something to push.
>
>Except for queuing when the message can't be delivered taken right away.

queuing is an entirely separate bit of mechanism and overhead, with equal 
overhead on top of push or pull.

the point that is being missed is polling permits only a subset of the 
behaviors and efficiencies that push permits.  Push demonstrates vastly 
better scaling properties.

Internet application protocols have intentionally chosen simplistic 
transaction models, in the past.  That has been fine for small scale 
activities.  As we try to permit serious, large-scale applications, we need 
to use the kinds of mechanisms that modern, large-scale transaction systems 
use.

And have used for a very long time.  (Which is a semi-polite way of 
suggesting that most of us need to realize that we are amateurs in the 
field of transaction processing. Amateurs usually find the work of 
professionals to be "more complicated" than we are used to...)

(For reference, please note the "we".  That is not just for politeness.  I 
very much include myself.  The difference, right now, is that I have been 
trying to appreciate the nature of this other area of expertise, rather 
than to automatically reject it because it makes protocol work a bit more 
complicated.)


>>So I have no idea how your are calculating a "multiplied" overhead.
>>Feel free to provide detail for your claim.
>
>I haven't invested any research time to back my claim.  I was just using 
>common sense.




>Let's say that 10 registrars are connected to the registry.  In a polling 
>system, the registry sits back and waits for requests from the 
>servers.  Much or most of the CPU time is dedicated to serving requests.

and pings and/or polling requests for non-existent data.  so, it is 
spending time for each client, doing unproductive overhead.


>In a pushing system, the registry has to be proactive and take up CPU 
>cycles from serving requests to push notifications to the registrars.

The server needs to get the data to the client, no matter what.  You seem 
to feel that sending it immediately, rather having to wait an arbitrary 
amount of time, somehow incurs onerous overhead.  It doesn't.

What it DOES do is a) impose delay, and b) force extra overhead from 
unproductive polling.


>That might not seem so bad at first, but, as I mentioned above in this 
>email, if the registrar is not connected or if an acknowledgment is not 
>returned, the registry has to take the time to queue this message 
>(dropping it then and there doesn't seem like a good idea) and try to 
>deliver it later.  Now the server  is performing housework for the client, 
>taking more time away from serving requests.
>
>This is where the "multiplied" part comes in:  the registry is now 
>performing delivery work for X registrars while the registrar clients 
>happily sit around waiting for something from the server while still 
>inundating the server with more requests.

Frankly I can't figure out the processing model that you are describing, 
since you seem to view queuing as intimately involved in the i/o model, 
although it does not need to be.


>What happens if there is a persistent queue for a particular registrar? If 
>there are a bunch of old (but not yet expired) transfer notifications 
>waiting around and a notification of low balance shows up, which do you 
>try to send the transfers first?  Do you implement a prioritized 
>queue?  If you do, should we include the message priorities in the 
>protocol spec?  Then registrars have to additionally deal with message 
>priorities.

None of this is relevant to the choice of mechanism that sends the "next" 
entry from the queue.

d/

----------
Dave Crocker  <mailto:dcrocker@brandenburg.com>
Brandenburg InternetWorking  <http://www.brandenburg.com>
tel +1.408.246.8253;  fax +1.408.273.6464