Re: Message Pushing and TCP Transport

[Daniel Manley <dmanley@tucows.com>]

(my apoligies to making these a little long, but it's required to 
describe the scenarios that polling and pushing could create.  the last 
couple of paragraphs contain questions and issues I have with pushing)

Dave Crocker wrote:

> At 05:23 PM 8/22/2001, Daniel Manley wrote:
>
>>> The usual response is "polling" but that means constantly polling, 
>>> and in this case constantly polling for something that is typically 
>>> not present.  Hence, substantial overhead, for negative benefit.
>>
>>
>> If you're using poll to ping a connection to keep it alive, then 
>> you're already doing it anyhow.
>
>
> And you are placing a load on the server that the registry operator 
> should not appreciate very much. 

Ok, let's say push replaces poll.  The registrars will still have to 
ping the registry to keep persistent connections alive.  Poll, IMHO, 
seemed like a very good candidate for pinging.  Without it, registrars 
will likely either turn to "hello" or "domain:check", both having no 
data modification effects on data.  

So then, the registrars are still pinging to stay alive and the registry 
is now also spontaneously sending notifications to the registrar.  The 
registrar then has to be ready to perform any of three actions at any 
time:  send requests and read the responses, ping the connection in 
periods of inactivity and read the in-bound pipe for notifications from 
the server.

Thinking also of a possible server design:  a multi-threaded or 
-processed server is working for many clients.  clientA issues a 
transfer for an object and places a message on an internal queue for 
clientB.  any of the threads working for clientB has to read the the 
queue when it has time (i.e. "poll" the queue) and then pushes the 
message to the registrar.  

I imagine my argument is getting old and worn out but... somewhere along 
the line, a process or thread will have to poll something to see if 
there's anything to do.  You could try to minimize that by making all 
the threads in the registry cognisant of each other so that these 
notifications could be pushed around, but as Scott says, if the message 
is undeliverable or unacknowledge, then the registry would likely want 
to queue it and then poll that queue later.

>
>> So I don't think there's a negative benefit. Pushing pushes the 
>> substantial overhead *multiplied* by each registrar client to the 
>> server, who's already overworked anyway.
>
>
> Actually, no.  Pushing has no wasted effort, unlike polling.  Hence 
> the overhead of push is only incurred when there is something to push. 

Except for queuing when the message can't be delivered taken right away.

>
>
> So I have no idea how your are calculating a "multiplied" overhead.  
> Feel free to provide detail for your claim. 

I haven't invested any research time to back my claim.  I was just using 
common sense.  

Let's say that 10 registrars are connected to the registry.  In a 
polling system, the registry sits back and waits for requests from the 
servers.  Much or most of the CPU time is dedicated to serving requests. 
 [most of this paragraph  can be skipped, the last sentence sums it up] 
Registrars process requests:  object registrations, modifications, 
transfers, etc... and pinging to keep idle connections alive.  If the 
registry has something to tell the registrar out of normal 
requests/responses (like a pending transfer notification), it places the 
message and accompanying data on a queue for the registrar to pick up at 
any time.  Some might contain time-sensitive data, and the registrar, in 
it's agreement with the registry *and* the suggestion from the protocol 
spec, knows that it should check this queue periodically to pick up 
messages.   So the registry in its role of serving registrars will 
retrieve queued messages if the requests ask for that.

In a pushing system, the registry has to be proactive and take up CPU 
cycles from serving requests to push notifications to the registrars. 
 That might not seem so bad at first, but, as I mentioned above in this 
email, if the registrar is not connected or if an acknowledgment is not 
returned, the registry has to take the time to queue this message 
(dropping it then and there doesn't seem like a good idea) and try to 
deliver it later.  Now the server  is performing housework for the 
client, taking more time away from serving requests.  

This is where the "multiplied" part comes in:  the registry is now 
performing delivery work for X registrars while the registrar clients 
happily sit around waiting for something from the server while still 
inundating the server with more requests.

What happens if there is a persistent queue for a particular registrar? 
 If there are a bunch of old (but not yet expired) transfer 
notifications waiting around and a notification of low balance shows up, 
which do you try to send the transfers first?  Do you implement a 
prioritized queue?  If you do, should we include the message priorities 
in the protocol spec?  Then registrars have to additionally deal with 
message priorities.

It just seems black and white to me:  the protocol says the registrar 
should poll for messages, the registry/registrar agreements and 
certification tests should enforce that, if messages expire or build up, 
then they could unloaded and an out-of-band report could be sent.  The 
registrar could choose to process that with a program, but it's likely 
that many of those messages would be useless or meaningless by then anyhow.

Dan