Re: publish/subscribe data between services

From: Buck Evan <buck_at_yelp.com>
Date: Tue, 8 Mar 2016 09:38:05 -0800

On Tue, Mar 8, 2016 at 1:07 AM, Laurent Bercot <ska-supervision_at_skarnet.org>
wrote:

> On 08/03/2016 02:41, Buck Evan wrote:
>
>> At this point I have a design issue. How do services become aware of each
>> others' port numbers? I've considered using ftrig or envdirs or both to
>> represent and notify this data. Still, I don't see how I can keep the
>> overall state consistent in the case of a single service's restart; I
>> don't
>> believe in general I can assume that its old port number will still be
>> available, which means that I will sometimes need to restart any other
>> services in the "playground" (collection of supervised developer services)
>> that know this port number. This implies dependency resolution, which is
>> too complex for my taste.
>>
>
> With a distributed port number allocation mechanism (i.e. what the kernel
> does) and no dependency resolution, you're going to have race conditions,
> there's no way around it.
>
> The only "correct" way to proceed is to have a centralized authority on
> what port belongs to what service, with proper serialization: a daemon that
> maintains a map of the associations, that updates it when it changes, and
> that services go through to query an association (or request a new port
> number), so it can do the operation in a guaranteed order that avoids
> race conditions


Is there any tooling to help me implement this without losing my mind?
It sounds a bit like zookeeper.

- but you'll always have the old race "A queries port of B,
> gets it, and B dies at that moment, and port gets reused",


I think I'm ok ignoring this case, as long as something notices that things
are inconsistent, sends the necessary restarts.
Eventual consistency is ok here as long as it's relatively quick.


> which you may
> mitigate by not using SO_REUSEADDR.
>



Many servers use SO_REUSADDR as a matter of course without making it
configurable.
I'm not sure I can control that in general.
Received on Tue Mar 08 2016 - 17:38:05 UTC