What about a feathers-µservice instead

[claustres]

Starting working on a FearthersJS-based micro-service architecture I wonder if this module could not be a little bit extended to cover this use case. Indeed it actually already covers two main features IMHO:

• connect (or plan to connect) FeathersJS with different "message bus" technologies (AMQP, Kafka, etc.)
• perform real-time dispatch of any service event on the bus
  It would be pity to develop again this in another module dedicated to micro-services.

So my proposal is to extend this module:

• add a small service proxy class that simply proxy requests/responses to a remote service using feathers-client (or the message bus ?)
• add an event in the Feathers app when a new service is created so this event would be propagated on the bus
• each node receiving this event will create a service proxy to the remote service so that any incoming request for the remote service will be dispatched to it

Let me know what you think and If you prefer to keep this module focused. In this case I would do something on my own. Thanks.

[claustres]

All things considered I believe using the message bus to proxy service requests is the best because it does not require to know something about the infrastructure topology while using feathers client will require you to know the IP adresses of the different machines. The knowledge of the infrastructure is delegated to the message bus/broker and you simply connect to its entry point.

[daffl]

I really like the idea but I'm wondering if feathers-sync is the right place for it. This module is intended for running multiple instance (e.g. Heroku dynos) of the same application to propagate all real-time events which I think is a different way of scaling.

Should we combine those two approaches into feathers-distributed? We've been debating for a while now to add message queue providers (and clients just like the ones for REST and Socket.io) so the additional piece then would be the broker that has the information on where the original services live.

[claustres]

Like the idea of feathers-distributed, distribution is actually the key point (by the way I initially wanted to start something with http://nats.io/ or http://nsq.io/). The pun would push something more like feathers-flock or feathers-swarm ;-)

Although the primary purpose of the feathers-sync module is to replicate a similar app I hardly see any limitations to synchronize different apps. I'd like to add that if the only purpose is to scale websockets there is no need to duplicate the running services if a single one can handle the workload. So you could use the module to simply sync a set of "gateways" handling socket requests for a running service instance that performs the "real" work.

So if I understand well you propose to add an abstraction over a messaging system so that we could imagine something similar to REST/Socket:

import amqp from `feathers-amqp`
...
app.configure(amqp({ brokerUrl: xxx }))

What I imagined was adding a similar set of events than the ones that already exist after a service has been called (created, etc.) for service requests (create, etc.). The purpose of the provider modules like feathers-amqp would then be to manage the mapping between incoming/outgoing Feathers events to incoming/outgoing provider messages:

Code would look something like:

amqp.bindOutgoing({
  eventIn: 'created', // Optional, defaults to * when just service is given
  serviceIn: 'xxx',
  topicOut: 'xxx' // Optional, defaults to service name
})
amqp.bindIncoming({
  topicIn: 'xxx',
  serviceOut: 'xxx' // Optional, defaults to topic name
})

Indeed one purpose of such brokers is also to make newly connected apps react to an existing set of events coming from existing applications.

Then each Feathers app should have its own service registry (ie the app), local/remote services should be callable transparently. The low-level event binding will be used by the services, eg:

// Remote services accept incoming requests from the broker
service.setup = (app, path) => {
  amqp.bindIncoming({
    topicIn: path,
    serviceOut: path
  })
}
// Proxy services emit requests to the broker
proxyService.setup = (app, path) => {
  amqp.bindOutgoing({
    serviceIn: path,
    topicOut: path
  })
}

The tricky part (v2) would be to also distribute hooks. Indeed because they are first class citizens like services to handle business logic you often face the situation where a module/app wants to push a hook on a service owned by another module/app.

PS Don't know if a client part as mentioned makes sense because most messaging systems already have such client modules, but I probably missed your point.

[claustres]

Just to let you know I started something here https://github.com/kalisio/feathers-distributed. I actually found https://github.com/dashersw/cote and it looks to be what I was looking for for a while. Started a couple of tests to illustrate the main concepts and it seems to work. Don't know if cote is the right tool for Feathers because it relies on a specific network tool but it eases to test what I was thinking about. Actually it can also uses redis for service discovery if you don't have such a network plugin so why not.

What is done in the initialization function of the plugin https://github.com/kalisio/feathers-distributed/blob/master/src/index.js is the following:

• each Feathers app creates a publisher to dispatch its locally registered services to other nodes.
• each Feathers app creates a subscriber to be aware of any remotely registered service from other nodes.

What is done by overriding app.use (but you might be aware of a better solution) is the following:

• each "local" Feathers service creates a responder to handle incoming requests from other nodes.
• each "local" Feathers service creates a publisher to dispatch service-level events to other nodes.

What is done when the app is aware of a new remotely registered service is the following:

• add a local Feathers service acting as a proxy by creating a requester to send incoming requests to other nodes.
• this Feathers service also creates a subscriber to be aware of service-level events coming from other nodes.

To see everything work together look at https://github.com/kalisio/feathers-distributed/blob/master/test/index.test.js.

Let me know what you think about it.

[subodhpareek18]

@claustres looks very interesting at first glance

[claustres]

We are seeking for help to make this production ready https://github.com/kalisio/feathers-distributed, sorry no paid job for now 😉 What we'd like to have first is some feedback on deployment in cloud providers, of course any PR is welcome ! We added an example to make things easier for testing https://github.com/kalisio/feathers-distributed/tree/master/example, it could be easily extended for real apps. Thanks