Add support for Flink and custom sinks.

[kl0u]

Currently the Runner has no support for sinks, apart from the console one, which is just for testing and debugging. We have to create a wrapper for the Dataflow sinks and the Flink ones.

[bakey]

@kl0u
What you mean is it that we need to add one FileBaseSink in Runner? Or any other forms support for sinks?

[kl0u]

Hello @bakey !

Thanks for wanting to get involved in the Beam Flink Runner.
So, concerning the sinks, this issue exists because also the Beam (or Google Dataflow) SDK version at the time we created the Flink Runner, did not support any Unbounded Sinks. Write operations were only allowed for batch workloads.

What would be interesting is to build support in the Beam Flink Runner for Flink Streaming sinks. This translates into creating a wrapper for sinks that implement the RichSinkFunction.

Cheers,
Kostas

[bakey]

@kl0u
Is it that means we have to add unbounded sinks support in Beam first , and then add translator in the runner to translate them to Flink?

[kl0u]

@bakey In the long run yes. But for people to start using Beam on top of Flink, we can just add something like a method like writeToFlinkSink.

[mxm]

Actually, we could realize sink support like in #3 using Write.Bound. The interface might be intended only for batch but it would work for streaming applications as well.