What is your recommendations on how to build up orchestrations with Durable Patterns?

[thomaseyde]

Hi,

Brilliant idea, this library of yours!

I tried to apply this to one of my workflows, but got scaling issues very quickly.

My simple setup below was still running after 40 minutes and the orchestration had more than 50,000 events. This design is not sustainable, and probably not how Durable Patterns is meant to be used.

Could you add your recommendations on what and not to do with Durable Patterns in Readme.md?

My setup:

// 50 customers with 1,000 files each

await patterns.WithContext(context)
              .RunActivity<ReadCustomers>() // return List<Customer>
              .FanOutFanIn<ReadInventoryFiles>(new FanOutFanInOptions(1, 10)) // Return List<CustomerFile>
              .FanOutFanIn<RemoveOldFiles>(new FanOutFanInOptions(10, 100)) // Pass-through
              .FanOutFanIn<RemoveProcessedFiles>(new FanOutFanInOptions(10, 100)) // Pass-through
              .FanOutFanIn<CreateTargetCsvFile>(new FanOutFanInOptions(10, 10)) // Pass-through
              .FanOutFanIn<AppendFileToCsv>(new FanOutFanInOptions(10, 10)) // Pass-through
              .FanOutFanIn<AppendFileToIndex>(new FanOutFanInOptions(10, 10)) // Pass-through
              .ExecuteAsync();

Adjusting FanOutFanInOptions did not improve much. But passing the same initial List<Customer> to all activities did. I see in retrospect that is what you do in your front page example.