project, packaging etc

[rambo]

Some quick points:

• Before this gets any bigger make it into an organization with features, website code etc as repos in the org.
• Make all coherent collections of features properly into packaged libraries (like waldo-opticalflow) from day one of development (use src structure), a little bit of extra diligence from the beginning will save huge amounts of work when trying to convert a random collection of quick and dirty scripts into something that not only "works on my machine" but is also usable by others.
• definitely separate raw data extraction (optical flow) from processing/filtering said data into something suitable for training (because it's far from free and computing resources to do the optical flow processing again and again and again)
• unit tests! CI.
  • remember that git-lfs exists, we absolutely can have also large test datasets without destroying the repo performance.
• Make a cookiecutter template(s) for project subcomponents, feel free to steal any and all ideas from https://gitlab.com/advian-oss?filter=template (also you can contact me if you have any questions about those or what I consider best practises for templates or docker images and whatnot)
• if/when you have multiple processes that need to communicate somehow take a look at https://gitlab.com/advian-oss/python-datastreamservicelib & https://gitlab.com/advian-oss/python-datastreamcorelib, we use that at work for realtime processing of image feeds and separation of responsibilities (one process deals with the image source and PUBlishes the frames, multiple analytics processes can then SUBscribe to the frame feed and process things in parallel, various components can be plugged together like lego to form complex pipelines)

[jaredb1011]

Great things to consider.

We'll definitely be consulting with a group of experienced devs to discuss our infrastructure.