Get "secrets" from something other than the `operations` repo

[dshemetov]

We'd like to make this repo self-contained and remove operations repo dependence. Some annoyances with the operations repo dependence:

• dependency updates are harder (see for example, the need to do simultaneous PRs in both repos Remove unused undefx depends from installation script operations#70 and Remove unused undefx depends from CI #935, in order to update dependencies)
• setup changes are harder (see for example, the work here that needs to change the setup.sh script in operations)

Merging Dockerfiles:

• remove dependence on and references to db docker definition in delphi/operations repo #1000 made the db Docker image self-contained in this repo
• Merge delphi_web_python Docker image from operations repo #1043 made the delphi_wb Docker image self-contained in this repo

What other work is there that needs doing?

Other repos using operations repo Dockerfiles (this is an organization-wide codebase search query):

• delphi-epidata-private
• www-epicast
• flu-contest

[melange396]

i looked at a bunch of the undefx dependencies while i was doing #1000. i think you can find virtually all of it with this command: grep -r "undef-analysis\|undefx\|py3test" .

• in addition to the removal from ci.yaml in Remove unused undefx depends from CI #935, you can also potentially remove references to the undefx repos or their location on the local filessystem from these files in this repo:

  • dev/local/install.sh
  • docs/epidata_development.md
  • docs/new_endpoint_tutorial.md

• then in deploy.json, remove the line that matches this as well as the comment line above it:

  • {"type": "py3test"}

• and finally under tests/ and integrations/ , remove lines from the many files that match these two:

  • # py3tester coverage target
  • __test_target__

• a number of our other repositories also have traces of this stuff:

  • repos flu-contest and utils have references in their own deploy.json, in some *.md documentation files, and the "coverage target" stuff in their test files [ NOTE: im not sure how we use (or if we are still using) deploy.json ]
  • repo operations also has it in the setup script, ./dev/docker/python/assets/setup.sh
  • repos github-deploy-repo and nowcast have the above as well as a couple actual imports in source code
    • the former seems like an easy excision, the latter will be harder as it uses this https://github.com/undefx/undef-analysis/blob/master/neldermead.py , and some part of the delphi-epidata build relies on the nowcast (though i dont know where the integration point is)

[krivard]

went looking to see if we had any github issues about moving away from github-deploy-repo and got briefly trapped here.

delphi-epidata-private was a fork that got shelved in late 2020. The intention was to have a parallel Epidata environment with no public access, so that we could store raw counts from DUA sources. Bringing back that effort is better off starting from scratch; we should archive the delphi-epidata-private repository.

www-epicast is an ILI crowdcasting tool that continues to run itself automatically every year, though the pandemic has essentially destroyed the relationship between ILI (fever+cough) and public health load. We should leave operations in place to continue to support www-epicast as it slowly rots from lack of maintenance.

flu-contest is an automated flu forecasting submission system. We should leave operations in place to continue to support flu-forecast as it slowly rots from lack of maintenance.

[melange396]

AFAICT, at this point in time, we are only using the operations repo for "secrets" stuff (ie, import delphi.operations.secrets as secrets, and that only really gets used in the src/acquisition/ and src/maintenance subdirectories (which is for periodically-run tasks (as opposed to long-running processes, like the API server)).

so this ~essentially "just" needs a new way to supply credentials to those jobs...

[melange396]

We should finalize #1212 and then we can move on to replacing delphi.operations.secrets with something else.

After that, "operations" stuff can come out of the environment and build stuff in:

• dev/local/install.sh
• dev/docker/python/setup.sh
• dev/local/setup.cfg
• .github/workflows/ci.yaml
• .github/workflows/performance-tests.yml
• .github/workflows/performance-tests-one-time.yml

Anything else that mentions "operations" seems to be in documentation/comments.