Jkolyer: Parallel File Upload #19
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* pfu-10: first test * pfu-10: sqlite connect/disconnect * pfu-10: config pytest * pfu-10: create db tables * pfu-10: removed unsued code Co-authored-by: Jonathan Kolyer <jonathankolyer@gmail>
* pfu-4: directory traversal starter * pfu-4: insert batch job, refactoring * pfu-4: create batch job model * pfu4: iterating on dir creation * pfu4: generate sample files for test * pfu-4: glob files * pfu-4: file model creation * pfu-4: file model insertion test * pfu-4: no duplicte file records * pfu-4: refactor db connect Co-authored-by: Jonathan Kolyer <jonathankolyer@gmail>
* pfu-3: file model upload shell * pfu-3: fixed pagination sql * pfu-3: added s3 mocks and file upload/retrieval * pfu-3: set s3 logging to critical * pfu-3: refactor s3 mocks * pfu-3: multi-file upload credentials fixed * pfu-3: use generator function to loop through files * pfu-3: async upload support Co-authored-by: Jonathan Kolyer <jonathankolyer@gmail>
* pfu-14: made uploader abstract base class * pfu-14: uploading metadata * pfu-14: reset_file_status; fixed permissions stat value * pfu-14: refactoring old code Co-authored-by: Jonathan Kolyer <jonathankolyer@gmail>
* pfu-5: refactoring models code * pfu-5: refactored models into separate files; passing tests * pfu-5: doc template * pfu-5: added doc templates * pfu-5: stubbing out gateway script * pfu-5: multi-proc code starters; refactor * pfu-5: filing out comments in file_model * pfu-5: comments for batch_model * pfu-5: base model comments * pfu-5: uploader comments * pfu-5: directories comments * pfu-5: iterating on multiprocess impl * pfu-5: moto workaround for multiprocessing * pfu-5: cleanup moto multiproc workaround * pfu-5: updated comments * pfu-5: fixed comment Co-authored-by: Jonathan Kolyer <jonathankolyer@gmail>
* pfu-1: gateway script * pfu-1: setting up for localstack * pfu-1: refactored dependency inversion on boto3 client for all evnironments * pfu-1: running concurrent/parallel on localstack * pfu-1: renamed localstack_url to endpoint_url * pfu-1: passing multiprocessing tests Co-authored-by: Jonathan Kolyer <jonathankolyer@gmail>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
File uploading service with the following features