Skip to content

[KP]add env for S3 and GCS credential #756

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

[KP]add env for S3 and GCS credential #756

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
fcb082c
fix
JJJ000 Feb 24, 2023
2d7511e
add gitignore
JJJ000 Feb 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,8 @@ celerybeat-schedule

# dotenv
.env
.env_dev
.env_prod

# docker
docker/config.py
Expand Down
7 changes: 4 additions & 3 deletions knowledge_repo/utils/gcs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,11 @@
import logging
import re
import json
from dotenv import load_dotenv

logger = logging.getLogger(__name__)
GOOGLE_AUTH_PATH = '.configs/google_auth.json'

load_dotenv()


def parse_gcs_uri(uri):
Expand All @@ -25,8 +27,7 @@ def parse_gcs_uri(uri):

def get_gcs_client():

with open(GOOGLE_AUTH_PATH) as source:
credentials_dict = json.load(source)
credentials_dict = os.getenv('GOOGLE_APPLICATION_CREDENTIALS')

credentials = ServiceAccountCredentials.from_json_keyfile_dict(
credentials_dict
Expand Down
6 changes: 3 additions & 3 deletions knowledge_repo/utils/s3.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,11 @@
import os
import json
import re
from dotenv import load_dotenv

logger = logging.getLogger(__name__)
AWS_S3_AUTH_PATH = '.configs/aws_s3_auth.json'

load_dotenv()

def parse_s3_uri(s3_uri):
"""Get s3_path for S3 Object URL
Expand All @@ -33,8 +34,7 @@ def get_s3_client():
:return: a boto3 client for s3 operations
"""

with open(AWS_S3_AUTH_PATH) as source:
credentials_dict = json.load(source)
credentials_dict = os.getenv('AWS_S3_CREDENTIALS')

return boto3.client(
"s3",
Expand Down
1 change: 1 addition & 0 deletions requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,4 @@ weasyprint==54.3
s3path==0.3.4
notion-client==2.0.0
gcloud==0.18.3
python-dotenv==0.21.1