|
| 1 | +# Data Source Syncer |
| 2 | + |
| 3 | +This CLI tool acts as a scheduled job that remotely syncs data to a given Grafana Cloud Logging data source. This ensures that the Grafana data source has the following set correctly: |
| 4 | + |
| 5 | +* Authentication by refreshing an OAuth2 access token periodically |
| 6 | +* The Google Cloud project ID |
| 7 | + |
| 8 | +By regularly refreshing the OAuth2 access token, you can configure Grafana to directly query Google Cloud Logging. |
| 9 | + |
| 10 | +[Google access tokens have a lifetime of 1 hour.](https://cloud.google.com/docs/authentication/token-types#at-lifetime) This script should be scheduled to run every 10 minutes to ensure you have an uninterrupted connection between Grafana and Google Cloud Logging. |
| 11 | + |
| 12 | +## Flags |
| 13 | + |
| 14 | +```bash mdox-exec="bash hack/format_help.sh datasource-syncer" |
| 15 | +Usage of datasource-syncer: |
| 16 | + -datasource-uids string |
| 17 | + datasource-uids is a comma separated list of data source UIDs to update. |
| 18 | + -grafana-api-endpoint string |
| 19 | + grafana-api-endpoint is the endpoint of the Grafana instance that contains the data sources to update. |
| 20 | + -grafana-api-token string |
| 21 | + grafana-api-token used to access Grafana. Can be created using: https://grafana.com/docs/grafana/latest/administration/service-accounts/#create-a-service-account-in-grafana |
| 22 | + -insecure-skip-verify |
| 23 | + Skip TLS certificate verification |
| 24 | + -project-id string |
| 25 | + Project ID of the Google Cloud Monitoring scoping project to query. Queries sent to this project will union results from all projects within the scope. |
| 26 | + -query.credentials-file string |
| 27 | + JSON-encoded credentials (service account or refresh token). Can be left empty if default credentials have sufficient permission. |
| 28 | + -tls-ca-cert string |
| 29 | + Path to the server certificate authority |
| 30 | + -tls-cert string |
| 31 | + Path to the server TLS certificate. |
| 32 | + -tls-key string |
| 33 | + Path to the server TLS key. |
| 34 | +``` |
| 35 | +## Scheduled the job using Cloud Run |
| 36 | +
|
| 37 | +To deploy and run a serverless data source syncer by using [Cloud Run](https://cloud.google.com/run) |
| 38 | +and [Cloud Scheduler](https://cloud.google.com/scheduler), do the following: |
| 39 | +
|
| 40 | +1. Choose a project to deploy the data source syncer in. We recommend choosing |
| 41 | + the [scoping project of a multi-project log scope](https://cloud.google.com/logging/docs/log-scope/create-and-manage). |
| 42 | + The data source syncer uses the configured Google Cloud project as the |
| 43 | + scoping project. |
| 44 | +
|
| 45 | + Next, configure and authorize a service account for the data source syncer. |
| 46 | + The following command sequence creates a service account and grants it |
| 47 | + several IAM roles. The first three roles let the service account |
| 48 | + read from the Cloud Logging API and generate service account tokens. The |
| 49 | + last two roles allow the service account to read the Grafana service account |
| 50 | + token from Secret Manager and to invoke Cloud Run: |
| 51 | +
|
| 52 | + ```bash |
| 53 | + # Replace the values if needed |
| 54 | + export PROJECT_ID=YOUR_LOGGING_PROJECT_ID |
| 55 | + export SA_DS_SYNCER=grafana-ds-syncer-sa |
| 56 | + export REGION=us-central1 # The Google Cloud region where you want to run your Cloud Run job, such as us-central1. |
| 57 | + |
| 58 | + |
| 59 | + gcloud config set project ${PROJECT_ID} \ |
| 60 | + && |
| 61 | + gcloud iam service-accounts create ${SA_DS_SYNCER} \ |
| 62 | + && |
| 63 | + gcloud projects add-iam-policy-binding ${PROJECT_ID} \ |
| 64 | + --member=serviceAccount:${SA_DS_SYNCER}@${PROJECT_ID}.iam.gserviceaccount.com \ |
| 65 | + --role=roles/logging.viewer \ |
| 66 | + --condition=None \ |
| 67 | + && \ |
| 68 | + gcloud projects add-iam-policy-binding ${PROJECT_ID} \ |
| 69 | + --member=serviceAccount:${SA_DS_SYNCER}@${PROJECT_ID}.iam.gserviceaccount.com \ |
| 70 | + --role=roles/logging.viewAccessor \ |
| 71 | + --condition=None \ |
| 72 | + && \ |
| 73 | + gcloud projects add-iam-policy-binding ${PROJECT_ID} \ |
| 74 | + --member=serviceAccount:${SA_DS_SYNCER}@${PROJECT_ID}.iam.gserviceaccount.com \ |
| 75 | + --role=roles/iam.serviceAccountTokenCreator \ |
| 76 | + --condition=None \ |
| 77 | + && \ |
| 78 | + gcloud projects add-iam-policy-binding ${PROJECT_ID} \ |
| 79 | + --member=serviceAccount:${SA_DS_SYNCER}@${PROJECT_ID}.iam.gserviceaccount.com \ |
| 80 | + --role=roles/secretmanager.secretAccessor \ |
| 81 | + --condition=None \ |
| 82 | + && \ |
| 83 | + gcloud projects add-iam-policy-binding ${PROJECT_ID} \ |
| 84 | + --member=serviceAccount:${SA_DS_SYNCER}@${PROJECT_ID}.iam.gserviceaccount.com \ |
| 85 | + --role=roles/run.invoker \ |
| 86 | + --condition=None |
| 87 | + ``` |
| 88 | +
|
| 89 | +1. Build the container image: |
| 90 | + ```bash |
| 91 | + # Create the artifact registry to store the image |
| 92 | + gcloud artifacts repositories create datasource-syncer-repo \ |
| 93 | + --repository-format=docker \ |
| 94 | + --location=${REGION} --description="Docker repository for Grafana data source syncer" |
| 95 | + |
| 96 | + gcloud builds submit --region=${REGION} --tag ${REGION}-docker.pkg.dev/${PROJECT_ID}/datasource-syncer-repo/datasource-syncer |
| 97 | + ``` |
| 98 | +
|
| 99 | +1. Determine the URL of your Grafana instance, for example |
| 100 | + `https://yourcompanyname.grafana.net` for a Grafana Cloud deployment. Your |
| 101 | + Grafana instance needs to be accessible from Cloud Run, meaning it |
| 102 | + needs to be accessible from the wider internet. |
| 103 | +
|
| 104 | + If your Grafana instance is not accessible from the wider internet, we |
| 105 | + recommend deploying the data source syncer on Kubernetes instead. |
| 106 | +
|
| 107 | +1. Choose the Grafana Cloud Logging data source to |
| 108 | + use for Cloud Logging, which can be either a new or a |
| 109 | + pre-existing data source, and then find and write down the data |
| 110 | + source UID. The data source UID can be found in the last |
| 111 | + part of the URL when exploring or configuring a data source, for example |
| 112 | + `https://yourcompanyname.grafana.net/connections/datasources/edit/`**_grafana_datasource_uid_**. |
| 113 | +
|
| 114 | + **Do not copy the entire datasource URL**. Copy only the unique identifier in |
| 115 | + the URL. |
| 116 | +
|
| 117 | +  |
| 118 | +
|
| 119 | +1. [Set up a Grafana service account](https://grafana.com/docs/grafana/latest/administration/service-accounts/#create-a-service-account-in-grafana) by creating the |
| 120 | + service account and generating a token for the account to use: |
| 121 | +
|
| 122 | + 1. In the Grafana navigation sidebar, click |
| 123 | + **Administration > Users and Access > Service Accounts**. |
| 124 | +
|
| 125 | + 1. Create the service account in Grafana by clicking **Add service |
| 126 | + account**, giving it a name, and granting it the "Data Sources > |
| 127 | + Writer" role. Make sure you hit the **Apply** button to assign the role. |
| 128 | + In older versions of Grafana, you can use the "Admin" role |
| 129 | + instead. |
| 130 | +
|
| 131 | + 1. Click **Add service account token**. |
| 132 | +
|
| 133 | + 1. Set the token expiration to "No expiration" and click **Generate |
| 134 | + token**, then copy the generated token to the clipboard for use as |
| 135 | + GRAFANA_SERVICE_ACCOUNT_TOKEN in the next step: |
| 136 | +
|
| 137 | +  |
| 138 | +
|
| 139 | +1. Set the following |
| 140 | + variables using the results of the previous steps: |
| 141 | +
|
| 142 | + ```bash |
| 143 | + # These values are required. |
| 144 | + export GRAFANA_INSTANCE_URL=YOUR_GRAFANA_INSTANCE_URL # The Grafana instance URL from step 2. This is a URL. Include "http://" or "https://". |
| 145 | + export GRAFANA_DATASOURCE_UID=YOUR_GRAFANA_DATASOURCE_UID # The Grafana data source UID from step 3. This is not a URL. |
| 146 | + export GRAFANA_SERVICE_ACCOUNT_TOKEN=YOUR_GRAFANA_SERVICE_ACCOUNT_TOKEN # The Grafana service account token from step 4. |
| 147 | + ``` |
| 148 | +
|
| 149 | +1. Create a secret in Secret Manager: |
| 150 | +
|
| 151 | + ```bash |
| 152 | + gcloud secrets create datasource-syncer --replication-policy="automatic" && \ |
| 153 | + echo -n ${GRAFANA_SERVICE_ACCOUNT_TOKEN} | gcloud secrets versions add datasource-syncer --data-file=- |
| 154 | + ``` |
| 155 | +
|
| 156 | +1. Run the following command to create a YAML file and name it |
| 157 | + `cloud-run-datasource-syncer.yaml`: |
| 158 | +
|
| 159 | + ```bash |
| 160 | + cat > cloud-run-datasource-syncer.yaml <<EOF |
| 161 | + apiVersion: run.googleapis.com/v1 |
| 162 | + kind: Job |
| 163 | + metadata: |
| 164 | + name: logging-datasource-syncer-job |
| 165 | + spec: |
| 166 | + template: |
| 167 | + spec: |
| 168 | + taskCount: 1 |
| 169 | + template: |
| 170 | + spec: |
| 171 | + containers: |
| 172 | + - name: datasource-syncer |
| 173 | + image: ${REGION}-docker.pkg.dev/${PROJECT_ID}/datasource-syncer-repo/datasource-syncer |
| 174 | + args: |
| 175 | + - "--datasource-uids=${GRAFANA_DATASOURCE_UID}" |
| 176 | + - "--grafana-api-endpoint=${GRAFANA_INSTANCE_URL}" |
| 177 | + - "--project-id=${PROJECT_ID}" |
| 178 | + env: |
| 179 | + - name: GRAFANA_SERVICE_ACCOUNT_TOKEN |
| 180 | + valueFrom: |
| 181 | + secretKeyRef: |
| 182 | + key: latest |
| 183 | + name: datasource-syncer |
| 184 | + serviceAccountName: ${SA_DS_SYNCER}@${PROJECT_ID}.iam.gserviceaccount.com |
| 185 | + EOF |
| 186 | + ``` |
| 187 | +
|
| 188 | + Then run the following command to create a Cloud Run job using the |
| 189 | + YAML file: |
| 190 | +
|
| 191 | + ```bash |
| 192 | + gcloud run jobs replace cloud-run-datasource-syncer.yaml --region ${REGION} |
| 193 | + ``` |
| 194 | +
|
| 195 | +1. Create a schedule in Cloud Scheduler to run the Cloud Run job |
| 196 | + every 10 minutes: |
| 197 | +
|
| 198 | + ```bash |
| 199 | + gcloud scheduler jobs create http datasource-syncer \ |
| 200 | + --location ${REGION} \ |
| 201 | + --schedule="*/10 * * * *" \ |
| 202 | + --uri="https://${REGION}-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/${PROJECT_ID}/jobs/logging-datasource-syncer-job:run" \ |
| 203 | + --http-method POST \ |
| 204 | + --oauth-service-account-email=${SA_DS_SYNCER}@${PROJECT_ID}.iam.gserviceaccount.com |
| 205 | + ``` |
| 206 | +
|
| 207 | + Then force run the scheduler you just created: |
| 208 | +
|
| 209 | + ```bash |
| 210 | + gcloud scheduler jobs run datasource-syncer --location ${REGION} |
| 211 | + ``` |
| 212 | +
|
| 213 | + It can take up to 15 seconds for the data source to be updated. |
| 214 | +
|
| 215 | +1. Go to your newly configured Grafana data source and verify the **Access Token |
| 216 | + ** field has the value `configured` and the **Project ID** field has your cloud project id. You might have to refresh the page. Once verified, go to the bottom |
| 217 | + of the page, select **Save & test**, and ensure you see a green checkmark saying that |
| 218 | + the datasource is properly configured. You need to select **Save & test** at |
| 219 | + least once to ensure that label autocompletion in Grafana works. |
0 commit comments