support metadata search #348

AliRamberg · 2025-07-20T14:31:00Z

closes #treeverse/lakeFS-Enterprise#454

This was tested with both S3 and GS buckets

CLAassistant · 2025-07-20T14:31:07Z

All committers have signed the CLA.

AliRamberg · 2025-07-26T17:47:39Z

charts/lakefs/templates/service-account.yaml

@@ -0,0 +1,10 @@
+{{- if .Values.serviceAccount.create }}


Though can be considered breaking change, it's kinda awkward we have these values for creating a new ServiceAccount yet we never do

@guy-har Do you think we can keep this? We've already had this value but never actually created the serviceAccout, only referenced it.

guy-har

Looks Good,
Added minor comments.

Also, how was this tested?

guy-har · 2025-07-27T09:45:22Z

charts/lakefs/templates/_arktika.tpl

+{{/*
+Arktika service port
+*/}}
+{{- define "arktika.port" -}}
+{{- default 80 (.Values.mds.service).port }}
+{{- end }}


Didn't find usage, also, in the deployment we are using port 8000

guy-har · 2025-07-27T09:50:18Z

charts/lakefs/templates/arktika-service.yaml

+apiVersion: v1
+kind: Service
+metadata:
+  name: arktika-server


Suggested change

name: arktika-server

name: {{ include "arktika.fullname" . }}

guy-har · 2025-07-27T09:51:47Z

charts/lakefs/templates/_arktika.tpl

+Arktika volumes
+*/}}
+{{- define "arktika.volumes" -}}
+{{- with .Values.mds.extraVolumes -}}


Does not exist in values file

I'm not sure it should be in the default values, because I thought that we shouldn't make these assumptions...
It's a consumer issue choosing what volumes or environment vars he needs. For our specific test case, I've created a secret with GS credentials which were then mounted using these generic templates.

Here's a snippet from my testing values, relevant for extraVolumes and extraVolumeMounts :

extraVolumes: - name: gcp-credentials secret: secretName: gcp-credentials extraVolumeMounts: - name: gcp-credentials mountPath: /app/credentials.json subPath: credentials.json extraEnvVars: - name: GOOGLE_APPLICATION_CREDENTIALS value: /app/credentials.json

guy-har · 2025-07-27T09:52:18Z

charts/lakefs/templates/_arktika.tpl

+Arktika volume mounts
+*/}}
+{{- define "arktika.volumeMounts" -}}
+{{- with .Values.mds.extraVolumeMounts -}}


Does not exist in values file

Check above comment

guy-har · 2025-07-27T09:59:33Z

charts/lakefs/templates/arktika-deployment.yaml

+  labels:
+    {{- include "arktika.labels" . | nindent 4 }}
+spec:
+  replicas: {{ .Values.mds.replicaCount }}


Not sure whats the best practice here, but if we currently don't support working with multiple replicas, should it be hard coded? Otherwise we might need to document it somewhere

Absolutely, thanks

AliRamberg · 2025-07-27T10:33:44Z

Also, how was this tested?

I've manually installed the chart with both GS and S3 block stores and performed couple commits on the main repository, made sure the metadata repository is updated with no cloud provider errors

charts/lakefs/values.yaml

charts/lakefs/templates/arktika-service.yaml

Isan-Rivkin · 2025-07-28T08:08:35Z

charts/lakefs/templates/_arktika.tpl

+helm.sh/chart: {{ include "lakefs.chart" . }}
+{{ include "arktika.selectorLabels" . }}
+{{- if .Chart.AppVersion }}
+app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}


I wonder - perhaps the version should be the mds image tag instead?
(Feel free to not do it and resolve if you think otherwise)

charts/lakefs/templates/arktika-deployment.yaml

Isan-Rivkin · 2025-07-28T08:25:42Z

charts/lakefs/templates/_arktika.tpl

+{{- end }}
+- name: {{ include "arktika.fullname" . }}-config
+  mountPath: /app/config.yaml
+  subPath: config.yaml


Will this not error during apply if config is empty?
Meaning by default Values.mds.config is null, so in runtime the ConfigMap has no key config.yaml in it's data spec at all, there's no such key.

I don't really mind as the service couldn't run without it, though we let the app catch it with a default value

WDYM? not sure I understand the response

charts/lakefs/values.yaml

Isan-Rivkin

Thanks! LGTM!
Left some unresolved / new comments but none of them is a blocker!
BTW Im sure you noticed but just in case there's git conflicts

Isan-Rivkin · 2025-07-30T15:51:50Z

charts/lakefs/values.yaml

+  config:
+      lakefs:
+        endpoint: "https://example.lakefs.io"
+        access_key_id: "AKIAIOSFOLEXAMPLE"
+        secret_access_key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
+      metadata_settings:
+        since: "1970-01-01T00:00:00+00"
+        max_commits: 100
+      repositories:
+        # example-repo-1:
+        #   branches:
+        #   - main
+        #   - dev
+        # example-repo-2:
+        #   branches:
+        #   - main
+        #   - feature-*


I would comment that out since it's dead config
and reference the docs, config structure may change
not a blocker

It is commented and I have referenced the docs above the mds block

AliRamberg · 2025-07-31T09:49:29Z

Waiting on releasing until we have a valid image of the default version 0.1.0.

AliRamberg force-pushed the feature/chart/support-mds branch 4 times, most recently from 14122f2 to 778f88a Compare July 26, 2025 17:43

AliRamberg commented Jul 26, 2025

View reviewed changes

AliRamberg force-pushed the feature/chart/support-mds branch from 778f88a to a34c3e1 Compare July 26, 2025 17:49

AliRamberg requested review from Isan-Rivkin and guy-har July 27, 2025 08:21

AliRamberg force-pushed the feature/chart/support-mds branch from a34c3e1 to 757a3f3 Compare July 27, 2025 08:35

guy-har requested changes Jul 27, 2025

View reviewed changes

AliRamberg force-pushed the feature/chart/support-mds branch from 2824b60 to 9ad1a90 Compare July 27, 2025 10:36

AliRamberg requested a review from guy-har July 27, 2025 10:37

AliRamberg force-pushed the feature/chart/support-mds branch from 9ad1a90 to afdb0a3 Compare July 27, 2025 10:38

Isan-Rivkin requested changes Jul 28, 2025

View reviewed changes

AliRamberg requested a review from Isan-Rivkin July 30, 2025 14:21

AliRamberg force-pushed the feature/chart/support-mds branch 2 times, most recently from 5b6a264 to 97e3dce Compare July 30, 2025 15:35

Isan-Rivkin approved these changes Jul 30, 2025

View reviewed changes

AliRamberg added 2 commits July 31, 2025 11:13

fix comments

da58136

comments and other stuff

2edf037

AliRamberg force-pushed the feature/chart/support-mds branch from 97e3dce to 2edf037 Compare July 31, 2025 08:14

guy-har approved these changes Jul 31, 2025

View reviewed changes

fix lints

d620719

fix health endpoint

935df1d

AliRamberg merged commit 0549a90 into master Aug 4, 2025
3 checks passed

AliRamberg deleted the feature/chart/support-mds branch August 4, 2025 13:25

	name: arktika-server
	name: {{ include "arktika.fullname" . }}

support metadata search #348

support metadata search #348

Uh oh!

Conversation

AliRamberg commented Jul 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CLAassistant commented Jul 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guy-har left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AliRamberg Jul 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AliRamberg commented Jul 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Isan-Rivkin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AliRamberg commented Jul 31, 2025

Uh oh!

Uh oh!

Uh oh!

AliRamberg commented Jul 20, 2025 •

edited

Loading

CLAassistant commented Jul 20, 2025 •

edited

Loading

AliRamberg Jul 27, 2025 •

edited

Loading

AliRamberg commented Jul 27, 2025 •

edited

Loading