Skip to content

K8s support for CEEMS API server #381

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 17, 2025
Merged

K8s support for CEEMS API server #381

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b905969
feat: Add k8s support for CEEMS API server
mahendrapaipuri Jul 16, 2025
3b92a37
refactor: Several style improvements and code updates
mahendrapaipuri Jul 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ ifeq ($(CGO_APPS), 1)
PROMU_CONF ?= .promu/.promu-cgo.yml
pkgs := ./pkg/sqlite3 ./pkg/api/cli \
./pkg/api/db ./pkg/api/db/migrator ./pkg/api/helper \
./pkg/api/resource ./pkg/api/resource/slurm ./pkg/api/resource/openstack \
./pkg/api/resource ./pkg/api/resource/slurm ./pkg/api/resource/openstack ./pkg/api/resource/k8s \
./pkg/api/updater ./pkg/api/updater/tsdb \
./pkg/api/http ./cmd/ceems_api_server \
./pkg/lb/backend ./pkg/lb/cli \
Expand Down Expand Up @@ -197,21 +197,26 @@ test-e2e: $(PROMTOOL) build pkg/collector/testdata/sys/.unpacked pkg/collector/t
./scripts/e2e-test.sh -s api-project-query
./scripts/e2e-test.sh -s api-project-empty-query
./scripts/e2e-test.sh -s api-project-admin-query
./scripts/e2e-test.sh -s api-project-query-k8s
./scripts/e2e-test.sh -s api-user-query
./scripts/e2e-test.sh -s api-user-admin-query
./scripts/e2e-test.sh -s api-user-query-k8s
./scripts/e2e-test.sh -s api-cluster-admin-query
./scripts/e2e-test.sh -s api-uuid-query
./scripts/e2e-test.sh -s api-running-query
./scripts/e2e-test.sh -s api-units-query-k8s
./scripts/e2e-test.sh -s api-admin-query
./scripts/e2e-test.sh -s api-admin-query-all
./scripts/e2e-test.sh -s api-admin-query-all-selected-fields
./scripts/e2e-test.sh -s api-admin-denied-query
./scripts/e2e-test.sh -s api-current-usage-query
./scripts/e2e-test.sh -s api-current-usage-experimental-query
./scripts/e2e-test.sh -s api-global-usage-query
./scripts/e2e-test.sh -s api-current-usage-admin-query
./scripts/e2e-test.sh -s api-current-usage-admin-experimental-query
./scripts/e2e-test.sh -s api-current-usage-query-k8s
./scripts/e2e-test.sh -s api-global-usage-query
./scripts/e2e-test.sh -s api-global-usage-admin-query
./scripts/e2e-test.sh -s api-global-usage-query-k8s
./scripts/e2e-test.sh -s api-current-usage-admin-denied-query
./scripts/e2e-test.sh -s api-current-stats-admin-query
./scripts/e2e-test.sh -s api-global-stats-admin-query
Expand Down Expand Up @@ -277,21 +282,26 @@ test-e2e-update: $(PROMTOOL) build pkg/collector/testdata/sys/.unpacked pkg/coll
./scripts/e2e-test.sh -s api-project-query -u || true
./scripts/e2e-test.sh -s api-project-empty-query -u || true
./scripts/e2e-test.sh -s api-project-admin-query -u || true
./scripts/e2e-test.sh -s api-project-query-k8s -u || true
./scripts/e2e-test.sh -s api-user-query -u || true
./scripts/e2e-test.sh -s api-user-admin-query -u || true
./scripts/e2e-test.sh -s api-user-query-k8s -u || true
./scripts/e2e-test.sh -s api-cluster-admin-query -u || true
./scripts/e2e-test.sh -s api-uuid-query -u || true
./scripts/e2e-test.sh -s api-running-query -u || true
./scripts/e2e-test.sh -s api-units-query-k8s -u || true
./scripts/e2e-test.sh -s api-admin-query -u || true
./scripts/e2e-test.sh -s api-admin-query-all -u || true
./scripts/e2e-test.sh -s api-admin-query-all-selected-fields -u || true
./scripts/e2e-test.sh -s api-admin-denied-query -u || true
./scripts/e2e-test.sh -s api-current-usage-query -u || true
./scripts/e2e-test.sh -s api-current-usage-experimental-query -u || true
./scripts/e2e-test.sh -s api-global-usage-query -u || true
./scripts/e2e-test.sh -s api-current-usage-admin-query -u || true
./scripts/e2e-test.sh -s api-current-usage-admin-experimental-query -u || true
./scripts/e2e-test.sh -s api-current-usage-query-k8s -u || true
./scripts/e2e-test.sh -s api-global-usage-query -u || true
./scripts/e2e-test.sh -s api-global-usage-admin-query -u || true
./scripts/e2e-test.sh -s api-global-usage-query-k8s -u || true
./scripts/e2e-test.sh -s api-current-usage-admin-denied-query -u || true
./scripts/e2e-test.sh -s api-current-stats-admin-query -u || true
./scripts/e2e-test.sh -s api-global-stats-admin-query -u || true
Expand Down
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ managers (SLURM, Openstack, k8s)
- Supports zero instrumentation eBPF based continuous profiling using
[Grafana Pyroscope](https://grafana.com/oss/pyroscope/) as backend
- Realtime access to metrics *via* Grafana dashboards or a simple CLI tool
- Access control to Prometheus and Pyroscope datasources in Grafana
- Multi-tenancy and access control to Prometheus and Pyroscope datasources in Grafana
- Stores aggregated metrics in a separate DB that can be retained for long time
- CEEMS apps are [capability aware](https://tbhaxor.com/understanding-linux-capabilities/)

Expand Down
87 changes: 65 additions & 22 deletions build/config/ceems_api_server/ceems_api_server.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -231,8 +231,10 @@ clusters: []
# #
# - id: default

# # Resource manager of the cluster. Currently only `slurm` is supported. In future,
# # `openstack` will be supported
# # Resource manager of the cluster. Currently supported managers:
# # - `slurm`
# # - `openstack`
# # - `k8s`
# #
# manager: slurm

Expand Down Expand Up @@ -437,28 +439,69 @@ clusters: []
# # Any other configuration needed to reach API server of the resource manager
# # can be configured in this section.
# #
# # Currently this section is used for Openstack resource manager
# # to configure API versions
# #
# # In the case of Openstack, this section must have two keys `api_service_endpoints`
# # and `auth`. Both of these are compulsory.
# # `api_service_endpoints` must provide API endpoints for compute and identity
# # services as provided in service catalog of Openstack cluster. `auth` must be the
# # same `auth` object that must be sent in POST request to keystone to get a API token.
# # Currently this section is used for Openstack and k8s resource managers
# # to configure API servers
# #
# extra_config: {}
# # api_service_endpoints:
# # compute: https://openstack-nova.example.com/v2.1
# # identity: https://openstack-keystone.example.com
# # auth:
# # identity:
# # methods:
# # - password
# # password:
# # user:
# # name: admin
# # password: supersecret

# # # In the case of Openstack, this section must have two keys `api_service_endpoints`
# # # and `auth`. Both of these are compulsory.
# # # `api_service_endpoints` must provide API endpoints for compute and identity
# # # services as provided in service catalog of Openstack cluster. `auth` must be the
# # # same `auth` object that must be sent in POST request to keystone to get a API token.
# # #
# # api_service_endpoints:
# # compute: https://openstack-nova.example.com/v2.1
# # identity: https://openstack-keystone.example.com
# # auth:
# # identity:
# # methods:
# # - password
# # password:
# # user:
# # name: admin
# # password: supersecret

# # # In the case of k8s, this section is used to configure the k8s API related config
# # #
# # # Path to the kube config file when out-of-cluster config file is used. If the
# # # ceems_api_server is running in a pod, in-cluster config will be used by default.
# # #
# # kubeconfig_file: ''
# # #
# # # List of annotation names where the name of the user that created/modified the pod is stored.
# # # The search for username will be stopped when the first annotation is found in the list.
# # # Therefore the order of the list is important.
# # #
# # # Default: [ceems.io/created-by]
# # #
# # username_annotations: []
# # #
# # # List of annotation names where the name of the project that the pod belongs to is stored.
# # # By default the project is always set to the namespace of the pod and when atleast one of
# # # annotation names are found in the pod spec, the namespace will be overridded by the value
# # # of the annotation
# # #
# # project_annotations: []
# # #
# # # List of GPU resource names in the cluster.
# # #
# # # Default: [nvidia.com/gpu, amd.com/gpu]
# # #
# # gpu_resource_names: []
# # #
# # # Path to the file that contains the list of namespaces and allowed users in each namespace.
# # # The format of the file must be as follows:
# # #
# # # users:
# # # ns1:
# # # - usr1
# # # - usr2
# # # ns2:
# # # - usr1
# # # - usr3
# # #
# # ns_users_list_file: ''

# A list of Updaters that will be used to update the compute unit metrics. This update
# step can be used to update the aggregate metrics of each compute unit in real time
# or to add complementary information to the compute units from on-premise third
Expand Down
21 changes: 10 additions & 11 deletions cmd/cacct/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ import (
"os/user"
"path/filepath"
"slices"
"sort"
"strings"
"time"

Expand Down Expand Up @@ -237,12 +236,12 @@ type field struct {
}

// titles return header titles for the table.
func (f field) titles() []interface{} {
func (f field) titles() []any {
if len(f.keys) <= 1 {
return []interface{}{f.title}
return []any{f.title}
}

t := make([]interface{}, len(f.keys))
t := make([]any, len(f.keys))
for i := range len(f.keys) {
t[i] = f.title
}
Expand All @@ -251,12 +250,12 @@ func (f field) titles() []interface{} {
}

// subtitles return header subtitles for the table.
func (f field) subtitles() []interface{} {
func (f field) subtitles() []any {
if len(f.keys) <= 1 {
return []interface{}{""}
return []any{""}
}

t := make([]interface{}, len(f.keys))
t := make([]any, len(f.keys))
for i, k := range f.keys {
t[i] = k
}
Expand Down Expand Up @@ -294,7 +293,7 @@ type Config struct {
}

// UnmarshalYAML implements the yaml.Unmarshaler interface.
func (c *Config) UnmarshalYAML(unmarshal func(interface{}) error) error {
func (c *Config) UnmarshalYAML(unmarshal func(any) error) error {
// Set a default config
*c = Config{}
c.API.UserHeaderName = "X-Grafana-User"
Expand All @@ -316,7 +315,7 @@ type WebConfig struct {
}

// UnmarshalYAML implements the yaml.Unmarshaler interface.
func (w *WebConfig) UnmarshalYAML(unmarshal func(interface{}) error) error {
func (w *WebConfig) UnmarshalYAML(unmarshal func(any) error) error {
// Set a default config
*w = WebConfig{}
w.HTTPClientConfig = http_config.DefaultHTTPClientConfig
Expand Down Expand Up @@ -726,15 +725,15 @@ func sortedKeys[K cmp.Ordered, V any](m map[K]V) []K {
i++
}

sort.Slice(keys, func(i, j int) bool { return keys[i] < keys[j] })
slices.Sort(keys)

return keys
}

func splitString(s, d string) []string { //nolint:unparam
var parts []string

for _, p := range strings.Split(s, d) {
for p := range strings.SplitSeq(s, d) {
if p != "" {
parts = append(parts, p)
}
Expand Down
1 change: 1 addition & 0 deletions cmd/ceems_api_server/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import (
"os"

"github.com/mahendrapaipuri/ceems/pkg/api/cli"
_ "github.com/mahendrapaipuri/ceems/pkg/api/resource/k8s"
_ "github.com/mahendrapaipuri/ceems/pkg/api/resource/openstack"
_ "github.com/mahendrapaipuri/ceems/pkg/api/resource/slurm"
_ "github.com/mahendrapaipuri/ceems/pkg/api/updater/tsdb"
Expand Down
12 changes: 6 additions & 6 deletions cmd/ceems_k8s_admission_controller/base/patch.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,14 @@ const (

// PatchOperation is an operation of a JSON patch https://tools.ietf.org/html/rfc6902.
type PatchOperation struct {
Op string `json:"op"`
Path string `json:"path"`
From string `json:"from"`
Value interface{} `json:"value,omitempty"`
Op string `json:"op"`
Path string `json:"path"`
From string `json:"from"`
Value any `json:"value,omitempty"`
}

// AddPatchOperation returns an add JSON patch operation.
func AddPatchOperation(path string, value interface{}) PatchOperation {
func AddPatchOperation(path string, value any) PatchOperation {
return PatchOperation{
Op: addOperation,
Path: path,
Expand All @@ -34,7 +34,7 @@ func RemovePatchOperation(path string) PatchOperation {
}

// ReplacePatchOperation returns a replace JSON patch operation.
func ReplacePatchOperation(path string, value interface{}) PatchOperation {
func ReplacePatchOperation(path string, value any) PatchOperation {
return PatchOperation{
Op: replaceOperation,
Path: path,
Expand Down
2 changes: 1 addition & 1 deletion cmd/ceems_k8s_admission_controller/http/handlers.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ func (h *admissionHandler) Serve(hook base.Hook) http.HandlerFunc {

var result *base.Result

var operation interface{}
var operation any

var version string

Expand Down
Loading