Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[helm chart] No permissions for secrets and configmaps, when deploying loki with rbac.namespaced = true #15845

Open
Mahagon opened this issue Jan 20, 2025 · 0 comments · May be fixed by #16776
Open

[helm chart] No permissions for secrets and configmaps, when deploying loki with rbac.namespaced = true #15845

Mahagon opened this issue Jan 20, 2025 · 0 comments · May be fixed by #16776
Labels
area/helm

Comments

@Mahagon
Copy link

Mahagon commented Jan 20, 2025

Describe the bug
loki-sc-rules containers inside the loki-backend-X pods can't access secrets and configmaps in the same namespace if rbac.namespaced is set to true. It seems that is because there is no deployment of a role & rolebinding for the permissions, just a deployment for a clusterrole & clusterrolebinding if rbac.namespaced is set to false.

To Reproduce
Steps to reproduce the behavior:

  1. Deploy the loki scalable with helm chart version 6.24.0
  2. Set rbac.namespaced to true.

Expected behavior
The Helm Chart should deploy a rolebinding and role to access the secrets and configmaps in it's namespace.
The following resolved the issue for me:

---
helmDefaults:
  createNamespace: false
---
repositories:
  - name: bedag
    url: https://bedag.github.io/helm-charts
---
releases:
  - name: rbac-workaround
    namespace: loki
    chart: bedag/raw
    version: 2.0.0
    values:
      - resources:
          - apiVersion: rbac.authorization.k8s.io/v1
            kind: Role
            metadata:
              name: loki-workaround-role
              namespace: loki
            rules:
            - apiGroups: [""]
              resources: ["configmaps", "secrets"]
              verbs: ["get", "list", "watch"]
          - apiVersion: rbac.authorization.k8s.io/v1
            kind: RoleBinding
            metadata:
              name: loki-workaround-rolebinding
            subjects:
            - kind: ServiceAccount
              name: loki
            roleRef:
              kind: Role
              name: loki-workaround-role
              apiGroup: rbac.authorization.k8s.io

Environment:

  • Infrastructure: Kubernetes on Azure
  • Deployment tool: Helmfile

Screenshots, Promtail config, or terminal output

loki level=info ts=2025-01-20T13:42:15.362411896Z caller=module_service.go:82 msg=starting module=store
loki level=info ts=2025-01-20T13:42:15.362454032Z caller=module_service.go:82 msg=starting module=index-gateway
loki level=info ts=2025-01-20T13:42:15.362485409Z caller=module_service.go:82 msg=starting module=ruler
loki level=info ts=2025-01-20T13:42:15.362516839Z caller=ruler.go:533 msg="ruler up and running"
loki level=info ts=2025-01-20T13:42:15.363236919Z caller=manager.go:164 user=fake msg="Starting rule manager..."
loki level=info ts=2025-01-20T13:42:15.987033187Z caller=compactor.go:428 msg="waiting until compactor is ACTIVE in the ring"
loki level=info ts=2025-01-20T13:42:16.147416754Z caller=compactor.go:432 msg="compactor is ACTIVE in the ring"
loki level=info ts=2025-01-20T13:42:16.147461995Z caller=loki.go:542 msg="Loki started" startup_time=7.231542407s
loki level=info ts=2025-01-20T13:42:18.256002376Z caller=scheduler.go:653 msg="this scheduler is in the ReplicationSet, will now accept requests."
loki-sc-rules {"time": "2025-01-20T13:45:32.953429+00:00", "taskName": null, "msg": "ApiException when calling kubernetes: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({'Audit-Id': '9caa01a2-0e33-4aee-bea4-d5fa70585293', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'f1378cdb-5a44-44bf-bbd8-4cbb26dee623', 'X-Kubernetes-Pf-Prioritylevel-Uid': '702eb1a3-a0ba-4152-8d34-1335458b80cf', 'Date': 'Mon, 20 Jan 2025 13:45:32 GMT', 'Content-Length': '372'})\nHTTP response body: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"secrets is forbidden: User \\\\\"system:serviceaccount:loki:loki\\\\\" cannot watch resource \\\\\"secrets\\\\\" in API group \\\\\"\\\\\" in the namespace \\\\\"loki\\\\\": RBAC: clusterrole.rbac.authorization.k8s.io \\\\\"kubernetes-service-get-access\\\\\" not found\",\"reason\":\"Forbidden\",\"details\":{\"kind\":\"secrets\"},\"code\":403}\\n'\n\n", "level": "ERROR"}
loki-sc-rules {"time": "2025-01-20T13:45:32.969258+00:00", "taskName": null, "msg": "ApiException when calling kubernetes: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({'Audit-Id': 'cd379e70-4170-4b47-89d1-1c36534e2867', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'f1378cdb-5a44-44bf-bbd8-4cbb26dee623', 'X-Kubernetes-Pf-Prioritylevel-Uid': '702eb1a3-a0ba-4152-8d34-1335458b80cf', 'Date': 'Mon, 20 Jan 2025 13:45:32 GMT', 'Content-Length': '381'})\nHTTP response body: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"configmaps is forbidden: User \\\\\"system:serviceaccount:loki:loki\\\\\" cannot watch resource \\\\\"configmaps\\\\\" in API group \\\\\"\\\\\" in the namespace \\\\\"loki\\\\\": RBAC: clusterrole.rbac.authorization.k8s.io \\\\\"kubernetes-service-get-access\\\\\" not found\",\"reason\":\"Forbidden\",\"details\":{\"kind\":\"configmaps\"},\"code\":403}\\n'\n\n", "level": "ERROR"}
loki-sc-rules {"time": "2025-01-20T13:45:32.973791+00:00", "taskName": null, "msg": "ApiException when calling kubernetes: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({'Audit-Id': 'e40215e1-a280-4757-b6e1-3e82386968ae', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'f1378cdb-5a44-44bf-bbd8-4cbb26dee623', 'X-Kubernetes-Pf-Prioritylevel-Uid': '702eb1a3-a0ba-4152-8d34-1335458b80cf', 'Date': 'Mon, 20 Jan 2025 13:45:32 GMT', 'Content-Length': '372'})\nHTTP response body: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"secrets is forbidden: User \\\\\"system:serviceaccount:loki:loki\\\\\" cannot watch resource \\\\\"secrets\\\\\" in API group \\\\\"\\\\\" in the namespace \\\\\"loki\\\\\": RBAC: clusterrole.rbac.authorization.k8s.io \\\\\"kubernetes-service-get-access\\\\\" not found\",\"reason\":\"Forbidden\",\"details\":{\"kind\":\"secrets\"},\"code\":403}\\n'\n\n", "level": "ERROR"}
...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/helm
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(helm): create namespaced RBAC when sidecar is enabled toVersus/loki
2 participants
@Mahagon @JStickler