Safety or usability: Why not both? Towards referential auth in k8s

Watch talk on YouTube

A talk by Google and Microsoft with the premise of better auth in k8s.

Baselines

  • Most access controllers have read access to all secrets -> They are not really designed for keeping these secrets
  • Result: CVEs
  • Example: Just use ingress, nginx, put in some Lua code in the config and e voilà: Service account token
  • Fix: No more fun

Basic solutions

  • Separate Control (the controller) from data (the ingress)
  • Namespace limited ingress

Current state of cross namespace stuff

  • Why: Reference TLS cert for gateway API in the cert team’s namespace
  • Why: Move all ingress configs to one namespace
  • Classic Solution: Annotations in contour that references a namespace that contains all certs (rewrites secret to certs/secret)
  • Gateway Solution:
    • Gateway TLS secret ref includes a namespace
    • ReferenceGrant pretty much allows referencing from X (Gateway) to Y (Secret)
  • Limits:
    • Has to be implemented via controllers
    • The controllers still have read all - they just check if they are supposed to do this

Goals

Global

  • Grant access to controller to only resources relevant for them (using references and maybe class segmentation)
  • Allow for safe cross namespace references
  • Make it easy for API devs to adopt it

Personas

  • Alex API author
  • Kai controller author
  • Rohan Resource owner

What our stakeholders want

  • Alex: Define relationships via ReferencePatterns
  • Kai: Specify controller identity (Serviceaccount), define relationship API
  • Rohan: Define cross namespace references (aka resource grants that allow access to their resources)

Result of the paper

Architecture

  • ReferencePattern: Where do i find the references -> example: GatewayClass in the gateway API
  • ReferenceConsumer: Who (Identity) has access under which conditions?
  • ReferenceGrant: Allow specific references

POC

  • Minimum access: You only get access if the grant is there AND the reference actually exists
  • Their basic implementation works with the kube API

Open questions

  • Naming
  • Make people adopt this
  • What about namespace-scoped ReferenceConsumer
  • Is there a need of RBAC verb support (not only read access)

Alternative

  • Idea: Just extend RBAC Roles with a selector (match labels, etc.)
  • Problems:
    • Requires changes to Kubernetes core auth
    • Everything bus list and watch is a pain
    • How do you handle AND vs OR selection
    • Field selectors: They exist
  • Benefits: Simple controller implementation

Meanwhile

  • Prefer tools that support isolation between controller and data-plane
  • Disable all non-needed features -> Especially scripting