Sponsored: Build an open source platform for ai/ml

Watch talk on YouTube

Jorge Palma from Microsoft with a quick introduction.

Baseline

  • Kubernetes is cool and all
  • Challenges:
    • Containerized models
    • GPUs in the cluster (install, management)

Kubernetes AI Tool chain (KAITO)

  • Kubernetes operator that interacts with
    • Node provisioner
    • Deployment
  • Simple CRD that describes a model, infra and have fun
  • Creates inference endpoint
  • Models are currently 10 (Hugginface, LLMA, etc.)