Building Container Images the Modern Way

Problems

• Dockerfiles are hard and not 100% reproducible
• Buildpacks are reproducible but result in large single-arch images
• Nix has multiple ways of doing things

Solutions

• Dagger as a CI solution
• Multistage docker images with distroless -> Small image, small attack surface
• Language specific solutions (ki, jib)

eBPF’s Abilities and Limitations: The Truth

A talk by Isovalent with a full room (one of the large ones).

Baseline

• eBPF lets you run custom code in the kernel -> close to hardware
• Typical use cases: Networking, Observability, Tracing/Profiling, security
• Question: Is eBPF truing complete and can it be used for more complex scenarios (TLS, LK7)?

eBPF verifier

• The verifier analyzes the program to verify safety
• Principles
  • Read memory only with correct permissions
  • All writes to valid and safe memory
  • Valid in-bounds and well-formed control flow
  • Execution on CPU time is bounded: sleep, scheduled callbacks, iterations, program actually completes
  • Acquire/release and reference count semantics

Demo: Game of life

• A random game of life map
• Implemented as a tetragon plugin
• Layout: Main control loop that loads the map, generates the next generation, and returns a next run function
• The timer callback pattern is used for infinite run

eBPF Limits & workarounds

• Instruction limit to let the verifier actually verify the program in reasonable time
  • Limit is based on: Instruction limit and verifier step limit
  • nowadays the limit it 4096 unprivileged calls and 1 million privileged instructions
• Only jump forward -> No loops
  • Is a basic limitation to ensure no infinite loops can ruin the day
  • Limitation: Only finite iterations can be performed
  • Loops: Newer versions support loops with upper bounds (for x=0;: x<100)
• Is the instruction limit hard?
  • Solution: subprogram (aka function) and the limit is only for each function -> x*subprogramms = x*limit
  • Limit: Needs real skill
• Programs have to terminate
  • Well eBPF really only wants to release the CPU, the program doesn’t have to end per se
  • Iterator: walk arbitrary lists of objects
  • Sleep on page fault or other memory operations
  • Timer callbacks (including the timer 0 for run me asap)
• Memory allocation
  • Maps are used as the memory management system

Result

• You can execute arbitrary tasks via eBPF
• It can be used for HTTP or TLS - it’s just not implemented yet™

What's New in Operator Framework?

By the nice operator framework guys at IBM and Red Hat. I’ll skip the baseline introduction of what an operator is.

Operator DSK

Build the operator

• Kubebuilder with v4 Plugins -> Supports the latest Kubernetes
• Java Operator SDK is not a part of Operator SDK, and they released 5.0.0
  • Now with server side apply in the background
  • Better status updates and finalizer handling
  • Dependent resource handling (alongside optional dependent resources)

Operator Lifecycle Manager

Manage the operator -> A operator for installing operators

OLM v1 APIs

• New API Set -> The old CRDs were overwhelming
• More GitOps friendly with per-tenant support
• Prescribes update paths (maybe upgrade)
• Support for operator bundles as k8s manifests/helm chart

OLM v1 Components

• Cluster Extension (User-Facing API)
  • Defines the app you want to install
  • Resolves requirements through CatalogD/depply
• CatalogD (Catalog Server/Operator)
• Depply (Dependency/Constraint solver)
• Applier (Rukoak/kapp compatible)

flowchart TD
    uapi(User facing api)-->|Can I find this operator|catalaogd
    catalogd-->|Check if all dependencies are checked|depply
    depply-->|Please install|kapp

flowchart LR
    oa(operator author)-->ba(Bundle and att to catalog)
    ba-->catalogd(Catalogd Handle unpackling)

    user-->ufa(User facing api)
    ufa-->|Resolve package|catalogd
    ufa-->|Create app on cluster|appcr(App CR / kapps)

Cryptographically Signed Swag: Cert-Manager’s Stamped Certificates

A talk by the cert manager maintainers that also staffed the cert manager booth. Humor is present, but the main focus is still the technical integration

Baseline

• Cert manager is the best™ way of getting certificates
• Poster features: Auto-renewal, ACME, PKI, HC Vault
• Numbers: 20M downloads 427 contributors 11.3 GitHub stars
• Currently on the graduation path

History

• 2016: Jetstack created kube-lego -> A operator that generated LE certificates for ingress based on annotations
• 2o17: Cert manager launch -> Cert resources and issuer resources
• 2020: v1.0.0 and joined CNCF sandbox
• 2022: CNCF incubating
• 2024: Passed the CNCF security audit and on the way to graduation

The booth works

How it came to be

• The idea: Mix the digital certificate with the classical seal
• Started as the stamping idea to celebrate v1 and send contributors a thank you with candles
• Problems: Candles are not allowed -> Therefor glue gun

How it works

• Components
  • Raspberry Pi with k3s
  • Printer
  • Cert manager
  • A Go-based Web-UI
• QR-Code: Contains link to certificate with private key

flowchart LR
    ui(UI in go)-->|Generate cert ressource|kubeapi
    kubeapi-->|Issue certificate|CertManager
    CertManager-->|Certificate|ui
    ui-->|print|Printer

What is new this year

• Idea: Certs should be usable for TLS
• Solution: The QR-Code links to a zip-download with the cert and private key
• New: ECDSA for everything
• New: A stable root ca with intermediate for every conference
• New: Guestbook that can only be signed with a booth issued certificate -> Available via script

Learnings

• This demo is just a private CA with cert manager -> Can be applied to any PKI-usecases
• The certificate can be created via the CR, CSI driver (create secret and mount in container), ingress annotations, …
• You can use multiple different Issuers (CA Issuer aka PKI, Let’s Encrypt, Vault, AWS, …)

flowchart LR
    ui-->|Input certificate subject details|CertManager
    cai(CA Issuer)-->|Source for certificate|CertManager
    CertManager-->|Creates|sr(Secret Ressource)

Conclusion

• This is not just a demo -> Just apply it for machines
• They have regular meetings (daily stand-ups and bi-weekly)

TikTok’s Edge Symphony: Scaling Beyond Boundaries with Multi-Cluster Controllers

A talk by TikTok/ByteDance (duh) focussed on using central controllers instead of on the edge.

Background

Global means non-china

• Edge platform team for CDN, livestreaming, uploads, real-time communication, etc.
• Around 250 cluster with 10-600 nodes each - mostly non-cloud aka bare-metal
• Architecture: Control plane clusters (platform services) - data plane clusters (workload by other teams)
• Platform includes logs, metrics, configs, secrets, …

Challenges

Operators

• Operators are essential for platform features
• As the feature requests increase, more operators are needed
• The deployment of operators throughout many clusters is complex (namespace, deployments, policies, …)

Edge

• Limited resources
• Cost implication of platform features
• Real time processing demands by platform features
• Balancing act between resources used by workload vs platform features (20-25%)

The classic flow

1. New feature gets requested
2. Use kubebuider with the SDK to create the operator
3. Create namespaces and configs in all clusters
4. Deploy operator to all clusters

Possible Solution

Centralized Control Plane

• Problem: The controller implementation is limited to a cluster boundary
• Idea: Why not create a single operator that can manage multiple edge clusters
• Implementation: Just modify kubebuilder to accept multiple clients (and caches)
• Result: It works -> Simpler deployment and troubleshooting
• Concerns: High code complexity -> Long familiarization
• Balance between “simple central operator” and operator-complexity is hard

Attempt it a bit more like kubebuilder

• Each cluster has its own manager
• There is a central multimanager that starts all the cluster specific manager
• Controller registration to the manager now handles cluster names
• The reconciler knows which cluster it is working on
• The multi cluster management basically just test all the cluster secrets and create a manager+controller for each cluster secret
• Challenges: Network connectivity
• Solutions:
  • Dynamic add/remove of clusters with go channels to prevent pod restarts
  • Connectivity health checks -> For loss the recreate manager gets triggered

flowchart TD
    mcm-->m1
    mcm-->m2
    mcm-->m3

flowchart LR
    secrets-->ch(go channels)
    ch-->|CREATE|create(Create manager + Add controller + Start manager)
    ch-->|UPDATE|update(Stop manager + Create manager + Add controller + Start manager)
    ch-->|DELETE|delete(Stop manager)

Conclusion

• Acknowledge resource constraints on edge
• Embrace open source adoption instead of build your own
• Simplify deployment
• Recognize your own opinionated approach and it’s use cases

Fluent Bit v3: Unified Layer for Logs, Metrics and Traces

The last talk of the conference. Notes may be a bit unstructured due to tired note taker.

Background

• FluentD is already graduated
• FluentBit is a daughter-project of FluentD (also graduated)

Basics

• FluentBit is compatible with
  • Prometheus (It can replace the Prometheus scraper and node exporter)
  • OpenMetrics
  • OpenTelemetry (HTTPS input/output)
• FluentBit can export to Prometheus, Splunk, InfluxDB or others
• So pretty much it can be used to collect data from a bunch of sources and pipe it out to different backend destinations
• Fluent ecosystem: No vendor lock-in to observability

Architectures

• The fluent agent collects data and can send it to one or multiple locations
• FluentBit can be used for aggregation from other sources

In the Kubernetes logging ecosystem

• Pod logs to console -> Streamed stdout/err gets piped to file
• The logs in the file get encoded as JSON with metadata (date, channel)
• Labels and annotations only live in the control plane -> You have to collect it additionally -> Expensive

New stuff

Limitations with classic architectures

• Problem: Multiple filters slow down the main loop

flowchart LR
    subgraph main[Main Thread/Event loop]
        buffer
        schedule
        retry
        fitler1
        filter2
        filter3
    end
    in-->|pipe in data|main
    main-->|filter and pipe out|out

Solution

• Solution: Processor - a separate thread segmented by telemetry type
• Plugins can be written in your favorite language (c, rust, go, …)

flowchart LR
    subgraph in
        reader
        streamner1
        processor2
        processor3
    end
    in-->|pipe in data|main(Main Thread/Event loop)
    main-->|filter and pipe out|out

General new features in v3

• Native HTTP/2 support in core
• Content modifier with multiple operations (insert, upsert, delete, rename, hash, extract, convert)
• Metrics selector (include or exclude metrics) with matcher (name, prefix, substring, regex)
• SQL processor -> Use SQL expression for selections (instead of filters)
• Better OpenTelemetry output

Networking

Who have I talked to today, are there any follow-ups or learnings?

Fastly

• They were nice and are always up to talk if we ever need something

Ozone

• An interesting tektone-based CI/CD solutions that also integrates with other platforms
• May be interesting for either ODIT.Services or some of our customers

Docker

• Talked to one salesperson just about the general conference
• Talked to one technical guy about docker build time optimization

Rancher/SUSE

• I just got some swag, a friend of mine got a demo focussing on runtime security

Kong

• They didn’t have any Insomina stickers and the insomnia guy apparently already left

Planetscale

• Said hi and talked a bit about the future of vitess
• Got some vitess stickers