Chaos on a timetable¶
kube-monkey deletes pods at random on apps that have opted in. It publishes the day's terminations each morning and runs them during working hours, while the people who own the service are still at their desks.
| Kind | Namespace | Name | Termination time |
|---|---|---|---|
| v1.Deployment | default | monkey-victim | 10:14:07 |
| v1.Deployment | payments | checkout-api | 11:32:55 |
| v1.StatefulSet | payments | ledger | 13:47:19 |
| v1.DaemonSet | observability | log-shipper | 15:02:41 |
kube-monkey writes this to the log at run_hour on each
of its run_days. Every termination lands between start_hour and
end_hour.
kube-monkey is an implementation of Netflix's Chaos Monkey for Kubernetes clusters. It randomly deletes pods in the cluster, encouraging and validating the development of failure-resilient services.
Safe by default¶
- Opt in, never opt out
- Nothing is touched until an app asks for it with a label. A team adopts chaos when it is ready, not when the cluster operator flips a switch.
- Office hours only
- Terminations are scheduled inside a window you choose, on the days you choose. Pods die while the people who own them are at their desks.
- Dry run until you say otherwise
- Out of the box kube-monkey only logs what it would have killed. You see the blast radius on paper before anything real happens.
What you can tune¶
- Blast radius
- Kill one pod, a fixed number, a fixed percentage, a random percentage up to a cap, or the whole app. Set per app, by a label.
- Prometheus metrics
- Schedules built, terminations attempted, pods actually deleted. Chart the chaos and alert on it like anything else.
- Notifications
- Post every attack to a Slack webhook or your own API, with placeholders for the victim, the time and the outcome.
Sixty seconds to your first termination¶
helm repo add kubemonkey https://asobti.github.io/kube-monkey/charts/repo
helm repo update
helm install kube-monkey kubemonkey/kube-monkey --namespace kube-system
That gives you a kube-monkey in dry run mode, touching nothing. Label an app to opt it in:
metadata:
labels:
kube-monkey/enabled: enabled
kube-monkey/identifier: monkey-victim
kube-monkey/mtbf: "2"
Then read Getting started for the walk through, including how to watch a real termination happen in debug mode before you trust it with a live namespace.
Talk to us¶
Join #kube-monkey on Kubernetes Slack, or open an issue on GitHub.
