Feature request: cleanup mechanism for connectivity test artifacts #3161
Description
Proposal / RFE
Connectivity test leaves test resources behind with no built-in cleanup mechanism.
Is your feature request related to a problem?
When running cilium connectivity test, the command deploys multiple namespaces (cilium-test-*) and associated resources (Pods, Services, DaemonSets, Policies).
After the tests complete successfully, all test resources remain running, and there is currently no built-in cleanup command or flag in the Cilium CLI to remove them.
While manual deletion of namespaces works, the lack of a discoverable cleanup mechanism or documentation makes this easy to miss.
Describe the solution you'd like
Add an explicit, opt-in cleanup mechanism for connectivity test artifacts, for example:
A dedicated flag, such as:
cilium connectivity test --cleanupIt’s possible that a cleanup mechanism already exists, but if so, it is not discoverable via the CLI.
Reviewing the output of cilium connectivity test --help does not reveal any flag or subcommand related to cleanup or teardown of test resources. Given the large number of available flags, this makes it difficult for users to know whether cleanup is supported or expected to be done manually.
As a result, users may reasonably assume that test resources are ephemeral or automatically removed, only to later discover multiple cilium-test-* namespaces and associated resources still running in the cluster.
~/Desktop/ cilium connectivity test --help
Validate connectivity in cluster
Usage:
cilium connectivity test [flags]
Flags:
--agent-daemonset-name string Name of cilium agent daemonset (default "cilium")
--agent-pod-selector string Label on cilium-agent pods to select with (default "k8s-app=cilium")
--all-flows Print all flows during flow validation
--assume-cilium-version string Assume Cilium version for connectivity tests
--chart-directory string Helm chart directory
--cilium-pod-selector string Label selector matching all cilium-related pods (default "app.kubernetes.io/part-of=cilium")
--collect-sysdump-on-failure Collect sysdump after a test fails
--conn-disrupt-dispatch-interval duration TCP packet dispatch interval
--conn-disrupt-test-restarts-path string Conn disrupt test temporary result file (used internally) (default "/tmp/cilium-conn-disrupt-restarts")
--conn-disrupt-test-setup Set up conn disrupt test dependencies
--conn-disrupt-test-xfrm-errors-path string Conn disrupt test temporary result file (used internally) (default "/tmp/cilium-conn-disrupt-xfrm-errors")
--connect-timeout duration Maximum time to allow initiation of the connection to take (default 2s)
--curl-image string Image path to use for curl (default "quay.io/cilium/alpine-curl:v1.10.0@sha256:913e8c9f3d960dde03882defa0edd3a919d529c2eb167caa7f54194528bde364")
--curl-insecure Pass --insecure to curl
--curl-parallel uint Number of parallel requests in curl commands (0 to disable)
-d, --debug Show debug messages
--dns-test-server-image string Image path to use for CoreDNS (default "registry.k8s.io/coredns/coredns:v1.12.4@sha256:986f04c2e15e147d00bdd51e8c51bcef3644b13ff806be7d2ff1b261d6dfbae1")
--echo-image string Image path to use for echo server (default "gcr.io/k8s-staging-gateway-api/echo-advanced:v20240412-v1.0.0-394-g40c666fd")
--external-cidr string IPv4 CIDR to use as external target in connectivity tests (default "1.0.0.0/8")
--external-cidrv6 string IPv6 CIDR to use as external target in connectivity tests (default "2606:4700:4700::/96")
--external-ip string IPv4 to use as external target in connectivity tests (default "1.1.1.1")
--external-ipv6 string IPv6 to use as external target in connectivity tests (default "2606:4700:4700::1111")
--external-other-ip string Other IPv4 to use as external target in connectivity tests (default "1.0.0.1")
--external-other-ipv6 string Other IPv6 to use as external target in connectivity tests (default "2606:4700:4700::1001")
--external-other-target string Domain name to use as a second external target in connectivity tests (default "k8s.io.")
--external-target string Domain name to use as external target in connectivity tests (default "one.one.one.one.")
--external-target-ca-name string Name of the CA secret for the external target. (default "cabundle")
--external-target-ca-namespace string Namespace of the CA secret for the external target.
--external-target-ipv6-capable External target is IPv6 capable
--flow-validation string Enable Hubble flow validation { disabled | warning | strict } (default "warning")
--force-deploy Force re-deploying test artifacts
--frr-image string Image path to use for FRR (default "quay.io/frrouting/frr:10.5.0@sha256:fc7f887ab4d8da06f481a4f8d59afded88b3c5823f03610a7e808f7eba45eeea")
--helm-values-secret-name string Secret name to store the auto-generated helm values file. The namespace is the same as where Cilium will be installed (default "cilium-cli-helm-values")
-h, --help help for test
--hubble Automatically use Hubble for flow validation & troubleshooting (default true)
--hubble-server string Address of the Hubble endpoint for flow validation (default "localhost:4245")
--include-conn-disrupt-test Include conn disrupt test
--include-conn-disrupt-test-egw Include conn disrupt test for Egress Gateway
--include-conn-disrupt-test-ns-traffic Include conn disrupt test for NS traffic
--ip-families strings Restrict test actions to specific IP families (default [ipv4,ipv6])
--json-mock-image string Image path to use for json mock (default "quay.io/cilium/json-mock:v1.3.9@sha256:c98b26177a5a60020e5aa404896d55f0ab573d506f42acfb4aa4f5705a5c6f56")
--junit-file string Generate junit report and write to file
--junit-property map Add key=value properties to the generated junit file
--k8s-version string Kubernetes server version in case auto-detection fails
--multi-cluster string Test across clusters to given context
--namespace-labels map Add labels to the connectivity test namespace
--node-cidr strings one or more CIDRs that cover all nodes in the cluster
--node-selector map Restrict connectivity pods to nodes matching this label
-p, --pause-on-fail Pause execution on test failure
--post-test-sleep duration Wait time after each test before next test starts
--print-flows Print flow logs for each test
--print-image-artifacts Prints the used image artifacts
--request-timeout duration Maximum time to allow a request to take (default 10s)
--retry uint Number of retries on connection failure to external targets (default 3)
--retry-delay duration Delay between retries for external targets (default 3s)
--secondary-network-iface string Secondary network iface name (e.g., to test NodePort BPF on multiple networks)
--service-type string Type of Kubernetes Services created for connectivity tests (default "NodePort")
--single-node Limit to tests able to run on a single node
--socat-image string Image path to use for multicast tests (default "docker.io/alpine/socat:1.8.0.3@sha256:b857f6b307559525fa7e233b4e79dfb9a9c9e1b555a6f7fbeddbbbf4270cc124")
--sysdump-cilium-bugtool-flags stringArray Optional set of flags to pass to cilium-bugtool command.
--sysdump-cilium-daemon-set-label-selector string The labels used to target Cilium daemon set (default "k8s-app=cilium")
--sysdump-cilium-envoy-label-selector string The labels used to target Cilium Envoy pods (default "k8s-app=cilium-envoy")
--sysdump-cilium-helm-release-name string The Cilium Helm release name for which to get values. If not provided then the --helm-release-name global flag is used (if provided)
--sysdump-cilium-label-selector string The labels used to target Cilium pods (default "k8s-app=cilium")
--sysdump-cilium-namespace string The namespace Cilium is running in. If not provided then the --namespace global flag is used (if provided)
--sysdump-cilium-node-init-selector string The labels used to target Cilium node init pods (default "app=cilium-node-init")
--sysdump-cilium-operator-label-selector string The labels used to target Cilium operator pods (default "io.cilium/app=operator")
--sysdump-cilium-operator-namespace string The namespace Cilium operator is running in. If not provided then the --namespace global flag is used (if provided)
--sysdump-cilium-spire-agent-selector string The labels used to target Cilium spire-agent pods (default "app=spire-agent")
--sysdump-cilium-spire-namespace string The namespace Cilium SPIRE installation is running in
--sysdump-cilium-spire-server-selector string The labels used to target Cilium spire-server pods (default "app=spire-server")
--sysdump-clustermesh-apiserver-label-selector string The labels used to target 'clustermesh-apiserver' pods (default "k8s-app=clustermesh-apiserver")
--sysdump-cni-config-directory string Directory where CNI configs are located (default "/etc/cni/net.d/")
--sysdump-cni-configmap-name string The name of the CNI config map (default "cni-configuration")
--sysdump-collect-logs-from-not-ready-agents Whether to collect logs from not ready Cilium agent pods (default true)
--sysdump-copy-retry-limit int Retry limit for file copying operations. If set to -1, copying will be retried indefinitely. Useful for collecting sysdump while on unreliable connection. (default 100)
--sysdump-debug Whether to enable debug logging
--sysdump-detect-gops-pid Whether to automatically detect the gops agent PID.
--sysdump-extra-label-selectors stringArray Optional set of labels selectors used to target additional pods for log collection.
--sysdump-hubble-flows-count int Number of Hubble flows to collect. Setting to zero disables collecting Hubble flows. (default 10000)
--sysdump-hubble-flows-timeout duration Timeout for collecting Hubble flows (default 5s)
--sysdump-hubble-generate-certs-labels string The labels used to target Hubble UI pods (default "k8s-app=hubble-generate-certs")
--sysdump-hubble-label-selector string The labels used to target Hubble pods (default "k8s-app=hubble")
--sysdump-hubble-relay-labels string The labels used to target Hubble Relay pods (default "k8s-app=hubble-relay")
--sysdump-hubble-ui-labels string The labels used to target Hubble UI pods (default "k8s-app=hubble-ui")
--sysdump-logs-limit-bytes int The limit on the number of bytes to retrieve when collecting logs (default 1073741824)
--sysdump-logs-since-time duration How far back in time to go when collecting logs (default 8760h0m0s)
--sysdump-node-list string Comma-separated list of node IPs or names to filter pods for which to collect gops and logs
--sysdump-output-filename string The name of the resulting file (without extension)
'<ts>' can be used as the placeholder for the timestamp (default "cilium-sysdump-<ts>")
--sysdump-profiling Whether to enable scraping profiling data (default true)
--sysdump-quick Whether to enable quick mode (i.e. skip collection of 'cilium-bugtool' output and logs)
--sysdump-tetragon-helm-release-name string The Tetragon Helm release name for which to get values.
--sysdump-tetragon-label-selector string The labels used to target Tetragon pods (default "app.kubernetes.io/name=tetragon")
--sysdump-tetragon-namespace string The namespace Tetragon is running in (default "kube-system")
--sysdump-tetragon-operator-label-selector string The labels used to target Tetragon operator pods (default "app.kubernetes.io/name=tetragon-operator")
--sysdump-tracing Whether to enable scraping tracing data
--sysdump-worker-count int The number of workers to use
NOTE: There is a lower bound requirement on the number of workers for the sysdump operation to be effective. Therefore, for low values, the actual number of workers may be adjusted upwards. Defaults to the number of available CPUs. (default 20)
--test strings Run tests that match one of the given regular expressions, skip tests by starting the expression with '!', target Scenarios with e.g. '/pod-to-cidr'
--test-concurrency int Count of namespaces to perform the connectivity tests in parallel (value <= 0 will be treated as 1) (default 1)
--test-conn-disrupt-image string Image path to use for connection disruption tests (default "quay.io/cilium/test-connection-disruption:v0.0.16@sha256:e8e3257b2c89543dc49a2d820f2d2d69c1fe60eaf1036fc1f1f7375bad8e6232")
--test-namespace string Namespace to perform the connectivity in (always suffixed with a sequence number to be compliant with test-concurrency param, e.g.: cilium-test-1) (default "cilium-test")
--timeout duration Maximum time to allow the connectivity test suite to take
-t, --timestamp Show timestamp in messages
--tolerations strings Extra NoSchedule tolerations added to test pods
-v, --verbose Show informational messages and don't buffer any lines
Global Flags:
--as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.
--as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.
--context string Kubernetes configuration context
--helm-release-name string Helm release name (default "cilium")
--kubeconfig string Path to the kubeconfig file
-n, --namespace string Namespace Cilium is running in. Can also be set via CILIUM_NAMESPACE env var (default "kube-system")Environment
Cilium versions:
- cilium-cli: v0.18.9 (compiled with go1.25.5 on darwin/arm64)
- cilium image (default): v1.18.3
- cilium image (stable): v1.18.5
Kubernetes version:
- Client Version: v1.34.1
- Kustomize Version: v5.7.1
- Server Version: v1.34.2-eks-b3126f4