The incident arrives late: someone detects anomalous outbound connections from a service in Kubernetes, but the pod was already rescheduled by auto-scaling and the original container no longer exists. In cloud this is not a rarity; it is the normal behavior of the system. The forensic challenge is not “getting into the container” (because it is no longer there), but reconstructing a reliable timeline from external, immutable, and previously enabled evidence.
If your incident response depends on “I connect to the host and look at /var/log”, in ephemeral environments you are going to lose every time. The only way to have useful forensics is to design before the incident: decide what telemetry is retained, how it is protected against tampering, and what mechanisms allow capturing state (disk/memory) under quarantine at the right moment.
What went wrong when the evidence disappears (and why it is not “bad luck”)
In an environment with auto-scaling, spot nodes, or serverless, the infrastructure is optimized for availability and cost, not for evidence retention. A compromised pod can live minutes; the node can be recycled; the runtime can clean layers; and local logs can go away with the container. If there was no centralized and immutable channel, the attacker does not need to delete anything: the platform does it for them.
The usual failure in companies is operational, not technical: it is assumed that “we will see” how to investigate when it happens. Then, in the middle of an IR, improvisation happens: logs are enabled late, configurations are changed (contaminating the scene), or platform is asked “not to scale” without a clear plan. That generates two real consequences: loss of traceability (you cannot attribute actions) and longer containment times (because you do not know what to touch without breaking production).
Reconstructing the timeline without the container: external evidence that does survive
When the container no longer exists, your forensics relies on signals that live outside the pod lifecycle: network, control plane, and execution. In practice, what saves incidents most is having retained network logs (to answer “who talked to whom, when, and how much”) and execution logs (to answer “what was executed and with what identity”). Without that, the investigation becomes a collection of hypotheses that are hard to prove.
Two sources are often mandatory in corporate cloud because of their forensic value and because of their low level of ambiguity: VPC Flow Logs (or equivalent) and Execution Logs of the compute layer (functions, gateways, runtimes, control plane auditing). The key is not only to enable them, but to retain them long enough, with integrity, and with a scheme that allows fast queries during IR.
- VPC Flow Logs: provide context on exfiltration, C2, and lateral movement at the network level even when the container died.
In a company, this makes it possible to correlate egress spikes with deployments, image changes, or pod rotations. It also helps to demonstrate impact: it is not the same a connection attempt blocked by SG/NACL as established sessions with bytes transferred.
- Execution Logs (and control plane auditing): capture calls, errors, invocations and, above all, identities and execution context.
In a typical real case, an attacker does not “break” Kubernetes; they abuse workload credentials. Execution logs and control plane logs are what tell you whether there was pod creation, role changes, secrets reads, or unusual invocations. If they are missing, internal attribution is lost: you do not know if it was a specific pod, a pipeline, or a human operator.
Containment without destroying evidence: snapshots under quarantine of disks and nodes
When you suspect compromise, the temptation is to restart or redeploy. That is often correct for availability, but it is bad for forensics if you do not capture state first. In cloud, the corporate equivalent of “cloning the disk” is creating snapshots (for example, EBS or managed disks) and treating them as evidence: with restricted permissions, defined retention, and operational chain of custody.
The important nuance: in containers, the container filesystem is usually ephemeral, but nodes and persistent volumes can contain critical pieces (downloaded binaries, attacker tools, dumps, artifacts in /tmp, runtime logs, or agent logs). If the pod used PV/PVC, the snapshot of the volume is one of the few “material” pieces of evidence that survive the death of the container.
- Snapshot of the persistent volume (if it exists): prioritize the disk that the workload mounted (for example, data or caches) before cleanup or rotation processes alter it.
This is useful when the attacker left artifacts in working paths or manipulated data. In corporate IR, in addition, it allows you to work in parallel: you contain in production and analyze outside, without blocking operations.
- Snapshot of the node disk (when applicable): if the node hosts runtime artifacts, agent logs, or layers that are not replicated elsewhere, capturing it can make the difference.
In practice, this requires discipline: tag the resource, isolate it (quarantine account/project or at least restrictive IAM) and prevent someone from “reusing” it for tests. A frequent antipattern is sharing snapshots for convenience between teams without access control; that breaks evidence integrity and complicates internal audits.
How to do it in practice: memory and state capture in Kubernetes before the pod dies
If the pod is still alive, your window is short. Memory can contain indicators that never reach disk: credentials in cleartext, active connections, process injection, or fileless payloads. In Kubernetes environments, what is viable is usually to rely on open-source tools and on kernel capabilities (eBPF / tracing) to extract signals without depending on getting “manually” into the container with interactive sessions that alter the state.
The operational point here is to have a playbook and permissions previously defined. In companies, the typical blocker is not technical: it is that the security team does not have RBAC to create a response pod/daemonset, or does not have access to the node. When that is negotiated on the fly, the pod is already dead and the only thing left is incomplete logs.
- Prepare a controlled response DaemonSet: deploy (only when authorized) an open-source forensic agent that can collect node artifacts and, if appropriate, memory dumps of processes.
In corporate production, this is handled as “break glass”: versioned manifests, reviewed by security and platform, and deployable with an explicit approval. The advantage is that you reduce improvisation and ensure that what you capture is consistent between incidents.
- Collect the volatile first: active connections, processes, memory maps, and recent runtime events before trying to copy files.
The practical consequence: if you start by “copying the filesystem”, many times you arrive late to what matters. Memory changes every second. In a cryptominer or reverse shell case, capturing connections and processes in the right minute can give you the C2 domain/IP and the injected binary, which speeds up containment across the entire fleet.
Example of IAM control (AWS) for snapshots with tags and isolation:
A realistic way to avoid mistakes is to allow only the creation of snapshots when the volume/disk is marked for IR, and deny deletion. This pattern reduces the probability that someone captures “whatever” or deletes evidence by accident.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowCreateSnapshotOnlyForIRTaggedVolumes",
"Effect": "Allow",
"Action": [
"ec2:CreateSnapshot",
"ec2:CreateTags"
],
"Resource": "*",
"Condition": {
"StringEquals": {
"ec2:ResourceTag/IR": "true"
}
}
},
{
"Sid": "DenyDeleteSnapshots",
"Effect": "Deny",
"Action": [
"ec2:DeleteSnapshot"
],
"Resource": "*"
}
]
}
How to validate it in AWS during an incident: verify in CloudTrail that CreateSnapshot was executed by the intended IR role, check that the snapshot has case tags (incident ID, owner, timestamp) and confirm that the role cannot execute DeleteSnapshot. If your CloudTrail query does not show the event, the snapshot may have been made in another account/region or with another identity (a finding in itself).
Recommendations for corporate environments
If the compromised container no longer exists, forensics depends on what survives the ephemeral lifecycle: network logs (such as VPC Flow Logs), execution logs, and control plane auditing. In practice, these records are what allow reconstructing a defensible timeline and reducing containment time without relying on assumptions.
When state needs to be captured, snapshots under quarantine (persistent volumes and, if applicable, node disks) are the most operational mechanism in cloud, as long as they are governed by IAM and by a tagging and retention process. And if the pod is still alive, the useful window is gained with prior preparation: permissions, playbooks, and open-source tools to collect memory/volatile signals before the scheduler does its job and erases the scene.
Interested in Cloud Security?
Technical analysis, hands-on labs and real-world cloud security insights.
Francisco Sepúlveda
Cloud Computing & Cybersecurity Instructor, Learning Experience Designer
Instructor and consultant in Cloud Computing, Cybersecurity and AI, with more than 25 years of hands-on experience in real enterprise environments. I design and deliver technical training programs, combining deep subject-matter expertise with learning experience design to make complex security concepts stick. I share practical, applied analysis on Cloud Security in Practice.
More about me →