Why Most Cloud Threat Hunting Programs Fail

The pattern is so consistent we have a shorthand for it internally: logs-rich, hunt-poor. The customer is paying seven figures a year for log ingest. Their SIEM contains terabytes a day. And yet, when an incident hits, the hunter cannot answer the questions that matter: which identity assumed the role at 02:14 UTC, what role chain did they walk, which object did they read.

Three failure modes recur:

• Wrong logs. Teams ingest VPC Flow Logs (cheap-per-flow, expensive-per-TB) and skip identity sign-in events (the actual epicentre of cloud attack chains). The result is network-heavy, identity-blind telemetry.
• Right logs, wrong fields. The team ingests CloudTrail but discards requestParameters, userAgent, or sessionContext to save space. The hunt query that needed those fields returns nothing — not because the activity did not happen, but because the evidence was thrown away upstream.
• Right logs, wrong window. Mandiant’s median dwell time has been 84-110 days for years. If your hot retention is 30 days and your cold retention is searchable only via an offline restore, your hunt cannot span the actual lifespan of the intrusion.

Each of those failure modes costs nothing to fix at architecture time and costs a forensic engagement to discover after the fact.

The Log Dependency Map: Technique → Log → Cost

Every detection or hunt query has a log dependency. If the log is not ingested with the right fields and retention, the detection does not exist — it is a stub. The map below shows the relationship between a sample of cloud-relevant ATT&CK techniques and the log source required to hunt them.

Figure 1 — Hunt viability collapses when any link in the chain is missing

Three observations matter for budgeting. First, identity logs (sign-in, IAM) and the management-plane audit log (CloudTrail / Activity Log / GCP Audit) carry the heaviest load — they are required for most persistence, privilege escalation, and lateral-movement techniques. Second, VPC Flow Logs cover surprisingly few cloud-native techniques on their own. They are essential for east-west crypto-mining or beacon detection, but for cloud-control-plane attacks they are largely orthogonal. Third, storage data events (S3, Blob, GCS) are the most expensive single source — enabling them globally is almost never the right call. Enable per-bucket, scoped to crown-jewel data.

AWS / Azure / GCP Coverage Matrix

The cross-cloud coverage matrix below maps a selection of MITRE ATT&CK Cloud Matrix techniques to the equivalent log sources in AWS, Azure, and GCP. Cell values indicate practical coverage: FULL means the log routinely contains the field needed to detect the technique; PART means coverage requires combining multiple log streams; NONE means the technique cannot be detected from this source.

What Happens When the Logs Are Missing

The abstract argument for full log coverage is unconvincing until you walk through what blind spots actually let through. Three concrete attack scenarios — each based on patterns observed across recent cloud incident response engagements:

Cost vs Security Trade-off: Order of Magnitude

The argument “we cannot afford comprehensive logs” almost always comes from a misallocation rather than a true budget ceiling. The scatter plot below shows the order-of-magnitude cost per terabyte ingested for the major cloud log sources, alongside the coverage they deliver against the MITRE Cloud Matrix.

Figure 2 — The efficient frontier favours identity + management-plane audit over network telemetry

Order-of-magnitude pricing (USD, planning estimates):

• CloudTrail management events: Approximately $80 per TB stored, free for the first copy to S3. The cheapest high-coverage source in cloud.
• Identity sign-in logs: Approximately $220 per TB for AAD diagnostic export or AWS SSO via CloudTrail. Highest coverage-per-dollar in the portfolio.
• VPC Flow Logs: Approximately $3,100 per TB when shipped to a SIEM at full fidelity, depending on per-GB ingest rates of the destination platform. Coverage is narrow.
• CloudTrail data events (S3 / Lambda): Approximately $2,800 per TB and they scale linearly with object operations. Selective enablement only.
• Full packet mirroring (VPC Traffic Mirroring / Azure vTAP): Approximately $18,000 per TB at hyperscale. Justified only for high-value workloads or regulatory mandates.

The implication is direct. If you have $500,000 a year for cloud telemetry, you do not buy 167 TB of VPC Flow Logs — you buy 6.2 PB of CloudTrail or 2.3 PB of identity logs. Network telemetry is a tactical addition for specific TTPs, not a baseline.

The Minimum Viable Log Set for Cloud Hunting

If you start a cloud hunting program from zero, this is the order in which to onboard log sources. Each tier delivers coverage that the next tier compounds.

Tier 1 — The non-negotiables (week one)

• Management-plane audit: CloudTrail (all regions, including the ones you do not use), Azure Activity Log (subscription level), GCP Cloud Audit Logs (Admin Activity, all projects). Retain hot for 90 days, cold-searchable for 12 months.
• Identity sign-in: AWS Identity Center sign-in events via CloudTrail, Azure AD / Entra ID Sign-In Logs (both interactive and non-interactive), GCP Cloud Identity audit. Pair with the identity provider in front (Okta, Ping, Auth0) — same retention.
• DNS query logs: Route 53 Resolver query logs, Azure DNS analytics, GCP Cloud DNS logging. Cheap, ubiquitously useful for C2 detection.

Tier 2 — The targeted additions (month one)

• CloudTrail data events on crown-jewel S3 buckets and Lambda functions only. Tag the buckets, route only tagged events to ingest.
• WAF and load-balancer access logs for any internet-facing application. Tier-2 retention (30 days hot, 90 cold).
• Kubernetes audit logs at the API-server level if you run EKS / AKS / GKE.

Tier 3 — The justified extensions (quarter one)

• VPC Flow Logs with custom fields enabled (TCP flags, packet size, pkt-src-aws-service) for the subnets running production workloads. Sample only on pre-prod.
• GuardDuty / Defender / SCC findings as enrichment, not as primary detection logic.
• Container runtime logs (Falco, Sysdig) for high-trust workloads.

Example: East-West Hunt via Athena on VPC Flow Logs

Once VPC Flow Logs are in place (whether shipped to a SIEM or, more economically, kept in S3 and queried via Athena), the workhorse hunt is east-west anomaly detection. The query below identifies instances initiating connections to internal IPs they have never connected to before within a 14-day baseline — a high-signal indicator of lateral movement or compromised-instance reconnaissance.

Athena SQL — East-West Anomaly Hunt (VPC Flow Logs)

-- Hypothesis: lateral movement from a compromised EC2 instance manifests
-- as connections to internal /16 destinations the source has never reached.
-- Baseline window: 14 days. Detection window: most recent 6 hours.

WITH baseline AS (
  SELECT
    srcaddr,
    dstaddr
  FROM   vpc_flow_logs
  WHERE  start_time BETWEEN DATE_ADD('day', -14, CURRENT_TIMESTAMP)
                        AND DATE_ADD('hour',  -6, CURRENT_TIMESTAMP)
    AND  action  = 'ACCEPT'
    AND  regexp_like(dstaddr, '^(10\.|172\.(1[6-9]|2[0-9]|3[01])\.|192\.168\.)')
  GROUP  BY 1, 2
),
recent AS (
  SELECT
    srcaddr,
    dstaddr,
    dstport,
    SUM(bytes)     AS bytes_out,
    SUM(packets)   AS packet_count,
    COUNT(*)       AS flow_count,
    MIN(start_time) AS first_seen,
    MAX(end_time)   AS last_seen
  FROM   vpc_flow_logs
  WHERE  start_time >= DATE_ADD('hour', -6, CURRENT_TIMESTAMP)
    AND  action  = 'ACCEPT'
    AND  regexp_like(dstaddr, '^(10\.|172\.(1[6-9]|2[0-9]|3[01])\.|192\.168\.)')
  GROUP  BY 1, 2, 3
)
SELECT
  r.srcaddr           AS source_eni,
  r.dstaddr           AS new_internal_dest,
  r.dstport,
  r.flow_count,
  r.bytes_out,
  r.first_seen,
  r.last_seen,
  -- enrichment: how many *other* new dests from same source?
  COUNT(*) OVER (PARTITION BY r.srcaddr) AS fanout_score
FROM   recent r
LEFT   JOIN baseline b
  ON   r.srcaddr = b.srcaddr AND r.dstaddr = b.dstaddr
WHERE  b.srcaddr IS NULL                -- destination never seen in baseline
  AND  r.dstport NOT IN (53, 123)       -- exclude DNS, NTP noise
  AND  r.flow_count >= 3                -- ignore one-shot probes
ORDER  BY fanout_score DESC, r.bytes_out DESC
LIMIT  500;

The query has three properties that matter. It is cost-bounded because Athena scans only the partitioned 14-day window. It is precision-tuned by excluding DNS/NTP and requiring at least three flows. And it includes a fanout score that surfaces compromised hosts touching many novel internal IPs — the canonical lateral-movement footprint.

Where This Fits in the Hunting Stack

This article is part of the Cloud Threat Hunting series. The log set described here is the input to the hunting workflow; the techniques you hunt for come from threat intelligence; the detection logic you author from those hunts becomes part of your detection engineering library; and the coverage you achieve across the MITRE Cloud Matrix is tracked in your MITRE coverage dashboard. The platform architecture page describes how those pipelines connect end-to-end.

FAQ