Cloud SIEM: Log Collection and Normalisation

Not every log source has equal investigative value, and cloud logging costs scale with volume, so source prioritisation is a real design decision rather than "collect everything." A reasonable starting set for most cloud environments covers control-plane activity, network flow, identity, and workload-level events.

Cloud logs generally reach a SIEM through one of two patterns: cloud-native log services push events to a streaming destination (an event bus, message queue, or storage bucket) that the SIEM subscribes to, or the SIEM's collector agents pull logs via API on a polling interval. Push-based collection (e.g., CloudTrail to an S3 bucket with event notifications, or subscription filters on CloudWatch Logs feeding a Kinesis stream) generally gives lower latency and lower API rate-limit risk than polling-based collection against a cloud provider's logging API.

Normalisation — mapping each source's native field names and event types to a common schema — is best done as close to ingestion as possible, either in a lightweight collector/forwarder or in the SIEM's ingest pipeline, rather than deferred to query time. Deferring normalisation to query time means every detection rule and every analyst has to know the quirks of each raw log format, which doesn't scale past a handful of sources.

Cross-account and cross-subscription aggregation adds another layer of design work in any organisation with more than a handful of cloud accounts. A common pattern is a dedicated logging or security account/subscription that every workload account forwards to — for example, an AWS Organizations setup using a delegated CloudTrail administrator account, or an Azure Lighthouse/management-group-scoped diagnostic setting — so that log collection configuration is enforced centrally rather than depending on every account team remembering to wire it up themselves.

Several open normalisation schemas exist and most modern SIEMs support at least one natively — the Open Cybersecurity Schema Framework (OCSF), Elastic Common Schema (ECS), and the older Common Event Format (CEF) are the most widely adopted. Picking one and mapping every source to it consistently is more valuable than which specific schema is chosen.

At minimum, normalised events should carry a consistent representation of: event timestamp (in a single timezone, ideally UTC), actor identity (user, service account, or role), source and destination network context, the action taken, the target resource, and the outcome (success/failure/denied). Detection rules and correlation searches written against these normalised fields keep working when a new log source is onboarded, rather than needing a rewrite per source.

{
  "event_time": "2026-03-14T09:12:47Z",
  "event_type": "iam.policy.attach",
  "actor": {
    "type": "iam_user",
    "id": "AIDAEXAMPLE123",
    "name": "svc-deploy"
  },
  "source": {
    "ip": "203.0.113.44",
    "cloud_provider": "aws",
    "account_id": "123456789012",
    "region": "us-east-1"
  },
  "target": {
    "resource_type": "iam_role",
    "resource_id": "arn:aws:iam::123456789012:role/app-role"
  },
  "action": "AttachRolePolicy",
  "outcome": "success",
  "raw_event_source": "cloudtrail"
}

VPC flow logs and DNS query logs are typically the highest-volume sources in a cloud environment, and ingesting them at full fidelity into a SIEM can dominate the licensing or ingestion cost of the whole pipeline. Common cost-control patterns include filtering out known-noisy, low-value traffic before ingestion (health checks, internal load balancer probes), routing full-fidelity raw logs to cheap object storage for retention and forensic replay while sending a filtered or summarised stream to the SIEM for active detection, and tiering retention so recent data sits in hot, queryable storage while older data moves to cold storage that's still retrievable for investigations.

Whatever filtering is applied, document it. An analyst investigating an incident needs to know what was deliberately excluded from the SIEM, so they know when to go back to the raw archive instead of trusting an apparent absence of evidence.

• •Route raw logs to cheap, durable storage (S3/Blob/Cloud Storage) as the source of truth, independent of SIEM retention limits
• •Apply sampling or filtering only to genuinely low-value, high-volume noise — not broadly across a source
• •Track ingestion cost per log source so filtering decisions are based on data, not guesswork
• •Set SIEM hot-tier retention based on realistic investigation timelines, with a documented path to pull older data back in when needed

A disproportionate number of investigation delays come down to timestamp handling rather than missing data. Cloud services log timestamps in different formats and occasionally different timezones by default; ingest pipelines that don't normalise to UTC consistently will produce a SIEM where event ordering across sources can't be trusted during an incident timeline reconstruction. Confirm normalisation preserves sub-second precision where the source provides it — correlating rapid API call sequences during an incident can depend on it.

Log pipelines silently break — an IAM permission changes, a subscription filter gets deleted, a service migrates and a new log source is added but never wired into collection. Coverage should be actively monitored, not assumed.

• •Alert on a drop in expected event volume per source (a sudden silence is often more urgent than a spike)
• •Periodically generate a known test event per source and confirm it's searchable in the SIEM within the expected latency
• •Maintain an inventory mapping every cloud account/subscription/project to its expected log sources, and audit against actual ingestion
• •Re-validate coverage whenever a new account, region, or service is onboarded — collection configuration doesn't propagate automatically