Federal Regulatory Data Hub change alerts: near-real-time OFAC sanctions, SAM debarments, and enforcement action webhooks

Service status: the public alert subscriptions, webhook delivery service, and sub-15-minute service levels described below are not currently live. This is a reference architecture. Timings in its tables are design targets, not production measurements.

A compliance team screening counterparties against the OFAC SDN list needs more than an API they can query. They need to know within minutes when a counterparty gets added. A defense contractor doing daily SAM debarment checks is missing the point: the question is not whether the entity is debarred today, but whether it became debarred since the last check. Building the Federal Regulatory Data Hub as a static query layer was the first step; building the change-detection and alerting layer on top of it is what makes it operationally useful.

This post lays out how such a system could detect changes across sources with different publication cadences, classify those changes into actionable event types, and deliver webhooks with at-least-once semantics.

Change detection architecture

Different datasets have different publication patterns, which means change detection cannot be uniform. The design uses three strategies depending on the source's update mechanism:

The OFAC conditional-GET approach polls the SDN publication endpoint every 10 minutes with an If-None-Match header using the stored ETag. OFAC typically updates the list between 10am–1pm ET on business days; the 10-minute window catches updates within two polling cycles. When OFAC returns HTTP 200 (content changed), the ingest pipeline runs immediately rather than waiting for the next scheduled cron.

Record-level diffing

Detecting that the OFAC XML file changed is not enough — we need to know which specific entries were added, modified, or removed to generate meaningful events. We maintain a hash index on normalized records and diff against it after each ingest.

-- Record hash index in Cloudflare D1
CREATE TABLE record_hashes (
  dataset_id    TEXT NOT NULL,
  record_id     TEXT NOT NULL,   -- source's primary key (OFAC uid, SAM CAGE, etc.)
  content_hash  TEXT NOT NULL,   -- SHA-256 of normalized record JSON
  last_seen_at  TIMESTAMPTZ NOT NULL,
  PRIMARY KEY (dataset_id, record_id)
);

-- Change detection query (after new ingest batch)
-- Returns (added, modified, removed) record_ids
WITH new_hashes AS (
  SELECT record_id, content_hash FROM staging_records WHERE dataset_id = ?
),
old_hashes AS (
  SELECT record_id, content_hash FROM record_hashes WHERE dataset_id = ?
)
SELECT
  n.record_id,
  CASE
    WHEN o.record_id IS NULL THEN 'added'
    WHEN n.content_hash != o.content_hash THEN 'modified'
  END AS change_type
FROM new_hashes n LEFT JOIN old_hashes o USING (record_id)
WHERE o.record_id IS NULL OR n.content_hash != o.content_hash
UNION ALL
SELECT o.record_id, 'removed'
FROM old_hashes o LEFT JOIN new_hashes n USING (record_id)
WHERE n.record_id IS NULL;

Record normalization before hashing strips fields that change on every ingest without indicating a meaningful change (e.g., OFAC's publish_date on the file header, microsecond-precision timestamps that drift due to ingest time). The normalized hash only covers fields that affect screening outcomes: entity name, aliases, sanctions programs, addresses, identification numbers, and remarks.

Event classification

Each detected change maps to one of six event types, ordered by operational urgency:

EVENT_TYPES = {
    # Highest priority — immediate compliance action required
    'entity_sanctioned':    'Entity added to OFAC SDN, Consolidated, or FinCEN list',
    'entity_debarred':      'Entity added to SAM exclusions, State debarment, or UFLPA',

    # High priority — enforcement action filed
    'enforcement_filed':    'New SEC enforcement, FDA warning letter, DOJ press release, CFPB order',

    # Medium priority — existing record updated
    'entity_sanctions_modified': 'Existing OFAC/FinCEN record changed (new program, address, alias)',
    'entity_debarment_modified': 'Existing SAM/debarment record changed (scope, dates, type)',

    # Lower priority — administrative updates
    'enforcement_resolved': 'SEC enforcement closed, CFPB consent order satisfied',
    'record_updated':       'Non-enforcement record update (address, SIC code, etc.)',
}

For OFAC records, distinguishing entity_sanctioned from entity_sanctions_modified uses the record hash delta: if the record_id did not exist in the previous hash index, it is a new sanction; if it existed and the hash changed, it is a modification. The same logic applies to SAM exclusions.

Proposed subscription filters

In the reference design, subscribers would configure alerts using a filter specification stored against an API key. Filters could target specific entities, datasets, event classes, or severity levels:

# Subscription filter specification (JSON)
{
  "filter": {
    # Watch specific entities by your own identifiers
    "entity_ids": ["DUNS:123456789", "LEI:529900...", "CIK:0001234567"],

    # Watch entire lists
    "datasets": ["ofac_sdn", "ofac_consolidated", "sam_exclusions"],

    # Watch by event type severity
    "min_severity": "high",   # high | medium | low

    # Watch by agency
    "agencies": ["ofac", "sam", "sec_edgar", "fda"],

    # Watch by entity name pattern (fuzzy-matched using existing pipeline)
    "entity_name_patterns": ["Acme Corp", "Global Trade Partners"]
  },
  "delivery": {
    "webhook_url": "https://compliance.example.com/regulatory-alerts",
    "hmac_secret": "<subscriber-provided-secret>"
  }
}

The proposed entity_ids filter maps subscriber-provided identifiers through the entity bridge (CIK, LEI, DUNS, UEI, NPI), then watches supported records linked to that identity. Coverage would depend on the identifiers and source routes actually available for the entity.

The entity_name_patterns filter would use the same three-pass fuzzy matching pipeline from the entity matching system: exact normalized match first, then Jaro-Winkler ≥ 0.88, then TF-IDF cosine ≥ 0.72. Matches below the exact threshold include a match_confidence field in the alert payload.

Webhook payload schema

# Webhook POST to subscriber endpoint
{
  "event_type": "entity_sanctioned",
  "idempotency_key": "sha256:f3c8a...",   # hash of (dataset + record_id + change_type + ingest_ts)
  "detected_at": "2026-04-21T14:08:33Z",
  "dataset": "ofac_sdn",
  "agency": "ofac",
  "severity": "high",

  "entity": {
    "entity_id": "OFAC_UID:23456",
    "name": "EXAMPLE TRADING CO LLC",
    "aliases": ["EXAMPLE TRADING", "ETC LLC"],
    "sanctions_programs": ["SDGT", "IRAN"],
    "countries": ["IR", "AE"],
    "entity_type": "entity",
    "match_confidence": 1.0   # 1.0 for entity_id filter, <1.0 for name pattern
  },

  "change": {
    "change_type": "added",   # added | modified | removed
    "changed_fields": null,   # array of changed field names for 'modified', null for 'added'
    "previous_programs": null # previous sanctions programs for 'modified' events
  },

  "source_url": "https://sanctionssearch.ofac.treas.gov/"
}

For modified events, changed_fields lists which specific fields changed: ["aliases", "sanctions_programs"] for an OFAC record that gained a new program. This lets subscribers implement their own logic about whether a modification is material — a new alias is low urgency, a new IRAN program designation is high urgency.

Delivery pipeline

In the reference design, change events would enter a Cloudflare Queue after the diff computation. Each event would be delivered to matched subscribers using at-least-once semantics; idempotency would be enforced on the subscriber side using the idempotency_key.

// Cloudflare Worker: webhook delivery queue consumer
export default {
  async queue(batch: MessageBatch<ChangeEvent>, env: Env): Promise<void> {
    for (const msg of batch.messages) {
      const event = msg.body;
      const subscribers = await matchSubscribers(event, env.DB);

      for (const sub of subscribers) {
        const delivered = await attemptDelivery(sub, event, env);
        if (delivered) {
          msg.ack();
        } else {
          msg.retry({ delaySeconds: backoffSeconds(msg.attempts) });
        }
      }
    }
  }
};

function backoffSeconds(attempt: number): number {
  // 30s, 5min, 20min, then DLQ after 3 retries
  const delays = [30, 300, 1200];
  return delays[Math.min(attempt - 1, delays.length - 1)];
}

The proposed HMAC scheme uses each subscriber's own secret, so a future subscriber could verify payload origin without relying on network location alone:

async function signedDelivery(
  url: string, payload: object, secret: string
): Promise<Response> {
  const body = JSON.stringify(payload);
  const key = await crypto.subtle.importKey(
    'raw', new TextEncoder().encode(secret),
    { name: 'HMAC', hash: 'SHA-256' }, false, ['sign']
  );
  const sig = await crypto.subtle.sign('HMAC', key, new TextEncoder().encode(body));
  const hex = Array.from(new Uint8Array(sig))
    .map(b => b.toString(16).padStart(2, '0')).join('');

  return fetch(url, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'X-Hub-Signature-256': `sha256=${hex}`,
      'X-Delivery-Id': crypto.randomUUID(),
    },
    body,
  });
}

Target detection latency by source

These are design targets, not observed production percentiles. In a deployed service, source polling cadence would dominate detection time; queue pickup, filter evaluation, and the subscriber's webhook response would determine delivery time.

Entity bridge disambiguation

A common scenario: a compliance team monitors a portfolio company under its trade name. When OFAC adds a subsidiary using a slightly different name, the name-pattern filter picks it up with match_confidence 0.81 (below the exact threshold, above the Jaro-Winkler threshold). The alert payload includes:

{
  "event_type": "entity_sanctioned",
  "entity": {
    "name": "ACME TRADING HOLDINGS LTD",      # new OFAC entry
    "match_confidence": 0.81,
    "matched_pattern": "Acme Corp",            # subscriber's watch pattern
    "possible_parent": {                        # entity bridge lookup
      "entity_id": "CIK:0001234567",
      "name": "ACME CORP",
      "relationship": "possible_subsidiary",   # based on SEC Exhibit 21
      "relationship_confidence": 0.72
    }
  }
}

The proposed possible_parent field would be populated by checking whether the newly sanctioned entity appears in the SEC Exhibit 21 subsidiary map for any entity the subscriber is watching. This is not a guaranteed match — it is a signal that warrants human review. The alert always includes the raw OFAC data so the compliance team can make the final determination.

Rate limiting and deduplication

A bulk OFAC update can simultaneously add 50 new entities — for example during a large Russia-linked sanctions package. Without rate limiting, all 50 events would land on the subscriber's webhook endpoint within 60 seconds, which most compliance workflows are not designed to handle.

The design groups bulk updates from one ingest run into batched delivery: changes from the same ingest cycle would be bundled into a webhook POST with an events array up to 25 entries. Changes beyond 25 are queued for subsequent delivery windows. This would reduce webhook volume during large sanctions packages while preserving event content fidelity.

The proposed idempotency key is a SHA-256 hash of dataset + record_id + change_type + ingest_timestamp. If the same ingest event is delivered twice due to queue retry, the subscriber can detect the duplicate by caching received idempotency keys. We recommend a 24-hour cache window; the same record will not generate a second event within 24 hours unless the record changes again.