Building Sub-5ms Ingestion Middleware for ServiceTitan and Jobber: Architecture & Deduplication Deep Dive

When you're handling inbound leads for high-end home services, you can't afford generic middleware boilerplate. If a lead hits your pipeline with a messed-up phone number or a duplicate webhook retry from a frantic homeowner spamming a contact button, your CRM gets polluted and dispatch gets a bad number.

I built our ingestion core to run sub-5ms in transient memory, stripping out the fluff before anything touches ServiceTitan or Jobber. Here is how the core loop handles idempotency, schema flattening, and deduplication.

Dual-Hash Idempotency Without Disk Bottlenecks

Hitting a database on every inbound webhook to check for duplicates adds unacceptable latency. Instead, we use an in-memory sliding-window ledger combined with a dual-hash fingerprint generator (DJB2 + FNV-1a):

JavaScript

const generateFingerprint = (str) => {
if (!str || !String(str).replace(/|/g, '').trim()) return null;
const s = String(str);
let djb2 = 5381;
let fnv = 2166136261;
for (let i = 0; i < s.length; i++) {
const char = s.charCodeAt(i);
djb2 = ((djb2 << 5) + djb2) + char;
fnv ^= char;
fnv = Math.imul(fnv, 16777619);
}
return (djb2 >>> 0).toString(16).padStart(8, '0') + '-' + (fnv >>> 0).toString(16).padStart(8, '0');
};

This gives us two layers of protection:

In-Batch Deduplication (seenInBatch): Catches identical payloads arriving simultaneously in the same execution burst.

Sliding TTL Ledger: Tracks recent events with strict time-to-live expiration (86400000ms for events, 3600000ms for contacts) while automatically pruning memory overflow so the heap doesn't bloat.

Flattening Ingress Chaos

Every form builder and ad network sends data differently. Google Lead Forms use user_column_data, Meta Lead Ads use field_data, and custom webhook forms dump nested objects into body or payload.

We flatten all of those disparate schemas into a single normalized payload before mapping them directly into clean, CRM-ready structures:
JavaScript

const allAdapters = {
servicetitan_v2: {
name: fullFormattedName,
type: isCommercialLead ? 'Commercial' : 'Residential',
address: {
street: street || rawAddress,
unit: unit || undefined,
city: city || undefined,
state: state || undefined,
zip: zip || undefined,
country: resolvedCountryIso3
},
contacts: [
...(cleanPhone ? [{ type: 'MobilePhone', value: cleanPhone, memo: phoneExtension ? 'ext ' + phoneExtension : undefined }] : []),
...(hasValidEmail ? [{ type: 'Email', value: cleanEmail }] : [])
]
},
jobber_graphql: {
firstName: firstName,
lastName: lastName,
companyName: resolvedCompany || undefined,
isCompany: Boolean(entityType === 'CORPORATE_OR_SYSTEM' || (isCommercialLead && resolvedCompany)),
phones: cleanPhone ? [{ number: cleanPhone, description: 'MAIN', primary: true }] : [],
emails: hasValidEmail ? [{ address: cleanEmail, description: 'MAIN', primary: true }] : [],
billingAddress: street ? {
street1: street,
street2: unit || undefined,
city: city || undefined,
province: state || undefined,
postalCode: zip || undefined,
country: resolvedCountryIso2
} : undefined
}
};

Edge Validation & DLQ Quarantine

If a lead has a bogus phone number (like 555-555-5555), profanity in the name field, or invalid email syntax, it never makes it to the CRM.

Instead, the pipeline flags the payload and routes it straight to a Dead-Letter Queue (DLQ) quarantine envelope, complete with automatic PII redaction (REDACT_DLQ_PII) so logs remain safe and inspectable.

Check out the full repository and documentation here:
KallowayThayer/KT-pipeline-core on GitHub

Operated strictly as an asynchronous engineering consultancy. No sales calls. kallowaythayer.com