Sales teams do not lose leads because nobody is trying. They lose them in the gaps between tools: a Meta form that has to be exported by hand, a WhatsApp enquiry buried in support chat, a landing page that posts to an inbox nobody checks, a provider that swears it sent the lead and nothing to prove otherwise.
The Lead Parsing System closes those gaps with one idea: every source is a connector that only knows how to verify and parse its provider; everything after that is shared. Codevibe built it first inside a holiday travel brand’s CRM, then ported the same module into a B2B group-booking platform for a second client, with different business rules on identical plumbing.
No two sources agree on anything.
What a lead looks like before it is a lead. Each provider has its own payload, its own way of proving who sent the request, and its own retry habits. Any system that takes them all has to absorb those differences somewhere.
“Only the verify and parse steps are connector-specific. Everything else, from normalising a phone number to retrying the CRM, is written once.”
One lead, start to finish.
Follow one enquiry from the moment a customer sends it to the moment an agent sees it. Five stops, one glowing lead, and a log that writes down every request, accepted or refused.
The story plays on its own. Pause, step through, or click a card.
- 01 · Provider → APIOne webhook per connection
Every source gets its own URL with its own secret. The provider posts to it; nothing else in the system knows or cares what that provider’s payload looks like. Before anything is checked, a delivery-log row is opened for the request.
REJECTED_SECRET - 02 · APIPaused connections acknowledge and drop
An admin can pause a source from the panel. A paused connection still answers 200, so the provider stops retrying, but nothing is stored. This is deliberate: a 4xx here would make Meta hammer the endpoint for days.
CONNECTION_DISABLEDCONNECTOR_MISSING - 03 · ConnectorThe connector proves who sent it
Meta and the WhatsApp Cloud API sign every request; the connector checks the signature over the exact bytes received. Forms can be signed too. A GET from Meta is a subscription handshake and is answered with its challenge.
REJECTED_SIGNATUREHANDSHAKE - 04 · ConnectorThe connector turns the payload into leads
One delivery can hold zero, one or many leads. For Meta, the webhook is only a notification: the connector fetches the actual answers from the Graph API with the page token, then maps the form’s questions onto our fields. Anything unmapped is kept in the message as “question: answer” lines, so nothing is lost.
INVALID_JSONERROR - 05 · PipelineNormalise to one shape
Phone numbers become digits with the country code, so a WhatsApp number and a form entry for the same person look identical. Emails are lowercased, dates become ISO, names are tidied. Campaign attribution is read from wherever the provider put it.
- 06 · PipelineGuard against noise and replays
A lead with no phone and no email cannot be contacted, so it is dropped and the reason is logged. A provider resending the same lead is recognised by its id and stored once. In the holiday CRM a repeat phone or email within 30 days is held as a duplicate rather than sent on; the group-booking platform treats every enquiry as a new trip.
NO_LEADDUPLICATE_ONLY - 07 · PostgreSQLStore with provenance
The lead is written with everything known about it: contact, trip intent, 31 attribution fields, the connection it came from, the provider’s timestamp and ours, and the untouched raw payload for debugging or replay.
ACCEPTED - 08 · ForwarderHand off, and keep trying
The moment a lead is stored, the forwarder hands it to the CRM in the same process. If the CRM is busy or down, the lead stays put with a retry time and a job sweeps every minute. Only a lead the CRM rejects as invalid is parked for a person to look at.
- 09 · CRMA lead in the inbox, exactly once
The CRM creates its own lead, keyed on the LPS lead id, so even a retry that races the first attempt cannot create twins. From here the business rules of that CRM take over: a round-robin to a destination expert in the holiday CRM, an unassigned inbox lead in the group-booking panel.
Retries that cannot create twins.
Handing a lead to the CRM is where lead systems usually leak: the CRM is down and the lead is lost, or a retry sends it twice and two agents call the same customer. Pick a scenario and watch what this system does instead.
The CRM is restarting or its database is busy. The lead must not be lost.
The provider retries for a while, then gives up. The lead sits inside Meta or the form tool until someone notices and exports it by hand.
Stored first, retried on a doubling schedule that never passes an hour. After ten tries it is parked for a person, never dropped.
A provider sending the same lead again is recognised by its id and skipped, even if two copies arrive at the same moment.
unique (connectionId, externalId)Only one send per lead at a time. A lead that is being sent cannot also be picked up by the minute sweep.
in-flight set · sweep batch 50The inbox lead is keyed on the LPS lead id. A second attempt gets the same lead back, never a new one.
idempotency row, same transaction, 180 daysSecrets that stay secret.
A connection holds real provider credentials: a Meta app secret, a page access token, a verify token. They are treated as the most sensitive thing in the system.
Encrypted at rest, in a format that explains itself
Each connector names its secret fields. A decorator around the store seals those fields on save with AES-256-GCM under a key that belongs to the LPS alone. The ciphertext carries its own version and IV, so encryption can be switched on for a running system: older plain values are still read, and sealed the next time the connection is saved.
Masked everywhere it is shown
Every API response masks a secret to its last four characters. When the settings form is saved with the mask still in place, the stored value is kept; only a new value replaces it. The webhook secret itself is 24 random bytes, shown once in the URL because the URL is what gets pasted into the provider, and replaced in one click when a URL leaks.
- Delivery log records signature headers as present or absent, never their value
- Query strings logged with secret, verify token and access token redacted
- Raw payloads visible only with the connections permission
Permissions per action, not per screen
Seeing the queue, acting on a lead and managing connections are three different grants. Admins hold all three by default; a sales manager can retry a stuck lead without ever seeing a webhook URL.
leadparsing:read | See the queue and the breakdowns |
leadparsing:manage | Retry a failed hand-off, send a duplicate on purpose |
leadparsing:connections | Connections, the delivery log, raw payloads |
Keys from the group-booking platform. The holiday-CRM copy uses ops.lead_parsing.read and .write, with connections and deliveries limited to the admin role.
Same plumbing. Different rules.
The strongest proof that the design holds is that it was ported. Connectors, pipeline, forwarder, secrets and delivery log are the same code in both products; what each business wants to happen to a lead lives in its intake adapter, not in the connectors.
Round-robin, working hours and the online rule are the holiday CRM’s. The group-booking platform’s documentation says it plainly: “There is no round-robin and no working-hours window.”
Five weeks, two platforms.
The dates below are commit and merge dates from the two repositories. The work ran on a feature branch and reached the holiday CRM’s main branch through 37 pull requests between 21 September and 11 October 2026; the port into the second platform was a single reviewed pull request. Tap a milestone to expand it.
A standalone app: four connectors, the verify → parse → normalise → dedupe → store pipeline, an event bus and a JSON store. 30 files, 2,012 lines.
PostgreSQL store with its migration, the CRM forwarder with retry and parking, secrets sealed with AES-256-GCM, the first unit tests, the admin screens, and round-robin assignment to destination experts.
Last-touch marketing attribution captured per lead, and every webhook request recorded with its outcome, redacted query and raw body.
The separate app, its URL and its tokens went away. The LPS became a module inside the CRM’s NestJS API; the forwarder calls the intake in process.
Departure-city matching with nearest-airport lookup, parsing of free-text form answers, destination and channel summaries, and redistribution of a departed agent’s leads through the same round-robin.
A CRM-form connector mirrors leads created on the CRM form, keyed on the CRM id, so one summary counts every lead whatever the entry point.
The module dropped into the group-booking platform in one pull request: 68 files, 9,389 lines, seven test files, three new tables and an intake adapter for that platform’s inbox.
The enquiry wizard, call-back and newsletter forms register with the API and flow through the website-form connection, with a direct fallback so a form is never lost.
A DoubleTick Bot Studio intake endpoint enriches a recent open lead or relays the conversation into the pipeline, where it gets the same round-robin and welcome.
The group-booking copy drops the 30-day duplicate window: for that business a repeat contact is a new trip. The holiday-CRM copy keeps it. Same engine, each platform’s own rule.
What each technology is for.
No logo wall. Each entry says what the technology does in this system and why it earned its place.
- NestJS 11
- The LPS is a NestJS module: a public webhook controller, a permission-guarded admin controller, a service that owns the store and the forwarder, and two cron jobs on @nestjs/schedule.
- PostgreSQL + Prisma
- Three tables: connections, leads and deliveries. Unique indexes do the replay and twin protection; a SELECT … FOR UPDATE on the round-robin cursor serialises assignment.
- Zod
- Each connector declares its config as a Zod schema. The API publishes it as JSON Schema and the panel renders the form from it, so a new connector needs no UI code. The intake payload contract is Zod too.
- node:crypto
- HMAC-SHA256 with a timing-safe compare for provider signatures; AES-256-GCM with a random IV for credentials at rest.
- React panels
- Inbound leads, deliveries and connections screens on TanStack Query, with a schema-driven connection dialog and a lead drawer that links each lead to the request that produced it.
- Vitest and Jest
- 96 unit tests in the group-booking copy (Vitest) and 150 in the holiday-CRM copy (Jest, Vitest for the admin), over normalisation, every connector, the pipeline, the forwarder, secrets, the webhook and the intake.
What is demonstrated, and what it is for.
Two lists, kept apart on purpose: what the code does today, and what that is designed to make possible. The second has not been measured yet, so it is not claimed as a result.
- 01
Four sources through one door
Website forms, Meta lead ads, WhatsApp (Cloud API or DoubleTick) and landing pages post to one webhook route and come out as one lead shape.
- 02
Every request is on record
Ten distinct outcomes, each with the redacted query, safe headers and the raw body. The first place to look when a provider says it sent a lead.
- 03
Retries without twins
In-process hand-off, exponential backoff to an hour, ten attempts, parking on a 400, and idempotency on the lead id at the CRM side.
- 04
Credentials sealed
App secrets, page tokens and verify tokens encrypted at rest in a self-describing format, masked in every response, and recognised when a form echoes the mask back.
- 05
Ported once, proven
The same module runs inside two unrelated products with different assignment, duplicate and notification rules.
- Leads from paid campaigns reach a salesperson without anyone exporting a CSV from Meta.
- A WhatsApp enquiry sent at night is greeted immediately and assigned when the team is in.
- Marketing can see paid versus organic by destination and platform from the same data the sales team works.
- Adding a new lead source is one connector file, not a project.
Every count, constant and date on this page was read from the two codebases and their commit history in October 2026. Where the copies differ, the page says which is which. Both clients are described by sector rather than named. No business metrics are claimed, because none have been measured yet.
Questions about this system.
A lead-ingestion module Codevibe built for travel CRMs. Each lead source, such as a website form, a Meta lead-ad form, a WhatsApp number or a campaign landing page, gets its own webhook URL. The module verifies each request, turns the provider’s payload into one lead shape, stores it with its attribution and raw payload, and hands it to the CRM with retries. Every request a provider sends is logged with its outcome.
Four provider connectors: website forms, landing pages, Meta lead ads (Facebook and Instagram) and WhatsApp through either the Meta Cloud API or DoubleTick. The holiday-CRM copy adds a connector that mirrors leads typed into the CRM by hand and an intake for DoubleTick Bot Studio conversations. A new source is one connector file plus one line in the registry; the panel form for its settings is generated from the connector’s schema.
Three layers. A provider resending the same lead is recognised by its id and stored once, backed by a unique index. The hand-off to the CRM runs in the same process with exponential backoff, ten attempts and a sweep every minute, and only a lead the CRM rejects as invalid is parked for a person. The CRM creates its lead idempotently on the LPS lead id, so a retry that races the first attempt cannot create twins.
Yes. That is how the second installation happened: the module was ported into the group-booking platform in one pull request, with a small intake adapter that creates that platform’s inbox lead. The connectors, pipeline, forwarder, secrets handling and delivery log came across unchanged; the business rules, such as what counts as a duplicate and who gets the lead, are set per platform.
Each connector declares which of its settings are secrets. Those are encrypted at rest with AES-256-GCM under a key that belongs to the LPS alone, in a self-describing format so encryption can be switched on without downtime. Every response masks them, and a settings form that echoes the mask keeps the stored value. Webhook URLs carry a per-connection secret that can be rotated in one click, and the delivery log records whether a signature header was present, never its value.
The repositories record the first commit on 11 September 2026, the module running inside the holiday CRM’s API by 22 September, and the port into a second platform merged on 2 October 2026. The dates on this page are commit and merge dates. No separate delivery deadline for this module is recorded in either repository, so none is claimed.
Yes. The engine is not travel-specific: a connector understands one provider, and an intake adapter creates whatever your CRM calls a lead. Start with the form below and we will come back within one business day.
Leads arriving from more places than you can watch?
One webhook per source, one shape for every lead, retries that cannot create twins, and a log of every request. We have built it twice; the third time starts with a conversation.