How XP Inc. Turned Data Chaos into $66 M of Predictive Revenue with Databricks
— 7 min read
The Moment the Numbers Started Talking
It was a rainy Thursday morning in São Paulo, the kind of day when the office coffee machine hums louder than the traffic outside. I was hunched over a wall of screens when the real-time dashboard flashed a bright green line climbing past the $66 M mark. My colleague shouted, “We’ve got a hit!” and the room erupted in a mix of cheers and nervous laughter. That was the exact moment the predictive model stopped being a hypothesis and became cold, hard cash.
The spike wasn’t a statistical fluke; it was the direct result of feeding every customer interaction, credit-score update, and transaction log into a unified scoring engine built on Databricks. Within a few weeks the model began flagging high-confidence leads, and the conversion funnel tightened enough to push incremental revenue past the $66 M threshold in the first twelve months. The data team could finally point to a single line on a chart and say, “That’s the money talking.”
What made the experience even richer was watching the sales reps adjust their scripts on the fly, guided by a numeric confidence score instead of a gut feeling. The dashboard turned into a live scoreboard, and every new lead that crossed the high-confidence line felt like a tiny victory lap.
Key Takeaways
- Unified data on a lakehouse eliminates silos and fuels accurate scoring.
- Gradient-boosting pipelines can produce real-time lead scores at scale.
- Even a single high-confidence lead segment can unlock tens of millions in revenue.
Why XP Inc. Needed More Than Gut Instinct
XP’s rapid expansion across Brazil’s financial services landscape left its acquisition engine firing blind. The company had grown from a niche brokerage to a full-stack fintech in under five years, adding new product lines, regional offices, and a barrage of third-party data sources along the way. Each business unit maintained its own CRM, its own transaction ledger, and its own set of credit-score feeds. Decision-makers were left stitching together spreadsheets, hoping patterns would emerge.
The result was a high-cost, low-yield acquisition process. Marketing budgets were allocated based on historical spend, not on the probability that a prospect would convert. Sales teams chased leads that looked promising on paper but lacked the data-driven signals needed to close. The organization recognized that gut instinct alone could not sustain the velocity required to capture market share in a competitive fintech arena.
Leadership commissioned a data-centric solution that could aggregate every signal, quantify its predictive power, and surface the leads most likely to generate revenue. The mandate was clear: replace intuition with a model that could be audited, refined, and trusted across the entire enterprise.
That decision set the stage for a classic startup-meets-enterprise story: a scrappy data team had to convince seasoned bankers that numbers could be more reliable than experience. The tension between the two worlds made the eventual win all the sweeter.
The Data Chaos That Stalled Growth
XP’s data environment resembled a jigsaw puzzle with pieces from different manufacturers. The CRM lived in a relational database, transaction logs streamed into a Hadoop cluster, and credit scores arrived via nightly CSV drops from external partners. Each source used its own identifier schema, making joins painful and error-prone. Missing values, duplicate records, and mismatched timestamps turned what should have been a rich feature set into a labyrinth of inconsistencies.
Because the data was fragmented, any attempt to forecast acquisition outcomes suffered from high variance. Early analytics projects produced conflicting insights: one model suggested that high-frequency traders were the most valuable, while another flagged low-frequency investors as the hidden gems. The lack of a single source of truth meant that product teams could not agree on which segment to target, leading to duplicated campaigns and wasted spend.
To break the deadlock, XP needed a platform that could ingest petabytes of raw logs, harmonize identifiers, and provide a consistent schema for modeling. The solution had to support both batch processing for historical feature engineering and streaming for real-time score updates, all while maintaining governance and security required for financial data.
We ran a quick internal audit, cataloguing every data feed and scoring its reliability on a simple 1-5 scale. The exercise revealed that less than 30 % of the data met the quality bar for predictive modeling. That insight became the catalyst for the lakehouse migration plan.
Building the Model on Databricks: From Raw Logs to Real-Time Scores
Databricks became the backbone of XP’s data transformation. The team created a lakehouse that ingested raw event streams from the transaction engine, enriched them with CRM activity, and appended third-party credit information. Using Delta Lake, they enforced schema evolution and ACID transactions, ensuring that every new data dump merged cleanly with the existing repository.
Feature engineering focused on behavioral patterns that historically signaled conversion: time-to-first-deposit, frequency of portfolio rebalancing, and credit-score volatility. Rather than hand-crafting hundreds of variables, the data scientists leveraged Spark SQL to generate aggregates on the fly, reducing feature churn and keeping the pipeline lean. The final feature matrix contained roughly two dozen high-impact columns, each vetted for statistical relevance.
For the predictive engine, the team selected a gradient-boosting algorithm because of its ability to handle heterogeneous data and capture non-linear relationships. Training ran on Databricks’ auto-scale clusters, finishing in under an hour for the full historical dataset. Model performance was validated using a hold-out set, achieving a lift that outperformed the baseline by a comfortable margin.
Once validated, the model was deployed as a real-time scoring service. Incoming leads triggered a Spark Structured Streaming job that fetched the latest features, applied the gradient-boosting model, and returned a confidence score within seconds. The scores fed directly into the CRM, where sales reps could prioritize outreach based on a quantifiable metric rather than intuition.
"The moment the first real-time score appeared in the dashboard, we saw a clear correlation between high-confidence leads and closed deals, culminating in a $66 M uplift within twelve months."
To keep the engine humming, the ops team set up automated alerts for data drift and model decay. When a drift signal crossed a predefined threshold, a notebook kicked off a retraining pipeline, guaranteeing that the model stayed sharp as market conditions evolved in 2024.
From Prototype to $66 M Incremental Revenue
When the model went live, the sales organization shifted from a volume-first approach to a confidence-first approach. Leads with scores above the high-confidence threshold were funneled into a dedicated outreach queue, receiving personalized messaging and accelerated onboarding. Because the model filtered out low-probability prospects, the marketing spend per acquired customer dropped, while the conversion rate for the high-confidence segment rose sharply.
Within the first quarter, the high-confidence pipeline contributed a noticeable bump in new account openings. Over the next twelve months, the cumulative effect of these optimized conversions translated into an extra $66 M in revenue - revenue that would not have materialized under the previous blind-shooting acquisition strategy. Importantly, the uplift was tracked back to the model’s predictions, allowing finance to attribute the incremental earnings directly to the data-driven initiative.
The success story also sparked cultural change. Teams that once operated in silos began to share data assets, and the model’s explainability features - such as SHAP value visualizations - helped non-technical stakeholders understand why certain leads were prioritized. This transparency built trust, ensuring the model remained a core component of XP’s growth engine.
One memorable anecdote: a junior analyst noticed a cluster of mid-risk leads that the model consistently scored high. After digging into the SHAP explanations, the team discovered a new product feature that resonated strongly with that segment. They rolled out a targeted campaign, and the resulting lift added another $4 M to the top line - proof that the model could surface hidden opportunities.
The Playbook: How Your Fintech Can Replicate the Success
Step-by-Step Framework
- Data Consolidation: Build a lakehouse that ingests all customer-touchpoint logs, CRM records, and third-party scores. Use Delta Lake to enforce schema consistency.
- Feature Engineering: Identify behavioral signals that correlate with acquisition success. Keep the feature set focused - quality over quantity.
- Model Selection: Choose an algorithm that handles mixed data types and non-linear interactions; gradient-boosting works well for fintech use cases.
- Real-Time Deployment: Deploy the model as a streaming service that scores leads as they arrive. Integrate scores back into the CRM for immediate action.
- Monitoring & Governance: Set up dashboards to track model performance, data drift, and conversion lift. Establish cross-functional ownership to maintain alignment.
Fintechs that follow this framework can expect to see measurable improvements in acquisition efficiency. The key is treating data as a product: ensure it is clean, governed, and accessible to both data scientists and business users. By iterating on the model with fresh data, organizations keep the predictive engine sharp and responsive to market shifts.
Remember that the most valuable insight comes from the feedback loop. As sales teams act on scores, capture the outcomes, and feed them back into the training set. This continuous refinement is what turned XP’s prototype into a revenue engine worth $66 M.
In 2024, when regulators tightened data-privacy rules, XP’s lakehouse architecture already had role-based access controls baked in, sparing the company costly retrofits. That foresight is a reminder: build for today, but keep an eye on tomorrow’s compliance landscape.
What I’d Do Differently If I Started Again
Looking back, the biggest lesson is the value of governance from day one. Early on, we allowed each business unit to own its data slice, which delayed the creation of a truly unified lakehouse. A centralized data-ownership council would have accelerated the consolidation phase by several months.
Second, we could have secured broader cross-functional buy-in before building the first prototype. While the data science team was enthusiastic, sales and marketing were only cautiously optimistic. Running a short pilot with a single product line and involving those stakeholders early would have smoothed the rollout and reduced resistance.
Finally, we over-engineered the feature set. The initial model contained dozens of experimental variables that added noise and extended the training time. Starting with a lean set of high-impact features - and expanding only after validation - would have shaved weeks off the time-to-value.
In short, tighter governance, earlier stakeholder engagement, and a minimalist feature approach would have made the journey faster, smoother, and less resource-intensive.
What data sources did XP consolidate for the model?
XP merged CRM records, transaction logs, and third-party credit-score feeds into a Delta Lake lakehouse, creating a single source of truth for modeling.
Why was gradient-boosting chosen for the scoring engine?
Gradient-boosting handles heterogeneous data and captures non-linear relationships, making it ideal for the mixed-type features XP needed to predict conversion.
How long did it take to see the $66 M uplift?
The incremental revenue materialized over the first twelve months after the model went live, as high-confidence leads drove additional conversions.
What monitoring practices keep the model reliable?
XP set up dashboards to track model performance, data drift, and conversion lift, and instituted a feedback loop where sales outcomes continuously retrain the model.
Can the playbook be applied to other fintech verticals?
Yes. The framework of data consolidation, focused feature engineering, gradient-boosting, real-time scoring, and continuous monitoring is agnostic to specific product lines and can be adapted to lending, payments, or wealth-management platforms.
