01 · Background
The platform and the workload
An enterprise procurement intelligence platform helps buyers and suppliers analyze purchasing behavior, market opportunities, and historical volume trends using Databricks.
One of its core capabilities analyzes how purchasing volume evolved over the previous 24 months. Suppliers use those results to understand purchasing trends, identify declining volume, recognize commercial opportunities, and evaluate customer behavior over time.
02 · Business Challenge
A core analytics pipeline had become unusable
By the time the investigation began, the Databricks job had already been running for several days. Execution had to be stopped after approximately USD 3,000 in compute resources had been consumed, with no realistic indication that the workload would finish successfully.
The issue was not simply a long-running background job. The pipeline supported a customer-facing analytical capability, so its inability to complete prevented the platform from delivering timely purchasing-volume insights.
03 · Technical Investigation
Tracing how the workload was actually executing
The investigation focused on the Spark UI: jobs, stages, task distribution, and the execution timeline. The objective was to understand why Spark was not distributing the work efficiently—not merely to identify which operation appeared slow.
That execution-level view made it possible to separate symptoms from the architectural cause. The bottleneck was rooted in how work was orchestrated before Spark had the opportunity to optimize it.
04 · Root Cause
Distributed compute constrained by centralized orchestration
The notebook processed each month individually across the previous 24 months. A Python for-loop coordinated the work through a ThreadPoolExecutor, effectively centralizing orchestration instead of allowing Spark to manage distributed execution.
This execution model produced severe data skew and limited Spark's ability to optimize task distribution. Adding compute would not address that underlying constraint: the architecture itself prevented the cluster from being used efficiently.
05 · Architecture Comparison
Execution Model Evolution
Before
- 24-Month Workload
- Python for-loop
- ThreadPoolExecutor
- Driver-managed orchestration
- Data Skew
- 126+ hour runtime
After
- 24-Month Workload
- Spark Dynamic Partition Overwrite
- Adaptive Query Execution
- Distributed Spark execution
- Balanced task distribution
- Under 30-minute runtime
The largest improvement came from redesigning the execution model rather than increasing compute resources.
06 · Optimization Strategy
Redesign the execution model
The solution was a structural redesign, not an attempt to tune the existing month-by-month implementation.
- Removed ThreadPoolExecutor
- Eliminated month-by-month execution
- Replaced manual replaceWhere processing
- Implemented Spark Dynamic Partition Overwrite
- Leveraged the existing partition strategy
- Enabled Adaptive Query Execution
- Allowed Spark to determine task distribution dynamically
07 · Results
Measured production improvement
| Measure | Before | After |
|---|---|---|
| Runtime | More than 126 hours | Under 30 minutes |
| Execution model | Driver-managed orchestration | Spark-native distributed execution |
| Data skew | Severe | Resolved |
Business Validation
After implementation, business stakeholders validated the generated results. The optimized pipeline achieved 100% agreement with business expectations. Runtime improvement without preserved business correctness would have had no value.
08 · My Role
Databricks technical lead
I served as the Databricks technical lead for the platform. My responsibilities included:
- Investigating production bottlenecks
- Defining the optimization strategy
- Redesigning the execution model
- Validating Spark execution
- Leading Databricks performance best practices
09 · Engineering Takeaways
What this project reinforced
Performance problems are often architecture problems rather than hardware problems.
Spark should orchestrate distributed execution whenever possible.
Understanding the Spark UI is often more valuable than increasing cluster size.
Correctness must always be validated with the business after optimization.