Kaggle Competition Winner Reveals Stacking Strategy with cuML

Kaggle Grandmaster Chris Deotte has unveiled the secrets behind his first-place victory in the April 2025 Kaggle competition. The challenge required participants to predict podcast listening times, and Deotte’s innovative approach centered on stacking models using NVIDIA’s cuML, a GPU-accelerated machine learning library, according to NVIDIA’s developer blog.

Understanding Stacking

Stacking is a sophisticated technique that combines predictions from multiple models to improve performance. Deotte’s strategy involved creating a three-level stack, starting with Level 1 models such as gradient boosted decision trees (GBDT), deep learning neural networks (NN), and other machine learning models like support vector regression (SVR) and k-nearest neighbors (KNN). These models were trained using GPU acceleration to enhance speed and efficiency.

Level 2 models were then trained using the outputs of Level 1 models, learning to predict targets based on different scenarios. Finally, Level 3 models averaged the outputs of Level 2 models, culminating in a robust predictive model.

Diverse Predictive Approaches

In the competition, Deotte explored various predictive approaches, including predicting the target directly, predicting the ratio of the target to episode length, predicting residuals from linear relationships, and predicting missing features. By employing diverse models with different architectures and hyperparameters, Deotte was able to identify the most effective strategies for the competition’s unique challenges.

Building the Stack

After developing hundreds of diverse models, Deotte constructed the final stack using forward feature selection. Level 1 model outputs, known as out-of-fold (OOF) predictions, were used as features for Level 2 models. Additional features, including engineered features like model confidence and average prediction, were also incorporated.

Multiple Level 2 models were trained, including GBDT and NN models, and a weighted average of their predictions formed the final Level 3 output. This advanced stacking technique achieved a cross-validation RMSE of 11.54 and a private leaderboard RMSE of 11.44, securing first place in the competition.

Conclusion

Deotte’s success demonstrates the power of GPU-accelerated machine learning with cuML. By rapidly experimenting with diverse models, he was able to develop an advanced solution that stood out in the competitive field. For more insights into his strategy, visit the NVIDIA developer blog.

Image source: Shutterstock