ICLR 2025: when does a model know it is uncertain?
In this article
A forecast is only as useful as the knowledge of how reliable it is. In a paper for ICLR 2025 we show how a transformer delivers honest prediction intervals without being retrained even once.
Every forecast is uncertain. That is not a flaw; it is in the nature of things: no one knows exactly how much will sell next week. It only becomes a problem when a model is uncertain and does not say so.
This is the question behind our paper “Predictive Inference Is Really Free with In-Context Learning”. It appeared in 2025 at the ICLR workshop “Quantify Uncertainty and Hallucination in Foundation Models”; ICLR is one of the most important conferences for machine learning. It was written by Sohom Mukherjee, Ivane Antonov, Kai Günder and Magnus Maichle.
Two kinds of uncertainty
When people talk about the uncertainty of a forecast, they often mean two different things. The first lies in demand itself: even someone who knows everything about a product cannot predict whether 40 or 60 units will sell next week. This variation is what the forecast band captures, which we wrote about in our article on NeurIPS 2024.
The second uncertainty lies in the model. Has it seen enough for this product to be reliable? Does the pattern fit what it has learned? This question becomes more important the more a model can do. A foundation model that also forecasts products it has never seen should also be able to say when it is struggling.
A proven idea with a catch
Statistics has a proven tool for this question: the jackknife. The idea is simple. You leave out one data point at a time during training and check how well the model predicts exactly that point. The errors of these leave-one-out predictions tell you how wide an honest prediction interval has to be. The jackknife+ variant even comes with a mathematical guarantee, without assumptions about the distribution of the data.
The catch: for every data point left out, the model has to be retrained. With a hundred data points, that is a hundred training runs. For a large neural network, that is practically impossible.
What we showed
This is where in-context learning comes in. A transformer that has learned to learn from context does not need to be retrained to leave out a data point. It is enough to remove it from the prompt. A hundred training runs become a hundred queries to the same model.
Our method is called in-context jackknife+, ICJ+ for short. We show that under mild assumptions it delivers the same kind of distribution-free guarantee as the classic jackknife+. The key is the stability of such transformers: one data point more or less in the prompt changes their prediction only to a limited extent.
In experiments on synthetic data for five classes of functions, the intervals of ICJ+ were as good as or better than those of the classic jackknife+ and the true confidence intervals, measured by coverage and width. The title, by the way, alludes to a well-known paper in statistics: “Predictive Inference Is Free with the Jackknife+-after-Bootstrap”. With us, uncertainty estimation really is free, because it costs no additional training.
Why this matters for inventory planning
For planning, an honest interval is worth more than a seemingly precise number. It helps decide how much safety stock makes sense. If the model is uncertain about a product, the order needs more buffer or a second look.
That second look matters to us. When a model recognizes where it is uncertain, it can direct the planner’s attention to the products where her knowledge makes the difference. Showing uncertainty openly is therefore one of the principles we brought from research into SOLISTIQ.
How these building blocks become a foundation model for entire assortments is the subject of our article on our ECIS 2025 paper. Want to know how reliable the forecasts for your assortment are?
Talk to us.
Further reading
- Mukherjee, S., Antonov, I., Günder, K. & Maichle, M. J. (2025). Predictive Inference Is Really Free with In-Context Learning. ICLR 2025 Workshop “Quantify Uncertainty and Hallucination in Foundation Models”. openreview.net
- Barber, R. F., Candès, E. J., Ramdas, A. & Tibshirani, R. J. (2021). Predictive Inference with the Jackknife+. The Annals of Statistics 49(1), 486–507. arXiv:1905.02928
- Kim, B., Xu, C. & Barber, R. F. (2020). Predictive Inference Is Free with the Jackknife+-after-Bootstrap. NeurIPS 2020. arXiv:2002.09025
- Angelopoulos, A. N. & Bates, S. (2023). Conformal Prediction: A Gentle Introduction. Foundations and Trends in Machine Learning 16(4), 494–591. arXiv:2107.07511
- Müller, S., Hollmann, N., Pineda Arango, S., Grabocka, J. & Hutter, F. (2022). Transformers Can Do Bayesian Inference. ICLR 2022. arXiv:2112.10510