A new confidence estimation method, XConf, is proposed for language models, which uses the model's accumulated experience to improve confidence estimation. XConf stores the model's past episodes, retrieves similar episodes for new tasks, and uses historical success rates to inform confidence. This approach outperforms existing methods on 23 of 24 comparisons and can improve delivered success rate by up to 8.7 points on agent tasks.