A study reveals a primacy bias in multimodal retrieval-augmented question answering systems, where the first passage retrieved is preferred over subsequent ones, affecting the accuracy of deployed systems. This has implications for the evaluation of knowledge-based visual question answering systems and suggests that recall@k is not a suitable metric for deployed KB-VQA systems.