DiaLong is a new benchmarking dataset and memory task designed to evaluate the ability of Large Language Models (LLMs) to retain and recall memories in long conversational contexts. The dataset consists of 100 human-created conversations augmented with LLMs, making it longer than existing datasets. The memory task involves classifying 40 candidate facts about a dialogue as true or false, directly testing the model's ability to actively retain and retrieve information. Testing on DiaLong shows that current LLMs lack sufficient memory performance, indicating a need for improvement to enable LLMs to maintain coherence, relevance, and human-like interaction in extended conversations. This improvement is crucial for fulfilling user needs and expectations in various conversational scenarios.