A new zeroth-order paradigm for aligning large language models (LLMs) with human preferences, called Comparison-based Preference Optimization (ComPO), is proposed. ComPO extracts directional information from preference pairs without directly optimizing a differentiable preference loss. Experiments demonstrate improvements over existing direct alignment methods.