Move fast and break people? Ethics, companion apps, and the case of Character.ai

Riffing off move fast and break things, the internal motto coined by Meta’s Mark Zuckerberg, this paper examines the ethical dimensions of human relationships with AI companions, focusing on Character.ai—a platform where users interact with AI-generated ‘characters’ ranging from fictional figures to representations of real people. Drawing on an assessment of the platform’s design, and the first civil lawsuit brought against Character.ai in the USA in 2024 following the suicide of a teenage user, this paper identifies unresolved ethical issues in companion-based AI technologies. These include risks from difficulty in separating AI-based roleplay from real life, unconstrained AI models performing edgy characters, reality detachment, and confusion by dishonest anthropomorphism and emulated empathy. All have implications for safety measures for vulnerable users. While acknowledging the potential benefits of AI companions, this paper argues for the urgent need for ethical frameworks that balance innovation with user safety. By proposing actionable recommendations for design and governance, the paper aims to guide industry, policymakers, and scholars in fostering safer and more responsible AI companion platforms.