Anthropic to users: be nice to Claude AI — new policy bars ‘abusive or cruel behavior’ to tech

Anthropic's new policy on user behavior towards its Claude AI introduces a specific prohibition against "abusive or cruel behavior," a clarification that stands out among other updates regarding election interference, surveillance, and weapon development. While the company states this applies only to "extreme cases" of repeated cruelty without purpose, it reflects an unusual dimension in how AI models are being governed.

This policy update surfaces against a backdrop of ongoing discussions about AI consciousness, with Anthropic co-founder Christopher Olah reportedly engaging with religious leaders on the topic. Although the policy document itself avoids explicit mention of AI consciousness, the company's internal "constitution" for Claude, reportedly based on a "soul doc," indicates an internal grappling with the model's "moral status, welfare, and consciousness."

The varied online reactions to Anthropic's new rule highlight a tension between practical AI usage and speculative concerns about AI sentience. While some users question the moral consideration given to a "computer program that is nothing but matrices/vectors/tokens," others criticize the company's approach, linking it to broader controversies around its valuation and affiliations. This division underscores the challenges in establishing social norms for human-AI interaction when the nature of AI itself remains a subject of intense debate and uncertainty.