ExpertCrowd and HUD Collaborate to Advance AI Training and Evaluation, Release SWE Benchmark

ExpertCrowd-SWE represents a new effort to assess the capabilities of advanced AI in software engineering. The benchmark, co-developed by ExpertCrowd and HUD, is designed for what are termed "frontier-grade LLMs and coding agents," indicating a focus on the most sophisticated AI systems currently available for software development.

The stated purpose behind ExpertCrowd-SWE is to address the increasing complexity of AI systems and their growing role in professional work. This suggests a move away from simpler, academic evaluations towards assessments that mirror the practical challenges and performance expectations found in professional software engineering environments.

The collaboration combines ExpertCrowd's experience in "expert-led task development" with HUD's "infrastructure for running complex RL environments." This approach implies that the benchmark tasks are crafted by human software engineering experts and executed within simulated, challenging conditions to rigorously test AI performance.

While the new benchmark aims to provide a more practical measure of AI's coding abilities, the initial announcement does not detail the specific types of tasks included in ExpertCrowd-SWE, the exact methodology used for developing these expert-led tasks, or any initial performance results from the LLMs or coding agents evaluated against it.