突破 0.669 跻身 Kaggle“Name That LLM”前 5 名：诱饵陷阱、双 SVC 及序列编码器

The "Name That LLM" competition tasked participants with identifying which of 18 generative language models, including those from Anthropic, Google, Zhipu AI, OpenAI, Moonshot AI, and Alibaba, authored short text passages. The competition's Balanced Accuracy Score metric was complicated by unseen prompts in the test set, a condition that could lead standard NLP classifiers to overfit by memorizing topic semantics rather than true model characteristics.

Competitors discovered that some top public notebooks contained subtly de-tuned hyperparameters, a "decoy camouflage" designed to prevent simple replication from surpassing original authors. Further, a significant structural difference existed between the training and hidden test sets: training data provided ~150 prompt groups with 18 passages each, while the hidden test set contained only 1 to 3 passages per prompt. This meant traditional cross-validation, which evaluated full 18-passage groups, inflated performance estimates by +8.5% but led to a -0.015 drop on the public Leaderboard.

To address these issues, the team found that evaluating out-of-fold on randomly sampled two passages per prompt group yielded a high Pearson correlation (r=0.91) with the live Kaggle Leaderboard. Their solution integrated a fine-tuned DeBERTa-v3 sequence classification head, adjusted balancing strategies to prevent model collapse on rare classes, and applied hard masking for specific "telltale tokens." These tokens represent patterns mathematically impossible for certain model providers, ensuring zero false assignments in such cases.