Probing Llama 3 and Qwen 2.5 for Psychometric Structure

Summary:This study evaluates 14 Llama 3 and Qwen 2.5 model variants using psychometric responses from 272 Taiwanese university students. Larger models reproduce and amplify the cross-scale correlations observed in humans, while geometric alignment with the human correlation matrix appears mainly in larger instruction-tuned models rather than base models. Activation probes, however, show systematically stronger slopes than behavioral responses. A synthetic control indicates that ridge probing can inflat

---

Research Question and Scope of the Claims

The study does not establish that large language models "understand" personality or psychological states. Instead, it separates three commonly conflated questions: whether models reproduce human cross-scale psychometric structure at the behavioral level, whether their activation spaces contain geometry aligned with that structure, and whether those internal directions actually mediate cross-persona predictions.

The experiments cover 14 variants from the Llama 3 and Qwen 2.5 families, ranging from 0.5B to 14B parameters and including both base and instruction-tuned models. Behavioral data come from 272 Taiwanese university students. Given responses on one psychological scale, the models predict responses on six others, allowing their cross-scale correlation matrix to be compared with the human matrix.

The findings have three distinct levels:

1. Larger models reproduce and amplify human cross-scale correlations in their outputs;
2. Larger instruction-tuned models exhibit internal geometry aligned with the human correlation matrix, while base models show little alignment under the assay;
3. Geometry strength is associated with behavioral amplification, but the evidence does not establish that the measured directions causally mediate cross-persona answers.

The paper therefore supports decodability and geometric alignment under the tested conditions, not a claim that the models possess psychological construct representations identical to those of humans.

How the Cross-Persona Paradigm Measures Structure

The cross-persona experiment does not simply ask the model, "How anxious are you?" Instead, real human scale responses are encoded as conditioning information. The model then predicts responses on the remaining scales, and the researchers compare the resulting correlation structure with that of the human sample.

If constructs such as neuroticism and depressive tendency covary in the human population, predictions for multiple scales should preserve similar dependencies when the model is conditioned on one scale. Larger models reproduce these dependencies and amplify their magnitude.

The authors also correct for measurement attenuation. Finite scale reliability systematically attenuates observed human correlations. The amplification persists after this correction, ruling out a simple artifact caused by different measurement-error structures.

Behavioral correlation alone, however, cannot locate the responsible representation inside the network. It also cannot distinguish genuine internal construct representation from surface patterns inherited from training data.

Testing Internal Geometric Alignment

Using contrastive direction analysis, the authors project activation differences generated under different scale conditions into a high-dimensional representation space. These directions are then compared with the geometry implied by the human seven-scale correlation matrix.

Larger instruct models show clearer alignment, whereas base models show little alignment under the current assay. One possible explanation is that instruction tuning improves controllable persona conditioning and the verbalization of psychological states, allowing stronger output-level regularity without necessarily changing what the base model internally encodes.

Across models, geometry strength correlates with behavioral amplification at r=0.68, p=0.008. Controlling for the logarithm of model size yields a partial correlation of r=0.69. This reduces the likelihood that the association is merely a byproduct of larger models producing more human-like behavior, although model-family coverage and sample size still limit broader generalization.

Readout Attenuation and the Pitfalls of Probe Slopes

The authors use a matched activation-probing paradigm to map psychological constructs onto internal representations. Probe slopes range from 1.38 to 1.77, whereas behavioral slopes range from 0.52 to 1.53. Internal signals therefore respond more strongly to conditioning than the final outputs do.

The authors call this gap "readout attenuation": a decodable signal exists internally, but nonlinear transformations, instruction adaptation, answer generation, and stylistic or safety constraints weaken its expression at the output.

The gap should not, however, be attributed entirely to a genuine model-internal mechanism. A synthetic control shows that ridge probing can produce inflated slopes even when the target structure is weak or absent, a familiar manifestation of probe overfitting and apparent effect inflation. After subtracting a baseline estimated with a reliability proxy from another analysis, the adjusted slopes remain associated with behavior at r=0.88.

The more defensible conclusion is that the assay contains both real geometric alignment and bias introduced by the probing method. "Readout attenuation" is an operational, empirical description rather than a causally established mechanism.

Engineering and Evaluation Trade-offs

The work is valuable for interpretability methodology, but probe slopes should not be treated as direct measurements of psychological construct strength.

• Correlation is not causation: An internal direction that predicts output does not prove that generation uses that direction. Activation intervention, directional ablation, or causal patching is still required.
• Scale is a confounder: Both behavioral amplification and internal alignment may change with size. The study controls for log model size, but the sample cannot fully exclude architecture, training-data, and alignment-tuning effects.
• Cultural and demographic boundaries: The human reference sample consists of Taiwanese university students. Generalization to other cultures, age groups, and clinical populations remains untested.
• Dependence on probe design: Ridge regression can amplify weak signals, and alternative regularization schemes, cross-validation protocols, or nonlinear probes may yield different slopes.
• Base-instruct differences are not cleanly isolated: Larger instruct models also have stronger behavioral modeling capability, so the current design cannot fully separate training stage, model scale, and conditional-expression ability.

Recommendations for Developers and Researchers

Researchers using activation probes to evaluate psychological or persona behavior should report reliability corrections, synthetic controls, train-test separation, and stability across probe families. Decodability and causal use should be tested separately, ideally through directional interventions or activation patching.

For mental-health products, the results also need to be combined with clinical validity, cross-cultural validation, bias audits, and human oversight. Model output must not be treated as a psychological diagnosis.

From an engineering perspective, the study is best used as a representation-diagnostic and evaluation-design tool rather than as a deployable personality meter. Its central contribution is to show that output-level psychometric structure, internal geometric alignment, and actual causal mediation are separate claims that require separate validation.

---

Source: ArXiv AI (cs.AI) (2026-11-11)

— END —

Tags: #LLM Interpretability Follow #Psychometrics Follow #Activation Probing Follow #Llama 3 Follow #Qwen 2.5 Follow