OTA-BERTurk Quantifies Cultural-Psychological Constructs in Ottoman Court Records
Summary:Researchers have built a computational text-analytic framework around 54,670 Ottoman Turkish court records from 21 Istanbul courts spanning 1513–1911. Its domain-adapted sentence embedding model, OTA-BERTurk, is used to quantify individualism, collectivism, and honor through theory-driven construct representations and a scoring pipeline. The reported results include 0.843 Recall@1 for document retrieval, 0.806 MRR for semantic textual similarity, and 89.58% accuracy on questionnaire-item classi
Researchers have turned 54,670 Ottoman court records into a reproducible system for measuring cultural-psychological constructs: OTA-BERTurk first encodes the text, after which theory-driven construct representations and a scoring pipeline map it along dimensions such as individualism, collectivism, and honor. The corpus covers 21 Istanbul courts from 1513 to 1911.
Technical mechanism: from domain embeddings to construct scores
OTA-BERTurk is presented as a sentence embedding model adapted to Ottoman legal language. In the standard sentence-embedding paradigm, a text (x) is mapped to a vector (h(x)=f_θ(x)), and document-query similarity can be computed with cosine similarity:
sim(x,y) = h(x)ᵀh(y) / (||h(x)|| ||h(y)||)
The resulting vector space supports document retrieval, semantic matching, and construct classification. The central contribution, however, is not a new Transformer architecture. It is the integration of domain-specific representations with a theory-driven measurement pipeline.
Conceptually, a construct can be represented by one or more vectors (r_{c,k}), with a construct score written as:
z_c(x) = Agg_k sim(h(x), r_c,k)
This is an illustrative formulation, not a confirmed equation from the paper. The Agg_k operator could involve questionnaire anchors, prototype texts, lexicons, thematic rules, or an ensemble, but the collected abstract does not specify the implementation. Chunking strategy, document-level weighting, and aggregation from documents to courts and historical periods must therefore be verified in the released code. Vector similarity alone is not equivalent to psychological measurement.
The corpus is itself a major research asset. With 54,670 records spanning roughly four centuries and 21 courts, it enables both longitudinal comparison and tests of model stability under changes in legal vocabulary, documentary conventions, and institutional context. The records are confined to Istanbul courts, however, and cannot be treated as direct evidence about the psychology of the entire Ottoman Empire.
Evaluation results: retrieval quality is not construct validity
The paper uses several complementary evaluation paths rather than relying on a single classification score:
- Document retrieval: Recall@1 is 0.843. Under the reported test protocol, 84.3% of queries place the relevant document first. This is not equivalent to an 84.3% accuracy rate for psychological judgments over the full corpus.
- Semantic textual similarity: MRR is 0.806. Because reciprocal rank strongly rewards top positions, this indicates that relevant answers are generally ranked highly. MRR, however, measures ordering quality rather than semantic agreement, human consistency, or construct validity.
- Questionnaire-item classification: Accuracy reaches 89.58%, testing whether the model can distinguish statements associated with different constructs.
- Expert and qualitative validation: The paper reports that high-scoring documents exhibit historically specific legal and institutional patterns. This supports interpretability but also suggests that the model may capture institutional language alongside psychological semantics.
Formally, Recall@1 = 1/N × Σ 1[top-1 relevant] and MRR = 1/N × Σ 1/rank_i. Their credibility depends on query construction, relevance labels, candidate-pool size, data splitting, and hard-negative selection, none of which are described in the collected abstract.
Ottoman court documents often contain fixed formulas, recurring legal language, and templated passages. A random document-level split can place near-duplicate templates in both training and test sets, inflating retrieval and classification results. Reproduction should therefore include grouped splits by court, period, and document template, together with a template-deduplicated evaluation.
Measurement invariance is also essential across four centuries. A term or documentary pattern may carry different construct meanings in early and late courts. Score changes can then reflect linguistic drift, institutional change, or recording practices rather than genuine psychological change. The abstract does not provide per-construct results, confidence intervals, expert sample sizes, or annotation agreement, so the 89.58% overall accuracy must not be interpreted as an accuracy rate for historical psychological measurement.
Engineering trade-offs: adaptation, indexing, and interpretability
Domain adaptation versus compute cost. A general-purpose multilingual sentence model is easy to deploy but may map Ottoman legal terminology, spelling variants, and documentary structures into inappropriate semantic neighborhoods. Domain adaptation can improve retrieval and classification, but it also requires careful control of training objectives, negative sampling, sequence length, and overfitting. The abstract does not disclose OTA-BERTurk's base model, pooling method, training loss, maximum input length, training hardware, or contamination checks.
Whether approximate vector search is necessary. For 54,670 documents, exhaustive cosine scoring is entirely feasible; HNSW or FAISS-style approximate indexes become more compelling only as the corpus scales. With a hypothetical 768-dimensional float32 embedding, the vectors alone occupy approximately 54,670 × 768 × 4 ≈ 168 MB, before metadata, database structures, and original text. Serving cost will depend more directly on average tokenized length, batching, and whether source passages are retained for auditing.
Interpretability versus compression. A single construct score is convenient for visualization and statistical comparison, but it removes disagreement, context, and within-document variation. A high honor score might reflect family relations, property disputes, litigation vocabulary, or religious norms, and a scalar cannot distinguish among those causes. An interactive interface should expose the triggering passages and nearby cases rather than presenting only a color-coded heat map.
Open assets and reproducibility. The paper states that models, construct representations, the scoring pipeline, and an interactive web interface will be released. That is valuable for reproducible scholarship. The collected material, however, provides no repository URL, release tag, license, dependency environment, split definition, or precomputed scores. Production use should pin model hashes, code commits, random seeds, and corpus versions while recording every post-processing step.
Reproduction and deployment workflow
- Audit the released assets. Check the base model, license, commercial restrictions, training-corpus documentation, tokenizer coverage of historical spellings and transcriptions, and whether the web interface uploads user data by default.
- Rebuild leakage-resistant splits. Create at least court-held-out, period-held-out, and template-held-out test sets. For retrieval tasks, publish the query-relevance protocol and hard negatives.
- Establish interpretable baselines. Compare OTA-BERTurk with off-the-shelf multilingual sentence models, bag-of-words or TF-IDF representations, lexicon-only scoring, and a same-capacity model without domain adaptation.
- Control documentary confounds. Include period, court, document type, length, and formulaic structure in stratified or regression analyses. Prefer within-court comparisons when estimating temporal change.
- Quantify uncertainty. Use court-level grouped bootstrapping or hierarchical models, and report variance across seeds for retrieval and classification.
- Validate the constructs. Have experts annotate independent samples without seeing model scores, report Fleiss κ or Krippendorff α, and test for differential item functioning across periods and courts.
- Preserve the evidence chain. Every aggregated score should link back to the source document, passage, model version, and construct rule. The web interface is suitable for exploration, but exploratory heat maps should not be presented directly as causal claims about historical populations.
Research value and limitations
The main contribution is a measurement protocol and open research infrastructure, not a new deep-learning architecture. It connects sentence embeddings, construct representations, document-level scoring, and historical text analysis into an auditable pipeline with direct value for digital humanities, social history, and computational cultural studies.
The boundaries are equally important. Court records are institutionally generated texts, not private psychological diaries. Whether modern definitions of individualism, collectivism, and honor transfer without loss into an Ottoman legal context remains a theoretical question requiring explicit justification and external validation. The reported association between high scores and legal or institutional patterns indicates that construct scores should be treated as variables to explain rather than direct observations of historical individuals' inner states.
The work is best regarded as a promising, resource-rich arXiv v1 preprint. Its real value will be established if the released code reproduces, grouped tests pass, and results survive independent historical review across periods.
— END —Source: ArXiv NLP/LLM (cs.CL) (2026-11-20)
Tags: #OTA-BERTurk #Sentence Embeddings #Computational History #Ottoman Archives
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments