Statistical principles

Ethnicolr reports name-pattern probabilities with their reference population, calibration status, and assumptions. This page defines those statistical claims and their limits.

What the probabilities mean

Every scored prediction returns a full probability distribution over classes (one column per class). After validated post-hoc calibration, hispanic = 0.83 means: among held-out evaluation names to which the model assigns ≈0.83, the true class is hispanic ≈83% of the time, under the evaluation weighting documented in each model’s stats file (see “Whose probabilities?” below).

Two caveats apply to any name-based model:

  1. Names are genuinely ambiguous. Even a perfect model cannot assign “Smith” to one class with certainty; the honest output is a spread-out distribution. Low confidence is information, not failure.

  2. The probability is about a population, not a person. p(race | name) is a statement about people who share a name, not a determination about an individual.

Calibration (temperature scaling)

Each model ships with a stats file (*_stats_pt.json) produced by scripts/model-training/calibrate_model.py:

  • The model’s deterministic held-out split is divided into a calibration half and an evaluation half.

  • A single temperature T is fitted on the calibration half by minimizing negative log-likelihood (Guo et al. 2017); shipped probabilities are softmax(logits / T). Temperature scaling never changes the ranking of classes, so accuracy is untouched.

  • Expected calibration error (ECE, 15 bins) and multiclass Brier score are reported before and after on the evaluation half, together with full reliability-diagram bins. Reliability plots are rendered in the model cards.

Calibration quality varies by source. Each artifact records both pre-scaling and post-scaling results so the package does not assume temperature scaling helped. Conformal coverage is checked separately on the evaluation half.

Whose probabilities? (weighting)

Several training sets deduplicate names or balance classes, so a naive held-out evaluation would answer “for a random unique name” rather than “for a random person”. Each stats file records its calibration_weighting:

  • census: person-weighted by construction (names sampled by population count, labels drawn from census race shares);

  • Florida: a source-disjoint, person-level sample of registered voters;

  • NC: names were deduplicated for training, so held-out rows are re-inflated by their pre-deduplication frequency in the voter file (person-weighted);

  • wiki: one row per unique notable person (Wikipedia/Wikidata); there is no natural person-frequency to inflate to, and the reference population is “people notable enough for Wikipedia”, not any national population.

Base rates and the target_prior= argument

Models trained on class-balanced data (Florida and North Carolina) output probabilities that answer: which class, if all classes were equally common? Applied to a real population this overstates rare classes by construction. The fix is a Bayes adjustment, exposed on each estimator that returns model probabilities:

estimate_florida_voter_surname(
    df,
    "last",
    target_prior={
        "asian": 0.03,
        "hispanic": 0.27,
        "nh_black": 0.15,
        "nh_white": 0.50,
        "other": 0.05,
    },
)

computes p_adj(y|x) ∝ p_cal(y|x) · π_target(y) / π_train(y), where π_train is read from the stats file. Pass the demographic margins of your population (a state, a county, a census tract). With tract-level margins this is the name-likelihood step of BISG-style methods (Elliott et al. 2009; Imai & Khanna 2016); for full BISG pipelines see surgeo (Python) or wru (R). Ethnicolr’s calibrated likelihoods are designed to compose with them, covering names absent from census dictionaries.

Conformal prediction sets (conformal_coverage=)

estimate_wikipedia_full_name(df, "last", "first", conformal_coverage=0.90)

For a model with a valid calibration artifact, this adds a race_set column: the smallest set of classes whose calibrated probability mass reaches the conformal quantile fitted on held-out data (split-conformal with adaptive prediction sets). The guarantee: among names exchangeable with the calibration data, the true class falls inside the set at least 90% of the time. Each stats file reports an empirical check (conformal_empirical_coverage). Valid shipped models are within 2 to 3 percentage points of nominal. Set sizes adapt per name: unambiguous names get singletons, ambiguous names get honestly larger sets. Supported levels: 0.80, 0.90, 0.95.

Fine print, stated plainly:

  • Coverage is marginal. It is averaged over names and not guaranteed per subgroup. Group-conditional coverage is reported in the model cards where it deviates materially.

  • The guarantee assumes your names are exchangeable with the calibration distribution (census sample, voter files, Wikipedia). Under distribution shift, treat the level as approximate.

  • conformal_coverage= cannot be combined with target_prior= because reweighting invalidates the stored quantiles. It also cannot be combined with uncertainty_level.

Monte Carlo dropout variation (uncertainty_level=)

Monte Carlo dropout summaries are not confidence intervals. The *_mc_mean, *_mc_std, *_mc_lower, and *_mc_upper columns answer a different question: how stable is the probability estimate itself under model uncertainty? A probability can be precisely estimated yet wrong, or noisy yet calibrated, so intervals and conformal sets complement rather than replace each other. The percentile range carries no frequentist coverage guarantee.

Responsible-use boundary and abstention

All outputs are name-pattern estimates tied to a reference population, not determinations about a person. They must not be used for individual or consequential decisions. Inference APIs report script_supported, abstained, and abstention_reason; blank inputs, unsupported scripts, dictionary misses without a fallback, and inputs with no known model features receive no class probabilities. Every result also reports the package version, model identifier, reference population, calibration status/reference, uncertainty method/level, and a SHA-256 revision of the complete runtime artifact bundle.

Dictionary estimators and the independence assumption

lookup_census_first_name, estimate_census_full_name, and estimate_voter_file_full_name are dictionary estimators: exact conditional frequencies from public tables (Census 2020 first-name and surname files; six-state voter-file name tables), no neural network involved for in-dictionary names.

Combining first and last names uses naive Bayes:

p(race | first, last) ∝ p(race | last) · p(race | first) / π(race)

which assumes first and last names are conditionally independent given race. This is an approximation. Culturally correlated first/last pairs (e.g. a distinctively Irish first name with an Irish surname) make the combined posterior somewhat overconfident, because the two names partially repeat the same evidence. The evidence_basis column records exactly what evidence each row’s estimate used, including surname-model fallback for out-of-dictionary surnames.

Reference populations differ and are part of each estimator’s meaning: census tables describe the 2020 US enumerated population; the voter dictionaries describe registered voters in AL/FL/GA/LA/NC/SC (their implied race marginal ships in rosenman_stats.json and anchors the target_prior= adjustment).

Exact intervals for census lookups

lookup_census_surname(..., uncertainty_level=0.95) and lookup_census_first_name(..., uncertainty_level=0.95) add Wilson score bounds computed from the published name counts. These capture sampling uncertainty in the published proportions. The uncertainty is near zero for common names, honest for rare ones. The 2020 counts additionally carry the Census Bureau’s disclosure-avoidance noise (±3 per cell at 95% probability), which the intervals do not model; for counts above a few hundred it is negligible.

The origin model’s reference population

estimate_wikipedia_origin predicts the likely country of origin of a name over 90 country classes, trained on Wikidata people using citizenship as the label. Two caveats are part of its meaning: the reference population is people notable enough for Wikipedia/Wikidata, and citizenship is a proxy for name origin. For migrants the two differ. Melting-pot citizenships and ambiguous historical unions are excluded from training for this reason; see the auditable mapping in scripts/data-acquisition/wiki/mappings/country_to_origin.csv. With 90 classes, single-country answers are often uncertain. Prefer the full distribution, conformal_coverage= sets, or aggregate columns into regions, e.g. df[["Sweden", "Norway", "Denmark", "Finland", "Iceland"]].sum(axis=1).

References

  • Guo, Pleiss, Sun & Weinberger (2017). On Calibration of Modern Neural Networks. ICML.

  • Elliott et al. (2009). Using the Census Bureau’s surname list to improve estimates of race/ethnicity. Health Services and Outcomes Research Methodology (BISG).

  • Imai & Khanna (2016). Improving Ecological Inference by Predicting Individual Ethnicity from Voter Registration Records. Political Analysis.

  • Imai, Olivella & Rosenman (2022). Addressing census data problems in race imputation via fully Bayesian Improved Surname Geocoding. Science Advances.

  • Rosenman, Olivella & Imai (2023). Race and ethnicity data for first, middle, and surnames. Scientific Data.

  • Angelopoulos & Bates (2023). Conformal Prediction: A Gentle Introduction. Foundations and Trends in Machine Learning.

  • Gelman, Hill & Vehtari (2020). Regression and Other Stories. (On calibrated probabilistic prediction and base rates.)