score several numeric columns jointly by Mahalanobis distance
Source:R/multiattr.R
score_mahalanobis.RdTreats the chosen columns as one vector-valued attribute and scores a candidate pair by the Mahalanobis distance between the RAW and ANON vectors,
Arguments
- dat_raw_anon
dataframe of raw_anon form
- targets
character vector of numeric column names, *before* the RAW_/ANON_ prefixing done by [join_raw_anon_data()]
- row_number
name of the row-number column *before* the RAW_/ANON_ prefixing done by [join_raw_anon_data()] (default: "ROW_NUMBER")
- cov_from
which side to estimate the covariance from: `"raw"` (default – the attacker's own reference population), `"anon"`, or `"pooled"` (the distinct records of both sides stacked). The default is a threat-model choice, not a tuning knob: a real attacker whitens with the population they hold. `"anon"` folds the release's own perturbation into the covariance and so is less fragile when \(S\) is ill-conditioned (measured 0.3350 against 0.2100 at kappa 2756), but it is still far below the weighted sum there, so it is a diagnostic rather than a rescue.
- ridge
non-negative shrinkage applied to the diagonal of the covariance matrix, as a multiple of its mean diagonal entry (default 1e-6). Needed because redundant columns – exactly the case Mahalanobis exists for – make \(S\) ill-conditioned or singular. It cannot repair an ill-conditioned result: a ridge large enough to matter turns the metric into the weighted sum it was supposed to improve on.
- squared
return the squared distance instead of the distance (default FALSE). The two give identical rankings; the square root is the default because it shares the units of the underlying columns.
- generalized
what to do when one of `targets` turns out to hold generalised values on the ANON side: `"stop"` (default), `"warn"` or `"ignore"`. A generalised column is also non-numeric, so this only decides which of the two errors is raised. See [score_containment()].
- .fn_name
name used in error messages; a function that wraps this one passes its own name so the message points at the function the user actually called
Value
a "reid_scores" table whose SCORE is the Mahalanobis distance (a distance: smaller is a better match)
Details
$$D = \sqrt{(x_{RAW} - x_{ANON})^\top S^{-1} (x_{RAW} - x_{ANON})}$$
where \(S\) is the covariance matrix of the reference population (by default the distinct RAW records).
WHY NOT JUST ADD THE COLUMNS UP. A weighted sum treats every column as an independent piece of evidence. Two columns that are strongly correlated carry roughly one column's worth of information but get two columns' worth of influence, which comes out of the budget of whatever independent column is competing with them. \(S^{-1}\) removes exactly that double counting, and at the same time up-weights directions in which the population barely varies – a small disagreement along such a direction is far more surprising, and therefore far more identifying, than the same disagreement along a direction the population is spread out over anyway.
WHEN THAT ARGUMENT FAILS (Issue #59). Up-weighting the low-variance directions is only right if the release did not *also* perturb the data along them. It is the ratio of the release's perturbation to the population spread **in the whitened directions** that decides the outcome, and strong correlation makes some of those directions narrow – so the metric is at its most fragile in exactly the situation it is recommended for.
Measured on a 3-column fixture (A, B correlated at `rho`; C independent; `success_analytic` over 200 records, the full sweep is in `docs/default-changes.md`):
| perturbation | rho | kappa(S) | weighted | mahalanobis |
| isotropic | 0.90 | 26.7 | 0.9600 | 0.9200 |
| isotropic | 0.99 | 273.4 | 0.9350 | 0.5600 |
| isotropic | 0.999 | 2755.6 | 0.8850 | 0.2100 |
| follows S | 0.99 | 273.4 | 0.8900 | 0.9800 |
| follows S | 0.999 | 2755.6 | 0.8400 | 0.9800 |
The same covariance, the same condition number, opposite conclusions: with an isotropic perturbation the whitened attack is 4.2x weaker than a plain weighted sum, and with a perturbation that follows the population covariance it is 1.2x stronger. **A high condition number therefore says the answer is fragile, not that it is wrong.** `score_mahalanobis()` warns above 100 and tells you to compare against `method = "weighted"`; do that before quoting either number.
Raising `ridge` is not a fix, only a retreat: at `ridge = 1` both regimes land on the weighted sum's own figure (0.8950 / 0.8650), which is to say the metric has stopped doing anything. That is why the default is left at 1e-6.
The covariance is estimated from *distinct records*, deduplicated by row number, not from the candidate pairs: a cross join repeats every record once per candidate, and estimating from the repeated rows would silently weight each record by how many candidates it happens to have.
Examples
set.seed(1)
n <- 30
a <- rnorm(n)
## correlated, but not so nearly-collinear that the whitening becomes
## fragile -- see the condition-number discussion above
raw <- data.frame(ROW_NUMBER = 1:n, A = a, B = 2 * a + rnorm(n, sd = 1))
anon <- raw
anon$A <- round(anon$A, 1)
anon$B <- round(anon$B, 1)
d <- join_raw_anon_data(raw, anon)
match_greedy(score_mahalanobis(d, c("A", "B")))
#> ANON_ROW_NUMBER RAW_ROW_NUMBER CONFIDENCE RESULT
#> 1 1 1 0.77448230 TRUE
#> 2 2 2 0.31058864 TRUE
#> 3 3 3 0.54306450 TRUE
#> 4 4 4 0.06153628 TRUE
#> 5 5 5 0.80633069 TRUE
#> 6 6 6 0.88255427 TRUE
#> 7 7 7 0.24250972 TRUE
#> 8 8 8 0.13189765 TRUE
#> 9 9 9 0.27747921 TRUE
#> 10 10 10 0.38906314 TRUE
#> 11 11 11 0.03243436 TRUE
#> 12 12 12 0.11594106 TRUE
#> 13 13 13 0.25516195 TRUE
#> 14 14 14 1.59610783 TRUE
#> 15 15 15 0.22796739 TRUE
#> 16 16 16 0.56235453 TRUE
#> 17 17 17 0.01497908 TRUE
#> 18 18 18 0.17696444 TRUE
#> 19 19 19 0.06519261 TRUE
#> 20 20 20 0.26970442 TRUE
#> 21 21 21 0.52462620 TRUE
#> 22 22 22 0.36862347 TRUE
#> 23 23 23 0.14017889 TRUE
#> 24 24 24 0.54764289 TRUE
#> 25 25 25 0.43158401 TRUE
#> 26 26 26 0.89098615 TRUE
#> 27 27 27 0.56631023 TRUE
#> 28 28 28 0.53997500 TRUE
#> 29 29 29 0.19050605 TRUE
#> 30 30 30 0.18428308 TRUE