Reduces a score table to one row per ANON record describing how clear-cut that record's best candidate is, under either of two measures.
Usage
reid_confidence(
scores,
method = c("margin", "tie"),
tolerance = reid_tie_tolerance()
)Arguments
- scores
a score table (see [score_num()])
- method
`"margin"` (default since Issue #44) or `"tie"` (the default up to and including Issue #16)
- tolerance
relative tolerance for deciding that two candidate scores are tied, default `sqrt(.Machine$double.eps)` (Issue #61). Without it `TIE_SIZE` and `MARGIN` are not invariant to a change of units: on a 200-record fixture, expressing the same values in 1/10 units left 198 of 200 records with a `MARGIN` below 1e-9 – non-zero only as an artefact of binary representation – and turned 108 genuine two-way ties into unique wins. Pass `tolerance = 0` for the exact `==` comparison used before #61; see `docs/default-changes.md`.
Value
a data frame with one row per ANON record, ordered by ANON_ROW_NUMBER, with columns ANON_ROW_NUMBER, N_CANDIDATES, BEST_SCORE, SECOND_SCORE, TIE_SIZE, MARGIN, SD_SCORE, ECCENTRICITY and CONFIDENCE. `CONFIDENCE` is `1 / TIE_SIZE` or `ECCENTRICITY` according to `method`; the other columns are reported either way so the choice can be second guessed without recomputing.
The two measures
- `"tie"`
`1 / TIE_SIZE`, where `TIE_SIZE` counts the RAW records sharing the best score. A calibrated probability: it is the chance that a uniform draw among the tied best candidates lands on the right one, *given* that the right one is in that group. Its weakness is resolution – every record whose best candidate is unique scores 1, however marginal that win was.
- `"margin"`
the eccentricity `(second best score - best score) / sd(candidate scores)`. It distinguishes a runaway winner from a photo finish, so it orders records that `"tie"` cannot separate. Because the runner-up is the second *best candidate* rather than the second distinct score, a tie at the top gives margin 0: it subsumes `"tie"`'s information instead of discarding it. It is **not** a probability – a value of 2 does not mean anything on a 0-1 scale – and it is only comparable between records scored the same way. Dividing by the record's own sd is what makes it comparable at all, since the raw margin inherits the units of the score.
Eccentricity does not carry between data sets
This is the single most important caveat about `"margin"`, and it applies with full force now that `"margin"` is the **default** (Issue #44). Eccentricity is a ratio to *the record's own* candidate spread, so its numeric range is a property of the score table, not of the risk. Measured maxima:
* dense 150-person two-attribute numeric fixture: **0.45** * sparse transaction-style scores (Issue #23): **4.86**
An order of magnitude apart, on data whose *risk* is not an order of magnitude apart – the dense fixture had 98 reidentified, and eccentricity still separated correct guesses from incorrect ones there (mean 0.167 against 0.007), which is what the measure is for. What it is not for is comparison across tables. So:
* **A threshold is not portable.** `min_confidence = 0.3` may attack the confident tenth of one table and every record of another. * **The constant 1.5 from Narayanan & Shmatikov does not transfer.** Their sparse scores put a true match several sd above the field; on dense data it rejects every record, and a tool that reports zero reidentifications because its threshold was on the wrong scale looks exactly like a tool reporting that the data is safe (docs/lessons-learned.md section 2). [match_greedy()] and [match_optimal()] warn when that happens, but the warning is a backstop, not a substitute for choosing the threshold. * **Two eccentricities are comparable only within one score table.** Do not read "data set A reached 0.9, B only 0.4" as A being riskier; use `success_analytic` or `max_risk` from [reid_evaluate()] for that, which are unaffected by the choice of confidence measure.
Pick thresholds from the observed distribution, never from a constant:
“` stats::quantile(reid_confidence(scores, "margin")$CONFIDENCE, 0.9) “`
`"tie"` has none of this problem – it is a probability in `(0, 1]` and means the same thing everywhere – which is the trade for its near-total lack of resolution. On the 150-person fixture above it took a single distinct value (1) across all 150 records, while `"margin"` took 150.
Changed defaults
`method` defaulted to `"tie"` when this function was added (Issue #16) and defaults to `"margin"` from Issue #44 onwards, as do the `confidence` arguments of [match_greedy()], [match_optimal()] and [reid_evaluate()]. Only the `CONFIDENCE` column moves: `TIE_SIZE`, `MARGIN`, `SD_SCORE` and `ECCENTRICITY` are reported under either setting, and the risk quantities in [reid_evaluate()] do not depend on the choice at all. Code that compared `CONFIDENCE` against a literal, or that fed it to `min_confidence`, will see different numbers than before; pass `method = "tie"` (or `confidence = "tie"`) for the old behaviour. See `docs/default-changes.md`.
Both measures are computed on the internally minimised scale, so a "similarity" score table gives the same answer as the distance table it negates to.
Edge cases
A record whose candidates all share one score has margin 0 and eccentricity 0: nothing distinguishes them. A record with a *single* candidate has no runner-up to be confused with, and its eccentricity is `Inf` – the honest limit, and harmless to a threshold sweep, which only ever sorts and compares.
References
Narayanan, A. and Shmatikov, V. (2008) Robust De-anonymization of Large Sparse Datasets. IEEE Symposium on Security and Privacy.
Examples
raw <- data.frame(ROW_NUMBER = 1:6, V = c(1, 1, 2, 5, 9, 14))
d <- join_raw_anon_data(raw, raw)
reid_confidence(score_num(d, "V"), method = "margin")
#> ANON_ROW_NUMBER N_CANDIDATES BEST_SCORE SECOND_SCORE TIE_SIZE MARGIN SD_SCORE
#> 1 1 6 0 0 2 0 5.240865
#> 2 2 6 0 0 2 0 5.240865
#> 3 3 6 0 1 1 1 4.647580
#> 4 4 6 0 3 1 3 2.898275
#> 5 5 6 0 4 1 4 3.076795
#> 6 6 6 0 5 1 5 5.240865
#> ECCENTRICITY CONFIDENCE
#> 1 0.0000000 0.0000000
#> 2 0.0000000 0.0000000
#> 3 0.2151657 0.2151657
#> 4 1.0350983 1.0350983
#> 5 1.3000542 1.3000542
#> 6 0.9540410 0.9540410