evaluate a reidentification attack: baselines, precision-recall, top-k and per-record risk

Takes the output of the score layer and reports how dangerous the attack actually is, in a form that can be read without already knowing the answer.

Usage

reid_evaluate(
  scores,
  seeds = 1:20,
  top_k = c(1, 5, 10),
  confidence = c("margin", "tie"),
  tolerance = reid_tie_tolerance()
)

Arguments

scores: a score table produced by a `score_*()` function or [combine_scores()]
seeds: integer vector of tie-break seeds used for the simulated success rate (default 1:20). Must contain at least 2 distinct values.
top_k: integer vector of guess budgets for the top-k hit rate (default `c(1, 5, 10)`); values larger than the number of candidates are dropped.
confidence: which attacker-visible confidence the precision-recall sweep should threshold on: `"margin"` (default since Issue #44, eccentricity) or `"tie"` (`1 / tie size`). `"tie"` is a calibrated probability but has almost no resolution on continuous scores – every record with a unique best candidate lands on 1, so the sweep collapses to a single point equal to the overall success rate. `"margin"` gives a distinct threshold per record and is what makes "attack the top 10 be right most of the time" visible. It is an ordering, not a probability (Issue #16), and its scale does not carry between data sets.
tolerance: relative tolerance for deciding that two candidate scores are tied, default `sqrt(.Machine$double.eps)` (Issue #61). Every risk figure here is built out of "how many candidates are at least as good as the true one", so an exact `==` made them depend on the units the input happened to use: the same 200-record data set expressed in 1/10 units moved `max_risk` from 0.5 to 1.0 and grew `precision_recall` from 3 rows to 93. Pass `tolerance = 0` for the pre-#61 behaviour; see `docs/default-changes.md`.

Value

an object of class "reid_evaluation": a list with

n_anon, n_raw, n_pairs: size of the problem
n_pairs_full, candidate_coverage: the size of the full cross join **over the records present in the score table**, and what fraction of it is present. A record dropped entirely is invisible to these two, which is why the next three exist beside them
n_true_missing, truth_coverage, truth_measurable: for how many ANON records the true RAW record is not a candidate at all, the complementary fraction, and whether at least one ANON record could have been reidentified
blocked: TRUE when the candidate set is not the full cross join, by either test: fewer rows than `n_anon * n_raw`, **or** `n_true_missing > 0`
n_zero_candidate: how many ANON records had an empty candidate set under containment (`candidate_count == 0`), or `NA` when the score table carries no such attribute. Independent of `blocked`: the table keeps its full shape, so neither test above can see it (Issue #101)
confidence: which confidence measure the sweep thresholded on
success_analytic: exact expected single-guess success rate
success_mean, success_sd, success_min, success_max, n_seeds: the same quantity simulated over `seeds`
per_seed: data frame of seed / success / trial / rate
baseline: data frame of method / rate for the random and mode baselines
lift: `success_analytic` divided by the random baseline
top_k: data frame of k / hit_rate
precision_recall: data frame of threshold / n_attacked / coverage / precision / recall
per_record: data frame of per-ANON-record risk, ordered by decreasing RISK
max_risk: the largest per-record risk

Details

**Baselines.** Two attacks that use no information are evaluated on the same data: `random`, which assigns each ANON record a RAW record uniformly at random, and `mode`, which ignores the ANON record entirely and always names the single RAW record that comes out best most often. A measured success rate must beat both before it means anything; `lift` is the ratio to the random baseline.

**Precision-recall.** The attacker can see `CONFIDENCE` but not the answer, so they can choose to attack only the records they are confident about. Sweeping a threshold over `CONFIDENCE` gives, at each coverage level, the precision they would achieve. This is what makes "10 precision" visible; the plain success rate averages it away.

**Variance.** The success rate is also simulated over `seeds` tie-break draws, so the point estimate comes with an sd and a range. A single run is one draw from a distribution, not a property of the data: on a tie-heavy 50-person fixture the rate ranges over \[0.02, 0.14\] depending only on the draw. `success_analytic` is the exact expectation of that same quantity; it agreeing with `success_mean` is a self-check on the implementation.

Changed defaults

`confidence` defaulted to `"tie"` before Issue #44 and to `"margin"` from #44 onwards, so `precision_recall` and the CONFIDENCE column of `per_record` **differ from what earlier versions reported for the same input.** On a 150-record continuous fixture the sweep went from 1 row (threshold 1, attack 150/150, precision 0.2467) to 150 rows (top row: threshold 0.3941, attack 1/150, precision 1.0000). `success_analytic`, `success_mean`, `baseline`, `lift`, `top_k`, `RISK` and `max_risk` are **unchanged** – the confidence measure only reorders records, it does not alter the risk. Pass `confidence = "tie"` to reproduce old numbers. See `docs/default-changes.md`.

Blocked candidate sets

A full cross join has exactly `n_anon * n_raw` rows. Anything smaller was filtered – by [block_candidates()], [lsh_candidates()], [top_k_candidates()] or by hand – and a filtered candidate set can only lower the measured rate, never raise it. That is checked here from the score table itself rather than from an attribute the caller has to remember to pass on, and reported as `blocked` / `candidate_coverage` / `n_true_missing`, which the print method shows above the success rate (Issue #36).

The row-count test alone is not enough, because `n_anon` and `n_raw` are counted from the score table and so only see the records that survived. When every surviving ANON record was offered every surviving RAW record the table is a **complete rectangle over a subset**, `nrow(scores) == n_anon * n_raw` holds, and the shape test cannot fire – which is what a release published as a single region, year or category produces. So `n_true_missing` (and `truth_coverage`) is measured against the ground truth, is treated as independent evidence of a filtered candidate set, and is printed whether or not the shape test fired (Issue #56).

When `n_true_missing == n_anon` **nothing at all was measured**: no ANON record could have been reidentified, so every rate is 0 by construction. That prints identically to a genuinely safe release, so this warns.

Empty candidate sets

Both tests above look at the *shape* of the candidate table, and Issue #101 is a failure that leaves the shape perfectly intact. [score_containment()] keeps every pair – an excluded candidate scores 1 rather than being dropped – so an ANON record whose published region contains no RAW record at all still has `n_raw` candidates, still has its true pair among them, and passes both tests: `blocked` FALSE, `n_true_missing` 0, `truth_coverage` 1. It simply cannot be reidentified. Every candidate ties, [match_greedy()] draws uniformly, and the record contributes exactly the random baseline; when it happens to every record the whole report reads `lift 1.00x`, which is what a unit written on one side only produces.

`n_zero_candidate` is read from the `candidate_count` attribute [score_containment()] attaches, and is `NA` for a score table that does not carry one. [score_containment()] itself warns; this is the same fact carried through to the printed report.

Examples

raw <- data.frame(ROW_NUMBER = 1:6, V = c(1, 1, 2, 2, 3, 3))
d <- join_raw_anon_data(raw, raw)
reid_evaluate(score_num(d, "V"), seeds = 1:10, top_k = c(1, 2))
#> reid evaluation: 6 ANON x 6 RAW record(s), 36 candidate pair(s)
#>   success rate   : 0.5000 exact | simulated mean 0.4333 sd 0.2629 range [0.1667, 0.8333] over 10 seeds
#>   baseline       : random 0.1667 | mode 0.1667   (lift vs random: 3.00x)
#>   top-k hit rate : k=1 0.5000  k=2 1.0000
#>   max per-record risk: 0.5000
#>   precision-recall (threshold on attacker-visible CONFIDENCE, margin):
#>     conf >= 0.0000 : attack 6/6 (100.0%)  precision 0.5000  recall 0.5000