build a reduced RAW/ANON candidate table by deterministic blocking

A drop-in replacement for [join_raw_anon_data()] that keeps only the pairs agreeing on a **blocking key** – the oldest and cheapest candidate reduction there is, and the one that makes a hundred-thousand-record assessment possible at all: the full join of 100,000 x 100,000 is 10^10 pairs.

Usage

block_candidates(
  raw,
  anon,
  keys,
  transform = NULL,
  row_number = "ROW_NUMBER",
  max_pairs = 1e+07,
  raw_header = "RAW_",
  anon_header = "ANON_"
)

Arguments

raw, anon: data frames, as for [join_raw_anon_data()]
keys: the blocking key. Either a character vector of column names – all of which must agree – or a list of such vectors, whose passes are unioned. Columns must exist in both `raw` and `anon`.
transform: optional named list of functions, applied to the column of that name on **both** sides before the keys are compared. Use it to block on a coarsened value, e.g. `list(AGE = function(x) x %/% 10)`.
row_number: name of the row-number column (default "ROW_NUMBER"), used to identify the true pairs when measuring recall
max_pairs: safety valve: stop rather than materialise more than this many candidate pairs (default 1e7). A key that barely discriminates produces one enormous block and no saving at all; failing loudly is better than exhausting memory.
raw_header, anon_header: column prefixes, as for [join_raw_anon_data()]

Value

a data frame in raw_anon form holding a *subset* of the pairs [join_raw_anon_data()] would produce, carrying a `blocking` attribute (a [print()]able "reid_blocking" record of the reduction and the recall).

Details

`keys` names the columns that must agree. Give several passes to take their **union**, which is the standard way to buy recall back: a record whose ZIP was perturbed is still reachable through the (AGE, SEX) pass. `transform` coarsens a column before comparison, so blocking on a *generalisation* of a quasi-identifier – decade of age, first three digits of a postcode – does not need a second column in the data.

**THIS UNDER-REPORTS RISK WHEN IT LOSES A TRUE PAIR.** If the true counterpart of an ANON record does not agree on any key, it is not a candidate, the record can never be reidentified, and the measured reidentification rate goes *down*. That is the direction a safety tool must not fail in quietly (`docs/lessons-learned.md` section 2), so this function measures its own recall, records it in the `blocking` attribute of the result, and warns when it is below 1. Recall is exact, not estimated: the true pairs are the ones sharing a `row_number`, which is the same ground truth [reid_evaluate()] scores against.

Blocking is only worth it when the key is *stable* under the anonymisation being assessed. A key the release perturbs (a noised age, a suppressed postcode) drops true pairs by construction; the recall figure will say so.

Examples