Skip to contents

score a set-valued column by min-hash estimated Jaccard distance

Source: R/setsim.R
score_minhash.Rd

Same score as `score_jaccard(method = "jaccard")`, estimated from `n_hash` min-hash components instead of computed exactly. The point is cost: exact Jaccard touches every element of both sets for every candidate pair, while min-hash reduces each record to a fixed-length signature once and then compares signatures.

Usage

score_minhash(
  dat_raw_anon,
  target,
  row_number = "ROW_NUMBER",
  split = ":",
  n_hash = 128L,
  seed = 0L,
  multiset = FALSE,
  generalized = c("stop", "warn", "ignore"),
  .fn_name = "score_minhash"
)

Arguments

dat_raw_anon

dataframe of raw_anon form

target

target column

row_number

name of the row-number column *before* the RAW_/ANON_ prefixing done by [join_raw_anon_data()] (default: "ROW_NUMBER")

split

character separating the elements, treated as a **literal string** and never as a regular expression (Issue #32)

n_hash

number of hash components (default 128)

seed

integer seed for drawing the hash family (default 0L), or NULL for the ambient RNG stream

multiset

if `TRUE`, repeated elements count with their multiplicities; the default `FALSE` reduces each record to a true set, which is the reading Issue #18 is about

generalized

what to do when `target` turns out to hold generalised values on the ANON side: `"stop"` (default), `"warn"` or `"ignore"`. A generalised value splits into a one-element set that intersects nothing, so every candidate pair scores 1 and the column silently carries no signal (Issue #100).

.fn_name

name used in error messages; a function that wraps this one passes its own name so the message points at the function the user actually called

Value

a "reid_scores" table whose SCORE is `1 - estimated Jaccard`

Details

It is an *estimate*, with standard error `sqrt(J (1 - J) / n_hash)` – about 0.044 at `J = 0.5` with the default 128 components. That error is symmetric, so it does not bias the reported reidentification rate in either direction, but it does blur genuinely close candidates. Use [score_jaccard()] unless the exact computation is actually too slow, and treat a min-hash result as a lower bound on the resolution of the exact one.

See also

[score_containment()] for generalised columns.

Examples

raw <- data.frame(
  ROW_NUMBER = 1:3,
  ITEMS = c("a:b:c:d", "c:d:e:f", "x:y:z:w"),
  stringsAsFactors = FALSE
)
match_greedy(score_minhash(join_raw_anon_data(raw, raw), "ITEMS", n_hash = 64))
#>   ANON_ROW_NUMBER RAW_ROW_NUMBER CONFIDENCE RESULT
#> 1               1              1   1.369095   TRUE
#> 2               2              2   1.369095   TRUE
#> 3               3              3   1.732051   TRUE