score a set-valued column ("A:B:C") by set overlap

For a column that holds a *set* of elements per record – the `_DIST` columns [transform_transaction_to_master()] builds out of a categorical column, a list of purchased items, of shops visited, of pages read – the evidence is which elements two records share, not any numeric distance between them. [score_dist()] reads such a column as a distribution of numbers and cannot handle a categorical one at all.

Usage

score_jaccard(
  dat_raw_anon,
  target,
  row_number = "ROW_NUMBER",
  split = ":",
  method = c("jaccard", "dice", "overlap", "tversky"),
  alpha = 1,
  beta = 1,
  multiset = FALSE,
  generalized = c("stop", "warn", "ignore"),
  .fn_name = "score_jaccard"
)

Arguments

dat_raw_anon

dataframe of raw_anon form

target

target column

row_number

name of the row-number column *before* the RAW_/ANON_ prefixing done by [join_raw_anon_data()] (default: "ROW_NUMBER")

split

character separating the elements, treated as a **literal string** and never as a regular expression (Issue #32)

method

one of `"jaccard"` (default), `"dice"`, `"overlap"` or `"tversky"`; see [set_similarity()] for the definitions

alpha, beta

Tversky asymmetry parameters, used only when `method = "tversky"`. `alpha` weights elements present in RAW but not ANON (what anonymisation removes), `beta` elements present in ANON but not RAW (much stronger evidence of a wrong match).

multiset

if `TRUE`, repeated elements count with their multiplicities; the default `FALSE` reduces each record to a true set, which is the reading Issue #18 is about

generalized

what to do when `target` turns out to hold generalised values on the ANON side: `"stop"` (default), `"warn"` or `"ignore"`. A generalised value splits into a one-element set that intersects nothing, so every candidate pair scores 1 and the column silently carries no signal (Issue #100).

.fn_name

name used in error messages; a function that wraps this one passes its own name so the message points at the function the user actually called

Value

a "reid_scores" table whose SCORE is `1 - similarity`, in \[0, 1\] (a distance: smaller is a better match)

See also

[score_minhash()] for a min-hash approximation of the same score, [lsh_candidates()] for candidate blocking, and [score_containment()] for generalised columns.

Examples

raw <- data.frame(
  ROW_NUMBER = 1:3,
  ITEMS = c("apple:beer:cod", "apple:donut", "egg:fig"),
  stringsAsFactors = FALSE
)
anon <- data.frame(
  ROW_NUMBER = 1:3,
  ITEMS = c("beer:cod", "donut", "fig:egg"),
  stringsAsFactors = FALSE
)
match_greedy(score_jaccard(join_raw_anon_data(raw, anon), "ITEMS"))
#>   ANON_ROW_NUMBER RAW_ROW_NUMBER CONFIDENCE RESULT
#> 1               1              1   1.732051   TRUE
#> 2               2              2   1.732051   TRUE
#> 3               3              3   1.732051   TRUE