Latest Update (and hopefully the last)

For paper 2, I used the following 5 sets of fields:
Set 1 = {name1, name2, last, mablast, bmo, bday byr, sex, ssn, stand_homephone, stand_workphone, street1, city1, state1, zip1, spname1, splast} stored in c0-c5
Set 2 = {name1, last, bmo, bday byr, sex, ssn, stand_homephone, street1, city1, state1, zip1} stored in cfs0-cfs5
Set 3 = {name1, laast, ssn, stand_homephone, street1} stored in sfs0-sfs5
Set 4 = {name1, last, ssn} stored in nfs0-nfs5
Set 5 = {name1, last, street1} stored in sss0-sss5

I created a few new functions in sdi_funcs.R and put the old ones in there. So you only need to run source("sdi_funcs.R") .

When you change the cutoff for approximate agreement or field set, change to the right dir in R, then use these R commands:
> sdi("0.edmu.score")
[1] "min score = 2059(154fps, 1905fns) which occurs at: "
[1] -0.9 -0.8 -0.7 -0.6 -0.5 -0.4 -0.3 -0.2 -0.1
> sdi("0.mu.score")
[1] "min score = 2059(154fps, 1905fns) which occurs at: "
[1] -0.9 -0.8 -0.7 -0.6 -0.5 -0.4 -0.3 -0.2 -0.1
> sdi("1.edmu.score")
[1] "min score = 1104(266fps, 838fns) which occurs at: "
[1] -1.5 -1.4
> sdi("1.mu.score")
[1] "min score = 2059(154fps, 1905fns) which occurs at: "
[1] -2.4 -2.3 -2.2 -2.1 -2.0 -1.9 -1.8 -1.7 -1.6 -1.5 -1.4 -1.3 -1.2 -1.1 -1.0
[16] -0.9 -0.8 -0.7 -0.6 -0.5 -0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2 0.3 0.4 0.5
[31] 0.6 0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2.0
[46] 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 3.0 3.1 3.2 3.3 3.4 3.5
[61] 3.6 3.7 3.8 3.9 4.0 4.1 4.2 4.3 4.4
> sdi("2.edmu.score")
[1] "min score = 1082(216fps, 866fns) which occurs at: "
[1] 0.4
> sdi("2.mu.score")
[1] "min score = 2059(154fps, 1905fns) which occurs at: "
[1] -2.4 -2.3 -2.2 -2.1 -2.0 -1.9 -1.8 -1.7 -1.6 -1.5 -1.4 -1.3 -1.2 -1.1 -1.0
[16] -0.9 -0.8 -0.7 -0.6 -0.5 -0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2 0.3 0.4 0.5
[31] 0.6 0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2.0
[46] 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 3.0 3.1 3.2 3.3 3.4 3.5
[61] 3.6 3.7 3.8 3.9 4.0 4.1 4.2 4.3 4.4
> edmu = scoreDist("validate.edmu.score")
[1] "Best divide: 0.000000"
[1] "Manual Review Pairs: 1089.000000"
[1] "FP: 232.000000 (0.391885%)"
[1] "FN: 857.000000 (1.447611%)"
> edmu = scoreDist("validate.mu.score")
[1] "Best divide: -2.400000"
[1] "Manual Review Pairs: 1966.000000"
[1] "FP: 151.000000 (0.255063%)"
[1] "FN: 1815.000000 (3.065827%)"
> scoreDiff()
[1] 2.222350
[1] -1.962576

This gets the correct cutoff for EDMU and MU needed for RunEM.java, scores the validation set, and calcualtes the mean score shift. The mean score shift for Dups is the average difference between the final score calculated by the standard linkage model (MU) and the extended linkage model (EDMU). A positive score for Dups means that EDMU gives (on average) a higher score to pairs that are duplicates. The mean score shift for Non-Dups is the same thing, but a lower score is better. A negative score for Non-Dups means that EDMU gives (on average) a lower score to pairs that are not duplicates.

To generate m() and u() function graphs for a field do this:
Go to C:\dev\paper2\paper2\bin and type:
java paper2.alignment.Aligner -t dpairs rpairs10 -f last > last.align
(substitute the field you want for 'last')

Take the last.align file. If you have source-ed sdi_funcs.R, plotmu() is included. Type:
plotmu("last.align")
determine the right maximum for the y-axis. Then type:
plotmu("last.align", maxy=0.8)

To plot the approximate agreement weight function for a field, type:
plotweight("last.align")
determine the right minimum and maximum for the y-axis. Then type:
plotweight("last.align", miny=-5, maxy=10)


To plot a series of distributions (like showDists or scoreDists or sdis):
plotDist()
it is automatically set up to print the distributions for the c0 directory. You can print other directories by using:
plotDist("c1").
whichMethod can be used to switch between EDMU and MU distributions. whichMethod={"edmu", "mu"}
whichPlot can be used to either use show or full score, sdi is not implemented yet. whichPlot = {"show", "score"}. Using show prints the first n-1 iterations and then plots the validation set using scoreDist.
printCross lets you put the cross on scoreDist plots. printCross={0,1}
n is the number of iterations you did (defaulted to 3)
normy is whether to normalize the y-axis to 0..1 instead of the best fit. normy={0,1}

How to redo for cutoffs

Retry try11 as cut0:
core fields,
initial params: 1,-1,
disagreement cutoff: 0

RunEM to get .0 files

in R:
change dir to c:\Program Files\R\R2.4.1\paper2
source("showDist.R")
source("scoreDist.R")
chage dir to c:\Program Files\R\R2.4.1\paper2\cut0
showDist("0.EDMU.score")
enter showDist EDMU cutoff (if it is good) in Java

showDist("0.MU.score")
enter showDist MU cutoff (if it is good) in Java

MU and EDMU cutoffs for 0. iteration should be the same (same starting params)

repeat for 1., 2., and validate. iteration steps.

Then do
edmu = scoreDist("validate.EDMU.score")
mu = scoreDist("validate.MU.score")
plot(edmu, type="l", xlim=c(0,0.02), ylim=c(.98,1))
lines(mu)

change cutoffs by changing hard coded variable approx in main (approx line 348)
change default field set in main (approx line 424)

name scoreDists
edmu
mu0 for 0 cutoff
mu1 for 1 cutoff
...
mu5 for 5 cutoff

then use roc.R and run roc() to get ROC curve and AUC for all values.

Otherwise use int(edmu) for AUC and above plot, lines combo.

Comparitive Tries

try11 - dpairs, rpairs10, core fields
EDMU: 105 (28, 77)
MU: 171 (73, 98)

try12b - dpairs, last_rpairs10, core fields
EDMU: 195 (58, 137)
MU: 271 (88, 183)

try13 - dpairs, last_zip_rpairs10, core fields
EDMU: 456 (294, 162)
MU: 517 (281, 236)

try14 - dpairs, last_bmo_byr_rpairs10, core fields
EDMU: 4175 (3342, 833)
MU: 4234 (3370, 464)

Updated Instructions

RunEM updated to take output directory and compare to regular M/U estimation.

java paper2.em.RunEM -Xmx1024m -v -o try11

1. It will load data from database where half=0 (excluding isDup=1 in random tables).
2. Use initial parameters to score records.

3. Output i.EDMU.score and i.MU.score files for iteration i.
4. Require input of a cutoff for EDMU and MU parameter recalculation.
5. Update the parameters based on the cutoff.

6. continue steps 3-5 for each iteration.
7. validate using half=1 and print validate.EDMU.score and validate.MU.score files

To find correct cutoffs, use showDist.R. (showDist("0.EDMU.score"))
To check FP/FN, use scoreDist.R. (scoreDist("0.EDMU.score"))

showDist shows density of all scores and outputs minimum near score mean.
scoreDist shows densities of divided scores (by table) and outputs minimum error rate.

To save PDFs of cutoff and distribution iterations, use printDist.R (printDist("try11"))

tries

try1-try4: Tries

try5: dpairs, rpairs10, core fields
105 (28,77)

try6: dpairs, rpairs10, small fields
107 (30, 77)

try7: dpairs, last_rpairs10, small fields
216 (79, 137)

try8: dpairs, last_zip_rpairs10, small fields
660 (471, 189)

try9: dpairs, last_zip_rpairs10, core fields
655 (493, 162)

try10: dpairs, last_bmo_byr_rpairs10, core fields
7558 (6733, 825)

NOTE: don't count pairs as FP if isDup=1, do more iterations.

Expectation Maximization

The new algorithm is all rolled into one.

Run RunEM in Eclipse. It will generate .0 - .N align and score files. Use showScore.R to find appropriate cutoff. Enter it into the RunEM console to update the parameters.

** more details coming **

Starting with 4,0 and using 3 iterations:
Best Divide: -1.1
Manual Review Pairs: 105
FP: 28 (0.047296%)
FN: 77 (0.130065%)

Results (fullscore)

EMU (estimated m&u values) all fields, skip missing values, EMU.score, EMU_1.score
Total: 797
FPs: 546
FNs: 251

EMU (estimated m&u values) core fields, skip missing values, EMU_2.score
Total: 626
FPs: 467
FNs: 159

OMU (estimated based on observed values) all fields, skip missing values, OMU.score, OMU_1.score
Total: 142
FPs: 39
FNs: 103

OMU (estimated based on observed values) core fields, skip missing values, OMU_2.score
Total: 141
FPs: 38
FNs: 103

EDMU (edit distance based on observed values), all fields, do not skip missing values, EDMU_1.score
Total: 601
FPs: 169
FNs: 432

EDMU (edit distance based on observed values), all fields, skip missing values, EDMU_2.score
Total: 128
FPs: 42
FNs: 82

EDMU (edit distance based on observed values), all fields, skip missing values, using edmu2 cutoffs, EDMU_3.score
Total: 123
FPs: 45
FNs: 78

EDMU (edit distance based on observed values), core fields, do not skip missing values, EDMU_4.score
Total: 163
FPs: 43
FNs: 120

EDMU (edit distance based on observed values), core fields, skip missing values, EDMU_5.score
Total: 107
FPs: 31
FNs: 76

EDMU (edit distance based on observed values), core fields, skip missing values, using edmu2 cutoffs, EDMU_6.score
Total: 122
FPs: 54
FNs: 68







1. With complete EDMU distribution including missing data edmu files in edmu1, but EDMU.score not saved

Total: 122 (?)
FPs: 39 (?)
FNs: 83 (?)


2. With complete EDMU distribution except missing data = 0 (code change for this fix. stored as EDMU-no-missing-data.score)

Total: 107
FPs: 31
FNs: 76

3. With partial EDMU distributions converging to account for small numbers in tail of distribution. Scores larger than last known value will receive last known value score. (edmu files in edmu2)

Total: 111
FPs: 42
FNs: 69

Hmmm. Don't know why.