Latest Update (and hopefully the last)

For paper 2, I used the following 5 sets of fields:
Set 1 = {name1, name2, last, mablast, bmo, bday byr, sex, ssn, stand_homephone, stand_workphone, street1, city1, state1, zip1, spname1, splast} stored in c0-c5
Set 2 = {name1, last, bmo, bday byr, sex, ssn, stand_homephone, street1, city1, state1, zip1} stored in cfs0-cfs5
Set 3 = {name1, laast, ssn, stand_homephone, street1} stored in sfs0-sfs5
Set 4 = {name1, last, ssn} stored in nfs0-nfs5
Set 5 = {name1, last, street1} stored in sss0-sss5

I created a few new functions in sdi_funcs.R and put the old ones in there. So you only need to run source("sdi_funcs.R") .

When you change the cutoff for approximate agreement or field set, change to the right dir in R, then use these R commands:
> sdi("0.edmu.score")
[1] "min score = 2059(154fps, 1905fns) which occurs at: "
[1] -0.9 -0.8 -0.7 -0.6 -0.5 -0.4 -0.3 -0.2 -0.1
> sdi("0.mu.score")
[1] "min score = 2059(154fps, 1905fns) which occurs at: "
[1] -0.9 -0.8 -0.7 -0.6 -0.5 -0.4 -0.3 -0.2 -0.1
> sdi("1.edmu.score")
[1] "min score = 1104(266fps, 838fns) which occurs at: "
[1] -1.5 -1.4
> sdi("1.mu.score")
[1] "min score = 2059(154fps, 1905fns) which occurs at: "
[1] -2.4 -2.3 -2.2 -2.1 -2.0 -1.9 -1.8 -1.7 -1.6 -1.5 -1.4 -1.3 -1.2 -1.1 -1.0
[16] -0.9 -0.8 -0.7 -0.6 -0.5 -0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2 0.3 0.4 0.5
[31] 0.6 0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2.0
[46] 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 3.0 3.1 3.2 3.3 3.4 3.5
[61] 3.6 3.7 3.8 3.9 4.0 4.1 4.2 4.3 4.4
> sdi("2.edmu.score")
[1] "min score = 1082(216fps, 866fns) which occurs at: "
[1] 0.4
> sdi("2.mu.score")
[1] "min score = 2059(154fps, 1905fns) which occurs at: "
[1] -2.4 -2.3 -2.2 -2.1 -2.0 -1.9 -1.8 -1.7 -1.6 -1.5 -1.4 -1.3 -1.2 -1.1 -1.0
[16] -0.9 -0.8 -0.7 -0.6 -0.5 -0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2 0.3 0.4 0.5
[31] 0.6 0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2.0
[46] 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 3.0 3.1 3.2 3.3 3.4 3.5
[61] 3.6 3.7 3.8 3.9 4.0 4.1 4.2 4.3 4.4
> edmu = scoreDist("validate.edmu.score")
[1] "Best divide: 0.000000"
[1] "Manual Review Pairs: 1089.000000"
[1] "FP: 232.000000 (0.391885%)"
[1] "FN: 857.000000 (1.447611%)"
> edmu = scoreDist("validate.mu.score")
[1] "Best divide: -2.400000"
[1] "Manual Review Pairs: 1966.000000"
[1] "FP: 151.000000 (0.255063%)"
[1] "FN: 1815.000000 (3.065827%)"
> scoreDiff()
[1] 2.222350
[1] -1.962576

This gets the correct cutoff for EDMU and MU needed for RunEM.java, scores the validation set, and calcualtes the mean score shift. The mean score shift for Dups is the average difference between the final score calculated by the standard linkage model (MU) and the extended linkage model (EDMU). A positive score for Dups means that EDMU gives (on average) a higher score to pairs that are duplicates. The mean score shift for Non-Dups is the same thing, but a lower score is better. A negative score for Non-Dups means that EDMU gives (on average) a lower score to pairs that are not duplicates.

To generate m() and u() function graphs for a field do this:
Go to C:\dev\paper2\paper2\bin and type:
java paper2.alignment.Aligner -t dpairs rpairs10 -f last > last.align
(substitute the field you want for 'last')

Take the last.align file. If you have source-ed sdi_funcs.R, plotmu() is included. Type:
plotmu("last.align")
determine the right maximum for the y-axis. Then type:
plotmu("last.align", maxy=0.8)

To plot the approximate agreement weight function for a field, type:
plotweight("last.align")
determine the right minimum and maximum for the y-axis. Then type:
plotweight("last.align", miny=-5, maxy=10)


To plot a series of distributions (like showDists or scoreDists or sdis):
plotDist()
it is automatically set up to print the distributions for the c0 directory. You can print other directories by using:
plotDist("c1").
whichMethod can be used to switch between EDMU and MU distributions. whichMethod={"edmu", "mu"}
whichPlot can be used to either use show or full score, sdi is not implemented yet. whichPlot = {"show", "score"}. Using show prints the first n-1 iterations and then plots the validation set using scoreDist.
printCross lets you put the cross on scoreDist plots. printCross={0,1}
n is the number of iterations you did (defaulted to 3)
normy is whether to normalize the y-axis to 0..1 instead of the best fit. normy={0,1}