Comparitive Tries

try11 - dpairs, rpairs10, core fields
EDMU: 105 (28, 77)
MU: 171 (73, 98)

try12b - dpairs, last_rpairs10, core fields
EDMU: 195 (58, 137)
MU: 271 (88, 183)

try13 - dpairs, last_zip_rpairs10, core fields
EDMU: 456 (294, 162)
MU: 517 (281, 236)

try14 - dpairs, last_bmo_byr_rpairs10, core fields
EDMU: 4175 (3342, 833)
MU: 4234 (3370, 464)

Updated Instructions

RunEM updated to take output directory and compare to regular M/U estimation.

java paper2.em.RunEM -Xmx1024m -v -o try11

1. It will load data from database where half=0 (excluding isDup=1 in random tables).
2. Use initial parameters to score records.

3. Output i.EDMU.score and i.MU.score files for iteration i.
4. Require input of a cutoff for EDMU and MU parameter recalculation.
5. Update the parameters based on the cutoff.

6. continue steps 3-5 for each iteration.
7. validate using half=1 and print validate.EDMU.score and validate.MU.score files

To find correct cutoffs, use showDist.R. (showDist("0.EDMU.score"))
To check FP/FN, use scoreDist.R. (scoreDist("0.EDMU.score"))

showDist shows density of all scores and outputs minimum near score mean.
scoreDist shows densities of divided scores (by table) and outputs minimum error rate.

To save PDFs of cutoff and distribution iterations, use printDist.R (printDist("try11"))

tries

try1-try4: Tries

try5: dpairs, rpairs10, core fields
105 (28,77)

try6: dpairs, rpairs10, small fields
107 (30, 77)

try7: dpairs, last_rpairs10, small fields
216 (79, 137)

try8: dpairs, last_zip_rpairs10, small fields
660 (471, 189)

try9: dpairs, last_zip_rpairs10, core fields
655 (493, 162)

try10: dpairs, last_bmo_byr_rpairs10, core fields
7558 (6733, 825)

NOTE: don't count pairs as FP if isDup=1, do more iterations.

Expectation Maximization

The new algorithm is all rolled into one.

Run RunEM in Eclipse. It will generate .0 - .N align and score files. Use showScore.R to find appropriate cutoff. Enter it into the RunEM console to update the parameters.

** more details coming **

Starting with 4,0 and using 3 iterations:
Best Divide: -1.1
Manual Review Pairs: 105
FP: 28 (0.047296%)
FN: 77 (0.130065%)

Results (fullscore)

EMU (estimated m&u values) all fields, skip missing values, EMU.score, EMU_1.score
Total: 797
FPs: 546
FNs: 251

EMU (estimated m&u values) core fields, skip missing values, EMU_2.score
Total: 626
FPs: 467
FNs: 159

OMU (estimated based on observed values) all fields, skip missing values, OMU.score, OMU_1.score
Total: 142
FPs: 39
FNs: 103

OMU (estimated based on observed values) core fields, skip missing values, OMU_2.score
Total: 141
FPs: 38
FNs: 103

EDMU (edit distance based on observed values), all fields, do not skip missing values, EDMU_1.score
Total: 601
FPs: 169
FNs: 432

EDMU (edit distance based on observed values), all fields, skip missing values, EDMU_2.score
Total: 128
FPs: 42
FNs: 82

EDMU (edit distance based on observed values), all fields, skip missing values, using edmu2 cutoffs, EDMU_3.score
Total: 123
FPs: 45
FNs: 78

EDMU (edit distance based on observed values), core fields, do not skip missing values, EDMU_4.score
Total: 163
FPs: 43
FNs: 120

EDMU (edit distance based on observed values), core fields, skip missing values, EDMU_5.score
Total: 107
FPs: 31
FNs: 76

EDMU (edit distance based on observed values), core fields, skip missing values, using edmu2 cutoffs, EDMU_6.score
Total: 122
FPs: 54
FNs: 68







1. With complete EDMU distribution including missing data edmu files in edmu1, but EDMU.score not saved

Total: 122 (?)
FPs: 39 (?)
FNs: 83 (?)


2. With complete EDMU distribution except missing data = 0 (code change for this fix. stored as EDMU-no-missing-data.score)

Total: 107
FPs: 31
FNs: 76

3. With partial EDMU distributions converging to account for small numbers in tail of distribution. Scores larger than last known value will receive last known value score. (edmu files in edmu2)

Total: 111
FPs: 42
FNs: 69

Hmmm. Don't know why.