Cross Validation

Cross Validation

1. Update tables dpairs and rpairs10 using queries like these:
1. ALTER TABLE `frequency`.`rpairs10` ADD COLUMN `half` TINYINT UNSIGNED NOT NULL DEFAULT 0 AFTER `isDup`;
2. ALTER TABLE `frequency`.`dpairs` ADD COLUMN `half` TINYINT UNSIGNED NOT NULL DEFAULT 0 AFTER `isDup`;
3. update rpairs10 set half = 1 order by rand() limit 59201;
4. update dpairs set half = 1 order by rand() limit 59201;
   
   These queries will split the datasets into two parts, one to train and one to test.
   
   
2. Update file paper2.alignment.Aligner
1. Change the SQL statement in LoopThroughFields() to include
   
1. "WHERE half = 0" (approximately line 90)
   
   
3. Run this set of commands: (found in massAlign.bat)
   java paper2.alignment.Aligner -t dpairs rpairs10 -f name1 > name1.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f name2 > name2.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f last > last.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f bmo > bmo.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f bday > bday.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f byr > byr.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f sex > sex.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f ssn > ssn.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f mablast > mablast.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f licno > licno.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f homephone > homephone.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f workphone > workphone.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f nok1name1 > nok1name1.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f nok1name2 > nok1name2.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f nok1last > nok1last.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f nok1rel > nok1rel.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f nok2name1 > nok2name1.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f nok2name2 > nok2name2.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f nok2last > nok2last.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f nok2rel > nok2rel.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f street1 > street1.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f city1 > city1.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f state1 > state1.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f zip1 > zip1.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f street2 > street2.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f city2 > city2.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f state2 > state2.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f zip2 > zip2.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f spname1 > spname1.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f spname2 > spname2.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f splast > splast.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f stand_homephone > stand_homephone.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f stand_workphone > stand_workphone.align
   java paper2.alignment.Aligner -t dpairs rpairs10 -f standstreet > standstreet.align
   
   This will make the distributions of the edit distances for both the dups and the random pairs.
   
   
4. Run the script in alignments3.R and then massAlign.R
   
   This will generate the massAlign.pdf file, .edmu, and .omu files for each field.
   
   .edmu files contain a weight for each edit distance difference between values. They start on a scale of -3 going up to the maximum edit distance observed in that field.
   -3 (the first value) means both records are missing a value
   -2 (the 2nd value) means the first record is missing a value
   -1 (the 3rd value) means the second record is missing a value
   0 (the 4th value) means the values are exactly the same
   1 (the 5th value) means the values have an edit distance of 1
   etc.
   
   .omu files contain an agreement weight and a disagreement weight
   If the values match exactly, they get the agreement weight.
   If they don't match exactly, they get the disagreement weight.
   If either record is missing a value, they get the score of 0 (or as defined by nullTreatment in paper2.linker.Linker)
   
   
5. Make the .emu files
   
   
6. Update file paper2.linker.Linker.
1. Change SQL statement in anotherWayToCompare() to include:
1. "WHERE half = 1" (approximately line 222)
   
2. Decide which fields you want to run and update the allFields array (approximately line 210)
   
3. Decide which compare method and update the whichWay variable in the main method (approximately line 375)
1. 2 means use EDMU probabilities
2. 3 means use OMU probabilites with real M and U values
1. M = observed % exact match on field for each field in dpairs
2. U = observed % exact match on field for each field in rpairs10
   
3. 4 means use EMU probabilites with estimated M and U values
1. M = ?
   
2. U = 1 / number of values per field
4. Note - using 3 or 4 has an agreement weight of log10(m/u), a disagreement weight of log10(1-m/1-u), and a score of 0 for missing values. (nullTreatment value of 2)
   
   
7. Run the commands (found in linkAll.bat)
   java -Xmx1024m paper2.linker.Linker -t dpairs rpairs10 -compare 2 -muT 0 -muN 2 > EDMU.score
   java -Xmx1024m paper2.linker.Linker -t dpairs rpairs10 -compare 3 -muT 0 -muN 2 > OMU.score
   java -Xmx1024m paper2.linker.Linker -t dpairs rpairs10 -compare 4 -muT 0 -muN 2 > EMU.score
   
   EDMU = edit distance m & u
   OMU = observed m & u
   EMU = estimated m & u
   
   
8. Run the script in fullscore.R and then run:
   
   fullscore("EDMU.score")
   fullscore("OMU.score")
   fullscore("EMU.score")