Final Test

The final test has 496,000 active queires (total of 2,304,000 query registrations and 1,808,000 query removals), and 415,104 documents. The size of the final test data file is 71 GB. All submissions were run with the final test, and the 5 top performing submissions were further investigated. Each of the top 5 submissions was run 3 times to account for randomness, and the average throughput is reported in the leaderboard.

Below is a step by step example to run the final benchmark on the winning submission:
1. Create a directory to save the downloaded files to, we'll call this directory sig13con in this example.
2. Download the final test driver source tarball and team phoenix tarball to sig13con.
3. Download the final test data file to sig13con. The file is 26 GB, so this might take a while!
4. Navigate to sig13con in a terminal then run the following commands:
sig13con> tar -xzf sigmod2013contest-final-testdriver.tar.gz
sig13con> tar -xzf phoenix.tar.gz
sig13con> tar -xzf final_test.tar.gz
sig13con> mv final_test.txt final-testdriver/test_data
sig13con> cd final-testdriver; make; cd ..
sig13con> cd phoenix; make; cd ..
sig13con> cp phoenix/libcore.so final-testdriver/
sig13con> cd final-testdriver
sig13con/final-testdriver> ./testdriver test_data/final_test.txt 3600
sig13con/final-testdriver> cat result.txt


For details on the final test driver, please read the included README file.

On our test machine (specs available on task details page) phoenix's implementation takes slightly less than an hour to finish the final test. If you're impatient and want to do a quick comparison between implementations you can put a lower time limit and see the throughput of an implementation without waiting for it to finish the entire final test data.