================================================== README FILE FOR Python script run-tbit-test-0.1.py ================================================== run-tbit-test-0.1.py is a python script that we have used to run TBIT tests in a slightly automated way. In this file we explain the code of this script and give examples of how we have used it to systematically run TBIT tests. BACKGROUND ========== The use of this script requires familiarity with the TBIT software. In order to achieve that: . Read the companion paper: http://www.aciri.org/tbit/tbit.ps . View the slides of the talk given at NANOG: http://www.aciri.org/tbit/nanog-talk.pdf STRUCTURE OF THE SCRIPT ======================= 1. The script has a set of routines to Execute and process the results of the following set of tests: - Reno: determines version of congestion control algorithm on remote host - ECN: does remote host negotiate an ECN connection - ECN_IPOnly: checks only to see if ECN-marked packets (in IP header) are dropped. - NewECN (*): Establishes an ECN-capble connection and counts number of marked packets and number of retransmissions. - SackSndr3P: using 3 packet drop, this test figures out if remote host responds correctly to SACKs. - Sack: does remote supports sack? (not correctness, just check if it does) - WinHalf: is the window reduced by half by remote host upon a packet drop? - Timestamp: does remote host use timestamps? (not correctness) - TahoeNoFR: if remote host's TCP is Tahoe, does it do fast retransmit? - SackRcvr: does remote host sends back SACKs correctly. (i.e. TBIT acts as sender) - TimeWait: Determines duration of time-wait period - DelACK: checks if remote host use delayed acknowledgments. - InitWin: Determines the initial window used by remote host - MinMSS(*): Determines the minimum segment size accepted by remote host - Reordering(*): Determines if a reordering took place using with a given MSS - Redirection (*): Determines if the "advertised" URL results in a redirection (*: new test; not "tbit-native". It may use of other tbit tests) There is an ExecuteTestNameTest() routine for each of the above tests (e.g. TestName = Reno, MinMSS,...), and a ProcessTestNameResults() routine for each test. 2. There are configuration variables based on the assumptions indicated below. ASSUMPTIONS: =========== The execution of run-tbit-test.0.1.py assumes the following: 1) A specific directory structure: there must be a directory for each test named: For example: ./reno-test ./minmss-test ./tahoenofr-test ... ... ./-test must be subdirectories that must exist in the directory from which run-tbit-test.0.1.py is run. For a given test, all the temporary results and final result processing will be done inside the associated subdirectory for the test. 2) The script also assumes the output is as provided in the current release of tbit. If the tbit code is changed such that its output changes significantly, then the result-processing routines in this script may fail. 3) TEST_DIR Variable: Set this variable to point to the working directory from which the test will be executed. 4) URL-LIST directory: TBIT tests are usually run against per-collected lists of URLs (web servers). The script assumes the existence of a a sub-directory called URL-LISTS where the specified url list will reside. 5) Both, running tbit and capturing packets from a network interface using tcpdump, require root access. We gain such access using the sudo command. If you don't have sudo access contact your system administrator to request your username to be added to the file /etc/sudoers. USING run-tbit-test.0.1.py ========================== The syntax for running the python script is as follows: run_tbit_test.py where: : list of URLs to be tested : Reno | ECN | Timestamp |... : MSS parameter - minimum pkt size to be requested : number of times to perform test on each server : file containing the processed results : 1 - only process results of previous experiment, 0 - do both experiment and processing The argument allows to execute a given test multiple times for each URL in the list. If the test succeeds on any given try, the successful result is recorded along with the iteration in which the test succeeded. If a test fails all the iterations for a given URL, the last failure mode is recorded in the results. Te argument allows to skip the execution of the test and proceed directly with the processing of the results. This is mainly to enable the development of or changes to the result-processing routines. EXAMPLE TEST RUNS ================= 1. Reno test: ------------ ./run-tbit-test-0.1.py url_list.txt Reno 128 5 TestScript_Reno.txt 0 For each server in "url_list.txt", this commands runs the Reno tbit test with an MSS of 128 bytes, 5 times, leaving the processed results in the file "TestScript.txt" and both test execution and processing of results will take place (last argument: 0). The file "TestScript_Reno.txt", which will be left in the reno-test directory, will contain the following contents: +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ www.tiscali.it 195.130.227.63 rx=4 to=0 version=Uncategorized rtt_unr=0 reorder=14 count=0 www.ebay.com 66.135.192.88 rx=3 to=1 version=TahoeNoFR rtt_unr=0 reorder=0 count=1 www.passport.net 65.54.230.254 rx=2 to=1 version=Reno rtt_unr=0 reorder=0 count=1 www.icra.org 66.221.165.93 rx=2 to=0 version=RenoPlus rtt_unr=0 reorder=4 count=1 www.mapquest.com 64.12.37.89 rx=2 to=0 version=NewReno rtt_unr=0 reorder=1 count=1 www.free.fr 213.228.0.42 rx=2 to=0 version=NewReno rtt_unr=0 reorder=1 count=1 www.drudgereport.com 66.28.209.210 rx=2 to=0 version=NewReno rtt_unr=0 reorder=0 count=1 ================================================ 3 NewReno rtt_unr 1 Reno rtt_unr 1 Uncategorized rtt_unr 1 TahoeNoFR rtt_unr 1 RenoPlus rtt_unr 7 Total ================================================ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ The given url list had 10 urls. For 7 of them the test succeeded and 3 failed. 2. ECN Test =========== ./run-tbit-test-0.1.py top10_large.txt ECN 128 5 TestNewScript_ECN.txt 0 This command is the same as before but now the ECN test is executed. The file "TestScript_ECN.txt", which will be left in the reno-test directory, will contain the following contents: Summary: ================================================ sr=1:se=0:ar=1:ae=0 5 sr=1:se=1:ar=1:ae=1 4 ERROR: 1 ------------------------------------------------ Total: 10 Meaning of results: ================================================ RST: reset NO_CONN: No connection sr=1: SYN/ACK received se=1/0: ECN_ECHO was set/not set in the SYN/ACK ar=1/0: Ack for the data packet (sent with ECT and CE bits set) received/not received ae=1/0: the ack had ECN_ECHO bit set/not set. ========================================================= www.tiscali.it 195.130.227.63 sr=1:se=1:ar=1:ae=1 www.icra.org 66.221.165.93 sr=1:se=1:ar=1:ae=1 www.free.fr 213.228.0.42 sr=1:se=1:ar=1:ae=1 www.drudgereport.com 66.28.209.210 sr=1:se=1:ar=1:ae=1 www.ebay.com 66.135.192.88 sr=1:se=0:ar=1:ae=0 www.google.com 216.239.53.99 sr=1:se=0:ar=1:ae=0 www.passport.net 65.54.230.254 sr=1:se=0:ar=1:ae=0 www.mapquest.com 64.12.37.89 sr=1:se=0:ar=1:ae=0 www.yahoo.co.jp 211.14.15.5 sr=1:se=0:ar=1:ae=0 www.microsoft.com 207.46.249.190 ERROR: NO_CONN ========================================================= 3. New ECN Test =============== The NewECN test is implemented part in the tbit tool and part in this script. The goal of the test is to download files from ECN capable servers and looking at the number of packet drops during the transmission as well as the number of packets that have set the ECN Capable Transport (ECT) and Congestion Experienced (CE) bits set. The idea is that if the losses are due to congestion *and* ECN is being actually used by routers in the path to the server *and* the server is actually implementing the ECN algorithm, we should observe certain correlation between the drop rate and the rate of packets marked with the ECT and CE bits. a) Given a list of web servers, check which servers are ECN-capable (see ECN test in TBIT's paper). b) For each ECN-capable server, establish a ECN-enabled session and download a large-enough file from the server. c) Capture a packet trace using tcpdump. d) Determine packet drops during the transmission. Currently, the script runs the tcptrace tool on the tcpdump trace and as a first approximation assumes a correlation between the number of retransmissions and the number of potential packet drops. e) Count the number of packets during the transmission that had the ECT and CE bits set. These five steps are implemented in the routine ExecuteNewECNTest(). The script is then run as follows: ./run-tbit-test-0.1.py top10_large.txt NewECN 128 5 TestNewScript_NewECN.txt 0 And the file "TestNewScript_NewECN.txt" will look like for this case: ==================================== ECN-Capable: 4 Not ECN-Capable: 5 Errors: 1 ==================================== www.tiscali.it 195.130.227.63 csnt:375 ssnt:372 cxmt:0 sxmt:55 cect:0 sect:315 cce:0 sce:0 www.icra.org 66.221.165.93 csnt:316 ssnt:313 cxmt:0 sxmt:108 cect:0 sect:203 cce:0 sce:0 www.free.fr 213.228.0.42 csnt:444 ssnt:440 cxmt:0 sxmt:156 cect:0 sect:282 cce:0 sce:0 www.drudgereport.com 66.28.209.210 csnt:98 ssnt:94 cxmt:0 sxmt:33 cect:0 sect:59 cce:0 sce:0 +++++++++++++++++++++++++++++++++++++++ 4. IsSack Test ============== ./run-tbit-test-0.1.py top10_large.txt Sack 128 5 TestNewScript_Sack.txt 0 ======================================== sack = 0: 5 sack = 1: 4 Errors: 1 Total: 10 ======================================== www.ebay.com 66.135.192.88 sack=0 www.passport.net 65.54.230.254 sack=1 www.yahoo.co.jp 211.14.15.5 sack=0 www.google.com 216.239.53.99 sack=0 www.mapquest.com 64.12.37.89 sack=1 www.tiscali.it 195.130.227.63 sack=0 www.icra.org 66.221.165.93 sack=0 www.free.fr 213.228.0.42 sack=1 www.drudgereport.com 66.28.209.210 sack=1 www.microsoft.com 207.46.249.190 ERROR: Could not establish contact after 2 retries ========================================= 5. SackSndr3P Test ================== ./run-tbit-test-0.1.py top6_large.txt SackSndr3P 128 1 SackSndr_NewScriptTest.txt 0 ======================================== sack = 0: 1 sack = 1: 0 no sack: 4 Errors: 1 Total: 6 ======================================== www.passport.net 65.54.230.254 sack = 0 www.nate.com 203.226.255.16 UNWANTED_PKT_REORDER 6. Redirection test =================== This test does not use the tbit tool. The test consists of simply checking if a requested URL gets redirected somewhere else. For each URL in the passed list, the test reports a binary outcome depending on whether or not a redirection took place. To run it: ./run-tbit-test-0.1.py testList100 Redirection 128 1 NewRedirection_Test.txt 0 ============================= Redirections: 2 Total: 3 ============================= cnn.com 64.236.16.52 Original:/ Redirect:http://www.cnn.com/ ---------------------- www.hotmail.com 207.68.173.245 Original:/ Redirect:http://loginnet.passport.com/login.srf?id=2&svc=mail&cbid=24325&msppjph=1&tw=0&fs=1&fsa=1&fsat=1296000&lc=1033&_ lang=EN ---------------------- 7. Reordering Test ================== This test makes use of the tbit tool. It simply runs a Reno test on a URL list using a given MSS. For each URL in the list, the test outcome indicates if a reordering took place when the Reno test was executed. A "Reordering vs. MSS" test would consists of running this test several times, varying the MSS. ./run-tbit-test-0.1.py testList100 Reordering 128 1 NewReordering_Test.txt 0 ===================== Summary: mss: 128 Reordering: 4 Total: 10 ===================== www.ebay.com 66.135.192.88 r = 0 www.passport.net 65.54.230.254 r = 0 www.google.com 216.239.53.99 r = 0 www.mapquest.com 64.12.37.89 r = 1 www.tiscali.it 195.130.227.63 r = 1 www.icra.org 66.221.165.93 r = 1 www.free.fr 213.228.0.42 r = 1 www.drudgereport.com 66.28.209.210 r = 0 www.microsoft.com 207.46.249.190 ERROR: Could not establish contact after 2 retries www.yahoo.co.jp 211.14.15.5 ####no data received for 4.000000 seconds 8. MinMSS Test ============== This test determines for each URL in the list, the minimum MSS accepted by the server. Only multiples of 2 are checked. To run it, not that the MSS argument should be one since the test will start from that values and it will go up searching for the minumum MSS: ./run-tbit-test-0.1.py url_list.txt MinMSS 1 1 NewMinMSS_Test.txt 0 ======================================== MSS 64: 2 MSS 128: 2 Errors: 2 Total: 6 ======================================== www.sohu.com 61.135.150.75 MSS = 64 accepted by server www.nate.com 203.226.255.16 MSS = 128 accepted by server www.sayclub.com 211.233.85.85 MSS = 64 accepted by server www.passport.net 65.54.230.254 MSS = 128 accepted by server www.yahoo.co.jp 211.14.15.5 ERROR: ####no data received for 4.000000 seconds www.google.com 216.239.53.99 ERROR: ####no data received for 4.000000 seconds 9. InitWin Test: =============== This test checks the minimum window size accepted by the server. To tun it: ./run-tbit-test-0.1.py top10_large.txt InitWin 128 1 ResultsInitWin_Test.txt 0 ================================================= Initial window in packets, with '128'-byte packets ================================================= Initial window Count 1 2 2 2 6 1 ----------------------------------------- Errors: 1 Total: 6 ----------------------------------------- www.sohu.com 61.135.150.75 1 www.nate.com 203.226.255.16 1 www.sayclub.com 211.233.85.85 2 www.passport.net 65.54.230.254 2 www.google.com 216.239.53.99 6 ======================================== 10. WinHalf Test ================ To run it: ./run-tbit-test-0.1.py top10_large.txt WinHalf 128 1 ResultsWinHalf_Test.txt 0 ======================================== Summary: Errors: 2 Total: 6 ======================================== www.nate.com 203.226.255.16 Window halved. 640 bytes outstanding www.sayclub.com 211.233.85.85 Window halved. 256 bytes outstanding www.passport.net 65.54.230.254 Window halved. 544 bytes outstanding www.google.com 216.239.53.99 Window halved. 512 bytes outstanding www.sohu.com 61.135.150.75 ERROR: Reordering before test completion www.yahoo.co.jp 211.14.15.5 ERROR: no data received for 4.000000 seconds 11. Timestamp test: ================== To run it: ./run-tbit-test-0.1.py top10_large.txt Timestamp 128 1 ResultsTimestamp_Test.txt 0 ======================== Summary: TS 0: 2 TS 1: 4 Errors: 0 Total: 6 ======================== www.sohu.com 61.135.150.75 Timestamp =0 www.nate.com 203.226.255.16 Timestamp =1 www.sayclub.com 211.233.85.85 Timestamp =1 www.passport.net 65.54.230.254 Timestamp =1 www.yahoo.co.jp 211.14.15.5 Timestamp =1 www.google.com 216.239.53.99 Timestamp =0 12. TahoeNoFR Test (Tahoe No Fast retransmit) ============================================= To run it: ./run-tbit-test-0.1.py top10_large.txt TahoeNoFR 128 1 ResultsTahoNoFR_Test.txt 0 ======================== Summary: Correct: 5 Errors: 1 Total: 6 ======================== www.sohu.com 61.135.150.75 rx=1 to=0 www.nate.com 203.226.255.16 rx=1 to=0 www.sayclub.com 211.233.85.85 rx=1 to=0 www.passport.net 65.54.230.254 rx=1 to=0 www.google.com 216.239.53.99 rx=1 to=0 www.yahoo.co.jp 211.14.15.5 ERROR: no data received for 4.000000 seconds 13. Sack Receiver Test (SackRcvr) ================================= To run it: ./run-tbit-test-0.1.py top10_large.txt SackRcvr 128 1 ResultsSackRcvr_Test.txt 0 ======================================== Summary: sack: 2 no sack: 4 Errors: 0 Total: 6 ======================================== www.nate.com 203.226.255.16 sack okay www.passport.net 65.54.230.254 sack okay ~ 14. TimeWait Test: ================= To run it: ./run-tbit-test-0.1.py top10_large.txt TimeWait 128 1 ResultsTimeWait_Test.txt 0 ======================================== Summary: correct: 2 errors: 4 Total: 6 ======================================== www.sayclub.com 211.233.85.85 #### 8.143952 < time wait < 10.143946 www.passport.net 65.54.230.254 #### -1.989935 < time wait < 0.010060 www.sohu.com 61.135.150.75 NO_DATA_RCVD www.nate.com 203.226.255.16 NO_DATA_RCVD www.yahoo.co.jp 211.14.15.5 NO_DATA_RCVD www.google.com 216.239.53.99 EARLY_RST 15. DeAck Test: ============== To run it: ./run-tbit-test-0.1.py top10_large.txt DelAck 128 1 ResultsDelAck_Test.txt 0 ======================================== Summary: Correct: 6 Errors: 0 Total: 6 ======================================== www.sohu.com 61.135.150.75 delack = 0 www.nate.com 203.226.255.16 delack = 0 www.sayclub.com 211.233.85.85 delack = 0 www.passport.net 65.54.230.254 delack = 0 www.yahoo.co.jp 211.14.15.5 delack = 1 www.google.com 216.239.53.99 delack = 0