Internet Research Needs Better Models: Why Models Matter
Why Models Matter:
S. Gorinsky and H. Vin,
Extended Analysis of Binary Adjustment Algorithms,
Technical Report TR2002-39, Department of Computer
The University of Texas at Austin, August 2002.
"It is still common to use Chiu-Jain model for comparison of
binary adjustment algorithms. This paper argues against such
practice... Chiu-Jain model is not suitable for trustworthy
conclusions about properties of an adjustment algorithm."
- W. Willinger, R. Govindan, S. Jamin, V. Paxson, and S. Shenker.
Scaling Phenomena in the Internet: Critically Examining Criticality,
Proc. Nat. Acad. Sci., 99, suppl. 1, pp. 2573--2580, February 19, 2002.
"We bring to bear a simple validation framework that aims at testing
whether a proposed model is merely evocative, in that it can reproduce
the phenomenon of interest but does not necessarily capture and
incorporate the true underlying cause, or indeed explanatory,
in that it also captures the causal mechanisms."
Y. Joo, V. Ribeiro, A. Feldmann, A. C. Gilbert, and W. Willinger,
TCP/IP traffic dynamics and network performance: A lesson in workload
modeling, flow control, and trace-driven simulations.
CCR, April 2001.
"Depending on the underlying assumptions about the inherent nature of
the dynamics of network traffic, very different conclusions can be
derived for a number of well-studied and apparently well-understood
problems in the area of performance evaluation."
- Reiner Ludwig, Almudena Konrad, Anthony D. Joseph, Randy H. Katz,
Optimizing the End-to-End Performance of Reliable Flows over Wireless
Proceedings of ACM MOBICOM 1999.
"An unrealistic error model of the wireless channel can completely
change the results of a performance analysis."
An Opinionated View of the Current State of IP Differentiated Services,
September 1999. In particular,
Part II: Issues in Evaluating Differentiated Services.
"Beware... sweeping conclusions based on very simple traffic models and
John Heidemann, Nirupama Bulusu, Jeremy Elson, Chalermek Intanagonwiwat,
Kun-chan Lan, Ya Xu, Wei Ye, Deborah Estrin, and Ramesh Govindan,
Effects of Detail in Wireless Network Simulation
SCS Multiconference on Distributed Simulation, January 2001.
"Too little detail can produce simulations that are
misleading or incorrect,
but adding detail ... can distract from the research
problem at hand."
Dangers in other Domains:
The Reckoning: Agency's '04 Rule Let Banks Pile Up New Debt,
October 2, 2008, New York Times.
"The agency ... decided to rely on the firms’ own computer models for
determining the riskiness of investments, essentially outsourcing the job
of monitoring risk to the banks themselves."
The Audio Slide Show is particularly interesting.
In Modeling Risk, the Human Factor Was Left Out,
November 4, 2008, New York Times.
"Today’s economic turmoil, it seems, is an implicit indictment of the
arcane field of financial engineering — a blend of mathematics,
statistics and computing. Its practitioners devised not only the exotic,
mortgage-backed securities that proved so troublesome, but also the
mathematical models of risk that suggested these securities were safe."
Wall Street’s Math Wizards Forgot a Few Variables
September 12, 2009, New York Times.
"In the aftermath of the great meltdown of 2008, Wall Street’s quants have
been cast as the financial engineers of profit-driven innovation run
But the real failure, according to finance experts and economists, was in
the quants’ mathematical models of risk that suggested the arcane stuff
was safe.... The risk models proved myopic, they say, because they were
too simple-minded. They focused mainly on figures like the expected
returns and the default risk of financial instruments. What they didn’t
sufficiently take into account was human behavior, specifically the
potential for widespread panic."
Proposed additions to this page can be sent to
Thanks for Andrei Gurtov for pointers to add to this page.
This material is based in part upon work supported by the National Science
Foundation under Grant No. 0230921.
Any opinions, findings, and conclusions or recommendations expressed
in this material are those of the author(s) and do not necessarily
reflect the views of the National Science Foundation.
Last modified: November 2008