Homework #6 - NIDS Evaluation - Due Monday Feb 10, 1PM

Read the paper Testing Intrusion Detection Systems: A Critique of the 1998 and 1999 DARPA Off-line Intrusion Detection System Evaluation as Performed by Lincoln Laboratory, John McHugh, ACM Transactions on Information and System Security, 3(4). November, 2000.

Note: this paper is a rarity in security systems work, and a somewhat unusual read. Ideally, I would have also assigned reading the Lippman et al. DISCEX '00 paper that forms the heart of the discussion here, though that would have been quite a load. If you're motivated to go ahead and read that too (great!), you should probably read it first. In any case, for following McHugh's discussion you may find it useful to refer to it.

It's easy when reading McHugh's critique to think that the Lippman et al. paper was obviously flawed. It's valuable to appreciate that this was not in fact at all clear at the time. So, as you read McHugh's paper, picture (1) that a major, influential study evaluating IDSs has just been released, and there's a lot of impressed buzz in the air about it, and (2) that the vast majority of researchers who read or heard about that study didn't see it as having any particular shortcomings, just that it reflected an impressive amount of hard work distilled into a bunch of valuable evaluation results.

In addition, the McHugh paper is on the long side (and you may find that some of it reads a bit slowly, though personally I find much of it interesting and sometimes fascinating), so you should budget a bit more time for reading it. You can skip section 6.2 and section 7 (though if you find the topic particularly interesting, section 7 is worth reading). Section 6.3 is subtle and somewhat hard to follow, but it's also an important set of considerations, so worth seeing if you can extract the main points from it. Don't worry with this paper about any fine-grained details.

Briefly write up your views of:

1. What are the main contributions of this paper?

2. What parts of the paper do you find unclear? (optional)

3. What parts of the paper are questionable? (That is, you think a conclusion may be wrong, or you have concerns with criticisms of the author's, or with approaches his proposes.)

4. Propose a particular step that the field of security research could take to strengthen the work that the discipline produces. For your discussion you can stay focussed on intrusion detection if you wish, but you do not have to. You can instead address other areas of security research (needn't be related to networking or the Internet), and it's fine to look at specific, focussed issues, or quite broadly at classes of problems, or something in between.

   Your step can align with some of McHugh's points providing you add to his existing discussion in a thoughtful fashion. If this is the case, delineate what you bring in addition to his framing.

   Be sure to clearly summarize any issues raised by the step that you discuss - what would be the barriers to the field adopting the approach? (Or argue why it really is a measure that the field can readily incorporate.)

   Note: I realize that students in the class might not have had significant exposure to thinking about these quality-of-research meta-issues. I'm not looking for you to go off and study the topic (though it might be helpful if you do a bit of informal investigation as a way to get ideas). I particularly want your thoughtful analysis given whatever assumptions you have about how the field works. If you're unsure that the assumptions are warranted, go ahead and state them explicitly.

   Finally, I'm only looking for a couple of paragraphs or so of text. If you want to write a bit more, that's okay, but not required.