An overview of testing issues in IDS

The paper gives a list of possible measurements which could be obtained as a result of testing the IDS.

Centres that are currently spending efforts on testing the IDS include the following:

1. University of California at DAVIS
2. IBM Zurich
3. MIT Lincoln Laboratory
4. Air Force Research Laboratory
5. MITRE
6. Neohapsis/ Network-Computing
7. The NSS Group
8. Network World Fusion

List of challenges as quoted include the following

1. Difficulties in Collecting Attack Scripts and Victim Software
2. Differing Requirements for Testing Signature Based vs. Anomaly Based
IDSs
3. Differing Requirements for Testing Network Based vs. Host Based IDSs
4. Four Approaches to Using Background Traffic in IDS Tests
- Testing using no background traffic/logs
- Testing using real traffic/logs
- Testing using sanitized traffic/logs
- Testing by generating traffic on a testbed network

Recommendations

1. Shared Datasets
2. Attack Traces
3. Cleansing Real Data
4. Sensor and Detector Alert Datasets
5. Real-life performance metrics
6. New Technologies