Anomaly Detection
-Misuse
-System Calls
-Adaptive
-Feature Selection
-Network Based
-Host Based
-Behavior Based
-Cooperative
-Cost Sensitive
The authors as quoted in my previous blog have collated relevant papers related to the classification mentioned above. This could act as a good starting point to understand more about the different anomaly based systems
Tuesday, December 15, 2009
Intrusion detection systems and models
Joseph S. Sherif, Tommy G. Dearmond, California Institute of Technology
Mahoney, M., “Computer Security: A Survey of Attacks and Defenses” Hyperlink docshow.net/ids.htm, 2000.
http://cs.fit.edu/~mmahoney/ids.html
According to Mohoney, there exists at least six types of attacks to the system
1. Worms - self replicating programs that spread across a network.
The three security flaws that the worms exploited were
a. a backdoor in the sendmail server program that allowed clients to execute remote commands on the server
b. a buffer overflow vulnerability in the fingerd server
c. weak passwords
2. Viruses - programs that replicate when a user performs some action such as running a program.
types discussed in this paper include the following
a. Boot sector infectors
b. File infectors
c. Email viruses
3. Server attacks - a client exploits a bug in the server to cause it to perform some unintended action.
Examples,
a. Microsoft IIS 4.0 web server
b. Cold Fusion web server
c. lpd attack
d. Sendmail buffer overflow
e. rpc.statd buffer overflow
4. Client attacks - a server exploits a bug in a client to cause it to perform some unintended action.
a. Microsoft Office 2000 UA Control
b. Microsoft Outlook
5. Network attacks (denial of service) - a remote attacker exploits a bug in the network software or weakness in the protocol to cause a server, router, or network to fail.
a. Winnuke
b. IRDP spoofing
c. Teardrop
d. Land
e. Ping of death
f. Smurf
g. Vengeance
h. SYN flood
i. Local network attacks
j. SNMP attacks
6. Root attacks - a user on a multiuser operating system obtains the priveliges of another user (usually root) by either
- Obtaining the other user's password, or
- Bypassing controls that restrict access.
NT attacks - NT password weaknesses
UNIX attacks - Shell script attacks, Dynamic library attacks, Directory tree escapes, Symbolic links in /tmp, Console attacks
Password Capture - Sniffer attacks, Weak encryption, Trojan attacks, Backdoor passwords, Stored passwords, Hardware key attacks
Mahoney, M., “Computer Security: A Survey of Attacks and Defenses” Hyperlink docshow.net/ids.htm, 2000.
http://cs.fit.edu/~mmahoney/ids.html
According to Mohoney, there exists at least six types of attacks to the system
1. Worms - self replicating programs that spread across a network.
The three security flaws that the worms exploited were
a. a backdoor in the sendmail server program that allowed clients to execute remote commands on the server
b. a buffer overflow vulnerability in the fingerd server
c. weak passwords
2. Viruses - programs that replicate when a user performs some action such as running a program.
types discussed in this paper include the following
a. Boot sector infectors
b. File infectors
c. Email viruses
3. Server attacks - a client exploits a bug in the server to cause it to perform some unintended action.
Examples,
a. Microsoft IIS 4.0 web server
b. Cold Fusion web server
c. lpd attack
d. Sendmail buffer overflow
e. rpc.statd buffer overflow
4. Client attacks - a server exploits a bug in a client to cause it to perform some unintended action.
a. Microsoft Office 2000 UA Control
b. Microsoft Outlook
5. Network attacks (denial of service) - a remote attacker exploits a bug in the network software or weakness in the protocol to cause a server, router, or network to fail.
a. Winnuke
b. IRDP spoofing
c. Teardrop
d. Land
e. Ping of death
f. Smurf
g. Vengeance
h. SYN flood
i. Local network attacks
j. SNMP attacks
6. Root attacks - a user on a multiuser operating system obtains the priveliges of another user (usually root) by either
- Obtaining the other user's password, or
- Bypassing controls that restrict access.
NT attacks - NT password weaknesses
UNIX attacks - Shell script attacks, Dynamic library attacks, Directory tree escapes, Symbolic links in /tmp, Console attacks
Password Capture - Sniffer attacks, Weak encryption, Trojan attacks, Backdoor passwords, Stored passwords, Hardware key attacks
Monday, December 14, 2009
Techniques to detect anomalies (IDS)
Susheela Sarasamma, Julie Huff, "Anomaly based techniques for intrusion detection systems", Northrop Grumman Mission Systems
In this presentation, the authors provide information on the various anomaly based techniques which can be used for the IDS.
General anomaly based techniques include, statistical, neural network and machine learning.
The authors have specifically discussed a couple of anomaly detection techniques for an IDS which include the following
1. A novel anomaly detection technique using Kohonen network
2. Conclusions on the test conducted on multi-level k map
Kinds of anomalies to be considered as outliers include the following
1. outlier detection
2. novelty detection
3. noice detection
4. deviation detection
5. exception mining
* Now suppose we have to come up with efficient testing techniques for anomaly based intrusion detection systems there are set of things which we might want to understand
- study the different anomaly based intrusion detection systems. As the test methodology might vary depending on the technique which has been adopted.
- research on the platform which might be required for performing the testing for such a system.
- in addition we might have to come to a decision or rather the focus of the research
- we might also want to understand the typical attacks in the case of the anomaly based intrusions
- audit trails how are they being used for testing purposes
- if we need to compose of a testing methodology, then how should the evaluation be done for it
In this presentation, the authors provide information on the various anomaly based techniques which can be used for the IDS.
General anomaly based techniques include, statistical, neural network and machine learning.
The authors have specifically discussed a couple of anomaly detection techniques for an IDS which include the following
1. A novel anomaly detection technique using Kohonen network
2. Conclusions on the test conducted on multi-level k map
Kinds of anomalies to be considered as outliers include the following
1. outlier detection
2. novelty detection
3. noice detection
4. deviation detection
5. exception mining
* Now suppose we have to come up with efficient testing techniques for anomaly based intrusion detection systems there are set of things which we might want to understand
- study the different anomaly based intrusion detection systems. As the test methodology might vary depending on the technique which has been adopted.
- research on the platform which might be required for performing the testing for such a system.
- in addition we might have to come to a decision or rather the focus of the research
- we might also want to understand the typical attacks in the case of the anomaly based intrusions
- audit trails how are they being used for testing purposes
- if we need to compose of a testing methodology, then how should the evaluation be done for it
Friday, December 11, 2009
Road Map
In the last quarter report, there were two potential streams identify for research
1. Testing of anomaly based intrusion detection systems
2. Modeling intrusion detection systems
Proceeding ahead with the testing of anomaly based IDS, we intended to do the following
1. Understand the anomaly based IDS models
2. How currently the IDS systems are tested. i.e. testing methodology, test scripts generation etc
3. How audit records are used to understand any intrusions in the system etc.
Currently, based on the information obtained. There are not many research papers supporting the testing of IDS systems.
When we are not getting sufficient information, the thought on what could have led to scenario is a question that comes up. In addition most of the testing related research have been carried out a few years ago, implying there is no currently existing research based on the literature review that has been obtained.
Given the situation what could be the steps ahead.
1. Irrespective of the availability of specific information, understanding the basic model of anomaly based IDS would help us in deriving better testing techniques.
2. Improvement of the test scripts could be an option which need to be worked on.
3. In addition, there was also some work related to the software platform for testing IDS undertaken by Prof. Biswanath Mukherjee and his associates.
4. The model itself could be looked at. One fact that should be considered is that anomaly based IDS does have the ability to raise an alarm for all possible situations which it was not trained before, hence there is greater probability of false alarms as well. Hence we could spend some effort here to understand how this could be avoided.
Though we have signature and anomaly based IDS. Current industry preference is signature since it is trained to identify fixed anomalies and it does most of the time. While in the case of anomaly based IDS, it is dynamic, but there is a probability if the system is trained to accept an intrusion as a normal behavior it cannot differentiate.
1. Testing of anomaly based intrusion detection systems
2. Modeling intrusion detection systems
Proceeding ahead with the testing of anomaly based IDS, we intended to do the following
1. Understand the anomaly based IDS models
2. How currently the IDS systems are tested. i.e. testing methodology, test scripts generation etc
3. How audit records are used to understand any intrusions in the system etc.
Currently, based on the information obtained. There are not many research papers supporting the testing of IDS systems.
When we are not getting sufficient information, the thought on what could have led to scenario is a question that comes up. In addition most of the testing related research have been carried out a few years ago, implying there is no currently existing research based on the literature review that has been obtained.
Given the situation what could be the steps ahead.
1. Irrespective of the availability of specific information, understanding the basic model of anomaly based IDS would help us in deriving better testing techniques.
2. Improvement of the test scripts could be an option which need to be worked on.
3. In addition, there was also some work related to the software platform for testing IDS undertaken by Prof. Biswanath Mukherjee and his associates.
4. The model itself could be looked at. One fact that should be considered is that anomaly based IDS does have the ability to raise an alarm for all possible situations which it was not trained before, hence there is greater probability of false alarms as well. Hence we could spend some effort here to understand how this could be avoided.
Though we have signature and anomaly based IDS. Current industry preference is signature since it is trained to identify fixed anomalies and it does most of the time. While in the case of anomaly based IDS, it is dynamic, but there is a probability if the system is trained to accept an intrusion as a normal behavior it cannot differentiate.
Wednesday, December 9, 2009
General definition of anomaly based intrusion detection
An Anomaly-Based Intrusion Detection System, is a system for detecting computer intrusions and misuse by monitoring system activity and classifying it as either normal or anomalous. The classification is based on heuristics or rules, rather than patterns or signatures, and will detect any type of misuse that falls out of normal system operation. This is as opposed to signature based systems which can only detect attacks for which a signature has previously been created.
In order to determine what is attack traffic, the system must be taught to recognise normal system activity. This can be accomplished in several ways, most often with artificial intelligence type techniques. Systems using neural networks have been used to great effect. Another method is to define what normal usage of the system comprises using a strict mathematical model, and flag any deviation from this as an attack. This is known as strict anomaly detection.
Source : - http://en.wikipedia.org/wiki/Anomaly-based_intrusion_detection_system
In order to determine what is attack traffic, the system must be taught to recognise normal system activity. This can be accomplished in several ways, most often with artificial intelligence type techniques. Systems using neural networks have been used to great effect. Another method is to define what normal usage of the system comprises using a strict mathematical model, and flag any deviation from this as an attack. This is known as strict anomaly detection.
Source : - http://en.wikipedia.org/wiki/Anomaly-based_intrusion_detection_system
IDS Testing Methodology
Nicolas J Puketza et al, "A methodology for testing intrusion detection systems"
In this paper, the authors try to test the performance objectives of the IDS systems
Precondition - IDS systems should be installed and configured appropriately.
They have considered the basic testing procedure as described below and brought in variations in it
1. create or select a set of test scripts
2. establish the desired conditions in the computing environment
3. start the IDS
4. run the test scripts
5. analyze the IDS's output
The different IDS Tests considered include the following
1. Intrusion Identification tests
- Basic detection tests
a. create a set of intrusion scripts
b. as much as possible, eliminate unrelated computing acitvity in testing environment
c. start the IDS
d. run the intrusion scripts
Another technique, test results include a number associting it to warning based on the level of abnormal behavior. This would assist the testers in better classification of the abnormal behavior.
- Normal Users test - Here instead of the intruder scripts, normal user scripts are introduced resulting in the understanding on how often normal scenarios are flagged as intrusions.
2. Resource Usage tests
This is done to understand the resource consumption of the IDS systems. The main test is the disk space. The procedure adopted is as follows:
1. eliminate unrelated activity in the test environment
2. start the IDS
3. run the test script for a measured period of time (e.g., one hour) and
4. calculate the total disk space used by the IDS to record the session associated with the script.
Evaluation:
In this paper, the authors try to test the performance objectives of the IDS systems
Precondition - IDS systems should be installed and configured appropriately.
They have considered the basic testing procedure as described below and brought in variations in it
1. create or select a set of test scripts
2. establish the desired conditions in the computing environment
3. start the IDS
4. run the test scripts
5. analyze the IDS's output
The different IDS Tests considered include the following
1. Intrusion Identification tests
- Basic detection tests
a. create a set of intrusion scripts
b. as much as possible, eliminate unrelated computing acitvity in testing environment
c. start the IDS
d. run the intrusion scripts
Another technique, test results include a number associting it to warning based on the level of abnormal behavior. This would assist the testers in better classification of the abnormal behavior.
- Normal Users test - Here instead of the intruder scripts, normal user scripts are introduced resulting in the understanding on how often normal scenarios are flagged as intrusions.
2. Resource Usage tests
This is done to understand the resource consumption of the IDS systems. The main test is the disk space. The procedure adopted is as follows:
1. eliminate unrelated activity in the test environment
2. start the IDS
3. run the test script for a measured period of time (e.g., one hour) and
4. calculate the total disk space used by the IDS to record the session associated with the script.
Evaluation:
Friday, December 4, 2009
An intrusion detection model
Denning, D. E. 1987. An intrusion detection model. IEEE Transactions of Software Engineer-
ing 13, 2, 222 - 232.
The model denning arrived at was for a generic purpose which could cater to any kind of environment and system vulnerability and the type of intrusion. He believed that most of the intrusions could be identified from the audit records that is created by the intrusion detection system. He proposed a model in lines of the IDES and claimed it to work well.
The model proposed by him was expected to cater to the following list of intrusions
1. attempted break-in
2. masquerading or successful break-in
3. penetration by legitimate user
4. leakage by legitimate user
5. Inference by legitimate user
6. Trojan horse
7. Virus
8. Denial of service
Model proposed by him:
Since his model was a general purpose one, he called it as an intrusion-detection expert system (IDES). The six main components of his model includes the following:
subjects
objects
audit records
profiles
anomaly records
activity rules
Denning also describes various metrics and statistical models using the audit records could be analyzed for detecting anomalies
The metrics defined are:
1. event counter - number of audit records satisfying some property occurring during a period
2. interval timer - length of time between two related events
3. resource measure - quantity of resources consumed by some action during a period as specified in the Resource-Usage field of the audit records.
Statistical models proposed include the following:
1. Operational model
2. Mean and standard deviation model
3. Multivariate model
4. Markov Process model
5. Time series model
In this paper, in addition to defining metrics and models, Denning also elaborates on the various profiles of data that could be generated for different sessions.
Thus this data could be used for designing intrusion detection system test cases
ing 13, 2, 222 - 232.
The model denning arrived at was for a generic purpose which could cater to any kind of environment and system vulnerability and the type of intrusion. He believed that most of the intrusions could be identified from the audit records that is created by the intrusion detection system. He proposed a model in lines of the IDES and claimed it to work well.
The model proposed by him was expected to cater to the following list of intrusions
1. attempted break-in
2. masquerading or successful break-in
3. penetration by legitimate user
4. leakage by legitimate user
5. Inference by legitimate user
6. Trojan horse
7. Virus
8. Denial of service
Model proposed by him:
Since his model was a general purpose one, he called it as an intrusion-detection expert system (IDES). The six main components of his model includes the following:
subjects
objects
audit records
profiles
anomaly records
activity rules
Denning also describes various metrics and statistical models using the audit records could be analyzed for detecting anomalies
The metrics defined are:
1. event counter - number of audit records satisfying some property occurring during a period
2. interval timer - length of time between two related events
3. resource measure - quantity of resources consumed by some action during a period as specified in the Resource-Usage field of the audit records.
Statistical models proposed include the following:
1. Operational model
2. Mean and standard deviation model
3. Multivariate model
4. Markov Process model
5. Time series model
In this paper, in addition to defining metrics and models, Denning also elaborates on the various profiles of data that could be generated for different sessions.
Thus this data could be used for designing intrusion detection system test cases
Follow - up survey - focus
The authors from the University of Minnesotta as explained in my previous blog had conducted an extensive study on the anomaly detection in different domains. The domain of interest to us are the intrusion detection systems, where-in they try to focus on the intrusions based on anomaly's in computer systems.
According to the authors the complexity with respect to anomaly detection in computer intrusion systems is the volume of data. In general labeled data corresponding to the expected behavior is available while those beyond expected are large in number and difficult to collate. In addition we understand that behavior beyond the normal and expected are considered to be an anomaly.
Denning in his paper classifies the types of intrusion detection systems to be host and network based.
(Denning, D. E. 1987. An intrusion detection model. IEEE Transactions of Software Engineer-
ing 13, 2, 222{232.)
According to the authors the complexity with respect to anomaly detection in computer intrusion systems is the volume of data. In general labeled data corresponding to the expected behavior is available while those beyond expected are large in number and difficult to collate. In addition we understand that behavior beyond the normal and expected are considered to be an anomaly.
Denning in his paper classifies the types of intrusion detection systems to be host and network based.
(Denning, D. E. 1987. An intrusion detection model. IEEE Transactions of Software Engineer-
ing 13, 2, 222{232.)
Anamoly detection
This is based on the technical report consisting of a survey conducted by Varun Chandola, Arindam Banerjee and Vipin Kumar from University of Minnessota.
They talk in detail on what anamolies are and how they could be identified by using different techniques.
They discuss about anomaly detection in terms of intrusions from the following perspective
1. Host based intrusion detection systems
2. Network intrusion detection systems
Apart from the intrusion detection systems, they also delve into fraud detections like credit card, mobile, insurance claim frauds and insider trading detections.
The different techniques that they discuss on to detect anomaly include the following
1. Neural network based
2. Bayesian network based
3. Support vector machine based
4. rule based
The authors also talk about the additional anomaly detection techniques from a mathematical perspective as well.
Among the research objectives that we have in place, one of them is to identify the testing technique that could be adopted to detect anomaly based intrusions.
Thus the technical report defined by the authors could act as an input to the way test data is being prepared for identifying intrusions. Effort needs to be added in order to achieve this. At the same time, this does sound to be a source to be taken into consideration.
The authors process of writing down the advantages and disadvantages of each technique they describe which could be used for anomaly detection could also add into the understanding.
One drawback which the authors quote is that, they believe the data considered for describing the various techniques in the report is not standard. It seems to be varying across. So the results that they have achieved might not give us a consistent understanding. Hence one of their future work would be to unify all the assumptions that they have considered.
They talk in detail on what anamolies are and how they could be identified by using different techniques.
They discuss about anomaly detection in terms of intrusions from the following perspective
1. Host based intrusion detection systems
2. Network intrusion detection systems
Apart from the intrusion detection systems, they also delve into fraud detections like credit card, mobile, insurance claim frauds and insider trading detections.
The different techniques that they discuss on to detect anomaly include the following
1. Neural network based
2. Bayesian network based
3. Support vector machine based
4. rule based
The authors also talk about the additional anomaly detection techniques from a mathematical perspective as well.
Among the research objectives that we have in place, one of them is to identify the testing technique that could be adopted to detect anomaly based intrusions.
Thus the technical report defined by the authors could act as an input to the way test data is being prepared for identifying intrusions. Effort needs to be added in order to achieve this. At the same time, this does sound to be a source to be taken into consideration.
The authors process of writing down the advantages and disadvantages of each technique they describe which could be used for anomaly detection could also add into the understanding.
One drawback which the authors quote is that, they believe the data considered for describing the various techniques in the report is not standard. It seems to be varying across. So the results that they have achieved might not give us a consistent understanding. Hence one of their future work would be to unify all the assumptions that they have considered.
Subscribe to:
Posts (Atom)
