Anomaly-Based Detection

Anomaly-based detection (see Figure 11-5) protects against unknown threats. An “anomaly” is anything that is abnormal. If any traffic is found to be abnormal from the baseline, then an alert is triggered by the IDS suspected of an intrusion. IDPS first creates a baseline profile that represents the normal behavior of the traffic. The baseline

profile is created by allowing the IDS system to learn the traffic over a period of time so that IDPS can study the traffic behavior during peak hours, non-peak hours, night hours, early hours of business, and as per your organizational network behavior. After learning, the traffic collected over a period of time is statistically studied and a baseline profile is created. Once the IDS is changed from learning mode to detection/prevention mode, it starts comparing the regular traffic with the profile that was created, and if any abnormality or deviation from the baseline profile is found, then an alert is triggered cautioning the possible intrusion or the intrusion is prevented, if it is configured for prevention mode. Customized profiles can also be created for specific traffic behavior such as the number of e-mails sent by a user and user access attempts.

Figure 11-5. Anomaly Detection

What is an anomaly? Here are some examples of anomalous behavior:

• Too many Telnet sessions on a single day

• HTTP traffic on a non-standard port

• Heavy SNMP traffic

For effective intrusion detection, IDS must have a robust baseline profile which covers the entire organization's network and its segments. It should cover normal traffic behavior of all the components which are aimed to be covered by the Intrusion Detection and Prevention System. Baseline profile can vary in complexity from a simple to a comprehensive content, depending on the characteristics of the network and its components. For example, a profile could include the following data:

• A web application logged in remotely by a specific set of users

• An application which has a specific acceptable password design

• Traffic during the peak hours and non-peak hours as defined by the organization

• Connectivity pattern from an external partner network

• Connecting from a set of mobile devices to the database server

The challenge of the anomaly-based detection method is creating an effective profile. The initial profile, sometimes referred to as the “training profile,” is generated by studying the traffic pattern over a period of time.

The time factor may vary from organization to organization. It could be a few hours to a few days. Once this profile is created, IDS is put into detection mode and every time there is a packet, a pattern is matched against the baseline profile. This baseline can be changed as and when required based on the traffic behavior. If any malicious activity

already exists from the beginning, while building the baseline profile, this activity will also become part of the baseline profile and such kind of malicious activity will thereby go undetected. Hence, anomaly detection does not necessarily detect each and every unknown attack. The limitation is based on the baseline profile you create. However, a system administrator was alerted by IDS to Microsoft DCOM DOS vulnerability without having a specific signature.

Types of Anomaly

Anomaly-based Intrusion Detection and Prevention Systems (IDPS) protect anomaly caused due to violation of protocols, and application payload. It also protects against Denial of Service attacks and Buffer overflow attacks.

Protocol Anomaly

Protocol anomaly refers to the anomaly in the protocol format and protocol behavior with respect to the Internet standards and specifications. There are many aspects in TCP and IP protocol that needs to be monitored, for example, different flags, SYN, ACK, and FIN, and their combination in TCP header and the reserved flags of IP header. The way IP fragmentation and reassembly is implemented is as per the standards. If this anomaly is not detected by the IDS, the end host may not process these unconventional packets and this may lead to the crash of the system. At the application level, IDPS must be able to do deep protocol parsing to understand application level protocol anomaly. It also requires a deep understanding of the application semantics in order to detect application payload anomaly.

Some other examples include:

• Unusual TCP segmentation and TCP flags combination

• Corrupt checksum

• Incorrect IP fragmentation and reassembly flags

• Erroneous source and destination port numbers

• Illegal protocol commands and its usage

• Running protocol on non-standard port

• Presence of shellcode in unexpected application protocol fields

• Misuse of protocol and protocol services

Statistical Anomaly Detection – Statistical DDoS

Denial of Service (DoS) and Distributed Denial of Service (DDoS) results in a burst of traffic on the network which is not normal. In order to overcome this kind of attack, baseline profiles are created on the normal flow of traffic, as described earlier, based on statistical modeling, such as Naïve Bayes, to determine anomalous packets on the

network. While learning the network traffic behavior, the function of statistical modeling is to compute the probability score for each of the data packets that is considered as normal traffic. The scores are computed based on the sampled data over a period of time and stored in a baseline profile. A threshold is set for each set of protocols and users. When the IDS is in monitoring mode, the data is checked against the baseline and the threshold. Whenever an anomalous packet is discovered and the scores are above threshold, then an alert is triggered. The reporting process will report only when the data is found to be anomalous for a sufficient period of time; otherwise, the IDPS will simply ignore the trace. Threshold can be set for different profiles, for different protocols, and for different users.3,4

When IDPS is in monitoring mode, if there is anything that is abnormal to the baseline, the system will generate an alert. But, it may turn out that the analysis results confirm that the alert found was a false positive. As a security administrator, one can expect a similar kind of traffic behavior appearing every other day and to minimize the spending of the same effort repeatedly, a threshold can be set so that anything within this threshold, the traffic is still considered normal and anything which exceeds this threshold is considered an intrusion. Thresholds can also be set for a set of users, or set of protocols.

Profiles based on the statistical measures can detect some of the DoS anomalies based on longand short-term distributions or bursts of peak (i.e., high) traffic. The normal baseline profiles are continuously being learned while the system is in detection mode and the baseline is re-created to adjust the changing traffic pattern to avoid

false positives.

By creating different profiles, DoS attacks can be prevented. For example, for each of the DoS attacks, a profile can be created. Knowing the pattern of SYN flood, a SYN flood DoS profile can be created. Whenever there is SYN flood traffic on the network, the IDS sensors can detect the SYN flood attack by comparing the network traffic with the SYN flood profile thus alerting a SYN flood attack. Similarly, UDP flood profile, TCP data segment profile, or ICMP flood profiles can be detected and alerted.

Though anomaly-based IDS has an advantage of detecting unknown attacks, defining rules for it is difficult. Each protocol must be analyzed, processed, and compared with a baseline. Any customized protocol makes it even more challenging.

Another major pitfall of anomaly detection is defining normal traffic while creating a baseline. Normal traffic has to be clean and should not have any malicious activity in the network. In case of any malicious activity during the learning process, then the baseline profile learns this and makes it harder to detect this intrusion or it may not even detect intrusion of such malicious traffic. For example, reconnaissance attacks such as fingerprint or directory traversal, which complies with network protocol, easily goes unnoticed since it complies with protocol and payload limitations. Some of the pros and cons of statistical anomaly detection are summarized in Table 11-4.

Table 11-4. Pros and Cons of Statistical Anamoly Detection

Pros Cons

Detects Unknown Attacks Prone to false positives Prevents DoS attacks, Buffer Overflows Longer detection time

Analyzing Intrusion may be difficult with Anomaly Difficulty in creating baseline

Stateful Protocol Analysis Detection

This method is similar to the anomaly-based detection, except that the profiles are created by the vendors who supply the sensor equipment (IDPS). The profiles are predetermined and made up of the generally accepted benign network traffic activity as specified by the standards. “Stateful” means that the IDPS has the capability to keep track of the state of the protocol both in network layer and application layers. For example, in case of a TCP connection establishment state, the IDS should remember all the connection states. Similarly, in case of authentication, the initial connection session is in an unauthorized state and IDS should remember these states. After an exchange of some information between the two parties, the client and the server, the user is authenticated and allowed access to the network. During this period, the traffic is benign and the IDPS should remember the state or it will lead to false positives.

The stateful protocol anomaly detection method uses profiles that have been created based on standards and specifications specified by the vendor who generally complies with most of the protocols from the standard bodies (Internet Engineering Task Force). If any vendor has implemented protocols, with variation to the standards, it would cause difficulty for the IDPS in detecting and analyzing the states. In such cases, IDPS protocol models also need to be updated for the customized protocol changes.

The primary drawback of this method is that they are processand memory-intensive like many protocols, and the IDPS has to keep track of their states simultaneously. Another problem is if an attack is within the generally

acceptable protocol behavior, then it can pass through. If the protocol implementation varies from operating system to operating system then IDPS may not perform well in detecting the intrusions. The pros and cons of this method are summarized in Table 11-5.

Table 11-5. Stateful Protocol Detection

Pros Cons

Stateful Inspection Resource intensive

Reasonable checks on the standard protocol before an alert

Cannot detect variations to the generally acceptable protocol behavior policy

Cannot detect any conflict between the standards and how they are implemented

 
< Prev   CONTENTS   Next >