Intrusion detection System is work as the network packet sniffer which based on the comparisons of packet contents with knows the virus signatures encapsulated as rules. It can initiate action and record all the events and information related to them in a log file and database. Intrusion detection is the process of monitoring the events occurring in a computer system or network and analyzing them for sign of intrusion. The rule language of an Intrusion Detection System (IDS) plays a critical role in its effectiveness. A rule language must be expressive, in order to describe attack patterns as precisely as possible. It must also allow for a matching algorithm with predictable and low complexity, in order to ensure robustness against denial-of-service attacks. Unfortunately, these requirements often conflict. We show, for instance, that a single rule, when coupled with a backtracking matching algorithm, can bring the processing rate down to nearly ONE packet per second. Performance vulnerabilities of this type are known for patterns described using regular expressions, and can be avoided by using a deterministic matching algorithm. Increasingly, however, rules are being written using the more powerful regex syntax, which includes non-regular features such as back-references. The matching algorithm for general regex's is based on backtracking, and is thus vulnerable to attacks [1] [2] [3].
2.2 What is Network Intrusion Detection System (NIDS)?
Network Intrusion Detection System (NIDS) is a system that emits alerts commensurate with abnormal or unauthorized events in the network. It is a data mining based system for detecting network intrusions. Input to the NIDS is data collected using data capturing unit. It is only capture the header of the packet and build one way session. Captured data are stored in float files. The analyst used NIDS to analyze these in a batch mode. Before data is fed into the anomaly detection module, a data filtering step is performed by the analyst to remove network traffic that the analyst in not interested in analyzing.
The first step in NIDS is extracting features that are used in the data mining analysis. Basic features include source and destination IP addresses, source and destination ports, protocol, flags, number of bytes and number of packets. Derived features include time window and connection-windows based features. The time-window based features are constructed to capture connections with similar characteristics in the last T seconds. These features are especially useful in separating sources of high volume connections per unit time from the rest of the traffic such as fast scanning activities. After the feature construction step, the known attack detection module is used to detect network connections that correspond to attacks for which signatures are available, and then to remove them from further analysis.
Next, the data is fed into the NIDS anomaly detection module that uses an outlier detection algorithm to assign an anomaly score to each network connection. A human analyst then has to look at only the most anomalous connections to determine if they are actual attacks or other interesting behavior. NIDS association pattern analysis module summarizes network connections that are ranked highly anomalous by the anomaly detection module. The analyst provides a feedback after analyzing the summaries created and decides whether these summaries are helpful in creating new rules that may be used in the known attack detection module [3] [4] [5].
2.3 Available Improvement method
2.3.1 FPGA device.
PCGA.png
Figure 2.1: Snort IDS and PCRE engine usage on CPU
Figure 2.1 show the snort IDS and PCRE engine usage on CPU. The heart of the automated compiling system lies in the way regular expressions from the SNORT database are converted to engines on FPGA. The initial step involves invoking the PCRE compiler software which is part of the PCRE package version 7.0. It is added a mechanism for the PCRE compiler to emit compiled opcodes from regular expressions to a local database. The AWK script parses the database and extracts the opcodes corresponding to the regular expression based rules. Thereafter the rules are processed by the opcode to VHDL script. This script generates storage block for each opcode that matches one or more characters. A second script processes the sequence of PCRE operators, in the list of opcodes and generates NFA based control logic in VHDL. A third script combines them to a VHDL entity that interacts with the memory interface module. Once all the regular expression engines are generated, the fourth script generates a payload buffer and a match data buffer. The payload buffer receives TCP / IP payload from the software and on each clock cycle retrieves a character into each of the engine through the memory interface module. FPGAs allow speedup of slow sequential software by efficient hardware execution, as well as executing multiple threads in parallel, thus augmenting the host processor. PCRE rules are segregated out from the SNORT. Database, they are compiled through the PCRE compiler. The PCRE compiler software is part of the PCRE package and converts PCRE regular expressions into PCRE opcodes. PCRE Opcodes are the instructions for the software based PCRE engine, and are denied in the pcre internal and is part of the PCRE package [7].
2.3.2 Exclusion-based Signature
An informal description of ExB is in its simplest and most intuitive form and then in more general ExB is based on the following simple reasoning: Suppose that we want to check whether a small input I contain a string s. Then, if there is at least one character in s that is not in I, then s is not in I.
This can be used to determine when a given string does not appear in the input string I. If, on the other hand, every character of s belongs to I, then we use a
Standard string searching algorithms (e.g., BoyerMooreHorspool) to confirm whether s is a substring of I. This is needed in case of false matches, for example, cases where every Character of s is in I, but not in the sequence they appear in I. This simple the matching problem, since we can efficiently determine whether a character c belongs to I by means of an occurrence bitmap. Specially, we first pre-process the input I, and for each character c that appears in string I, we mark the corresponding element on the (256element) bitmap. After preprocessing, we know that string I contains the jth character only if the jth element of the bitmap is marked. The pseudo code for preprocessing input and for matching a string s on input the algorithm can be generalized for pairs of characters, with the intention of reducing the probability of false matches. Instead of recording the occurrence of single characters in string I, it is possible to record the appearance of each pair of consecutive characters in string I. In the matching process, instead of determining whether each character of s appears in I, the algorithm then checks whether each pair of consecutive characters of s appears in I. If a pair is found that does not appear in I, ExB knows that s is not in I. Generalizing further, ExB can use bit strings of arbitrary length, instead of just 8bit characters. ExB records all (byte aligned) bit strings of length I. The size of the bit string exposes a tradeoff: larger bit strings are likely to result in fewer false matches, but also increase the size of the occurrence bitmap, which could, in turn, increase capacity misses and degrade performance [8].
2.3.3 Linux Kernel and Application
A user land application is classified as a CPU- or I/O-bound. A CPU-bound application is an application whose speed is limited by CPU speed and availability, whereas an I/O-bound application is an application whose execution speed is limited by the speed of I/O devices. A CPU-bound application requires heavy computational cycles with infrequent intermediate I/O operations, whereas an I/O-bound application computational cycles. In other words, a CPU-bound application often consumes all of its CPU quantum, whereas an I/O-bound application often gives up CPU before the end of its quantum because of I/O operations. In order to provide fair sharing of CPU cycles and to prevent starvation of I/O-bound processes in priority of I/O application to compete and to be able to preempt CUP-bound applications.
Snort NIDS can be classified as a CPU- or I/O-bound process, depending primarily on the incoming traffic and snort configuration and rule actions. For example, under normal traffic with no logging, Snort behaves as a CPU bound process, with most of its execution time spent on processing and analyzing packets by the detection engine. However, under malicious traffic with alerting and logging, snort behaves as an I/O-bound process, with heavy disk or database I/O operations involved in alerting and logging every malicious packet. In conclusion such settings would make it possible for Snort to consume all of its CPU quantum and to resume running as soon as its I/O operation is completed [9].
2.3.4 Improve NIDS
A NIDS with insufficient capacity skips packets and may manifest inadequate detection during heavy traffic. A resourceful hacker may overload such NIDS with a flood of smoke screen packets to sneak past the real attack packets, which the NIDS may skip. In addition to possible inefficiencies in the NIDS application and in the underlying general purpose commodity PC hardware, inefficient packet capturing support in most operating system.
The packet capturing system at the sensor can be modeled as a finite length receiving buffer, where the incoming packets get queued at rate. The receiving host CPU services this queue at a rate which is given by the reciprocal of the total packet receiving and processing response time.
An efficient packet capturing will lower the receiving response component of processing response time and increase the system capacity. Other systemic limitations of the underlying platform which introduce task response jitters may also cause packet loss, even when the total average task response may be well below the incoming packet rate limit. The packet loss occurs when the queue length, a random variable exceeds the finite buffer capacity [10].
2.4 Understand Linux Performance
2.4.1 Linux Memory architecture
mm.png
Figure 2.2: Linux Kernel Memory Architecture
Figure 2.1 show that differences in the way the Linux kernel hast to address memory in 32-bit and 64-bit systems. On 32-bit architectures such as the IA-32, the Linux kernel can directly address only the first gigabyte of physical memory (896 MB when considering the reserved range). Memory above the so-called ZONE_NORMAL must be mapped into the lower 1 GB. This mapping is completely transparent to applications, but allocating a memory page in ZONE_HIGHMEM causes a small performance degradation.
On the other hand, with 64-bit architectures such as x86-64 (also known as EM64T or AMD64 respectively), ZONE_NORMAL extends all the way to 64GB or to 128 GB in the case of IA-64 systems. As the figure show above, the overhead of mapping memory pages from ZONE_HIGHMEM into ZONE_NORMAL can be eliminated by using a 64-bit architecture [11].
2.4.2 Linux Performance Metric
2.4.2.1 Processor metric
CPU utilization
This is probably the most straightforward metric. It describes the overall utilization per processor. On xSeries architectures, if the CPU utilization exceeds 80% for a sustained period of time, a processor bottleneck is likely.
Runable processes
This value depicts the processes that are ready to be executed. This value should not exceed 10 times the amount of physical processors for a sustained period of time; otherwise a processor bottleneck is likely.
Blocked
Processes that cannot execute as they are waiting for an I/O operation to finish. Blocked processes can point you toward an I/O bottleneck.
User time
Depicts the CPU percentage spent on user processes, including nice time. High values in user time are generally desirable because, in this case, the system performs actual work.
System time
Depicts the CPU percentage spent on kernel operations including IRQ and softirq time. High and sustained system time values can point you to bottlenecks in the network and driver stack. A system should generally spend as little time as possible in kernel time.
Idle time
Depicts the CPU percentage the system was idle waiting for tasks.
Nice time
Depicts the CPU percentage spent on re-nicing processes that change the execution order and priority of processes.
Context switch
Amount of switches between threads that occur on the system. High numbers of context switches in connection with a large number of interrupts can signal driver or application issues. Context switches generally are not desirable because the CPU cache is flushed with each one, but some context switching is necessary.
Waiting
Total amount of CPU time spent waiting for an I/O operation to occur. Like the blocked value, a system should not spend too much time waiting for I/O operations; otherwise you should investigate the performance of the respective I/O subsystem [11].
2.4.2.2 Memory Metric
Free memory
Compared to most other operating systems, the free memory value in Linux should not be a cause for worries. As explained in 1.3, "The virtual memory manager" on page 5, the Linux kernel allocates most unused memory as file system cache, so subtract the amount of buffers and cache from the used memory to determine (effectively) free memory.
Swap usage
This value depicts the amount of swap space used. As described in 1.3, "The virtual memory manager" on page 5, swap usage only tells you that Linux manages memory really efficiently. Swap In/Out is a reliable means of identifying a memory bottleneck. Values above 200 to 300 pages per second for a sustained period of time express a likely memory bottleneck.
Buffer and cache
Cache allocated as file system and block device cache. Note that in Red Hat Enterprise Linux 3 and earlier, most free memory will be used for cache. Under Red Hat Enterprise Linux 4 you can specify the amount of free memory allocated as cache via the page_cache_tuning entry in /proc/sys/vm.
Slabs
Depicts the kernel usage of memory. Note that kernel pages cannot be paged out to disk.
Active versus inactive memory
Provides you with information about the active use of the system memory. Inactive memory is a likely candidate to be swapped out to disk by the kswapd daemon [11].
2.5 Software
2.5.1 Snort
Snort is a popular NIDS which widely used to audit network packets and compare them with a database of known attack signatures. It is a open source software programmed by C language, which conforms to GNU. Campus network and the information technology organization that support these networks are facing security threats that are increasing in both size and complexity. It is a platform in dependent, light network intrusion detection system, and it also works as network traffic sniffer and log record tool. Snort employs network information search technique based on rules. Pattern matching is performed on each packet to conduct intrusion detection [12] [13].
Snort is a libpcap-based [PCAP94] packet sniffer and logger that can be used as a lightweight network intrusion detection system (NIDS). It features rules based logging to perform content pattern matching and detect a variety of attacks and probes, such as buffer overflows [ALE96], stealth port scans, CGI attacks, SMB probes, and much more. Snort has real-time alerting capability, with alerts being sent to syslog, Server Message Block (SMB) "WinPopup" messages, or a separate "alert" file. Snort is configured using command line switches and optional Berkeley Packet Filter [BPF93] commands. The detection engine is programmed using a simple language that describes per packet tests and actions. Ease of use simplifies and expedites the development of new exploit detection rules
2.5.2 Snort Signature
Snort signature is the rules file which content all the information of suspicious packet. The format consists rule action, protocol, IP address, port number and rule option. These are the main information to check the arrival packet.
Rule action is the such as alert, log and pass action when the protocol, IP address, port number and rule option is match with the arrival packet. Protocol can be TCP, UDP and ICMP and IP address and port number is categories in the source IP address, source port address, destination IP address and destination port address. These are the information require to check the packet.
Rule option has a lot of options, the purpose of the options to reduce false positive or false negative. So it is the part which cannot omit by the rule creator.
2.5.3 Snort versus tcpdump
Figure 2.3: Typical Snort telnet packet display
Figure 2.4: The same telnet packet as displayer by tcpdump
Snort is cosmetically similar to tcpdump but is more focused on the security applications of packet sniffing. The major feature that Snort has which tcpdump does not is packet payload inspection. Snort decodes the application layer of a packet and can be given rules to collect traffic that has specific data contained within its application layer. This allows Snort to detect many types of hostile activity, including buffer overflows, CGI scans, or any other data in the packet payload that can be characterized in a unique detection fingerprint.
Another Snort advantage is that its decoded output display is somewhat more user friendly than tcpdump's output. Snort does not currently lookup host names or port names while running, which is a function that tcpdump can perform. Snort is focused on collecting packets as quickly as possible and processing them in the Snort detection engine. Performing run-time host name lookup is not conducive to high performance packet analysis. Figure 1 shows typical Snort output for a telnet banner display, and Figure 2 shows the same packet as displayed by tcpdump.
One powerful feature that Snort and tcpdump share is the capability to filter traffic with Berkeley Packet Filter (BPF) commands. This allows traffic to be collected based upon a variety of specific packet fields. For example, both tools may be instructed via BPF commands to process TCP traffic only. While tcpdump would collect all TCP traffic, Snort can utilize its flexible rules set to perform additional functions, such as searching out and recording only those packets that have their TCP flags set a particular way or containing web requests that amount to CGI vulnerability probes. The SHADOW IDS from the Naval Surface Warfare Center is based on tcpdump and uses extensive BPF filtering.
2.5.4 Logging/Alerting System
The alerting and logging subsystem is selected at run-time with command line switches. There are currently three logging and five alerting options. The logging options can be set to log packets in their decoded, human readable format to an IP-based directory structure, or in tcpdump binary format to a single log file. The decoded format logging allows fast analysis of data collected by the system. The tcpdump format is much faster to record to the disk and should be used in instances where high performance is required. Logging can also be turned off completely, leaving alerts enabled for even greater performance improvements.
Alerts may be sent to syslog, logged to an alert text file in two different formats, or sent as WinPopup messages using the Samba smbclient program. The syslog alerts are sent as security/authorization messages that are easily monitored with tools such as swatch [SWT93]. WinPopup alerts allow event notifications to be sent to a user-specified list of Microsoft Windows consoles running the WinPopup software. There are two options for sending the alerts to a plain text file; full and fast alerting. Full alerting writes the alert message and the packet header information through the transport layer protocol. The fast alert option writes a condensed subset of the header information to the alert file, allowing greater performance under load than full mode. There is a fifth option to completely disable alerting, which is useful when alerting is unnecessary or inappropriate, such as when network penetrations tests are being performed.
2.5.5 Snortby
Snorby is a new and modern Snort IDS front-end. The basic fundamental concepts behind snorby are simplicity and power. The project goal is to create a free, open source and highly competitive application for network monitoring for both private and enterprise use. With a simple keystroke or a mouse click, analysts can quickly classify an event into one of the many preconfigured classifications or into custom classifications relevant to an organization. Use classifications to organize events into helpful categories for follow-up investigations or for tuning your alert rule sets.
Unlike most network security monitoring applications, Snorby integrates with new and existing OpenFPC,Solera DS Appliances, and Solera's DeepSeeinstallations to give analysts full packet and session data
Moreover Snorby is 100% open source! The snorby* source code is licensed under version 3 of the permissive GNU General Public License. Snorby uses highcharts for metrics and reporting and has it's own licensing which can be found here. Snorby is actively and passionately developed and has ushered in a large and enthusiastic support community
2.5.6 Libnet
Libnet is a generic networking API that provides access to several protocols. It is not designed as a 'all in one' solution to networking. Currently many features that are common in some network protocols are not available with Libnet, such as streaming via TCP/IP. We feel that Libnet should not provide specific features that are possible in other protocols. If we restrict Libnet to the minimal needed to communicate (datagram/packets) then this allows it to support more interfaces.
Libnet supports the following platforms and compilers:
Unix (gcc) -- tested with Linux, FreeBSD and OSF/1
DOS (djgpp)
Windows (djgpp+RSXNTDJ, Mingw32, or MSVC)
Other platforms should be fairly easy to support, especially if GNU CC and GNU Make will run on them. Libnet supports UDP/IP on UNIX, Windows, and, provided the Winsock is not version .0 or greater, in a DOS box under Windows. It supports IPX in DOS, Windows, and UNIX, and serial linking in DOS, Windows, UNIX, and BeOS. On all platforms there is a `local host' driver which is a sort of internal loopback device
2.5.7 Libdnet
libdnet provides a simplified, portable interface to several low-level networking routines, including
network address manipulation
kernel arp(4) cache and route(4) table lookup and manipulation
network firewalling (IP filter, ipfw, ipchains, pf, PktFilter, ...)
network interface lookup and manipulation
IP tunnelling (BSD/Linux tun, Universal TUN/TAP device)
raw IP packet and Ethernet frame transmission
And it is support languages,C, C++, Python and Perl, Ruby.
2.5.8 Data Acquisition (DAQ)
Data acquisition is the process of sampling signals that measure real world physical conditions and converting the resulting samples into digital numeric values that can be manipulated by a computer. Data acquisition systems (abbreviated with the acronym DAS or DAQ) typically convert analog waveforms into digital values for processing. The components of data acquisition systems include:
Sensors that convert physical parameters to electrical signals.
Signal conditioning circuitry to convert sensor signals into a form that can be converted to digital values.
Analog-to-digital converters, which convert conditioned sensor signals to digital values.
Data acquisition applications are controlled by software programs developed using various general purpose programming languages such as BASIC, C, Fortran, Java, Lisp, Pascal. COMEDI is an open source API (application program Interface) used by applications to access and controls the data acquisition hardware. Using COMEDI allows the same programs to run on different operating systems, like Linux and Windows.
DAQ software is needed in order for the DAQ hardware to work with a PC. The device driver performs low-level register writes and reads on the hardware, while exposing a standard API for developing user applications. A standard API such as COMEDI allows the same user applications to run on different operating systems, e.g. a user application that runs on Windows will also run on Linux and BSD
DAQ hardware is what usually interfaces between the signal and a PC. It could be in the form of modules that can be connected to the computer's ports (parallel, serial, USB, etc.) or cards connected to slots in the mother board. Usually the space on the back of a PCI card is too small for all the connections needed, so an external breakout box is required. The cable between this box and the PC can be expensive due to the many wires, and the required shielding.
2.5.9 Basic Analysis and Security Engine (BASE)
BASE is the Basic Analysis and Security Engine. It is based on the code from the Analysis Console for Intrusion Databases (ACID) project. This application provides a web front-end to query and analyze the alerts coming from a SNORT IDS system.
BASE is a web interface to perform analysis of intrusions that snort has detected on your network. It uses a user authentication and role-base system, so that you as the security admin can decide what and how much information each user can see. It also has a simple to use, web-based setup program for people not comfortable with editing files directly.
BASE is supported by a group of volunteers. They are available to answer any questions you may have or help you out in setting up your system. They are also skilled in intrusion detection systems and make use of that knowledge in the development of BASE.
2.5.10 ADODB
ADOdb is a database abstraction library for PHP and Python based on the same concept as Microsoft's ActiveX Data Objects. It allows developers to write applications in a fairly consistent way regardless of the underlying database system storing the information. The advantage is that the database system can be changed without re-writing every call to it in the application. Some reasons why ADOdb is popular include:
Designed for speed - It is probably the fastest open source database abstraction library available for PHP.
Provides extensive portability support - date and type-handling and portable schema creation. See portable sql tips
Support many enterprise features - database backed sessions (with session expiry notification), SQL code generation, pivot tables, SELECT LIMIT emulation for all databases, performance monitoring.
Easy to learn - especially have Window's programming experience, as it uses many ADO conventions.
Extensive QA - every release is unit-tested on Access, MySQL, PostgreSQL, MS SQL, Oracle 11g.
Mature - continuously developed since August 2000. Has a large community of users.
Powerful Active Record support - Very reasonable licensing terms (BSD). This means that you can incorporate (and even compile) it into your software applications royalty-free without asking the author's permission, provided you include license.txt in your release. Also dual-licensed (Lesser GPL).
2.5.11 Barnyard and Barnyard2
Barnyard is an add-on tool for snort which allows logging or alerting operations to be offloaded from snort. It is a Unified Log Reader which read the Unified log file by snort. With help of barnyard is able to improve the performance of snort like:
Unified output logging is comparatively fast
Offloads performance-intensive logging operations from Snort (e.g. database logging)
This frees Snort to focus on packet inspection
Barnyard2 is a fork of the original barnyard project, designed specifically for Snort's new unified2 file format. The new version has the following features:
Offloads output processing of your Snort alert files to a dedicated process, minimizing dropped packets in Snort itself.
Parses unified2 files.
Uses similar configuration syntax to that of Snort to simplify deployment.
Supports all Snort output plug-in (except alert_sf_socket) as well as two additional plug-in (Sguil and CEF).
2.5.12 Oinkmaster & pulled pork
Oinkmaster is a toll that will help the snort update and manage the snort rules. It will work on most platforms that can run Perl script. Oinkmaster can be user to update and manage the VRT licensed rules, the community rules, the bleeding-snort, rules and other third party rules including local rules.
The current version of Oinkmaster does not handle so_rules and also had a few minor shortcomings. With the advent of dcerpc2 and thus the decrease in text based rules for such things, a utility/app/script is needed that can handle so_rules and thus Pulled_Pork was conjured
2.5.13 COMMVIEW
CommView is a powerful network monitor and analyzer designed for LAN administrators, security professionals, network programmers, home users…virtually anyone who wants a full picture of the traffic flowing through a PC or LAN segment. Loaded with many user-friendly features, CommView combines performance and flexibility with an ease of use unmatched in the industry.
This application captures every packet on the wire to display important information such as a list of packets and network connections, vital statistics, protocol distribution charts, and so on. You can examine, save, filter, import and export captured packets, view protocol decodes down to the lowest layer with full analysis ofover 70 widespread protocols. With this information, CommView can help you pinpoint network problems and troubleshoot software and hardware.
CommView includes a VoIP analyzer for in-depth analysis, recording, and playback of SIP and H.323 voice communications. The function of CommView includes:
View detailed IP connections statistics: IP addresses, ports, sessions, etc.
Reconstruct TCP sessions.
Map packets to the application that is sending or receiving them.
View protocols distribution, bandwidth utilization, and network nodes charts and tables.
Generate traffic reports in real time.
Browse captured and decoded packets in real time.
Search for strings or hex data in captured packet contents.
Import and export packets in Sniffer®, EtherPeekâ„¢, AiroPeekâ„¢, Observer®, NetMon, and Tcpdump formats, export packets in hex and text formats.
Configure alarms that can notify you about important events, such as suspicious packets, high bandwidth utilization, unknown addresses, etc.
Create your own plug-ins for decoding any protocol.
Exchange data with your application over TCP/IP.
Export any IP address to Smart Whois for quick, easy IP lookup.
Capture loopback traffic
