This case study looks at the typical experience of a Georgia Army National Guard. The GaARNG turned to distributed patch management in an effort to reduce the cost of distributed centralized operations centers. The resulting centralization of desktop administration and support follows on the success of an earlier effort to use virtualization to consolidate and centralize server management. The problems faced by the GaARNG are similar to those facing other National Guard Units, as well as many businesses.
The development of today's distributed patch management systems has created a progressive structure for managing organizational computer systems. This is known as updating computers for known vulnerabilities (on both servers and workstations), this is essential for maintaining a secure computing system. Patches are composed of several different computer "fixes" that help maintain, secure and control the computer systems for an organization. In this term paper I will take a closer look at some of the issues that the Georgia Army National Guard (GaARNG) had with patch management and the solution that was implemented.
In the GaARNG patch management is usually under the overall responsibility of the Information Assurance Manager (IAM). The department that performs patch management is known as Information Technology Division or simply ITD. ITD has a Systems Administrator or (Sys Admin) that provides network and database management, which is viewed as part of patch management or they may be viewed as co-equal parts of a total patch management system. GaARNG used Windows Update Server (WUS) and System Management Server (SMS) to manage 1,000 Windows XP clients, 1, 500 Windows Vista, 90 Windows 2003 Servers, 1 Linux and HP server.
What is Patch Management? Patch management is one of the most important processes to repair vulnerabilities of operating systems and software. Since an institute or a company has distributed hierarchical structure and the structure consists of many heterogeneous Patch, it is not easy to update patches timely. Patch management framework with patch profiling mechanism and patch dependency solving mechanism. There it's no secret that identifying and correcting network security holes is critical to protecting any business from harmful attacks, the process of vulnerability assessment and remediation often gets overlooked as a critical component of sound security practices. Because it is an ongoing process, many companies avoid proper vulnerability auditing until disaster strikes and they are forced to react. Some businesses fail to learn the lesson of proactive vulnerability assessment and remediation. Despite all the attention that firewalls, anti-virus applications and Intrusion Detection System (IDS) receive, security vulnerabilities still plague organizations. The implementation of these tools often leads administrators into believing that their networks are safe from intruders. There is a complex threat environment of malware, spyware, disgruntled employees and aggressive international hackers, developing and enforcing a strict and regular network security policy that incorporates on-going vulnerability assessment is critical to maintaining business continuity. Firewalls and IDS are independent layers of security. Firewalls merely examine network packets to determine whether or not to forward them on to their end destination. Firewalls screen data based on domain names or IP addresses and can screen for low-level attacks. They are not designed to protect networks from vulnerabilities and improper system configurations. Nor can they protect from malicious internal activity or rogue assets inside the firewall. Similarly, an IDS inspects all inbound and outbound network activity and identifies suspicious patterns. IDS can be either passive or reactive in design, but either way they rely on signature files of known attacks to prevent intrusion. Most sophisticated attacks can easily trick IDS and penetrate networks. Likewise, an IDS will not protect against vulnerabilities that may be exploited by remotely executed code. A vulnerability assessment system, on the other hand, will look at the network and pinpoint the weaknesses that need to be fixed/patched - before they ever get breached. With over 80 new vulnerabilities announced each week, a company's network is only as secure as its latest vulnerability assessment. An ongoing vulnerability assessment process, in combination with proper remediation, will help ensure that the network is fortified to withstand the latest attacks.
In the GaARNG there is an established Configuration Control Board or CCB. The CCB managements any network infrastructure changes as well as any changes to software. Each client workstation is imaged with the CCB approved master image. If any changes are requested they must to be approved by the CCB. The CCB board members are comprised of key personnel from ITD and Data Processing Installation or DPI. CCB is vital to every stage of the Patch Management process. As with all system modifications, patches and updates must be tracked through the change management process. Like any environmental change, plans submitted through change management must have associated contingency and back out plans. Also, information on risk mitigation should be included in the change management solution. How are desktop patches going to be scheduled and rolled out to prevent mass outages and service desk overload? Monitoring and acceptance plans should be included. How will updates be certified as successful? There should be specific milestones and acceptance criteria to guide the verification of the patches' success, and to allow for the closure of the update in the change management system.
Vulnerability management is another keep section of patch management. Vulnerability management will focus on the immediate risk to the GaARNG network infrastructure and furnishs Computer Network Defense (CND) alerts to potential compromises. Vulnerability management takes a leading role in monitoring the network and develops countermeasures to be taken on those vulnerabilities that are active, while providing assistance with reducing risk and vulnerability remediation.
Once a patch was deployed, many computers were not often or never updated. Most software patches are known as installing software or operating system fixes. I associate a patch with a band-aid. The band-aid covers up the scratch or wound but does not heal the skin, an update patch is the same way, and it covers up the opening but doesn't repair the entire system. This perception is accurate, and as soon as the patch is release it should be tested and installed.
The increase of quickly circulating malicious code and worms targeting recognized vulnerabilities on systems that are not kept up to date, and the downtime along with the expense to bring a system back up are prominent causes for the GaARNG to focus on patch management. These malware threats and increasing interest on regulatory compliance (IE: HIPAA) has evaluated the GaARNG to improve control over the Information Technology assets that have Personal Identifiable Information (PII). The increase in Virtual Private Network (VPN), which gives the mobile user the ability to remote back into the network, is forcing patch management to become a major priority. Patches are normally used to install software fixes and to add new functionality. Others provide operational enhancements of legacy systems through driver optimization. Additionally, patching will provide more options to current network management tools. On an enterprise level these improvements on software service packs and firmware upgrades will also enhanced functionality on certain computers.
A key component of all patch management strategies is the intake and vetting of information regarding both security and utility patch releases. You must know which security issues and software updates are relevant to your environment.
An organization needs a point person or team that is responsible for keeping up to date on newly released patches and security issues that affect the systems and applications deployed. This team should also take the lead in alerting those responsible for the maintenance of systems to security issues or updates to the applications and systems they support. Intelligence gathering can be accomplished by subscribing to vendor supplied alerts, free RSS feeds, or paid professional service subscriptions. Each alerting mechanism will provide specific benefits and have specific shortcomings.
Not all vendors offer alerting mechanisms for their security and utility patch releases. Those that do typically offer this service free of charge. Vendors have historically announced patch release, not development, although that is starting to change of late. Vendors will typically not monitor or alert users to the various stages of exploit code development. Some security vendors, researchers and popular social networking sites offer breaking vulnerability news feeds. These feeds are generally free, and can offer advice before a patch is released, as well as potential work-around. Advice from 3rd party sources ranges from lame to authoritative, and may not be vendor recommended or supported. Paid services have hit the market, offering deep-dive analysis, consultative recommendations, and advance warning. Prices for these services can range from a few hundred dollars a month to hundreds of thousands a year. Public web sites and mailing lists should be regularly monitored at a minimum, including Bugtraq, SecurityFocus lists, and patchmanagement.org. A comprehensive and accurate Asset Management system can be pivotal in determining whether all existing systems have been considered when researching and processing information regarding vulnerabilities and patches.
The recommended method for applying patches is to use some form of patch deployment automation. Users and administrators should not be permitted to apply patches arbitrarily. While this should be addressed at a policy and procedural level with acceptable use policies, change management processes, and established maintenance windows, it may also be appropriate to apply additional technical controls to limit when and by whom patches can be applied. Even for smaller businesses, the savings that can be realized through deployment automation can be significant. Imagine, patching one system for developing an image, testing it in a virtualized environment that mimics your production environment, and then at the press of a button, consistently upgrading your entire organization to a more secure configuration.
The benefits of using deployment automation are reduced time spent patching, reduced human error factored into each deployment exercise, significant reductions in overtime and associated costs. Decrease in downtime because patching is done in non-working hours, or often as a background task. Consistent operating system and application image across the environment, reducing service desk calls. Auditing reports, including asset inventory, licensing, and other standard reports.
Applying security and utility patches in a timely manner is critical, however these updates must be made in a controlled and predictable fashion, properly assessed and prioritized. Without an organized and controlled patch deployment process, system state will tend to drift from the norm quickly, and compliance with mandated patch levels will diminish.
GaARNG Patch Management Strategies:
With major software vendors now reporting more than 8,000 software vulnerabilities 1 each year, eradicating all of the potential threats to your network endpoints is a daunting task. Not to mention the rising amount of threats that attack Patch configurations. Fortunately, there is a way. Lumension Security's Patch Management and Remediation Solution allows you to automate the collection, assessment and deployment of software patches and to create and deploy remediation packages that address a wide range of configuration related issues. This comprehensive solution significantly streamlines the patch management process and adds custom capabilities to address configuration issues, ultimately reducing the incidents of worms, Trojans, viruses and targeted malicious attacks. Lumension's Patch Management and Remediation Solution is comprised of three market-leading security products; PatchLink Updateâ„¢, PatchLink Developers Kitâ„¢ and the PatchLink Security Management Consoleâ„¢. These three products work together seamlessly to:
Reactive Patch Management Strategy
The main goal in reactive patch management is to reduce the impact of an outage. Reactive patching occurs in response to an issue that is currently affecting the running system, and that needs immediate relief. The most common response to such a situation is usually to apply the latest patch or patches, which might be perceived as being capable of fixing the issue. Unfortunately, if the patch implementation does not work, you are often left worse off than before you applied the patch.
There are several main reasons why this approach is not effective: even if a known problem appears to go away, you don't know whether the patch or patches actually fixed the underlying problem or simply masked the symptoms. The patches might have simply changed the system in such a way as to obscure the issue for now. Applying patches in a reactive patching session introduces a considerable element of risk. When you are in a reactive patching situation, you must try to minimize risk at all costs. In proactive patching, you can and should have tested the change you are applying. In a reactive situation, if you apply a large number of changes, you still may not have identified root cause. Also, there's a greater chance that the changes you applied will have negative consequences elsewhere on the system, which leads to more reactive patching.
So, even when you experience an issue that is affecting the system, spend time investigating root cause. If a fix can be identified from such investigation, and that fix involves applying one or more patches, then at least the change is minimized to just the patch or set of patches required to fix the problem. Depending on the severity of the problem, the patch or patches that fix the issue will be installed at one of the following times: Immediately to gain relief. At the next regular maintenance window, if the problem is not critical or a workaround exists. During an emergency maintenance window that is brought forward to facilitate applying the fix.
Identifying Patches for Reactive Patching
Identifying patches that are applicable in a reactive patching scenario can often be complex. In a lot of cases, depending on support, contact with official vendor channels will be initiated, but as a starting point, you should do some analysis. There is no single standard way of analyzing a technical issue, because each issue involves different choices. Using debug level logging and examining log files usually provides some troubleshooting guidance. Also, a proper recording system that records changes to the system should be considered. Recent system configuration changes can be investigated as possible root causes.
Proactive Patch Management Strategy
The main goal in proactive patch management is to prevent unplanned downtime. The idea behind proactive patching is that in most cases, problems that can occur have already been identified, and patches have already been released. So, the problem becomes mainly one of identifying the most important patches, and applying them in a safe and reliable manner.
In all cases of proactive patching, it is assumed that the system is functioning normally. Why patch a system that is functioning normally, since any change implies risk and downtime? As with any system that is functioning normally, there is always the chance that some underlying, known issue can cause a problem. Such issues can include the following: Memory corruption that has not yet caused a problem. Data corruption, which is typically unnoticed until the data is re-read. Latent security issues.
Security issues are a good example of the value of proactive patching. Most security issues are latent issues, meaning they exist in the system, but are not causing issues yet. It is important to take proactive action to prevent security vulnerabilities from being exploited.
In comparison to reactive patching, proactive patching generally implies more change, and additional planning, for regularly scheduled maintenance windows and testing.
Proactive patching is the strategy of choice. Proactive patching is recommended mainly for the following reasons: proactive patching reduces unplanned downtime. Proactive patching prevents systems from experiencing known issues. Proactive patching provides the ability to plan ahead and do appropriate testing before deployment. Planned downtime for proactive maintenance is usually much less expensive than unplanned downtime for addressing issues reactively.
Security patch management requires a separate strategy because it requires you to be proactive and yet requires reactive patching's sense of urgency. In other words, security fixes deemed relevant to the environment might need to be installed proactively, before the next scheduled maintenance window. The same general rules apply to security patches as to proactively or reactively applying utility patches. Plan, test, and automate.
All security patches should be assessed independently. Although vendors have begun to standardize on a single patch assessment methodology, they cannot take into account the most important factors, the environmental factors. They are also reticent in bringing attention to exploit code development, and have been prone to understating the severity and impact that vulnerabilities in their products pose. The framework for analysis that vendors are adopting, and is strongly recommended for all businesses, is the Common Vulnerability Scoring System, or CVSS. CVSS was developed by the NIST team, is now in its second version, and has spawned a series of other assessment tools to solve a multitude of problems from malware naming conventions to configuration issues.
Security patch planning should be performed based on the factored risk rating of the vulnerability, and a standard sliding patch window adopted for each of the platforms in use, that ties directly back to the rating. If the organization is smaller, consider a single monthly window for applying all missing security patches. Medium sized organizations might consider having a second maintenance period every month, as they are more likely to have multiple platforms present (IE: Windows & Unix). Larger, Enterprise environments might consider having several maintenance periods as well. In the case of larger environments, complexity of the platforms in use and their inter-dependencies must be taken into account. If the front-end systems are going to be down for utility patches and updates, it may be a perfect time to apply security patches for the back-end databases, for instance. Make certain that no matter what the size of the organization or the nature of the patch, a back out plan has been developed and tested.
Regular audit and assessment helps gauge the success and extent of patch management efforts. There are typically two phases in the auditing and assessment portion of the patch management program. Verification and Validation. You are essentially trying to answer two very different questions: Verification - What systems need to be patched? Validation - Are the systems that were supposed to be patched, actually patched and protected?
The audit and assessment component will help answer these questions, but there are dependencies. The most critical success factor here is accurate and effective asset management information. The major requirement for any asset management system is the ability to accurately track deployed hardware and software throughout the enterprise, including remote users and office locations. Ideally, asset management software will allow the administrator to generate reports that will be used to drive the effort toward consistent installation of patches and updates across the organization.
System discovery is an essential component of the audit and assessment process. While asset management systems can help administer and report on known systems, there are likely a number of systems that have been unknowingly or intentionally excluded from inventory databases and management infrastructures. System discovery tools can help uncover these systems and assist in bringing them under the umbrella of formal asset management and patch management compliance
Regardless of the tools used, the goal is to discover all systems within your environment and assess their compliance with the organization's patch and configuration policies and standards.
The Georgia Army National Guard decided on Patchlink UPDATE consists of both client-side and server-side components for critical patch management and basic software distribution.
Client-side: PatchLink has a patent pending technology and is the leading company in automated
Server-side: PATCHLINK is based on PatchLink's proven technology for automated patch detection and deployment, enabling the management and distribution of critical patches and software packages that resolve known security vulnerabilities and other stability issues with. The company has successfully fulfilled customer patch requirements since mid-1996. PATCHLINK UPDATE Server runs on Windows 2000 Server with Service Pack 2 or later, including Windows 2003 Server.
Internet Information Services (IIS) must be enabled on the server.
Secured Automatic Replication: The replication service is a server-side component that retrieves the latest critical updates from the private PATCHLINK UPDATE Master Archive. As new updates are added to the PATCHLINK UPDATE Master Archive, their meta data is downloaded automatically. If patches are marked as critical, they are downloaded and cached for rapid deployment. Each patch has an installer, prerequisite signature and fingerprint identification. Information is sent in one direction only; from the Master Archive to the user's PATCHLINK UPDATE Server. All information is encrypted, CRC checked, compressed, digitally signed, and downloaded over a 128-bit SSL connection. The SSL connection validates and confirms the authenticity of the patch source.
Patch detection and deployment: Focusing solely on technology to solve the patch management issue is not the best answer. Installing patch management software and vulnerability assessment tools without supporting policies, standards, guidelines, and oversight will be a wasted effort. Instead, solid patch management programs will team technological solutions with policy and operationally-based components that work together to address each organization's unique needs.
When new patches are released, the PATCHLINK UPDATE Server downloads the proper fingerprint from the PATCHLINK UPDATE Master Archive and then checks to see if there are any computers that meet the profile by sending the fingerprints to the workstations to be scanned. The administrator is then notified of the new patch and its impact to the work environment. The report matrix quickly informs the administrator which computers or groups need the patch and which do not. The administrator simply selects an individual computer or a group and then deploys. The administrator can set the time of the deployment and decide whether or not to reboot after the patch installation.
In a managed data center, the administrator creates a group for each cluster of servers. This will help the administrator manage large numbers of servers easily. Administrators can test all critical updates published from the PATCHLINK UPDATE Master Archive service before they are deployed to client computers on the network. After the testing has been successful, the administrator can then deploy the patch to all or just a group of servers. The use of agent policies will help the administrator to setup the hours of operation for each group of servers.
In summary, today's network environments are dynamic, requiring a multitude of defense measures to effectively prevent attacks and efficiently mitigate vulnerabilities across the entire enterprise. Organizations must not only be aware of threats, but also the impact of those threats on their infrastructure. Security administrators require a solution that can put them in a position to rapidly and effectively respond. Vulnerability assessment solutions provide the constant monitoring and evaluation of security events to help security and IT personnel stay ahead of the attacks and ensure ongoing business continuity
