"IT consolidation is a major undertaking that can require escalating upfront capital costs to achieve long-term cost savings. It can also take between six months and two years to execute. As such, these investments often face senior executive — if not board-level — scrutiny. A business case built or vetted by a major consultancy has a better chance of approval due to higher perceived credibility of the methodologies and rigor behind the business cases built by these firms." (Source: James Staten, principal analyst, Forrester Research)
One of the more common sights in enterprises is to see an incremental approach to addressing big problems. From an engineering perspective, this is fine, and really quite a good problem solving tactic. However, when it comes to making meaningful change from a security perspective, I have come to seriously question the utility of incremental changes.
The problem, as I see it, is that attackers are not evolving their methods incrementally so much as exponentially. You might call this a semantics game, but let me put it to you this way: if your enemy is growing their capabilities exponentially, and you're growing your defenses linearly, then do you think there's any way to catch up? If your answer here isn't "obviously you can't catch up" then let's look at a quick picture to illustrate my point (using x^2 vs 2x).
As you can see in the above diagram, there is an initial window of opportunity during which a linear progress will actually out-pace an exponential progression. However, one you get past those first two iterations, it's off to the races, and you're not the winner.
The problem, of course, is the law of finite resources. Organizations generally do not have unlimited budget or unlimited staffing to tackle big problems. The full-time focus of the enterprise is not generally on security, and thus all security countermeasures are seen, to some degree, as a drag on the bottom line. In contrast, the malfeasants of the world who are developing attacks are essentially engaged in enterprises whose focus is on security, and whose bottom line is bolstered by advancements in attack techniques and tactics.
This situation provides an interesting abstraction on the concept of asymmetric warfare. We've typically thought of information security as asymmetric, but from the perspective that 1-2 hackers can wreak havoc against an entire enterprise of hundreds or thousands. However, if you look at resource allocation and profit motive, you can actually flip the scenario such that you have 1-2 resources dedicated full-time (in excess of 40 hours/week, to boot) on developing and executing advanced attacks, while the average enterprise has a staff of dozens who are generally working 40 hours/week, but whose time is distributed across multiple projects and multiple fronts. To make matters worse, while many of those tasks assigned within the enterprise are necessary and important (e.g. firewall administration), they actually have a very low impact on preventing successful attacks.
Case-in-point, look at spam. If your organization has a mail server, then you probably receive spam. If your personnel are very active in the digital communications realm, then their email addresses are likely out there, and getting hammered with spam. Firewall administration really has very little to offer in preventing spam from infecting your mail server. The firewall does its job in limiting inbound access to only port 25/tcp, but after that, where's the additional value?
Expanding on this case, then, we see ourselves in an arms race with an enemy far more capable of investing and reinvesting in exponential growth of its program, while you really don't have that luxury. As such, a different tactic is much needed.
The Forklift Approach
Given that incremental changes generally result in a linear progression against an exponentially evolving attacker, it is then necessary to introduce leaps in capabilities over time in order to make up lost ground. If defensive technologies are able to grow effectively, the ideal is to be able to surpass attackers' capabilities over the short-term, buying time to put aside a "war fund" for the next leap project.
In graphic form, the following diagram shows the potential impact of using what I call the "forklift approach." Every few years, as the linear vs exponential progression reaches a significant degree of disparity, it is in the best interest of the enterprise to introduce a surge in capabilities in order to lift their organization to the next level. In this diagram, I've simply added +30 at year 4, +30+50 at year 8, and +30+50+70 at year 12.
One might wonder if this is a reasonable or sustainable approach. After all, the enterprise is already running on limited resources. The solution, I believe is make strategic use of temporary help in order to forklift the enterprise to a new level of operations. Temporary help can either be acquired through directing contracting of specialists to help with a project, or it can be a wholesale outsourcing to consultants of a project. In practice, what we're talking about here is bringing in external help every 3-5 years to help conduct a mass upgrade of infrastructure, policies, and practices.
Why a Surge of External Help?
Yes, it's true, I'm a consultant, and thus this post is a bit self-serving. However, there are a few good reasons to bring in outsiders with specialized experience to help forklift your enterprise to a new level of operational capabilities.
1) Consultants/Contractors are project-focused. One of the most common failings in the enterprise is expecting full-time employees to pick up an additional project on top of their existing work. With incremental changes, this approach is usually adequate, because the enterprise has accepted that it can only make small progress over time. In contrast, consultants/contractors are specifically geared toward project-based work, and the purpose of bringing them on-board would be to focus just on a specific project. External help like this can then accelerate a project realistically, achieving in weeks or months what would normally take several months to years.
2) Consultants/Contractors have less invested in the status quo. Another key challenge in introducing revolutionary and evolutionary change within the enterprise is that, frankly, change is hard to handle psychologically. Your existing personnel have much time and energy invested into the way things are, and to suggest that they now need to do things differently can be seen implicitly as an affront to all their hard work. Consultants/contractors do not generally suffer this same malaise. In fact, quite the opposite can be true, in that bringing in a fresh set of eyes can reveal areas for significant improvements as well as short-cuts for achieving those outcomes.
3) Consultants/Contractors bring specialized experience to the table. Another key challenge for full-time employees within the enterprise is continually upgrading their skills over time. Training can be an expensive component. While it's expected by employees that their employers will provide some professional development opportunities, there is an inherent contradiction in that incremental training opportunities will over time have limited utility to operations. Instead, a quick short-cut is to bring in specialists from outside who've already been trained in a given technology or process and rely on them to not only upgrade the enterprise, but to also pass along that new knowledge as part of the transition process. This point possibly flies in the face of established organization development opinions, but I believe that it's an important point. It's one thing to send your sysadmin to training on the latest version of an OS or database. It's another thing entirely to expect them to be able to go to training on a completely new technology or process and then be able to come back and implement it with minimal support. Consulting firms, in particular, are much better positioned for performing these types of projects, and if you structure the statement of work properly, you can also realize the forklift in skills within your internal staff along the way.
4) Consultants/Contractors have motivation to learn new technologies. Investing in new technologies is already an expensive proposition, but investing time and money into vendor presentations and training before you even choose or implement the product adds hidden costs that are rarely considered. Consultants/contractors, however, are very motivated to learn new technologies that can then be taken and applied in various organizations. Rather than invest into local fishing expeditions for new technologies, it may in the long-term be a better investment to identify the problem that needs solving and then bring in an external firm to present solutions, through even outsourcing the implementation, as discussed in #3 above.
The How: Build a War Fund
The biggest challenge to the proposed "forklift approach" is in how to fund it. There may be a number of different approaches, but what I submit makes the most logical sense is to establish a cyclical approach to wholesale improvements and then establish a "war fund" that is built up over the duration of cycle, expended at the end of the cycle, and then re-established. Initially, this planning will be rough, because there won't be much of a basis for estimating the costs. It's hard to know how much an organization will grow, or how much new technologies will cost. For a standard mid-sized organization, we could literally be talking about several million dollars every 5 years, in addition to routine operational costs.
The key is to protect the fund against raiding. It's one thing to jump the gun on a forklift project because the planned lifespan simply hasn't played out as planned. It's another thing entirely to keep going to well, robbing Peter to pay Paul. Budgeting and economics is certainly not be strong suit, but it seems to me that by taking a strategic planning approach with the realization that every 3-5 years a major overhaul and upgrade will be necessary will allow for adequate budgetary planning.
I Reject Your Reality and Substitute My Own
The objective of this post is to highlight an alternative approach to enterprise improvements than what seems to be the common practice of relying strictly on incremental improvements. In the end, it will ultimately be up to your organization to decide whether or not setting aside monies each year for a large upgrade every 3-5 years is feasible. Furthermore, it may make sense to have a large upgrade planned every year on a key sub-set of the enterprise environment, such that the recurring cost normally put into a war fund would simply become an overhaul line item. This approach does seem to be part of most practices each year.
Where I get concerned with an incremental or staggered approach is that there are many buried costs that get overlooked. Incremental improvements oftentimes are hindered by a resistance to change by existing personnel. Also, as Guy Kawasaki discusses in his posted "The Art of Innovation", it can be difficult to "jump to the next curve" from within. Simply put: when all you have is a hammer, then everything looks like nails, and this can be a real problem when what you really need to do is deploy a completely different connector tool that does not work with a hammer.
Overall, the really important point here is that every organization (including consulting firms) need fresh eyes every few years to ensure that what seems like progress is really innovation and advancement, rather than incremental changes in the same old tired thing. And don't forget that no matter how good your environment looks, somewhere else there is probably an even better environment. Just because your grass looks green compared to the dirt field that used to exist doesn't mean that it's as full and rich as it could be at the peak of its potential. Or, if you've peaked out in potential, it's time to find a new peak.
