I've finally had a chance to sit down and read "PCI Shrugged: Debunking Criticisms of PCI DSS" by Anton Chuvakin and Ben Rothke. My response to the article can also be construed as a follow-up to my earlier post, here.
Overall, I take great issue with the article and assertions made. My primary critique is that they're debunking "myths" or "criticisms" that are based in reality, not in abstract and obscure theoretical PCI realms. It does not matter what the PCI Security Council wants us to believe about the DSS. It only matters how the DSS is used and perceived by those required to comply with it. The simple fact is that the DSS is a checklist (see the SAQ for example) that has a very narrow scope (billing systems with credit card numbers) and that does not look at larger concepts like risk or maturity.
That being said, allow me to do a section-by-section rebuttal.
Complaint: PCI is a Distracter from Security and Risk Management?
So, here's the thing: PCI does raise the visibility and priority of security in certain organizations. But, PCI absolutely, positively does not do anything for risk management. Moreover, as a result, PCI is seen as a destination rather than a vehicle, resulting in the problems we see today wherein organizations claim "compliance" in spite of compromises.
"everything about PCI is core security"
--> This is, unfortunately, nowhere near true. PCI is generally focused only on systems and networks where cardholder data is stored, processed, or transmitted. In a reasonable organization, this is only a small slice out of the overall enterprise. While it is true that some provisions apply more broadly - or at least should be - it seems to be quite uncommon for this to happen.
"Previous data security efforts, such as Sarbanes-Oxley, have encouraged a check-box approach to compliance."
--> SOX isn't a data security effort, it's about corporate accountability, and assertions to the contrary are a perversion of SOX by organizations like the AICPA and ISACA, who had a direct vested interest in creating this confusion. There have not been other data security efforts - PCI DSS is a first and unique effort in this area. That it falls far short is not surprising given this truth, though in the modern age of information security we should expect more.
"organizations that have developed a formal information security program will find that PCI compliance is useful for security and not an onerous distraction"
--> PCI doesn't encourage building a security program or organization - it merely encourages compliance for those systems in-scope for the regulation. While some of the requirements may have broader applicability to organizations, it is then even more expensive to do so, and especially in the current economic environment, organizations are balking.
Let's put it plainly like this: in every quality process improvement initiative there is a short-term spike in cost in order to gain long-term efficiencies and cost savings. In the current economic environment - and even in the environment of the last 3-5 years - organizations are neither willing nor able to shoulder the additional burden. The average L1 merchant spent $2.7m in 2008 on PCI compliance, and the average L2 merchant spent $1.1m. Where did that money come from? Who's footing that bill?
"While an organization can attempt to pursue PCI DSS compliance for compliance sake without regard to security, such irresponsible behavior can hardly be blamed on PCI DSS standard itself."
--> It's called survivalist and minimalist behavior. What's the incentive here? Organizations are compelled to achieve compliance at the lowest cost possible. Most organizations are not Level 1 merchants, which means they're self-assessing. The card brands are not particularly incentivized to cut off merchants or processors or banks. Merchants are not particularly incentivized to burn millions on compliance with limited benefit. In fact, one could go so far as to argue that if the card brands are going to only fine non-compliant organizations in amounts less than US$100,000/month, then for the average L1 or L2 merchant, it's actually cheaper to be fined than to pursue compliance! (see this report for info on where the $2.7m and $1.1m numbers come from).
Moreover, let's look at the impact of fines. Say an organization is working on compliance, albeit slowly. Say that the PCI Security Council decides to enforce a fine. Does the fine not work against the compliance objective by taking money away from compliance efforts? Would it not be better that, rather than fine, organizations be required to foot the bill for Council-approved security management consultants to come in and fix things? Just a thought...
Complaint: Data Breaches Prove PCI DSS Useless?
This complaint is particularly interesting. Is a bad regulation useless? Probably not, but nor do I think most rational people are arguing that PCI is useless. It is, however, frustrating and distracting. It imposes a burden that does not accomplish the goals stated. The DSS does not align with the assertions made in the "PCI Shrugged" article, nor with the assertions made by Visa, MasterCard, or the PCI Security Council.
"One should not assume that compliance necessarily means that breaches can't occur."
--> This is a very good statement, and one that I support wholeheartedly. There is a gross misconception amongst non-security professionals that compliance means much of anything. However, this also raises the question: what is the point of the standard, or compliance? What is the benefit?
"A simpler explanation applies here: they were breached despite being PCI DSS compliant."
--> Yes, indeed. So, what was the point of achieving PCI compliance? The problem here is that PCI compliance does not secure the enterprise, but rather just the narrow environment containing the cardholder data. Moreover, "clever" scoping further infringes on the effectiveness or compliance.
"It is surprising to the authors that security professionals will hold the view that following an external guidance document can guarantee 100% security to any organization."
--> It's surprising to me that the authors would think any security professional would hold that view. There are no guarantees, especially in information security. Maybe somebody should share this with the PCI Security Council, because they don't seem to fully agree as they de-certify organizations that were previously declared compliant. Is the loss of compliant status due to bad QSAs, or because security is a moving target?
"In much the same way as a doctor cannot guarantee the health of the patient, neither PCI nor any other regulatory guidance can guarantee that there will not be breaches. 100% PCI compliance does not guarantee an entity is 100% secure or even as secure as they need to be."
--> If PCI does not ensure the security of organizations - not even reasonable security/assurance - then what's the point? The argument that "something is better than nothing" does not necessarily hold if that "something" doesn't actually solve any problems! Moreover, what is the problem being addressed? If you go to the doctor for a headache and then die outside the office from an aneurism, then that doctor may very well be culpable for missing the diagnosis.
Complaint: PCI is Just Security Theater?
"For many, PCI compliance means emptying their plates via yet another compliance checklist. They often do the bare minimum in the hope that they can gain compliance and make the QSA go away. At times they may even lie to their QSA or on the Self Assessment Questionnaire (SAQ)."
--> Lying is an interesting thought... what if the people completing the SAQ aren't qualified to complete it, but don't know any better? One of the biggest faults with the PCI DSS is that the majority of organizations subject to compliance are then self-assessing their degree of compliance. What is the incentive for organizations to report anything other than being most or fully compliant? Moreover, who is filling out the SAQ in the first place? What if these unqualified respondents really do think they're secure? It strikes me as incredibly unreasonable to believe or expect that every self-assessor with a SAQ in hand will actually understand half of what is contained therein. If QSAs can't get it consistently right, how can we reasonable expect self-assessors to be any better?
"Far too many organizations have an audit-based mentality with the frame of mind of evading the auditor, as opposed to a risk-based mentality of protecting the cardholder data."
--> Wait, I'm sorry, where does PCI mandate a risk management approach to securing the enterprise, or even to protecting cardholder data? Oh, that's right, it doesn't. It provides a prescriptive list of things to implement to protect cardholder data, and then provides a checklist questionnaire (SAQ) for self-assessment (or auditors - aka QSAs - for Level 1 merchants). Of course organizations have an audit-based mentality - that's what they've been brought up to expect and believe! This notion is further reinforced by requiring them to hire auditors.
"PCI DSS is a good start of a security program, not its end. Checklists do have their place in security, but a security program cannot be reduced to a checklist; attempts to pretend that an organization can 'follow the checklist to become secure' are guaranteed to fail."
--> Duh. Except that "good start" part. The place for a checklist is in procedures that describe in detail how to execute a specific task. Unfortunately, the PCI DSS reads like such a thing; it's a checklist for securing your cardholder data. It is not a standard for building a security program, and in fact it even wanders outside the lines of security at points, such as with configuration management. Unfortunately, all of its guidance can be applied in such a narrow scope that there are limits to the overall usefulness of the requirements.
What the Future Holds
"Not only is PCI not dead, it is alive and well and maturing. In its current version 1.2, it is still evolving, but it is clearly the best we have."
--> It's not the best we could have, and that's my point. The "it's the best we have" or "it's as good as we could do" arguments are simply unacceptable. The simple fact is that the PCI DSS was written by people who were not properly qualified for the effort. It's written like an audit checklist precisely because it was authored by auditors! The fundamental problem with the PCI DSS is that it is now being described as something that it was never written as in the first place. It's not a security management standard - it's a cardholder data security standard. Accept it and get over it. Have a problem with this? Great - so do I! The PCI DSS is definitely not the best we can do, but to improve it will mean completely rewriting it from scratch. (btw, I'd love to be involved in PCI DSS v2.0 that completely rewrites the darned thing)
"The authors challenge anyone to find a better standard or regulation."
--> Just because you can't find something better today doesn't mean there can't be something better. I would, however, argue that there are lots of models and frameworks that are much better suited to securing the enterprise, whether that be *ahem* the TEAM Model or SSE-CMM or FAIR or even the ISO standards. The point here is that we know how to write good standards, so why are we settling for this thing (PCI DSS)?
Conclusions
"Most attacks against PCI boil down to we don't like it or PCI is useless, rather than a direct critique of the standard, or ways in which in can be improved. "
--> This is patently false. Many of us have provided very specific critique of PCI DSS and have provided very specific suggestions for improvement, and yet nobody on the PCI Security Council is listening. If you're not aware of specify critiques or recommendations, then you're probably not paying close attention to infosec discussions in this area.
"PCI has taken the masses of security illiterate companies and forced many of them into some semblance of security."
--> Again, patently false. It has spawned nothing short of blatant lies and security theater. Lots of hand-waving to give the aura of security with nothing behind it. PCI is a prescriptive standard designed to protect card brands, passing the buck to the merchants. Why is it the responsibility of the merchants? Why isn't it the responsibility of the card brands themselves? Moreover, again, what is the incentive for either party? The only reasonably expected outcome here is security theater.
"PCI is not perfect; but neither is the world in which we live."
--> Oh, I see, the old "fallible humans" argument. So, we're supposed to accept this garbage because "well, golly, humans are imperfect." The authors admin that the PCI DSS does not secure the enterprise. In practice, we've found that it also does not consistently secure cardholder data. It puts a burden on merchants who are not particularly incentivized to do anything with it. It's not the merchants' data, so why is it their responsibility to secure it? The entire model is broken. This is where we need to see major improvements before we see anything further from the PCI Security Council. Fix the card transaction/processing model, and then see what's left to secure. If done well, the merchants should have very little role or responsibility, and the risks associated with cardholder data breaches should decrease dramatically.
