SAS 70 and the Lies Auditors Tell

A Little Background

To understand SAS 70, you need to first understand that there are 2 types of audits with nuanced differences. I like Wikipedia's description, so here ya go:

"There are two types of service auditor reports. A Type I service auditor’s report includes the service auditor's opinion on the fairness of the presentation of the service organization's description of controls that had been placed in operation and the suitability of the design of the controls to achieve the specified control objectives. A Type II service auditor’s report includes the information contained in a Type I service auditor's report and also includes the service auditor's opinion on whether the specific controls were operating effectively during the period under review."

So, in a nutshell, Type 1 is combination paper- and interview-based exercise meant to identify and document controls in place, with the auditor's report provide an opinion on the presentation of the materials. As we'll note below, however, this opinion can be rather sketchy. The Type II report does actual tests of the effectiveness of the controls identified in the Type I report.

I've seen a Type I take anywhere from 3-9 months, depending on how mature the organization is in terms of documented controls, and depending largely on the tenacity of the auditor. A SAS 70 15 years ago was a vastly different beast than the one we see today. Moreover, for better or worse I've even now seen large tech organizations making use of a SAS 70 for internal audit (or, so they claimed), since in such large orgs your internal auditors are effectively 3rd parties. But is any of this based on reality?

The Standard's Got No Clothes

If you read the full text of SAS 70, the first thing you should notice is that it does not contain a controls framework. It is merely an audit standard that prescribes the process that must be followed when performing this particular type of audit. For those more familiar with PCI DSS, this reality will seem quite strange. An audit standard that provides no controls framework?

This dirty little secret of SAS 70 has been abused for more than a decade, particularly by the likes of people aligned with ISACA and the AICPA who have hidden agenda. The Type I report is meant to identify controls and then document auditor opinion on those controls. But what is the basis for this opinion? The standard has no defined controls, and was in fact released in April 1992! We are following an audit standard that was written well before eCommerce was a reality.

The sad fact is that, more often than not, organizations are being held to either a subjective set of controls for comparison or, worse, a framework that may not have been disclosed or may not have any relevance to their given environment (e.g. CObIT, COSO). Auditors generally bring to the table their experience, but sadly they are rarely current or former security practitioners, nor are they typically all that technical (though this is less the case today than 10 years ago). Especially if you're dealing with the Big N audit firms, it is not unusual to get extremely green and inexperienced auditors making recommendations without having an even remotely adequate background to provide such advice.

Instead, many auditors will cling to their control framework of choice - typically CObIT - and then try to jam that cross-wise down the throat of the audited organization, whether their guidance makes any sense at all. This is, in fact, part of a greater conspiracy by the organizations who train and certify these auditors to push their own mantras regardless of fit to the client organization. Why? For the bottom line, of course. The more closely your organization aligns to the auditor's audit framework, then the more easily the auditors can audit your organization, reducing the amount of time they spend on the audit, and thus increasing their operating profitability since most audits are performed on a flat-rate or long-term contract basis.

ISACA, AICPA, and SOX

One of the more fascinating aspects of the audit business, and a part that often rears its head during SAS 70 audits, is the background role of ISACA and the AICPA. This role first became most apparent to me when I encountered the initial push for Sarbanes–Oxley Act of 2002 (SOX) compliance audits. SOX is a very poorly written Act that makes vague hand-waving motions about requiring better internal controls over financial processes and systems. It was drafted hastily in the wake of Enron scandal and really serves as a warning of what can happen if industry leaves regulation to Congress in the event of an ethical collapse.

Fast-forward to the relevant section here... SOX Section 404... the requirement for annual reporting (by public companies) on the assessment of internal controls as pertain to financial reporting. Note the key qualifier here in that it highly limits scope. SOX has been taken by the Public Company Accounting Oversight Board (PCAOB) and the SEC and been turned into this rather egregious and nasty audit tarpit wherein publicly traded companies are now given very explicit requirements of duties to perform. In general, a risk-based approach must be used to managed these environments, with a risk assessment performed on a regular basis. However, if you listen to the ISACA/AICPA/PCAOB folks - who are oftentimes the same people, or at heavily allied - then you will see a general alignment around COSO for risk management, and ultimately a unified push toward control frameworks like CObIT.

The interesting thing here is that SOX 404 does not mandate these frameworks. In fact, SOX 404 doesn't mandate much of anything at all. Instead, in empowering the PCAOB, and by extension the AICPA and ISACA, SOX 404 has been used to try and force publicly traded companies to use COSO and CObIT, even if companies have other preferred approaches that better match their environments.

Given this psuedo-sanctioned pseudo-codification of COSO and CObIT under SOX 404 enforcement activities, it is often then further misconstrued that other audits, such as SAS 70, should be based on these frameworks. Of course, nothing could be further from the truth. Unfortunately, many auditors will implicitly use these frameworks as the basis for their audit guidance, oftentimes with dissonant results that steer organizations toward practices that may not make much sense in their given context whatsoever.

Auditor Independence

One of my biggest complaints with the role and empowerment of auditors today is the erosion of auditor independence. This attribute is in fact one of the key facets of SOX - one that is oftentimes downplayed by the auditors themselves. There are a couple key aspects to auditor important that are important.

First, on its face, external auditors must not have a vested interest in the financial performance of the company they're auditing. If they do have a vested interest, then a conflict exists wherein the auditor must choose between providing a favorable audit report that could positively affect financial performance, or providing a negative audit report that could negative affect financial performance. This aspect of the rule is fairly self-evident.

However, the second aspect of auditor importance - and one that is frequently overlooked or easily dismissed - pertains to ensuring that the auditors are not in fact running the organization that they are auditing. As a general rule, auditors are not security experts, nor are they business experts, nor are they policy experts, and so on. A good auditor will be expert in audit process, plain and simple, and will understand their role in the overall organization. Unfortunately, many auditors are ambitious and thus wish to transcend their role as "mere auditors."

Unfortunately, if your risk and/or IT organizations are driven or run by auditors (internal OR external), then you have a serious conflict of interest in-hand that must be addressed, because the auditors are effectively writing the rules that they are then testing. That is, the rules exist in a vacuum that most likely lacks any sort of contextual relevance. More importantly, without independence, you cannot rely on the opinion of the auditors because they are codifying their biases into the rules.

It is thus imperative that auditors have minimal vested interest in the performance of the organization. For this reason, I would caution the use of internal auditors without first putting in place very strict barrier to prevent them from having a role in writing the rules that they will later be asked to inspect as part of enforcement. This is not to say that internal audit does not have a role, especially in large organizations. Instead, it is to say that auditors should have their roles well-defined and intentionally segregated from the rest of risk management and operations to help ensure that their objectivity can be optimized.

It's All About Scope

Back to SAS 70... let's talk about one of its biggest weaknesses; a weakness that in fact applies to many audits, including PCI DSS. That weakness is scope. A standard practice in audit and assessment, and one that is particularly exercised with SAS 70, is to define a limited scope. Again, bear in mind that SAS 70 does not itself define a controls framework. As such, there is no real starting point or basis for audit. Instead, it's up to the auditor and the client to define what is to be audited. That scope can be as small or large as the client desires, with a particular tendency toward it being small. Sometimes 3rd party requirements will apply, such as contractual obligations, but otherwise the scope is purely discretionary.

The problem here, of course, is that a client can eventually claim that they've "passed" a SAS 70 audit while barely testing anything at all. And this is of course allowed and perfectly acceptable, though if the reality were to get out, then it may end up doing more harm than good.

At the same time, scope can be an excellent shield against auditor intrusiveness. SOX is again an excellent example. In the first year of compliance audits, it was not uncommon for the auditors to come in and tell organizations what would and would not be in scope. For the most part, this egregious violation of auditor independence resulted in massive audits that went well beyond the necessary scope. Even now I've seen cases where auditors will come in as part of the annual SOX audit and try to say that a certain area will be part of an "extra" value-add audit, even though that extra area is not part of the approved scope, and which has not really been prepared for auditor review.

As such, it is important for client organizations to spend a reasonable amount of time defining the audit scope before the auditors arrive, and then fight vehemently against any changes in scope that auditors may push that do not have direct bearing on the desired audit activities. Do not let your auditors push you around. You're on the hook for the scope - sometimes legally (as is the case with SOX) - and thus you need to be satisfied with the audit scope.

A Snapshot in Time

I think it's conventional wisdom now, but it bears citing nonetheless. Audits, as assessments, are merely a snapshot in time. And, let's be honest, a lot can happen in a short period of time, even in the largest, slaggiest corporations. Consider, for example, a recent client. They were undertaking remediation ahead of a Type II evaluation period based in large part on Type I findings. The problem, however, was that the entire operational leadership team had changed during the course of the Type I audit. Findings that had likely been relevant and accurate at the beginning of the Type I audit 4 months earlier were now being challenged aggressively because new leaders had come onboard and had taken a more charitable view of their own teams.

An audit or assessment is only accurate to the time at which it was performed. If it takes a month or two to close out the report, then there's a strong chance that things will change. This temporal limitation can be very vexatious in trying to perform remediation later.

Certify What?

One of the more annoying things I hear on a regular basis is that such-n-such organization has completed their "SAS 70 certification." Well, umm, no, that's not true. There is no such thing as a "SAS 70 certification." How could there be? As already noted ad nauseum, SAS 70 does not define a controls framework. As such, there's nothing to certify against. Contrast this with ISO 27001/27002, where a list of control objectives and controls are defined, and you will see the folly in claiming a SAS 70 "certification."

What you get from a SAS 70 is an auditor's report, plain and simple. Nothing more, nothing less. It's a report with opinions, either on the completeness of the in-scope controls, or on the operating effectiveness of in-scope controls, but it is still just a report.

Where I really choke on this notion of "SAS 70 certification" is when I find it in contracts. It's essentially a term that cannot be met. Lawyers and business people generally ignore the discrepancy and just do the needful in getting a SAS 70 completed, but if you wanted to really hinge on the fine details, this could be a real interesting test of competency. Personally, if I were writing contract terms, I think that I would be tempted to put in requirements like this just to test the competency of the other side as a basis for gauging whether we should even be working with them. Of course, IANAL, and there's probably a good reason for that... ;)

Final Thoughts

To wrap this up, here are the key take-aways for this post...
* There is a role for audit, but it must be limited and controlled.
* SAS 70 is just an audit process, it does not define controls.
* Beware auditor bias and false assertions.
* Beware attempts to under-scope or over-scope audits.
* Bear in mind the temporal limitations of all audits and assessments.
* In the end, audit is just one piece of the overall puzzle.