SOC 2 Challenges – Population Completeness

By Christopher Moschella, CPA, CISA, Risk Advisory Services Senior Manager

SOC 2 Challenges – Population Completeness

By Christopher Moschella, Risk Advisory Services Senior Manager

This is Part 2 of a two-part series on the two biggest challenges we see most clients face in SOC audits.

These concepts are equally applicable to other types of SOC audits including SOC 1, SOC 3, and SOC for Cybersecurity.

Generating complete populations during the SOC 2 audit

In Part 1 of our series on SOC 2 challenges, we discussed common challenges related to generating evidence to demonstrate that certain control activities occurred. When testing controls, auditors need a population of events from which to draw a sample. The auditors will then test the evidence for each sample.

Auditors are required to obtain complete populations from which they can draw the sample. The completeness can sometimes be very simple or very complex.

Simple Populations

The populations for controls that occur on a set periodic basis, e.g., daily, weekly, etc., are easy to generate. If the control occurs once each week, the population is simply the weeks of the year, and the auditor will randomly select a certain number of weeks and request the control evidence for the selected weeks.

Complex Populations

Other controls do not occur on a set schedule. Rather they occur as needed. For example, a control over the timely deactivation of computer access for a terminated employee occurs when a termination occurs. As a result, the auditors would request a list of all terminations that occurred during the examination period. From that list, the auditors will draw sample and test the underlying evidence to ensure the access was deactivated timely.

With complex populations, the auditor is required to gain comfort over the completeness of the populations. That is, they need to obtain evidence that everything that should be in the population is in the population.

Generating Populations from a System of Record

Auditors always prefer populations that are generated from a computer system rather than from manually maintained lists. For example, to test computer access for terminated employees, the auditor is likely to request the population be generated from an HR system, rather than from a manually maintained list of terminations.

There are, however, circumstances where a manual list is the only conceivable way of maintaining a population. Such as, if a company issues physical keys to employees, and retains a list of people with the keys. In those circumstances, the auditor will likely be able to rely upon a manually maintained list provided that the entity can demonstrate that access to modify the list is limited to appropriate staff.

Avoiding Bias in the Population

Depending on the system from which the population is generated, the population may contain bias and therefore be unusable to your auditor. Let’s revisit our example of the population to test the timely deactivation of access for terminated employees. Organizations may identify two systems from which their auditor could draw their population: the payroll system or the ticketing system from which terminations are communicated to the IT team.

In this example the payroll system is likely to be freer of bias than the ticketing system. This is because the control itself starts within the ticketing system by initiating the communication via a ticket. If we select our population from the ticketing system, then we are excluding the possibility that an employee was terminated and a ticket was never created. This creates inherent bias in the population in the favor of passing the test. The better data source would therefore be the payroll system.

Obtaining/Observing Query Parameters

In addition to the population itself, your auditors will need to document how the population was generated. After all, if you just gave them a spreadsheet, they would not know what instructions were provided to the computer to generate the population, which creates doubt about the completion of the population.

For example, many reports allow you to pick the start date and end date indicating the time span from which activity which should be captured in the report. Let’s look one last time at our terminations example. If the audit period occurs from January 1 to December 31, and you provide the auditor a report that stops at October 31, then the population is missing an entire quarter of data.

Auditors need evidence of completeness for their workpapers, and your system needs to be able to capture this information. This evidence can come in a few forms:

  • Screenshots of query parameters: If the system has a user interface where you can specify the query parameters, the auditors will likely require an image of the parameters you used to generate the report.
  • Copy of the query language: If a custom query needs to be used to create the report, the auditor will likely want a copy of the query language used to generate the report so that they can verify the results in the report were not inappropriately excluded.
  • Observation of the query: If the previous options will not work in a given circumstance, they auditor may need to observe the process of where the report is generated.
Difficult Populations

There are certain types of populations that regularly create challenges for service organizations.

  • Software Changes – Companies that create software, either as an outsourced developer or as part of a software-as-a-service (SaaS), will need to generate a list of production software builds and/or production deployments. The list should be such that each item on the list can be agreed to appropriate request documents, testing, approvals, etc.Ideally, the population will be generated as close to the build/deployment process as possible and the individuals with access to deploy/build should not also have the ability to alter the logs generated during the deployment/build process.Depending on the organization’s software development processes, certain events in the source code control system could be used as a population as well.  However, we recommend speaking with your SOC 2 services provider to discuss this in more detail.
  • System Changes – Companies frequently need to make non-programming changes to production environments, frequently in response to customer requests. For example, a service provider who hosts a web application may be asked to whitelist a new client IP address in the firewall.  In such circumstances, the company should be able to generate a list of firewall configuration changes from the firewall, which generally requires optional logging functions to be enabled and for reporting functions to be built in.
  • Access Changes – Most systems, in our experience, are able to generate reports that indicate when an account was created and when an account was disabled. Many systems, however, lack the ability to generate lists of changes in access that occur throughout the year. This includes one of the most common systems in use, Microsoft Active Directory.  As a result, many service organizations implement an Active Directory auditing tool that is able to log and report on changes in access throughout the year.Service organizations who do not have the ability to create a report of these access changes generally need to create access reports periodically throughout the year, usually weekly, and stitch them together to create a list of access changes.  This is time consuming for both the service organization and the auditor, and if possible, should be avoided by using native reporting tools.

For some of your SOC controls, your auditor will need a population of events that trigger the control activity from which they can draw a sample to test. Additionally, the auditor has a requirement to obtain comfort that the populations provided to them are complete. This comfort is typically derived from the service organization having systems with sufficient reporting capability to generate complete reports. However, generating complete populations is often trickier than you might think. It is critically important for organizations to think about their controls and how they would generate both a population for the auditor and to demonstrate to the auditor that the population generated is complete.

Keiter provides SOC Exam and Exam Readiness Services for all types of SOC exams. Our SOC services are provided by our Risk Advisory Services Team, which is led by experienced CPAs, CISAs, and technology professionals with industry experience. Our Risk Advisory Teams services a variety of industries and business types.

Please contact us to discuss your company’s SOC needs. Risk Advisory Services Team | Email | Call: 804.747.0000

SOC 2 Challenges – Audit Evidence

Additional SOC Resources

Share this Insight:

About the Author

Christopher Moschella

Christopher Moschella, CPA, CISA, Risk Advisory Services Senior Manager

Chris is a Senior Manager in Keiter’s Risk Advisory Services. Chris has a strong combination of IT skills, which range from IT audit and internal control assessments, including general computer controls and application controls, to full stack web development. Most recently, Chris developed a Cybersecurity web application that assesses an organization’s resistance to social engineering attacks. Chris shares his cybersecurity insights on our blog.

More Insights from Christopher Moschella

The information contained within this article is provided for informational purposes only and is current as of the date published. Online readers are advised not to act upon this information without seeking the service of a professional accountant, as this article is not a substitute for obtaining accounting, tax, or financial advice from a professional accountant.


Contact Us