Lessons Learned from Analyzing 20+ Penetration Test Reports: A Critique and Reflection

Rachel Park and I, both seasoned professionals in ethical hacking and penetration testing, recently embarked on a revealing engagement. Commissioned by a nationally recognized client with a household name, our mission was to analyze over 20 penetration test reports from a diverse array of firms, encompassing both high-profile and lesser-known entities. In our inaugural collaboration with this client, the experience proved to be profoundly enlightening. This blog post seeks to articulate our reflections and convey the insights gained from scrutinizing a substantial number of reports crafted by our peer organizations and colleagues.

Reflections on the Reports:

The Price of a Name: The Illusion of Security Through Brand Recognition

When it comes to hiring a penetration testing firm, many organizations seem to place a premium on brand recognition. The assumption is that a well-known name must equate to high-quality service. However, our analysis indicates that this is far from the truth. Many reports from these 'big-name' firms were disproportionately voluminous, often exceeding 100 pages for engagements with a relatively small scope. But when you dig into the details, a large part of these reports consisted of boilerplate language, filler content, and unactionable or generic recommendations.

This raises several critical issues:

• Resource Drain: Delving into extensive reports exceeding 100 pages, often filled with boilerplate language and generic content from well-known penetration testing firms, proves to be a time-consuming task. This exhaustive process demands substantial manpower for analysis, diverting resources that could be more efficiently employed elsewhere within the organization.

• Quality Over Quantity: While comprehensive reports are valuable, the sheer volume can compromise clarity and specificity. The excessive length of these reports, often laden with filler content, undermines their effectiveness, making it challenging for clients to distill actionable insights and prioritize necessary actions.

• Unactionable Recommendations: Recommendations within these voluminous reports frequently suffer from being too broad or generic, rendering them practically useless. This lack of specificity can foster a false sense of security, as the suggested actions may not be tailored to the actual findings, potentially leaving critical vulnerabilities unaddressed.

So, the question that organizations must ask themselves is: "Are we paying for quality, or are we paying for a logo and a sense of false security?"

Lack of Understanding: The Danger of Inaccurate Reporting

Our analysis revealed a troubling trend among pentesters—a lack of understanding about the vulnerabilities they discovered. This was evident through inconsistencies between the screenshots of vulnerabilities and the accompanying narratives. These discrepancies ranged from minor misunderstandings to significant issues, like a pentester being a step away from compromising an entire domain but marking the vulnerability as 'low risk.'

These inaccuracies pose severe problems:

• Misleading Information: Inaccurate assessments have the potential to misguide organizations about their true security standing, creating a dangerous scenario where critical vulnerabilities might be downplayed. This misrepresentation can hinder the identification and prompt mitigation of serious security threats.

• Credibility Concerns: Glaring errors in pentesting assessments not only compromise the credibility of the testing firm but also cast shadows on the broader cybersecurity industry. The trustworthiness of the entire sector is at risk when such inaccuracies occur during security evaluations.

• Security Risk Mismanagement: Incorrectly classifying the risk level of a vulnerability can lead to the inappropriate allocation of resources, leaving critical systems exposed and vulnerable. This mismanagement poses a tangible threat to the overall security posture of an organization.

Vulnerability Scans Masquerading as Pentests: The Deception

One of the most concerning observations was the number of reports that were essentially vulnerability scans labeled as penetration tests. These reports mostly contained generic findings that could be uncovered by any open-source scanning tool and lacked any indication of a manual, in-depth analysis by a human.

This is a concern for several reasons:

• Insufficient Depth: While automated scans have their merits, they possess limitations that underscore the value of a seasoned penetration tester. Relying solely on vulnerability scans confines you to predefined parameters and restricts exploration to the limits of the scanner’s (or vendor’s) database. The refined skills of an adept professional hacker surpass these limitations, delivering actionable insights.

• Insufficient Detection: Automated scans, despite their advantages, fall short in recognizing "acceptable" Active Directory configurations exploitable by an experienced threat actor. Identifying attack pathways formed by chaining multiple misconfigurations together demands the nuanced understanding of a human expert in the field.

• Misleading Confidence: Organizations may cultivate a false sense of security, assuming a thorough assessment with automated scans, when, in reality, these tools merely scratch the surface. A comprehensive evaluation requires the expertise of a skilled professional capable of uncovering deeper vulnerabilities.

• Compliance Concerns: Sole reliance on automated scans for security assessments may lead to regulatory challenges, especially if manual testing is a mandated requirement. Ensuring compliance demands a balanced approach that integrates the insights of human testers alongside automated tools.

Lessons Learned and Recommendations:

Do Your Homework: The Importance of Due Diligence

If you're in the market for a penetration testing firm, it's imperative to do your homework well in advance. This entails more than just searching for companies with a high profile or impressive client list. Schedule meetings with potential firms to discuss their methodologies in depth. A reputable firm will be willing to share their standard operating procedures, the types of vulnerabilities they commonly encounter, and how they go about communicating these findings to their clients. During these discussions, be alert for red flags such as vague or generic answers. Evasive responses to direct questions about their methodology or past findings are often a sign that the firm may not have the expertise or transparency you're seeking. Due diligence doesn't stop at the service level; scrutinize client testimonials, and if possible, talk to past or current clients to gauge their satisfaction and the quality of work performed.

Question the Tools: Beyond Automated Scanners

When discussing methodologies, it's essential to delve into the tools the firm employs for penetration testing. Ask them to list their favorite or most valuable full-scope penetration testing tools. While automated scanners like Tenable Nessus are helpful for initial vulnerability assessments, they should not be the only tools in a pentester's arsenal. If the firm mentions these as their primary tools without elaborating how they fit into a more extensive, manual testing framework, consider this a significant red flag. Experienced pentesters integrate automated scanning as a starting point but rely on a range of other tools and manual techniques for a comprehensive assessment. Their toolkit should include utilities for web application testing, network analysis, and even social engineering tests, all integrated into a strategic framework that goes beyond what automated tools can achieve.

Be Clear About Your Needs: Define Your Objectives

Before you even reach out to potential penetration testing firms, it's crucial to have a clear understanding of what you're looking to achieve with the engagement. Is the pentest being driven by regulatory requirements, or is it part of a broader risk management strategy? Are you interested in a full-scope test that looks at your entire digital landscape, or are you focused on a specific web application or business process? Being clear about your needs will not only help you select the most suitable firm but also ensure that the engagement delivers real value. For example, if compliance is your primary driver, make sure the firm has experience with the specific regulations that affect your business. If you're concerned about a particular aspect of your digital infrastructure, look for firms that specialize in that area. Clearly defined objectives make it easier to communicate your needs effectively, ensuring a more successful outcome.

Though the process of conducting this analysis has been somewhat disconcerting, its value cannot be emphasized enough. The primary takeaway is the noteworthy variation in the quality and depth of penetration tests across the industry. As a client, the responsibility lies with you to ensure that the services you invest in align with your organization's specific needs in both scope and quality. It goes beyond the monetary aspect; it's about safeguarding your organization's digital assets, reputation, and, fundamentally, its future. Hence, maintaining vigilance throughout the process is crucial. Pose pointed questions about methodologies, tools, and past engagements. Be attentive to red flags like vagueness or evasion, and be willing to delve deeper when necessary. Most importantly, keep your organization's security requirements sharply in focus. By taking these proactive steps, you'll be better equipped to select a penetration testing firm that not only offers value for your investment but also delivers a truly effective service.

To read more from our experts on a range of security topics, check out our other blogs here.

Published By: Chris Neuwirth, Senior Penetration Tester and Rachel Park, Penetration Tester

Publish Date: October 18, 2023