Rachel Park and I, both seasoned professionals in ethical hacking and penetration testing, recently embarked on a revealing engagement. Commissioned by a nationally recognized client with a household name, our mission was to analyze over 20 penetration test reports from a diverse array of firms, encompassing both high-profile and lesser-known entities. In our inaugural collaboration with this client, the experience proved to be profoundly enlightening. This blog post seeks to articulate our reflections and convey the insights gained from scrutinizing a substantial number of reports crafted by our peer organizations and colleagues.
When it comes to hiring a penetration testing firm, many organizations seem to place a premium on brand recognition. The assumption is that a well-known name must equate to high-quality service. However, our analysis indicates that this is far from the truth. Many reports from these 'big-name' firms were disproportionately voluminous, often exceeding 100 pages for engagements with a relatively small scope. But when you dig into the details, a large part of these reports consisted of boilerplate language, filler content, and unactionable or generic recommendations.
This raises several critical issues:
So, the question that organizations must ask themselves is: "Are we paying for quality, or are we paying for a logo and a sense of false security?"
Our analysis revealed a troubling trend among pentesters—a lack of understanding about the vulnerabilities they discovered. This was evident through inconsistencies between the screenshots of vulnerabilities and the accompanying narratives. These discrepancies ranged from minor misunderstandings to significant issues, like a pentester being a step away from compromising an entire domain but marking the vulnerability as 'low risk.'
These inaccuracies pose severe problems:
One of the most concerning observations was the number of reports that were essentially vulnerability scans labeled as penetration tests. These reports mostly contained generic findings that could be uncovered by any open-source scanning tool and lacked any indication of a manual, in-depth analysis by a human.
This is a concern for several reasons:
If you're in the market for a penetration testing firm, it's imperative to do your homework well in advance. This entails more than just searching for companies with a high profile or impressive client list. Schedule meetings with potential firms to discuss their methodologies in depth. A reputable firm will be willing to share their standard operating procedures, the types of vulnerabilities they commonly encounter, and how they go about communicating these findings to their clients. During these discussions, be alert for red flags such as vague or generic answers. Evasive responses to direct questions about their methodology or past findings are often a sign that the firm may not have the expertise or transparency you're seeking. Due diligence doesn't stop at the service level; scrutinize client testimonials, and if possible, talk to past or current clients to gauge their satisfaction and the quality of work performed.
When discussing methodologies, it's essential to delve into the tools the firm employs for penetration testing. Ask them to list their favorite or most valuable full-scope penetration testing tools. While automated scanners like Tenable Nessus are helpful for initial vulnerability assessments, they should not be the only tools in a pentester's arsenal. If the firm mentions these as their primary tools without elaborating how they fit into a more extensive, manual testing framework, consider this a significant red flag. Experienced pentesters integrate automated scanning as a starting point but rely on a range of other tools and manual techniques for a comprehensive assessment. Their toolkit should include utilities for web application testing, network analysis, and even social engineering tests, all integrated into a strategic framework that goes beyond what automated tools can achieve.
Before you even reach out to potential penetration testing firms, it's crucial to have a clear understanding of what you're looking to achieve with the engagement. Is the pentest being driven by regulatory requirements, or is it part of a broader risk management strategy? Are you interested in a full-scope test that looks at your entire digital landscape, or are you focused on a specific web application or business process? Being clear about your needs will not only help you select the most suitable firm but also ensure that the engagement delivers real value. For example, if compliance is your primary driver, make sure the firm has experience with the specific regulations that affect your business. If you're concerned about a particular aspect of your digital infrastructure, look for firms that specialize in that area. Clearly defined objectives make it easier to communicate your needs effectively, ensuring a more successful outcome.
Though the process of conducting this analysis has been somewhat disconcerting, its value cannot be emphasized enough. The primary takeaway is the noteworthy variation in the quality and depth of penetration tests across the industry. As a client, the responsibility lies with you to ensure that the services you invest in align with your organization's specific needs in both scope and quality. It goes beyond the monetary aspect; it's about safeguarding your organization's digital assets, reputation, and, fundamentally, its future. Hence, maintaining vigilance throughout the process is crucial. Pose pointed questions about methodologies, tools, and past engagements. Be attentive to red flags like vagueness or evasion, and be willing to delve deeper when necessary. Most importantly, keep your organization's security requirements sharply in focus. By taking these proactive steps, you'll be better equipped to select a penetration testing firm that not only offers value for your investment but also delivers a truly effective service.
To read more from our experts on a range of security topics, check out our other blogs here.
Published By: Chris Neuwirth, Senior Penetration Tester and Rachel Park, Penetration Tester
Publish Date: October 18, 2023
Security news, tips, webinars, and more straight to your inbox.