VAs, Scans and PenTests; not the same thing

I’d like to define a couple of subjects that seem to be confused often in the industry of application security; Vulnerability Assessment (VA), Vulnerability Scan (VA Scan) and Penetration Test (PenTest). They are often used interchangeably, and the differences do not seem to be well-understood; I have seen this misunderstanding used against many clients who have purchased these services and am hoping clear definitions will help us all.

A Vulnerability Assessment (VA) (sometimes called a security assessment) is an assessment of the security of a system, in attempts to find all possible vulnerabilities. It generally involves using multiple scanning tools, manual exploration and evaluation, as well as examination of all security controls (a lock on a door, a login screen, or input validation are all security controls). The Assessor does not exploit vulnerabilities that are found (for instance they see the door is unlocked, but they do not enter), they just report them, along with information on how to fix each of the vulnerabilities. This sometimes includes a security review of the design and/or threat modelling, questionnaires or interviews, and generally takes days or weeks, not hours or minutes. Sometimes the security assessor will create a proof of concept (POC) to explain a vulnerability with more clarity, but to be clear, that is not the focus of this exercise.

In the past when I was hired to do a penetration test, I would often describe a VA, as if that’s what they wanted, and they would say “yes, do that”. My contract would say “PenTest”, but I would conduct a vulnerability assessment.

In the past I often had requests for “a quick VA” or “VA Scan”, which as it turns out meant “one scan with a vulnerability assessment tool” and no other activities, such a manual investigation of the results. This can be done in as few as a few hours or even minutes if your target is small, and the person performing the task does not need advanced training or skills to perform the task. There are many VA tools on the market; Nessus, Nexpose, OpenVAs and Azure Security Center (for Azure cloud infrastructure only) are all used for scanning infrastructure, while Microsoft Security Risk Detection, Burp Suite, Zed Attack Proxy, NetSparker, Acunetix, AppScan, and App Spider are for scanning web apps. Doing “a quick scan” with any of these tools will net you a list of vulnerabilities, and many of them will be true positives (as opposed to false positives); it is most certainly a worthwhile venture. It is not, however, as thorough as a Vulnerability Assessment or Penetration Test, and there will remain many other issues that are not uncovered if you leave it at that.

A Penetration Test is another beast entirely. A PenTest seeks to find vulnerabilities and then exploit them, to prove real-world risk. Sometimes penetration tests can cause damage (exploits, if not done very carefully, can leave a mess), and sometimes the scope of a PenTest can call for the tester to collect “trophies” to prove they did the things they claim.

It is very rare that I write an exploit or feel the need to exploit vulnerabilities I find when testing*. Most of the times in my career when I have exploited something everyone just ended up pissed off at me; from the first PenTest I ever did as a sub-contract when I ruined a live prod server and the person that hired me had to explain what happened, to creating proof of concept exploits that embarrass management into doing “the right thing”, to breaking a Drupal CMS implementation so badly that they had to restore the database AND the app server (Drupal CMS itself was completely unusable) from backup. It’s nice that I impressed people, but I honestly would prefer to spend that extra time helping the developers fix what I have found and re-testing the fixes, rather than showing off whatever talent I have for burning things down.

Special note on ethics: I have seen many consultants who offer these services pass off a quick scan as a full VA or Pentest, charging for 10 days what took them only 1 day to perform. I have also seen many of these same consultants sub-contract this work out to others who they pay less (and with who they share your sensitive data with!), but they do not credit these individuals in the reports or contracts resulting in you having no idea who had access to your systems and data. When writing contracts for such services it would be wise to be explicit in what you are paying for, as well as who will do the work and what information must remain confidential. I am sad to report that I have met many consultants who have bragged about doing these types of (in my opinion) unethical practices. Buyer beware.

I would suggest that performing a proper VA against all of your custom applications as well as large COTS implementations (Customizable Off The Shelf system, such as SharePoint) is a best practice for Enterprise businesses. Not only would you be amazed at the things that you find, (assuming you fix the issues) you will have taken serious measures to avoiding a data breach in the future, as insecure software is still, sadly, the top reason for data breaches (as per the Verizon Breach Reports 2016, 2017, and 2018).

I hope this article helps instil a bit of clarity in our industry.

For this and more, check out my book, Alice and Bob Learn Application Security and my online training academy, We Hack Purple!

** When I did testing, I did exploit XSS using alert boxes, regularly, because it’s 100% safe to do so. And also blind SQL with timers and errors, but to be clear I am very careful to only perform safe exploits when testing. I can feel myself putting my foot right into my mouth with this note…