Exploring Hidden Attack Surfaces: Performing Better Content Discovery

Exploring Hidden Attack Surfaces is a new blog post series curated by BLACKBIRD Technologies for the community to help you on your way to discover any attack surfaces that most current tools are not capable of.

Part 2 is dedicated to performing better content discovery so that you can have that extra edge over other researchers and directly also increase your chances of finding more security vulnerabilities.

Part 1 was about finding more subdomains. Click here to read "Exploring Hidden Attack Surfaces: The Secret To Finding More Subdomains".

In the previous part of this post series, we went over the secret to finding more subdomains. The same concept applies to performing better content discovery.
To perform better content discovery, you need to make sure to combine as many data sources and methods as possible.

In this article, we will go over several methods that you can copy in your content discovery process to help you find even more links, URLs, files, app routes, parameters and API endpoints to help you expand your attack surface and increase the chances of catching a web vulnerability.

Keep reading as at the end of this post, I will show you a solution to help you automate almost all of the mentioned methods.

Crawling:

Most links and files are referenced on each web page that you'll come across. A basic web crawler is capable of finding all sorts of links in these web pages.

For example, here are some HTML elements to parse for links:

• A-Tags & Buttons
• Images, video's, SVG-files
• HTML Forms
• Iframes
• Meta-tags
• Script-tags (JS enumeration)

More advanced web crawlers are also capable of clicking buttons, posting forms and performing other events that would result in finding more links.

A more effective approach is using a headless web crawler as it can mimic a navigator's behavior. SPIDER X—BLACKBIRD Security Scanner's built-in content discovery scanner—uses a similar approach.
It navigates your target using a headless web browser and is also able to intercept every incoming and outgoing HTTP request which often yields better results.

Moreover, a headless web crawler is also able to parse elements that are dynamically rendered on the client side. This is often the case with targets that use some popular JavaScript frameworks and libraries for example.

Public Sources:

There are several public sources available that may have indexed your target at some point in the past. These indexed results often contain interesting links and references that you always want to check out.

A few sources may include:

• Search engines (like Google, Bing, DuckDuckGo, etc.)
• Public internet archives (like WaybackMachine, OTX Vault, PublicWWW, URLScan, etc.)
• Public developer platforms (like Github and Gitlab)

JavaScript File Enumeration:

JavaScript files are a gold mine for penetration testers and bug bounty hunters. They contain all sorts of references and must always be part of your reconnaissance process.

There are automated JS file parsers that can quickly read several JavaScript files and extract links, URLs, parameters and more.

It is also recommended to monitor these for changes. As changes often include newly released app routes or API endpoints that have just made it into production, it would be ideal to test them for web security vulnerabilities as well.

Forced Browsing (Bruteforcing):

Some links or files are not referenced anywhere, and that's where forced browsing more known as bruteforcing comes into play.

Forced browsing or bruteforcing is a technique where you directly request a certain path or file and observe the response. The main aim is to identify hidden or referenced links, files and/or parameters.

This is mostly done through an automated tool that loads up a wordlist with filenames and paths and requests each one of them.
As most web servers are configured to return a 404 status code for a non-existing resource, tools can easily distinguish whether they should mark a certain path as valid or not.

There is also a more sophisticated way of bruteforcing that SPIDER X—BLACKBIRD Security Scanner's built-in content discovery scanner—employs that's called targeted bruteforcing.

Targeted bruteforcing is a lesser-known technique that involves 2 separate steps. It is also a more effective approach and it likely also yields more accurate results.

The first step is to identify the technologies used by the target. The second step is loading a matching wordlist that then could be used for bruteforcing.

An example would be a website that is hosted on an IIS web server that is written in ASP.NET. Having a wordlist that is curated specifically for IIS targets will yield not only more but also better results in general.

Automation:

There are several open-source tools available that can automate some of the content discovery methods mentioned above. But as of now, none exists that covers all of them at once.

However, SPIDER X—BLACKBIRD Security Scanner's built-in content discovery scanner—can do all of that. With capabilities of:

• Headless web crawling
• Request/response intercepting
• Targeted bruteforcing
• JavaScript file enumeration
• Query & body parameter discovery
• And support for fetching public sources

SPIDER X is currently one of the most advanced closed-sourced tools on the market right now. If you would like to try it out on your own list of targets, try out a quick demo today!

This post was part 2 of the new small post series "Exploring Hidden Attack Surfaces". If you've learned something new feel free to share this post with your friends and/or colleagues!

BLACKBIRD Technologies is a security company that provides a cloud-based web app pentesting platform with tens of manual and automated web app pentesting tools, including a fully automated web vulnerability scanner!