Domain discovery techniques and Recon-ng automation [part 1]

Hello Readers !

Recon is extremely important in pentest jobs. As an example, using recon and public information you can find open TCP ports of your targets without never scanning it...

Today we want to share with you all the techniques we use while mapping subdomains of a domain name.🌵

What we want to achieve :

1. Discover as much DNS entries for a domain [example.com] as possible and their associated IPs using various techniques;
2. Manage how we expand the scope not to fall out of scope totally, we need to have a reasonable ground to integrate new targets;
3. If possible, discover domain vulnerabilities while reckoning;
4. Try to make everything automatic ...


Let's start our tentative to list all techniques.

1. Subdomains discovery techniques

Several tools exist per technique so we wont show all existing tools but only the ones we deem useful or important. As it might be useful to store and organize results, ✅ we will try to present all techniques within the recon-ng framework as much as possible, as the framework is storing results and can be later exported under various formats.

Please install custom additional modules if you want to organize everything in recon-ng and not miss some techniques there. You can create a "custom" folder in /usr/share/recon-ng/modules and copy the modules there. They will be loaded during recon-ng restart. Then you can use them inside with "use custom/abc". 
 
We will use the following pattern to present each technique:

Technique name 
Description, explanation of the technique 

Tools command example

AXFR Zone transfer
This technique gives access to all records if one of the DNS server is mis-configured. If this is working, you can stop here since you have all existing records.

recon-ng custom recon/domains-hosts/axfr module
dnsrecon -t axfr -d example.com, outside of recon-ng dnsrecon.py is a robust dns enumeration tool which include lots of techniques listed here  

 

Brute force
We want to discover existing records by trying all of them using a word-list, such as a.example.com, b.example.com and so on... 🐻 

recon-ng recon/domains-hosts/brute_hosts module, a better wordlist than default might be needed
subbrute.py example.com, subbrute.py uses an innovative technique to be slightly faster than standard brute forcer (it queries multiple recursive resolvers with multi thread at the same time) 

Databases / Data repos
This can include a lot of sources. We want to query databases created by third parties to find new records. We will use as filter : domain include example.com. 

recon-ng recon/domains-hosts/hackertarget (database)
recon-ng recon/domains-hosts/netcraft module (database)
recon-ng recon/domain-hosts/builtwith module (database)
recon-ng recon/domain-hosts/threatcrowd module (passive DNS)
recon-ng recon/domains-hosts/shodan_hostname (database) (requires API key)
scans.io databases, download full DNS dataset and use jq (json query) on it
aquatone-discover -d example.com, new tool in town, it queries several databases at the same time without too much config

 
SPF Record
We collect extra domain and do reverse PTR query on the IP ranges specified in the Sender Policy Framework (SPF) which are usually trusted IPs of the domain.

recon-ng recon/domains-hosts/mx_spf_ip module  
dnsrecon -s -t std -d example.com

Reverse Whois
Using whois on the domain, we cross check other existing IP ranges with the same contact emails. 

recon-ng custom recon/domains-domains/threatcrowd_domain module
whois example.com | grep -v registrar | egrep [emails regex]
and curl http://viewdns.info/reversewhois/?q=email, reverse whois are usually not free, this one is. 

Search engines crawlers 
We rely on the data indexed by the search crawlers to find new domain name. 🐛

recon-ng recon/domains-hosts/google_site_web module
recon-ng recon/domains-hosts/bing_domain_web
sublist3r.py -d example.co, has some query to additional crawlers like Ask or DnsDumpster, simple

Certificate names
We check certificates to detect new subdomain names, for instance there are 2 certificates matching example.com, ssl1.example.com and ssl2.example.com.  

recon-ng recon/domains-hosts/certificate_transparency module

SSL SAN Alternative names 
We check a certificate alternatives names for other valid domain name, for instance the SAN subject alternative name can include anotherssl.example.com. 

recon-ng recon/domains-hosts/ssl_san module

Once you have gathered all these domains entries, you might need to organize them with corresponding IPs (for instance in CSV format) and remove duplicates (sort -u). Now that you have a list of IPs  you can apply another "transform round" (maltego style 🐈) and techniques on them to get more subdomains, such as:  



Whois
Using IPs found, we can expand to ranges CIDR / Netblocks from Whois databases. Warning! this can increase greatly or break the scope. You need to use it with care, as you don't want to expand to Amazon AWS  or Microsoft netblocks and companies.

recon-ng custom recon/hosts-netblocks/arin
dnsrecon -d example.com -w

Resolving IPs for all domains 
Simply resolving IPs for domains found without IPs

recon-ng recon/hosts-hosts/resolve module

Cohost / Vhost shared IPs 
Using third parties databases again, we want to check other vhosts from example.com which resides on our IPs list 

technique not available in recon-ng although you can cross check records 
curl hackertarget API (database), one request gives you all vhosts 
scans.io database, download full DNS dataset and jq based on IPs, then grep hostname

Reverse PTR
We check if the associated name to the IP gives a new domain name in the scope, such as 1.2.3.4 PTR newptrhost.example.com

recon-ng recon/hosts-hosts/reverse_resolve module
recon-ng recon/netblocks-hosts/reverse_resolve module 

dnsrecon -r 1.2.3.4

Now if you want to check other domains to expand the scope:


Resolve PTR and add new domain
We resolve all found IPs and new domains. Warning! this can greatly extend or break the scope, such as adding google.com in targets.

recon-ng recon/hosts-domains/migrate_hosts

Other TLDs similar domains
We query similar domain names with another Top Level Domain TLD, such as example.org. This is slow. 

recon-ng recon/domains-domains/brute_suffix module 
dnsrecon -d example.com -t tld

That's it ! Think we are missing a technique ? Feel free to add it in comment.

 

2. Scripting recon-ng for automated scan

We love recon-ng. 💓 It can be scripted really easily. Let's create a script which takes a domain as input and run all the above mentioned modules, then export results to CSV file. As for hosts information gathering, we did not deem necessary to use API with keys services. So the script can run without any API keys.

First part, we create a new workspace, select it and add the domain target, then run modules and export. The script assume you have the custom scripts in "custom" directory, such as /usr/share/recon-ng/modules/custom. The full script can be found on gist.

[...]
{
# prepare workspace
echo "workspaces delete $domainattacked";
echo "workspaces add $domainattacked";
echo "workspaces select $domainattacked";
echo "add domains $domainattacked";
# run modules in blog post order
# axfr
echo "use custom/axfr" ;
echo "run"
# brute
# [...] # report
echo "use reporting/csv";
echo "set FILENAME $fileoutput.csv";
echo "set TABLE hosts";
echo "run";
} > script.reconng
# and now run it
recon-ng --no-analytics -r $(pwd)/script.reconng 


Now anytime you want to recon a domain, you can simply run the script "./domainrecon_with_reconng.sh goblinsecurity.net recongoblin" and wait for results. The output is a CSV file presenting domain / IP and source of information.

"www.goblinsecurity.net","92.222.9.113","","","","","brute_hosts"

3. Domain vulnerabilities

Would not be it cool to gather extra information on vulnerabilities for the domain while we are scanning ? 🎰

NXDOMAIN / Subdomain takeover vulnerabilities

This vulnerability received quite a lot of attention lately. The basis of it is : the DNS admin forgot to remove a CNAME record which is now pointing to a domain which an external attacker can register.

not available in recon-ng
aquatone-discover -d example.com && aquatone-takeover -d example.com, aquatone is simple and does the job

 

Weak SPF, spoof emails without (?) detection 

Emails can ALWAYS be spoofed. Please read this again and remember it. Whatever you do, the from: field can always be spoofed. 🐇 However, we can, using SPF record in DNS TXT, detect unauthorized senders. Sometimes, these SPF records are not well configured. 

not available in recon-ng
dig +short example.com TXT | grep spf --color, output example.com.        53    IN    TXT    "v=spf1 -all"

You need to check that no "~all" or "-all" is present or that a "+all" exists in SPF. The -all means no one is an authorized sender for this domain (bar the exceptions before this statement). "~all" is also interesting because it is called SoftFail and sometimes mail servers do not tag them as SPAM, dangerous or phishy. Finally, +all means everyone is an authorized sender, which is kinda dangerous tag.

In this case, there is a good probability that your phishing email wont be detected as SPAM, dangerous or phishy by the mail servers and so by the clients. That way they wont be filtered or detected and increase your chance. For instance, looking at source of the email, you will see in headers:

received-spf: None (EXCHANGE-EDGE.[...]) 

Public vulnerabilities databases

Some websites maintain public list of web vulnerabilities, allowing you to find vulnerabilities just by querying. 

recon-ng recon/domains-vulnerabilities/xssed
recon-ng recon/domains-vulnerabilities/punkspider
recon-ng recon/domains-vulnerabilities/xssposed

Dorking

Dorking (from "Google dorking") is the art of finding vulnerabilities or interesting results using only search terms and keys. As Google tries to limit bots making dorks request, you need to assist the script in resolving captcha while running google hacking databases search terms.

recon-ng recon/domain-vulnerabilities/ghdb
phantomjs dorks.js google -D all_dorks.txt -s "example.com" -o result.txt, dorks.js does the job outside of recon-ng, still need to resolve captchas

This technique can be transposed to other search engines too, such as github.

recon-ng recon-ng recon/companies-multi/github-miner, then 
recon-ng recon/profiles-repositories/github_repos , then
recon/repositories-vulnerabilities/gists_search
recon/repositories-vulnerabilities/github_dorks

python github-dork.py -u dev-nepal, does the job outside of recon-ng and has longer dorks list 

And that's it ! Our mapping is finished. I hope you found some techniques useful for you future recon jobs. 🐸 Stay focused, in part 2, we want to pivot from this pile of information to companies, contacts, profiles, repositories and more.