Blog
Nov 9, 2021
Pythonizing Nmap
Tristram (aka gh0x0st) shares with us some tips for using python to automate nmap and other parts of your penetration testing process.
45 min read
by Tristram (aka gh0x0st)
This blog post was originally published by Tristram (aka gh0x0st) on GitHub and has been reposted with permission from the author.
When I started to get into this field, I tried my best to stick to manual workflows to get the lay of the land. As time passed by and I gained more experience the more I found that I needed to find better ways to be more efficient with my time. One place where I spent too much time was performing my initial enumeration with Nmap on larger scale assessments, staying organized as well as writing reports with the information I have collected.
I believe that automation is crucial for some aspects of a penetration test and Python is a tool to help us facilitate this. Allow me to show you various ways you can enhance your workflows by incorporating Python into your Nmap processes.
Boots on The Ground
Opening Remarks on Nmap Wrappers
Keep in mind that there is no one size fits all when it comes to Nmap scans. Before you run any sort of Nmap wrapper you should always look at the parameters that are in play and craft them to meet your needs and applicable scenarios. For your convenience here are the individual nmap commands I have incorporated in these scripts.
I like to keep the structure of my nmap commands consistent in a TARGET PORT OMIT SCAN SPEED VERBOSITY OUTPUT format.
Stage | Nmap Command | Requires Root |
---|---|---|
Host Discovery – ICMP Echo | nmap TARGET -n -sn -PE -vv -oX OUTPUT | Yes |
Host Discovery – ICMP Netmask | nmap TARGET -n -sn -PM -vv -oX OUTPUT | Yes |
Host Discovery – ICMP Timestamp | nmap TARGET -n -sn -PP -vv -oX OUTPUT | Yes |
Host Discovery – Port Scanning | nmap TARGET -PS21,22,23,25,80,113,443 -PA80,113,443 -n -sn -T4 -vv -oX OUTPUT | Yes |
Port Scanning (Top 1000) | nmap TARGET –top-ports 1000 -n -Pn -sS -T4 –min-parallelism 100 –min-rate 64 -vv -oX OUTPUT | Yes |
Service Detection | nmap TARGET -p PORTS -n -Pn -sV –version-intensity 6 –script banner -T4 -vv -oX OUTPUT | No |
OS Detection | nmap TARGET -n -Pn -O -T4 –min-parallelism 100 –min-rate 64 -vv -oX OUTPUT | Yes |
SSL Ciphers | nmap TARGET -p PORTS -n -Pn –script ssl-enum-ciphers -T4 -vv -oX OUTPUT | No |
SSL Certs | nmap TARGET -p PORTS -n -Pn –script ssl-cert -T4 -vv -oX OUTPUT | No |
Port Scanning (1-65535) | nmap TARGET -p- -n -Pn -sS -T4 –min-parallelism 100 –min-rate 128 -vv -oX OUTPUT | Yes |
Using Subprocess with Nmap
The subprocess (https://docs.python.org/3/library/subprocess.html) library allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. This library will make it easy for us to make calls to Nmap as well as manage the output effectively. Because we are going to use subprocess to call a program with parameters, we must pass our arguments as a list. What makes this tricky is each parameter will need to be its own element. However, Python makes this easy for us by using the shlex (https://docs.python.org/3/library/shlex.html) library.
This library takes in a string and it will split each space delimiter parameter as its own element in the list. I frequently see scripts do this manually but it’s not necessary. There may be some reading this and wonder why we are using a library when we can just use the built-in split() method from a string. These two approaches do nearly the same thing. The difference being is the split() method will create a list based on the delimiter and shlex.split() will create a delimited list intelligently based on how the shell interprets the input.
What this means is if you have any parameters passed to Nmap that contains spaces within quotes, then split() will break your input when delimiting on spaces whereas shlex.split() will break it down appropriately. In a nut shell, if you do not plan on using spaces where you shouldn’t, split() will work just fine, but out of my own habit, I incorporate shlex.split() to build my arguments for subprocess.
You can see an example of what this looks like below:
Reading STDOUT and STDERR is also relatively easy to do if you care about capturing both within your scripts. You can declare the values of the stdout/stderr arguments as subprocess.PIPE. Finally, you can read the data passed from stdout and stderr by using communicate() and declare them in variables respectively.
I invite you to look at the man page for subprocess to see if there’s any other tricks that you could find useful as it expands a lot further than what I have provided here.
https://docs.python.org/3/library/subprocess.html#subprocess.Popen.communicate
Parsing Nmap XML
One of my favorite features of Nmap is the ability to output our scan results to XML files. This enables us to parse through them to generate reports or use the output to generate input parameters for other Nmap operations. Let’s look at an example of the XML output:
Parse for Live Hosts
The first case where this will be useful for us is to determine which hosts from our host discovery probes are considered up. To facilitate this task in python we’ll take advantage of the xml.etree.ElementTree (https://docs.python.org/3/library/xml.etree.elementtree.html) library. Our helper function will take in the path of an XML file we designate and parse out the hosts that are flagged as being ‘up’.
Since I use all possible discovery probes I use parseDiscoverXml() to take in the results from all the Xml files, then I use a second helper function to remove any duplicates and output them space delimited so I can use those at the target input values for future Nmap calls.
#!/usr/bin/python3 import xml.etree.ElementTree as ET def parseDiscoverXml(in_xml): live_hosts = [] xml_tree = ET.parse(in_xml) xml_root = xml_tree.getroot() for host in xml_root.findall('host'): ip_state = host.find('status').get('state') if ip_state == "up": live_hosts.append(host.find('address').get('addr')) return live_hosts def convertToNmapTarget(hosts): hosts = list(dict.fromkeys(hosts)) return " ".join(hosts) def main(): hosts = parseDiscoverXml('/home/tristram/Scans/Stage_1/icmp_echo_host_discovery.xml') hosts += parseDiscoverXml('/home/tristram/Scans/Stage_1/icmp_netmask_host_discovery.xml') hosts += parseDiscoverXml('/home/tristram/Scans/Stage_1/icmp_timestamp_host_discovery.xml') hosts += parseDiscoverXml('/home/tristram/Scans/Stage_1/tcp_syn_host_discovery.xml') print(f"Flagged Hosts: {len(hosts)}") print(f"Unique Hosts: {len(list(dict.fromkeys(hosts)))}") print(f"Nmap Format Example: {convertToNmapTarget(hosts)}") if __name__ == '__main__': main()
Parse for Accessible Ports
Now that we have a way to easily construct a list of available hosts, we can move onto port scanning. After this operation is finished, we’ll need a way to programmatically parse the ports that considered available to our attacker machine. Our port scanning stages scripts will produce files called top_1000_portscan.xml / full_portscan.xml respectively.
What we will do with this file is parse through every host in the hosts element and for each host we will loop through every port in the ports element. After it flags a port that’s found to be open it’ll keep all the results in a list with each element in a ” ,” format. When we start our service scanning, we’ll split the results so we can designate our target host and target hosts respectively in future Nmap calls. This allows to programmatically generate our Nmap commands with the necessary target ip addresses and ports.
#!/usr/bin/python3 import xml.etree.ElementTree as ET def parseDiscoverPorts(in_xml): results = [] port_list = '' xml_tree = ET.parse(in_xml) xml_root = xml_tree.getroot() for host in xml_root.findall('host'): ip = host.find('address').get('addr') ports = host.findall('ports')[0].findall('port') for port in ports: state = port.find('state').get('state') if state == 'open': port_list += port.get('portid') + ',' port_list = port_list.rstrip(',') if port_list: results.append(f"{ip} {port_list}") port_list = '' return results def main(): targets = parseDiscoverPorts('/home/tristram/Scans/Stage_2/top_1000_portscan.xml') for target in targets: element = target.split() target_ip = element[0] target_ports = element[1] print(f'Nmap Format Example: nmap {target_ip} -p {target_ports}') if __name__ == '__main__': main()
Parsing XML in Memory
The previous examples showed you how you can parse XML files that are on disk, but you are also able to parse XML without relying on XML on disk by changing a few approaches. Specifically, we’ll tell Nmap to output XML to stdout and we will store that in a variable. The output itself will be stored in the first element in the tuple as a bytes-like object. We will just need to make a few changes but can borrow nearly the entire function we created before.
The only changes we need to make will be to remove ET.parse and xml_tree.getroot() and replace with ET.fromstring which parses XML from a string directly into an Element, which is the root element of the parsed tree.
Personally, I do not use this approach as much as I like to have the XML files on disk so I can use with other operations. Keep in mind that if you wanted to write your tool that works with everything in memory then you should keep an eye on your system resources. Some Nmap scans can produce quite large output files and you do not want to bog down your system or lose data in the event of a system crash.
#!/usr/bin/python3 import xml.etree.ElementTree as ET import subprocess import shlex def nmapMemory(target): args = shlex.split(f"/usr/bin/nmap {target} -T4 -Pn -n -vv -sS -min-parallelism 100 --min-rate 64 --top-ports 1000 -oX -") return subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate() def parseDiscoverPortsMemory(in_xml): results = [] port_list = '' #xml_tree = ET.parse(in_xml) #xml_root = xml_tree.getroot() xml_root = ET.fromstring(in_xml.decode('utf-8')) for host in xml_root.findall('host'): ip = host.find('address').get('addr') ports = host.findall('ports')[0].findall('port') for port in ports: state = port.find('state').get('state') if state == 'open': port_list += port.get('portid') + ',' port_list = port_list.rstrip(',') if port_list: results.append(f"{ip} {port_list}") port_list = '' return results def main(): target = '127.0.0.1' xml_output = nmapMemory(target)[0] targets = parseDiscoverPortsMemory(xml_output) for target in targets: element = target.split() target_ip = element[0] target_ports = element[1] print(f'Scanning: {target_ip} against ports {target_ports}') if __name__ == '__main__': main()
Importing Nmap XML into SQLite Databases
Since we have learned previously how to parse Nmap XML using Python we can also take those results and import them into SQLite Databases. From there you could use that database to build input parameters, generate reports or keep historical information from past engagements. The sqlite3 (https://docs.python.org/3/library/sqlite3.html) library does virtually all the hard work for us. Keep in mind that this library requires us to work with tuples when you receive results back from the database. They work just like lists except you cannot change the element values.
Let’s look at how this can be done with a simple use case scenario for storing the results from a host discovery scan and whether an IP is up or not based on the results from an ICMP Echo scan.
- Creating the Database File
- Inserting Data into a Table
- Selecting Content from a Table
Creating the Database File
This is where we’ll create the actual database file on disk. Within the create_db function you can setup your tables and the values you want to store. If you want some ideas on what sort of tables could work for you then consider peaking at section 6 of this post before moving on as those sections produce CSV tables with various amounts of information.
#!/usr/bin/python3 import sqlite3 def create_connection(db_file): conn = None try: conn = sqlite3.connect(db_file) except Exception as e: print(e) return conn def create_db(conn): createHostDiscoveryTable="""CREATE TABLE IF NOT EXISTS HostDiscovery ( id integer PRIMARY KEY, IP text NOT NULL, Status text NOT NULL, ICMP_Echo text NOT NULL);""" try: c = conn.cursor() c.execute(createHostDiscoveryTable) except Exception as e: print(e) def main(): db_file = 'PythonizingNmap.db' conn = create_connection(db_file) create_db(conn) if __name__ == '__main__': main()
Inserting Data into a Table
Now that our table is created, we can define our insert_content function to insert our parsed XML data directly into the HostDiscovery table. Granted I hardcoded some values here this would be a good function to parameterize to make it more dynamic. Keep note that we are inserting our data as a tuple.
#!/usr/bin/python3 import sqlite3 import xml.etree.ElementTree as ET def create_connection(db_file): conn = None try: conn = sqlite3.connect(db_file) except Exception as e: print(e) return conn def insert_content(conn, content): sql = ''' INSERT INTO HostDiscovery(IP,Status,ICMP_Echo) VALUES(?,?,?) ''' cur = conn.cursor() cur.execute(sql, content) return cur.lastrowid def main(): # Database Connection db_file = 'PythonizingNmap.db' conn = create_connection(db_file) # Parse XML in_xml_echo = '/home/tristram/Scans/Stage_1/icmp_echo_host_discovery.xml' # Load ICMP Echo XML xml_tree_echo = ET.parse(in_xml_echo) xml_root_echo = xml_tree_echo.getroot() # Load ICMP Echo XML for host in xml_root_echo.findall('host'): echo_ip = host.find('address').get('addr') echo_state = host.find('status').get('state') echo_reason = host.find('status').get('reason') # Insert results into database insert_content(conn, (echo_ip, echo_state, echo_reason)) conn.commit() if __name__ == '__main__': main()
Selecting Content from a Table
After our data is inserted into the database what you can do from here is up to you! You could use this to store results from past engagements or even use it as a working database where you can build other automated workflows that utilize information selected from the database itself. In this example here we are selecting all the hosts that are considered ‘up’.
#!/usr/bin/python3 import sqlite3 def create_connection(db_file): conn = None try: conn = sqlite3.connect(db_file) except Exception as e: print(e) return conn def select_content(conn): sql = """SELECT IP FROM HostDiscovery WHERE Status = 'up' """ cur = conn.cursor() cur.execute(sql) rows = cur.fetchall() return rows def main(): db_file = 'PythonizingNmap.db' conn = create_connection(db_file) live_hosts = select_content(conn) for host in live_hosts: print(f'Live: {host[0]}') if __name__ == '__main__': main()
Python Nmap Wrapper Scripts
Now that we’ve gone through parsing the XML files from Nmap we can use this approach to programmatically generate input parameters for other Nmap operations where we need to designate target IPs and/or ports. I have included below some thoughts around a staged approach to Nmap enumeration. Keep in mind that the code snippets provided are intended to act as blueprints for you to build upon.
As you read these examples you will find cases where we scan individual IPs at a time, resulting in multiple XML output files and others where I have a single scan targeting all the IPs resulting in a single XML output file. I did this intentionally for you to weight the benefits of parsing through individual XML files vs a single XML file. One option allows you to pass in a single file into your functions where the others require you use a for loop. If you use individually XML files it would be easier to review the results for a specific machine vs picking out the bits you want a in a larger file.
Stage 1 – Host Discovery
With this step the objective is to determine whether something exists at a particular IP based on the response to your probes. You’ll typically encounter straight ICMP restrictions at the firewall, but there are cases where there’s misconfigurations or even intended configurations where specific ICMP types are permitted. Because of this I like to take advantage of ICMP ECHO, ICMP TIMESTAMP and ICMP NETMASK probes by sending them individually.
Outside of ICMP probes, another approach you will likely have to take is to run a port scan with a small subset of ports to solicit a response from the firewall. In these cases, a RESET or SYN-ACK from the firewall denotes a live host at that IP address. I combine both half open scans and ack scans (https://nmap.org/book/host-discovery-strategies.html) with a very small subset of ports to try. By combining all five of these probes together you can craft yourself a scripted host discovery solution to enhance your chances of discovering a live host.
Granted during this stage all you want is to know is whether a host is up. However, I like to expand on this a little more by reporting how each host responds to each of the probes. You may identify hosts that allow ICMP and if the client believes they are blocking ICMP across the board from the internet it could be helpful for them to be aware.
#!/usr/bin/python3 import shlex import subprocess import os import sys def sendIcmpEcho(target, out_xml): out_xml = os.path.join(out_xml,'icmp_echo_host_discovery.xml') nmap_cmd = f"/usr/bin/nmap {target} -n -sn -PE -vv -oX {out_xml}" sub_args = shlex.split(nmap_cmd) subprocess.Popen(sub_args, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate() makeInvokerOwner(out_xml) def sendIcmpNetmask(target, out_xml): out_xml = os.path.join(out_xml,'icmp_netmask_host_discovery.xml') nmap_cmd = f"/usr/bin/nmap {target} -n -sn -PM -vv -oX {out_xml}" sub_args = shlex.split(nmap_cmd) subprocess.Popen(sub_args, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate() makeInvokerOwner(out_xml) def sendIcmpTimestamp(target, out_xml): out_xml = os.path.join(out_xml,'icmp_timestamp_host_discovery.xml') nmap_cmd = f"/usr/bin/nmap {target} -n -sn -PP -vv -oX {out_xml}" sub_args = shlex.split(nmap_cmd) subprocess.Popen(sub_args, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate() makeInvokerOwner(out_xml) def sendTcpSyn(target, out_xml): out_xml = os.path.join(out_xml,'tcp_syn_host_discovery.xml') nmap_cmd = f"/usr/bin/nmap {target} -PS21,22,23,25,80,113,443 -PA80,113,443 -n -sn -T4 -vv -oX {out_xml}" sub_args = shlex.split(nmap_cmd) subprocess.Popen(sub_args, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate() makeInvokerOwner(out_xml) def makeInvokerOwner(path): uid = os.environ.get('SUDO_UID') gid = os.environ.get('SUDO_GID') if uid is not None: os.chown(path, int(uid), int(gid)) def is_root(): if os.geteuid() == 0: return True else: return False def main(): if not is_root(): print('[!] The discovery probes in this script requires root privileges') sys.exit(1) target = '127.0.0.1' sendIcmpEcho(target, os.getcwd()) sendIcmpNetmask(target, os.getcwd()) sendIcmpTimestamp(target, os.getcwd()) sendTcpSyn(target, os.getcwd()) if __name__ == '__main__': main()
Stage 2 – Port Scanning (Top 1000)
I do not find services running on non-standard ports too often in production. Because of this I focus on the ports that have a higher ratio as defined in the nmap-services file. This will help save you time while finding the ports that are likely to be accessible. Based on the nmap author’s research (https://nmap.org/book/performance-port-selection.html), scanning the top 1000 ports will catch roughly 93% of the TCP ports. The statistics here are in your favor and you’ll find most of the ports within a reasonable amount of time.
#!/usr/bin/python3 import xml.etree.ElementTree as ET import subprocess import shlex import os import sys def parseDiscoverXml(in_xml): live_hosts = [] xml_tree = ET.parse(in_xml) xml_root = xml_tree.getroot() for host in xml_root.findall('host'): ip_state = host.find('status').get('state') if ip_state == "up": live_hosts.append(host.find('address').get('addr')) return live_hosts def convertToNmapTarget(hosts): hosts = list(dict.fromkeys(hosts)) return " ".join(hosts) def tcpSynPortScan(target, out_xml,): out_xml = os.path.join(out_xml,'top_1000_portscan.xml') nmap_cmd = f"/usr/bin/nmap {target} --top-ports 1000 -n -Pn -sS -T4 --min-parallelism 100 --min-rate 64 -vv -oX {out_xml}" sub_args = shlex.split(nmap_cmd) subprocess.Popen(sub_args, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate() makeInvokerOwner(out_xml) def makeInvokerOwner(path): uid = os.environ.get('SUDO_UID') gid = os.environ.get('SUDO_GID') if uid is not None: os.chown(path, int(uid), int(gid)) def is_root(): if os.geteuid() == 0: return True else: return False def main(): if not is_root(): print('[!] TCP/SYN scans requires root privileges') sys.exit(1) hosts = parseDiscoverXml('/home/tristram/Scans/Stage_1/icmp_echo_host_discovery.xml') hosts += parseDiscoverXml('/home/tristram/Scans/Stage_1/icmp_netmask_host_discovery.xml') hosts += parseDiscoverXml('/home/tristram/Scans/Stage_1/icmp_timestamp_host_discovery.xml') hosts += parseDiscoverXml('/home/tristram/Scans/Stage_1/tcp_syn_host_discovery.xml') target = convertToNmapTarget(hosts) tcpSynPortScan(target, os.getcwd()) if __name__ == '__main__': main()
Stage 3 – Service Detection
Service scanning is something that will catch inexperienced pen testers off guard when they discover that a simple service scan, they run all the time on CTFs just alerted a blue team to their presence an hour in on their assessment. Allow me to provide you an example of what I’m talking about and look at an example from the nmap-service-probes file:
If you come across any server that uses ports 515,1028,1068,1503,1720,1935,2040,3388,3389 then nmap, with the default options, will eventually use the TerminalServer probes. Here’s the problem. If you have a client that uses a Cisco IPS for example that sits in front of that server and it sees x03 x0bx06xe0 | destined to any port that isn’t 3389, then it’s going to flag you thinking you’re trying to connect to RDP on a non-standard port. Because of this as a rule of thumb I put a hard stop on letting nmap try to service probe anything on those ports so I block those off the bat in the config file on line 29 Exclude T:9100-9107,T:515,T:1028,T:1068,T:1503,T:1720,T:1935,T:2040,T:3388.
The problem doesn’t stop there though. If you run into a port that nmap cannot figure out, it will try every possible probe up the intensity level, which by default is 7 (https://nmap.org/book/man-version-detection.html). If you look at the snippet below, there is another terminal server probe that is set to rarity 7, so those probes would be included. To prevent that from happening, I set my intensity version to 5 or 6 via –version-intensity depending on how paranoid I am.
If you have access to lab network with some sort of IDS/IPS it would be great practice for you to see what type of scans trigger alerts and what you can do to prevent them from happening.
#!/usr/bin/python3 import xml.etree.ElementTree as ET import subprocess import shlex import os def parseDiscoverPorts(in_xml): results = [] port_list = '' xml_tree = ET.parse(in_xml) xml_root = xml_tree.getroot() for host in xml_root.findall('host'): ip = host.find('address').get('addr') ports = host.findall('ports')[0].findall('port') for port in ports: state = port.find('state').get('state') if state == 'open': port_list += port.get('portid') + ',' port_list = port_list.rstrip(',') if port_list: results.append(f"{ip} {port_list}") port_list = '' return results def serviceScan(target_ip, target_ports, out_xml): out_xml = os.path.join(out_xml,f'{target_ip}_services.xml') nmap_cmd = f"/usr/bin/nmap {target_ip} -p {target_ports} -n -Pn -sV --version-intensity 6 --script banner -T4 -vv -oX {out_xml}" sub_args = shlex.split(nmap_cmd) subprocess.Popen(sub_args, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate() def main(): in_xml = '/home/tristram/Scans/Stage_2/top_1000_portscan.xml' targets = parseDiscoverPorts(in_xml) for target in targets: element = target.split() target_ip = element[0] target_ports = element[1] print(f'Scanning: {target_ip} against ports {target_ports}') serviceScan(target_ip, target_ports, os.getcwd()) if __name__ == '__main__': main()
Stage 4 – OS Detection
I’m a bit torn on using the OS discovery scan over the internet. Sometimes it does not provide me anything useful and other times it provides me a gold mine with unsupported operating systems. I will run this scan just to see and will try to verify through other types of enumeration, such as identifying os requirements for the running software if I’m able. If I’m on the network probing a device, I’ll typically use this all the time if I’m on the internal network but over the internet it all depends on if I have anything else to work off from first.
#!/usr/bin/python3 import xml.etree.ElementTree as ET import subprocess import shlex import os import sys def parseDiscoverXml(in_xml): live_hosts = [] xml_tree = ET.parse(in_xml) xml_root = xml_tree.getroot() for host in xml_root.findall('host'): ip_state = host.find('status').get('state') if ip_state == "up": live_hosts.append(host.find('address').get('addr')) return live_hosts def convertToNmapTarget(hosts): hosts = list(dict.fromkeys(hosts)) return " ".join(hosts) def osScan(targets, out_xml): out_xml = os.path.join(out_xml,f'osdetection.xml') nmap_cmd = f"/usr/bin/nmap {targets} -n -Pn -O -T4 --min-parallelism 100 --min-rate 64 -vv -oX {out_xml}" sub_args = shlex.split(nmap_cmd) subprocess.Popen(sub_args, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate() makeInvokerOwner(out_xml) def makeInvokerOwner(path): uid = os.environ.get('SUDO_UID') gid = os.environ.get('SUDO_GID') if uid is not None: os.chown(path, int(uid), int(gid)) def is_root(): if os.geteuid() == 0: return True else: return False def main(): if not is_root(): print('[!] TCP/IP fingerprinting (for OS scan) requires root privileges.') sys.exit(1) hosts = parseDiscoverXml('/home/tristram/Scans/Stage_1/icmp_echo_host_discovery.xml') hosts += parseDiscoverXml('/home/tristram/Scans/Stage_1/icmp_netmask_host_discovery.xml') hosts += parseDiscoverXml('/home/tristram/Scans/Stage_1/icmp_timestamp_host_discovery.xml') hosts += parseDiscoverXml('/home/tristram/Scans/Stage_1/port_host_discovery.xml') target = convertToNmapTarget(hosts) osScan(target, os.getcwd()) if __name__ == '__main__': main()
Stage 5 – SSL Ciphers
For the most part I try to keep NSE scripts for more targeted enumeration, apart from ssl-enum-ciphers and ssl-certs. The NSE script ssl-enum-ciphers is particularly useful for when your target under regulatory requirements and aren’t supposed to be using unsafe TLS configurations. Some of the NSE scripts can be noisy so weigh the benefit of what you are trying to learn about a target vs the risk of being busted.
The results of this NSE script exports nicely into XML and I’ll show you how you can convert these results into a CSV format so you can easily move into a report further down.
#!/usr/bin/python3 import xml.etree.ElementTree as ET import subprocess import shlex import os def parseDiscoverPorts(in_xml): results = [] port_list = '' xml_tree = ET.parse(in_xml) xml_root = xml_tree.getroot() for host in xml_root.findall('host'): ip = host.find('address').get('addr') ports = host.findall('ports')[0].findall('port') for port in ports: state = port.find('state').get('state') if state == 'open': port_list += port.get('portid') + ',' port_list = port_list.rstrip(',') if port_list: results.append(f"{ip} {port_list}") port_list = '' return results def sslCipherScan(target_ip, target_ports, out_xml): out_xml = os.path.join(out_xml,f'{target_ip}_ssl_ciphers.xml') nmap_cmd = f"/usr/bin/nmap {target_ip} -p {target_ports} -n -Pn --script ssl-enum-ciphers -T4 -vv -oX {out_xml}" sub_args = shlex.split(nmap_cmd) subprocess.Popen(sub_args, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate() def main(): in_xml = '/home/tristram/Scans/Stage_2/syn_port_scan.xml' targets = parseDiscoverPorts(in_xml) for target in targets: element = target.split() target_ip = element[0] target_ports = element[1] print(f'Scanning: {target_ip} against ports {target_ports}') sslCipherScan(target_ip, target_ports, os.getcwd()) if __name__ == '__main__': main()
Stage 6 – SSL Certs
I like to include this step because from time to time misconfigured or poorly crafted SSL certificates can reveal quite a bit of information. For example, if you identify a web server that’s accessible to the internet and it has a certificate signed by an internal CA then there is a good chance that web server is behind reverse proxy or a server on the private network being NAT’d to the internet which could lead to a damaging foothold if you can identify an exploitable condition.
#!/usr/bin/python3 import xml.etree.ElementTree as ET import subprocess import shlex import os def parseDiscoverPorts(in_xml): results = [] port_list = '' xml_tree = ET.parse(in_xml) xml_root = xml_tree.getroot() for host in xml_root.findall('host'): ip = host.find('address').get('addr') ports = host.findall('ports')[0].findall('port') for port in ports: state = port.find('state').get('state') if state == 'open': port_list += port.get('portid') + ',' port_list = port_list.rstrip(',') if port_list: results.append(f"{ip} {port_list}") port_list = '' return results def sslCertScan(target_ip, target_ports, out_xml): out_xml = os.path.join(out_xml,f'{target_ip}_ssl_certs.xml') nmap_cmd = f"/usr/bin/nmap {target_ip} -p {target_ports} -n -Pn --script ssl-cert -T4 -vv -oX {out_xml}" sub_args = shlex.split(nmap_cmd) subprocess.Popen(sub_args, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate() def main(): in_xml = '/home/tristram/Scans/Stage_2/syn_port_scan.xml' targets = parseDiscoverPorts(in_xml) for target in targets: element = target.split() target_ip = element[0] target_ports = element[1] print(f'Scanning: {target_ip} against ports {target_ports}') sslCertScan(target_ip, target_ports, os.getcwd()) if __name__ == '__main__': main()
Stage 7 – Port Scanning (1-65535)
I intentionally run this step last because it takes a long time if you have a lot of hosts. Based on the stats from the first port scan we’ll only have a 7% chance of finding anything new so the return on investment of is particularly low. However, this stage is still something worth digging into a little bit. Obviously, we want our scans to run as fast as possible but we’re too noisy we might trip an alarm, especially since we’re scanning the entire TCP port range.
If you have a lot of time to spare, consider the low and slow approach. If you’re not concerned about alerts, then play around with the timing and performance parameters (https://nmap.org/book/man-performance.html). I’ve found –min-parallelism 100 –min-rate 128 to be a good sweet spot between speed and reliably.
#!/usr/bin/python3 import xml.etree.ElementTree as ET import subprocess import shlex import os import sys def parseDiscoverXml(in_xml): live_hosts = [] xml_tree = ET.parse(in_xml) xml_root = xml_tree.getroot() for host in xml_root.findall('host'): ip_state = host.find('status').get('state') if ip_state == "up": live_hosts.append(host.find('address').get('addr')) return live_hosts def convertToNmapTarget(hosts): hosts = list(dict.fromkeys(hosts)) return " ".join(hosts) def tcpSynPortScan(target, out_xml,): out_xml = os.path.join(out_xml,'65535_portscan.xml') nmap_cmd = f"/usr/bin/nmap {target} -p- -n -Pn -sS -T4 --min-parallelism 100 --min-rate 128 -vv -oX {out_xml}" sub_args = shlex.split(nmap_cmd) subprocess.Popen(sub_args, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate() makeInvokerOwner(out_xml) def makeInvokerOwner(path): uid = os.environ.get('SUDO_UID') gid = os.environ.get('SUDO_GID') if uid is not None: os.chown(path, int(uid), int(gid)) def is_root(): if os.geteuid() == 0: return True else: return False def main(): if not is_root(): print('[!] TCP/SYN scans requires root privileges') sys.exit(1) hosts = parseDiscoverXml('/home/tristram/Scans/Stage_1/icmp_echo_host_discovery.xml') hosts += parseDiscoverXml('/home/tristram/Scans/Stage_1/icmp_netmask_host_discovery.xml') hosts += parseDiscoverXml('/home/tristram/Scans/Stage_1/icmp_timestamp_host_discovery.xml') hosts += parseDiscoverXml('/home/tristram/Scans/Stage_1/port_host_discovery.xml') target = convertToNmapTarget(hosts) tcpSynPortScan(target, os.getcwd()) if __name__ == '__main__': main()
Generating Reports from Nmap XML
Depending on the size of your engagement the process of transcribing your notes into a report can be quite tedious. Thankfully this is another place where Python comes to the rescue. We can take the same XML files we were working with before to generate CSV files that we can then use to import into the report format of our choosing.
The scripts for this can be a little confusing with the all the loops so I added comments to help describe each step. Keep a mental note that if there are multiple tables shown in a section that means that script will create that many tables.
Detected Hosts
IP | Status | ICMP Echo | ICMP Netmask | ICMP Timestamp | Port |
---|---|---|---|---|---|
192.168.0.100 | down | no-response | no-response | no-response | no-response |
192.168.0.101 | up | echo-reply | no-response | timestamp-reply | reset |
192.168.0.102 | up | echo-reply | no-response | no-response | syn-ack |
192.168.0.103 | down | no-response | no-response | no-response | no-response |
#!/usr/bin/python3 import xml.etree.ElementTree as ET import csv def main(): # File Paths in_xml_port = '/home/tristram/Scans/Stage_1/tcp_syn_host_discovery.xml' in_xml_echo = '/home/tristram/Scans/Stage_1/icmp_echo_host_discovery.xml' in_xml_netmask = '/home/tristram/Scans/Stage_1/icmp_netmask_host_discovery.xml' in_xml_timestamp = '/home/tristram/Scans/Stage_1/icmp_timestamp_host_discovery.xml' # Load Port XML xml_tree_port = ET.parse(in_xml_port) xml_root_port = xml_tree_port.getroot() # Load ICMP Echo XML xml_tree_echo = ET.parse(in_xml_echo) xml_root_echo = xml_tree_echo.getroot() # Load ICMP Netmask XML xml_tree_netmask = ET.parse(in_xml_netmask) xml_root_netmask = xml_tree_netmask.getroot() # Load ICMP Timestamp XML xml_tree_timestamp = ET.parse(in_xml_timestamp) xml_root_timestamp = xml_tree_timestamp.getroot() # CSV File with open('detected_hosts.csv', 'w') as file: writer = csv.writer(file) # CSV Headers writer.writerow(['IP', 'Status', 'ICMP Echo', 'ICMP Netmask', 'ICMP Timestamp', 'Port']) # Load SYN Port XML for host in xml_root_port.findall('host'): host_status = 'down' master_ip = host.find('address').get('addr') port_state = host.find('status').get('state') port_reason = host.find('status').get('reason') # Load ICMP Echo XML for host in xml_root_echo.findall('host'): echo_ip = host.find('address').get('addr') echo_state = host.find('status').get('state') echo_reason = host.find('status').get('reason') # Load ICMP Netmask if master_ip == echo_ip: for host in xml_root_netmask.findall('host'): netmask_ip = host.find('address').get('addr') netmask_state = host.find('status').get('state') netmask_reason = host.find('status').get('reason') # Load ICMP Timestamp if master_ip == netmask_ip: for host in xml_root_timestamp.findall('host'): timestamp_ip = host.find('address').get('addr') timestamp_state = host.find('status').get('state') timestamp_reason = host.find('status').get('reason') if master_ip == timestamp_ip: if port_state == 'up' or echo_state == 'up' or netmask_state == 'up' or timestamp_state == 'up': host_status = 'up' # Write results to row writer.writerow([master_ip, host_status, echo_reason, netmask_reason, timestamp_reason, port_reason]) if __name__ == '__main__': main()
Detected Hosts Without Ports
IP | Port | Service |
---|---|---|
192.168.0.104 | no-response | no-response |
192.168.0.105 | no-response | no-response |
192.168.0.106 | no-response | no-response |
192.168.0.107 | no-response | no-response |
#!/usr/bin/python3 import xml.etree.ElementTree as ET import csv def main(): # Path to directory with host XML files in_xml = '/home/tristram/Scans/Stage_2/top_1000_portscan.xml' # CSV Data with open('detected_hosts_no_ports.csv', 'w') as file: writer = csv.writer(file) # CSV Headers writer.writerow(['IP', 'Port', 'Service']) # Load Top 1000 Port Scan xml_tree = ET.parse(in_xml) xml_root = xml_tree.getroot() # Cycle through each host for host in xml_root.findall('host'): ip_address = host.findall('address')[0].attrib['addr'] ports_element = host.findall('ports') port_child = ports_element[0].findall('port') open_ports = [] # Within each host cycle through the ports for port in port_child: if port.findall('state')[0].attrib['state'] == 'open': port_id = port.attrib['portid'] open_ports.append(port_id) # Write results to row if len(open_ports) == 0: writer.writerow([ip_address, 'no-response', 'no-response']) if __name__ == '__main__': main()
Detected Hosts With Ports + Services
IP | Port | Service |
---|---|---|
192.168.0.108 | 443 | https |
192.168.0.109 | 443 | https |
192.168.0.110 | 25 | tcpwrapped |
192.168.0.111 | 443 | https |
192.168.0.112 | 25 | Microsoft Exchange smtpd |
443 | https |
#!/usr/bin/python3 import xml.etree.ElementTree as ET import csv import os def main(): # Path to directory with host XML files in_path = '/home/tristram/Scans/Stage_3/' # CSV Data with open('detected_hosts_with_ports.csv', 'w') as file: writer = csv.writer(file) # CSV Headers writer.writerow(['IP', 'Port', 'Service']) for file in sorted(os.listdir(in_path)): if file.endswith(".xml"): if "no_ports.xml" in file: writer.writerow([file.split('_no_ports.xml')[0], 'no-response', 'no-response']) else: # Load Service Scan in_xml = os.path.join(in_path, file) xml_tree = ET.parse(in_xml) xml_root = xml_tree.getroot() # Cycle through each host for host in xml_root.findall('host'): ip_once = True ip_address = host.findall('address')[0].attrib['addr'] ports_element = host.findall('ports') port_child = ports_element[0].findall('port') open_ports = [] # Within each host cycle through the ports for port in port_child: if port.findall('state')[0].attrib['state'] == 'open': port_id = port.attrib['portid'] service_name = port.find('service').get('product') if service_name == None: service_name = port.find('service').get('name') open_ports.append([port_id, service_name]) # Within each port cycle through the open ports if len(open_ports): for op in open_ports: # Ensure we only notate the IP once to keep it clean if ip_once == True: # Write results to row writer.writerow([ip_address, op[0], op[1]]) ip_once = False else: # Write results to row writer.writerow([None, op[0], op[1]]) else: # Write results to row if none writer.writerow([ip_address, 'none', 'none']) if __name__ == '__main__': main()
Detected Hosts With Guessed Operating Systems
IP | Port |
---|---|
192.168.0.113 | Unknown |
192.168.0.114 | D-Link DCS-6620G webcam or Linksys BEFSR41 EtherFast router |
192.168.0.115 | Linux 4.9 |
192.168.0.116 | Linux 2.6.32 |
192.168.0.117 | FreeBSD 9.0-RELEASE – 10.3-RELEASE |
#!/usr/bin/python3 import xml.etree.ElementTree as ET import csv def main(): # Path top the os scan file in_xml = '/home/tristram/Scans/Stage_4/osdetection.xml' # CSV Data with open('detected_hosts_os.csv', 'w') as file: writer = csv.writer(file) # CSV Headers writer.writerow(['IP', 'OperatingSystem']) # Load OS Scan XML xml_tree = ET.parse(in_xml) xml_root = xml_tree.getroot() # Cycle through each host for host in xml_root.findall('host'): ip_address = host.findall('address')[0].attrib['addr'] try: os_element = host.findall('os') os_name = os_element[0].findall('osmatch')[0].attrib['name'] except IndexError: os_name = 'Unknown' # Write results to row writer.writerow([ip_address, os_name]) if __name__ == '__main__': main()
Detected Hosts TLS Protocols
TLSv1.0
IP | Port | Protocol |
---|---|---|
192.168.0.118 | 443 | TLSv1.0 |
192.168.0.119 | 25 | TLSv1.0 |
192.168.0.120 | 443 | TLSv1.0 |
192.168.0.121 | 443 | TLSv1.0 |
192.168.0.122 | 5061 | TLSv1.0 |
TLSv1.1
IP | Port | Protocol |
---|---|---|
192.168.0.123 | 443 | TLSv1.1 |
192.168.0.124 | 25 | TLSv1.1 |
192.168.0.125 | 443 | TLSv1.1 |
192.168.0.126 | 443 | TLSv1.1 |
192.168.0.127 | 5061 | TLSv1.1 |
SSLv3.0
IP | Port | Protocol |
---|---|---|
192.168.0.128 | 443 | SSLv3.0 |
192.168.0.129 | 25 | SSLv3.0 |
192.168.0.130 | 443 | SSLv3.0 |
192.168.0.131 | 443 | SSLv3.0 |
192.168.0.132 | 5061 | SSLv3.0 |
#!/usr/bin/python3 import xml.etree.ElementTree as ET import csv import os def main(): # Protocol Report Lists tls_v10_report = [] tls_v11_report = [] ssl_v3_report = [] # Flagged Lists flagged_tls_v10 = '' flagged_tls_v11 = '' flagged_ssl_v3 = '' # File Path in_path= '/home/tristram/Scans/Stage_5/' # Cycle through each XML file for file in os.listdir(in_path): # Load each XML file if file.endswith(".xml"): in_xml = os.path.join(in_path, file) xml_tree = ET.parse(in_xml) xml_root = xml_tree.getroot() # Cycle through each host for host in xml_root.findall('host'): ip = host.find('address').get('addr') ports_element = host.findall('ports') port_element = ports_element[0].findall('port') # Cycle through each port for scanned_port in port_element: port_id = scanned_port.get('portid') script_element = scanned_port.find('script') if script_element: # Cycle through each protocol protocol_element = script_element.findall('table') for tls_protocol in protocol_element: protocol_version = tls_protocol.attrib['key'] # Stage data for TLSv1.0 Table if protocol_version in 'TLSv1.0': flagged_tls_v10 += protocol_version + ',' # Stage data for TLSv1.1 Table if protocol_version in 'TLSv1.1': flagged_tls_v11 += protocol_version + ',' # Stage data for SSLv3 Table if protocol_version in 'SSLv3': flagged_ssl_v3 += protocol_version + ',' # Load TLSv1.0 Data if flagged_tls_v10: flagged_tls_v10 = flagged_tls_v10.strip(',').split(',') tls_v10_report.append([ip,port_id,flagged_tls_v10 ]) flagged_tls_v10 = '' # Load TLSv1.1 Data if flagged_tls_v11: flagged_tls_v11 = flagged_tls_v11.strip(',').split(',') tls_v11_report.append([ip,port_id,flagged_tls_v11 ]) flagged_tls_v11 = '' # Load SSLv3.0 Data if flagged_ssl_v3: flagged_ssl_v3 = flagged_ssl_v3.strip(',').split(',') ssl_v3_report.append([ip,port_id,flagged_ssl_v3 ]) flagged_ssl_v3 = '' # CSV Data for TLSv1.0 with open('detected_tls_v10.csv', 'w') as csvFile: writer = csv.writer(csvFile) # CSV Headers writer.writerow(["IP", "Port", "Protocol"]) # Cycle through each staged element in list for server in tls_v10_report: row_ip = server[0] row_port = server[1] # Write results to row writer.writerow([row_ip,row_port,'TLSv1.0']) # CSV Data for TLSv1.1 with open('detected_tls_v11.csv', 'w') as csvFile: writer = csv.writer(csvFile) # CSV Headers writer.writerow(["IP", "Port", "Protocol"]) for server in tls_v11_report: row_ip = server[0] row_port = server[1] # Write results to row writer.writerow([row_ip,row_port,'TLSv1.1']) # CSV Data for SSLv.3 with open('detected_ssl_v3.csv', 'w') as csvFile: writer = csv.writer(csvFile) # CSV Headers writer.writerow(["IP", "Port", "Protocol"]) for server in ssl_v3_report: row_ip = server[0] row_port = server[1] # Write results to row writer.writerow([row_ip,row_port,'SSLv3.0']) if __name__ == '__main__': main()
Detected SSL Certificates
IP | Port | CommonName | IssuerCommon | CertStart | CertEnd |
---|---|---|---|---|---|
192.168.0.133 | 443 | stay.example.com | DigiCert Global CA G2 | 2020-05-06 | 2021-05-06 |
192.168.0.134 | 443 | off.example.com | DigiCert Global CA G2 | 2019-11-06 | 2020-11-05 |
192.168.0.135 | 443 | ronins.example.com | DigiCert Global CA G2 | 2020-05-04 | 2021-05-05 |
192.168.0.136 | 443 | lawn.example.com | DigiCert Global CA G2 | 2019-11-15 | 2020-11-14 |
#!/usr/bin/python3 import xml.etree.ElementTree as ET import csv import os import re def main(): # Regular Expressions reg_date = r'd{4}-d{2}-d{2}' # File Path in_path= '/home/tristram/Scans/Stage_6/' # Reports cert_info_list = '' cert_info_report = [] # Cycle through each XML file for file in os.listdir(in_path): if file.endswith(".xml"): # Load XML in_xml = os.path.join(in_path, file) xml_tree = ET.parse(in_xml) xml_root = xml_tree.getroot() # Cycle through each host for host in xml_root.findall('host'): ip = host.find('address').get('addr') ports_element = host.findall('ports') port_element = ports_element[0].findall('port') # Check Every Port for scanned_port in port_element: port_id = scanned_port.get('portid') script_element = scanned_port.find('script') # SSL Cert Top Level if script_element: ssl_element = script_element.findall('table') # Cycle through each certificate for ssl in ssl_element: # Subject Field if ssl.attrib.get('key') == 'subject': for data in ssl: if data.attrib.get('key') == 'commonName': host_common_name = data.text # Issuer Field if ssl.attrib.get('key') == 'issuer': for data in ssl: if data.attrib.get('key') == 'commonName': issuer_common_name = data.text # Validity Field if ssl.attrib.get('key') == 'validity': for data in ssl: if data.attrib.get('key') == 'notBefore': cert_start = data.text cert_start = re.findall(reg_date, cert_start)[0] if data.attrib.get('key') == 'notAfter': cert_end = data.text cert_end = re.findall(reg_date, cert_end)[0] cert_info_list = f"{ip},{port_id},{host_common_name},{issuer_common_name},{cert_start},{cert_end}" # Load Data cert_info_list = cert_info_list.split(',') cert_info_report.append(cert_info_list) # Reset results for next host cert_info_list = '' # CSV Data with open('detected_ssl_certs.csv', 'w') as csvFile: writer = csv.writer(csvFile) # CSV Headers writer.writerow(["IP", "Port","CommonName","IssuerCommon","CertStart", "CertEnd"]) for ci in cert_info_report: if len(ci) == 6: row_ip = ci[0] row_port = ci[1] row_cn = ci[2] row_ion = ci[3] row_cs = ci[4] row_ce = ci[5] # Write results to row writer.writerow([row_ip,row_port,row_cn, row_ion, row_cs, row_ce]) if __name__ == '__main__': main()
Detected Cipher Suites
These reports are built to flag insecure cipher suites like that of virtually any vulnerability scanner. The risk levels were determined by the grade threshold output by Nmap’s ssl-enum-ciphers NSE script (https://nmap.org/nsedoc/scripts/ssl-enum-ciphers.html). My own preference is to treat F, E and D as high risk and C as a moderate risk, but you can tweak that within the script itself.
Detected High Risk Ciphers
IP | Port | Cipher Suite |
---|---|---|
192.168.0.154 | 443 | “TLS_RSA_WITH_NULL_SHA (F) |
TLS_RSA_WITH_NULL_MD5 (F)” | ||
192.168.0.155 | 443 | “TLS_RSA_EXPORT1024_WITH_RC2_CBC_56_MD5 (D) |
TLS_RSA_WITH_NULL_SHA (F) | ||
TLS_RSA_EXPORT1024_WITH_RC4_56_SHA (D) | ||
TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5 (E) | ||
TLS_RSA_EXPORT_WITH_DES40_CBC_SHA (E) | ||
TLS_RSA_WITH_NULL_MD5 (F) | ||
TLS_RSA_EXPORT_WITH_RC4_40_MD5 (E) | ||
TLS_RSA_EXPORT1024_WITH_RC4_56_MD5 (D)” |
Detected Moderate Risk Ciphers
IP | Port | Cipher Suite |
---|---|---|
192.168.0.156 | 443 | “TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA (C) |
TLS_RSA_WITH_RC4_128_MD5 (C) | ||
TLS_RSA_WITH_3DES_EDE_CBC_SHA (C) | ||
TLS_RSA_WITH_RC4_128_SHA (C)” |
#!/usr/bin/python3 import xml.etree.ElementTree as ET import csv import os def main(): # Cipher Risk Lists ciphers_list = [] flagged_ciphers = '' # Grade Threshold Lists high_risk_grades = ['D','E','F'] moderate_risk_grades = ['C'] high_risk_ciphers = [] moderate_risk_ciphers = [] high_risk_flagged_list = '' moderate_risk_flagged_list = '' # Path to directory with host XML files in_path= '/home/tristram/Scans/Stage_5/' for file in os.listdir(in_path): if file.endswith(".xml"): # Load XML in_xml = os.path.join(in_path, file) xml_tree = ET.parse(in_xml) xml_root = xml_tree.getroot() # Cycle through each host for host in xml_root.findall('host'): ip = host.find('address').get('addr') ports_element = host.findall('ports') port_element = ports_element[0].findall('port') # Cycle through every port for scanned_port in port_element: port_id = scanned_port.get('portid') script_element = scanned_port.find('script') if script_element: script_element = script_element.findall('table') # Cycle through script element for tls_protocol in script_element: # Cycle through TLS protocol for protocol in tls_protocol: if protocol.attrib.get('key') == 'ciphers': # Cycle through each cipher for entry in protocol: for en in entry: if en.attrib.get('key') == 'name': name = en.text if en.attrib.get('key') == 'strength': grade = en.text # Risk based on grade if grade in high_risk_grades: high_risk_flagged_list += f"{name} ({grade})" + ',' if grade in moderate_risk_grades: moderate_risk_flagged_list += f"{name} ({grade})" + ',' # Stage flagged data for current host if high_risk_flagged_list: high_risk_flagged_list = list(set(high_risk_flagged_list.strip(',').split(','))) high_risk_ciphers.append([ip,port_id,high_risk_flagged_list]) if moderate_risk_flagged_list: moderate_risk_flagged_list = list(set(moderate_risk_flagged_list.strip(',').split(','))) moderate_risk_ciphers.append([ip,port_id,moderate_risk_flagged_list]) # Reset results for next host high_risk_flagged_list = '' moderate_risk_flagged_list = '' # Create High Risk Report with open('high_risk_tls_ciphers.csv', 'w') as file: writer = csv.writer(file) # CSV Headers writer.writerow(["IP", "Port","Cipher Suite"]) for hrc in high_risk_ciphers: row_ip = hrc[0] row_port = hrc[1] row_cs = 'rn'.join(hrc[2]) # Write results to row writer.writerow([row_ip,row_port,row_cs]) # Create Moderate Risk Report with open('moderate_risk_tls_ciphers.csv', 'w') as file: writer = csv.writer(file) # CSV Headers writer.writerow(["IP", "Port","Cipher Suite"]) for mrc in moderate_risk_ciphers: row_ip = mrc[0] row_port = mrc[1] row_cs = 'rn'.join(mrc[2]) # Write results to row writer.writerow([row_ip,row_port,row_cs]) if __name__ == '__main__': main()
Detected 3DES Ciphers
IP | Port | Cipher Suite |
---|---|---|
192.168.0.137 | 443 | “TLS_RSA_WITH_3DES_EDE_CBC_SHA |
TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA” | ||
192.168.0.138 | 443 | TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA |
192.168.0.139 | 443 | “TLS_RSA_WITH_3DES_EDE_CBC_SHA |
TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA” |
#!/usr/bin/python3 import xml.etree.ElementTree as ET import csv import os def main(): # Cipher Risk Lists ciphers_list = [] flagged_ciphers = '' # Path to directory with host XML files in_path= '/home/tristram/Scans/Stage_5/' for file in os.listdir(in_path): if file.endswith(".xml"): # Load XML in_xml = os.path.join(in_path, file) xml_tree = ET.parse(in_xml) xml_root = xml_tree.getroot() # Cycle through each host for host in xml_root.findall('host'): ip = host.find('address').get('addr') ports_element = host.findall('ports') port_element = ports_element[0].findall('port') # Cycle through every port for scanned_port in port_element: port_id = scanned_port.get('portid') script_element = scanned_port.find('script') if script_element: script_element = script_element.findall('table') # Cycle through script element for tls_protocol in script_element: # Cycle through TLS protocol for protocol in tls_protocol: if protocol.attrib.get('key') == 'ciphers': # Cycle through each cipher for entry in protocol: for en in entry: if en.attrib.get('key') == 'name': name = en.text if en.attrib.get('key') == 'strength': # Check for targeted cipher if '3DES' in name: flagged_ciphers += name + ',' # Stage flagged data for current host if flagged_ciphers: flagged_ciphers = list(set(flagged_ciphers.strip(',').split(','))) ciphers_list.append([ip,port_id,flagged_ciphers]) # Reset results for next host flagged_ciphers = '' # Create NULL Cipher Report with open('detected_3des_ciphers.csv', 'w') as file: writer = csv.writer(file) # CSV Headers writer.writerow(["IP", "Port","Cipher Suite"]) for entry in ciphers_list: row_ip = entry[0] row_port = entry[1] row_cs = 'rn'.join(entry[2]) # Write results to row writer.writerow([row_ip,row_port,row_cs]) if __name__ == '__main__': main()
Detected RC4 Ciphers
IP | Port | Cipher Suite |
---|---|---|
192.168.0.148 | 443 | “TLS_RSA_WITH_RC4_128_MD5 |
TLS_RSA_WITH_RC4_128_SHA” | ||
192.168.0.149 | 443 | “TLS_RSA_EXPORT1024_WITH_RC4_56_SHA |
TLS_RSA_WITH_RC4_128_MD5 | ||
TLS_RSA_WITH_RC4_128_SHA | ||
TLS_RSA_EXPORT1024_WITH_RC4_56_MD5 | ||
TLS_RSA_EXPORT_WITH_RC4_40_MD5″ |
#!/usr/bin/python3 import xml.etree.ElementTree as ET import csv import os def main(): # Cipher Risk Lists ciphers_list = [] flagged_ciphers = '' # Path to directory with host XML files in_path= '/home/tristram/Scans/Stage_5/' for file in os.listdir(in_path): if file.endswith(".xml"): # Load XML in_xml = os.path.join(in_path, file) xml_tree = ET.parse(in_xml) xml_root = xml_tree.getroot() # Cycle through each host for host in xml_root.findall('host'): ip = host.find('address').get('addr') ports_element = host.findall('ports') port_element = ports_element[0].findall('port') # Cycle through every port for scanned_port in port_element: port_id = scanned_port.get('portid') script_element = scanned_port.find('script') if script_element: script_element = script_element.findall('table') # Cycle through script element for tls_protocol in script_element: # Cycle through TLS protocol for protocol in tls_protocol: if protocol.attrib.get('key') == 'ciphers': # Cycle through each cipher for entry in protocol: for en in entry: if en.attrib.get('key') == 'name': name = en.text if en.attrib.get('key') == 'strength': # Check for targeted cipher if 'RC4' in name: flagged_ciphers += name + ',' # Stage flagged data for current host if flagged_ciphers: flagged_ciphers = list(set(flagged_ciphers.strip(',').split(','))) ciphers_list.append([ip,port_id,flagged_ciphers]) # Reset results for next host flagged_ciphers = '' # Create NULL Cipher Report with open('detected_rc4_ciphers.csv', 'w') as file: writer = csv.writer(file) # CSV Headers writer.writerow(["IP", "Port","Cipher Suite"]) for entry in ciphers_list: row_ip = entry[0] row_port = entry[1] row_cs = 'rn'.join(entry[2]) # Write results to row writer.writerow([row_ip,row_port,row_cs]) if __name__ == '__main__': main()
Wrapping Up
There was a lot of information presented here as well as a lot of Python code. It is my hope that you found it useful and perhaps sparked some inspirational fires for you to think about designing your own enumeration tools or other automated workflows. I invite you to look at your own processes and see if the information you have learned here can be used to help enhance your own processes.
Tristram
About the Author
Tristram (gh0x0st) develops strategies and implements controls to defend healthcare from malicious entities as well as validate security controls through penetration testing. You can find Tristram on GitHub as well as the OffSec Discord as one of our moderators.
Latest from OffSec
OffSec News
Evolve APAC 2024: Key Insights
Discover key insights from Evolve APAC 2024 on building skills, career growth, and tackling cybersecurity challenges with expert advice.
Nov 21, 2024
8 min read
Enterprise Security
How to Use Assessments for a Skills Gap Analysis
Discover how OffSec’s Learning Paths help organizations perform skills gap analyses, validate expertise, and strengthen cybersecurity teams.
Nov 19, 2024
4 min read
Enterprise Security
The Human Side of Incident Response
Effective incident response requires decision-making, adaptability, collaboration, stress management, and a commitment to continuous learning.
Nov 8, 2024
5 min read