THE

SPRAWL

  •  
  •  
  •  
  • Nmap Scripting Engine became a part of the mainline codebase with the release of Nmap 4.21ALPHA1 back in December, 2006. Today, the NSE library has grown to more than 400 scripts covering an amazing array of different network technologies (from SMB vulnerability checks to Stuxnet detection and everything in between). The power of NSE lies in its versatile library collection which allows easy interaction with most major network services and protocols.

    The challenge

    Often times it is necessary to scan a large number of hosts for a new vulnerability which does not yet have signatures in existing scanning engines. The challenge is to quickly develop a signature and scan your enterprise to enumerate hosts running the application and check them for the vulnerability.

    You may be familiar with a scripting language (e.g. Python, Perl, etc.) and are likely to quickly prototype a vulnerability check for the application. However, once faced with scanning hundreds or thousands of IP addresses, you will quickly realize that what works great for one or two targets is terribly inefficient for a larger number of hosts.

    Nmap to the rescue! By using a combination of embedded Lua language and a powerful collection of libraries you will be able to develop the same solution that will work on a large scale all while using nmap's lightning fast host and port scanning engine.

    Imlementation

    Nmap Scripting Engine scripts are implemented using Lua programming language, Nmap API and a number of really powerful NSE Libraries. For the purposes of this article, let's imagine a fictional vulnerability in a fictional web application called ArcticFission. Just like many other web applications it can be detected by the presence of a certain file, let's call it /arcticfission.html. The vulnerability can be detected by analyzing the content of this file, again let's assume the file contains a version string which we will extract using regular expressions and compare against a known vulnerable value. Sounds pretty simple, so let's get started!

    Skeleton Code

    In the tradition of K&R, let's begin with a script that will simply print out 'Hello World' for all open HTTP ports. Open a text editor and write the following snippet into http-vuln-check.nse in your home directory.

    -- The Head Section --
    -- The Rule Section --
    portrule = function(host, port)
        return port.protocol == "tcp"
                and port.number == 80
                and port.state == "open"
    end
    
    -- The Action Section --
    action = function(host, port)
        return "Hello world!"
    end
    

    NOTE: Anything beginning with -- is a comment.

    NSE scripts consist of three sections:

    • The Head Section contains meta-data which describes script's functionality, author, impact, category and other descriptive data. This section will be left blank for now; however, we will populate it later once the sample nse script is more complete.
    • The Rule Section defines necessary conditions for the script to execute. This section must contain at least one function from this list: portrule, hostrule, prerule, postrule. For the purposes of this tutorial (and the majority of scripts), I will concentrate on the portrule which can perform checks on both host and port properties before deciding to run. In the script above, portrule takes advantage of nmap's API to check for an open TCP port 80.
    • The Action Section defines the script logic. In the tradition of K&R, I will simply output "Hello world!" for any open port 80. It is important to note that script output displayed during nmap's execution will be based on the string returned by this section.

    Let's run the above script as follows:

    # nmap --script http-vuln-check thesprawl.org -p 22,80,443
    
    Starting Nmap 6.25 ( http://nmap.org ) at 2013-01-09 02:36 EST
    Nmap scan report for thesprawl.org (108.59.3.64)
    Host is up (0.023s latency).
    rDNS record for 108.59.3.64: web219.webfaction.com
    PORT    STATE    SERVICE
    22/tcp  filtered ssh
    80/tcp  open     http
    |_http-vuln-check: Hello world!
    443/tcp open     https
    
    Nmap done: 1 IP address (1 host up) scanned in 1.77 seconds
    

    http-vuln-check was triggered for an open TCP port 80 just as we have specified. It was not triggered for a filtered ssh port or an open https port since they were not specified in the portrule function. The output of the action function was displayed in the script output.

    Using NSE Libraries

    What makes NSE particularly powerful is its excellent collection of utility libraries. For example, we can simplify the generation of portrule by using a function from a library to check for http ports. Here is an updated script using the shortport library containing such common definitions:

    local shortport = require "shortport"
    
    -- The Rule Section --
    portrule = shortport.http
    
    -- The Action Section --
    action = function(host, port)
        return "Hello world!"
    end
    

    Repeating the same nmap scan will yield slightly different results:

    # nmap --script http-vuln-check thesprawl.org -p 22,80,443
    
    Starting Nmap 6.25 ( http://nmap.org ) at 2013-01-09 02:51 EST
    Nmap scan report for thesprawl.org (108.59.3.64)
    Host is up (0.032s latency).
    rDNS record for 108.59.3.64: web219.webfaction.com
    PORT    STATE SERVICE
    22/tcp  open  ssh
    80/tcp  open  http
    |_http-vuln-check: Hello world!
    443/tcp open  https
    |_http-vuln-check: Hello world!
    
    Nmap done: 1 IP address (1 host up) scanned in 0.19 seconds
    

    The script has now executed on port 443/tcp as well as 80/tcp. This is because shortport.http will be true for any of the likely HTTP ports (80, 443, 631, 7080, 8080, 8088, 5800, 3872, 8180, 8000). Even more exciting, shortport.http will actually match based on nmap's service detection for any "http", "https", "ipp", "http-alt", "vnc-http", "oem-agent", "soap", "http-proxy" services running on a non-standard port as well. That's some powerful stuff! Check NSE Library shortport documentation for additional details.

    Service Detection

    Let's shift our focus to the script's logic contained in the action section. The vulnerability we are trying to identify is in a web application, so the first step would be to identify our fictional ArcticFission application is indeed running on a webserver. In order to perform the detection, I will attempt to retrieve "/arcticfission.html" from a webserver and observe the HTTP return code as follows:

    local shortport = require "shortport"
    local http = require "http"
    
    -- The Rule Section --
    portrule = shortport.http
    
    -- The Action Section --
    action = function(host, port)
    
        local uri = "/arcticfission.html"
        local response = http.get(host, port, uri)
        return response.status
    
    end
    

    In the code snippet above, I have used NSE Library http to quickly retrieve and process web pages.

    # nmap --script http-vuln-check localhost thesprawl.org -p 80,443
    Starting Nmap 6.25 ( http://nmap.org ) at 2013-01-09 03:44 EST
    Nmap scan report for localhost (127.0.0.1)
    Host is up (0.000040s latency).
    Other addresses for localhost (not scanned): 127.0.0.1
    PORT    STATE SERVICE
    80/tcp  open  http
    |_http-vuln-check: 200
    443/tcp open  https
    |_http-vuln-check: 200
    
    Nmap scan report for thesprawl.org (108.59.3.64)
    Host is up (0.023s latency).
    rDNS record for 108.59.3.64: web219.webfaction.com
    PORT    STATE SERVICE
    80/tcp  open  http
    |_http-vuln-check: 404
    443/tcp open  https
    |_http-vuln-check: 404
    
    Nmap done: 2 IP addresses (2 hosts up) scanned in 8.63 seconds
    

    The above output shows two webservers with and without 'arcticfission.html' file. Notice that 'http' library works transparently for https and http ports, so you don't have to implement any additional TLS/SSL logic.

    Let's introduce additional code to only return script output for services which are running the vulnerable web application:

    local shortport = require "shortport"
    local http = require "http"
    
    -- The Rule Section --
    portrule = shortport.http
    
    -- The Action Section --
    action = function(host, port)
    
        local uri = "/arcticfission.html"
        local response = http.get(host, port, uri)
    
        if ( response.status == 200 ) then
            return response.body
        end
    
    end
    

    In the script above, I am returning HTTP response body in case the response.code is equal to 200.

    Notice that not returning anything or returning an empty string ("") will result in no script output being displayed at all (even script's name):

    # nmap --script http-vuln-check localhost thesprawl.org -p 80,443
    
    Starting Nmap 6.25 ( http://nmap.org ) at 2013-01-09 04:03 EST
    Nmap scan report for localhost (127.0.0.1)
    Host is up (0.000049s latency).
    Other addresses for localhost (not scanned): 127.0.0.1
    PORT    STATE SERVICE
    80/tcp  open  http
    | http-vuln-check: <html>
    | <head>
    | <title>ArcticFission 1.0</title>
    | </head>
    | <body>
    | <h1>Welcome to ArcticFission 1.0</h1>
    | </body>
    |_</html>
    443/tcp open  https
    | http-vuln-check: <html>
    | <head>
    | <title>ArcticFission 1.0</title>
    | </head>
    | <body>
    | <h1>Welcome to ArcticFission 1.0</h1>
    | </body>
    |_</html>
    
    Nmap scan report for thesprawl.org (108.59.3.64)
    Host is up (0.023s latency).
    rDNS record for 108.59.3.64: web219.webfaction.com
    PORT    STATE SERVICE
    80/tcp  open  http
    443/tcp open  https
    
    Nmap done: 2 IP addresses (2 hosts up) scanned in 8.64 seconds
    

    At this point we can clearly identify hosts running ArcticFission and are now ready to check for the actual vulnerability.

    Vulnerability Detection

    Most of the time it is possible to detect a vulnerability by simply detecting service version. In this case, our fictional server returns a banner with the version number. Let's extract the version number from the title and display it in the script's output:

    local shortport = require "shortport"
    local http = require "http"
    local string = require "string"
    
    -- The Rule Section --
    portrule = shortport.http
    
    -- The Action Section --
    action = function(host, port)
    
        local uri = "/arcticfission.html"
        local response = http.get(host, port, uri)
    
        if ( response.status == 200 ) then
            local title = string.match(response.body, "<[Tt][Ii][Tt][Ll][Ee][^>]*>ArcticFission ([^<]*)</[Tt][Ii][Tt][Ll][Ee]>")
            return title
        end
    
    end
    

    Using some straightforward regex from Lua's string library, we can extract and display page title:

    # nmap --script http-vuln-check localhost -p 80,443
    
    Starting Nmap 6.25 ( http://nmap.org ) at 2013-01-09 04:17 EST
    Nmap scan report for localhost (127.0.0.1)
    Host is up (0.000031s latency).
    Other addresses for localhost (not scanned): 127.0.0.1
    PORT    STATE SERVICE
    80/tcp  open  http
    |_http-vuln-check: 1.0
    443/tcp open  https
    |_http-vuln-check: 1.0
    

    At this point, all we have to do is compare the extracted value against a known vulnerable version and report appropriate results:

    local shortport = require "shortport"
    local http = require "http"
    local string = require "string"
    
    -- The Rule Section --
    portrule = shortport.http
    
    -- The Action Section --
    action = function(host, port)
    
        local uri = "/arcticfission.html"
        local response = http.get(host, port, uri)
    
        if ( response.status == 200 ) then
            local title = string.match(response.body, "<[Tt][Ii][Tt][Ll][Ee][^>]*>ArcticFission ([^<]*)</[Tt][Ii][Tt][Ll][Ee]>")
    
            if ( title == "1.0" ) then
                return "Vulnerable"
            else        
                return "Not Vulnerable"
            end
        end
    end
    

    And here is the final output:

    # nmap --script http-vuln-check localhost -p 80,443
    
    Starting Nmap 6.25 ( http://nmap.org ) at 2013-01-09 04:24 EST
    Nmap scan report for localhost (127.0.0.1)
    Host is up (0.000034s latency).
    Other addresses for localhost (not scanned): 127.0.0.1
    PORT    STATE SERVICE
    80/tcp  open  http
    |_http-vuln-check: Vulnerable
    443/tcp open  https
    |_http-vuln-check: Vulnerable
    
    Nmap done: 1 IP address (1 host up) scanned in 8.08 seconds
    

    Another approach to version detection (and a possible way to eliminate false positives) is to generate and compare page hash against a known value. For this version of the script I am going to use the NSE Library openssl:

    local shortport = require "shortport"
    local http = require "http"
    local stdnse = require "stdnse"
    local openssl = require "openssl"
    
    -- The Rule Section --
    portrule = shortport.http
    
    -- The Action Section --
    action = function(host, port)
    
        local uri = "/arcticfission.html"
        local response = http.get(host, port, uri)
    
        if ( response.status == 200 ) then
            local vulnsha1 = "984c6f159d5b5baba8fe23dfa5372d047ed1de2e"
            local sha1 = string.lower(stdnse.tohex(openssl.sha1(response.body)))
    
            if ( sha1 == vulnsha1 ) then
                return "Vulnerable"
            else
                return "Not Vulnerable"
            end
        end
    end
    

    The above script will produce same output.

    Adding a bit of stealth

    It is important to fully test your script's execution especially when using third party libraries. For example, here is a snippet from the debugging output:

    # nmap --script http-vuln-check localhost -p 80,443 --script-trace
    Starting Nmap 6.25 ( http://nmap.org ) at 2013-01-11 01:42 EST
    NSOCK (0.0490s) nsi_new (IOD #1)
    NSOCK (0.0790s) TCP connection requested to 127.0.0.1:80 (IOD #1) EID 8
    NSOCK (0.0790s) nsi_new (IOD #2)
    NSOCK (0.0790s) SSL connection requested to 127.0.0.1:443/tcp (IOD #2) EID 17
    NSOCK (0.0800s) Callback: CONNECT SUCCESS for EID 8 [127.0.0.1:80]
    NSE: TCP 127.0.0.1:56968 > 127.0.0.1:80 | CONNECT
    NSOCK (0.0800s) Callback: SSL-CONNECT SUCCESS for EID 17 [127.0.0.1:443]
    NSE: TCP 127.0.0.1:55825 > 127.0.0.1:443 | CONNECT
    NSE: TCP 127.0.0.1:56968 > 127.0.0.1:80 | 00000000: 47 45 54 20 2f 61 72 63 74 69 63 66 69 73 73 69 GET /arcticfissi
    00000010: 6f 6e 2e 68 74 6d 6c 20 48 54 54 50 2f 31 2e 31 on.html HTTP/1.1
    00000020: 0d 0a 55 73 65 72 2d 41 67 65 6e 74 3a 20 4d 6f   User-Agent: Mo
    00000030: 7a 69 6c 6c 61 2f 35 2e 30 20 28 63 6f 6d 70 61 zilla/5.0 (compa
    00000040: 74 69 62 6c 65 3b 20 4e 6d 61 70 20 53 63 72 69 tible; Nmap Scri
    00000050: 70 74 69 6e 67 20 45 6e 67 69 6e 65 3b 20 68 74 pting Engine; ht
    00000060: 74 70 3a 2f 2f 6e 6d 61 70 2e 6f 72 67 2f 62 6f tp://nmap.org/bo
    00000070: 6f 6b 2f 6e 73 65 2e 68 74 6d 6c 29 0d 0a 43 6f ok/nse.html)  Co
    00000080: 6e 6e 65 63 74 69 6f 6e 3a 20 63 6c 6f 73 65 0d nnection: close 
    00000090: 0a 48 6f 73 74 3a 20 6c 6f 63 61 6c 68 6f 73 74  Host: localhost
    000000a0: 0d 0a 0d 0a                                         
    ...
    

    In the trace above, the NSE Library 'http' is using a default User-Agent string "Mozilla/5.0 (compatible; Nmap Scripting Engine; http://nmap.org/book/nse.html)". You may want to change this and possibly other connection parameters for performance or security reasons. There are two options to change the user-agent string. First (easiest) is to simply include an extra command-line script argument to override default user agent:

    # nmap --script http-vuln-check localhost -p 80,443 --script-args="http.useragent='Mozilla/5.0 (compatible; ArcticFission)'"
    

    Alternatively, it is possible to override the User-Agent header parameter (or other request parameters) in the script itself as follows:

    local shortport = require "shortport"
    local http = require "http"
    local stdnse = require "stdnse"
    local string = require "string"
    
    -- The Rule Section --
    portrule = shortport.http
    
    -- The Action Section --
    action = function(host, port)
    
        local uri = "/arcticfission.html"
    
        local options = {header={}}
        options['header']['User-Agent'] = "Mozilla/5.0 (compatible; ArcticFission)"
    
        local response = http.get(host, port, uri, options)
    
        if ( response.status == 200 ) then
            local title = string.match(response.body, "<[Tt][Ii][Tt][Ll][Ee][^>]*>ArcticFission ([^<]*)</[Tt][Ii][Tt][Ll][Ee]>")
    
            if ( title == "1.0" ) then
                return "Vulnerable"
            else        
                return "Not Vulnerable"
            end
        end
    end
    

    Packaging the Script

    There are several important pieces of metadata that I have omitted until this point. In case you decide to distribute the script, you may want to include description, author information, license as well as identify and classify the script based on its function and impact. Below is a fully marked up example:

    -- The Head Section --
    description = [[Sample script to detect a fictional vulnerability
    in a fictional ArcticFission 1.0 web server]]
    author = "iphelix"
    license = "Same as Nmap--See http://nmap.org/book/man-legal.html"
    categories = {"default", "safe"}
    
    local shortport = require "shortport"
    local http = require "http"
    local stdnse = require "stdnse"
    local string = require "string"
    
    -- The Rule Section --
    portrule = shortport.http
    
    -- The Action Section --
    action = function(host, port)
    
        local uri = "/arcticfission.html"
    
        local options = {header={}}
        options['header']['User-Agent'] = "Mozilla/5.0 (compatible; ArcticFission)"
    
        local response = http.get(host, port, uri, options)
    
        if ( response.status == 200 ) then
            local title = string.match(response.body, "<[Tt][Ii][Tt][Ll][Ee][^>]*>ArcticFission ([^<]*)</[Tt][Ii][Tt][Ll][Ee]>")
    
            if ( title == "1.0" ) then
                return "Vulnerable"
            else        
                return "Not Vulnerable"
            end
        end
    end
    

    At last, you may want to include some documentation in the NSEDoc format. Script documentation may include special tags which will be processed by the documentation system (e.g @output for script output, @args for script arguments, @usage for sample command line parameters, etc.). Here is the final script example:

    -- The Head Section --
    description = [[Sample script to detect a fictional vulnerability
    in a fictional ArcticFission 1.0 web server]]
    
    ---
    -- @usage
    -- nmap --script http-vuln-check <target>
    -- @output
    -- PORT   STATE SERVICE
    -- 80/tcp open  http
    -- |_http-vuln-check: Vulnerable
    
    author = "iphelix"
    license = "Same as Nmap--See http://nmap.org/book/man-legal.html"
    categories = {"default", "safe"}
    
    local shortport = require "shortport"
    local http = require "http"
    local stdnse = require "stdnse"
    local string = require "string"
    
    -- The Rule Section --
    portrule = shortport.http
    
    -- The Action Section --
    action = function(host, port)
    
        local uri = "/arcticfission.html"
    
        local options = {header={}}
        options['header']['User-Agent'] = "Mozilla/5.0 (compatible; ArcticFission)"
    
        local response = http.get(host, port, uri, options)
    
        if ( response.status == 200 ) then
            local title = string.match(response.body, "<[Tt][Ii][Tt][Ll][Ee][^>]*>ArcticFission ([^<]*)</[Tt][Ii][Tt][Ll][Ee]>")
    
            if ( title == "1.0" ) then
                return "Vulnerable"
            else        
                return "Not Vulnerable"
            end
        end
    end
    

    Parsing the output

    With your custom vulnerability checking script up and running it is now necessary to parse the results and produce a meaningful report. Unfortunately older 'gnmap' output does not include script output, so we will have to parse the newer 'xml' output:

    #!/usr/bin/env python
    # nmap-xml-parse by iphelix
    import sys
    from xml.dom.minidom import parse
    
    if len(sys.argv) != 2:
        print "Usage: %s nmap_output.xml"
        sys.exit(1)
    
    nmap = parse(sys.argv[1])
    
    for host in nmap.getElementsByTagName("host"):
        addresses = [addr.getAttribute("addr") for addr in host.getElementsByTagName("address")]
    
        for port in host.getElementsByTagName("port"):
            portid = port.getAttribute("portid")
    
            for script in port.getElementsByTagName("script"):
                if script.getAttribute("id") == "http-vuln-check":
                    output = script.getAttribute("output")
    
                    for address in addresses:
                        print "%s,%s,%s" % (address, portid, output)
    

    The above Python script can be used as follows:

    # nmap --script http-vuln-check localhost -p 80,443 -oA http-vuln
    
    Starting Nmap 6.25 ( http://nmap.org ) at 2013-01-11 02:47 EST
    Nmap scan report for localhost (127.0.0.1)
    Host is up (0.000066s latency).
    Other addresses for localhost (not scanned): 127.0.0.1
    PORT    STATE SERVICE
    80/tcp  open  http
    |_http-vuln-check: Vulnerable
    443/tcp open  https
    |_http-vuln-check: Vulnerable
    
    Nmap done: 1 IP address (1 host up) scanned in 8.08 seconds
    # ./nmap-xml-parse.py http-vuln.xml
    127.0.0.1,80,Vulnerable
    127.0.0.1,443,Vulnerable
    

    The output can be easily parsed as a CSV file. Feel free to adopt the script to your specific needs.

    Vulnerability Management

    There are several problems with the above generic vulnerability discovery script. First it lacks vulnerability meta-data that may be useful to users of the script, furthermore adding any additional output results in an increased parsing complexity. Second issue is the need for an additional output parsing script to aggregate vulnerabilities across the scan. Both of these problems can be easily solved with another excellent Nmap library simply called 'vulns'.

    NSE Vulnerability Library

    The NSE vulns library was developed by Djalal Harouni and Henri Doreau in order to standardize vulnerability presentation and ease vulnerability management. Let's modify the previous example script in order to support the library:

    -- The Head Section --
    description = [[Sample script to detect a fictional vulnerability
    in a fictional ArcticFission 1.0 web server]]
    
    ---
    -- @usage
    -- nmap --script http-vuln-check <target>
    -- @output
    -- PORT    STATE SERVICE
    -- 80/tcp  open  http
    -- | http-vuln-check: 
    -- |   VULNERABLE:
    -- |   ArcticFission 1.0 Vulnerability
    -- |     State: VULNERABLE
    -- |     IDs:  CVE:CVE-XXXX-XX
    -- |     References:
    -- |_      http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-XXXX-XX
    
    
    author = "iphelix"
    license = "Same as Nmap--See http://nmap.org/book/man-legal.html"
    categories = {"default", "safe"}
    
    local shortport = require "shortport"
    local http = require "http"
    local stdnse = require "stdnse"
    local string = require "string"
    local vulns = require "vulns"
    
    -- The Rule Section --
    portrule = shortport.http
    
    -- The Action Section --
    action = function(host, port)
    
        -- The Vuln Definition Section --
        local vuln = {
            title = "ArcticFission 1.0 Vulnerability",
            state = vulns.STATE.NOT_VULN, --default
            IDS = { CVE = 'CVE-XXXX-XX' }
        }   
        local report = vulns.Report:new(SCRIPT_NAME, host, port)
    
        local uri = "/arcticfission.html"
    
        local options = {header={}}
        options['header']['User-Agent'] = "Mozilla/5.0 (compatible; ArcticFission)"
    
        local response = http.get(host, port, uri, options)
    
        if ( response.status == 200 ) then
            local title = string.match(response.body, "<[Tt][Ii][Tt][Ll][Ee][^>]*>ArcticFission ([^<]*)</[Tt][Ii][Tt][Ll][Ee]>")
    
            if ( title == "1.0" ) then
                vuln.state = vulns.STATE.VULN
            else        
                vuln.state = vulns.STATE.NOT_VULN
            end
        end
    
        return report:make_output(vuln)
    end
    

    There are several changes made to the original script. First, notice the vulnerability definition table containing detailed information about the vulnerability:

    -- The Vuln Definition Section --
    local vuln = {
        title = "ArcticFission 1.0 Vulnerability",
        state = vulns.STATE.NOT_VULN, --default
        IDS = { CVE = 'CVE-XXXX-XX' }
    }
    

    This section can actually be expanded with more standardized vulnerability descriptors such as disclosure date, CSV scores, risk factors, etc.

    Next, notice that vulnerability state is recorded in the above structure using vulns.STATE.VULN and vulns.STATE.NOT_VULN variables with the latter being set as default:

    if ( title == "1.0" ) then
        vuln.state = vulns.STATE.VULN
    else        
        vuln.state = vulns.STATE.NOT_VULN
    end
    

    At last (and the most powerful part) we have added an automatic report generator using the 'make_output' function.

    Here is an updated nmap script output:

    # nmap --script http-vuln-check localhost -p 80,443
    
    Starting Nmap 6.25 ( http://nmap.org ) at 2013-01-11 02:52 EST
    Nmap scan report for localhost (127.0.0.1)
    Host is up (0.000038s latency).
    Other addresses for localhost (not scanned): 127.0.0.1
    PORT    STATE SERVICE
    80/tcp  open  http
    | http-vuln-check: 
    |   VULNERABLE:
    |   ArcticFission 1.0 Vulnerability
    |     State: VULNERABLE
    |     IDs:  CVE:CVE-XXXX-XX
    |     References:
    |_      http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-XXXX-XX
    443/tcp open  https
    | http-vuln-check: 
    |   VULNERABLE:
    |   ArcticFission 1.0 Vulnerability
    |     State: VULNERABLE
    |     IDs:  CVE:CVE-XXXX-XX
    |     References:
    |_      http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-XXXX-XX
    

    At the expense of a few additional lines of code, you gain automatically generated vulnerability report standardized across a growing number of nmap scripts.

    Aggregating output

    The true magic begins when performing vulnerability aggregation using the 'vulns' library. As described in the documentation, let's write the following code snippet to 'vulns-post-process.nse':

    local stdnse = require "stdnse"
    local vulns = require "vulns"
    
    local FID -- my script FILTER ID
    
    prerule = function()
      FID = vulns.save_reports()
      if FID then
        return true
      end
      return false
    end
    
    postrule = function()
      if nmap.registry[SCRIPT_NAME] then
        FID = nmap.registry[SCRIPT_NAME].FID
        if vulns.get_ids(FID) then
          return true
        end
      end
      return false
    end
    
    prerule_action = function()
      nmap.registry[SCRIPT_NAME] = nmap.registry[SCRIPT_NAME] or {}
      nmap.registry[SCRIPT_NAME].FID = FID
      return nil
    end
    
    postrule_action = function()
      return vulns.make_output(FID) -- show all the vulnerabilities
    end
    
    local tactions = {
      prerule = prerule_action,
      postrule = postrule_action,
    }
    
    action = function(...) return tactions[SCRIPT_TYPE](...) end
    

    Now execute nmap with both scripts running at the same time:

    # nmap --script http-vuln-check,vulns-post-process localhost thesprawl.org -p 80,443
    
    Starting Nmap 6.25 ( http://nmap.org ) at 2013-01-11 03:12 EST
    Nmap scan report for localhost (127.0.0.1)
    Host is up (0.000051s latency).
    Other addresses for localhost (not scanned): 127.0.0.1
    PORT    STATE SERVICE
    80/tcp  open  http
    | http-vuln-check: 
    |   VULNERABLE:
    |   ArcticFission 1.0 Vulnerability
    |     State: VULNERABLE
    |     IDs:  CVE:CVE-XXXX-XX
    |     References:
    |_      http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-XXXX-XX
    443/tcp open  https
    | http-vuln-check: 
    |   VULNERABLE:
    |   ArcticFission 1.0 Vulnerability
    |     State: VULNERABLE
    |     IDs:  CVE:CVE-XXXX-XX
    |     References:
    |_      http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-XXXX-XX
    
    Nmap scan report for thesprawl.org (108.59.3.64)
    Host is up (0.012s latency).
    rDNS record for 108.59.3.64: web219.webfaction.com
    PORT    STATE SERVICE
    80/tcp  open  http
    443/tcp open  https
    
    Post-scan script results:
    | vulns-post-process: 
    |   Vulnerability report for 108.59.3.64: NOT VULNERABLE
    |
    |   Vulnerability report for 127.0.0.1: VULNERABLE
    |   Target: localhost (127.0.0.1)  Port: 80/http
    |   ArcticFission 1.0 Vulnerability
    |     State: VULNERABLE
    |     IDs:  CVE:CVE-XXXX-XX
    |     References:
    |       http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-XXXX-XX
    |     Reported by scripts: http-vuln-check
    |
    |   Target: localhost (127.0.0.1)  Port: 443/https
    |   ArcticFission 1.0 Vulnerability
    |     State: VULNERABLE
    |     IDs:  CVE:CVE-XXXX-XX
    |     References:
    |       http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-XXXX-XX
    |_    Reported by scripts: http-vuln-check
    Nmap done: 2 IP addresses (2 hosts up) scanned in 0.42 seconds
    

    As you can see, it is now possible to aggregate and report on results across multiple hosts. The script operates by saving vulnerability in nmap's registry which is persistent across multiple host scans and later formats combined output as a postrule script.

    In fact, it is possible to further format the post-scan output. Here is an updated 'vulns-post-process' script which will output a list of vulnerable hosts:

    local stdnse = require "stdnse"
    local vulns = require "vulns"
    
    local FID -- my script FILTER ID
    
    prerule = function()
      FID = vulns.save_reports()
      if FID then
        return true
      end
      return false
    end
    
    postrule = function()
      if nmap.registry[SCRIPT_NAME] then
        FID = nmap.registry[SCRIPT_NAME].FID
        if vulns.get_ids(FID) then
          return true
        end
      end
      return false
    end
    
    prerule_action = function()
      nmap.registry[SCRIPT_NAME] = nmap.registry[SCRIPT_NAME] or {}
      nmap.registry[SCRIPT_NAME].FID = FID
      return nil
    end
    
    postrule_action = function()
      local filter = {state = vulns.STATE.VULN}
      local list = vulns.find(FID, filter)
      if list then
        local out = {}
        for _, vuln_table in ipairs(list) do
          local ip = vuln_table.host.ip
          local port = vuln_table.port.number
          local state = vulns.STATE_MSG[vuln_table.state]
          local title = vuln_table.title
          table.insert(out, string.format("%s:%d - %s - %s", ip, port, title, state))
    
        end
        return stdnse.format_output(true, out)
      end
    end
    
    local tactions = {
      prerule = prerule_action,
      postrule = postrule_action,
    }
    
    action = function(...) return tactions[SCRIPT_TYPE](...) end
    

    Here is an updated nmap output:

    # nmap --script http-vuln-check,vulns-post-process localhost thesprawl.org -p 80,443
    
    Starting Nmap 6.25 ( http://nmap.org ) at 2013-01-11 04:15 EST
    Nmap scan report for localhost (127.0.0.1)
    Host is up (0.000067s latency).
    Other addresses for localhost (not scanned): 127.0.0.1
    PORT    STATE SERVICE
    80/tcp  open  http
    | http-vuln-check: 
    |   VULNERABLE:
    |   ArcticFission 1.0 Vulnerability
    |     State: VULNERABLE
    |     IDs:  CVE:CVE-XXXX-XX
    |     References:
    |_      http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-XXXX-XX
    443/tcp open  https
    | http-vuln-check: 
    |   VULNERABLE:
    |   ArcticFission 1.0 Vulnerability
    |     State: VULNERABLE
    |     IDs:  CVE:CVE-XXXX-XX
    |     References:
    |_      http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-XXXX-XX
    
    Nmap scan report for thesprawl.org (108.59.3.64)
    Host is up (0.012s latency).
    rDNS record for 108.59.3.64: web219.webfaction.com
    PORT    STATE SERVICE
    80/tcp  open  http
    443/tcp open  https
    
    Post-scan script results:
    | vulns-post-process: 
    |   127.0.0.1:80 - ArcticFission 1.0 Vulnerability - VULNERABLE
    |_  127.0.0.1:443 - ArcticFission 1.0 Vulnerability - VULNERABLE
    Nmap done: 2 IP addresses (2 hosts up) scanned in 13.44 seconds
    

    The final output was generated using the postrule_action function by enumerating a list of vulnerabilities and extracting only relevant information:

    postrule_action = function()
      local filter = {state = vulns.STATE.VULN}
      local list = vulns.find(FID, filter)
      if list then
        local out = {}
        for _, vuln_table in ipairs(list) do
          local ip = vuln_table.host.ip
          local port = vuln_table.port.number
          local state = vulns.STATE_MSG[vuln_table.state]
          local title = vuln_table.title
          table.insert(out, string.format("%s:%d - %s - %s", ip, port, title, state))
    
        end
        return stdnse.format_output(true, out)
      end
    end
    

    Notice that the final list only includes hosts identified as vulnerable. This is due to an additional filter state = vulns.STATE.VULN which ignores not vulnerable hosts when executing the vulns.find().

    Where to go from here

    I hope you are just as excited about Nmap scripting as I am at this point. The best way from here is to study documentation of individual libraries and write more complex scripts covering protocols other than http. Nmap was always an amazing tool, but with the addition of the NSE engine and with the open source community support it may very well become a de facto vulnerability scanning tool on par with many commercial offerings. Thanks, Fyodor and all of the Nmap developers.

    External Links and References

    Published on January 8th, 2013 by iphelix

    sprawlsimilar

    nmap

    nmap (Network MAPper) is a network port scanner with service version and operating system detection engines. The tool was originally developed by Fyodor and published in Phrack Issue 51 in 1997. The tool is command line although a number of GUIs exist. nmap runs on a variety of platforms including Linux, *BSD, Windows, and others. Read more.

    port scanning

    Discovering open ports on a networked system is an important reconnaissance step used to enumerate potentially vulnerable services. In this article you will learn a number of techniques used to perform fast and reliable port scans while bypassing many trivial defenses. Read more.


    sprawlcomments

    All original content on this site is copyright protected and licensed under Creative Commons - Attribution, NonCommercial, ShareAlike 4.0 International.

    π
    ///\oo/\\\