Author Topic: GyoiThon is a growing penetration test tool using Machine Learning.  (Read 339 times)

0 Members and 1 Guest are viewing this topic.

Offline Flavio58

GyoiThon is a growing penetration test tool using Machine Learning.
« Reply #1 on: May 11, 2018, 03:08:45 AM »
Advertisement
https://github.com/gyoisamurai/GyoiThon

Overview
GyoiThon is a growing penetration test tool using Machine Learning.

GyoiThon identifies the software installed on web server (OS, Middleware, Framework, CMS, etc...) based on the learning data. After that, it executes valid exploits for the identified software using Metasploit. Finally, it generates reports of scan results. GyoiThon executes the above processing automatically.

Processing steps
Processing flow
GyoiThon executes the above "Step1" - "Step4" fully automatically.
User's only operation is to input the top URL of the target web server in GyoiThon.

It is very easy!
You can identify vulnerabilities of the web servers without taking time and effort.

Processing flow
Step 1. Gather HTTP responses.
GyoiThon gathers several HTTP responses of target website while crawling.
The following are example of HTTP responses gathered by GyoiThon.

Example.1
HTTP/1.1 200 OK
Date: Tue, 06 Mar 2018 03:01:57 GMT
Connection: close
Content-Type: text/html; charset=UTF-8
Etag: "409ed-183-53c5f732641c0"
Content-Length: 15271

...snip...
Example.2
HTTP/1.1 200 OK
Date: Tue, 06 Mar 2018 06:56:17 GMT
Connection: close
Content-Type: text/html; charset=UTF-8
Set-Cookie: f00e68432b68050dee9abe33c389831e=0eba9cd0f75ca0912b4849777677f587;
path=/;
Content-Length: 37496

...snip...
Example.3
HTTP/1.1 200 OK
Date: Tue, 06 Mar 2018 04:19:19 GMT
Connection: close
Content-Type: text/html; charset=UTF-8
Content-Length: 11819

...snip...

 <script src="/core/misc/drupal.js?v=8.3.1"></script>
Step 2. Identify product name.
GyoiThon identifies product name installed on web server using following two methods.

1. Based on Machine Learning.
By using Machine Learning (Naive Bayes), GyoiThon identifies software based on a combination of slightly different features (Etag value, Cookie value, specific HTML tag etc.) for each software. Naive Bayes is learned using the training data which example below (Training data). Unlike the signature base, Naive Bayes is stochastically identified based on various features included in HTTP response when it cannot be identified software in one feature.

Example.1
Etag: "409ed-183-53c5f732641c0"
GyoiThon can identify the web server software Apache.
This is because GyoiThon learns features of Apache such as "Etag header value (409ed-183-53c5f732641c0). In our survey, Apache use combination of numeral and lower case letters as the Etag value. And, Etag value is separated 4-5 digits and 3-4 digits and 12 digits, final digit is 0 in many cases.

Example.2
Set-Cookie: f00e68432b68050dee9abe33c389831e=0eba9cd0f75ca0912b4849777677f587;
GyoiThon can identify the CMS Joomla!.
This is because GyoiThon learns features of Joomla! such as "Cookie name (f00e6 ... 9831e) " and "Cookie value (0eba9 ... 7f587). In our survey, Joomla! uses 32 lower case letters as the Cookie name and Cookie value in many cases.

Training data (One example)
Joomla! (CMS)
Set-Cookie: ([a-z0-9]{32})=[a-z0-9]{26,32};
Set-Cookie: [a-z0-9]{32}=([a-z0-9]{26,32});
...snip...
HeartCore (Japanese famous CMS)
Set-Cookie:.*=([A-Z0-9]{32});.*
<meta name=["'](author)["'] content=["']{2}.*
...snip...
Apache (Web server software)
Etag:.*".*-[0-9a-z]{3,4}-[0-9a-z]{13}")[\r\n]
...snip...
2. Based on String matching.
Of course, GyoiThon can identify software by string matching also used in traditional penetration test tools. Examples are shown below.

Example.3
<script src="/core/misc/drupal.js?v=8.3.1"></script>
GyoiThon can identify the CMS Drupal.
It is very easy.

Step 3. Exploit using Metasploit.
GyoiThon executes exploit corresponding to the identified software using Metasploit and it checks whether the software is affected by the vulnerability.

Link with Metasploit

Running example
  • exploit/multi/http/struts_code_exec_exception_delegator, target: 1, payload: linux/x86/shell/reverse_nonx_tcp, result: failure
  • exploit/multi/http/struts_code_exec_exception_delegator, target: 1, payload: linux/x86/shell/reverse_tcp, result: failure
  • exploit/multi/http/struts_code_exec_exception_delegator, target: 1, payload: linux/x86/shell/reverse_tcp_uuid, result: failure
  • exploit/multi/http/struts_code_exec_exception_delegator, target: 1, payload: linux/x86/shell_bind_ipv6_tcp, result: failure
  • exploit/multi/http/struts_code_exec_exception_delegator, target: 1, payload: linux/x86/shell_bind_tcp, result: failure


...snip...

  • exploit/linux/http/apache_continuum_cmd_exec, target: 0, payload: generic/custom, result: failure
  • exploit/linux/http/apache_continuum_cmd_exec, target: 0, payload: generic/debug_trap, result: failure
  • exploit/linux/http/apache_continuum_cmd_exec, target: 0, payload: generic/shell_bind_tcp, result: failure
  • exploit/linux/http/apache_continuum_cmd_exec, target: 0, payload: generic/shell_reverse_tcp, result: failure
  • exploit/linux/http/apache_continuum_cmd_exec, target: 0, payload: generic/tight_loop, result: bingo!!

Step 4. Generate scan report.
GyoiThon generates a report that summarizes vulnerabilities.
Report's style is html.

sample gyoithon_report
Demonstration movie.
IMAGE ALT TEXT HERE

https://www.youtube.com/watch?v=jmi43eZOE9w

Usage
Step.0 Initialize Metasploit DB
Firstly, you initialize metasploit db (postgreSQL) using msfdb command.

root@kali:~# msfdb init
Step.1 Launch Metasploit Framework
You launch Metasploit on the remote server that installed Metasploit Framework such as Kali Linux.

root@kali:~# msfconsole
______________________________________________________________________________
|                                                                              |
|                   METASPLOIT CYBER MISSILE COMMAND V4                        |
|______________________________________________________________________________|
     \\                                  /                      /
      \\     .                          /                      /            x
       \\                              /                      /
        \\                            /          +           /
         \\            +             /                      /
          *                        /                      /
                                  /      .               /
   X                             /                      /            X
                                /                     ###
                               /                     # % #
                              /                       ###
                     .       /
    .                       /      .            *           .
                           /
                          *
                 +                       *

                                      ^
####      __     __     __          #######         __     __     __        ####
####    /    \\ /    \\ /    \\      ###########     /    \\ /    \\ /    \\      ####
################################################################################
################################################################################
# WAVE 4 ######## SCORE 31337 ################################## HIGH FFFFFFFF #
################################################################################
                                                          https://metasploit.com


      =[ metasploit v4.16.15-dev                         ]
+ -- --=[ 1699 exploits - 968 auxiliary - 299 post        ]
+ -- --=[ 503 payloads - 40 encoders - 10 nops            ]
+ -- --=[ Free Metasploit Pro trial: http://r-7.co/trymsp ]

msf >
Step.2 Launch RPC Server
You launch RPC Server of Metasploit following.

msf> load msgrpc ServerHost=192.168.220.144 ServerPort=55553 User=test Pass=test1234
  • MSGRPC Service: 192.168.220.144:55553
  • MSGRPC Username: test
  • MSGRPC Password: test1234
  • Successfully loaded plugin: msgrpc

msgrpc options   description
ServerHost   IP address of your server that launched Metasploit. Above example is 192.168.220.144.
ServerPort   Any port number of your server that launched Metasploit. Above example is 55553.
User   Any user name using authentication (default => msf). Above example is test.
Pass   Any password using authentication (default => random string). Above example is test1234.
Step.3 Edit config file.
You have to change following value in config.ini

...snip...

[GyoiExploit]
server_host      : 192.168.220.144
server_port      : 55553
msgrpc_user      : test
msgrpc_pass      : test1234
timeout          : 10
LHOST            : 192.168.220.144
LPORT            : 4444

...snip...
config   description
server_host   IP address of your server that launched Metasploit. Your setting value ServerHost in Step2.
server_port   Any port number of your server that launched Metasploit. Your setting value ServerPort in Step2.
msgrpc_user   Metasploit's user name using authentication. Your setting value User in Step2.
msgrpc_pass   Metasploit's password using authentication. Your setting value Pass in Step2.
LHOST   IP address of your server that launched Metasploit. Your setting value ServerHost in Step2.
Step.4 Edit target file.
GyoiThon accesses target server using host.txt.
So, you have to edit host.txt before executing GyoiThon.

sample of host.txt
target server => 192.168.220.148
target port => 80
target path => /oscommerce/catalog/
192.168.220.148 80 /oscommerce/catalog/
You have to separate IP address, port number and target path using single space.

Note
Current gyoithon.py is provisional version that without crawling function. We'll upgrade gyoithon.py by April 9. Then, target path will be unnecessary.
Step.5 Run GyoiThon
You execute GyoiThon following command.

local@client:~$ python gyoithon.py
Step.6 Check scan report
Please check scan report using any web browser.

local@client:~$ firefox "gyoithon root path"/classifier4gyoithon/report/gyoithon_report.html
Tips
1. How to add string matching patterns.
signatures path includes four files corresponding to each product categories.

local@client:~$ ls "gyoithon root path"/signatures/
signature_cms.txt
signature_framework.txt
signature_os.txt
signature_web.txt
signature_cms.txt
It includes string matching patterns of CMS.
signature_framework.txt
It includes string matching patterns of FrameWork.
signature_os.txt
It includes string matching patterns of Operating System.
signature_web.txt
It includes string matching patterns of Web server software.
If you want to add new string matching patterns, you add new string matching patterns at last line in each file.

ex) How to add new string matching pattern of CMS at signature_cms.txt.

tikiwiki@(Powered by TikiWiki)
wordpress@<.*=(.*/wp-).*/.*>
wordpress@(<meta name="generator" content="WordPress).*>

...snip...

typo@.*(href="fileadmin/templates/).*>
typo@(<meta name="generator" content="TYPO3 CMS).*>
"new product name"@"regex pattern"
[EOF]
Note
Above new product name must be a name that Metasploit can identify. And you have to separate new product name and regex pattern using @.
2. How to add learning data.
signatures path includes four files corresponding to each product categories.

local@client:~$ ls "gyoithon root path"/classifier4gyoithon/train_data/
train_cms_in.txt
train_framework_in.txt
train_os_in.txt
train_web_in.txt
train_cms_in.txt
It includes learning data of CMS.
train_framework_in.txt
It includes learning data of FrameWork.
train_os_in.txt
It includes learning data of Operating System.
train_web_in.txt
It includes learning data of Web server software.
If you want to add new learning data, you add learning data at last line in each file.

ex) How to add new learning data of CMS at train_cms_in.txt.

joomla@(Set-Cookie: [a-z0-9]{32}=.*);
joomla@(Set-Cookie: .*=[a-z0-9]{26,32});

...snip...

xoops@(xoops\.js)
xoops@(xoops\.css)
"new product name"@"regex pattern"
[EOF]
Note
Above new product name must be a name that Metasploit can identify. And you have to separate new product name and regex pattern using @.
And you have to delete trained data (*.pkl).

local@client:~$ ls "gyoithon root path"/classifier4gyoithon/trained_data/
train_cms_out.pkl
train_framework_out.pkl
train_web_out.pkl
local@client:~$ rm "gyoithon root path"/classifier4gyoithon/trained_data/*.pkl
3. How to change "Exploit module's option".
When GyoiThon exploits, it uses default value of Exploit module options.
If you want to change option values, please input any value to "user_specify" in exploit_tree.json as following.


"unix/webapp/joomla_media_upload_exec": {
    "targets": {
        "0": [
            "generic/custom",
            "generic/shell_bind_tcp",
            "generic/shell_reverse_tcp",

...snip...

        "TARGETURI": {
            "type": "string",
            "required": true,
            "advanced": false,
            "evasion": false,
            "desc": "The base path to Joomla",
            "default": "/joomla",
            "user_specify": "/my_original_dir/"
        },
Above example is to change value of TARGETURI option in exploit module "exploit/unix/webapp/joomla_media_upload_exec" to "/my_original_dir/" from "/joomla".

Operation check environment
Kali Linux 2017.3 (for Metasploit)
Memory: 8.0GB
Metasploit Framework 4.16.15-dev
ubuntu 16.04 LTS (Host OS)
CPU: Intel(R) Core(TM) i5-5200U 2.20GHz
Memory: 8.0GB
Python 3.6.1(Anaconda3)
docopt 0.6.2
jinja2 2.10
msgpack-python 0.4.8
pandas 0.20.3


Consulente in Informatica dal 1984

Software automazione, progettazione elettronica, computer vision, intelligenza artificiale, IoT, sicurezza informatica, tecnologie di sicurezza militare, SIGINT. 

Facebook:https://www.facebook.com/flaviobernardotti58
Twitter : https://www.twitter.com/Flavio58

Cell:  +39 366 3416556

f.bernardotti@deeplearningitalia.eu

#deeplearning #computervision #embeddedboard #iot #ai

 

Related Topics

  Subject / Started by Replies Last post
0 Replies
204 Views
Last post April 11, 2018, 12:19:32 AM
by Flavio58
0 Replies
1527 Views
Last post May 28, 2018, 07:11:54 PM
by Ruggero Respigo
0 Replies
88 Views
Last post June 01, 2018, 01:00:26 PM
by Flavio58
0 Replies
446 Views
Last post June 02, 2018, 03:49:22 PM
by Flavio58
0 Replies
111 Views
Last post June 18, 2018, 03:48:47 PM
by Flavio58

Sitemap 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326