logstash解析naxsi日志的问题

目前在用naxsi防火墙，使用elk来做一个日志分析，遇到问题如下：

naxsi作为waf会产生error日志，目前我打开了NAXSI_EXLOG日志选项，因为这个选项可以看到具体的请求内容。

对于同一个请求，naxsi会产生2行或者3行的日志，格式如下：

2017/10/23 17:45:36 [error] 744#0: *19 NAXSI_EXLOG: ip=192.168.141.232&server=192.168.182.141&uri=/sqli-labs/Less-11/&%20or%20'1'='1'%20xxxxxxxxxx, client: 192.168.141.232, server: _, request: "POST /sqli-labs/Less-11/ HTTP/1.1", host: "192.168.182.141:8000", referrer: "1.1.1.1"  2017/10/23 17:45:36 [error] 744#0: *19 NAXSI_EXLOG: ip=192.168.141.232&server=192.168.182.141&uri=/sqli-labs/Less-11/&%20or%20'1'='1'%20xxxxxxxxxx, client: 192.168.141.232, server: _, request: "POST /sqli-labs/Less-11/ HTTP/1.1", host: "192.168.182.141:8000", referrer: "1.1.1.1"  2017/10/23 17:45:36 [error] 744#0: *19 NAXSI_FMT: ip=192.168.141.232&server=192.168.182.141&uri=/sqli-labs/Less-11/&learning=0&vers=0.55.3&total_processed=4&total_blocked=4&block=1&cscore0=$SQL&score0=22&cscore1=$XSS&score1=40&zone0=BODY&id0=1009&var_name0=passwd&zone1=BODY&id1=1013&var_name1=passwd, client: 192.168.141.232, server: _, request: "POST /sqli-labs/Less-11/ HTTP/1.1", host: "192.168.182.141:8000", referrer: "1.1.1.1"  

这是同一个请求产生的结果，因为每次请求都会有一个id值在里面，这个是19:

问题：如何取出NAXSI_EXLOG里面的content，跟NAXSI_FMT里面的结果合并到一起？

我写的logstash和正则如下：

DA1 d{4}/d{2}/d{2}  TM1 d{2}:d{2}:d{2}  LEVEL (w+)  NUM1 d+(?:#0: *)  NUM2 d+  EXLOG NAXSI_EXLOG  FMT NAXSI_FMT  ID1 (d+)  ZONE w+  VAR1  (.*)  CONTENT (.*)  T3 w+  T4 HTTP/1.1", host: "(.*)", referrer: "  HOST (.*)    NAXSI %{DA1:date1}s%{TM1:time}s[%{LEVEL:level}]s%{NUM1:num1}%{NUM2:num2}s(?<logtype>NAXSI_EXLOG):sw+=%{HOST:client_ip}&server=%{HOST:hostname}&uri=%{PROG:filepath}&%{T3:method}s%{HOST:uri}sHTTP/1.1",shost:s"%{HOST:host22}"    NAXSI2 %{DA1:date1}s%{TM1:time}s[%{LEVEL:level}]s%{NUM1:num1}%{NUM2:num2}s(?<logtype>NAXSI_EXLOG):sw+=%{HOST:client_ip}&server=%{HOST:hostname}&uri=%{PROG:filepath}&%{T3:method}s%{HOST:uri}sHTTP/1.1",shost:s"%{HOST:host22}",sreferrer:s"(?<referrer>(.*))    FMT %{DA1:date1}s%{TM1:time}s[%{LEVEL:level}]s%{NUM1:num1}%{NUM2:num2}s(?<logtype>NAXSI_FMT):sip=%{HOST:ip}&server=%{HOST:server}&uri=%{UNIXPATH:uri}&learning=%{HOST:learing}&vers=%{HOST:vers}&total_processed=%{HOST:toal_processed}&total_blocked=%{HOST:blocked}&block=%{HOST:block}&cscore0=%{HOST:attack}&score0=%{HOST:score0}&cscore1=%{HOST:xss}&score1=%{HOST:score}&zone0=%{WORD:args}&id0=%{NUMBER:id}&var_name0=%{HOST:varname},sclient:s%{HOST:ip3},sserver:s(.*)srequest:s"%{T3:method}s%{HOST:uri}sHTTP/1.1",shost:s"%{HOST:host22}  

logstash.conf:

input {   file {         path => "/usr/local/nginx/logs/naxsi.err"         type => "naxsi-error"         start_position => "beginning"     }     }     filter {      if [type] == "naxsi-error" {      grok {          patterns_dir => "/opt/logstash-5.5.1/pattern"          match => [ "message" , "%{NAXSI2}",                 "message" , "%{NAXSI}",                 "message" , "%{FMT}"              ]        }      # aggregate {      #   task_id => "%{num2}"      #       code => "map['sql_duration'] = 0"      #   end_of_task => true      #   }    }  }  output {    if [type] == "naxsi-error" {      elasticsearch {         hosts => ["localhost"]         index => "nxapi"             document_id => "%{num2}"          }       }  }  

原文出处：threathunter -> https://threathunter.org/topic/59eea28fec721b1f1966ed59