Splunk App for McAfee Web Gateway

Version: 4.0.11

Date: 20 October 2022

  1. About
  2. Where to install this App
  3. Quick Start
  4. Get Data In
  5. Overview of Sourcetypes and Log Formats
  6. Configure a custom log format (mcafee:webgateway:custom) on MWG
  7. Upgrade from 3.07
  8. Configuration examples
  9. Onboarding checklist
  10. Detailed description of the mcafee:webgateway:custom Log Format
  11. Other logs
  12. Next Steps / Action Plan
  13. FAQ
  14. Dashboard Views
  15. Troubleshooting
  16. Summary of changes
  17. Contributors, Attributions
  18. Copyright
  19. Disclamer
  20. Contact, Support and Feedback
  21. Additional information

About

This Splunk App for McAfee Web Gateway allows rapid insights and operational visibility into McAfee Web Gateway (MWG) and McAfee Web Gateway Cloud Service (WGCS) deployments. It provides field extraction and CIM field mapping using all available types of access logs (default and custom McAfee Web Gateway log, McAfee Web Gateway Cloud Service), facilitates fast incident response and troubleshooting.

In 2022 McAfee Web Gateway (MWG) was renamed to SkyHigh Secure Web Gateway (SWG).

List of abbreviations used in this document:

AbbreviationMeaning
MWGMcAfee Web Gateway
WGCSMcAfee Web Gateway Cloud Service
UFSplunk Universal Forwarder

Product Compatibility:

ProductVersion(s)
Splunk6.6+, 7.x, 8.x, 9.x
MWG7.6+, 8.x, 9.x, 10.x, 11.x
WGCSAPI v5

Currently there are 85 different charts and tables grouped in 22 views

  Applications
      Applications by Hits
      Applications by Volume
      Top Blocked Applications by Hits
      Top Applications by Volume
      Top Applications by Hits
      Top Application Statistics
  Audit
      Failed Logins
      Activity by Action
      Activity by Source_Type
      Activity by User
      User Activity by Appliance
  Authentication
      Top IP by Failed Auth
      Top User-Agents by Failed Auth
      Top Destination Hosts by Failed Auth
      Top User-Agents + IPs by Failed Auth
      Top User-Agents + DestHost by Failed Auth
      Top IPs + DestHost by Failed Auth
      Top IPs + User-Agent + DestHost by Failed Auth
      Multiple Logins from diff IPs
      Multiple Usernames coming from a single IP
      Authentication Method Statistics
  Connections
      Long running transactions
  DNS
      Timechart DNS resolution time
      Timechart DNS resolution time distribution (including Cached)
      Timechart DNS resolution time distribution (excluding Cached)
      DNS distribution (1ms - 200ms)
      DNS distribution (all)
  Errors
      Error Analysis
  HTTP
      Timechart HTTP Method
      HTTP Method Statistics
      HTTP Request Headers Statistics
      HTTP Response Headers Statistics
  Easy Search
      Status Code Overview
      Web Usage by URL Category
      Web Usage by URL Category Area Graph
      Top User-Agents
      Users + IPs
      IP Addresses by Hits Graph
      Top Hosts by Hits
      Top Blocked Domains by Hits
      Top Rules by Hits
      Events
  Malware
      Malware
      Top Users by blocked Malware
  Media Types
      Media Types
      Top Media Types by Volume
      Top Media Types by Hits
      EXE Uploads/Downloads
      Macro Uploads/Downloads
      EXE and Macro Uploads/Downloads with Magic Bytes Mismatch
      Encrypted Files
  Network
      Top unreachable Servers
  Performance
      Connect to Server Latency
      Total Transaction Duration distribution
      Client-Side Latency
      DNS resolution Latency distribution
      Time in Externals Distribution
  Protocols
      Protocols by Hits
      Protocols by Hits (Percent)
      Protocols by Volume
      Protocols by Volume (Percent)
  Potential Risks
      Top SRC with high Ratio of High Risk Requests
      Unusual Ports
      Requests to IP Addresses
      CONNECT Requests to IP Addresses
      Very long URLs
      Very large request and response Headers
      Non-resolvable Domains, potential DGA (Domain Generation Algorithm)
  Rules
      Top Rules
      Block Rules Overview
      Top Block Rules
      Rule Complexity/Performance
      Slowest Rule Execution
      Time in Rule Engine Distribution
      Time in Rule Engine over Time
  Security Posture
      Content Scan is possible Ratio
  SSL
      SSL Versions by Hits (Server)
      SSL Versions by Hits (Client)
      SSL Ciphers by Hits (Server)
      SSL Ciphers by Hits (Client)
      SSL KeyExchangeBits by Hits (Server)
      SSL KeyExchangeBits by Hits (Client)
      SSL Ciphers (Server)
      SSL Versions (Server)
      Client Certificate Requested
      SSL-related blocks
      Expired Certificate
      Certificate Issuers
  Summary
      Requests / Block Ratio
      Traffic Overview
  Traffic
      Top Inbound Traffic by Source
      Top Inbound Traffic by Destination
      Top Outbound Traffic by Source
      Top Outbound Traffic by Destination
  Uploads
      Uploads
  URL Filter
      URL Categories
      Blocked by URL Filter or by Web Reputation
      Top URL Categories by Volume
      Top URL Categories by Hits
      Geolocation Stats
      High Risk Destinations
      Not categorized Domains - Chart
      Top not categorized Domains - Table
  User-Agents
      User-Agent Statistics

Where to install this App
InstanceApp for McAfee Web GatewayAdd-on for McAfee Web Gateway
Standalone (all-in-one) Splunk+-
Search Head+-
Indexer-+
Syslog/Log Server with Universal Forwarder-+

Quick Start

Install Splunk directly on MWG and configure it to monitor local log folder:

  1. Configure a custom log format (mcafee:webgateway:custom) on MWG
  2. Install Splunk on the same MWG
  3. Install Splunk App for McAfee Web Gateway on Splunk
  4. CLI: Allow Splunk to read splunk.log: setfacl -m u:splunk:rx /opt/mwg/log/user-defined-logs
  5. Configure a local file monitor
Step-by-step walkthrough: https://youtu.be/96oRco3MTu0

Configure MWG to send logs via TCP to Splunk

  1. Configure a custom log format (mcafee:webgateway:custom) on MWG
  2. Configure MWG to send events via UDP/TCP
  3. Install Splunk App for McAfee Web Gateway on Splunk
  4. Configure Splunk network input to accept logs from MWG
Step-by-step walkthrough: https://youtu.be/vYy6ddpGkNw

Get Data In

MWG can write logs to hard disk or/and send them via Syslog. Splunk can read log files locally, get them via network input (Syslog or raw UDP/TCP steam) or get them from a UF that is installed on a log server or on MWG itself. All these methods combined produce many possible ways to get MWG logs into Splunk:

Method / Link to configuration exampleDescriptionReal time
Local file monitorSplunk is installed directly on MWG and monitors the log file folderYes, up to 30 sec delay
Local UDP/TCP inputSplunk is installed directly on the MWG and gets log files sent using Syslogyes
Syslog UDP/TCPMWG sends logs via UDP/TCP to syslog collector or directly to Splunkyes
Syslog TCP+TLSMWG sends logs via TCP, encrypted with TLS, to syslog collector or directly to Splunkyes
UFInstall UF on MWG to monitor log file folderyes, up to 30 sec delay
Log pushing from MWG to a log serverUse pushing (FTP/FTPS/SCP/SFTP/HTTP/HTTPS) from MWG to a log serverno
Log pulling from MWGPulling logs from MWG via API, scp or rsyncno
Log pulling from WGCSPulling logs via WGCS APIno

Installing UF directly on MWG and configure UF to forward events to Splunk indexer is a recommended and most reliable method!

Further consideration:

Overview of Sourcetypes and Log Formats

 There are several possible log formats that can be used. Compare your logs with example below to find out the current format.

On-premise Web Gateway

Log FormatSourcetype# of MWG fields# of CIM fieldsAverage log line length (HTTPS Scanner enabled)Comment/Example
Default Access Logmcafee:webgateway:default1417~700 BytesDefault log format with a fixed structure, provides only minimal subset of fields. Use it only if no MWG modification is possible.

[26/Feb/2021:14:40:23 +0100] "" 192.168.2.n 200 "GET https://example.com/test&adk=1473563476 HTTP/2.0" "Web Ads" "Minimal Risk" "image/gif" 286 538 "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0) Gecko/20100101 Firefox/86.0" "" "0" "Google"
Legacy Log for the Splunk App v.3.0.7MWGaccess32627~650 BytesCustomized log format with a fixed structure, provides more fields than the default log, including some timings and transferred bytes. Wasteful information like User-Agent string is shortened. Consider it obsolete.

[26/Feb/2021:14:40:23 +0100]status="200/0" srcip="192.168.2.n" user="" profile="-" dstip="-" dhost="example.com" urlp="443" proto="HTTPS/https" mtd="GET" urlc="Web Ads" rep="0" mt="image/gif" mlwr="-" app="Google" bytes="538/539/289/286" ua="FF86.0-10.0" lat="0/0/59/434" rule="Last Rule" url="https://example.com/test&adk=1473563476"
Custom Log (recommended)mcafee:webgateway:custom50-10050-100~600-1800 BytesNew custom modular log format (described in details below), logs fields can be added/removed as needed, provides full CIM coverage and deep insights for analytics and rapid troubleshooting. Despite the significantly larger amount of provided information, the size of the log has changed insignificantly. This new format provides up to 3x higher information density than the default log format.

2021-02-26 14:40:23 +0100 204 allowed 192.168.2.n https GET example.com 443 775/58 88/1 up="/test" ua="FF86-10.0" a="Google" c="wa" dip=142.250.185.nn kex=112/112 cntx sccc=1302/1302 sslp=1.3/1.3 sslicn="GTS CA 1O1,GlobalSign" sslcn="example.com" crtdays=-66 ctmt0 rul="L" rn=13/44 srcp=63298 conrt=0 b=744/239 psrcip=192.168.2.n psrcp=20010 piv=2.0/2.0 r=0 t=0/0/86/87/56/56/3/4/28

Web Gateway Cloud Service (WGCS)

WGCS log format provides a subset of required fields, there are several API versions:

Log FormatSourcetype# of MWG fields# of CIM fieldsAverage log line length (HTTPS Scanner enabled)Comment/Example
WGCS API version 5mcafee:webgateway:wgcs_v52828~300-400 Bytes "user_id","username","source_ip","http_action","server_to_client_bytes","client_to_server_bytes","requested_host","requested_path","result","virus","request_timestamp_epoch","request_timestamp","uri_scheme","category","media_type","application_type","reputation","last_rule","http_status_code","client_ip","location","block_reason","user_agent_product","user_agent_version","user_agent_comment","process_name","destination_ip","destination_port" "-1","142.250.185.nn","142.250.185.nn","GET","206","1040","example.com","/test","OBSERVED","","1626329868","2021-07-15 06:17:48","https","Business, Software/Hardware","application/x-empty","","Minimal Risk","Internal Request handled","200","8.65.16.n","","","Other","","","","78.47.250.n","443"
WGCS API version 6mcafee:webgateway:wgcs_v6 (not supported yet)2828~300-400 BytesNo new fields are introduced. All fields from versions 1 – 5 are downloaded. Starting with API version 6, an error message is sent with the response to a download request that has timed out.
WGCS API version 7mcafee:webgateway:wgcs_v7 (not supported yet)2828~300-450 BytesAll fields from versions 1 – 6 are downloaded, plus these fields: pop_country_code referer ssl_scanned av_scanned_up av_scanned_down rbi
WGCS API version 8mcafee:webgateway:wgcs_v8 (not supported yet)2828~300-500 Bytes All fields from versions 1 – 7 are downloaded, plus these fields: dlp client_system_name filename pop_egress_ip pop_ingress_ip proxy_port

Configure a custom log format (mcafee:webgateway:custom) on MWG

  1. Extract the file Splunk_Log_XXXXXX.xml (where XXXXXX is the version) from the MWG folder of the application package.

  2. Import Splunk_Log_XXXXXX.xml file in MWG into the Default Log Handler: Policies > Rule Sets > Log Handler, right click on "Default" and select Add > Rule Set from Library



  3. In the new window that appears, click on the "Import from file" button, then choose the xml file and click OK.


  4. click "Auto-Solve Conflicts..." > select "Solve by referring to existing objects" and click OK to import the RuleSet.


  5. If some of the imported RuleSets/Rules marked red - that means some properties like Header.Request.GetAll (available on MWG 10.x+) are not available in the current MWG version. Just delete such rules or upgrade MWG to the latest 10.x+ version. If a TLS RuleSet shown red, it needs to be modified as described below in the Troubleshooting section.

  6. The Log configuration has a modular structure, you can choose to send just a preconfigured minimal set of fields or select any subset from available fields. The log ruleset contains several parts (see numbering on the next screenshot):

    1. Required rulesets for CIM conformed logging.
    2. Web Data Model ruleset where a log line from the previously prepared fields are built.
    3. Additional rulesets where other fields are added as needed.
    4. DEBUG ruleset helps to verify that the log lines built correctly.
    5. Write Splunk.log - final log line modifications, performance monitoring of the Splunk ruleset itself and writing the Splunk log to the hard disk.
    6. Send via Syslog.
    7. RuleSet Library - optional templates that can to be copied into appropriate Policy Rule Sets (Opener, Media Type Filter etc.) to be able to get information that is usually not available in the logging cycle.

    Here are most important modifications that you can do in additional Rulesets (block of RuleSets #3 on the previous screenshot).

    RulesetPossible modifications
    SplunkDomains not to log - some domains can be excluded from logging completely.
    Set Timestampchoose the right timestamp. The ISO format with a time zone is selected by default. Other options are ToGMT, ISO8601, unix epoch and ToWebReporter formats. If you change the timestamp format on MWG then you have to adjust the TIME_FORMAT setting in local/props.conf on Splunk Indexer.
    Client IPConnection.IP property is used by default. Deselect it and select Client.IP if you have downstream proxies or loadbalancer between the client and MWG.
    URL Categoriesadd internal domains to "internal Domains" list to avoid them to being shown as "uncategorized"
    Headerson MWG older than version 10.x some rules will be marked in red if they are not compatible - delete them or upgrade MWG to the newest 10.x version or later.
    TLSdisable this ruleset if HTTPS Scanner is not enabled
    -To get the correct Rule statistics you must create one last ruleset with a rule named "Last Rule" which is applied to all cycles (Request, Response, Embedded).
    RuleSet LibraryOpener, Hashes/Body, Malware, Media Type, Uploads - to get some of the required information, additional rules need to be placed in the corresponding Policy Rule Sets. If you skip this step, some tables and graphs will be empty. Watch a YouTube video on the Splunkbase for step by step instructions.


    Create a "Last Rule Set" with an empty "Last Rule" as a most bottom rule in the Rule Sets Tree:

    Copy Rules to Certification Verification Rule Set to be able to log information about certification parameters:

Upgrade from 3.07
  1. Create a backup of MWG config, export MWG Log Rules, backup your current app.
  2. Check if there are any custom changes in the old MWG Log Rules or in the app.
  3. Check which sourcetype is currently used - MWGaccess3 or "default". MWGaccess3 works with new version without any changes, the "default" is named "mcafee:webgateway:default".
  4. Upgrade an App via GUI or CLI.
  5. Follow the installation instructions for version 4.x.x.
  6. Modify "index_and_sourcetype" macro to include an index and a sourcetype (i.e. 'index=proxy AND sourcetype="mcafee:webgateway:custom"')
  7. It is recommended to switch from default or MWGaccess3 to new mcafee:webgateway:custom log format.

Configuration examples