This Splunk App for McAfee Web Gateway allows rapid insights and operational visibility into McAfee Web Gateway (MWG) and McAfee Web Gateway Cloud Service (WGCS) deployments. It provides field extraction and CIM field mapping using all available types of access logs (default and custom McAfee Web Gateway log, McAfee Web Gateway Cloud Service), facilitates fast incident response and troubleshooting.
MWG can write logs to hard disk or/and send them via Syslog. Splunk can read log files locally, get them via network input (Syslog or raw UDP/TCP steam) or get them from a UF that is installed on a log server or on MWG itself. All these methods combined produce many possible ways to get MWG logs into Splunk:
| Log Format | Sourcetype | # of MWG fields | # of CIM fields | Average log line length (HTTPS Scanner enabled) | Comment/Example |
|---|
| Default Access Log | mcafee:webgateway:default | 14 | 17 | ~700 Bytes | Default log format with a fixed structure, provides only minimal subset of fields. Use it only if no MWG modification is possible.
[26/Feb/2021:14:40:23 +0100] "" 192.168.2.n 200 "GET https://example.com/test&adk=1473563476 HTTP/2.0" "Web Ads" "Minimal Risk" "image/gif" 286 538 "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0) Gecko/20100101 Firefox/86.0" "" "0" "Google"
|
| Legacy Log for the Splunk App v.3.0.7 | MWGaccess3 | 26 | 27 | ~650 Bytes | Customized log format with a fixed structure, provides more fields than the default log, including some timings and transferred bytes. Wasteful information like User-Agent string is shortened. Consider it obsolete.
[26/Feb/2021:14:40:23 +0100]status="200/0" srcip="192.168.2.n" user="" profile="-" dstip="-" dhost="example.com" urlp="443" proto="HTTPS/https" mtd="GET" urlc="Web Ads" rep="0" mt="image/gif" mlwr="-" app="Google" bytes="538/539/289/286" ua="FF86.0-10.0" lat="0/0/59/434" rule="Last Rule" url="https://example.com/test&adk=1473563476" |
| Custom Log (recommended) | mcafee:webgateway:custom | 50-100 | 50-100 | ~600-1800 Bytes | New custom modular log format (described in details below), logs fields can be added/removed as needed, provides full CIM coverage and deep insights for analytics and rapid troubleshooting. Despite the significantly larger amount of provided information, the size of the log has changed insignificantly. This new format provides up to 3x higher information density than the default log format.
2021-02-26 14:40:23 +0100 204 allowed 192.168.2.n https GET example.com 443 775/58 88/1 up="/test" ua="FF86-10.0" a="Google" c="wa" dip=142.250.185.nn kex=112/112 cntx sccc=1302/1302 sslp=1.3/1.3 sslicn="GTS CA 1O1,GlobalSign" sslcn="example.com" crtdays=-66 ctmt0 rul="L" rn=13/44 srcp=63298 conrt=0 b=744/239 psrcip=192.168.2.n psrcp=20010 piv=2.0/2.0 r=0 t=0/0/86/87/56/56/3/4/28
|
| WGCS version 5 | mcafee:webgateway:wgcs_v5 | 28 | 28 | ~300-400 Bytes | WGCS log format provides a subset of required fields
"user_id","username","source_ip","http_action","server_to_client_bytes","client_to_server_bytes","requested_host","requested_path","result","virus","request_timestamp_epoch","request_timestamp","uri_scheme","category","media_type","application_type","reputation","last_rule","http_status_code","client_ip","location","block_reason","user_agent_product","user_agent_version","user_agent_comment","process_name","destination_ip","destination_port"
"-1","142.250.185.nn","142.250.185.nn","GET","206","1040","example.com","/test","OBSERVED","","1626329868","2021-07-15 06:17:48","https","Business, Software/Hardware","application/x-empty","","Minimal Risk","Internal Request handled","200","8.65.16.n","","","Other","","","","78.47.250.n","443"
|
McAfee Web Gateway based on RedHat/CentOS 7 and inherits some settigs that rate-limit syslog. Read https://www.ibm.com/support/pages/how-disable-rsyslog-rate-limiting and https://access.redhat.com/solutions/1417483 to modify or disable rate-limiting in /etc/rsyslog.conf (using MWG UI) and /etc/systemd/journal.conf .
rsyslog.conf:
Why a new log format? Neither the default nor the previously used MWGaccess3 log formats provide enough information for SIEM to be useful. For example these formats provide very limited information about download/upload risky files. Many SIEM correlation rules will not work properly if a transferred file was embedded as a part of a composite object (zip, iso, docx, etc.) or has different/faked media-type header or extension.
Instead of logging a URL as-is, MWG splits the URL into usable parts which will be put together on Splunk's end.
By default, the query string is not logged. You can enable it in the Web Data Model ruleset if needed.
| MWG field | CIM field | Comment |
|---|
| Timestamp | - |
| Property | Example | TIME_FORMAT / Comment |
| DateTime.ToISOString | 2010-03-22 11:45:12) | %Y-%m-%d %H:%M:%S |
| DateTime.ToISOString with Milliseconds | 2010-03-22 11:45:12.123 | %Y-%m-%d %H:%M:%S.%3N |
| DateTime.ToISOString with Milliseconds and timezone | 2010-03-22 11:45:12.123 -0600 | %Y-%m-%d %H:%M:%S.%3N %z |
| DateTime.ToISOString and timezone | 2010-03-22 11:45:12 -0600 | %Y-%m-%d %H:%M:%S %z |
| DateTime.ToGMTString | Mon, 22 March 2010 11:45:36 GMT | %a, %d %B %Y %H:%M:%S %Z |
| DateTime.ToISO8601String | 2016-01-26T11:45:36.695Z | this time format can produce unexpected output, don't use it |
| DateTime.ToNumber | Unix epoch time - 1512915182 | %s |
| DateTime.ToWebReporterString | [29/Oct/2010:14:28:15 +0000] | \[%d/%b/%Y:%H:%M:%S %z\] |
|
| Connection.IP / Client.IP | src | Client.IP takes the value of X-Forwarded-For header |
| Authentication.UserName | user | |
Message.TemplateName, Block.ID,
Response.StatusCode, Protocol.FailureDescription,
BytesFromServer, Command.Name,
Action.Names | action | The action taken by the proxy: allowed, blocked, error or auth. Various MWG properties are used to calculate correct action field. |
| URL | url | Don't enable it, Splunk build URL based on uri components |
| URL.Categories | category | MWG will try to categorize URL retroactively even if URL Filter was skipped in the Policy Rule Sets. Add your internal domains to "internal Domains" list to avoid them be marked as "uncategorized" |
Header.Response.Get(Content-Type)
MediaType.FromHeader | http_content_type | The content-type of the requested HTTP resource as reported by the web server (can be wrong, faked or missing) |
| Header.Request.Get(User-Agent) | http_user_agent | A short string (FF68-10.0 for Firefox 68 on Windows 10) |
| LastSentLastReceivedServer | response_time | FSFRS-LSFRS+LSLRS is used to calculate response_time that includes sending time |
| Header.Request.Exists(Referer) | http_referrer | The HTTP referrer used in the request. The W3C specification and many implementations misspell this as http_referer. Use a FIELDALIAS to handle both key names. This field is disabled by default. |
| URL.Domain of Header.Request.Exists(Referer) | http_referer_domain | The domain name contained within the HTTP referrer used in the request. Disabled by default. |
| Response.StatusCode | status | The HTTP response code indicating the status of the proxy request. MWG doesn't distinguish between status sent by web server and status set by proxy, so this value can be misleading. Use action field to see what the proxy action was. |
| URL.Protocol | - | http/https/ftp etc. Used to re-build url |
| Command.Name | http_method | GET/POST/PUT/OPTIONS etc |
| URL.Host | dest | The host of the requested resource |
| URL.Port | dest_port | The port of the requested resource |
| BytesToServer | bytes_out | The number of outbound bytes transferred |
| BytesFromServer | bytes_in | The number of inbound bytes transferred |
| TimeInTransaction | duration | The time taken by the proxy event, in milliseconds |
| URL.Path | uri_path | The path of the resource served by the webserver or proxy |
| URL.ParametersString | uri_query | Not enabled by default. You can enable it for all requests or selectively |
| Application.Name | App | The application detected or hosted by the server/site such as WordPress, Splunk, or Facebook |
| Cache.Status eq TCP_HIT | cached | Indicates whether the event data is cached or not. Not enabled by default. |
| Header.Get(Cookie) | cookie | The cookie file recorded in the event. Not enabled by default. |
| URL.Destination.IP | dest_ip | It is important to record the destination IP at the moment of the request. A hostname can be resolved to several IPs (think "moving target" CDN) so a DNS resolution a second later can lead to wrong result. Be aware that MWG can be unable to do DNS resolution by itself and it can be a different IP after all if MWG is behind upstream proxies. |
| URL.Domain | url_domain | The domain name contained within the URL of the requested HTTP resource. It is extracted from hostname based on Public Suffix List |
| Header.Request.GetAll | - | Returns a concatenated string of all the original request headers (separated by \r\n) as received from client. |
| Header.Response.GetAll | - | Returns a concatenated string of all the original response headers (separated by \r\n) as received from server. |
| Header.Request.Get(Via) | - | Via header in request |
| Header.Response.Get(Via) | - | Via header in response |
| Header.Response.Get(Location) | - | Location header in response |
| Client.KeyExchangeBits | - | Normalized strength (symmetric) of the weakest link during the key exchange. Helps to detect outdated client software |
| Server.KeyExchangeBits | - | Normalized strength (symmetric) of the weakest link during the key exchange. Helps to detect outdated servers which required special handling |
| Server.Handshake.CertificateIsRequested | - | True, if the web server requests a client certificate (during the initial SSL handshake) [*] |
| ClientContext.IsApplied | - | A clue if HTTPS Scanner is enabled for this request |
| Server.Cipher | - | Description of cipher/algorithms between proxy and server (e.g. ECDHE-RSA-AES256-GCM-SHA384) |
| Client.Cipher | - | Description of cipher/algorithms between client and proxy (e.g. ECDHE-RSA-AES256-GCM-SHA384) |
| SSL.Server.Protocol | - | SSL/TLS protocol used between proxy and server (e.g. TLSv1.2 TLSv1.1 TLSv1.0 SSLv3.0 unknown). |
| SSL.Client.Protocol | - | SSL/TLS protocol used between client and proxy (e.g. TLSv1.2 TLSv1.1 TLSv1.0 SSLv3.0 unknown) |
| SSL.TransparentCNHandling | - | true for ssl connections where the CN is not known until the server handshake is done |
| Server.CertificateChain.Issuer.CNs | ssl_issuer_common_name | The issuer common names of the certificate chain (bottom-up including the self-signed root CA, empty without certificate verification) [*] |
| SSL.Server.Certificate.CN | ssl_subject_common_name | The common name of the server certificate [*] |
| Server.Certificate.SHA2-256Digest | ssl_hash | The hex-encoded sha2-256 digest of the server certificate [*] |
| Server.Certificate.AlternativeCNs | - | This list stores all alternative subject names stored in the server certificate's extensions section [*] |
| Server.Certificate.DaysExpired | ssl_end_time | Stores how many days the server certificate is expired. Negative values mean that it is still valid [*] |
| DNS.Lookup(URL.Host) | - | List of IP addresses of URL.Host if there are more than one. |
| DNS.Lookup.Reverse(URL.Destination.IP) | - | List of hostnames for the destination IP. Very often it does not equal the requested hostname |
| Body.NumberOfChildren | - | Number of embedded objects for archive or document [*] |
| Body.NestedArchiveLevel | - | The current archive level, used to calculate the max level of the embedded object [*] |
| IsCompositeObject | - | True, if current file is composite (archive or office document) [*] |
| Body.IsEncryptedObject | - | True, if current object is encrypted |
| Antimalware.Proactive.Probability | - | Malware probability value |
| Antimalware.Infected | used for:
file_name
file_hash | True, if virus was found, false otherwise |
| Antimalware.VirusNames | signature | List of names of found viruses |
| Application.Reputation | - | reputation of the application |
| Authentication.Method | authentication_method | authentication method (NTLM, Kerberos, etc.) |
| Authentication.Realm | - | authentication realm (i.e. AD directory name) |
| Authentication.UserGroups | - | User Groups, can be filtered with "Authentication UserGroups to log" list |
| Authentication.FailureReason.Message | signature (?) | Human readable authentication failure reason description |
| Authentication.Failed | action (in Authentication DM) | It is true if credentials were provided but the authentication has failed |
| Cache.IsCacheable | - | True, if the response is cacheable and web cache is enabled |
| Cache.Status | - | TCP_HIT for a web cache hit, TCP_MISS_RELOAD for a miss, TCP_MISS_VERIFY if the data in the cache was outdated, TCP_MISS_BYPASS for bypass based on I/O load |
| Cache.IsFresh | - | True, if the response is validated or not read from web cache |
| MagicBytesMismatch | - | True, if Mime Type from header doesn't match to detected Mime Type [*] |
| EnsuredTypes | - | List of Mime Types detected by signatures (with high probability of detection) |
| NotEnsuredTypes | - | List of Mime Types detected by signatures (with low probability of detection) |
| IsMediaStream | - | Determine if current transaction is media stream |
| StreamDetector.Probability | - | Probability value for media stream detection |
| StreamDetector.MatchedRule | - | Returns name of matched streaming detection rule |
| Rules.CurrentRule.Name | - | The name of the currently evaluated rule |
| Rules.EvaluatedRules | - | List of all IDs of rules/rule sets, which have been evaluated |
| Rules.FiredRules | - | List of all IDs of rules/rule sets, where the condition was true |
| Proxy.IP | - | Stores the Webgateway IP |
| Proxy.Port | - | Stores the Webgateway port |
| Client.ProcessName | - | Stores the process name that initiated the connection, e.g. provided by MCP |
| Client.SystemInfo | - | Client System Information (provided by MCP) |
| DNS.Lookup.Reverse(client_ip) | src_ip | Hostname of the client |
| Connection.Protocol | - | The protocol that the client uses to communicate with the proxy (HTTP, HTTPS, FTP, IFP, SSL, ICAP, XMPP, TCP or SOCKS) |
| Connection.Port | src_port | Stores the port of the client |
| Connection.RunTime | - | Connection run time (current time minus start time) in seconds |
| BytesFromClient | - | Number of bytes received from the client for this request |
| BytesToClient | - | Number of bytes sent to the client for this request |
| Tunnel.Enabled | - | True, if a HTTP or HTTPS tunnel was enabled - the server response bypassed the response cycle |
| Proxy.Outbound.IP | - | Stores the IP which is used as the Outbound Source IP by Webgateway when connecting to onward server |
| Proxy.Outbound.Port | - | The port which is used as the source port by Webgateway when connecting to onward server |
| ProtocolAndVersion | - | protocol and version of the request/response (HTTP/1.1, HTTP/2.0) |
| Error.ID | - | ID of error |
| Error.Message | - | Name of error |
| URL.Reputation | severity (?) | Returns the web reputation value for the current URL. Range is from -127 to 127, where -127 means 'Minimal Risk' and 127 means 'High Risk'. |
| URL.Geolocation | - | Returns the geolocation of the current URL. The geolocation is the code of the country in which the webserver is located, that hosts the requested resource. The country code is given in ISO 3166 notation. Note: The setting "Disable local GTI database" must be enabled in the URL Filter settings; otherwise this property is not filled. |
| TimeInRuleEngine | - | Milliseconds currently spent in rule engine. If used in log handler, time consumed by the rule engine from start to the end of a transaction |
FirstSentFirstReceivedServer
LastSentLastReceivedServer
FirstReceivedFirstSentClient
LastReceivedLastSentClient
LastSentFirstReceivedServer
| - | Time between first byte sent to server and first byte returned from server in milliseconds etc... |
| HandleConnectToServer | - | Time to connect to a server in milliseconds |
| ResolveHostNameViaDNS | - | Time to resolve a host name via DNS |
| TimeInExternals | - | Milliseconds currently spent waiting for external responses, e.g. from AV scanner, domain controller for NTLM authentication or URL cloud categorization |
This App, documentation and MWG logging ruleset are licensed under Creative Commons BY-ND 3.0
