Category: Networking

Palo Alto SSL Inbound Inspection Issues on Cached Sessions

February 2, 2019 / Dallin Warne / 0 Comments

With the majority of web traffic now served over HTTPS, it is important to decrypt traffic to give visibility to network security monitoring (NSM) tools. The Palo Alto Networks next-generation firewall can decrypt inbound traffic quite effectively.

However, there is one gotcha when enabling this feature on production systems with live traffic. Beware of SSL session caching!

Identifying the SSL decryption transition issue

When I first tested SSL inbound inspection in my Palo Alto firewall, it was in a lab environment and it worked great! The URLs were showing up in the logs, I did not get any SSL errors (decrypt-error,
decrypt-unsupport-param, or decrypt-cert-validation) and it all seemed to work fine. I submitted a change request and was on my marry way.

Then I enabled the feature on a system with a fair amount of active traffic. The results were startling. There was a huge spike in “decrypt-error” logs I couldn’t explain. Enough users were complaining that I ended up reverting the change, puzzled at why what worked flawlessly in the lab didn’t work in production.

I had two leads to what the cause was. The first was Palo Alto’s 8.0 and 8.1 documentation on the “decrypt-error” session reason end saying:

“The session terminated because you configured the firewall to block SSL forward proxy decryption or SSL inbound inspection when firewall resources or the hardware security module (HSM) were unavailable. This session end reason is also displayed when you configured the firewall to block SSL traffic that has SSH errors or that produced any fatal error alert other than those listed for the decrypt-cert-validation and decrypt-unsupport-param end reasons. “
Palo Alto Networks Pan-OS® Administrator’s Guide 8.0 & 8.1

The second clue was the error that appeared in browser windows of some clients who had an active connection to the server at the time. For Google Chrome it was “ERR_SSL_VERSION_INTERFERENCE” and “ERR_SSL_PROTOCOL_ERROR” for Samung’s browser on Android. Firefox and Microsoft Edge gave similar messages.

This led me to believe that clients with cached SSL sessions were attempting to resume their SSL sessions. When that happened the firewall treated the connections as if they were a new connection but would produce a fatal error when it didn’t receive the expected payloads for a new session.

Resolution to resumed SSL sessions

To resolve this, I tried changing the decryption profile settings such as disabling “Unsupported Mode Checks,” but to no avail. On affected clients I tried clearing the SSL cache and even restarting the machines but that did not correct the issue.

Finally, I reduced the SSL session cache timeout setting on the server itself to 60 seconds. When that happened, the issue disappeared!

I wouldn’t recommend shutting SSL session caching entirely as there could be a huge performance impact to the server, but a 60 second timeout leading up to and immediately after enabling the policy should be adequate. If possible, keep it 60 seconds for as long as what the value was previously. I also found decreasing the timeout even after enabling SSL inbound inspection immediately worked.

You can find documentation for SSL session timeout settings for Nginx F5, Apache, and IIS. (At the time of writing I have not tested the parameters on each of these.)

Some forum posts suggest restarting the server(s) will also clear the server’s SSL session cache and force a new negotiation, although I did not test this. This isn’t always feasible if sessions are shared across multiple backend servers, there is a load balancer at play, or engineers are turning on decryption for a large number of servers.

When I contacted Palo Alto about this issue, they told me, “(T)here is feature request (FR ID: 5786) in addition to Jira PAN-80072,” that they did not have a work-around, and that “the only thing to be done now is to wait till further notice.” I am disappointed they do not publish the known issues surrounding decryption and did not have a work-around readily available as this would have saved me hours of troubleshooting, research, and some downtime.

If you found this write-up useful, I ask you let people on Twitter and LinkedIn know. If you liked this post, check out Is Network Security Monitoring Dead in the Age of Encryption?

https://www.maxpixel.net/Cyber-Space-Hacking-Hacker-Cyber-Security-Hack-1944688

Is Network Security Monitoring Dead in the Age of Encryption?

January 7, 2019 / Dallin Warne / 0 Comments

Over the last several years we have seen encryption become more pervasive. Does it now make sense for security teams to invest in network security monitoring solutions?

With the strong push for encryption on everything from websites to hard drives, encryption is becoming a standard practice for most organizations. Reviewing the graph below from Google’s Transparency Report, we see that a majority of web traffic is now HTTPS.

Percentage of pages loaded over HTTPS in Chrome by country according to Google’s Transparency Report

Encryption is permeating other protocols. In September 2018, CloudFlare announced a new protocol that hides the server name during the SSL handshake. RFC 7858 (DNS-over-TLS) and RFC 8484 (DNS-over-HTTPS) both were proposed this decade and are already implemented by some organizations. (Note that DNSSEC doesn’t encrypt dns queries, but ensures they are authenticated.) SMB and SNMP in their third versions also include cryptographic capabilities. Microsoft’s Remote Desktop protocol now incorporates SSL, and SSH has always been encrypted.

It seems that just about all data transmitted over a network is encrypted or is moving in that direction. It is these reasons that some vendors push to move security monitoring to the endpoint where the machine decrypts the information anyways. Is network security monitoring dead in the coming age of encryption? Continue reading

Getting Started with Intrusion Detection System (IDS) Bro: Installation

June 22, 2017 / Dallin Warne / 0 Comments

This is part two of a four part series on getting started with the Bro IDS. See part one on installing the Bro prerequisites. This post is about installing and preparing Bro.

Bro Compilation and Installation

Now that the prerequisites are taken care of, it is time to compile and install Bro. I downloaded Bro 2.5 IDS from bro.org and extracted it. After entering the directory, I ran

./configure --with-pfring=/usr/src/PF_RING --with-pcap=/opt/pf_ring-6.5.0 --prefix=/opt/bro

Below is the output from my ./configure command. It is okay to see failures on some of the lines since some items might not be needed for your system. If you followed this guide, you should see successful messages for GeoIP, gperftools, and PF_RING as highlighted in the output below. (Note, I skipped installing GeoIP so my message will show false below.)

Getting Started with Bro Intrusion Detection System (IDS)

June 6, 2017 / Dallin Warne / 2 Comments

If you have a computer network then you need to ensure an intrusion detection system (IDS) is a part of your cybersecurity strategy. The value of monitoring the traffic on your network far outweighs the cost of a breach. Although most IDS systems are commercial, there are a few open-source IDS solutions.

Snort and Suricata are popular open-source firewall/IDS solutions, but come with a few drawbacks. For a small operation they may work well, but for medium or larger networks they can bring more work and less value. Their key drawback at this time is that Snort/Suricata-capable devices do not communicate with other capable devices on the network, nor are they centrally managed. With cyberattacks becoming more sophisticated, a security-conscious organization needs a better solution.

There is a third major player in the open-source IDS game. The Bro Network Security Monitor, developed originally by higher education, provides both a network protocol analyzer and a security tool. It’s strength is the ability to correlate traffic across multiple Bro devices on a network, and add additional and customizable plugins. In other words, instead of having multiple independent IDS boxes on your network, you could have a single clustered system that correlates information across the network.

Initial Setup of VM-100: Part 2

January 12, 2017 / Dallin Warne / 0 Comments

This is part two of a a two-part series to configure a Palo Alto Networks firewall in a virtual environment. Palo Alto Firewalls are a great asset for any organization as it includes many advanced features to detect and stop bad network traffic.

Configuring the Palo Alto

At this point, the virtual environment is setup (see part 1). I am plugged into my router and can access the ESXi box and Palo Alto from the internal network. Now it’s time to configure the Palo Alto.

Initial Setup of Palo Alto Virtual Firewall: Part 1

January 5, 2017 / Dallin Warne / 0 Comments

The Palo Alto Networks firewall is quite an amazing piece of engineering. This state-of-the-art firewall not only includes traditional firewalling on layer 3 and 4, but it also provides application-level firewall capabilities, user-level policies, DDoS protection, threat prevention, and a whole lot more. In short, it makes a network and security guy like me drool.

Sony Blu-ray Player BDP-S3200: “Internet Connection: Failed”

December 31, 2016 / Dallin Warne / 0 Comments

I recently came across a problem with a Sony Blu-Ray player, specifically the model BDP-S3200 running software version M19.R.0071. When I open the network status screen, it shows valid IPV4 network settings such as a good IP address, a subnet mask, DNS servers, etc. However, there is a glaring message on this screen that reads, “Internet Connection: Failed.” I can use the internet browser just fine to access different websites like Youtube and Google. However, when attempting a software update of the player, it would report that it did not have an internet connection. And to add to the confusion, that same player would perform the software update and see it has a valid internet connection when it is connected on a different network. When I performed the network diagnostics, it reports there is an error connecting to the DHCP server.