by Philip Leslie; ProOnGo Expense
If you are new to the Intuit Partner Platform, one step in the process that might have caught your eye is the security review that happens once you've finished development and internal testing. It's a checkpoint that helps maintain a high quality bar on apps that are destined for Intuit App Center or live customers, and emphasizes the unique security expectations that come with your request to access sensitive QuickBooks data.
If you've already pencilled in some time on your calendar to navigate the security scan, you are no doubt trying to size the effort involved in preparing for the scan, going through the scan, and following up to solve any vulnerabilities discovered during the scan. The purpose of this post is to give you a practical perspective on some of the time, effort, and resources that go into successfully passing the scan. As the developer of ProOnGo Expense, we've been through the scan several times, and so I'll speak from first-hand experience.
Part 1: Preparing for the Security Scan
Some startups that are launching a web app for the first time start out with the "deploy first, test second" strategy; i.e., going straight from testing on localhost to deploying to a public-facing "live server", with no intermediate "staging server". If that describes your current arrangement, it's time to graduate to a deployment strategy that provides some level of isolation on a staging or test server. Why? The security scan will definitely include attacks where the test is clearly trying to peg the CPU (sometimes on a sustained basis, sometimes momentarily):
Equally certain, is that your site will be exposed to SQL injection, XML injection, XSS attacks, and various SSL and TLS-related validation. All of these are things you'll want to experience first on a staging/test server, to harden your security in an isolated environment. Your server logs will suddenly become much more interesting than usual, with tens of thousands of requests like this trivial example:
https://example.com/yourscript.php?page=1%3Ciframe+src%3Djavascript%3Aalert%2868140%29%3E
Although my observation is that the tests are designed to be as non-destructive as possible, the dynamic nature of the tests combines with the sheer volume and variety of test cases, makes it impossible to predict what impact they would have on your system.
At ProOnGo, here are some of the steps we take to prepare for a security scan. Our terminology is AWS-centric, but the concepts apply equally to Azure and other cloud-hosting options:
- Web Server Machine Image & Instance: We create a machine image of our staging server, frozen at a last-known-good changelist, and we spawn a new/isolated/separate instance of the staging server to be the target of the security scan.
- Database Image & Instance: We bring up a database instance that is conforms to the normal database schema used by our live server, but containing none of the live customer data that our production database contains. Then, we dump output containing MRU timestamps of the DB tables, so that later on we can easily determine which tables were written to (and which ones weren't) during the test.
- Session-Handling: We isolate our session-handling infrastructure similarly; ensuring that any SESSION-related attacks are sandboxed from having any production impact (I'm looking in your direction, shared instances of memcached)
- Outbound Network Scrub: We scrub-and-disable certain parts of our source tree that lead to calls to external service providers; i.e., just because we're going through a security scan, does not mean that our partners are prepared to. We disable certain code that triggers outbound mails, eCommerce actions, and other services.
- Log-Clearing: We clear our web server logs, so that we don't have to wonder what portion of the logs were caused by the security scan.
While taking these and other actions, you'll also want to preview the security questionnaire. This is one of the easiest things you can do to save yourself time: if you spot anything in the questionnaire where you know or believe you have a vulnerability, save yourself some time by mitigating up front. The alternative — failing the test, mitigating, potentially re-testing, and all of the coordination that goes along with it — will undoubtedly delay the launch date of your production app.
Part 2: Going through the Security Scan
One of the terrific things that's happened during all of the security scans we've been party to, is that we've been able to request that the bulk of the testing happen within a particular 2-3 day time period. Further, each time we've agreed on a particular time-of-day for the test to start. That's been helpful because we can literally be viewing the tail of the web server log at the very minute that the testing is scheduled to begin, to be sure that it's off to a successful start. Although we do our best to be sure that our instances are ready for testing, there's nothing like seeing the web traffic start to flow to know that the security scan process is underway.
If you watch your web server logs and your diagnostics related to disk I/O, CPU, and memory use, you'll get the added benefit of watching to see whether your various throttling, fail-over, and/or auto-scaling triggers get tripped and whether they respond the way you designed. It can be difficult to get a full grasp of that in a post-mortem analysis; we prefer to watch this happen interactively. Another benefit of watching your logs interactively is that you start to see patterns of the kinds of attacks that are being tried, and perhaps that knowledge will help you manually scrutinize parts of the system that you know are not getting the full brunt of the testing.
We've also had situations where our throttling precautions caused us to dramatically slow or stop the acceptance of web traffic from the test, and it was helpful to catch that early on and work out a plan with the security test team to get them "the access they needed" while still taking credit for the fact that our throttling did what it was supposed to do.
We've never seen a scan take fewer than several hours, and we've never had it take more than three days, but as always "your mileage may vary".
Part 3: Following-Up on the Security Scan
Sometime after the scan (likely within a week), you'll find yourself on the phone with the penetration testers that administered the scan, walking through a report of their findings. The first time (and every time since) we've appreciated the level of detail in the report. In many cases, the findings will come with an example of an HTTP request that demonstrates the defect. You may or may not have access to the same tools that the penetration testers used, but if you sharpen your cURL command-line skills, you'll have a good chance at repro'ing many of the HTTP POST, HTTP GET, TRACE, or TRACK attacks.
As for XSS attacks, you'll likely find yourself at a browser window pasting in a URL with a frightening string of characters in the params. You might need to have your web server send a "X-XSS-Protection" header, to cause your webkit browser act naive about XSS (therefore acting a little bit like an older less cautious web browser). If you are a Chrome or Firebug fan, or a fan of IE's F12 developer tools, you'll want your console, call stack, and network activity at-the-ready so that you can see the exploit in action.
For SQL injection attacks, you'll probably find yourself diffing the MRU timestamps on your SQL tables, before and after running a cURL command line or browser-based copy-and-pasted URL.
For SSL-related vulnerabilities (like weak cipher availability, CRIME, BEAST, certificate chain problems, and others), you might find yourself using one of the web-based scanners like this excellent one from Qualys, as you work to mitigate the vulnerability and test your fix.
Once you've mitigated the vulnerabilities uncovered by the penetration testers, you'll touch base again to explain your fixes, and to work together to plan any next-steps — the eventual outcome being that you will fill out the security questionnaire to apply for production access.
See You Next Time: And, there will be a next time…
Once you are granted production access, know that the security scans will become part of the rhythm of your business — they happen on a recurring basis. The good news is, you'll get better and more efficient and handling them over time (nine runs in, we're getting pretty good at it).
Good luck navigating the security scan process, I wish you smooth sailing and a rapid arrival on Intuit App Center — see you there!
Leave a Reply