Everyone has a role to play in security, from startups to government entities to global corporations. Even the most security-minded companies, like financial institutions, still encounter challenges.

‍

For startups in growth mode, Bob Aman, one of Bulwark Security’s founders, provides insights and advice on DevSecOps and protecting customer data in companies who are trying to establish themselves and enter growth mode.

‍

Cee Ng: What is an example scenario of a common way that malicious actors can attack a business? 

‍

Bob Aman: There are many attack paths that can lead to a compromise of customer data, and many development teams may find it to be challenging to fully enumerate them.

‍

Let me give an example of a scenario I encountered a few years ago that caught me by surprise. We’d been investing in mitigation of cross-site scripting vulnerabilities in the product we were responsible for securing and we were starting to feel confident about our hardening efforts. We’d deployed content security policy headers to reduce our exposure, we’d completed a recent security assessment that hadn’t found any XSS issues, and we’d been doing our own careful internal scanning, testing, and evaluation. Beyond that, our platform seemed to have lower inherent risk because all user data was siloed. User data was only ever visible to the person who entered it, so for an XSS threat scenario, even if you found this type of vulnerability, we expected that you could only use it against yourself and your own account. I was feeling optimistic about our risk exposure from that threat. Overly-optimistic, it turned out.

‍

One day we got a really interesting report from an external security researcher working on our behalf, suggesting they’d found something in one of our support systems. Notably, those systems weren’t externally accessible, but the researcher had used an XSS payload designed to phone home. Around a month after they’d placed the XSS payload, a staff member had opened the researcher’s account profile in our internal tool, triggering the vulnerability as soon as they did so.

‍

Cee: Seems like overlooking the security of internal tools and support software might be a major risk for a business. Could you give some examples of how hackers might compromise customer support functions?

Bob: The researcher didn’t have the ability to inspect the internal software when they crafted the payload. They had to take a shot in the dark and we’d overlooked that this was even a possibility. We’d been focusing all our efforts on the customer-accessible product but hadn’t been evaluating our internal tools the same way. That was really eye-opening and motivated a strategy shifting in how we hardened our software.

‍

It’s easy to get into a mode where you get tunnel vision with respect to the risks in what you can see. A bit like what they say about icebergs, there’s all this risk exposure that’s less visible because it’s just not customer-facing.

‍

‍Cee: What is a way to address this? ‍

‍

Bob: The real challenge with securing customer data lies in the multitude of potential ways the data can be compromised. You’re hard-pressed to successfully secure against a threat you haven’t recognized, so it’s important to use techniques designed to help uncover them. Threat modeling is a practice I recommend. I’ve particularly appreciated Kelly Shortridge’s writing on this topic and I’ve gotten great value from her work around security decision trees.

‍

Sit down with the people responsible for each software component. You really want the widest range of perspectives possible in the room, both because it can otherwise be easy to miss something otherwise and also because it helps to build shared context around exactly what you’re trying to protect a system from. Threat modeling is a bit like a brainstorming session, but for bad things that could happen. When we don’t properly threat model the systems we’re building, especially during the design phase of a project lifecycle, we’re leaving way too much to chance.

‍

Ultimately, the result of a threat modeling exercise should be an artifact that can be used for a number of different purposes. It’ll be useful when creating and deploying mitigations for the threats identified, since we want to confirm that the mitigation really does address the attack path it was designed for. It can be useful for anyone doing a security assessment, where it may help focus their efforts on areas that may be less well-defended. It can even be useful when starting a new project or service, since it may give you a starting point for what proved helpful the last time around. I’ve also found it’s been helpful to embed security decision trees created during threat modeling in my design documents for a project.

‍

Cee: What are common ways that businesses are harmed by hacking attempts?

‍

Bob: Impact can range from direct financial consequences such as fraudulent wire transfers to more abstract issues like brand damage associated with a data breach or loss of availability. Personally, I’m most motivated by any potential impact on customers. It’s gotten to a point where we all joke about how many years of identity theft protection services we’ve all collectively racked up, but I want to believe that we can do much better as an industry for the customers we serve.

‍

Cee: For all stakeholders involved to be security-first in development aka DevSecOps, what steps should founders, investors, founding engineers, and individual contributors take to influence security considerations to reduce risk and prioritize time to be spent in security operations, especially before launching each feature? What is the trade-off of not practicing reasonable security protocols during product development? 

‍

Bob: Security in practice is full of unavoidable trade-offs. Particularly when stacking security up against the user experience. We should prioritize both where we can, but also be deliberate about where we’re making our trade-offs. We could choose to deploy seven-factor authentication, but then we’d have no users. Security doesn’t need to be user-hostile, but it does need to be effective.

‍

It’s easy to focus on growth at all costs as an early stage startup, but always remember that customer trust is hard to earn and even harder to earn back. The main thing to avoid is the creation of a heavy security process for software delivery. Make security as self-serve as possible for developers. You want to get out of the way as much as possible while still delivering on your customer’s expectations for a secure product.

‍

Try to prioritize avoiding common security mistakes that are difficult to fix later like embedding sensitive secrets in code. Bake automated checks and security tests into your continuous integration system. Automate as much of your security process as you can and then treat false positives and wasted developer effort as a cardinal sin.

‍

Ideally, when selecting components for your tech stack, prioritize those widely recognized as "secure by default." A secure default configuration allows teams to focus on the product implementation rather than hardening efforts that become necessary when dependencies haven’t been designed this way.

‍

Cee: What advice and recommendations do you have for early-stage companies who are preparing to enter growth mode? 

‍

Bob: It’s fair to acknowledge that security measures can slow people and the development process down. Day one security necessarily looks very different from a more mature organization. At the same time, companies need to recognize that the increased attention that comes with growth also applies to folks with ill intentions. Events like funding announcements, for instance, can sometimes trigger increased threat activity.

‍

Cee: What advice would you have for founders who are researching tech stacks for their product development and how to make decisions on which one to invest in?

‍

Bob: Reach out to trusted colleagues who have a security background and get their take on what you’re doing. Ask them what they think of your stack and what you need to watch out for. There are many frameworks out there that do a great job of balancing the needs of development velocity, the quality of user experience, and product security, but also an unfortunate many that do not. It’s helpful to know upfront what to expect.

‍

Take into consideration what your team has past experience with. Resist the urge to build on the shiny new framework. Velocity is typically going to be much higher with something mature and trustworthy and usually the security story is a lot better too. Tech stacks should be boring. Save the excitement for your actual product.

‍

Cee: Let’s say all of the above security measurements have been addressed, what is the next topic in security operations to be discussed internally as an agenda item?

‍

‍Bob: At the point where you have a product and it's in the market and you’re approaching product-market fit — that’s a good time to start spending some additional focus on your authentication story. You might’ve started off with a simple login form at the beginning of your product’s journey, but now you need to move a bit further down the attack tree. Phishing, customer email account compromise, credential stuffing, and sometimes even session hijacking may all start being more common problems at this stage.

‍

If you’re not delegating to a third party authentication provider, this is a good time to start thinking about implementing 2FA or something like “magic links” as part of your authentication flow. We’ve got a lot of evidence at this point that 2FA is incredibly effective when implemented correctly. It’s also wise to collect a lot of telemetry during deployment of a 2FA feature to help identify and address any sources of friction that might have been introduced along the way.

‍

Cee: What happens to the company reputation, customer perception, and brand loyalty in the event of a data breach? How can companies prepare their responses when facing highly automatable attacks?

‍

‍Bob: There have been some recent high-profile cases involving large numbers of customer accounts compromised via a technique known as “credential stuffing”. This threat relies on a fairly common user behavior where passwords are reused between different accounts. All it takes is one data breach leaking a user’s credentials and now everywhere that credential has been reused is at risk. Unfortunately, it’s not uncommon for people to be exposed to multiple breaches, and indeed, despite being a security professional, my own personal accounts have appeared in 36 of them! Generally users can’t do much to avoid appearing in a breach, they can only prevent subsequent consequences to their personal security.

What commonly happens is that threat actors will simply guess that a user has an account on a different service and that they kept the same password as the site that was breached. Typically, somewhere under 3% of the time, that guess will be right. While that may seem low at first glance, it adds up quickly when an adversary makes several million guesses.

‍

This method of account take-over has grown in popularity due to its effectiveness and can be a major source of fraud. For most companies, these types of security incidents lead to loss of customer trust even when a major contributing factor was user behavior. In my mind, it remains primarily the company’s responsibility to address these issues since they have the most visibility into what’s happening.

‍

For publicly traded companies, this type of activity may result in having to make an SEC filing related to the incident. State attorneys general have also indicated interest in credential stuffing, with the NYSAG office releasing a report on their investigation in January 2022.

‍

Unfortunately, many traditional security measures have proven ineffective against credential stuffing techniques, such as account lockouts after some number of failed attempts and simple IP address bans. Because credential stuffing will usually make only a single guess per account, lockouts don’t help, while lockouts also increase friction for legitimate users in the process. Additionally, it’s common practice to cycle through IP addresses during attacks. I’ve worked on cases where a single threat actor cycled through tens of thousands of unique IPs in a single day with no sign that they might exhaust their available IP pool.

‍

The response playbook to such incidents varies widely among companies and it can be particularly difficult for earlier stage companies that do not yet have mature security and anti-fraud functions yet. Adoption of 2FA or passwordless authentication is usually the best approach, but that’s often easier said than done in practice. Detection methods, such as bot mitigation can help significantly, but companies should carefully weigh the risks associated with false positives.

‍

Cee: How should founders initiate efforts to safeguard customer data and mitigate potential business compromises within their organizations?

‍

Bob: You can't protect something when you don't know it's there. Step one is trying to find out what you are trying to protect. We often refer to that as "doing inventory" — you inventory the things you need to protect, such as customer data, finances, physical hardware, software deployments, and credentials.