We believe that Grammarly’s users should have transparency into how their data is protected. One of the main ways that we protect users is by catching and resolving vulnerabilities in our systems before attackers can exploit them. In this post, we’ll share how our vulnerability management program at Grammarly keeps our development pipeline and user data secure.
Meeting the vulnerability management challenge at Grammarly
In recent years, we’ve invested considerably in our vulnerability management program. Previously, like many other companies, we relied on several vulnerability platforms to automate our security assessments. Each tool had a different user interface and console, and the results of these separate tools provided a fragmented view that we needed to consolidate manually. At times, after detecting a vulnerability, we encountered delays in addressing it due to challenges in identifying the right contacts for remediation and assessing its potential impact.
Prioritizing which vulnerabilities to address first also posed challenges. The Common Vulnerability Scoring System (CVSS) provides a standardized way of scoring vulnerabilities and offers mitigation factors, like Temporal and Environmental scores, that contextualize them further. However, it’s critical to interpret these scores in the context of your organization’s unique environment, assets, and risk appetite, as the vendor cannot go beyond the base score and doesn’t have enough data or capabilities to automate setting the Temporal or Environmental score. For instance, a vulnerability like CVE-2021-4428 – Log4j, which has the highest base score of 10, would often require a high priority for remediation, but the priority may be lower for a back-end system with minimal access. To understand the true priority of each case, we need to use the CVSS score as an initial indication of the vulnerability’s severity, which can then be combined with other contextual and environmental factors to determine its actual risk and prioritization.
We created a custom vulnerability data ingestion and prioritization workflow to obtain a consolidated view of vulnerabilities and better prioritize our remediation efforts. As a result, security engineers at Grammarly can now obtain crucial context on our asset exposure, business roles, and the types of data being affected. Using this information, we can prioritize our efforts more effectively and rapidly reduce risk.
The next section will show how we achieved this in more detail.
How we assess, prioritize, and remediate vulnerabilities
We are continuously conducting rolling assessments of our development infrastructure and pipeline. This is necessary because new vulnerabilities in cloud systems, open-source systems, operating systems, and development tools arise daily.
“Work on what matters” is one of our most important tenets as a security team. When we detect a vulnerability, we don’t just look at the immediate exposure and severity score—we understand the full context to make sure we’re prioritizing effectively. This means modeling the following:
- Attack paths: An attack path is a chain of points that reach an asset of value, such as customer data. We look at what devices or systems can interact with the affected service to determine if there are high-risk attack paths exposed by this vulnerability.
- Data criticality: Data regulated by industry, government, or our internal policy mandates is of the utmost importance to protect.
- Security intelligence: We continuously identify adversaries, study their attack techniques, and replay these techniques within our environment. This lets us learn their tactics, techniques, and procedures (TTPs). We correlate TTPs with our vulnerability reports to understand which vulnerabilities reside in systems that attackers are most likely to try to exploit.
Regarding remediation, we work on updating and patching our systems and automating tasks whenever feasible. For instance, if a vulnerability is announced in one of the developer libraries our teams use, we’ll instantly upgrade our teams’ libraries to the same version for everyone. If a vulnerable library or other component appears in a container, we’ll update the base container image and eradicate the issue systemwide at scale.
In addition, we maintain an accurate and up-to-date inventory of internal assets and their owners. This helps us engage with the right people to make fixes in minutes or seconds.
Metrics, dashboards, and how Grammarly continuously improves our vulnerability management program
Measurement is critical to improvement, and we’ve centered our vulnerability management program around a core set of metrics:
- Mean time to discover: Time from detecting where a vulnerability is in our system to publicly documenting it
- Coverage: The portion of our development environment that we are covering
- Scan failures: How often do our scans fail (error, crash, time-out, broken configuration, unsupported technology, etc.)?
- Unhealthy Tickets: Number of tickets not meeting our quality standards, which are (1) must have an owner, (2) must have a severity, (3) must have a due date
- False Positives: We monitor false positive rates and keep them below 30%. Why not zero? We worry about false negatives.
- Mean time to fix: Time from discovering a vulnerability to fully resolving it (including rolling out the fix)
- Out of SLA: We monitor for issues that exceed our mean time to fix for Critical (30 days) and High (90 days).
It’s one thing to track these metrics, but one of our tenets is that we are never done improving. This is why we look at our key metrics every week, analyze what has changed for better or worse, and brainstorm ways we can be better. We actively examine our data to learn from past situations and improve our tools and processes.
Finally, so that the appropriate stakeholders always have access to the right vulnerability management information, we provide dashboards tailored to different roles:
- Security leadership: We provide security leaders with a high-level overview of the status of our program. This includes the number of vulnerabilities uncovered in our assessments, the percentage of those that have been remediated, and trends over time.
- Engineering leadership: We provide engineering leaders with insights on the state of security in their space, including a list of security vulnerabilities to resolve, upcoming and existing out-of-SLA issues, and their team remediation velocity.
- Engineers: We provide team members with data relevant to their role, such as a prioritized manifest of vulnerabilities assigned to them and auto-remediation code changes they need to approve.
We’re proud of how far we’ve come with our vulnerability management program. The work is ongoing as we continuously assess our systems for new vulnerabilities, prioritize the resulting updates, and validate that patches are in place. In addition, we constantly measure how well we’re doing to identify ways we can improve.
Managing vulnerabilities is a project that’s never done, and it’s another example of how we strive every day to live up to our users’ trust in Grammarly. If that mission resonates with you, check out our open roles and consider joining Grammarly today.
