The GoldenGate of Grammarly Business

The Golden Gate of Kyiv is an impressive medieval gateway, which has provided a spectacular entrance to Ukraine’s capital city since 1024. In those days, such gateways offered crucial protection to large urban areas surrounded by dangers and uncertainty. They played an important role in a city’s growth and prosperity.

We named the application gateway for Grammarly Business after Kyiv’s Golden Gate. It helps protect our services and authenticate different kinds of Grammarly Business users. In this article, we’ll talk about why we needed such a gateway, how we built GoldenGate, and how it’s helped us scale our engineering efforts. 

Early days

Grammarly Business was the kind of project that grew from a small idea to one of our company’s leading initiatives in a very short period of time. We had to move fast, keep delivering new features, and maintain a highly stable and available system. It was exciting, but we knew we were accumulating technical debt and would have to pay it back sooner or later. 

This debt was particularly concentrated in one place: a service called Institution. This service was a monolith involved in everything related to Grammarly Business and all its features, and it had many responsibilities. Some were product-related, like group subscription, user management, and applying different business rules. Others were related to security and networking, like rate limiting, circuit breaking, retries, and providing our web application firewall (WAF).

Our Institution service quickly became a bottleneck. We were working on a single repository with a single GitLab pipeline, creating a ton of merge conflicts and traffic jams. We wanted to move to an architecture where each new Grammarly Business feature would have its own separate microservice. But we needed to avoid duplicating all the common logic related to security and networking in each microservice, since this would only create new technical debt.

Cornerstone

We quickly realized that the best option would be to move all the nonproduct functionality to a separate service that all our future microservices could use. In other words, this service would be a gateway for all our publicly facing endpoints.

An application gateway is a well-known pattern for solving these kinds of problems, and there are many popular frameworks that can help with the implementation. 

Grammarly Business’s back-end is mostly written in Java, and this is the language that engineers in our team are most familiar with. With this in mind, we had three major frameworks to consider: 

• Zuul 2 from Netflix
• Cloud Gateway from Spring
• API Gateway from AWS

Zuul 2 and Spring Cloud Gateway have a lot in common. Zuul 2 includes some cool features like service discovery and load balancing—but in our case, we were already getting these features from AWS, where we host our services. So it came down to Cloud Gateway (Spring) or API Gateway (AWS). 

We built a proof of concept for each technology to get some hands-on experience and insight into the pros and cons. API Gateway provides features like WAF and rate limiting out of the box and doesn’t require a separate Redis cluster setup—it was easier to set up than Cloud Gateway, where we needed an EC2 cluster with an Application Load Balancer. 

However, we wanted to create a generic service that could be used by many different teams, so good test coverage and flexibility were must-haves. With a Java service based on Cloud Gateway, we could achieve the complete test coverage we needed, from low-level unit tests to high-level integration tests between GoldenGate and upstream services in a real environment. Plus, we could use a single repository powered by Spring profiles to address the needs of different teams. These two factors tipped the scales in favor of Cloud Gateway.

Building GoldenGate

Laying the foundation

Once we decided to use the Spring Cloud Gateway framework, we started our work. We implemented all the nonproduct responsibilities (rate limiting, firewall, circuit breaker, retries, web security, and authentication) pretty fast because we had to do only a small number of customizations on top of what Cloud Gateway already provided.

We used LEGO-like route builders with filters for each of these responsibilities (except WAF, where we used what was already available from AWS). We happily reused filters provided by Spring where possible (rate limiting, circuit breaking).

Here’s an example: 

@Configuration
public class BrandNewFeatureRoutesConfiguration {
    @Bean
    public RouteLocator brandNewFeatureRoutes(@Value("${brandNewFeature.url}") String uri,
                                              @Value("${brandNewFeature.requestsPerSecond}") int requestsPerSecond,
                                              GoldengateRouteLocatorBuilder goldengateRouteLocatorBuilder) {
        GoldengateRouteConfig routeConfig = GoldengateRouteConfig.builder()
          .uri(uri) // upstream service URI
          .rateLimiterConfig(requestsPerSecond) // per user/IP rate-limiting
          .circuitBreakerConfig(defaultCircuitBreakerConfig()) // default circuit-breaking configuration
          .permissions("brandNewFeature") // subset of permissions required by upstream service for authorization
          .build();
        return goldengateRouteLocatorBuilder.routes()
          .goldengateRoute("brandNewFeatureRoute", "/brandNewFeature/**", routeConfig) // routing rule
          .build(); 
    }
}

Now, each public endpoint that we wanted to expose to our users could easily be enriched by any number of pluggable functionalities.

Building the walls 

Our next important milestone was quality assurance. To make sure that any contribution to our service would be smooth and safe, we implemented a lightweight system-tests framework.

Using a few building blocks, each contributor could now easily cover their integration with GoldenGate using automated tests and ensure that their endpoint is accessible via GoldenGate in a real environment.

Here’s an example:

@Test
void routeToBrandNewFeatureRulesEndpointAuthorized() {
    var institution = testInstitutionService.createInstitution(); // creates a real institution (Grammarly Business organization entity) on a real QA/pre-production environment
    var institutionUser = testInstitutionService.createInstitutionUser(institution); // creates a real user and links them to an institution
    var uri = "/brandNewFeature/v1/institution/%s/rule".formatted(institution.getId());
    var expectedResponse = "{ "rules" : []}";

    routeSystemTestHelper.assertRouteIsUpAndRunningForUser(
          institutionUser, uri, HttpStatus.OK, expectedResponse); // performs the HTTP call and asserts that the response is as expected
}

@Test
void routeToBrandNewFeatureRulesEndpointUnauthorized() {
    var uri = "/brandNewFeature/v1/institution/123456/rule";
    var response = gatewayWebClient.get()
         .uri(uri)
         .exchangeToMono(clientResponse -> Mono.just(clientResponse.statusCode()))
         .block();

    assertThat(response).isEqualTo(HttpStatus.UNAUTHORIZED);
}

Raising the gates

Now we’d covered the key functionality and quality assurance for GoldenGate, but we still had one unsolved problem. Most of our upcoming feature services would need to handle role-based authorization. Some Grammarly Business features should be available for each organization user, while others are only available for account owners or contributors. It would be great to avoid reinventing the wheel in each feature service with the help of GoldenGate.

After a couple of brainstorming sessions, and inspired by Netflix’s Passport, we introduced UserPassport. It’s a simple protobuf object that gets populated with essential user information by GoldenGate’s custom Authentication HttpFilter, such as the user’s ID and what features they can access. Each public endpoint routed through GoldenGate will now receive a separate HTTP header containing essential information about the user.

This approach helped us to

1. Avoid overloading Grammarly services with excessive calls to fetch user information
2. Provide the information required for authorization to upstream services

The next step was to design a common authorization approach for all our services. We could potentially use Spring Security here, but it didn’t meet all our custom requirements. We ended up with the following flow:

We wrote a simple authorization library that parses the UserPassport from the HTTP header, fetches the required user data, and uses the data for authorization. This simplified the authorization process drastically. Now it’s just two steps: 

1. Import the authorization library.
2. Add @HasPermission on your public endpoint.

That’s it. Of course, there is a permission management routine that lies under the hood and handles things like registering permissions for new endpoints and assigning them to different user roles. But that’s a topic for another blog post!

Inside the gates

Within the first few months of launching GoldenGate in 2021 and migrating our services to use it, we started to see major benefits. 

• Our core Institution service shrank in size drastically. 
• We removed thousands of lines of code from our repository.
• A ton of feature-related business logic was extracted to separate services, making it much easier to work on multiple features in parallel.
• Our bottleneck was gone, so our feature delivery pace increased many times as well. 

With the addition of its “city walls,” the Grammarly Business team was able to thrive. What had been a group of less than 20 people (with fewer than a dozen engineers) quickly almost tripled in size, evolving in multiple directions simultaneously. 

In fact, our single engineering team turned into three separate ones. And GoldenGate played an important role.

Fast-forward to 2022. . . 

Currently, 11 Grammarly Business feature services use GoldenGate. These services power brand tones, style guides, snippets, security client controls, and more.

GoldenGate turned out to be a very cheap solution to our problems. Thanks to the asynchronous nature of Spring Cloud Gateway and its nonblocking request processing, GoldenGate uses CPU very efficiently. Our Cloud Gateway–based solution processes thousands of requests per second and in comparison to AWS Gateway REST API pricing (which was our alternative option) costs us 10 times less.

It’s always rewarding to eliminate technical debt and help your team scale. Speaking of scaling—Grammarly Business is hiring! Check out our careers page.