This post is by Grammarly software engineers Anton Pets and Yaroslav Voloshchuk. See part 1 here.
In our last post, we examined some of Grammarly engineering’s process-based methods of reducing complexity, derived from research, experience, and classic computer science guidelines. In this post, we’ll look at how we reduce complexity in the code itself, starting with our complexity pyramid, which informs our choices throughout the front-end development process and helps ensure that engineering hours are invested well. We hope to provide you with ideas for designing your code, educating yourself, and leveraging classic ideas of computer science to reduce complexity—from the start of project planning all the way to the final PR merge.
Complexity can creep in from anywhere: from new challenges or familiar problems. The latter can be especially insidious because we’re likely to be less suspicious of common issues. Consider email validation, a commonly encountered task. Shouldn’t it be a straightforward matter of working out the correct regex, validating it, and calling it done?
Alas, regex for email validation isn’t simple. And once you finally nail down a long string of regex pattern, you still need to send an email with a validation link that the user has to click. This creates obstacles: An extra step means a drop in conversions. Users sometimes can’t find the emails, or maybe just won’t take that extra step of opening to click.
We might address this problem by starting to send email addresses to the server—to check the MX records for the domains—while the user fills out the form. It’s not a bad fix, but it brings the inherent complexity of MX record validation into play.
Because that’s how complexity grows: By solving one issue you might add several new ones. Figuring out how to avoid doing so is difficult—but we’ve worked through a few processes that can help.
The structure of code complexity
There are two types of complexity in software: structural and behavioral. Structural complexity represents the built connection between your application’s components. Behavioral complexity comes into play when these components interact: when they start sending messages to one another at runtime. Behavioral complexity is strongly connected to how a user interacts with the system, whereas structural complexity is more about how engineers build the system. In this article, we will mostly focus on structural complexity.
In code, complexity occurs at increasing levels of abstraction. The complexity pyramid shown above is one way to think about how an engineer will encounter them. Let’s climb the pyramid to see what might help us at each level. We’ll need to reconsider basic code blocks that we might take for granted as having mastered: statements, functions and methods, and modules and classes. We’ll even need to reconsider how we think about applications themselves.
But first, we need to discuss some of the major tools in our arsenal for minimizing complexity up the pyramid.
Allies against complexity
Functional programming: Almost a silver bullet
In the previous post, we agreed that there is no silver bullet to fight software complexity. But a central part of overcoming complexity is ensuring that your code works predictably. One way we do this at Grammarly is by using statically typed functional programming.
Consider this level of excitement about FP:
We introduced static typing at Grammarly, which had helped us vastly improve our code quality but left us still struggling with unpredictability and complexity. That’s when functional programming entered our toolkit.
One particularly challenging task we encountered was synchronizing a user’s text between front-end and back-end. If a user clicks on the Correct Mistake button, for example, the error correction event should first go to the server. Only then should we correct the text in the browser and send the updated document to the server. If done in the wrong order, our back-end tries to handle the correction event on the already corrected text—and consequently produces an incorrect state. The two events—correcting a mistake and updating text—are examples of side effects. Coordinating multiple asynchronous side effects in a complex UI system is difficult and can be a limitless source of bugs.
We were able to address some of these side-effect issues once we began using functional programming, which was introduced to our team by colleagues that had backgrounds in F#. The first FP citizen in our repository was Option (aka Maybe) type, which is now used all over our codebase. Later, we discovered existing FP libraries that provide nuanced ready-to-use tools for controlling side effects elegantly. Because FP abstractions are universal (they operate like extremely forceful design patterns), the FP-powered solutions are easier to understand and reuse. We diminished our side-effect problems by using this functional approach, and now we always prefer declarative FP solutions to imperative or class-based code.
Types: An underrated (and critical) ally
Another major ally to consider in the fight against complexity: the use of types.
Though we know it can seem to add complexity when you adapt an existing codebase to have types, our own experience is that incorporating types merely surfaces unavoidable complexity already in the code.
So if your code is clumsy: Yes, adding types will result in clumsy types. Though this can itself be a step toward conquering complexity. Having a combination of different function calls with complex parameters can make it hard to understand the underlying logic. But when types are used, it becomes obvious what are the inputs and outputs of functions—and, consequently, it becomes easier to understand the behavior.
Without types, there is no verification before code execution (except code linters, which are more of a crutch used in the absence of types). There are no very compelling reasons not to use types, as static typing doesn’t add overhead at runtime in almost any available languages compiled in JavaScript. Some examples: TypeScript, Elm, PureScript, Reason, and OCaml.
Grammarly’s Editor was originally written in JavaScript because there were no popular statically typed languages that could compile into JavaScript at the time. Later, as the complexity of the app grew, the Editor became harder to support, and it was not easy to improve a big codebase without compile-time verification. So we started to research options for compiled-to-JavaScript languages that could meet our needs. We analyzed many options, including TypeScript, PureScript, and Scala.js. In the end, we chose TypeScript because it has a robust ecosystem and community of support. Even more, TypeScript is a superset of JavaScript, and so it was easy to adopt.
We completely rewrote our Editor in TypeScript so we could make a fair comparison with JavaScript. Long before the last commit was made, we knew we preferred TypeScript, so we used it to develop Denali, the latest generation of the Grammarly Editor.
How does this language switch relate to functional programming and our journey to limit complexity? JavaScript has all the necessary attributes to be an effective functional language: first-class functions, lambda functions, and closures. But functions in JavaScript are not pure, and data is not immutable. Furthermore, the lack of static typing makes it harder to compose neat functions because composition requires some contracts to be respected, and there are some other advantages of types in FP. For this reason, TypeScript is a better option for FP—especially now that we have such FP libraries as FP-TS and Funfix—and thereby a better ally for limiting complexity.
So even though there is no one fix to eliminate complexity entirely, we agree with Mark Seeman’s points in “Yes Silver Bullet”: Statically typed functional programming is as close to a silver bullet as we’ve found.
Structural Complexity
Now that we’ve discussed functional programming and the benefit of using types, we can return to our pyramid fully outfitted for success. Let’s address complexity from the bottom up, starting at the level of statements.
Statements: Unchanging variables and handling predictable exceptions
Some of functional programming’s most helpful characteristics come from some pretty elemental guidelines. To start: When possible, we use readonly variables, which keep the changes created by our functions compact and predictable, and our parameters unchanged.
We also try to contain unexpected effects by using for/while loops only in performance-sensitive places. for/while loops are powerful tools but are notorious for creating problems (there’s a reason Apple headquarters’ original address was 1 Infinite Loop). We use them judiciously and only when we know they’re the best option.
Finally, we try to perform idiomatic handling for empty values and errors with Try and Option. Option is an object type that may or may not have a value; this type is used to prevent undefined and null objects and their resulting errors. Try is an object that can be completed either successfully or unsuccessfully, without the need to catch exceptions in your application. By using these, you can be less defensive in your programming.
In the code above, we get an Option—that is, an optional parameter—and try to parse it with Try. First, we do it in one way, and if that doesn’t work, we try to recover in another way, which can enable your code to support two different formats of stored app configuration (in the case of json.parse, a JSON string). Then we try to validate it; if the validation worked, we merge it with our basic config and return it. If at some point the values we needed to proceed through the entire function are not there, we return the default value. And as we discussed with readonly variables the original variables supplied as parameters are unchanged and still available to other functions as needed.
Functions: Clear, pure, and SOLID
When possible, we prefer pure functions, and we try to avoid conditional if statements for switching behavior.
For instance, we have a function that writes logs, and when we run it in the local environment, it writes to the console. Instead of making one function with conditional checks, we make it polymorphic with two different implementations. The correct one will be used depending on the parameters supplied. Here’s an example:
For polymorphism, it is important to respect two important principles from SOLID—a classic set of computer science principles that we keep in mind throughout our work. First, the single responsibility principle states that each function or method should be responsible for just one thing. This makes our code more deliberate, clear, and predictable. Second, the Liskov substitution principle states that we should be able to easily switch from one function implementation to another.
Another way of ensuring clarity is to avoid class methods that can leave the class in a disassembled state. For example, when we create an alert in the user’s text (an object that stores information about a specific error or an issue, including its position in text and possible ways to fix the error or issue), we must register this alert with the positionManager, which is an entity that controls the absolute position of the alert in text.
If we haven’t registered the alert, we can’t do anything with it, so if an operation happens without registering the alert, we’ll get an exception. To avoid this, we should design our system so the object is provided what it needs from the start.
The composition is critical, and we use it frequently with functions, classes, and objects. When used with functions, it opens up a whole stream of possibilities in the form of memoization, piping, and other useful perks. We can also use composition to divide function logic from error recovery—e.g., to isolate handling JSON parsing errors from the actual usage of the parsing result. This then empowers us to use and combine different error-handling strategies inside a function without rewriting its logic.
Finally, it’s essential to write functions with a clear and easy-to-use interface. In the example below, the parse function takes a data parameter, and what it does internally may not be immediately evident in its twenty lines of code (to say nothing of much longer functions). If you specify that the function returns Try, engineers using your function later will be grateful, and future engineers will find it much easier to maintain the codebase.
Modules and classes
Although types bring considerable benefits at each level of the complexity pyramid, their utility becomes most evident at the class and module level. At this level, engineers start to think in terms of software architecture. We think through the design of our modules, how they interact, and their APIs, using types to describe these aspects. Then we implement our architecture, integrating work from other Grammarly engineers that might use different libraries or frameworks. Having clear types in place ensures that we can manage these different sources of complexity for this commit—and future ones too.
Invariance in types
In general, when constructing types, it’s best to ensure that they cannot express an incorrect or somehow contradictory state. Let’s imagine, for instance, that we are implementing an interface for a bulb switcher. We might inadvertently create an unpredictable state, like trying to switch off a bulb that’s not on. The solution would be creating a specialized bulb interface that does not allow such behavior. Consider the code sample below.
In some languages, such a design can be achieved with the help of algebraic data types.
Algebraic data types
This principle dictates that we cannot read data from a state that is incorrect according to the type system. By avoiding contradictory states, we can avoid complexity.
Correct data structures for better performance
Strong types help ensure that you’re using the right data structures and not defaulting to simple arrays and objects, which can be suboptimal. A few examples:
- Set vs. Array: If you need only unique values, use set instead of array, as it provides a quick uniqueness check.
- Prefix tree vs. Hash table: This is an important distinction for search functionality. For instance, if you needed to create an English-language dictionary that will be frequently searched but rarely modified, then you might use a prefix tree. Looking up data will be faster, even in a worst-case scenario, compared to an imperfect hash table that contains many key collisions. Moreover, using a hash table for this would require a huge amount of allocated memory, which is especially important to consider for front-end apps.
Data heading/encapsulation
This classic software engineering principle dictates that our variables, functions, and classes should have the smallest possible scope. Avoid exporting or importing unnecessary artifacts. Hide nonessential variables inside closures instead of storing them inside a class. Try not to clutter your namespaces.
Separation of concerns
This principle dictates that responsibilities are distinctly separated. For example, imagine that you need to implement functionality for bank-transaction processing. In addition to the core business logic, this task also requires authorization, logging, tracking, and other functions. You will want to isolate all these concerns into separate modules. Mixing them all into a single function or class risks turning into unsupportable spaghetti code.
Composition over inheritance
Inheritance is one of the ways to implement polymorphism. When we use it for composition, however, we end up with highly coupled, inflexible, and fragile architectures. Composition is usually preferable.
Applications and subsystems
Before we move on from discussing the structural complexity of building an app, we wanted to talk about complexity at the level of the application itself. This can sit between the code level—which we just discussed—and the behavioral areas. There are some shared principles that can help a project move cleanly from discussion through to implementation.
To start, we try not to reinvent the wheel when writing code, because less new code means less maintenance. If we have a need our current code and libraries don’t address, often there’s an existing library that will cover it—and our job, when we’re planning out a project’s architecture and code, is to find it. Design patterns and frameworks are similar. Many common problems are already well solved and don’t require a new or different solution. Sometimes engineers feel ambitious and are drawn to a challenge, but good planning can mean the difference between a successful project and one that requires a difficult retro.
We also suggest using a trustworthy runtime library—we like FP-TS, Ramda, Immutable.js, and lodash/fp. We keep our list of tools up to date through ongoing team self-education and sharing resources and papers with one another. We like dev.to, The Morning Paper, Dan Abramov’s blog, and this extensive repo of functional programming resources.
Behavioral complexity
Though behavioral complexity is a subject that’s too big for one blog post—much less a section of one—it’s important to examine how it fits into our discussion about structural complexity. If we look back at our complexity pyramid, we’re now going all the way back to the base of product requirements. Behavioral complexity is a concern from that early stage and continues running alongside all the structural concerns we’ve been discussing.
Behavioral complexity in front-end applications at Grammarly is often caused by the asynchronous nature of human-computer interaction. We have struggled with common front-end challenges like callback hell and orchestrating application actions via event emitters. But one thing has made our life noticeably better: functional reactive programming (FRP). We found it especially effective to manage application state. We even created our own state management framework, Focal, which leverages our favorite parts of React and RxJS.
To illustrate how one might deal with this type of complexity, let’s look at an example of functional reactive programming usage. Here we are waiting for the user’s mouse to get into a certain area of Grammarly’s Editor; after that, we will start tracking mouse moves. Our task is to find DOM nodes under the pointer. For performance reasons, we don’t want to immediately notify the system about the mouse leaving the area, so we add a 200-millisecond delay. Finally, we filter duplicates of those states so we don’t overload the system with unnecessary actions.
There are a lot of complicated things going on here—and keep in mind that this code would be much longer if it were written in imperative style. In fact, just to show this complex sample, we had to violate some of the principles discussed above. Namely, the single responsibility principle is not respected, and some functions are not separated into their own modules.
But perhaps this itself is the best illustration of complexity: Even just demonstrating and explaining how to manage, it can be so complicated you might need to question whether you’re taking the right approach. You might even have to go back to the beginning and make sure you planned things out correctly. (But don’t worry—we won’t do that to you in this post.)
Takeaways
Complexity in software engineering is—no surprise—a very complex subject. We can never fully eliminate it, but with the use of long-lived principles (such as SOLID) and the use of recent engineering advances (such as recent functional programming libraries like ft-ts), we can reduce it enough to make reliable software that isn’t a burden to maintain. Even just incorporating one or two of the above ideas might make your team’s work measurably easier. With a strongly typed language here and the adoption of readonly variables there, adding a new feature becomes that much less burdensome.
