Is Complexity Bad?
The answer is… it’s complicated. 😀
Our new friend George Brocklehurst from thoughtbot’s NYC office recently gave an excellent talk at Dia&Co., where we proudly sponsored and hosted the NYC.rb Meetup group. Ruby on Rails represents a significant part of our technology stack, and a few of us have been attending this Meetup group (which is the largest Ruby Meetup group in the world, by the way!) for the past year. We are so grateful for this community to be right in our backyard, and this is just the beginning of the Dia&Co. engineering team contributing back to the Ruby community that we have benefitted so much from.
Anyways, one part of George’s talk which really resonated with me was this question as to whether complexity was bad. Funny enough, I have recently had similar conversations with other engineers, and the answer to this question brings up an illuminating distinction.
Two Different Types of Complexity
You can essentially think of complexity as a distinction between two different types. Accidental and necessary complexity. Necessary complexity is okay, but accidental complexity will absolutely ruin your day as a programmer. Let’s discuss each.
Necessary Complexity
Necessary complexity arises as a result of the problem space in which you are operating. If you are Stripe, you need to deal with things like the banking system, which are super complicated, and an overall schlep. In the schlep sometimes there are enormous business opportunities, as is the case with Stripe. In order to process the large volume of customer payments that we do, and manage subscriptions for a growing customer-base, it would be a real shame if we also had to write and maintain code to deal with actually communicating with each individual financial institution. Stripe handles this for us and for so many other companies like ours, and as a result, they have become a multi-billion dollar business.
Examples of our necessary complexity include maintaining a search index of tens of thousands of product variants, knowing which of these product variants are in-stock in real-time, and being able to provide the most efficient search interface possible for our styling team, both from a UI perspective and from a backend performance perspective. This is just the business that we are in, so any complexity arising out of this business need is okay, because it’s non-optional.
Accidental Complexity
This is bad news! Accidental complexity is a controller method that has 103 lines of code. Accidental complexity is 20 small things that are not all that difficult to understand on their own, but that are all in one place that makes understanding how the entire module works at one time a near superhuman feat. Accidental complexity, to put it mildly, will ruin your day!
We have a few areas of accidental complexity in our application. One of those happens to be in the controller action which powers the search functionality for our stylists. We’re doing things like building parameter hashes and redirecting to a default search for the first-run experience when a stylist begins working on a brand new box for a customer. We’re initializing an instance of our box analyzer tool which basically acts as a linter of sorts to ensure compliance with the guidelines for items provided by our merchandising teams. We’re building further options hashes to pass to our Elasticsearch service as well as to our recommendation systems. We’re constructing two vastly different Elasticsearch queries based on two completely different types of search methods, all in the same controller action. We’re merging results returned from the application database, the recommender services, and Elasticsearch, and finally, tracking all of this in analytics for our data warehouse.
To put it mildly, this is not one of our favorite areas of the application to maintain. It masks bugs and introduces performance problems, which is bad for our stylists. It makes code hard to understand and change, which is bad for our engineers. Finally, it makes things take longer to build and change, which is bad for the business.
How Did We Get Here
Those who are unaware of history are destined to repeat it.
No one on our engineering team set out with the intention to introduce this sort of accidental complexity. The software originally needed to do a subset of the functionality and business logic that it currently is responsible for, and at that point, it was far more appropriate for that logic to occur all inside of this controller action. Then as time progressed, and more features were required to support the growing business, this module fell victim to the classic frog boil problem. Little by little, more responsibility was added, until the entire module was, in essence, boiled alive.
Our Plan
Fortunately, one of our newer engineers on the team, Lisa Nguyen, who is not quite yet indoctrinated and immune to our previously questionable coding practices 😉, noticed this while working on an iteration of our inventory search tool. She took an opportunity to refactor out some of this accidental complexity, which will benefit our stylists, our business, and our team. As it turns out, making code they touch simpler, clearer, easier to work with, and faster to change, out of habit, opportunistically, and continuously is one of the traits of exceptional engineers, and something we definitely screen for when choosing who to hire on our engineering team.
If this way of thinking about building software and systems is appealing to you, I encourage you to reach out to me personally about joining our engineering team at Dia&Co. We are solving a worthwhile, massive, and real customer problem, and doing it with a world-class product, engineering, and business team.
