Go Ahead and Re-Invent the Wheel
On the surface, it seems obvious that we would leverage software that has already been built if it suits our needs. The underlying theory behind open source software and service-oriented architecture are, at least in part, based on this principal. In a similar vein, there was an article that got a lot of attention recently on Medium entitled “Coding is Over”. In this article, the author goes on to explain how we should stop rebuilding basic CRUD applications and how we should be able to use a single solution for solving similar problems. To be fair, sometimes we should. Authentication, click tracking, payment services, front-end widgets and the like all make sense to probably not rebuild. However, for most others applications, a one-size-fits-all solution built by another team is so often the wrong choice. Here’s why.
Cost
Inevitably, during planning someone will point out that we shouldn’t “reinvent the wheel” and team X has already build something like solution Y we are discussing and we should leverage what’s already been built. On the surface, not building something vs. building something seems like an easy choice. However, we aren’t considering the fully loaded cost for the not building option.
Recently, my team was evaluating whether to use an existing system to store customer preferences or build our own solution, which essentially would be a simple CRUD application. The existing service stored a customer’s library of music and the partner team claimed that it could easily be extended to handle the storing of a customer’s preferences. They had a well-defined API, were scaled for production and clients had already integrated with their client interface packages. On the surface, it seems like we should use their service, right? Well, one problem was that they couldn’t get to our feature request for another 2–3 months due to other priorities. Also, there was another team that had a similar set of requirements to store settings and we needed to work with them to get aligned on what the complete setting storage system needed to do. The full set of requirements needed to be baked and handed off to them to work on. Problems could arise if requirements were to change after that handoff and work had already begun. Very quickly the lead developer and myself had spent dozens of hours in meetings, gathering requirements and replying to email and not one line of code had been written. Just the process of planning to build a one-size-fits-all CRUD application took weeks of time out of our schedule.
Long story short, after a few weeks of this, the team and I made the decision to create a simple new settings storage service from scratch that was optimized for our use case. For example, a schema with a composite key, an API design that met our specific client requirements and flexibility in the weight parameter for the future use cases. We were able to complete the service end-to-end and have it in production in just over a month. It would have been even faster if we decided to roll out our own from the beginning before attempting the silver bullet solution. Lesson learned.
Dependency Management
Using another team’s service or application will hinder you in a way that isn’t immediately obvious. Once you take that dependency and something breaks or you need a new critical feature has to be added, one of two things happen. (1) You file an issue with the partner team, work with them to get it on their backlog and bug them constantly to finish it up so you can use the change. More often than not, they not only have your needs to consider, but the needs of other teams that are using their system. This goes beyond just functionality, but latency, scalability and API contracts. Do they create a new API just for you? Will your traffic pattern change the load profile in a significant way? Which version of their client interface are you using? When can they schedule this with consideration to the other priorities they have on their plate? All questions any good service owner will ask. The back-and-forth can go on for weeks. I’ve seen email threads go on for well over 50 responses in the process of asking a dependent team to make a seemingly simple change. (2) You make the change yourself as a guest developer. This saves time in terms of scheduling with a partner team, but now you are responsible for all of the nuances of their service and getting a check-in to pass the partner’s teams criteria. Also, you have to spend potentially huge time investments in learning their systems, codebase and test framework. All of this will likely take up as much or more time then waiting on the partner team to do it in the first place. Neither options are great.
Build a Better Mouse Trap
Another reason to build new is competitive advantage. From a startup perspective, if you are simply re-purposing an underlying open source technology, there’s nothing stopping the competition from doing the same. If you are using a service another team built at a larger company, you are tied to the core functionality of what that service does.
Don’t get me wrong, there are plenty of amazing tools and frameworks out there to help bring an idea to life and to help get a startup off the ground or allow a larger company to not waste man hours on something that isn’t their core competency. However, for the “secret sauce” of the application, re-using existing solutions is quite the opposite of how technology companies gain a competitive advantage and delight customers. Let me stress technology companies, whose core offering is the technology. Amazon doesn’t convert shoppers at the rate it does because it uses an off the shelf web framework to deliver product detail pages and PayPal for a checkout experience. Facebook doesn’t render amazingly fast on all devices because it used a popular ORM and a set of templates from Wix. These companies have made the uniqueness of their technology their defining aspect, which meant completely re-inventing the wheel.
Even within an organization there can be advantages gained from building a new solution when a similar one already exists. Earlier this year, my team started a project to deliver personalized ranking functionality. We started by evaluating the existing ranking functionality contained within a large org-wide service that could have served our use case. However, we had a new modeling technique and ranking algorithm that we wanted to try and we had very, very low (sub 20ms) latency requirements. Due to the complexity of trying to squeeze efficiencies out of the existing ranking service and the lack of flexibility to plug in new machine learning models, we made the decision to build a new data pipeline that did the model generation offline and then created a thin service that simply looked up a given customer in the generated data model, applied the scores and returned the re-ranked content. With our new solution, we had new modeling capabilities we never had before and we were able to deliver on our crazy strict latency goals. This never would have happened had we not re-invented the wheel.
Developer Happiness
A final aspect to consider is developer happiness. Developers are the lifeblood of technology companies and as leaders of an organization, keeping our engineering teams engaged, excited and passionate about what they are building should be a top priority. Find me an engineer that would rather attend meetings, follow-up on email and coerce another team to prioritize something vs. build a fresh solution for the specific goal they have and I will delete this post right now.
Passionate software developers got into the profession because they love to write code and build things, even if a similar thing has been built before. There’s always a fresh way to look at a problem, which will often yield better results. Not to mention, there is much more to be learned by building something from scratch vs. just calling an API and trusting that everything will work as advertised.
What can we do?
Seriously consider rolling your own solution for your use case, even if it is somewhat redundant with something that already exists. When we take this approach, we have full control over the prioritization of requirements and bug fixes for the application, we have the ability to make something unique and amazing that could serve as a competitive advantage and we are keeping our workforce engaged and productive. When a team independently owns their system, they can efficiently scale it to our needs, optimize it for the exact use cases and work to improve performance and create the best experiences for customers.
Rolling our own solution of course comes at a cost, but when we factor in the actual hours and lack of freedom required to take a dependency on another team/system, we quickly realize that in many cases, overall throughput is greatly increased with building our own solutions. If you are honest about the hours spent, you can actually quantify the time savings in a spreadsheet to help justify your decision. As mentioned, there are plenty of instances where it does make sense to use what is already there, especially when the complexity of what you are building is high. Each case should be evaluated individually, but the next time your team is faced with the “make or buy” decision, strongly consider make.
