Subsystems: The Best Way to Build a Scalable SaaS Platform

If you’re the CTO or lead developer at a SaaS company, there’s a good chance a visionary founder or excited marketing director already refers to your technology as a “platform.” Now it’s up to you to make that promise a reality.

It’s not as daunting as it sounds, because platforms are nothing more than a collection of interconnected yet independent subsystems. Each subsystem is responsible for some discrete unit of work that takes a set of known inputs and produces a set of known outputs while adhering to a contract.

By building discrete subsystems, we can quickly tweak a subsystem, push the change into production, and even run A-B or canary testing on the new subsystem — all with no downtime. As long as we adhere to the contract.

Winner, winner, chicken dinner!

Bonus: You can build new features into a subsystem before developing the UI/UX, allowing separate engineering teams to work more independently. 

At Leasecake, we recently started building out an integration subsystem. We already had integration points in our software, but they were largely soft, point-to-point integrations. That’s a great start, especially when you are moving fast and adapting to an ever-shifting market. But for our long-term health, it was time to build an integration subsystem.

Here’s the 30,000-foot-view of what we need to accomplish:

• Abstraction of domain entity
• Guaranteed processing of integration request upon confirmed receipt
• Ubiquitous workflow for all integrations
• Ability to integrate data with any open API
• Horizontal scaling

Abstraction of Domain Entities

Abstraction of data is actually the simple ingredient in our solution. I learned a lot about data abstraction at my first startup, where we processed a lot of abstract product data.

So at Leasecake, we started by asking questions: What are some characteristics all our potential integrations will share?  What does a calendar or email integration share with an accounting integration?

The answer is simpler than you may think. Let’s look at an abstract representation of an integration and break it down.

{
“requestGUID”: “”,
“topic”: “”,
“customerId”:””,
“document”: {
}
}

• requestGUID allows the calling entity a way to uniquely identify the request. We have to assume that integrations are long running workflows that will occur asynchronously so this will allow the calling entity to call back for updates rather than registering a callback. Registering a callback more tightly coupled the subsystem so we opted out of that.
• topic specifies what the request is. A topic could be something like “journal entry” or “slack notification”. 
• customerId identifies who is making the request.
• document is the actual data needed. Note that we’ve completely abstracted the data into a subdocument as this will be domain specific. Each domain will need its own topic configurator. That’s the only real code needed to to accommodate a new integration.

That’s basically it.

• Who is making the request?
• What type of request is it?
• And what are the data points needed to make the request?

Requesting an integration is a simple JSON object passed in through a RESTful API.

Guaranteed Processing

Let’s start by agreeing on what I mean by “guaranteed.”

With a best-of-breed cloud implementation, we guarantee an acknowledged request will process as long as an availability zone (AZ) doesn’t get wiped out. Furthermore, we can scale this horizontally across AZs and effectively have a nearly 100 percent guarantee of processing. 

On the front end, we have REST (although other services will likely lean towards GraphQL). Once we get the request, we quickly process it by handing it off to a message broker.

Modern message brokers are fast, scalable, and durable. Everything we need to guarantee processing once it’s been received. The message broker provides a receipt, we pass that back to the caller, and a state engine pulls the request off the queue and starts the workflow. 

In the event of a catastrophic failure, we recover the message broker queue, recover the state engine, check for the last point of agreement, and then restart the workflow. Other pieces of the workflow do their part and inform the state engine along the way. 

Piece of cake.

Ubiquitous Workflow

More abstraction, please.

What are the abstracted steps we need to perform?  Again pretty simple (the devil is always in the details, though):

1. Receive a request
2. Find a topic subscriber
3. Configure the topic for the customer
4. Hand off the meta data to a request manager
5. Manage the request to the third party

Of course, the magic happens in the steps. But there’s a lot of power in abstracting every workflow down to these steps. Configuring the topic is where all of the heavy lifting will happen.

Have a journal entry to pass on to Quickbooks? Write a configurator for it. What about a larger ERP like Oracle? Write a configurator. What about calendar entries in G Suite? Configurator. All the other steps are the same regardless of what you want to integrate with.

Sweet!

Integrate Data With Any Open API

Arguably, the hardest piece is the actual call into a third party. Yes, RESTful calls are fairly easy, but authenticating on behalf of a client adds complexity. Additionally, what if we want to integrate with an on-premise solution? 

We took a typical agile approach to this at Leasecake, crawl before you walk and walk before you run. Our “jacks or better to open” bet was to go with the most-requested integration. Our starting point was a fair amount of hard coding for the request. We learn from that, do another, learn some more — it’s your basic, “lather, rinse, repeat” approach.

This is a work in progress for us, so look for a future blog post on integration as we learn and adapt.