A composable middleware for Shopify

In today's fast-paced digital landscape, companies they need to be able to integrate with ecommerce platforms like Shopify quickly and efficiently. For many companies, however, developing integrations can be a long and expensive process.

For many companies, creating internal integrations can be a significant investment in terms of Time and money. It may require hiring additional developers or consultants, investing in new technologies, and dedicating significant resources to the project. In addition, creating a custom integration can take many months, depending on the complexity of the integration and the availability of internal resources.

Given the complexity and cost of creating integrations internally, many companies turn to digital agencies to help them develop them. By taking advantage of the agency's knowledge and experience, they can develop their integrations with less risk, although it can also be time-consuming.

The following diagram illustrates a custom middleware that is being built between an ERP system and Shopify. The blue area reflects the custom code that must be written for the integration.

Custom middleware/integration with Shopify is all work and no fun

As mentioned before, building middleware or an integration like this has several disadvantages:

High initial cost‍

Building middleware or integration from scratch can be a lengthy and costly process. The entire integration project must be covered, from requirements collection to design, development, testing and deployment. The cost may include hiring specialized developers, the purchase of hardware and software, and other infrastructure expenses. This initial cost can pose a significant barrier to entry for many companies, especially smaller ones with limited resources.

High maintenance cost

‍Once the integration is built, ongoing maintenance costs can also be high. As systems change and evolve, the integration may have to be updated, modified, or even completely rebuilt. To ensure that the integration continues to meet the needs of the company, continuous development, technical support, documentation and training may be necessary. This can increase the total cost and complexity of the project, making it difficult to manage.

Requires a team of expert developers

‍Building a middleware or integration from scratch requires a team of expert developers with specialized skills and knowledge. Adding new members to the team can be difficult, as they need to understand the entire system and its underlying architecture. In addition, developers with these skills and knowledge can be hard to find, especially for smaller companies that may not have the resources to compete with large companies for the best talent.

When a team of developers creates custom integrations for multiple customers, they often end up creating similar integrations with nuances unique to each customer. This results in individual codebases for each integration, which hinders and makes future development and maintenance difficult and expensive.

In addition, having to maintain a lot of customized integration projects can leave the team feeling overwhelmed and unable to focus on delivering new business value to customers. This can lead to burnout, where equipment becomes fatigued and demotivated, resulting in decreased productivity and lower quality work.

A team that tries to maintain many custom integrations simultaneously will struggle to keep up with the pace

El The problem of exhaustion also affects customers, as the team may no longer be able to provide the same level of support and expertise they were known for, resulting in a decline in service quality. In addition, if key team members who knew everything about a particular client's project leave, the knowledge gap can create significant challenges for the client and may result in the need for additional training or hiring new staff to take charge of the project.

A better way to integrate with Shopify is needed because current integration methods, such as creating custom integrations, can be costly, time-consuming, and difficult to maintain. In addition, these methods may not be able to meet specific business requirements or adapt to changes in the e-commerce market. By using connectors designed specifically for Shopify, companies can benefit from quick and simple integrations that meet their unique needs, while reducing costs and freeing up resources for other important tasks.

Can we build a composable middleware solution for Shopify?

Composable middleware solutions have become increasingly popular in recent years, as companies seek to create more flexible and scalable integrations. These solutions use a modular approach to integration development, allowing companies to connect multiple systems and services using a variety of pre-built components or “middleware.”

In the case of Shopify integrations, this could include middleware components for transferring orders to an ERP or OMS system, capturing or refunding order payments, managing shipping and fulfillment, and synchronizing products and customers.

If we could translate business integration needs with Shopify into generic functional requirements, we could build a composable middleware solution for Shopify. The ideal result would allow us maintain several integration flows to Shopify regardless of the customer's nuances, while at the same time it would allow us to communicate without any customer system.

What does Shopify recommend?

Shopify recommends the following guidelines when building integrations to ensure a robust and scalable solution. One approach is use webhooks to keep external systems up to date, while separating the management of webhooks from data processing and transformation.

The webhook processor receives and stores webhooks from Shopify, verifying their digital signature to protect them against counterfeiting, and sends them along with their associated data to the queue. The queue maintains a list of webhook data for processing, while the synchronization tool extracts this data and transforms it to synchronize it with other systems, such as ERP, PIM or OMS. The reconciliation work checks periodically for missing data and sends a synthetic webhook to the queue to ensure proper synchronization.

Cloud computing and data storage solutions, such as message queues, are recommended to manage traffic peaks.

Novicell middleware for Shopify

A few years ago, a Novicell team embarked on a journey to try to achieve just that: build a composable middleware for Shopify that could standardize integrations between several external systems and Shopify.

However, instead of creating integrations based on data, we take into account business requirements, we create business-based use cases and implement them. For example, instead of writing code that can “create a capture-type transaction for an amount of 100 euros”, we write code that can “accept an invoice note from an external system and capture the card payment accepted in the specified order”. In this way, we can provide value to companies without burdening them with technical implementation details. In addition, it allows us to change implementation details in the future when there is a better way to do so with little or no impact on the company.

No company is the same as another, so too We added the relevant extension points. This way, when we export orders from Shopify to customer systems, we can export exactly the data they need. In the same way, when we need to receive instructions from the ERP to capture order payments in Shopify, we define an input based on the business (and compatible with Shopify). ERP implementers can then choose to send data based on our data model or we can handle differences in the “adapter” component.

The following diagram illustrates the mentality resulting from aspiring to build middleware solutions that could be distributed to each customer with an “adapter” - code that translates data from systems external to Shopify and vice versa.

In doing so, we have been able to reap benefits for both the team and our customers:

1. Reduced development time. Less code to write means we can finish in less time than before.
2. Increased reliability. The “basic” components have a series of automated tests to try to detect errors in time.
3. Standardized documentation and training. Since middleware implementations based on our connectors share most of the same code, it allows us to write standardized documentation and incorporate new employees more quickly into all projects.
4. Lower maintenance costs. Maintenance costs are often lower because the “core” component can be developed centrally and distributed to customer implementation projects.

Complex integration processes are built and maintained in the green “Core” component, while mapping and data exchange with customer-specific systems are managed in the blue “Adapter” component.

We have organized our middleware in three separate connectors: one for products, one for orders and one for customers. With this modularity, we can achieve greater flexibility in development, since individual components can be upgraded or replaced without affecting the rest of the system. In addition, we can manage scalability and achieve performance requirements scaling each connector. On the other hand, smaller, centered components are easier to troubleshoot and maintain, since problems can be isolated to specific components.

The “Adapter” component also acts as an anti-corruption layer, so that changes to the “core” do not require changes in the connection with the ERP system. For example, if we later discover a better way to implement a specific use case, or if Shopify makes changes to its platform and APIs, we can Centrally make changes to the connector or connectors and distribute them to the client's middleware with minor changes or no changes to the existing adapter code, which means that there is hardly any need to do work on the ERP system.

Data flow

As mentioned before, we built our middleware connectors for Shopify based on the business needs of our customers. As a result, each connector implements a particular set of supported use cases that most of our customers share.

If a customer requires a particular customization, we can then investigate if the functional requirement is something we should build in the connector, handle in the adapter component, or build together with the connectors in the middleware layer.

Product connector

The middleware product connector works primarily with product and stock data. We built the adapter component to accept data from different systems, sources, or a combination of them. The data is then assigned to our standardized connector input and sent to the connector. From there, predefined, company-driven use cases are put in place to orchestrate Shopify API calls in the specific order in which they should be made.

If the product data is sent to the connector, but it hasn't changed since the last time the data was sent, then no API requests are sent to Shopify. This is important, particularly with large catalogs, because if all the data were forwarded to Shopify over time, then it would take much longer to update all the products and, more importantly, the stock.

Another advantage is that the product connector can receive product and stock information independently: we can store the information before receiving the product information. Once we receive the product information, we check to see if we've received the stock information before, and if so, we immediately update it in Shopify. It also works the other way around.

Order connector

The Middleware order connector works primarily with orders, transactions, and fulfillment. The adapter component is built for each customer, allowing us to send and receive data from any ERP or other system they may have. The data we receive is assigned to a standard connector input and sent to the connector. At this point, business-oriented use cases are executed based on input and orchestrate communications with Shopify.

When the order connector needs to send data back to the ERP or another system, it does so through the adapter component, where we specify what Shopify data we need, how it should be formatted and how it should be sent.

Communications with Shopify are carried out through the use of webhooks, as well as reconciliation work to ensure that the data is up to date.

When exporting orders to external systems, we can prevent the same order from being shipped multiple times. In addition, through the adapter component, we can specify which validation rules an order must pass in Shopify before being exported.

If the ERP system accidentally provides the same invoice note or credit note, the order connector can avoid charging or refunding the customer too much by ignoring duplicate documents.

Finally, if the order connector is configured to talk to multiple Shopify stores, we can automatically adjust the inventory of the items in the order in stores other than the one where the order was placed. This can reduce the risk of overselling low-stock items before the ERP can respond with updated stock levels.

Client connector

The Middleware client connector works primarily with customer data. We built the adapter component to accept input from external systems. The data is assigned to our standardized connector input and sent to the connector. From there, Shopify API calls are orchestrated to create or update the customer, addresses, and custom fields.

When a customer is synchronized with Shopify, any identifiable customer data in the connector is deleted.

What about Low-Code integration platforms?

Although Low-Code integration platforms can be a quick and easy way to connect a series of systems, it's important to keep in mind that your built-in connectors do not always meet the requirements businesses specific to a specific company. Many of these connectors are designed for data-driven integrations and may not provide the level of customization needed to create a fully optimized workflow.

In addition, some Low-Code integration platforms may be based on older technologies, such as Shopify's REST API, instead of the newer and more advanced GraphQL API. By using the REST API, companies can miss out on the latest features and updates that are only available through the GraphQL API. This could potentially limit a company's ability to take advantage of Shopify's rapid pace of innovation and adapt to changing market trends.

On the contrary, our connectors are designed with a deep understanding of Shopify APIs and of the specific needs of companies. By taking advantage of the latest Shopify technologies and updates, our connectors aim to provide updated and streamlined workflows.

On the other hand, it is also possible to combine the best of both worlds - Low-Code integration platform to handle the enterprise-level middleware layer with our connectors for integration with Shopify. Instead of the connectors “talking” to the ERP integration points, they would communicate with the integration platform.

An opportunity for your company?

If you want to know more details about our connector and understand how it can be integrated into your company's digital ecosystem, contact us. We will be happy to tell you more details about the implementation and all the advantages it can bring to your business