How to Create a Custom Checkout Engine from Scratch

December 30, 2022
13 min read

The checkout page is the final step of every online purchase, where customers enter their payment and shipping information. Though the checkout process may appear straightforward, there is a lot of work that goes into creating a secure, user-friendly checkout page. In this article, we'll explore whether you need a custom checkout engine and how to create it.

Do I need a custom checkout engine?

There may be many reasons to create a custom checkout. You could need it for selling online, integrating it with your SaaS, or using it for a game-changing product. Before we dive into creating a custom checkout, let's consider when you'll need to build it.

As a business, you want to keep things as simple as possible. Therefore, it may be unreasonable to build a custom checkout engine if existing solutions solve your problems.

For example, if you want to start selling online and don't have any unique requirements, the best way is to use platforms like Shopify, BigCommerce, or Magento. These have a built-in checkout page with features like Apple and Google Pay, tipping, or saving information for the next checkout session.

If you're building a custom SaaS application or platform, various checkout solutions are also available. You can look at Stripe Checkout, which allows you to accept payments for one-time purchases or subscriptions with a few lines of code. Another popular example is, which provides an API for processing payments, calculating fees, and managing subscriptions.

However, if you have a unique business model or product requirements and can't use any existing checkout solutions, don't worry. This article will guide you through everything you need to successfully build a fully-fledged checkout engine and start accepting payments online.

How the checkout works

Alright, you've decided that you need a custom checkout. But how does it work? What system powers the page with a few text fields? There's no single answer — you can create it to fit your needs, adding components such as 3-D Secure or third-party integrations. However, some core components are essential for any checkout system. In the following steps, we'll examine them.

What components does the basic checkout engine include?

  • High-level components:
  • A form for customers to fill in their payment and delivery information.
  • The API, the core of the checkout engine.
  • A platform for managing customer or store data.
  • Payment components:
  • A payment gateway, a service that processes payment information.
  • A payment processor, a service that processes transactions.
  • Security components:
  • A 3-D Secure system.
  • A tokenization system.

High-level components

It's essential to understand that a custom checkout system consists of three parts:

  • A form for customers to fill in their payment and delivery information. This could be a web page or a mobile/laptop app screen, depending on your product.
  • An API to process the customer's information, create an order and perform other necessary actions. This is the system your development team will create, connecting all the checkout components.
  • A platform for managing data, which you or other merchants can use to manage customers and their purchases.

The form should be simple and convenient to use, so customers won’t struggle to fill it in and you won’t lose them. You can find best practices for designing a checkout form here.

The API is the core of your checkout engine, responsible for all internal operations such as money transactions and creating orders. The internal components we’ll cover in this article will communicate directly with your API.

You can create a platform for managing data yourself, or integrate an existing one. We’ll discuss this further.

Meet the payment gateway

Suppose you have a checkout form, where the customer fills in their payment and delivery information. What comes next? How do you process this information?

The first API component is the payment gateway. It takes the customer's payment and delivery information, encrypts it, and sends it to the payment processor. When the customer fills out the form, their data is sent to the payment gateway through the API.

There are many payment gateways to choose from, such as Authorize.Net, USAePay, and others. Pick the one that best suits your needs.

Payment processor

The payment gateway sends the payment information to the payment processor, which relays transactions between the customer's and seller's banks. For example, when the customer's money is transferred to the seller's bank account, the payment processor handles this operation.

You don't need to worry about the payment processor; the payment gateway typically handles all its operations, so there's no need to do it manually or select the payment processor and monitor its work.

A platform for managing data — existing or custom?

Suppose you've chosen and integrated a payment gateway (which already contains a payment processor), and your checkout system works like this: customers fill in their payment and shipping information on the checkout form, then your payment gateway creates the transaction using a payment processor, and a certain amount of money is transferred from the customer's bank account to the seller's.

You also need to manage your customers: view the delivery address for the purchase, make a refund if necessary, view purchase analytics, etc. If you're selling goods online, you also need to manage the products (add, edit, or remove them from your store).

So you need a system to manage everything that happens before and after the checkout process. You can build it, of course, but it would be a lot of work. The simplest (and best) option is to use one of the existing platforms and integrate your custom checkout there. We recommend Shopify or BigCommerce, as they are robust, have many useful features, and are trusted by thousands of merchants worldwide.

If you're building a custom checkout for other merchants, another advantage of using an existing platform is that the more merchants already use it, the more likely they are to try your custom checkout that integrates with their platform.

Integrating checkout with the platform

Once you've chosen the platform, you need to integrate your checkout system there. For example, once the order is created and processed through the payment gateway, you need to create the order on the platform so you or other merchants can view it. Similarly, if the order is refunded on the platform, you need to use the same payment gateway to refund it on the API side. This is a two-way integration.

The integration with the platform depends on the platform itself. In most cases, your custom checkout needs to be installed as an app to access the merchants' stores.

The basic checkout engine is ready

The basic checkout engine is illustrated above. It includes a form for customers to fill out their payment and delivery information, a payment gateway to process transactions (using a payment processor), and a platform to manage customers, purchases, and products.

You can choose components according to your needs and integrate your engine with multiple ones, like multiple payment gateways and platforms.

For a more extended checkout engine, you may need components like 3-D Secure, tokenization, or third-party integrations. We'll discuss how to implement them in the following steps.

Third-party integrations

Integrating third-party services, such as analytics, with your engine can be done on the API side, connecting them just like any other component. Alternatively, they can be integrated with a platform of your choice.

While the second option may appear easier, there can be unexpected consequences. For example, if you're creating a checkout for other merchants integrated with an existing platform, a merchant may install a third-party app on the platform that affects the checkout. This app may work well with the default platform's checkout, but your custom checkout engine may not recognize it. Therefore, you must consider all third-party apps that can be installed on the platform and adjust your checkout's logic accordingly.

The illustration below outlines two ways to integrate third-party apps: connecting them to your API and the platform.

3-D Secure

3-D Secure (or 3DS) is a protocol designed to add an extra layer of security to online transactions. After customers enter their valid payment information, they must confirm the transaction. For example, their bank may request confirmation by sending a security code or a Push notification.

Another reason to use 3-D Secure is that once the customer confirms the transaction, their bank takes responsibility for it. This is called a liability shift. In the event of fraud, the customer's bank will compensate them, not your checkout system.

Fortunately, you don't need to implement 3DS manually. Services like PAAY provide 3DS functionality that you can integrate with. The transaction must be verified with 3-D Secure before authorizing it in the payment gateway. Learn more about 3-D Secure integration here.

Tokenization — introduction

The diagram above illustrates how customer payment information, such as card number, expiration date, and CVC, is sent to the payment gateway through your API. According to Payment Card Industry Data Security Standard (PCI DSS), any system that processes or transmits payment data must pass the PCI DSS compliance validation. This involves meeting 12 requirements, which international payment systems regularly review.

In other words, if your system touches a customer's payment information at least once, regardless of where (on the web page or server side) or what is done with it, you must pass the PCI DSS compliance. This makes things more complex.

Tokenization is a useful solution here. It helps your system not interact with payment information and not even see it. During tokenization, a service that takes responsibility for the customer's payment info (and is PCI DSS compliant) exchanges sensitive data with a non-sensitive token. This token contains no information and cannot be converted to sensitive data, so your system can use it without PCI DSS Compliance validation.

Tokenization — implementation

The illustration above presents a problem: the checkout cannot pass payment information to the tokenization service without adhering to the PCI DSS. To implement tokenization, two additional steps are necessary.

Let's use TokenEx tokenization provider as an example. The checkout form, where customers input their payment info, communicates directly with TokenEx. This exchange of sensitive data is done with a token. Afterward, the token is passed to the API.

In the API, operations requiring payment info (e.g. authorizing transactions with payment gateways) are processed through the TokenEx proxy. This proxy exchanges the token with payment info before the request is sent, so our API is unaware of the payment info, but the payment gateway is.


With tokenization and 3-D Secure, the checkout workflow is as follows:

  1. The customer fills out the checkout form connected to the tokenization service, and their payment information is replaced with a non-sensitive token.
  2. The token is sent to the 3-D Secure service, which requests additional transaction confirmation.
  3. After the user confirms the transaction (passing 3-D Secure), the token and delivery information is sent to the payment gateway through the tokenization proxy. The proxy then replaces the token with payment information.
  4. The payment gateway encrypts and transmits the payment information to the payment processor.
  5. The payment processor performs the transaction (for example, transferring money from the customer's bank account to the merchant's).
  6. The engine creates an order on the platform, allowing you or your merchant to interact with it.
  7. When the merchant takes any action on the platform (e.g. refunds an order), the platform sends a webhook (event) to your API.
  8. Your API then refunds the appropriate transaction using the same payment gateway and notifies the platform that the operation is complete (so the merchant can see that the order was refunded on the platform).

This article has covered the basics of creating a custom checkout engine. We've discussed the components, how they interact, what 3-D Secure and tokenization are, and how they can make your life easier. You may still face challenges on your journey, just as we did while creating Eye4Fraud RapidCheckout. If you have any questions or need consultation, please contact us at

Elijah Atamas

CEO, Softcery
© 2023 Softcery. All rights reserved.