How to Create a Custom Checkout Engine from Scratch

The checkout page is the final step of every online purchase, where customers enter their payment and shipping information. Though the process may appear straightforward, much work goes into creating a secure, user-friendly checkout page.

This article will help you determine whether you need a custom checkout engine and how to create one.

Do I need a custom checkout engine?

There may be many reasons to create a custom checkout. You could use it to sell online, integrate it with your SaaS, or create a game-changing product. Before we dive into creating a custom checkout, let's consider when you'll need to build it.

As a business, you want to keep things as simple as possible. Therefore, building a custom checkout engine may be unreasonable if existing solutions solve your problems.

For example, if you want to start selling online and don't have any unique requirements, the best way is to use platforms like Shopify, BigCommerce, or Magento. These have a built-in checkout page with features like Apple and Google Pay, tipping, or saving information for the next checkout session.

If you're building a custom SaaS application or platform, various checkout solutions are also available:

• One popular example is Stripe Checkout, which allows you to accept payments for one-time purchases or subscriptions with a few lines of code.
• Another is Checkout.com, which provides an API for processing payments, calculating fees, and managing subscriptions.

However, if you have a unique business model or product requirements and can't use any existing checkout solutions, don't worry. This article will guide you through everything you need to successfully build a fully-fledged checkout engine and start accepting payments online.

How the checkout works

All right, you've decided to use a custom checkout. But how does it work? What system powers the page with a few text fields? There's no single answer—you can customize it to your needs, adding components such as 3-D Secure or third-party integrations. However, some core components are essential for any checkout system. In the following steps, we'll examine them.

What components does the basic checkout engine include?

• High-level components:
  • A form for customers to fill in their payment and delivery information.
  • An API, the core of the checkout engine.
  • A platform for managing customer or store data.
• Payment components: 
  • A payment gateway—a service that processes payment information.
  • A payment processor—a service that processes transactions.
• Security components:
  • A 3-D Secure system.
  • A tokenization system.

High-level components

Let's take a closer look at the high-level components.

A form for customers to fill in their payment and delivery information

Depending on your product, this could be a web page or a mobile/laptop app screen. The form should be simple and convenient to use, so customers won’t struggle to fill it in, and you won’t lose them. You can find best practices for designing a checkout form here.

An API, the core of the checkout engine

The API is responsible for all internal operations, such as money transactions and order creation. The internal components we’ll cover in this article communicate directly with your API.

A platform for managing customer or store data

This platform allows you or other merchants to manage customers and their purchases. You can create a platform for managing data or integrate an existing one. We’ll discuss this further.

Meet the payment gateway

Suppose you have a checkout form where the customer fills in their payment and delivery information. What comes next? How do you process this information?

The first API component is the payment gateway. It encrypts the customer's payment and delivery information and sends it to the payment processor. When the customer fills out the form, their data is sent to the payment gateway through the API.

There are many payment gateways to choose from, such as Authorize.Net, USAePay, and others. Pick the one that best suits your needs.

Payment processor

The payment gateway sends the payment information to the payment processor, which relays transactions between the customer's and seller's banks. For example, when the customer's money is transferred to the seller's bank account, the payment processor handles this operation.

You don't need to worry about the payment processor; the payment gateway typically handles all its operations, so you don't need to select it or monitor its work manually.

A platform for managing data—existing or custom?

Suppose you've chosen and integrated a payment gateway (which already contains a payment processor). Your checkout system works like this:

1. Customers fill in their payment and shipping information on the checkout form.
2. Then, your payment gateway creates the transaction using a payment processor.
3. A certain amount of money is transferred from the customer's bank account to the seller's.

You also need to manage your customers: view the delivery address for the purchase, make a refund if necessary, view purchase analytics, etc. If you're selling goods online, you also need to manage the products (add, edit, or remove them from your store).

So, you need a system to manage everything that happens before and after checkout. You can build one, but it would be a lot of work. The simplest (and best) option is to use one of the existing platforms and integrate your custom checkout. We recommend Shopify or BigCommerce, as they are robust, have many useful features, and are trusted by thousands of merchants worldwide

If you're building a custom checkout for other merchants, another advantage of using an existing platform is that the more merchants already use it, the more likely they are to try your custom checkout that integrates with their platform.

Integrating checkout with the platform

Once you've chosen the platform, you need to integrate your checkout system there. For example, once the order is created and processed through the payment gateway, you need to create it on the platform so you or other merchants can view it.

Similarly, if the order is refunded on the platform, you need to use the same payment gateway to refund it on the API side. This is a two-way integration.

The way integration works depends on the platform. In most cases, your custom checkout must be installed as an app to access the merchants' stores.

The basic checkout engine is ready. You can see it illustrated above. It includes a form for customers to fill out their payment and delivery information, a payment gateway to process transactions (using a payment processor), and a platform to manage customers, purchases, and products.

You can choose components according to your needs and integrate your engine with multiple ones, like multiple payment gateways and platforms.

You may need components like 3-D Secure, tokenization, or third-party integrations for a more extended checkout engine. In the following steps, we'll discuss how to implement them.

Third-party integrations

Integrating third-party services, such as analytics, with your engine can be done on the API side, connecting them like any other component. Alternatively, they can be integrated with a platform of your choice.

While the second option may appear easier, it can have unexpected consequences. For example, if you're creating a checkout for other merchants integrated with an existing platform, a merchant may install a third-party app that affects the checkout. This app may work well with the default platform's checkout, but your custom checkout engine may not recognize it.

Therefore, you should consider all third-party apps that can be installed on the platform and adjust your checkout logic accordingly.

The illustration below outlines two ways to integrate third-party apps:

1. Connecting them to your API.
2. Connecting them to the platform.

3-D Secure

3-D Secure (or 3DS) is a protocol designed to add an extra layer of security to online transactions. After customers enter their valid payment information, they must confirm the transaction. For example, their bank may request confirmation by sending a security code or a Push notification.

Another reason to use 3-D Secure is that once the customer confirms the transaction, their bank takes responsibility for it. This is called a liability shift. In the event of fraud, the customer's bank will compensate them, not your checkout system.

Fortunately, you don't need to implement 3DS manually. Services like PAAY provide 3DS functionality that you can integrate with. The transaction must be verified with 3-D Secure before authorization is made in the payment gateway:

Tokenization

Introduction

The diagram above illustrates how customer payment information, such as card number, expiration date, and CVC, is sent to the payment gateway through your API. According to the Payment Card Industry Data Security Standard (PCI DSS), any system that processes or transmits payment data must pass the PCI DSS compliance validation. This involves meeting 12 requirements, which international payment systems regularly review.

In other words, if your system interacts with a customer's payment information even once—regardless of where (on a web page or server-side) or how it is processed—you must pass the PCI DSS compliance. This makes things more complicated.

Tokenization is a useful solution here. It helps your system not interact with payment information and not even see it. During tokenization, a service that takes responsibility for the customer's payment info (and is PCI DSS compliant) exchanges sensitive data with a non-sensitive token. This token contains no information and cannot be converted to sensitive data, so your system can use it without PCI DSS Compliance validation.

Implementation

The illustration above presents a problem: the checkout cannot pass payment information to the tokenization service without adhering to the PCI DSS. Two additional steps are necessary to implement tokenization.

Let's use the TokenEx tokenization provider as an example. The checkout form, where customers input their payment information, communicates directly with TokenEx. Sensitive data is exchanged using a token, which is then passed to the API.

In the API, operations requiring payment information (e.g., authorizing transactions with payment gateways) are processed through the TokenEx proxy. This proxy exchanges the token with payment information before the request is sent, so our API is unaware of the payment information, but the payment gateway is.

Conclusion

With tokenization and 3-D Secure, the checkout workflow is as follows:

1. The customer fills out the checkout form connected to the tokenization service, and their payment information is replaced with a non-sensitive token.
2. The token is sent to the 3-D Secure service, which requests additional transaction confirmation.
3. After the user confirms the transaction (passing 3-D Secure), the token and delivery information are sent to the payment gateway through the tokenization proxy. The proxy then replaces the token with payment information.
4. The payment gateway encrypts and transmits the payment information to the payment processor.
5. The payment processor performs the transaction (for example, transferring money from the customer's bank account to the merchant's).
6. The engine creates an order on the platform, allowing you or your merchant to interact with it.
7. When the merchant takes any action on the platform (e.g., refunds an order), the platform sends a webhook (event) to your API.
8. Your API then refunds the appropriate transaction using the same payment gateway. It notifies the platform that the operation is complete (so the merchant can see that the order was refunded on the platform).

This article has covered the basics of creating a custom checkout engine. We've discussed the components, how they interact, 3-D Secure and tokenization, and how they can make your life easier. However, just as we did while creating Eye4Fraud RapidCheckout, you may still face challenges on your journey. If you have any questions or need consultation, please contact us at hi@softcery.com.