Dsco Platform Standard API Integration Guide for Retailers & Suppliers

Transparency & Other Guides

This is a combined retailer/supplier guide intentionally so both parties transparently understand each other's integration requirements and responsibilities as success in e-commerce transactions is a two-party dance.

Overview

This guide explains how a partner, a retailer or supplier, would use Rithum's Dsco Platform APIs to integrate to Dsco whether doing a dropship integration or a marketplace integration.

Best Practices

Use API over file integration

CommerceHub strongly recommends partners use these APIs as opposed to CSV/EDI files as the basis of their integration if the partner generates a lot of orders, retailers, or fulfills a lot of orders, suppliers.

While a file-based integration will work for partners who play at truly massive scale, and a few partners have chosen to use the file approach, the API approach provides a much more featured, rich and highly-performant approach. Most features are available to file-integrated partners but some features simply don't lend themselves well to be represented in a two-dimensional file structure. Also, the Dsco platform focuses on wringing as much latency as possible out of the sharing of information between you and your trading partners to help you compete and win in the marketplace with the best possible experience for your end consumers. APIs facilitate a better overall experience for your customers on the whole than do file integrations.

It's OK to mix integration approaches

Some partners choose a file-based integration simply because they have done file-based integrations in the past and it feels safer and faster to get going.

The Dsco platform allows a partner to use a file integration for some workflows, getting shipment updates for example, while one at a time migrating to an API integration for other workflows, creating orders via API perhaps.

This then allows a partner to get launched quickly and then gradually migrate to the more featured and performant API approach.

Pay attention to limits

Each API describes any limits imposed, such as how many times a second the API may be called or the max size in bytes of the request payload.

Use batch APIs for batch use cases

Batch APIs are more work for you the integrator. However, the batch APIs exist for a reason: to provide a robust solution at scale for batch use cases, cases where a lot of orders or inventory or shipments need to be sent to Dsco all at once. In these cases, please use the batch APIs. While it will be more work up front, you will be glad you did in the long run.

Use Streams

Whenever possible use streams.

Error Handling

You should implement an exponential backoff-retry algorithm to retry API calls sent to Dsco that fail when the HTTP Response Type indicates the error is a transient error, or one that will go away with time (see list of HTTP codes below).

The Dsco platform recommends the backoff-retry algorithm retry for up to two hours after which the partner should send a request to Rithum's Dsco platform support (support@dsco.io) to ask whether there is an issue with the Dsco APIs. In the event that this occurs, please send the complete API request made and the HTTP error code that is being seen by the partner in the support request.

Here are some resources describing the algorithms we recommend a partner implement...

Foundational Concepts

The APIs Change

Dsco will not break backward compatibility on an API when making changes. Dsco will add new APIs and new optional attributes to existing APIs. Partners should ensure their validators do not break when Dsco adds new APIs or adds attributes to existing APIs.

Authentication

Retailer & Supplier

Retailers are demand channels. Orders originate with retailers. Suppliers fulfill demand by shipping orders. Dsco exists to pair supply with demand intelligently. There are a number of different words that are largely synonymous.

Object or Data Object

The fundamental data objects that flow between a partner and Dsco and Dsco and a partner's own trading partners.

Object Actor Description
Order Created by a retailer A consumer order a retailer sends into Dsco which is then shared with a one or more trading partners
Shipment Created by a retailer A tracking number with the specific order line item(s) packaged in the shipment (aka package)
Item Created by a supplier A single variant of a product with both inventory and catalog attributes
Item Inventory Created by a supplier The item attributes that describe how much of this item is or will be available and where in the world it is located (warehouse)
Item Catalog Created by a supplier, optionally enhanced by a retailer The item attributes that describe the item's physical attributes and categorization such as category, description, image, size, etc.
Warehouse Created by a supplier, optionally with codes mapped by a retailer A physical location that houses items
Invoice Created by a supplier A request for payment based on an accounting of what was shipped
Return Created by either a retailer or supplier Represents the intention for some part of an order to be returned with an accounting of what was actually received
Cancel Created by a supplier Indicates part or all units of an order will not be fulfilled by the supplier
Assortment Created by either a retailer or supplier A named collection of items. Retailers use them to drive automation such as which items are ready for automatic listing or which are not yet fully compliant and so forth. Suppliers use them to decide which items to share with which of the retailers they work with.
Dynamic Assortment Created by a retailer An assortment that uses retailer-defined rules to automatically include or exclude items from a named assortment.

Direction

Indicates who is sending an Object to whom.

Workflow

A workflow is an object with a direction. order-in represents an order being sent into Dsco by a partner. order-out is a workflow where Dsco is sending an order out to a partner. inventory-in represents a supplier sending an inventory update into Dsco. inventory-out represents Dsco sending inventory updates to a retailer received from the retailer's trading partner(s).

Synchronous vs Batch APIs

Let's say you want to create an order or update inventory. You have a choice. Do you want to create each order one at/update item inventory one at a time with a separate HTTP request for each object or do you want to send a group of objects all at once, which will be processed asynchronously? Dsco provides two types of async APIs:

Async APIs require that you the partner call the API and then later check back to see what worked and what didn't. Perhaps you call a batch API to create 10 orders in a single request. You will later have to call back into the corresponding Change Log API to determine if the 10 orders were created successfully. A pain, we know, but there is a significant benefit. If you are willing to use our batch APIs, Small Batch and Large Batch, with the annoying need to check back into make sure everything worked, you will effectively work around the API limits as Dsco will be doing a lot more work on your behalf all at once. If you have a batch use case, meaning you generate a lot of orders all at once as a retailer or a lot of inventory updates as a supplier, and you elect to send them one at a time, then there is a likelihood that you are going to be throttled and have to accommodate this fact. The choice is yours but remember when you choose the path of least resistance because it's easier even if it isn't the right pattern for what you are actually doing there are usually consequences.

Dynamic Assortment

Retailers may create dynamic assortments which are named groups of items whose membership in the assortment is automatically determined in near real-time by the rules specified by the retailer on the dynamic assortment. Dsco's inventory and catalog streams may be filtered by a given assortment using assortmentId to only include those items that are members of an assortment. Then a retailer may retrieve the data events of a given assortment-driven stream to cause different automatic and manual behaviors.

Here are examples of some dynamic assortments created by retailers to date to illustrate how the feature is used.

Ready to Auto List Assortment

Retailers will create catalog/inventory streams that are only populated with items that are completely ready to be automatically listed to the retailer's website but aren't yet listed. Retailers may include rules such as...

Ready for Manual Curation Assortment

Retailers will create catalog/inventory streams that are populated with items that are completely ready to be listed but a retailer merchandiser is meant to manually curate these items to determine if they will be listed or not. Retailers may include rules such as...

Country-based Assortment

Retailers will create catalog/inventory streams that are populated with items that may be sold in different countries. So, perhaps an inventory stream for items that may be sold in North America and another for the UK and another for Germany. The rule would look at the locationsCanBeSold catalog attribute to determine whether to include or exclude an item from a stream.

Streams

Backstory

Retailers and suppliers need to get data from Dsco. Retailers need to get shipments, item inventory item catalog data, invoices and suppliers need to get orders, returns, etc. Ideally, Dsco would push new orders or shipments or updated inventory to the retailer or the supplier as the case may be. Some time ago, Dsco provided webhooks to do just that. Dsco no longer provides webhooks. They are too flimsy. It is not acceptable that even a single webhook is not received and processed successfully by a partner. Dsco knows that a partner received a data event but not that it was processed successfully. Only the partner knows that. Alos, even with robust retries, there were times when a webhook event did not get processed by a partner, resulting in late orders, undersells and oversells on rare occasions. Dsco eventually created an API that a partner could call every hour to check for any webhooks that were rejected by the partner's infrastructure. What's the point of "pushing" if the partner has to call into get these missed data events; rare though they were the fact that it could happen meant partners had to implement the API to check if some didn't get through. At some point, all involved decided there must be a better way.

Fundamentals

Streams is a more modern, more robust way to "push" data to a partner. Here's how it works.

You create a stream using the Create Stream API in which you designate which data objects you want automatically populated in the stream, including query criteria specific to the type of object you selected. Then, Dsco will populate your stream with any new or updated object of the given type that matches the query criteria. You then call an API to get events in the stream.

All streams are created initially with a single partition, from which you read events in your stream. A partition is a sub-stream in your stream, providing additional read scale for you. Dsco ensures that like data events always go to the same partition. For example, all inventory updates for a given SKU will always go to the same partition, maintaining partial ordering of data events by item. Thus, partners can spin up independent copies of the same app that read from different partitions simultaneously to achieve whatever read scale is needed to ensure the fast processing of data events from the stream.

How is this Dsco pushing me events?

Consider that cell phones use an approach known as micro-polling under certain circumstances to become aware of push notifications meant for your phone. Micro-polling is a technique to perform near constant, light-weight queries using an API to determine if there is work to be done. A stream contains the data you need to be aware of. You will write an application that is constantly running and micro-polling a stream partition for any new events as soon as they are there and pulling them down.

How do I know there are events to be retrieved?

Each stream maintains a checkpoint which is the last position in the stream that you have confirmed that you read data events up to and successfully processed the data events for. When your app starts up it will call the List Streams API to get the stream definition of your stream which includes the position for each stream partition. Then, you will call the Get Stream Events from Position API to micro-poll for events in that stream partition. As you get an event and successfully process it, you will call the Update Stream Position API to tell the stream where you have successfully read and processed data events up to.

How is this more robust than webhooks?

Only you, the partner, know that you both received and processed a data event successfully. Some very small percent of Webhooks, even with robust retries, failed to get through to a partner for whatever reason. Also, there were occasions when a partner would receive the event just fine, but then some other back office system in the partner's infrastructure would be down or having problems and the event would not get processed successfully. The partner would then have to reach out to Dsco support to have a webhook resent, which took time to find and resend. Some partners went ahead and implemented a tool for their internal teams to use that would call a Dsco API to pull down order or inventory updates that were not processed successfully by the partner.

Streams are better. The events in each stream partition are guaranteed to persist for up to 90 days. That means the partner may call the Get Stream Events in Range API to retrieve events from position A to position B. A position is just a timestamp. So, if a partner's infrastructure was having issues yesterday for 30 minutes, the partner could easily spin up an app instance to pull down all data events in that 30 minute period and reprocess them in order. The partner doesn't have to call anyone. The partner is in the driver's seat for failure scenarios that don't result in a sidecar process since each stream partition is essentially a persistent database that lasts for 90 days.

Sample Java App

Dsco has created a github repository that is a Java sample app that authenticates to the APIs and demonstrates how to create a streaming app using the Dsco Streams APIs soup to nuts.

Partition Best Practices

It is uncommon for you to need more than a few partitions per stream in order to keep up with reading the volume of data events you receive with very little delay. Some of our largest retailers are able to read all data events and process them with very little delay while receiving millions and millions of data events a day with only 8 partitions and even then they only need this many partitions when receiving millions and millions of inventory update events a day. Our streams are robust and performant. You can create additional partitions after you go live if you find you aren't reading all the events fast enough to keep the delay between when a data event arrives in your stream and when you can pull it down. Of course, it isn't a good idea to constantly be creating and removing partitions since this involves the Dsco system re-partitioning events across your partitions which introduces some temporary delay. However, the fact that Dsco supports this means you can learn what your real production throughput requirements are as part of your initial launch preparation and increase the number of partitions on the fly as you see real traffic. Feel free to reach out to support@dsco.io if you seek further guidance on how many partitions to use.

Stream Operations

Sync Stream Operation

Partners may call the Create Stream Operation API with an operationType set to sync to cause Dsco to go query all objects in its database that match the stream's query params and act as if every single object changed so the stream is populated with an event for every object. This can be useful when a partner wants to true up all data of a given object type. This operation is intelligent in that it inserts all the objects into the stream all at once, pausing updates until done. This can take some time if the amount of data to be placed in the stream as a result of the sync operation is very large.

Stream data events that were placed in the stream as a result of this operation will have the source attribute set to the value sync.

Set Partition Owner Stream Operation

Partners may call the Create Stream Operation API with an operationType set to setpartitionowner to change the partition owner value associated with a stream partition. A partition owner is just an arbitrary string ID that a partner wants to associate with a stream partition. Some partners use this in containerized environments to associate an arbitrary string value with a partition so on container startup in a say a kubernetes or docker cluster a given container will know which partition it is meant to be associated with.

Repartition Stream Operation

Partners may call the Create Stream Operation API with an operationType set to repartition to kick off a repartition. This operation expects that you have used the Update Stream API previously to change the number of partitions and are initiating this stream operation to cause the changed partitions to be enforced, rebalancing objects across the remaining partitions.

Event Types

Go to the Get Stream Events from Position API and click on the 200 response to see the definition of events that are placed in streams. Note that the payload attribute type differs based on the stream type. Also note that the eventReasons attribute, describing why this event was placed in the stream, may be set differently for different stream types.

Retailer API Integration Structure

Retailers need to send into Dsco consumer orders. They need to be able to get from their trading partners item catalog and inventory updates, shipments, cancels, invoices and returns. Retailers need to be able to send into Dsco item data to enrich and override item catalog definitions. Retailers need to use assortments (manual and Dynamic) that are used to group items based on the retailer's rules that model their business goals and objectives.

API-In Workflows

Workflow Sync API Batch APIs Description
order-in Create Order Small Batch | Large Batch Retailers send new consumer orders into Dsco. Dsco then shares the order with the appropriate trading partner(s) of the Retailer.
return-in Create Return Small Batch | Large Batch A retailer may initiate a return, meaning tell the supplier that a return is expected, by calling this API to send in the return to Dsco.
complete-return-in Complete Return Small Batch | Large Batch A retailer may respond to a supplier-initiated return, marking the return as complete, by calling this API to send the fact of the completed return into Dsco.
catalog-in - Update Catalog Attribution Large Batch A retailer using the Advanced Catalog product may define its catalog category taxonomy, with required and optional attributes at each category, using this API.
channel-override-in - Upate Channel Overrides Small Batch | Large Batch A retailer using the Advanced Catalog product may define item-level overrides to augment and enrich item data before Dsco shares these items with the retailer. This allows a retailer to both override individual content attributes with its own values as well as accommodate a retailer's own back office listing requirements, such as perhaps Partner SKU, by ensuring Dsco has all data the retailer requires for a SKU to be listed on the item itself. Retailers will often create Dynamic Assortments that use retailer-provided rules to include or exclude items from the assortment, allowing automation to take items in a Dynamic Assortment and cause them to be listed to a retailer's website only when completely ready.
cancel-in Request Order Cancel - A retailer sends into Dsco a request to cancel an order.

Stream Out Workflows

Assortment Query Params

The inventory and catalog streams support the assortmentId query param, meaning only include events for items that are a member of the assortment specified. Since retailers may create Dynamic Assortments

inventory-out Workflow

API: see Streams Section
Object Type: item inventory\

Get item inventory data from Dsco.

All inventory updates from all of the retailer's trading partners that match the inventory stream query params will result in an event being placed in the stream. Also, a Sync Stream Operation may cause events to flow into this stream.

Use the assortmentId query param to filter to only those items that are a member of the manual or Dynamic Assortment.

Use the clearQuantityForStoppedItems query param to cause Dsco to zero out inventory quantity for items that a retailer has stopped.

Use the includeUnchangedItems query param to include inventory and catalog updates from the retailer's trading partners even if an inventory update didn't result in any change to any inventory attributes.

Use the quantityChangeOnly query param to only include inventory updates that actually changed the inventory quantity available attribute.

catalog-out Workflow

API: see Streams Section
Object Type: item catalog\

Get item catalog data from Dsco.

This stream is only available for retailers using the Advanced Catalog product.

All catalog updates from all of the retailer's trading partners that match the catalog stream query params will result in an event being placed in the stream. Also, a Sync Stream Operation may cause events to flow into this stream.

Use the assortmentId query param to filter to only those items that are a member of the manual or Dynamic Assortment.

Use the compliancePercent query param to filter objects to only include catalog items that have a status of not_compliant and that match the provided compliance percentage.

Use the complianceStatus query param to filter objects to only include catalog items that have the given status.

Use the isListed query param to filter objects to only include catalog items that are or are not listed on the retailer's website.

invoice-out Workflow

API: see Streams Section
Object Type: invoice\

Get invoices from Dsco.

All invoices created or updated by any of the retailer's trading partners that match the invoice stream query params will result in an event being placed in the stream. Also, a Sync Stream Operation may cause events to flow into this stream.

Use the eventTypes query param to filter to only new invoices or just updated invoices.

orderitemchange-out Workflow

API: see Streams Section
Object Type: orderitemchange\

Get newly shipped or canceled order line items.

This stream will be populated by any order line items that change to become shipped or canceled for any of a retailer's orders.

Use the statuses query param to filter to only newly shipped or only newly canceled order line items.

return-out Workflow

API: see Streams Section
Object Type: return\

Get newly created or updated returns.

This stream will be populated by any return that is created or modified that is associated with a retailer.

Use the statuses query param to filter to returns in a given status: created, acknowledged or completed.

Supplier API Integration Structure

Suppliers need to get from Dsco orders they are meant to fulfill. They need to be able to send into Dsco item catalog and inventory updates, shipments, cancels, invoices and returns.

API-In Workflows

Workflow Sync API Batch APIs Description
catalog-in - Update Catalog Small Batch | Large Batch For suppliers working with retailers using the Advanced Catalog product, the supplier sends new/updated item catalog data into Dsco. Dsco then validates catalog data and shares validation result and the catalog data with the appropriate trading partner(s) of the Supplier.
inventory-in Update Single Inventory Small Batch | Large Batch The supplier sends new/updated item inventory data into Dsco. Dsco shares the inventory data with the appropriate trading partner(s) of the Supplier.
shipment-in Create Shipment Small Batch The supplier sends order shipments into Dsco. Dsco shares the shipments with the appropriate trading partner of the Supplier.
cancel-in Cancel Order Item Small Batch The supplier sends cancellations into Dsco. Dsco shares the cancels with the appropriate trading partner of the Supplier.
acknowledge-order-in Acknowledge Orders - The supplier sends order acknowledgments into Dsco to indicate an intention to fulfill the order.
return-in Create Return Small Batch | Large Batch A supplier may initiate a return, meaning tell the retailer that a return is expected, by calling this API to send in the return to Dsco. This may happen when an order is rejected at the door by an end consumer and the shipment is returned to sender.
complete-return-in Complete Return Small Batch | Large Batch A supplier may respond to a retailer-initiated return, marking the return as complete, by calling this API to send the fact of the completed return into Dsco.
invoice-in Create Invoice Small Batch A supplier sends invoices into Dsco to get paid for order units shipped. Dsco shares invoices with the supplier's correct trading partner.

Api-Out Workflows

Workflow API Description
order-out Get Order Suppliers may use this synchronous API to get a single order they are meant to fulfill out of Dcsco.
order-out Get Orders Suppliers may use this synchronous API to get orders they are meant to fulfill out of Dcsco.
return-out Get Returns Suppliers may use this synchronous API to get returns associated with the supplier.

Stream-out Workflows

order-out Workflow

API: see Streams Section
Object Type: order\

A supplier uses this stream to get newly created orders from Dsco.

Each time an order is created/updated Dsco will populate this stream with any order from all of the supplier's trading partners shared with the supplier to be fulfilled that match the order stream query params. Also, a Sync Stream Operation may cause events to flow into this stream.

return-out Workflow

API: see Streams Section
Object Type: return\

Get newly created or updated returns.

This stream will be populated by any return that is created or modified that is associated with a supplier.

Use the statuses query param to filter to returns in a given status: created, acknowledged or completed.

© 2024 Rithum Holdings, Inc., together with its subsidiaries, all rights reserved, protected under U.S. and international copyright law. Rithum and the Rithum logo are trademarks of Rithum, LLC