Our data format is really very simple. The only real requirement is that there must be a unique ID, and some meta data, for every product.
The Intelligent Reach data feed management platform is a powerful tool that optimises our client’s product data across more than 1,500 channels. When we onboard a new customer to the platform we have two main priorities:
1. To find the most powerful and stable way of integrating clients’ product and performance data into our platform.
2. To onboard the client’s data in the most frictionless way, demanding the least possible effort from the client’s IT team, without compromising the integration.
To answer these requirements, we typically work with the client’s own data and use machine learning, and internal expertise, to map the client's data to our master feed schema in the platform.
Our master feed is already mapped, through templates and API integrations, to more than 1,500 partner channels. So onboarding, and syndication to those channels, really is a relatively simple matter of understanding, importing, merging and testing clients’ data feeds.
As Mark Batson, IR’s Director of Onboarding and Training, puts it:
“The majority of technical questions we receive stem from the fact we don’t impose a rigid API schema. To the client’s technical team this can seem almost too easy. So, a very common question during onboarding would be whether the API supports parents, and variant data structures, or does the client have to break it all out? And our answer is, that they don’t need to worry about that.”
The Intelligent Reach data format is very simple. If you look at the company’s developer documentation, the only requirements are that there must be a unique ID and some metadata about each product. And IR opens that schema up so that anything and everything can go in there. If inventory data is also required, it comprises a price and a quantity per product ID.
The Intelligent Reach platform is genuinely self-serve and designed to be reliable and easy to use. That reliability and ease of use come from the underlying data being as solid and comprehensive as it can be. For this reason, during onboarding, Intelligent Reach’s onboarding and customer success teams get deeply involved. This is called ‘concierge onboarding’ and typically involves three stages across a 15-working day project.
Stage 1. Sourcing the product data.
“We typically say to the customer, please give us everything you have. Better to have it and not need it than to need it and not have it.” Mark Batson, Director of Onboarding and Training.
Intelligent Reach aims to do a product data feed audit, a product demo and the onboarding framing around the same time.
First, there is a low-level feed audit. If the project goes forward, the onboarding team will take that audit and if there is anything obviously wrong with the data, they will feed that back to the customer and offer to fix it in a live demo of the platform.
If the main source feed comes from an e-commerce platform, such as Magento, Shopify or SalesForce, the team starts there. The Intelligent Reach platform has connectors for most e-commerce platforms, so all that’s needed in this scenario is a login (i.e. API credentials). Stepping out of that perfect world, sometimes some of the data is in an e-commerce platform but the rest of it is elsewhere. Or, perhaps, some data is in a PIM and the rest is in an ERP system. In this scenario, the team deploys the Intelligent Reach API, which is designed to work with data from multiple, often bespoke, sources.
Magento 2 - How we Integrate with e-commerce platforms
When onboarding we work closely with new clients to import their data into a master data feed in the Intelligent Reach platform. During that process, we separate data into three categories: "Product Attributes", "Product Inventory" and "Orders"
This allows us to set up a different refresh frequency for each data type:
- Product Inventory - anywhere up to real time but at a greater cadence than attributes
- Product Attributes - once or twice per day, mostly to capture new products and deletions but can run more frequently if needed
- Orders - up to real time
Typically we will build out these connections on our client's behalf, during onboarding. In this example, the client would provide us with Magento 2 credentials and we would integrate Magento to extract the required data to create our own master version of the client's feed:
Parameters
- Magento2BaseUrl: domain of site i.e. https://www.domain.com.au/
- Magento2StoreView: [optional] will be added to URL here "/rest{storeView}V1"
- Magento2AuthSignature: [optional] authorisation algorithm to use - defaults to "HMAC-SHA1", but we have used "HMAC-SHA256"
Authentication Credentials
Authentication can either be by OAuth (recommended) or by username
Oauth:
- Magento2ConsumerKey
- Magento2ConsumerSecret
- Magento2AccessToken
- Magento2TokenSecret
Username:
- Magento2Username
- Magento2Password
Feed Options
- Magento2ProductApiPageSize: [optional] Page size for products. Defaults to 300. Only change if Magento instance is slow.
- Magento2InventoryApiPageSize: [optional] Page size for inventory. Defaults to 300. Only change if Magento instance is slow.
- Magento2GetCategoryPageSize: [optional] Page size for categories. Defaults to 300. Only change if Magento instance is slow.
- Magento2GetProductAttributePageSize: [optional] Page size for product attributes. Defaults to 300. Only change if Magento instance is slow.
Inventory can be pulled by using "V1/stockItems/lowStock" or "V1/inventory/dump-stock-index-data/website/base".
lowStock:
Magento doesn't allow delta updates of inventory – but they have a method that returns all inventory that is below a certain threshold. So we run two azure functions - one runs a few times per day (see recommended timings at the end), and gets all inventory, the other runs frequently , and gets low stock items so will return fewer items.
- Magento2RealTimeInventoryLowStockQtyCeiling: [optional] When calculating low stock in real time, we only get items with stock less than this (so its quicker). Defaults to 10
- Magento2FullInventoryLowStockQtyCeiling: [optional] When calculating low stock in real time, we only get items with stock less than this (so its quicker). Defaults to 10000000
inventory dump:
- UseInventoryDumpStock: true uses the inventory dump method
Order Options
Magento2MarketplaceOrderApiKey: API Key from IR system
Magento2MarketplaceOrderSecretKey: API Secret from IR system
Magento2MarketplaceOrderSubscriptionKey: IR Api Gateway subscription key
FeedNameToCustomerGroupIdMapping: [optional] allows mapping from our feed name to CustomerGroupId in the order. Format is {feed name}::{customer group id};;etc
Magento2CustomerIsGuest: [optional] whether to make all orders guest checkout, or assign against user (via email address)
Magento2DeliveryShippingMethod: [optional] shipping.method to send in order
Magento2InStorePickupShippingMethod: [optiona] shipping.method to send in order if Instore Pickup.
Magento2StandardShippingMethod: [optional] shipping_description to send in order, if not specified it will send the ShippingService in the IR order.
Magento2StoreId: [optional] store_id to send in order.
Magento2StoreName: [optional] store_name to send in order
Magento2OrderStatus: [optional] status to send in order. Defaults to "processing"
Magento2OrderState: [optional] state to send in order. Defaults to "processing"
StoreIdToNameMapping: [optional] Mapping to change storeId from IR order into store name. This is used in click & collect orders and modifies the shipping_description field. Defaults to storeID of not specified. Format is {storeId}::{outputName};;etc
CustomExtensionAttributeMapping: [optional] Values in here are added to entity.extension_attribute to send in order. Format is {key}::{value};;etc
Magento2CurrencyCode: [optional] *_currency_code to send in order. Defaults to AUD
Magento2CreateInvoice: [optional] Whether to create invoice with order. Defaults to false.
Magento2FeedTypeToStoreView: [optional] allows changing storeview based on Feed Name. {feed name}::{store view};;etc. If not specified will use Magento2StoreView if setup.
Magento2AllowOrderRefund: [optional] Whether to process refunds from IR (not all marketplaces support refunds yet). Defaults to true.
Magento2GetOrdersPageSize: [optional] Sets page size to get orders back. Defaults to 300
AddDeliveryNoteAsOrderComment: [optional] Whether to add delivery not as an Order Comment. Defaults to false. Delivery notes are always added to customer_note in IR.
Magento2LogOrders: [optional] whether to log orders to Azure logger - please be aware of GDPR issues around using this. Defaults to false
Magento Access Setup
The client will also need to provide access to the following areas in Magento:
Sales
Catalog
Carts
The Intelligent Reach data schema is very flexible and powerful. If a client looks at the developer documentation, the main requirement is for a unique product ID and the metadata associated with it (such as colour, size, style, and any other data that is available; and if inventory is included (for example, to power marketplaces and BOPIS campaigns) it typically comprises just the price and the current quantity associated with each product ID. The structure itself is owned by the customer. So when the onboarding team debugs or does any kind of work on the data, it's on the customer’s terms. All the data is unpacked and mapped to IR’s master feed format, after which it can be rolled out to 1,500 partner channels. That's the core part of the onboarding process.
The team has a high level of confidence in this data extraction, manipulation and mapping process and uses artificial intelligence and machine learning to constantly drive and improve it.
What are the key Google shopping attributes?
Your product data feed is a file that contains information about your catalogue of products. It is similar in structure to a spreadsheet, with each row representing an individual product, or variant, and each cell defining an attribute of the product (e.g., name, price, size, colour etc).
Before doing anything else you must ensure that your master product feed is in the best possible shape because it is the source and the centre of all your product marketing. A minimum viable master data feed is one where:
- The feed is fully populated with all the key product attributes, especially brand, name, colour, size and product type – see the full list below.
- Words and phrases are normalised around everyday words that people use when they search for products.
The minimum standard in a Google Shopping feed is to populate all key fields with normalised attributes. E.g., yellow is a more common search term than ‘daiquiri’.
Stage Two. Reviewing the data.
“We try to leverage everything useful in the client product feeds. And to check if anything crucial is missing.” Mark Batson, Director of Onboarding and Training.
If the product metadata contains information about fabrics, the company would write an extraction rule that retrieves that data and populates it to the master feed as a material attribute.
If some products don't have a GTIN number, Intelligent Reach would try to get it from another source or add any data shortfalls to a data improvement plan. Often the client has supplementary files that can be added as lookup lists to backfill / add missing information directly into the platform.
On top of that, a data connector can be used to add performance data, because connecting even mediocre product data with real-time performance data allows performance to be optimised straight away. The platform can improve impressions, clicks and ROAS while the client is plugging the gaps in the underlying data, rather than sitting dormant for six months.
None of the onboarding processes tend to involve the client’s engineering staff to any great degree. The day-to-day technical contacts tend to be technical project leads or product managers if there is a role of that sort within the business.
As the project progresses, the client’s engineering team may get involved, perhaps by doing a weekly stand-up and sending questions to the onboarding team. Responses are sent back quickly and the process is usually very light touch.
The majority of the questions typically stem from the lack of a formal API schema, so although that's not a negative, it can seem so open ended that the client’s engineering team wants reassurance. Some typical answers might be:
- When the data is unpacked, it is mapped to the mandatory fields and recommended fields in IR’s own master feed, which ensures everything necessary is in place to connect that data to IR’s 1,500 partner channels.
- If the client has special things they want to do with product ads, for example on Auto Trader, then the IR teams will discuss with them where they can get this additional data and how they will make that work in the feeds.
- Even though the platform is a SaaS system, Intelligent Reach offers ‘concierge onboarding’ to make sure that the integration is the best it can be, right from the start.
Stage Three. Launch Activity
"This is where the client makes the feeds live and starts running campaigns. Again, the onboarding and customer success teams are very active at this stage. reacting to anything and everything that the platform is telling them." Mark Batson, Director of Onboarding and Training.
At this stage, Google might point out that there are some issues with some products in the feed. For example, some of the links might not structured quite correctly. There are some things that you can't evaluate until they’ve gone live. So the team is on hand to make any changes as soon as anything arises. That's usually the biggest part of the project, getting all the test order structures agreed and signed off. And then monitoring them to make sure everything is working perfectly.
Intelligent Reach is not just testing shopping sites, at this stage, it’s supporting every kind of marketplace, and every possible thing a marketplace might want to do. There's a lot of testing that has to happen to ratify the order integration, and synching, across platforms, is working correctly.
That's a huge part of the process but it’s essential, and invaluable because what the teams are testing at this stage will cover every marketplace. So clients can roll out to any, and every, channel they like in future.
“Unbelievable onboarding from the IR team; smooth, reassuring and amazing communication”
Hannah Bennett – Head of digital, Paul Smith
Products not showing on Google Shopping or other important sales channels?
Our platform identifies and eliminates product data issues! Book your free product data feed audit now.
Book Now