---

What is FeedLogic?

FeedLogic is a technology used to process disparate data feeds into a common, unified format.

FeedLogic is ideally suited for use in price comparison projects. By unifying data feed content provided by multiple retailers into a single unified feed, comparison of complex product or service sectors becomes trivial.

Standalone use, or integrate into custom projects

FeedLogic is a data feed processing framework, written in the Microsoft™ .NET language.

The framework can easily be integrated into custom projects by consuming the FeedLogic code.

For generic projects, a standalone Windows application is used to manage the processing of feeds.

Multi-format feed processing

The FeedLogic framework uses objects known as "data processors" to read each feed to be unified. The framework includes data processors for the most common data feed formats:

• XML
• CSV (Comma Separated Values). The most common separator characters are supported - comma, tab, or pipe.
• Microsoft Excel™ Worksheets

FeedLogic reads both locally and remotely stored data feeds. If a data feed is stored remotely at a web or ftp server, then FeedLogic downloads the feed prior to processing it. Downloaded data feeds can be cached for a period of time if necessary to prevent repeated downloads. This is especially useful during development.

FeedLogic supports feeds that are archived using ZIP or GZIP compression, and are decompressed automatically.

FeedLogic also includes two additional data processors that offer a unique feed reading capability:

• Amazon ECS Data Processor - interact with the Amazon product catalogue to perform custom searches of products.
• Robot Data Processor - interact with the Kapow Robosuite™ webscraping platform (available separately). Now you can treat any website as a data feed. Even if a retailer does not offer a data feed, you can construct your own based upon the data on their website!

The modular nature of the framework means new data processors can easily be developed.

Intuitive processing of data feeds

FeedLogic features a unique logical approach to processing data feeds.

First, the unified feed format is defined. This is a list of field names, data-types, and other related information that describe the unified format.

A data processor is then created for each feed that is to be processed. Each data processor has a number of actions defined. During processing, FeedLogic applies these actions to every data element within the feed. The defined actions are used to transform the raw data in the feed to the unified format.

FeedLogic supports a wide range of actions:

• Basic value extraction.
• RegEx extraction, allowing a regular expression to be applied to feed data to isolate specific data. Full support for named RegEx groups.
• Transformation of feed values - Trim, Replace, RegEx Replace, Strip Bad Characters, Pre/postpend additional text, and more!
• Manually setting values in the unified output.
• Validation of feed data.
• Conditional IF/ELSE logic (if the feed contains this data, then perform this set of actions, otherwise perform another set of actions).
• Looping (loop through repeating blocks of XML).
• Comparison of feed data with other data extracted from the same feed block.
• Skipping the current feed element.

When using the FeedLogic application, the definition of these actions is performed using a simple graphical interface. Actions can be cut/pasted, imported/exported, and even moved around the action tree.

This unique approach means that, should the format of a feed change, modifying the actions of the data processor becomes trivial.

Multi-threaded for Performance

FeedLogic allows multiple data processors to run concurrently on a single computer. This maximises the number of feed elements per hour that can be processed. On a relatively modern PC (2GHZ Core2Duo), typically around 5,000,000 to 7,000,000 feed elements can be processed per hour^*.

_{* The architecture of the application integrating FeedLogic, and the complexity of the actions being applied to the feed may affect these benchmarks.}

'Clustering' Supports Unlimited Capacity

FeedLogic includes clustering functionality that allows the processing of feeds to be performed on multiple remote computers. By installing a simple Windows Service, any PC instantly becomes capable of processing feeds, while being controlled and monitored remotely from the main FeedLogic application.

By spreading the workload across multiple computers, an unlimited number of feed elements can be processed. A cluster of 4 relatively inexpensive computers can typically process around 20,000,000 to 28,000,000 feed elements per hour^*.

Need for more capacity? Simply add more computers to the cluster!

Efficient Processing

FeedLogic automatically detects elements of the data feed that have not changed, so that unnecessary procesing can be skipped for these items.

If a feed only has a handful of new elements added per day, then it is possible for the new feed elements to be fully processed, but the old elements to be skipped.

Integrated Feed Analysis

The FeedLogic framework includes a set of feed analysis tools. These tools make it easy to analyse data feeds in order to quickly create processing actions.

• Identify all unique values for each field in a data feed. It is quite common in a feed with thousands of feed elements for each field to have only a small number of unique values.
• RegEx testing suite.