1. an event associated with an entity comes in, say a user (the entity) visits a page (the event) on a web site
2. the features associated with the entity are retrieved, say features that describe the user and his or her behavior at the site, and the features are updated with the event
3. the features are used as the input to a predictive model, which produces a score, say the likelihood that the user will click on an ad in a certain category
4. an action is taken on the basis of the score, say to display an ad in a certain category if the score is above a threshold.

This may be easier to remember if you use the acronym EEFM for Event, Entity, Features, and Model. If the events are all available at the same time, the feature vectors for all the entities can be created at the same time and the features can all be scored “in batch”.

At Open Data Group, we used the Predictive Model Markup Language (PMML) to express our models in XML so that models may be built in a development environment with one application and then easily deployed in a production environment with another application. We also use an open source scoring engine (Augustus) to deploy our models in operational environments.

In practice, when actually deploying models, it is usually a bit more complicated.

First, various business rules are usually used to process the event prior to using the associated state as input to the predictive model (pre-processing). Second, the score of the model is usually processed by additional business rules (post-processing) prior to selecting an action. For example, if the event is associated with the visit of someone who is likely to be under 18, the site may choose different types of ads as part of the pre-processing. Second, as part of the post-processing, inventory rules and rules about how often to show ads (exposure rules) may be used to exclude certain ads.

Until recently, pre- and post-processing usually needed to be coded manually and couldn’t be expressed easily in PMML.

That has changed with Augustus 0.5.2. With this version of Augustus, Python code can be embedded in the PMML file to express pre- and post-processing rules, as well as to combine multiple models to produce scores, and to process multiple models and scores in a variety of different ways.

We call this augmented processing and in our experience over the past several months it has significantly simplified the deployment of predictive models into operational systems.

Here is an example from a white paper that we are writing about augmented processing using Augustus. With segmented modeling, you can use multiple models in different segments to score an event and in this way produce multiple scores. Assume that you want to use the minimum score produced in this way as the score for the event. Here is some Augustus code to do this that can be embedded in the PMML file for the segmented model:

def action():
    segmentScores = []
    for segment in segments:
        segmentScores.append(segment.score()[PREDICTEDVALUE])

    if len(segmentScores) == 0:
        finalScore = MISSING
    else:
        finalScore = min(segmentScores)

    output.xmlopen(“Event”, attrib={“number”: eventNumber})
    output.xmlfield(“Score”, finalScore)

Augustus 0.5.x was a rewrite of the PMML-compliant scoring engine with two important changes:

1. Augustus now has the ability to build models in batch mode using static data files as usual, but also in a streaming mode in which data is read only once.
2. Augustus now supports custom processing, a mechanism that allows the user to embed arbitrary Python code inside of Augustus. This provides a simple mechanism, for example, to combine multiple models and produce a single score.

Augustus 0.5.x uses the latest PMML specification 4.1 from December of 2011.
The latest version, Augustus 0.5.2.0, was released on April 4, 2012. Its model coverage is: Tree, Regression, Baseline, Rule Set, Naïve-Bayes, and Cluster.

AdReady is an Open Data Group Business Partner and is using the latest Augustus for the development and deployment of sophisticated statistical models for use in a large, robust, and scalable targeted ad-bidding system. AdReady’s Real-Time-Bidding (RTB) enables advertisers to bid more effectively on precise inventory and audience targets. Bid requests need to be collected, scored against segmented models, and responded to in less than 100 ms.

By using Augustus, AdReady can grow using elastic capacity and score on VM instances, bypassing the normal (Augustus) data pipeline and send events directly into the scoring engine processing loop from their Python MVC web framework (Django over Apache). AdReady described their use of Augustus in a nice blog post today.

To learn more about building a scalable analytics platform using Augustus, please contact us at info at opendatagroup.com.

The fastest way forward is to use a public cloud, external experts, and to do some quick experiments and prototyping. At this point, for many companies, there is a problem. It is quite common these days for companies to have policies that prohibit placing proprietary data, or data that contains information that can identify customers, on public clouds. Providing access to this data to third parties is also usually quite difficult.

One practical approach is to replace actual data with simulated data, and, instead of using public clouds, to use instead private clouds operated by third parties. This requires using data simulators that produce realistic data. For example, large data is rarely normally distributed, but more often follows power laws or similar types of distributions.

As a reminder, a private cloud is a cloud that is used exclusively by a single organization. It may be managed by the organization or by a third party; and, it may exist on premise (an in-house private cloud) or off premise (a third-party private cloud). In contrast, in a public cloud, the cloud infrastructure is made available to the general public, or a large group, and is owned by an organization selling cloud services (a cloud service provider). In this post, we assume that private third party clouds are also single tenant clouds; that is, only one client’s data is on the cloud at a time and the cloud is sanitized between use by different clients.

In more detail, one approach for moving your analytics to clouds is:

• use simulated data following realistic simulations, instead of actual data;
• supplement in-house expertise with third party experts who specialize in analytics and cloud computing;
• use third party private clouds instead of public clouds to decrease risk or perceived risk;
• experiment with different analytic approaches and different analytic infrastructures;
• agree on APIs up front and transfer technology by transferring code that uses these APIs.

Open Data Group offers a service to our clients called Rapid Analytic Prototyping, or RAP, that uses this approach help our clients rapidly develop and deploy predictive models over big data.

Mistake 1. Underestimating the time required to get the data. This is probably the most common mistake in modeling projects. Getting the data required for analytic projects usually requires a special request to the IT department. Any special requests made to IT departments can take time. Usually, several meetings are required between the business owners of the analytic problem, the statisticians building the models, and the IT department in order to decide what data is required and whether it is available. Once there is agreement on what data is required, then the special request to the IT department is made and the wait begins. Project managers are sometimes under the impression that good models can be built without data, just as statisticians are sometimes under the impression that modeling projects can be managed without a project plan.

Mistake 2. There is not a good plan for deploying the model. There are several phases in a modeling project. In one phase, data is acquired from the IT department and the model is built. A statistician is usually in charge of building the model. In the next phase, the model is deployed. This is the responsibility of the IT department. This requires providing the model with the appropriate data, post-processing the scores produced by the model to compute the associated actions, and then integrating these actions into the required business processes. Deploying models is in many cases just as complicated or more complicated than building the models and requires a plan. A good standards-compliant architecture can help here. It is often useful for the statistician to export the model as PMML. The model can then be imported by the application used in the operational system.

Mistake 3. Building predictive models without an analytic strategy. Companies do not undertake major IT projects without having an IT strategy and a senior executive to champion the project, but the same cannot be said about analytic projects. It is a best practice for a company or organization to develop an analytic strategy and to assign a senior executive to execute the strategy. The executive then makes sure that the analytic strategy aligns with the corporate strategy and that all required resources are in place for the project to succeed. For an analytic project so succeed: the data required to support the project must be available; there must be modelers (or statisticians) available to develop the models; the modelers must have the right software tools; the models that are built must be able to be deployed; deployed models must enable actions that increase revenues, decrease risks, or improve the efficiency of business processes; and finally the improvements produced by the models must be tracked and reviewed.

Mistake 4. Trying to build the perfect model. Another common mistake is trying to build the perfect statistical model. Usually, the impact of a model will be much higher if a model that is good enough is deployed and then a process is put in place that: i) reviews the effectiveness of the model frequently with the business owner of the problem; ii) refreshes the model on a regular basis with the most recent data; and, iii) rebuilds the model on a periodic basis with the lessons learned from the reviews.

Mistake 5. The predictions of the model are not actionable. From this point of view, the model is evaluated not just by its accuracy but instead by measures that directly support a specified strategy. For example, the strategy might be to increase sales by recommending another product after an initial product is selected. Here the relevant measure might be the incremental revenue generated by the recommendations. The actions could be present up to three additional products to the shopper. The scores might be a score from 1 to 1000. The products with the highest three scores are then presented. This is a simple example. Unfortunately, in most of the projects that I have been involved with determining the appropriate actions and measures often requires an iterative process to get it right. I have developed a framework called SAMS to help ensure that the predictions of statistical models are actionable that will be the subject of a future post.

As digital systems grow larger and more complex, it can become very challenging to monitor their health and status, which is the first step in detecting potential problems, identifying the root causes, and taking appropriate preventive actions. These types of systems can contain thousands of different data feeds, data flows and processes. A problem with just one of them can interrupt payments, ads, and status updates, respectively. Often there are hourly, daily, weekly and seasonal variations in the data that complicates the detection of problems.

One way to gain some insight into this problem is to look at the origins in the 1920’s of statistical quality control. Walter Andrew Shewhart (1891 – 1967) was an engineer from 1918 to 1924 at the Western Electric Company, which manufactured hardware for the Bell Telephone Company. From 1925 to 1956 he was a member of the Technical Staff of Bell Telephone Company [ASQ].

Shewhart introduced control charts in the 1920's and they are still relevant today for quality control and related problems.

One of the problems that concerned him was identifying potential issues in factory assembly lines. For example, the dimensions and weight of metal parts that are sampled from an assembly can be recorded. He distinguished between two types of variations in these measurements:

• Common cause of variation (or noise) occurs as a normal part of the manufacturing process.
• A special cause of variation is not part of the normal manufacturing process, but represents a problem.

One of the goals of statistical quality control is to distinguish between these two types of variation and to quickly identify special causes of variation.

Shewhart introduction control charts as a tool for distinguishing between common and special causes of variation. A control chart had a central line and upper and lower control limits. When the measurement exceeded either the upper or lower control limits, it was considered a potential special cause of variation and investigated. Usually, the upper and lower control limits were three standard deviations above and and below the mean.

As anyone who has investigated potential data quality problems knows, identifying roots causes of potential problems is not easy and Shewhart also introduced a four step approach to these types of investigations that became known as the Shewhart Cycle, the Deming Cycle or the Plan-Do-Check-Act Cycle:

• Plan. Identify an opportunity or potential problem and make a plan for improving it or changing it.
• Do. Implement the change on a small scale and collect the appropriate data.
• Check. Use data to analyze statistically the results of the change and determine whether it made a difference.
• Act. If the change was successful, implement it on a wider scale and continuously monitor and improve your results. If the change did not work, begin the cycle again.

These same ideas are still used today as the basis for health and monitoring systems. Well designed digital systems these days are designed from the ground up so that appropriate log data is produced. Instead of a single assembly line producing physical items, there are thousands or millions of digital processes producing (nearly) continuous digital data. Often this data is available through an http interface and is continually collected.

Below is a dashboard from the open source Augustus system. Instead of a control chart, a change detection model is used, such as a CUSUM or GLR statistical model [Poor]. Instead of building a single model, a model for each cell in a multi-dimensional cube of models is built [Bugajski]. Instead of looking at the charts each day, an online dash board is used that is at the hub of an operations center.

Baseline and change detection models for each cell in a multi-dimensional data cube of models can be built easily using the open source Augustus system.

Open Data Group worked with the Data Mining Group to develop Predictive Model Markup Language (PMML) versions of change detection models and cubes of models. Both of these contributions were included in the most release of the PMML standard, version 4.1, which was finished in December of 2011.

This is an updated version of a post that first appeared on November 29, 2009.

References

[ASQ] ASQ, The History of Quality – Overview, retrieved from www.asq.org.

[Bugajski] Joseph Bugajski, Chris Curry, Robert L. Grossman, David Locke and Steve Vejcik, Data Quality Models for High Volume Transaction Streams: A Case Study, Proceedings of the Second Workshop on Data Mining Case Studies and Success Stories, ACM 2007

[Poor] H. Vincent Poor and Olympia Hadjiliadi, Quickest Detection, Cambridge University Press, 2008.

Best practices for managing big data and building empirically derived and statistically valid models over it are now emerging and the tutorial described some of these.

It is often a challenge once statistical models are built over big data to deploy these in operational systems. The open source Augustus system developed by Open Data Group provides an efficient means to deploy statistical and data mining models in operational systems and to quickly refresh models once they are deployed. Models deployed with Augustus can be updated simply by reading an XML file that uses the Predictive Model Markup Language or PMML. Augustus can both produce and read PMML files.

He also updated a FAQ about predictive analytics and data mining that you can here

The essential idea of PMML is that a predictive model, and more generally a statistical or data mining model, should not be thought of as code, but rather abstracted and described as metadata about the underlying data that it models. The PMML standard specifies an XML format for this metatadata.

The reason that this point of view is important is so that one application (the model producer) can produce the model, while another application (the model consumer) can use the model for scoring data. The model consumer can be integrated into production and operational systems and models can then be updated simply by reading new PMML files.

With this approach predictive models in operational systems can be updated quickly and easily. In contrast, when predictive models are viewed as code and new code is added to operational systems, a careful QA process is required before any new code can be deployed.

The upcoming version of PMML (PMML version 4.1, which should be released in the Fall of 2011) supports multiple models. This PMML feature was championed by Open Data Group over the past few years based upon its experience building predictive models over big data.

As the amount of data increases, building predictive models using multiple models (segmented models, hierarchical models, and related techniques) is absolutely critical. For big data, there is really no alternative.

I expect that with the explosion of big data and big data analytics, and with PMML’s support for multiple models, that PMML will begin to be an essential component of any analytic infrastructure that supports big data.