Grossman is widely recognized as an expert in large data, including design of analytic architectures deployed in cloud environments involving Petabytes of data.   Dr. Grossman has served in a variety of key advisory capacities over his career to assist government agencies meet the complex challenges of information technology, security and oversight.

In addition to his work with Open Data’s commercial and government clients, Grossman is a faculty member at the University of Chicago, where he is the Director of Informatics at the Institute for Genomics and Systems Biology, a Senior Fellow at the Computation Institute, and a Professor of Medicine in the Section of Genetic Medicine.  He also serves on the NASA Advisory Council, chairs the Open Cloud Consortium, founded the Data Mining Group, is a Visiting Professor at the Booth School of Management, and is a frequent speaker and author on the area of big data and intensive computing.

Open Data is proud of the well deserved recognition, and pleased that Robert Grossman’s  significant contributions  are being honored in this way.

More about the Fed100 Award

The event was kicked off Christopher Steiner, author of Automate This: How Algorithms Came to Rule Our World, who talked about some of the ways that algorithms are changing our lives, ranging from high speed trading to medical diagnoses.

In addition to myself, the panel included:

• Rayid Ghani, Chief Scientist of the Obama for America 2012 campaign
• George John, Founder and CEO, rocketfuel
• Keary Philips, Allstate Insurance Company
• Rishad Tobaccowala, Chief Strategy and Innovation Officer, VivaKi

Rayid Ghani spoke about some of the ways that predictive analytics was used to help persuade some of those who may not have voted to actually register to vote and later to show up at the polls and to vote.

I discussed that as important as algorithms are, they are sometimes best thought of in the context of: i) what are the Concepts and abstractions used to model the problem; ii) what are the Algorithms used to compute with these abstractions; iii) and what are the Devices that the algorithms run on? CAD for short.

It is interesting to look at big data from the perspective of the concepts, algorithms and devices. We are better at predictive analytics today not just because we have better algorithms, but also because we have made significant progress on the concepts and abstractions that underlie predictive analytics and on the devices we use.

For example, 20 years ago with big data and predictive analytics, the focus was on building a single statistical model and looking for knowledge; we generally used regression algorithms to analyze data; and we used high end workstations for the computations. Today, with big data, we tend to think of collections of models (ensembles, cubes of models, etc.) and focus the actions (not the knowledge) that are possible; we would more typically use algorithms that compute trees or support vector machines; and we do computations over clusters of workstations.

There is more about CAD in Chapter 1 of my book on the Structure of Digital Computing.

They illustrated the lecture with an example of building a predictive model over data provided by the City of Chicago about CTA busses using Amazon’s Elastic MapReduce.

You can find some of the materials for the lecture on the web page tutorials.opendatagroup.com. The materials also contain links to some best practices for deploying analytics in operational systems using PMML and PMML-compliant scoring engines, such as Augustus.

The slides and some related materials can be downloaded from tutorials.opendatagroup.com.

The 3.5 hour tutorial consists of 12 modules:

1. Introduction
2. Building Predictive Models – EDA and Building Features
3. Case Study: MalStone
4. Working with Multiple Models: Ensembles and Segments
5. Case Study: CTA
6. Deploying Predictive Models Using PMML-based Scoring Engines
7. Three Ways to Build Models over Hadoop Using R
8. Case Study: Building Trees over Big Data
9. Improving the Impact of a Model In Operations – The SAMS Methodology
10. Case Study: AdReady
11. Quantifying the Lift of a Predictive Models and Improving It
12. Case Study: Matsu

Open Data Group helped pioneer some of the technology behind topics 2, 3, 6, 7, 8 and 9. For example, you can follow the links to learn more about the MalStone Benchmark, the Multiple Model component of the DMG’s PMML standard, and Project Matsu, which uses MapReduce to process and analyze images.

If you are at the Strata Conference, please stop by to say hello.

In this post, the point of view is that a business owner of a problem in a company that requires a model and is considering whether to build the model in-house, outsource the model to a vendor providing analytic services, or simply to give up on building a model and produce a report instead. I don’t recommend the latter option, but unfortunately, in practice, it is all too common.

Broadly speaking, from a business owner’s point of view, there are several phases required to build a model for a new project. The process looks a bit different from the modeler’s point of view. It is also a bit simpler if the same model has been built before and all that is required is to update the model using new data. Here are the basic steps required to build a model from the business owner’s point of view.

1. Working with IT to obtain all the data required for the project and making it available to the modeler.
2. Answering questions from the modeler about the data.
3. Agreeing upon the output of the model.
4. Reviewing the first model with the modeler.
5. Reviewing the second and subsequent models with the modeler.
6. Working with IT to deploy the model.

All steps except for Step 1 and Step 6 are collaborative between the business owner and the modeler. At the beginning of many projects, Step 3 looks obvious. It turns out that it is often not so obvious until the project is towards the end, the data has been cleaned, and the deployment well underway. One way to understood why this is so is because often one doesn’t have a good understanding of the most appropriate output of a model until the data has been cleaned and there is a good understanding of how the model will be deployed in operational systems.

Let’s look at this same process now from the viewpoint of the modeler. To simplify, the following steps are required:

1. Waiting for the data.
2. Cleaning the data.
3. Asking the business owner questions about the data.
4. Agreeing upon the output of the model.
5. Developing a set of features for the model.
6. Estimating the parameters of the model.
7. Building a measure to evaluate the model.
8. Evaluating the model using the measure.
9. Developing post-processing rules for the scores produced by the model.
10. Repeating the steps above for the second and subsequent versions of the model until everyone is happy, or there is no more time or funding left.
11. Deploying the model.

Building a new model requires completing all the steps above. Generally, a series of models (version 1 of the model, version 2 of the model, etc.) are produced and reviewed by the business owner and the modeler (Step 10). The more time available for Step 10, the better the quality of the model.

To understand these steps a bit better, it might be helpful to review post about the SAMS Methodology. The SAMS methodology explains how to think of models in terms of the Scores they produce, the Actions these enable, the Measures used to evaluate the actions, and whether these actions support a targeted Strategy or not.

Sometimes a model has been built before and only some of these steps need to be repeated. For example, refreshing a model only require completing steps 6 and 8 for a series of models. Rebuilding a model usually only requires repeating Steps 5, 6, 8 and 9 for a series of models.

Sometimes, the data is supplied in a standard format (for example, it is provided by a third party) and the deployment uses a standard format (for example, only a list is required that contains a list of names and corresponding offers). In this case, after a model has been built once, all that is required when a business owner supplies new data is to perform Steps 6 and 8. Call this a standard model. Standard models are substantially less work to build then models that require completing all the steps above. These more labor intensive models are often called custom models.

Most requests for models fit into some standard categories of models. For example, models that predict whether a prospect will respond to an offer (response models), whether a customer will remain a customer (attrition models), whether a customer will keep current with their payments (credit model), whether a transaction is valid or fraudulent (fraud models), etc.

Sometimes, models that don’t fit into these familiar categories of models are built. Call these new types of models. A new type of model also requires that the modeler develop new types of features, new types of measures for evaluating the models, etc. New types of custom models are the most labor intensive to build.

In practice, it usually takes four to six months or longer to build a custom model, once the data has arrived. As the size and complexity of the data grows, each of the steps usually requires more time.

There is pretty good practice out there of how to build data warehouses. There is not a lot of good practice or knowledge out there about how to build statistical models over big data.

The quote above is from an interview by WashingtonExec of Open Data Group Founder Robert Grossman. He was interviewed about big data, predictive modeling and related topics. You can find the interview here.

In the interview, he briefly discusses some of the rules he has developed over the years for building predictive models over big data. One of the top three is: “Do you have an environment where you can deploy the models you build into operational systems?”

Open Data Group often uses the Augustus system for deploying models into operational systems. Augustus is open source and follows the PMML standard. It supports segmented models and pre-processing of the inputs to models and post-processing of the scores produced by models. Augustus support for pre- and post-processing was described in a recent post.

He was also asked about the disruptive nature of predictive modeling over big data.

Real Time Scoring of Data Using Precomputed Analytic Models

Perhaps the most important way that the term is used is to describe the real time scoring of data stream using an analytic model. The most common standard for describing analytic models is the Predictive Model Markup Language or PMML. In the modeling environment, a trained individual aggregates the data, cleans and preprocesses the data, and then uses modeling software to build an analytic model, which can be exported as a PMML file, as in the top have of the diagram below.

This PMML model can then be deployed in operational environments, as in the lower half of the diagram. A stream of data can then be scored using the analytic model in “real time.” Notice that in this case, the model does not change automatically, it simply scores the data. Of course, in practice, the team that builds the analytic models also rebuilds them from time to time.

Real Time Scoring of Data Using Continuously Updated Models

In the first example, the analytic model was built over all of the data and did change until it was rebuilt on new data. Some analytic models have the property that they can be updated incrementally with each new data record that comes along.

A good example, is the nearest neighbor (NN) model. The NN model contains all the data over which it was built and simply assigns a record to be scored, the label of the nearest record in the model. There is an obvious incremental version of the algorithm that consists of the model that results from adding each new data point to the current model to produce the new model.

Note that in practice continuously updated models like this run in computers that have a finite amount of storage and a finite amount of disks.

The subject of streaming analytics is concerned with how to build analytic models in which data is presented incrementally and there is a finite amount of memory and storage.

Real time analytics with continuously updated models is essentially the first use case but instead of periodically updating the model manually, models are updated with each new record that they consume.

Event Stream Processing

More recently, systems such as Storm and S4, use an elastic scale out architecture to process streams of data in parallel in real time. Most often rules are used to process the data, but analytic models could also be used.

In a later post, we will discuss this third example of real time processing in more detail.

You can think of this as roughly the analogy for analytics and predictive modeling of the Capability Maturity Model for Software that was developed by Software Engineering Institute at CMU about 20 years ago.

I recently finished a survey of over 25 companies that I used to fine tune this model. Here is a high level summary of the five different analytic maturity levels of an organization:

Level 1 – Analytic reporting. An Analytic Maturity Level 1 organization has the ability to analyze data and to build reports.

Level 2 – Analytic modeling. An Analytic Maturity Level 2 organization has the ability to build predictive models over data.

Level 3 – Repeatable analytics. An Analytic Maturity Level 3 organization uses a repeatable process for building and deploying analytic models.

Level 4 – Enterprise level analytics. An Analytic Maturity Level 4 organization uses a repeatable process for building analytic models throughout an enterprise and integrates these models together to improve operations.

Level 5 – Strategy driven analytics. An Analytic Maturity Level 5 organization has an analytic strategy and a senior leader in charge of the analytics strategy and the analytic governance.

On June 25, 2012, I’ll be speaking at Predictive Analytics World in Chicago on the Analytic Maturity Model and some steps that organizations can take to improve their Analytic Maturity Level.

The following two forthcoming publications contain more detail about the Analytic Maturity Model:

1. Robert L. Grossman, The Strategic Dimensions of Data, Chapter 10, Open Data Press, 2012, to appear.
2. Robert L. Grossman and Kevin Siegel, An Organizational Maturity Model for Analytics, to appear.

1. an event associated with an entity comes in, say a user (the entity) visits a page (the event) on a web site
2. the features associated with the entity are retrieved, say features that describe the user and his or her behavior at the site, and the features are updated with the event
3. the features are used as the input to a predictive model, which produces a score, say the likelihood that the user will click on an ad in a certain category
4. an action is taken on the basis of the score, say to display an ad in a certain category if the score is above a threshold.

This may be easier to remember if you use the acronym EEFM for Event, Entity, Features, and Model. If the events are all available at the same time, the feature vectors for all the entities can be created at the same time and the features can all be scored “in batch”.

At Open Data Group, we used the Predictive Model Markup Language (PMML) to express our models in XML so that models may be built in a development environment with one application and then easily deployed in a production environment with another application. We also use an open source scoring engine (Augustus) to deploy our models in operational environments.

In practice, when actually deploying models, it is usually a bit more complicated.

First, various business rules are usually used to process the event prior to using the associated state as input to the predictive model (pre-processing). Second, the score of the model is usually processed by additional business rules (post-processing) prior to selecting an action. For example, if the event is associated with the visit of someone who is likely to be under 18, the site may choose different types of ads as part of the pre-processing. Second, as part of the post-processing, inventory rules and rules about how often to show ads (exposure rules) may be used to exclude certain ads.

Until recently, pre- and post-processing usually needed to be coded manually and couldn’t be expressed easily in PMML.

That has changed with Augustus 0.5.2. With this version of Augustus, Python code can be embedded in the PMML file to express pre- and post-processing rules, as well as to combine multiple models to produce scores, and to process multiple models and scores in a variety of different ways.

We call this augmented processing and in our experience over the past several months it has significantly simplified the deployment of predictive models into operational systems.

Here is an example from a white paper that we are writing about augmented processing using Augustus. With segmented modeling, you can use multiple models in different segments to score an event and in this way produce multiple scores. Assume that you want to use the minimum score produced in this way as the score for the event. Here is some Augustus code to do this that can be embedded in the PMML file for the segmented model:

def action():
    segmentScores = []
    for segment in segments:
        segmentScores.append(segment.score()[PREDICTEDVALUE])

    if len(segmentScores) == 0:
        finalScore = MISSING
    else:
        finalScore = min(segmentScores)

    output.xmlopen(“Event”, attrib={“number”: eventNumber})
    output.xmlfield(“Score”, finalScore)

Augustus 0.5.x was a rewrite of the PMML-compliant scoring engine with two important changes:

1. Augustus now has the ability to build models in batch mode using static data files as usual, but also in a streaming mode in which data is read only once.
2. Augustus now supports custom processing, a mechanism that allows the user to embed arbitrary Python code inside of Augustus. This provides a simple mechanism, for example, to combine multiple models and produce a single score.

Augustus 0.5.x uses the latest PMML specification 4.1 from December of 2011.
The latest version, Augustus 0.5.2.0, was released on April 4, 2012. Its model coverage is: Tree, Regression, Baseline, Rule Set, Naïve-Bayes, and Cluster.

AdReady is an Open Data Group Business Partner and is using the latest Augustus for the development and deployment of sophisticated statistical models for use in a large, robust, and scalable targeted ad-bidding system. AdReady’s Real-Time-Bidding (RTB) enables advertisers to bid more effectively on precise inventory and audience targets. Bid requests need to be collected, scored against segmented models, and responded to in less than 100 ms.

By using Augustus, AdReady can grow using elastic capacity and score on VM instances, bypassing the normal (Augustus) data pipeline and send events directly into the scoring engine processing loop from their Python MVC web framework (Django over Apache). AdReady described their use of Augustus in a nice blog post today.

To learn more about building a scalable analytics platform using Augustus, please contact us at info at opendatagroup.com.