Predictive Model Tutorial - Interpreting the Results

Interpreting the Predictive Model Results

The following results are from the OfficeStar Tutorial data set that loads automatically when you select the Tutorial link in the Enginius Dashboard and run with analysis parameters indicated in Running a Predictive Modeling Analysis article. 

Confusion matrix

The confusion matrix section of the report assesses the model performance. The confusion matrix contains two matrices of the same data: numerical counts and percentages.

The diagonal elements of both matrices indicate the convergence of the actual (observed) and predicted data.  High values on diagonal represent a high correlation between observed and predicted behaviors (called hit rate).  You can also use the confusion matrix to compute indices called “Recall” and “Precision” which also provide additional metrics for evaluating a model’s performance. 

Model predictions

The model predictions table shows how well the model compares to actual results. The online and Word/Powerpoint/PDF version of the report will give you an excerpt of the data table. To see the full data table, output the report inExcel format.

Gain chart and lift

A gain chart is a useful representation of how good a predictive model is at identifying the most favorable responses (e.g., “Yes” in the above table). 

The x-axis represents the population ordered in decreasing order of choice likelihood of the favorable response, and the y-axis represents proportion of the total number of favorable choices. The diagonal on the Gain chart represents the predictions of a model that predicts the choices of each individual randomly, and the red line predicts the choices based on actual data (i.e., Truth). The other two lines predict the performance of the choice model. The choice model’s performance gets better as the green lines (representing the model) depart from the performance of the random model, and approach the predictions based on truth (observed data). When we reach 100% of the ordered list, all models recover fully the total number of favorable choices.

The dashed green line represents the gain chart obtained on the entire calibration data, without cross-validation, whereas the green area represents the same obtained by cross-validation. The latter sometimes provides degraded but more realistic performance results that reduces the influence of outliers in the calibration data.

Lift is defined as the improvement in model performance at different percentile levels of the ordered list of the population depicted on the horizontal axis. If by selecting the top 10% of the ordered list, we can reach 18.7% of the individuals who make the appropriate choice (i.e., respond favorably), the focal model performs 1.87 times better than a model that makes random assignments. In that case, the lift at the 10-percentile level is 1.87.

The 'truth' is the true number of favorable responses in the ordered list. Improvement defines how well the truth is recovered by the model. An improvement of 100% means that all the favorable responses were recovered perfectly.

Discrete Elasticities

Elasticity is a measure of how responsive a target variable is to a change in the value of a predictor. Specifically, elasticity is defined as a ratio of percentage change in the target variable (Y) in response to a 1% change in a predictor (X), so that Elasticity = (% change in Y) / (1% change in X).

To compute the elasticities, Enginius follows these steps:

1. Predict the target variable Y for each individual at the current values of X for each individual. Average Y across respondents to obtain Y0.
2. Increase the values of X (for each observation) by 1% and predict the target variable Y at these new values. Average across respondents to obtain Y1.
3. Compute elasticities as (Y1 – Y0) / Y0.

Keep in mind that, when X is discrete, an increase of 1% in X is meaningless. For instance, if X = 1 means that the color is red, X = 1.01 has no useful interpretation and the elasticity computations do not lead to interpretable results.

Here is an example of elasticities computed by Enginius:

It does not make sense to interpret the elasticity of Gender. For Age, these results suggest that if the age of everyone increases by 1% from their current ages (someone a bit older), the overall probability of choosing alternative “1” will increase by 0.34%, and the overall probability of choosing alternative “0” will decrease by 0.29%.

Point Elasticities

For several predictive models, it is possible to obtain point estimates for elasticities using analytical formula (this is described in the Appendix for the case of conditional logit model (Choice between multiple alternatives, one line per alternative (0/0/1)). For the example data in the tutorial, the following elasticities are obtained for List Price. 

The elasticity matrix summarizes how sensitive the predicted choice shares are to changes in the list price of each brand. The diagonal elements represent own-price elasticities, indicating how the share of a brand changes when its own price increases. The off-diagonal elements represent cross-price elasticities, showing how the share of one brand responds when the price of another brand changes. Because these are aggregate-share elasticities, they reflect the percentage change in the average predicted market share.

For example, the own-price elasticity for All is −1.872, meaning that a 1% increase in the price of All reduces its predicted choice share by about 1.87%. Similarly, the own-price elasticities for Tide, Wisk, and Yes are −1.136, −1.768, and −2.889 respectively, indicating that each brand’s share decreases when its own price rises.

The cross-price elasticities indicate substitution effects between brands. For instance, a 1% increase in the price of Tide increases the predicted share of All by about 0.823%, while a 1% increase in the price of Wisk increases All’s share by about 0.325%. Likewise, a 1% increase in the price of All raises Tide’s share by about 0.398% and Wisk’s share by about 0.372%. These positive cross elasticities reflect the fact that when the price of one brand increases, consumers tend to substitute toward competing brands.

As with other elasticity calculations, these results should be interpreted cautiously when alternatives differ substantially or when the variable of interest is categorical. In such cases, the implied substitution patterns may appear unrealistic. For example, a simple model might predict that a price increase for a mainstream brand such as Honda would increase the choice probability of both close competitors (e.g., Toyota) and distant luxury brands (e.g., BMW) by similar percentages. For this reason, elasticity interpretations are most meaningful when applied to comparable alternatives and continuous variables such as price.