Feb
22
2011
Is there a winner in this battle between Crystal Ball and ModelRisk? To quote that way-too-often-quoted reply: It depends. Some users will value certain technical capabilities over others. Some users will value user-friendliness over accuracy. If there is to be a group deployment of a MCA spreadsheet package, usability may trump technical capabilities overall. Does it matter if one package has more distributions to choose from if there are only three that are of interest for your particular class of stochastic problems? Would it matter what kind of correlation enforcement method is used if, as in many manufactured assemblies, there is practically no correlation between separate components? Probably not. But if they do (as in financial and insurance applications), there will be a clear winner.
Based on the discrete distribution-fitting and correlation/copula-fitting capabilities of CB and MR in previous posts, I have contrasted the capabilities in the following table with pros and cons. Even though I am an engineer by nature, I have attempted to step back and objectively consider the much larger class of risk analysts. That would include the fields of scientific research, finance, insurance, and transactional processes, to name a few.
| |
PROS
|
CONS
|
|
Discrete Distribution Fitting
|
|
|
|
CRYSTAL BALL
|
- Ability to rapidly select classes of PDFs for fitting operation.
- Fitting operation requires at least 9 clicks.
- Provides hypothesis test p-values (null = fitted PDF).
|
- Fits at most 6 PDFs.
- Only 1 fitting criterion available.
- Cannot fit to data with less than 15 samples.
- Allows for PDF parameter uncertainty via cell-referencing other PDFs.
- No penalty for over-fitting.
|
|
MODELRISK
|
- Fits up to 11 PDFs. (Does not include Zero-Inflated or Zero-Truncated.)
- 3 fitting criteria available.
- Can fit to minimum number of samples required by PDF MLE calculations (less than 15).
- Allows for uncertainty in fitted PDF parameters.
- Over-fitting penalties.
|
- Some PDFs fitted are adaptations or combinations of other PDFs. (Not really a negative but should be considered when comparing # of PDFs available for fitting with other MCA codes.)
- Fitting operation requires up over 18 clicks.
- No hypothesis test p-values provided.
|
|
Correlation / Copula Fitting
|
|
|
|
CRYSTAL BALL
|
- Implements Spearman's coefficient to allow accurate simulation between PDFs of different nature.
- Adding fitted correlation to existing PDFs requires up to 15 clicks.
- Ability to easily turn off defined correlations prior to simulation.
|
- No immediate flexibility to adjust for defined patterns observed in scatter plots.
- No fitting criteria employed.
|
|
MODELRISK
|
- Implements Copulas to allow more flexibility in modeling interacting variables.
- 5 different bi-variate copulas to select from.
- 3 fitting criteria available.
- Correlation behavior can be manipulated directly in spreadsheet.
- Ability to link data to Empirical copulas, allowing immediate adjustment.
|
- Correlation behavior must be connected to simulated variables via Vose formula arguments.
- Adding fitted correlation behavior requires over 20 clicks plus necessary manual modification of simulated variables.
- Correlation behavior can be turned off via spreadsheet manipulation techniques.
|
|
Usability of Discrete PDFs and Correlations
|
|
|
|
CRYSTAL BALL
|
- Organizes PDFs & correlations in hidden worksheet, preventing unintentional modification (error-proofing).
- Automatic color-coding during CB Assumption creation.
|
- All CB Assumptions and Correlations must be manually re-created if fitted data is modified or added to.
- Necessary to create new CB Assumption to simulate same PDF in two-or-more transfer function input cells.
|
|
MODELRISK
|
- Vose function call-outs to data allow instant updating when fitted data is modified or added to.
- Not necessary to create new Distribution Object in order to simulate same PDF in two-or-more transfer function input cells.
|
- User must organize both Distribution and Copula MR Objects for easy recognition and operation.
- Less protection from unintentional modification of PDFs and correlation behavior.
|
A few caveats should be mentioned: The data set utilized to demonstrate the CB and MR capabilities has a sample size of 34. Unfortunately, if the sample size goes below 15, there is no facility in CB for distribution fitting. Why? Statistically speaking, a small sample size provides less confidence about the distribution fit. The creators of CB saw it fit to prevent this from happening by placing a lower limit on the sample size. But savvy statisticians know this relation very well. MR assumes the user is one of these types. And to counteract the greater range of confidence around a fitted PDF parameter, MR can incorporate an Uncertainty element that we have yet to describe.
We have also not attempted to fit continuous distributions, which are probably of more interest to the reader than discrete ones. In addition, we have ignored particular classes of MR distributions that are really neat (zero-inflated, zero-truncated, aggregate, combined, sliced and diced, etc…). There are also interesting empirical-fitting options for both distributions and copulas in MR. I hope to return to these in future posts.
Certainly, discrete distribution-fitting and correlation enforcement are not the only two capabilities that one should base a software purchasing decision on. I would urge the reader to place an importance ranking on the summarized feature list being judged and make an informed decision with your priorities in place. For my money, MR is the clear winner in this battle. Your winner may be different.
Karl Luce is a Solutions Specialist who has a deep passion for using Monte Carlo analysis as a Six Sigma tool and quantifying uncertainty in the universe. His past roles included training Crystal Ball and quality concepts as a Design for Six Sigma (DFSS) Master Black Belt and a Six Sigma Transactional Black Belt. He received his BS in Aeronautical/Astronautical Engineering from M.I.T. in 1985. Prior to joining Oracle in 2005, Karl held analytical and managerial positions within Fortune 500 corporations in the defense aerospace and automotive industries.