A detailed comparison of the top Monte-Carlo Simulation Tools for Microsoft Excel

There are very few performance comparisons available when considering the acquisition of an Excel-based Monte Carlo solution. It is with this in mind and a bit of intellectual curiosity that we decided to evaluate Oracle Crystal Ball, Palisade @Risk, Vose ModelRisk and Frontline Risk Solver in terms of speed, accuracy and precision. We ran over 20 individual tests and 64 million trials to prepare comprehensive comparison of the top Monte-Carlo Tools.

Modeling in Excel or with any other tool for that matter is defined as the visual and/or mathematical representation of a set of relationships. Correlation is about defining the strength of a relationship. Between a model and correlation analysis, we are able to come much closer in replicating the true behavior and potential outcomes of the problem / question we are analyzing. Correlation is the bread and butter of any serious analyst seeking to analyze risk or gain insight into the future.

Given that correlation has such a big impact on the answers and analysis we are conducting, it therefore makes a lot of sense to cover how to apply correlation in the various simulation tools. Correlation is also a key tenement of time series forecasting…but that is another story.

In this article, we are going to build a simple correlated returns model using our usual suspects (Oracle Crystal Ball, Palisade @RISK , Vose ModelRisk and RiskSolver). The objective of the correlated returns model is to take into account the relationship (correlation) of how the selected asset classes move together. Does asset B go up or down when asset A goes up – and by how much? At the end of the day, correlating variables ensures your model will behave correctly and within the realm of the possible.

One of the cool things about professional Monte-Carlo Simulation tools is that they offer the ability to fit data. Fitting enables a modeler to condensate large data sets into representative distributions by estimating the parameters and shape of the data as well as suggest which distributions (using these estimated parameters) replicates the data set best.

Fitting data is a delicate and very math intensive process, especially when you get into larger data sets. As usual, the presence of automation has made us drop our guard on the seriousness of the process and the implications of a poorly executed fitting process/decision. The other consequence of automating distribution fitting is that the importance of sound judgment when validating and selecting fit recommendations (using the Goodness-of-fit statistics) is forsaken for blind trust in the results of a fitting tool.

Now that I have given you the caveat emptor regarding fitting, we are going to see how each tools offers the support for modelers to make the right decisions. For this reason, we have created a series of videos showing comparing how each tool is used to fit historical data to a model / spreadsheet. Our focus will be on :

• Oracle Crystal Ball 11.1.2.1 Standard
• Palisade @RISK 5.7 Professional
• Vose ModelRisk 4.0 Professional
• Frontline Risk Solver 11.01 Professional

The goal of this comparison is to see how each tool handles this critical modeling feature.  We have not concerned ourselves with the relative precision of fitting engines because that would lead us down a rabbit hole very quickly – particularly when you want to be empirically fair.

Over the last 3 months, we have seen 3 of the 4 major players in the Excel Monte-Carlo Simulation arena introduce new releases. We hear a lot of talk about which tool is best and the truth is there is no perfect answer – it’s a personal thing dictated by user skill, preference and need.

For this reason, we have created a series of videos showing comparing how each tool is used to apply Monte-Carlo simulation to a model / spreadsheet. Our focus will be on :

• Oracle Crystal Ball 11.1.2.1
• Palisade @RISK 5.7
• Vose ModelRisk Standard 4.0
• Frontline Risk Solver 11.0
• Comparison

To keep the playing field level, we have used a simple additive model, which is simply defining a series of distributions (i.e. costs, budget items…), summing them up and analyzing the resulting sensitivity analysis. We have kept things simple, so we are not correlating any of the variables nor using any fancy math.

As you will see, there are definite differences AND similarities regarding how these packages tackle building a model. We are going to focus on those relating to inserting and copying input distributions as well as defining and analyzing model outputs. The objective is to compare the ease, usability and efficiency of each tool and give people the opportunity to choose for themselves which tool reflects their needs and preferences better.

Is there a winner in this battle between Crystal Ball and ModelRisk? To quote that way-too-often-quoted reply: It depends. Some users will value certain technical capabilities over others. Some users will value user-friendliness over accuracy. If there is to be a group deployment of a MCA spreadsheet package, usability may trump technical capabilities overall. Does it matter if one package has more distributions to choose from if there are only three that are of interest for your particular class of stochastic problems? Would it matter what kind of correlation enforcement method is used if, as in many manufactured assemblies, there is practically no correlation between separate components? Probably not. But if they do (as in financial and insurance applications), there will be a clear winner.

Correlation behavior in ModelRisk is enforced with the use of copulas. Copulas offer more flexibility in accurately simulating real data scatter-plot patterns than do single-value correlation coefficients. While this advantage is clear for financial and insurance applications, its implementation in an MCA spreadsheet simulator can make the difference between universal adoption and rejection by a majority of the intended user group. Let us now use ModelRisk (MR) to enforce the correlation behavior between Duke Basketball offense scores and their opponents' scores, based on the '09/'10 historical data.

In our quest to simulate future Duke Basketball scores, we have taken past historical data of individual games during the '09/'10 season and fitted probability distributions to that data. Two PDFs are generated; one for Duke's scores (offense) and one for their opponents' scores (defense). We have used both Crystal Ball and ModelRisk to perform this task. Is there something missing in our PDF formulations?

Let the battle begin anew. We continue our journey in uncertainty modeling, having understood how to fit distributions to data using Crystal Ball (CB). How does that experience compare to what ModelRisk (MR) has to offer?

Open the Duke 09_10 Scores spreadsheet with ModelRisk loaded in the Excel environment. We will first create the MR Objects representing the fitted PDFs. (Just as with the CB exercise, it is good practice to examine a variety of best-fitting distributions, rather than blindly accepting the top dog.) Then, in distinctly separate cells, we will create the VoseSimulate functions that behave as sampled values from the PDFs modeled by the MR Objects.

As with Crystal Ball, ModelRisk has the ability to fit distributions to historical data. The analyst looking to describe the variation of a Monte Carlo Analysis input can use "fitting" windows to select data and manipulate other options. How does the ModelRisk (MR) fitting experience stack up against the Crystal Ball (CB) methods and options? There are some important differences one should understand about MR before fitting PDFs to the Duke 09_10 Scores spreadsheet.

Let us assume we have a batch of historical data in a spreadsheet. Our mission-of-the-moment is to use this data and fit probability distributions that describe its past variability (or uncertainty). Consider using either Crystal Ball or ModelRisk to do this task. We offer free trials of both to registered users. If you register here, you can get yours too. Try fitting the same data using these two different packages. Let us know how and why one is better than the other. In demonstrating these capabilities, we gain first-hand experience on the usability and capabilities of the alternatives and which features compared have more priority. The best way to judge is to try them out for yourself.