Bayesian Reasoning using R
Management Research

Bayesian Reasoning using R

Robert Brown

Gender Inference from a Specimen Measurement

Share:

Print

Rate article:

No rating
Rate this article:
No rating
Bayesian Reasoning: Gender Inference from a Specimen Measurement

Imagine that we have a population of something composed of two subset populations that, while distinct from each other, share a common characteristic that can be measured along some kind of scale. Furthermore, let’s assume that each subset population expresses this characteristic with a frequency distribution unique to each. In other words, along the scale of measurement for the characteristic, each subset displays varying levels of the characteristic among its members. Now, we choose a specimen from the larger population in an unbiased manner and measure this characteristic for this specific individual. Are we justified in inferring the subset membership of the specimen based on this measurement alone? Baye’s rule (or theorem), something you may have heard about in this age of exploding data analytics, tells us that we can be so justified as long as we assign a probability (or degree of belief) to our inference. The following discussion provides an interesting way of understanding the process for doing this. More importantly, I present how Baye’s theorem helps us overcome a common thinking failure associated with making inferences from an incomplete treatment of all the information we should use. I’ll use a bit of a fanciful example to convey this understanding along with showing the associated calculations in the R programming language.

Note: If you don't care about the R code at all and just want to follow the basic reasoning, you can hide all the code at once by selecting Code/Hide All Code from the drop down list in the top right corner of this page. If you want to play around with the R code yourself, you can download the code by selecting Code/Download Rmd.

Suppose we are aliens from another planet conducting scientific research on this strange group of bipedal organisms called humans. Humans are sexually dimorphic, presenting themselves as female and male genders. Each gender expresses its height across a distribution, and the means of each gender population are distinct. Years of collecting data about the humans reveals that the mean height for adult females is 5.33 bliks; males, 5.83 bliks. (“Bliks” are our unit distance measurement, like feet or meters.) Both genders show a similar bell-shaped distributed (i.e., Normal distribution) variation around their mean heights with a standard deviation of 0.2.

gender <- c("Female", "Male") adult.m <- c(5.33, 5.83) # Mean adult height by gender. adult.sd <- 0.2 # Adult height standard deviation. Constant across gender.

Another piece of information that we have collected about the human population is the base rate frequency, or the proportion, of each subpopulation in the total population. These proportions provide some valuable information, particularly if the proportion of one subpopulation is is larger than the others. For example, among the Zargons, chartreuse gills show up in 85% of the population compared to 15% for those with magenta gills. If we randomly select a member of the Zargon population for our zoo exhibit, we most likely will select one with chartreuse gills. In the language of Bayesian reasoning, we can call this base rate the prior probability (or degree of belief) that a randomly selected individual from the total population should be classified as a member of one category or another. In the generic notation, this would be Prob(Hypothesis), but in the context of our current situation, we might write Prob(Gender). Hold on to this concept, as we will return to it in just two shakes of a proton dislocator.

Unfortunately, a recent viral epidemic nearly wiped out the population of males, leaving the females over-represented as 90% of the population as compared to the historically approximately even distribution of females to males.

# Set up a vector to contain the current Prob(Gender). base.rate.freq.fem <- 0.9 # The female base rate frequency. base.rate.freq <- c(base.rate.freq.fem, 1 - base.rate.freq.fem) # (females, males)

Imagine that on an exploratory mission to Earth, we discover an injured human to whom we intend to apply medical attention before we ship it off to our galactic zoo; however, due to its injuries and its extreme emotional state, we can’t determine its gender. The only data we can obtain is its height (Yes, we can travel across vast distances of space, but we have to use this crude means to make a measurement. Bear with me. This is just an example, not a sci-fi novel.), which we observe to be 5.65 bliks. Based on this one sample of measured information, we must infer the gender of the specimen before we proceed. How can we use the information we have on hand to make a rational inference?

# The measured height of our unidentified specimen. # units = Bliks specimen.height <- 5.65

First, we need to determine the probable height of any member of a gender based on our current characterization of the gender populations. Again, in the language of Bayesian reasoning, we call this the likelihood function, which tells us the probability of observing some kind of evidence, like a measurement, that is conditional on the selection of a given population or conditional on a given hypothesis being the case. We usually write this as Prob(Evidence|Hypothesis). In the context of our problem, we might write Prob(Height|Gender).

Using the dnorm(...) function, we can calculate the probability distribution across the height domain for each gender. Note that the dnorm(...) function produces the probability density at a given point.

# Here we set up an index for the height domain, with a step size of # 0.04 bliks. step.size <- .04 height <- seq(4, 7, by = step.size) # Calculate the height probability distribution for each gender across the # height domain. In this case, the result will be an array of shape (2 x # height). We apply the transpose funtion to this array to reorient it to # (height x 2). likelihood.height <- t(sapply(height, function(h) dnorm(h, adult.m, adult.sd))) # Recast the array as a data frame in preparation for plotting in ggplot. df.likelihood.height <- data.frame(height, likelihood.height) colnames(df.likelihood.height) <- c("Height", gender)

If we plot this function, we observe the probability distribution across the height domain. Again, we also refer to this as the likelihood function.

# Recast the rectangular data frame to a relational format such that the height # index is the leftmost variable. melt.df.likelihood.height <- melt(df.likelihood.height, id.vars = "Height", variable.name = "Gender", value.name = "Probability") height.distr <- ggplot(melt.df.likelihood.height, aes( x = Height, y = Probability, group = Gender, colour = Gender )) + geom_line(aes(group = Gender, colour = Gender)) + geom_vline(xintercept = specimen.height) + xlab("Height [bliks]") + ylab("Probability Density") + ggtitle("Height Distribution of Human Adults", subtitle = "Prob(Height|Gender)") print(height.distr)

It’s tempting to look at these distributions and conclude that the proper inference about the gender of our specimen would be the population with the greatest probability at the specimen’s measured height. After all, the height of adult humans is one of the most easily observable and available aspects of this strange species that tends to cover up all the distinguishing dimorphic characteristics with something they call “clothes.” A quick reference to our intuition might tell us that height is a good measure to make an inference about gender. In this case, we measured the height of our specimen at 5.65 bliks (the vertical black line). Based on this height compared to the distributions, we might infer that the specimen is male since the likelihood at r specimen.height bliks is higher for males than females.

Unfortunately, this approach leaves out the information we have about the current prevalence (or the base rate frequency or prior degree of belief) of the genders in the population. Remember the example of the Zargons? We typically could infer the subpopulation based on the observation of a dominantly expressed characteristic. But in this case, we have two characteristcs we could consider, and they seem to work against each other; that is, a randomly selected adult human after the viral post-apocalypse is now mostly likely female, but the height characteristic information we have seems to lead us to infer that we have a male. Can we combine both pieces of information to adjust the strength of our inclination in a reasonable manner? Fortunately, all that we have to do to include both pieces of information in our judgment is to weight the likelihood of height for a given population proportionally to the prevalence of the population. We can achieve this by multiplying the liklihood function by the base rate probability.

likelihood.height.gender <- t(base.rate.freq * t(likelihood.height)) # Recast the array as a data frame in preparation for plotting in ggplot. df.likelihood.height.gender <- data.frame(height, likelihood.height.gender) colnames(df.likelihood.height.gender) <- c("Height", gender)
# Recast the rectangular data frame to a relational format such that the height # index is the leftmost variable. melt.df.likelihood.height.gender <- melt(df.likelihood.height.gender, id.vars = "Height", variable.name = "Gender", value.name = "Probability") height.distr.gender <- ggplot(melt.df.likelihood.height.gender, aes( x = Height, y = Probability, group = Gender, colour = Gender )) + geom_line(aes(group = Gender, colour = Gender)) + geom_vline(xintercept = specimen.height) + xlab("Height [bliks]") + ylab("Probability Density") + ggtitle("Probability-Weighted Height Distribution of Human Adults", subtitle = "Prob(Height|Gender) * Prob(Gender)") print(height.distr.gender)

Now we observe that the probability weighted height distributions most likely ought to change our inference about the gender of our specimen. All we need to do now is find the conditional likelihoods at the measured height and normalize them with the marginal probability at that height, where the marginal probability is just the sum of the conditional likelihoods across genders at the measured height.

conditional.likelihood <- base.rate.freq * dnorm(specimen.height, adult.m, adult.sd) # cast into an array. conditional.likelihood <- array(conditional.likelihood, dim = c(1, 2)) colnames(conditional.likelihood) <- gender

The conditional likelihoods for each gender at the measured height is:

Female Male 0.50 0.13 
# The marginal probability is the total probability of the product of the prior # and the likelihood at the specified measured observation. marginal.prob <- sum(conditional.likelihood)

The marginal probability, which is just the sum of the two probabilities in our conditional likelihood is:

[1] 0.63

Now we divide the conditional likelihood values by the marginal probability value.

# The probability (rational inference) of gender associated with sampled height. posterior.prob.gender <- conditional.likelihood / marginal.prob

We should accept now as our updated belief (i.e., posterior probability) about the gender of our specimen to be:

Female Male 0.79 0.21 

Our final observation is that Female should be our rational inference for the gender of our specimen based on the prior probability weighted likelihood of height as a function of gender.

In short, what we calculated was prob(Gender|Height)=prob(Gender)*prob(Height|Gender)/(marginal_probability) where the marginal probability is sum(prob(Gender) * prob(Height|Gender)) Then we selected the gender argument with the highest probability.

The term sum(prob(Gender) * prob(Height|Gender)) is a normalizing factor. Once the quotient has been found, the sum of probabilities should equal to 1.

With regard to our example, this exercise has shown us that tall females among the population of all adult females are somewhat rare. However, tall females among the entire post-apocalyptic population of adult humans (in which females are the most prevalent subpopulation) are more prevalent than just-below-average height males, even though such males would be more prevalent than tall females if the the subpopulations were more equal in size. Therefore, finding a tall individual should lead us to infer, given no other information about them, that the individual is most likely female. Combining both pieces of information allows us to temper our intuition that would have been based on only one of the available pieces of information.

An interesting question that might arise from these insights is: what is the base rate frequency of the populations of females and males that might instill complete ambiguity in us about the gender of the specimen based on the measured height and the likelihood height functions of each gender? The answer is simple. All we have to do is recognize that

base.rate.freq.fem * Female_Likelihood|specimen.height = (1 - base.rate.freq.fem) * Male_Likelihood|specimen.height

and then solve for base.rate.freq.fem. Let’s call this special base rate frequency indif.base.rate.freq.fem. Rearranging we get

Likelihood.ratio = Male_Likelihood|specimen.height / Female_Likelihood|specimen.height indif.base.rate.freq.fem = Likelihood.ratio / (1 + Likelihood.ratio)
# reform example equations into R statements likelihood.ratio <- dnorm(specimen.height, adult.m[2], adult.sd) / dnorm(specimen.height, adult.m[1], adult.sd) indif.base.rate.freq.fem <- likelihood.ratio / (1 + likelihood.ratio)
Female Male 0.71 0.29 

Therefore, if we find ourselves in a situation in which the proportion of females and males matches those in our indifference calculation above, we really won’t know how which gender would most likely be indicated from our height measurement alone. More invasive tests might be required. :O

In general, what this example shows us is that combining background information can play a significant role in helping us make less biased judgments. Instead of thinking about just the distribution of a characteristic within a subpopulation with which we are most familiar, we need also to think about the distribution of the subpopulations that exhibit the characteristic we might choose to measure in order to make an inference about which subpopulation a specimen might belong. Not taking the full context into account of a situation is what often leads us to make bigoted socially toxic judgments, entertain superstitions, or allocate marketing and sales resources to markets that are least receptive to our product messaging.

In the next few installments on this discussion, I plan to dive a little more deeply in to the Bayesian reasoning process with increasingly more complex examples that will remain still rather simple to implement in R code. If you don’t come for the code, I do hope you entertain the thinking process that can help improve executive judgment and decision making.

LS0tCnRpdGxlOiAnQmF5ZXNpYW4gUmVhc29uaW5nOiBHZW5kZXIgSW5mZXJlbmNlIGZyb20gYSBTcGVjaW1lbiBNZWFzdXJlbWVudCcKYXV0aG9yOiAiUm9iZXJ0IEQuIEJyb3duIElJSSIKZGF0ZTogIjgvNC8yMDE3IgpvdXRwdXQ6CiAgaHRtbF9ub3RlYm9vazogZGVmYXVsdAogIGh0bWxfZG9jdW1lbnQ6IGRlZmF1bHQKLS0tCgpgYGB7ciBsaWJyYXJpZXMsIGVjaG8gPSBGQUxTRX0KbGlicmFyeShnZ3Bsb3QyKQpsaWJyYXJ5KHJlc2hhcGUyKQpgYGAKCmBgYHtyIHNldHVwLCBpbmNsdWRlPUZBTFNFfQprbml0cjo6b3B0c19jaHVuayRzZXQoZWNobyA9IFRSVUUpCmBgYAoKQmFjayB0byBbSW5jaXRlISBEZWNpc2lvbiBUZWNobm9sb2dpZXNdKGh0dHA6Ly93d3cuaW5jaXRlZGVjaXNpb250ZWNoLmNvbSkuCgpJbWFnaW5lIHRoYXQgd2UgaGF2ZSBhIHBvcHVsYXRpb24gb2Ygc29tZXRoaW5nIGNvbXBvc2VkIG9mIHR3byBzdWJzZXQgCnBvcHVsYXRpb25zIHRoYXQsIHdoaWxlIGRpc3RpbmN0IGZyb20gZWFjaCBvdGhlciwgc2hhcmUgYSBjb21tb24gY2hhcmFjdGVyaXN0aWMgCnRoYXQgY2FuIGJlIG1lYXN1cmVkIGFsb25nIHNvbWUga2luZCBvZiBzY2FsZS4gRnVydGhlcm1vcmUsIGxldCdzIGFzc3VtZSB0aGF0IAplYWNoIHN1YnNldCBwb3B1bGF0aW9uIGV4cHJlc3NlcyB0aGlzIGNoYXJhY3RlcmlzdGljIHdpdGggYSBmcmVxdWVuY3kgCmRpc3RyaWJ1dGlvbiB1bmlxdWUgdG8gZWFjaC4gSW4gb3RoZXIgd29yZHMsIGFsb25nIHRoZSBzY2FsZSBvZiBtZWFzdXJlbWVudCBmb3IgCnRoZSBjaGFyYWN0ZXJpc3RpYywgZWFjaCBzdWJzZXQgZGlzcGxheXMgdmFyeWluZyBsZXZlbHMgb2YgdGhlIGNoYXJhY3RlcmlzdGljIAphbW9uZyBpdHMgbWVtYmVycy4gTm93LCB3ZSBjaG9vc2UgYSBzcGVjaW1lbiBmcm9tIHRoZSBsYXJnZXIgcG9wdWxhdGlvbiBpbiBhbiAKdW5iaWFzZWQgbWFubmVyIGFuZCBtZWFzdXJlIHRoaXMgY2hhcmFjdGVyaXN0aWMgZm9yIHRoaXMgc3BlY2lmaWMgaW5kaXZpZHVhbC4gCkFyZSB3ZSBqdXN0aWZpZWQgaW4gaW5mZXJyaW5nIHRoZSBzdWJzZXQgbWVtYmVyc2hpcCBvZiB0aGUgc3BlY2ltZW4gYmFzZWQgb24gCnRoaXMgbWVhc3VyZW1lbnQgYWxvbmU/IEJheWUncyBydWxlIChvciB0aGVvcmVtKSwgc29tZXRoaW5nIHlvdSBtYXkgaGF2ZSBoZWFyZCAKYWJvdXQgaW4gdGhpcyBhZ2Ugb2YgZXhwbG9kaW5nIGRhdGEgYW5hbHl0aWNzLCB0ZWxscyB1cyB0aGF0IHdlIGNhbiBiZSBzbyAKanVzdGlmaWVkIGFzIGxvbmcgYXMgd2UgYXNzaWduIGEgcHJvYmFiaWxpdHkgKG9yIGRlZ3JlZSBvZiBiZWxpZWYpIHRvIG91ciAKaW5mZXJlbmNlLiBUaGUgZm9sbG93aW5nIGRpc2N1c3Npb24gcHJvdmlkZXMgYW4gaW50ZXJlc3Rpbmcgd2F5IG9mIHVuZGVyc3RhbmRpbmcKdGhlIHByb2Nlc3MgZm9yIGRvaW5nIHRoaXMuIE1vcmUgaW1wb3J0YW50bHksIEkgcHJlc2VudCBob3cgQmF5ZSdzIHRoZW9yZW0gaGVscHMKdXMgb3ZlcmNvbWUgYSBjb21tb24gdGhpbmtpbmcgZmFpbHVyZSBhc3NvY2lhdGVkIHdpdGggbWFraW5nIGluZmVyZW5jZXMgZnJvbSBhbiAKaW5jb21wbGV0ZSB0cmVhdG1lbnQgb2YgYWxsIHRoZSBpbmZvcm1hdGlvbiB3ZSBzaG91bGQgdXNlLiBJJ2xsIHVzZSBhIGJpdCBvZiBhIApmYW5jaWZ1bCBleGFtcGxlIHRvIGNvbnZleSB0aGlzIHVuZGVyc3RhbmRpbmcgYWxvbmcgd2l0aCBzaG93aW5nIHRoZSBhc3NvY2lhdGVkIApjYWxjdWxhdGlvbnMgaW4gdGhlIFIgcHJvZ3JhbW1pbmcgbGFuZ3VhZ2UuCgpgYGB7fQpOb3RlOiBJZiB5b3UgZG9uJ3QgY2FyZSBhYm91dCB0aGUgUiBjb2RlIGF0IGFsbCBhbmQganVzdCB3YW50IHRvIGZvbGxvdyB0aGUKYmFzaWMgcmVhc29uaW5nLCB5b3UgY2FuIGhpZGUgYWxsIHRoZSBjb2RlIGF0IG9uY2UgYnkgc2VsZWN0aW5nIENvZGUvSGlkZSBBbGwKQ29kZSBmcm9tIHRoZSBkcm9wIGRvd24gbGlzdCBpbiB0aGUgdG9wIHJpZ2h0IGNvcm5lciBvZiB0aGlzIHBhZ2UuIElmIHlvdSB3YW50CnRvIHBsYXkgYXJvdW5kIHdpdGggdGhlIFIgY29kZSB5b3Vyc2VsZiwgeW91IGNhbiBkb3dubG9hZCB0aGUgY29kZSBieSBzZWxlY3RpbmcKQ29kZS9Eb3dubG9hZCBSbWQuCmBgYAoKU3VwcG9zZSB3ZSBhcmUgYWxpZW5zIGZyb20gYW5vdGhlciBwbGFuZXQgY29uZHVjdGluZyBzY2llbnRpZmljIHJlc2VhcmNoIG9uIHRoaXMKc3RyYW5nZSBncm91cCBvZiBiaXBlZGFsIG9yZ2FuaXNtcyBjYWxsZWQgaHVtYW5zLiBIdW1hbnMgYXJlIHNleHVhbGx5IGRpbW9ycGhpYywKcHJlc2VudGluZyB0aGVtc2VsdmVzIGFzIGZlbWFsZSBhbmQgbWFsZSBnZW5kZXJzLiBFYWNoIGdlbmRlciBleHByZXNzZXMgaXRzIApoZWlnaHQgYWNyb3NzIGEgZGlzdHJpYnV0aW9uLCBhbmQgdGhlIG1lYW5zIG9mIGVhY2ggZ2VuZGVyIHBvcHVsYXRpb24gYXJlIApkaXN0aW5jdC4gWWVhcnMgb2YgY29sbGVjdGluZyBkYXRhIGFib3V0IHRoZSBodW1hbnMgcmV2ZWFscyB0aGF0IHRoZSBtZWFuIGhlaWdodApmb3IgYWR1bHQgZmVtYWxlcyBpcyA1LjMzIGJsaWtzOyBtYWxlcywgNS44MyBibGlrcy4gKCJCbGlrcyIgYXJlIG91ciB1bml0IApkaXN0YW5jZSBtZWFzdXJlbWVudCwgbGlrZSBmZWV0IG9yIG1ldGVycy4pIEJvdGggZ2VuZGVycyBzaG93IGEgc2ltaWxhciAKYmVsbC1zaGFwZWQgZGlzdHJpYnV0ZWQgKGkuZS4sIE5vcm1hbCBkaXN0cmlidXRpb24pIHZhcmlhdGlvbiBhcm91bmQgdGhlaXIgbWVhbiAKaGVpZ2h0cyB3aXRoIGEgc3RhbmRhcmQgZGV2aWF0aW9uIG9mIDAuMi4KCmBgYHtyIGhlaWdodF9zdGF0c30KZ2VuZGVyIDwtIGMoIkZlbWFsZSIsICJNYWxlIikKYWR1bHQubSA8LSBjKDUuMzMsIDUuODMpICMgTWVhbiBhZHVsdCBoZWlnaHQgYnkgZ2VuZGVyLgphZHVsdC5zZCA8LSAwLjIgIyBBZHVsdCBoZWlnaHQgc3RhbmRhcmQgZGV2aWF0aW9uLiBDb25zdGFudCBhY3Jvc3MgZ2VuZGVyLgpgYGAKCkFub3RoZXIgcGllY2Ugb2YgaW5mb3JtYXRpb24gdGhhdCB3ZSBoYXZlIGNvbGxlY3RlZCBhYm91dCB0aGUgaHVtYW4gcG9wdWxhdGlvbiAKaXMgdGhlIGJhc2UgcmF0ZSBmcmVxdWVuY3ksIG9yIHRoZSBwcm9wb3J0aW9uLCBvZiBlYWNoIHN1YnBvcHVsYXRpb24gaW4gdGhlIAp0b3RhbCBwb3B1bGF0aW9uLiBUaGVzZSBwcm9wb3J0aW9ucyBwcm92aWRlIHNvbWUgdmFsdWFibGUgaW5mb3JtYXRpb24sIApwYXJ0aWN1bGFybHkgaWYgdGhlIHByb3BvcnRpb24gb2Ygb25lIHN1YnBvcHVsYXRpb24gaXMgaXMgbGFyZ2VyIHRoYW4gdGhlIApvdGhlcnMuIEZvciBleGFtcGxlLCBhbW9uZyB0aGUgWmFyZ29ucywgY2hhcnRyZXVzZSBnaWxscyBzaG93IHVwIGluIDg1JSBvZiB0aGUgCnBvcHVsYXRpb24gY29tcGFyZWQgdG8gMTUlIGZvciB0aG9zZSB3aXRoIG1hZ2VudGEgZ2lsbHMuIElmIHdlIHJhbmRvbWx5IHNlbGVjdCBhCm1lbWJlciBvZiB0aGUgWmFyZ29uIHBvcHVsYXRpb24gZm9yIG91ciB6b28gZXhoaWJpdCwgd2UgbW9zdCBsaWtlbHkgd2lsbCBzZWxlY3QgCm9uZSB3aXRoIGNoYXJ0cmV1c2UgZ2lsbHMuIEluIHRoZSBsYW5ndWFnZSBvZiBCYXllc2lhbiByZWFzb25pbmcsIHdlIGNhbiBjYWxsIAp0aGlzIGJhc2UgcmF0ZSB0aGUgcHJpb3IgcHJvYmFiaWxpdHkgKG9yIGRlZ3JlZSBvZiBiZWxpZWYpIHRoYXQgYSByYW5kb21seSAKc2VsZWN0ZWQgaW5kaXZpZHVhbCBmcm9tIHRoZSB0b3RhbCBwb3B1bGF0aW9uIHNob3VsZCBiZSBjbGFzc2lmaWVkIGFzIGEgbWVtYmVyIApvZiBvbmUgY2F0ZWdvcnkgb3IgYW5vdGhlci4gSW4gdGhlIGdlbmVyaWMgbm90YXRpb24sIHRoaXMgd291bGQgYmUKUHJvYihIeXBvdGhlc2lzKSwgYnV0IGluIHRoZSBjb250ZXh0IG9mIG91ciBjdXJyZW50IHNpdHVhdGlvbiwgd2UgbWlnaHQgd3JpdGUKUHJvYihHZW5kZXIpLiBIb2xkIG9uIHRvIHRoaXMgY29uY2VwdCwgYXMgd2Ugd2lsbCByZXR1cm4gdG8gaXQgaW4ganVzdCB0d28Kc2hha2VzIG9mIGEgcHJvdG9uIGRpc2xvY2F0b3IuCgpVbmZvcnR1bmF0ZWx5LCBhIHJlY2VudCB2aXJhbCBlcGlkZW1pYyBuZWFybHkgd2lwZWQgb3V0IHRoZSBwb3B1bGF0aW9uIG9mIG1hbGVzLCAKbGVhdmluZyB0aGUgZmVtYWxlcyBvdmVyLXJlcHJlc2VudGVkIGFzIDkwJSBvZiB0aGUgcG9wdWxhdGlvbiBhcyBjb21wYXJlZCB0byB0aGUKaGlzdG9yaWNhbGx5IGFwcHJveGltYXRlbHkgZXZlbiBkaXN0cmlidXRpb24gb2YgZmVtYWxlcyB0byBtYWxlcy4KCmBgYHtyIGdlbmRlcl9mcmVxfQojIFNldCB1cCBhIHZlY3RvciB0byBjb250YWluIHRoZSBjdXJyZW50IFByb2IoR2VuZGVyKS4KYmFzZS5yYXRlLmZyZXEuZmVtIDwtIDAuOSAjIFRoZSBmZW1hbGUgYmFzZSByYXRlIGZyZXF1ZW5jeS4KYmFzZS5yYXRlLmZyZXEgPC0gYyhiYXNlLnJhdGUuZnJlcS5mZW0sIDEgLSBiYXNlLnJhdGUuZnJlcS5mZW0pICMgKGZlbWFsZXMsIG1hbGVzKQpgYGAKCkltYWdpbmUgdGhhdCBvbiBhbiBleHBsb3JhdG9yeSBtaXNzaW9uIHRvIEVhcnRoLCB3ZSBkaXNjb3ZlciBhbiBpbmp1cmVkIGh1bWFuIHRvCndob20gd2UgaW50ZW5kIHRvIGFwcGx5IG1lZGljYWwgYXR0ZW50aW9uIGJlZm9yZSB3ZSBzaGlwIGl0IG9mZiB0byBvdXIgZ2FsYWN0aWMgCnpvbzsgaG93ZXZlciwgZHVlIHRvIGl0cyBpbmp1cmllcyBhbmQgaXRzIGV4dHJlbWUgZW1vdGlvbmFsIHN0YXRlLCB3ZSBjYW4ndCAKZGV0ZXJtaW5lIGl0cyBnZW5kZXIuIFRoZSBvbmx5IGRhdGEgd2UgY2FuIG9idGFpbiBpcyBpdHMgaGVpZ2h0IChZZXMsIHdlIGNhbgp0cmF2ZWwgYWNyb3NzIHZhc3QgZGlzdGFuY2VzIG9mIHNwYWNlLCBidXQgd2UgaGF2ZSB0byB1c2UgdGhpcyBjcnVkZSBtZWFucyB0bwptYWtlIGEgbWVhc3VyZW1lbnQuIEJlYXIgd2l0aCBtZS4gVGhpcyBpcyBqdXN0IGFuIGV4YW1wbGUsIG5vdCBhIHNjaS1maSBub3ZlbC4pLAp3aGljaCB3ZSBvYnNlcnZlIHRvIGJlIDUuNjUgYmxpa3MuIEJhc2VkIG9uIHRoaXMgb25lIHNhbXBsZSBvZiBtZWFzdXJlZAppbmZvcm1hdGlvbiwgd2UgbXVzdCBpbmZlciB0aGUgZ2VuZGVyIG9mIHRoZSBzcGVjaW1lbiBiZWZvcmUgd2UgcHJvY2VlZC4gSG93IGNhbgp3ZSB1c2UgdGhlIGluZm9ybWF0aW9uIHdlIGhhdmUgb24gaGFuZCB0byBtYWtlIGEgcmF0aW9uYWwgaW5mZXJlbmNlPwoKYGBge3Igc3BlY19oZWlnaHR9CiMgVGhlIG1lYXN1cmVkIGhlaWdodCBvZiBvdXIgdW5pZGVudGlmaWVkIHNwZWNpbWVuLgojIHVuaXRzID0gQmxpa3MKc3BlY2ltZW4uaGVpZ2h0IDwtIDUuNjUKYGBgCgpGaXJzdCwgd2UgbmVlZCB0byBkZXRlcm1pbmUgdGhlIHByb2JhYmxlIGhlaWdodCBvZiBhbnkgbWVtYmVyIG9mIGEgZ2VuZGVyIGJhc2VkIApvbiBvdXIgY3VycmVudCBjaGFyYWN0ZXJpemF0aW9uIG9mIHRoZSBnZW5kZXIgcG9wdWxhdGlvbnMuIEFnYWluLCBpbiB0aGUgCmxhbmd1YWdlIG9mIEJheWVzaWFuIHJlYXNvbmluZywgd2UgY2FsbCB0aGlzIHRoZSBsaWtlbGlob29kIGZ1bmN0aW9uLCB3aGljaCAKdGVsbHMgdXMgdGhlIHByb2JhYmlsaXR5IG9mIG9ic2VydmluZyBzb21lIGtpbmQgb2YgZXZpZGVuY2UsIGxpa2UgYSBtZWFzdXJlbWVudCwKdGhhdCBpcyBjb25kaXRpb25hbCBvbiB0aGUgc2VsZWN0aW9uIG9mIGEgZ2l2ZW4gcG9wdWxhdGlvbiBvciBjb25kaXRpb25hbCBvbiBhIApnaXZlbiBoeXBvdGhlc2lzIGJlaW5nIHRoZSBjYXNlLiBXZSB1c3VhbGx5IHdyaXRlIHRoaXMgYXMgClByb2IoRXZpZGVuY2V8SHlwb3RoZXNpcykuIEluIHRoZSBjb250ZXh0IG9mIG91ciBwcm9ibGVtLCB3ZSBtaWdodCB3cml0ZSAKUHJvYihIZWlnaHR8R2VuZGVyKS4KClVzaW5nIHRoZSBgYGBkbm9ybSguLi4pYGBgIGZ1bmN0aW9uLCB3ZSBjYW4gY2FsY3VsYXRlIHRoZSBwcm9iYWJpbGl0eQpkaXN0cmlidXRpb24gYWNyb3NzIHRoZSBoZWlnaHQgZG9tYWluIGZvciBlYWNoIGdlbmRlci4gTm90ZSB0aGF0IHRoZQpgYGBkbm9ybSguLi4pYGBgIGZ1bmN0aW9uIHByb2R1Y2VzIHRoZSBwcm9iYWJpbGl0eSBkZW5zaXR5IGF0IGEgZ2l2ZW4gcG9pbnQuCgpgYGB7ciBnZW5kZXJfZGVuc2l0eX0KIyBIZXJlIHdlIHNldCB1cCBhbiBpbmRleCBmb3IgdGhlIGhlaWdodCBkb21haW4sIHdpdGggYSBzdGVwIHNpemUgb2YKIyAwLjA0IGJsaWtzLgpzdGVwLnNpemUgPC0gLjA0CmhlaWdodCA8LSBzZXEoNCwgNywgYnkgPSBzdGVwLnNpemUpCgojIENhbGN1bGF0ZSB0aGUgaGVpZ2h0IHByb2JhYmlsaXR5IGRpc3RyaWJ1dGlvbiBmb3IgZWFjaCBnZW5kZXIgYWNyb3NzIHRoZQojIGhlaWdodCBkb21haW4uIEluIHRoaXMgY2FzZSwgdGhlIHJlc3VsdCB3aWxsIGJlIGFuIGFycmF5IG9mIHNoYXBlICgyIHgKIyBoZWlnaHQpLiBXZSBhcHBseSB0aGUgdHJhbnNwb3NlIGZ1bnRpb24gdG8gdGhpcyBhcnJheSB0byByZW9yaWVudCBpdCB0bwojIChoZWlnaHQgeCAyKS4KbGlrZWxpaG9vZC5oZWlnaHQgPC0gdChzYXBwbHkoaGVpZ2h0LCBmdW5jdGlvbihoKQogIGRub3JtKGgsIGFkdWx0Lm0sIGFkdWx0LnNkKSkpCiAgICAgICAgICAgICAgICAgICAgIAojIFJlY2FzdCB0aGUgYXJyYXkgYXMgYSBkYXRhIGZyYW1lIGluIHByZXBhcmF0aW9uIGZvciBwbG90dGluZyBpbiBnZ3Bsb3QuCmRmLmxpa2VsaWhvb2QuaGVpZ2h0IDwtIGRhdGEuZnJhbWUoaGVpZ2h0LCBsaWtlbGlob29kLmhlaWdodCkKY29sbmFtZXMoZGYubGlrZWxpaG9vZC5oZWlnaHQpIDwtIGMoIkhlaWdodCIsIGdlbmRlcikKYGBgCgpJZiB3ZSBwbG90IHRoaXMgZnVuY3Rpb24sIHdlIG9ic2VydmUgdGhlIHByb2JhYmlsaXR5IGRpc3RyaWJ1dGlvbiBhY3Jvc3MgdGhlCmhlaWdodCBkb21haW4uIEFnYWluLCB3ZSBhbHNvIHJlZmVyIHRvIHRoaXMgYXMgdGhlIGxpa2VsaWhvb2QgZnVuY3Rpb24uCgpgYGB7ciBwbG90X2dlbmRlcl9kZW5zaXR5fQojIFJlY2FzdCB0aGUgcmVjdGFuZ3VsYXIgZGF0YSBmcmFtZSB0byBhIHJlbGF0aW9uYWwgZm9ybWF0IHN1Y2ggdGhhdCB0aGUgaGVpZ2h0CiMgaW5kZXggaXMgdGhlIGxlZnRtb3N0IHZhcmlhYmxlLgptZWx0LmRmLmxpa2VsaWhvb2QuaGVpZ2h0IDwtIG1lbHQoZGYubGlrZWxpaG9vZC5oZWlnaHQsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaWQudmFycyA9ICJIZWlnaHQiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgIHZhcmlhYmxlLm5hbWUgPSAiR2VuZGVyIiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICB2YWx1ZS5uYW1lID0gIlByb2JhYmlsaXR5IikKCmhlaWdodC5kaXN0ciA8LSBnZ3Bsb3QobWVsdC5kZi5saWtlbGlob29kLmhlaWdodCwKICAgICAgICAgICAgICAgICAgICAgICBhZXMoCiAgICAgICAgICAgICAgICAgICAgICAgeCA9IEhlaWdodCwKICAgICAgICAgICAgICAgICAgICAgICB5ID0gUHJvYmFiaWxpdHksCiAgICAgICAgICAgICAgICAgICAgICAgZ3JvdXAgPSBHZW5kZXIsCiAgICAgICAgICAgICAgICAgICAgICAgY29sb3VyID0gR2VuZGVyCiAgICAgICAgICAgICAgICAgICAgICAgKSkgKwogIGdlb21fbGluZShhZXMoZ3JvdXAgPSBHZW5kZXIsIGNvbG91ciA9IEdlbmRlcikpICsKICBnZW9tX3ZsaW5lKHhpbnRlcmNlcHQgPSBzcGVjaW1lbi5oZWlnaHQpICsKICB4bGFiKCJIZWlnaHQgW2JsaWtzXSIpICsKICB5bGFiKCJQcm9iYWJpbGl0eSBEZW5zaXR5IikgKwogIGdndGl0bGUoIkhlaWdodCBEaXN0cmlidXRpb24gb2YgSHVtYW4gQWR1bHRzIiwgc3VidGl0bGUgPSAiUHJvYihIZWlnaHR8R2VuZGVyKSIpCnByaW50KGhlaWdodC5kaXN0cikKYGBgCgpJdCdzIHRlbXB0aW5nIHRvIGxvb2sgYXQgdGhlc2UgZGlzdHJpYnV0aW9ucyBhbmQgY29uY2x1ZGUgdGhhdCB0aGUgcHJvcGVyIAppbmZlcmVuY2UgYWJvdXQgdGhlIGdlbmRlciBvZiBvdXIgc3BlY2ltZW4gd291bGQgYmUgdGhlIHBvcHVsYXRpb24gd2l0aCB0aGUgCmdyZWF0ZXN0IHByb2JhYmlsaXR5IGF0IHRoZSBzcGVjaW1lbidzIG1lYXN1cmVkIGhlaWdodC4gQWZ0ZXIgYWxsLCB0aGUgaGVpZ2h0IG9mCmFkdWx0IGh1bWFucyBpcyBvbmUgb2YgdGhlIG1vc3QgZWFzaWx5IG9ic2VydmFibGUgYW5kIGF2YWlsYWJsZSBhc3BlY3RzIG9mIHRoaXMgCnN0cmFuZ2Ugc3BlY2llcyB0aGF0IHRlbmRzIHRvIGNvdmVyIHVwIGFsbCB0aGUgZGlzdGluZ3Vpc2hpbmcgZGltb3JwaGljIApjaGFyYWN0ZXJpc3RpY3Mgd2l0aCBzb21ldGhpbmcgdGhleSBjYWxsICJjbG90aGVzLiIgQSBxdWljayByZWZlcmVuY2UgdG8gb3VyCmludHVpdGlvbiBtaWdodCB0ZWxsIHVzIHRoYXQgaGVpZ2h0IGlzIGEgZ29vZCBtZWFzdXJlIHRvIG1ha2UgYW4gaW5mZXJlbmNlIGFib3V0CmdlbmRlci4gSW4gdGhpcyBjYXNlLCB3ZSBtZWFzdXJlZCB0aGUgaGVpZ2h0IG9mIG91ciBzcGVjaW1lbiBhdCBgYGByIHNwZWNpbWVuLmhlaWdodGBgYApibGlrcyAodGhlIHZlcnRpY2FsIGJsYWNrIGxpbmUpLiBCYXNlZCBvbiB0aGlzIGhlaWdodCBjb21wYXJlZCB0byB0aGUKZGlzdHJpYnV0aW9ucywgd2UgbWlnaHQgaW5mZXIgdGhhdCB0aGUgc3BlY2ltZW4gaXMgbWFsZSBzaW5jZSB0aGUgbGlrZWxpaG9vZCBhdApgYGByIHNwZWNpbWVuLmhlaWdodGBgYCBibGlrcyBpcyBoaWdoZXIgZm9yIG1hbGVzIHRoYW4gZmVtYWxlcy4KClVuZm9ydHVuYXRlbHksIHRoaXMgYXBwcm9hY2ggbGVhdmVzIG91dCB0aGUgaW5mb3JtYXRpb24gd2UgaGF2ZSBhYm91dCB0aGUgCmN1cnJlbnQgcHJldmFsZW5jZSAob3IgdGhlIGJhc2UgcmF0ZSBmcmVxdWVuY3kgb3IgcHJpb3IgZGVncmVlIG9mIGJlbGllZikgb2YgdGhlCmdlbmRlcnMgaW4gdGhlIHBvcHVsYXRpb24uIFJlbWVtYmVyIHRoZSBleGFtcGxlIG9mIHRoZSBaYXJnb25zPyBXZSB0eXBpY2FsbHkKY291bGQgaW5mZXIgdGhlIHN1YnBvcHVsYXRpb24gYmFzZWQgb24gdGhlIG9ic2VydmF0aW9uIG9mIGEgZG9taW5hbnRseSBleHByZXNzZWQKY2hhcmFjdGVyaXN0aWMuIEJ1dCBpbiB0aGlzIGNhc2UsIHdlIGhhdmUgdHdvIGNoYXJhY3RlcmlzdGNzIHdlIGNvdWxkIGNvbnNpZGVyLAphbmQgdGhleSBzZWVtIHRvIHdvcmsgYWdhaW5zdCBlYWNoIG90aGVyOyB0aGF0IGlzLCBhIHJhbmRvbWx5IHNlbGVjdGVkIGFkdWx0Cmh1bWFuIGFmdGVyIHRoZSB2aXJhbCBwb3N0LWFwb2NhbHlwc2UgaXMgbm93IG1vc3RseSBsaWtlbHkgZmVtYWxlLCBidXQgdGhlCmhlaWdodCBjaGFyYWN0ZXJpc3RpYyBpbmZvcm1hdGlvbiB3ZSBoYXZlIHNlZW1zIHRvIGxlYWQgdXMgdG8gaW5mZXIgdGhhdCB3ZSBoYXZlCmEgbWFsZS4gQ2FuIHdlIGNvbWJpbmUgYm90aCBwaWVjZXMgb2YgaW5mb3JtYXRpb24gdG8gYWRqdXN0IHRoZSBzdHJlbmd0aCBvZiBvdXIKaW5jbGluYXRpb24gaW4gYSByZWFzb25hYmxlIG1hbm5lcj8gRm9ydHVuYXRlbHksIGFsbCB0aGF0IHdlIGhhdmUgdG8gZG8gdG8KaW5jbHVkZSBib3RoIHBpZWNlcyBvZiBpbmZvcm1hdGlvbiBpbiBvdXIganVkZ21lbnQgaXMgdG8gd2VpZ2h0IHRoZSBsaWtlbGlob29kCm9mIGhlaWdodCBmb3IgYSBnaXZlbiBwb3B1bGF0aW9uIHByb3BvcnRpb25hbGx5IHRvIHRoZSBwcmV2YWxlbmNlIG9mIHRoZQpwb3B1bGF0aW9uLiBXZSBjYW4gYWNoaWV2ZSB0aGlzIGJ5IG11bHRpcGx5aW5nIHRoZSBsaWtsaWhvb2QgZnVuY3Rpb24gYnkgdGhlIApiYXNlIHJhdGUgcHJvYmFiaWxpdHkuCgpgYGB7ciBsaWtlbGlob29kX2hlaWdodF9nZW5kZXJ9Cmxpa2VsaWhvb2QuaGVpZ2h0LmdlbmRlciA8LSB0KGJhc2UucmF0ZS5mcmVxICogdChsaWtlbGlob29kLmhlaWdodCkpCgojIFJlY2FzdCB0aGUgYXJyYXkgYXMgYSBkYXRhIGZyYW1lIGluIHByZXBhcmF0aW9uIGZvciBwbG90dGluZyBpbiBnZ3Bsb3QuCmRmLmxpa2VsaWhvb2QuaGVpZ2h0LmdlbmRlciA8LSBkYXRhLmZyYW1lKGhlaWdodCwgbGlrZWxpaG9vZC5oZWlnaHQuZ2VuZGVyKQpjb2xuYW1lcyhkZi5saWtlbGlob29kLmhlaWdodC5nZW5kZXIpIDwtIGMoIkhlaWdodCIsIGdlbmRlcikKYGBgCgpgYGB7ciBwbG90X2NvbmRfaGVpZ2h0fQojIFJlY2FzdCB0aGUgcmVjdGFuZ3VsYXIgZGF0YSBmcmFtZSB0byBhIHJlbGF0aW9uYWwgZm9ybWF0IHN1Y2ggdGhhdCB0aGUgaGVpZ2h0CiMgaW5kZXggaXMgdGhlIGxlZnRtb3N0IHZhcmlhYmxlLgptZWx0LmRmLmxpa2VsaWhvb2QuaGVpZ2h0LmdlbmRlciA8LSBtZWx0KGRmLmxpa2VsaWhvb2QuaGVpZ2h0LmdlbmRlciwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpZC52YXJzID0gIkhlaWdodCIsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdmFyaWFibGUubmFtZSA9ICJHZW5kZXIiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgIHZhbHVlLm5hbWUgPSAiUHJvYmFiaWxpdHkiKQoKaGVpZ2h0LmRpc3RyLmdlbmRlciA8LSBnZ3Bsb3QobWVsdC5kZi5saWtlbGlob29kLmhlaWdodC5nZW5kZXIsCiAgICAgICAgICAgICAgICAgICAgICAgYWVzKAogICAgICAgICAgICAgICAgICAgICAgIHggPSBIZWlnaHQsCiAgICAgICAgICAgICAgICAgICAgICAgeSA9IFByb2JhYmlsaXR5LAogICAgICAgICAgICAgICAgICAgICAgIGdyb3VwID0gR2VuZGVyLAogICAgICAgICAgICAgICAgICAgICAgIGNvbG91ciA9IEdlbmRlcgogICAgICAgICAgICAgICAgICAgICAgICkpICsKICBnZW9tX2xpbmUoYWVzKGdyb3VwID0gR2VuZGVyLCBjb2xvdXIgPSBHZW5kZXIpKSArCiAgZ2VvbV92bGluZSh4aW50ZXJjZXB0ID0gc3BlY2ltZW4uaGVpZ2h0KSArCiAgeGxhYigiSGVpZ2h0IFtibGlrc10iKSArCiAgeWxhYigiUHJvYmFiaWxpdHkgRGVuc2l0eSIpICsKICBnZ3RpdGxlKCJQcm9iYWJpbGl0eS1XZWlnaHRlZCBIZWlnaHQgRGlzdHJpYnV0aW9uIG9mIEh1bWFuIEFkdWx0cyIsIHN1YnRpdGxlID0gIlByb2IoSGVpZ2h0fEdlbmRlcikgKiBQcm9iKEdlbmRlcikiKQpwcmludChoZWlnaHQuZGlzdHIuZ2VuZGVyKQpgYGAKCk5vdyB3ZSBvYnNlcnZlIHRoYXQgdGhlIHByb2JhYmlsaXR5IHdlaWdodGVkIGhlaWdodCBkaXN0cmlidXRpb25zIG1vc3QgbGlrZWx5IApvdWdodCB0byBjaGFuZ2Ugb3VyIGluZmVyZW5jZSBhYm91dCB0aGUgZ2VuZGVyIG9mIG91ciBzcGVjaW1lbi4gQWxsIHdlIG5lZWQgdG8gCmRvIG5vdyBpcyBmaW5kIHRoZSBjb25kaXRpb25hbCBsaWtlbGlob29kcyBhdCB0aGUgbWVhc3VyZWQgaGVpZ2h0IGFuZCBub3JtYWxpemUgCnRoZW0gd2l0aCB0aGUgbWFyZ2luYWwgcHJvYmFiaWxpdHkgYXQgdGhhdCBoZWlnaHQsIHdoZXJlIHRoZSBtYXJnaW5hbCBwcm9iYWJpbGl0eSBpcyBqdXN0IHRoZSBzdW0gb2YgdGhlIGNvbmRpdGlvbmFsIGxpa2VsaWhvb2RzIGFjcm9zcyBnZW5kZXJzIGF0IHRoZSBtZWFzdXJlZCBoZWlnaHQuCgpgYGB7cn0KY29uZGl0aW9uYWwubGlrZWxpaG9vZCA8LQogIGJhc2UucmF0ZS5mcmVxICogZG5vcm0oc3BlY2ltZW4uaGVpZ2h0LCBhZHVsdC5tLCBhZHVsdC5zZCkKCiMgY2FzdCBpbnRvIGFuIGFycmF5Lgpjb25kaXRpb25hbC5saWtlbGlob29kIDwtIGFycmF5KGNvbmRpdGlvbmFsLmxpa2VsaWhvb2QsIGRpbSA9IGMoMSwgMikpCmNvbG5hbWVzKGNvbmRpdGlvbmFsLmxpa2VsaWhvb2QpIDwtIGdlbmRlcgoKYGBgCgpUaGUgY29uZGl0aW9uYWwgbGlrZWxpaG9vZHMgZm9yIGVhY2ggZ2VuZGVyIGF0IHRoZSBtZWFzdXJlZCBoZWlnaHQgaXM6CgpgYGB7ciwgZWNobz1GQUxTRX0KcHJpbnQocm91bmQoY29uZGl0aW9uYWwubGlrZWxpaG9vZFsxLCBdLCAyKSkKYGBgCgpgYGB7cn0KIyBUaGUgbWFyZ2luYWwgcHJvYmFiaWxpdHkgaXMgdGhlIHRvdGFsIHByb2JhYmlsaXR5IG9mIHRoZSBwcm9kdWN0IG9mIHRoZSBwcmlvcgojIGFuZCB0aGUgbGlrZWxpaG9vZCBhdCB0aGUgc3BlY2lmaWVkIG1lYXN1cmVkIG9ic2VydmF0aW9uLgptYXJnaW5hbC5wcm9iIDwtIHN1bShjb25kaXRpb25hbC5saWtlbGlob29kKQpgYGAKClRoZSBtYXJnaW5hbCBwcm9iYWJpbGl0eSwgd2hpY2ggaXMganVzdCB0aGUgc3VtIG9mIHRoZSB0d28gcHJvYmFiaWxpdGllcyBpbiBvdXIgY29uZGl0aW9uYWwgbGlrZWxpaG9vZCBpczoKCmBgYHtyLCBlY2hvPUZBTFNFfQpwcmludChyb3VuZChtYXJnaW5hbC5wcm9iLCAyKSkKYGBgCgpOb3cgd2UgZGl2aWRlIHRoZSBjb25kaXRpb25hbCBsaWtlbGlob29kIHZhbHVlcyBieSB0aGUgbWFyZ2luYWwgcHJvYmFiaWxpdHkgdmFsdWUuCgpgYGB7cn0KIyBUaGUgcHJvYmFiaWxpdHkgKHJhdGlvbmFsIGluZmVyZW5jZSkgb2YgZ2VuZGVyIGFzc29jaWF0ZWQgd2l0aCBzYW1wbGVkIGhlaWdodC4KcG9zdGVyaW9yLnByb2IuZ2VuZGVyIDwtIGNvbmRpdGlvbmFsLmxpa2VsaWhvb2QgLyBtYXJnaW5hbC5wcm9iCmBgYAoKV2Ugc2hvdWxkIGFjY2VwdCBub3cgYXMgb3VyIHVwZGF0ZWQgYmVsaWVmIChpLmUuLCBwb3N0ZXJpb3IgcHJvYmFiaWxpdHkpIGFib3V0IHRoZSBnZW5kZXIgb2Ygb3VyIHNwZWNpbWVuIHRvIGJlOgoKYGBge3IsIGVjaG89RkFMU0V9CnByaW50KHJvdW5kKHBvc3Rlcmlvci5wcm9iLmdlbmRlclsxLCBdLCAyKSkKYGBgCgpPdXIgZmluYWwgb2JzZXJ2YXRpb24gaXMgdGhhdCBgYGByIGdlbmRlclt3aGljaC5tYXgocG9zdGVyaW9yLnByb2IuZ2VuZGVyKV1gYGAgCnNob3VsZCBiZSBvdXIgcmF0aW9uYWwgaW5mZXJlbmNlIGZvciB0aGUgZ2VuZGVyIG9mIG91ciBzcGVjaW1lbiBiYXNlZCBvbiB0aGUgCnByaW9yIHByb2JhYmlsaXR5IHdlaWdodGVkIGxpa2VsaWhvb2Qgb2YgaGVpZ2h0IGFzIGEgZnVuY3Rpb24gb2YgZ2VuZGVyLgoKSW4gc2hvcnQsIHdoYXQgd2UgY2FsY3VsYXRlZCB3YXMKYGBgcHJvYihHZW5kZXJ8SGVpZ2h0KT1wcm9iKEdlbmRlcikqcHJvYihIZWlnaHR8R2VuZGVyKS8obWFyZ2luYWxfcHJvYmFiaWxpdHkpYGBgCndoZXJlIHRoZSBtYXJnaW5hbCBwcm9iYWJpbGl0eSBpcwpgYGBzdW0ocHJvYihHZW5kZXIpICogcHJvYihIZWlnaHR8R2VuZGVyKSlgYGAKVGhlbiB3ZSBzZWxlY3RlZCB0aGUgZ2VuZGVyIGFyZ3VtZW50IHdpdGggdGhlIGhpZ2hlc3QgcHJvYmFiaWxpdHkuCgpUaGUgdGVybQpgYGBzdW0ocHJvYihHZW5kZXIpICogcHJvYihIZWlnaHR8R2VuZGVyKSlgYGAKaXMgYSBub3JtYWxpemluZyBmYWN0b3IuIE9uY2UgdGhlIHF1b3RpZW50IGhhcyBiZWVuIGZvdW5kLCB0aGUgc3VtIG9mCnByb2JhYmlsaXRpZXMgc2hvdWxkIGVxdWFsIHRvIDEuCgpXaXRoIHJlZ2FyZCB0byBvdXIgZXhhbXBsZSwgdGhpcyBleGVyY2lzZSBoYXMgc2hvd24gdXMgdGhhdCB0YWxsIGZlbWFsZXMgYW1vbmcgCnRoZSBwb3B1bGF0aW9uIG9mIGFsbCBhZHVsdCBmZW1hbGVzIGFyZSBzb21ld2hhdCByYXJlLiBIb3dldmVyLCB0YWxsIGZlbWFsZXMgCmFtb25nIHRoZSBlbnRpcmUgcG9zdC1hcG9jYWx5cHRpYyBwb3B1bGF0aW9uIG9mIGFkdWx0IGh1bWFucyAoaW4gd2hpY2ggZmVtYWxlcwphcmUgdGhlIG1vc3QgcHJldmFsZW50IHN1YnBvcHVsYXRpb24pIGFyZSBtb3JlIHByZXZhbGVudCB0aGFuIGp1c3QtYmVsb3ctYXZlcmFnZQpoZWlnaHQgbWFsZXMsIGV2ZW4gdGhvdWdoIHN1Y2ggbWFsZXMgd291bGQgYmUgbW9yZSBwcmV2YWxlbnQgdGhhbiB0YWxsIGZlbWFsZXMKaWYgdGhlIHRoZSBzdWJwb3B1bGF0aW9ucyB3ZXJlIG1vcmUgZXF1YWwgaW4gc2l6ZS4gVGhlcmVmb3JlLCBmaW5kaW5nIGEgdGFsbAppbmRpdmlkdWFsIHNob3VsZCBsZWFkIHVzIHRvIGluZmVyLCBnaXZlbiBubyBvdGhlciBpbmZvcm1hdGlvbiBhYm91dCB0aGVtLCB0aGF0CnRoZSBpbmRpdmlkdWFsIGlzIG1vc3QgbGlrZWx5IGZlbWFsZS4gQ29tYmluaW5nIGJvdGggcGllY2VzIG9mIGluZm9ybWF0aW9uCmFsbG93cyB1cyB0byB0ZW1wZXIgb3VyIGludHVpdGlvbiB0aGF0IHdvdWxkIGhhdmUgYmVlbiBiYXNlZCBvbiBvbmx5IG9uZSBvZiB0aGUKYXZhaWxhYmxlIHBpZWNlcyBvZiBpbmZvcm1hdGlvbi4KCkFuIGludGVyZXN0aW5nIHF1ZXN0aW9uIHRoYXQgbWlnaHQgYXJpc2UgZnJvbSB0aGVzZSBpbnNpZ2h0cyBpczogd2hhdCBpcyB0aGUKYmFzZSByYXRlIGZyZXF1ZW5jeSBvZiB0aGUgcG9wdWxhdGlvbnMgb2YgZmVtYWxlcyBhbmQgbWFsZXMgdGhhdCBtaWdodCBpbnN0aWxsCmNvbXBsZXRlIGFtYmlndWl0eSBpbiB1cyBhYm91dCB0aGUgZ2VuZGVyIG9mIHRoZSBzcGVjaW1lbiBiYXNlZCBvbiB0aGUgbWVhc3VyZWQKaGVpZ2h0IGFuZCB0aGUgbGlrZWxpaG9vZCBoZWlnaHQgZnVuY3Rpb25zIG9mIGVhY2ggZ2VuZGVyPyBUaGUgYW5zd2VyIGlzIHNpbXBsZS4KQWxsIHdlIGhhdmUgdG8gZG8gaXMgcmVjb2duaXplIHRoYXQKCmBgYHt9CmJhc2UucmF0ZS5mcmVxLmZlbSAqIEZlbWFsZV9MaWtlbGlob29kfHNwZWNpbWVuLmhlaWdodCA9ICgxIC0gYmFzZS5yYXRlLmZyZXEuZmVtKSAqIE1hbGVfTGlrZWxpaG9vZHxzcGVjaW1lbi5oZWlnaHQKYGBgCgphbmQgdGhlbiBzb2x2ZSBmb3IgYGBgYmFzZS5yYXRlLmZyZXEuZmVtYGBgLiBMZXQncyBjYWxsIHRoaXMgc3BlY2lhbCBiYXNlIHJhdGUKZnJlcXVlbmN5IGBgYGluZGlmLmJhc2UucmF0ZS5mcmVxLmZlbWBgYC4gUmVhcnJhbmdpbmcgd2UgZ2V0CgpgYGB7fQpMaWtlbGlob29kLnJhdGlvID0gTWFsZV9MaWtlbGlob29kfHNwZWNpbWVuLmhlaWdodCAvIEZlbWFsZV9MaWtlbGlob29kfHNwZWNpbWVuLmhlaWdodAppbmRpZi5iYXNlLnJhdGUuZnJlcS5mZW0gPSBMaWtlbGlob29kLnJhdGlvIC8gKDEgKyBMaWtlbGlob29kLnJhdGlvKQpgYGAKCmBgYHtyfQojIHJlZm9ybSBleGFtcGxlIGVxdWF0aW9ucyBpbnRvIFIgc3RhdGVtZW50cwpsaWtlbGlob29kLnJhdGlvIDwtIGRub3JtKHNwZWNpbWVuLmhlaWdodCwgYWR1bHQubVsyXSwgYWR1bHQuc2QpIC8gZG5vcm0oc3BlY2ltZW4uaGVpZ2h0LCBhZHVsdC5tWzFdLCBhZHVsdC5zZCkKaW5kaWYuYmFzZS5yYXRlLmZyZXEuZmVtIDwtIGxpa2VsaWhvb2QucmF0aW8gLyAoMSArIGxpa2VsaWhvb2QucmF0aW8pCmBgYAoKYGBge3IsIGVjaG89RkFMU0V9CiMgY2FzdCBpbnRvIGFuIGFycmF5LgppbmRpZi5iYXNlLnJhdGUuZnJlcSA8LSBhcnJheShjKGluZGlmLmJhc2UucmF0ZS5mcmVxLmZlbSwgMSAtIGluZGlmLmJhc2UucmF0ZS5mcmVxLmZlbSksIGRpbSA9IGMoMSwgMikpCmNvbG5hbWVzKGluZGlmLmJhc2UucmF0ZS5mcmVxKSA8LSBnZW5kZXIKCnByaW50KHJvdW5kKGluZGlmLmJhc2UucmF0ZS5mcmVxWzEsIF0sIDIpKQpgYGAKVGhlcmVmb3JlLCBpZiB3ZSBmaW5kIG91cnNlbHZlcyBpbiBhIHNpdHVhdGlvbiBpbiB3aGljaCB0aGUgcHJvcG9ydGlvbiBvZgpmZW1hbGVzIGFuZCBtYWxlcyBtYXRjaGVzIHRob3NlIGluIG91ciBpbmRpZmZlcmVuY2UgY2FsY3VsYXRpb24gYWJvdmUsIHdlIHJlYWxseQp3b24ndCBrbm93IGhvdyB3aGljaCBnZW5kZXIgd291bGQgbW9zdCBsaWtlbHkgYmUgaW5kaWNhdGVkIGZyb20gb3VyIGhlaWdodAptZWFzdXJlbWVudCBhbG9uZS4gTW9yZSBpbnZhc2l2ZSB0ZXN0cyBtaWdodCBiZSByZXF1aXJlZC4gOk8KCkluIGdlbmVyYWwsIHdoYXQgdGhpcyBleGFtcGxlIHNob3dzIHVzIGlzIHRoYXQgY29tYmluaW5nIGJhY2tncm91bmQgaW5mb3JtYXRpb24gCmNhbiBwbGF5IGEgc2lnbmlmaWNhbnQgcm9sZSBpbiBoZWxwaW5nIHVzIG1ha2UgbGVzcyBiaWFzZWQganVkZ21lbnRzLiBJbnN0ZWFkIG9mCnRoaW5raW5nIGFib3V0IGp1c3QgdGhlIGRpc3RyaWJ1dGlvbiBvZiBhIGNoYXJhY3RlcmlzdGljIHdpdGhpbiBhIHN1YnBvcHVsYXRpb24gCndpdGggd2hpY2ggd2UgYXJlIG1vc3QgZmFtaWxpYXIsIHdlIG5lZWQgYWxzbyB0byB0aGluayBhYm91dCB0aGUgZGlzdHJpYnV0aW9uIG9mCnRoZSBzdWJwb3B1bGF0aW9ucyB0aGF0IGV4aGliaXQgdGhlIGNoYXJhY3RlcmlzdGljIHdlIG1pZ2h0IGNob29zZSB0byBtZWFzdXJlIGluCm9yZGVyIHRvIG1ha2UgYW4gaW5mZXJlbmNlIGFib3V0IHdoaWNoIHN1YnBvcHVsYXRpb24gYSBzcGVjaW1lbiBtaWdodCBiZWxvbmcuIApOb3QgdGFraW5nIHRoZSBmdWxsIGNvbnRleHQgaW50byBhY2NvdW50IG9mIGEgc2l0dWF0aW9uIGlzIHdoYXQgb2Z0ZW4gbGVhZHMgdXMgCnRvIG1ha2UgYmlnb3RlZCBzb2NpYWxseSB0b3hpYyBqdWRnbWVudHMsIGVudGVydGFpbiBzdXBlcnN0aXRpb25zLCBvciBhbGxvY2F0ZQptYXJrZXRpbmcgYW5kIHNhbGVzIHJlc291cmNlcyB0byBtYXJrZXRzIHRoYXQgYXJlIGxlYXN0IHJlY2VwdGl2ZSB0byBvdXIgcHJvZHVjdAptZXNzYWdpbmcuCgpJbiB0aGUgbmV4dCBmZXcgaW5zdGFsbG1lbnRzIG9uIHRoaXMgZGlzY3Vzc2lvbiwgSSBwbGFuIHRvIGRpdmUgYSBsaXR0bGUgbW9yZSAKZGVlcGx5IGluIHRvIHRoZSBCYXllc2lhbiByZWFzb25pbmcgcHJvY2VzcyB3aXRoIGluY3JlYXNpbmdseSBtb3JlIGNvbXBsZXggCmV4YW1wbGVzIHRoYXQgd2lsbCByZW1haW4gc3RpbGwgcmF0aGVyIHNpbXBsZSB0byBpbXBsZW1lbnQgaW4gUiBjb2RlLiBJZiB5b3UKZG9uJ3QgY29tZSBmb3IgdGhlIGNvZGUsIEkgZG8gaG9wZSB5b3UgZW50ZXJ0YWluIHRoZSB0aGlua2luZyBwcm9jZXNzIHRoYXQgY2FuCmhlbHAgaW1wcm92ZSBleGVjdXRpdmUganVkZ21lbnQgYW5kIGRlY2lzaW9uIG1ha2luZy4KCkJhY2sgdG8gW0luY2l0ZSEgRGVjaXNpb24gVGVjaG5vbG9naWVzXShodHRwOi8vd3d3LmluY2l0ZWRlY2lzaW9udGVjaC5jb20pLgo=

Comments

Collapse Expand Comments (0)
You don't have permission to post comments.

Oracle Crystal Ball Spreadsheet Functions For Use in Microsoft Excel Models

Oracle Crystal Ball has a complete set of functions that allows a modeler to extract information from both inputs (assumptions) and outputs (forecast). Used the right way, these special Crystal Ball functions can enable a whole new level of analytics that can feed other models (or subcomponents of the major model).

Understanding these is a must for anybody who is looking to use the developer kit.

Why are analytics so important for the virtual organization? Read these quotes.

Jun 26 2013
6
0

Since the mid-1990s academics and business leaders have been striving to focus their businesses on what is profitable and either partnering or outsourcing the rest. I have assembled a long list of quotes that define what a virtual organization is and why it's different than conventional organizations. The point of looking at these quotes is to demonstrate that none of these models or definitions can adequately be achieved without some heavy analytics and integration of both IT (the wire, the boxes and now the cloud's virtual machines) and IS - Information Systems (Applications) with other stakeholder systems and processes. Up till recently it could be argued that these things can and could be done because we had the technology. But the reality is, unless you were an Amazon, e-Bay or Dell, most firms did not necessarily have the money or the know-how to invest in these types of inovations.

With the proliferation of cloud services, we are finding new and cheaper ways to do things that put these strategies in the reach of more managers and smaller organizations. Everything is game... even the phone system can be handled by the cloud. Ok, I digress, Check out the following quotes and imagine being able to pull these off without analytics.

The next posts will treat some of the tools and technologies that are available to make these business strategies viable.

Multi-Dimensional Portfolio Optimization with @RISK

Jun 28 2012
16
0

Many speak of organizational alignment, but how many tell you how to do it? Others present only the financial aspects of portfolio optimization but abstract from how this enables the organization to meets its business objectives.  We are going to present a practical method that enables organizations to quickly build and optimize a portfolio of initiatives based on multiple quantitative and qualitative dimensions: Revenue Potential, Value of Information, Financial & Operational Viability and Strategic Fit. 
                  
This webinar is going to present these approaches and how they can be combined to improve both tactical and strategic decision making. We will also cover how this approach can dramatically improve organizational focus and overall business performance.

We will discuss these topics as well as present practical models and applications using @RISK.

Reducing Project Costs and Risks with Oracle Primavera Risk Analysis

.It is a well-known fact that many projects fail to meet some or all of their objectives because some risks were either: underestimated, not quantified or unaccounted for. It is the objective of every project manager and risk analysis to ensure that the project that is delivered is the one that was expected. With the right know-how and the right tools, this can easily be achieved on projects of almost any size. We are going to present a quick primer on project risk analysis and how it can positively impact the bottom line. We are also going to show you how Primavera Risk Analysis can quickly identify risks and performance drivers that if managed correctly will enable organizations to meet or exceed project delivery expectations.

.

 

Modeling Time-Series Forecasts with @RISK


Making decisions for the future is becoming harder and harder because of the ever increasing sources and rate of uncertainty that can impact the final outcome of a project or investment. Several tools have proven instrumental in assisting managers and decision makers tackle this: Time Series Forecasting, Judgmental Forecasting and Simulation.  

This webinar is going to present these approaches and how they can be combined to improve both tactical and strategic decision making. We will also cover the role of analytics in the organization and how it has evolved over time to give participants strategies to mobilize analytics talent within the firm.  

We will discuss these topics as well as present practical models and applications using @RISK.

The Need for Speed: A performance comparison of Crystal Ball, ModelRisk, @RISK and Risk Solver


Need for SpeedA detailed comparison of the top Monte-Carlo Simulation Tools for Microsoft Excel

There are very few performance comparisons available when considering the acquisition of an Excel-based Monte Carlo solution. It is with this in mind and a bit of intellectual curiosity that we decided to evaluate Oracle Crystal Ball, Palisade @Risk, Vose ModelRisk and Frontline Risk Solver in terms of speed, accuracy and precision. We ran over 20 individual tests and 64 million trials to prepare comprehensive comparison of the top Monte-Carlo Tools.

 

Excel Simulation Show-Down Part 3: Correlating Distributions

Escel Simulation Showdown Part 3: Correlating DistributionsModeling in Excel or with any other tool for that matter is defined as the visual and/or mathematical representation of a set of relationships. Correlation is about defining the strength of a relationship. Between a model and correlation analysis, we are able to come much closer in replicating the true behavior and potential outcomes of the problem / question we are analyzing. Correlation is the bread and butter of any serious analyst seeking to analyze risk or gain insight into the future.

Given that correlation has such a big impact on the answers and analysis we are conducting, it therefore makes a lot of sense to cover how to apply correlation in the various simulation tools. Correlation is also a key tenement of time series forecasting…but that is another story.

In this article, we are going to build a simple correlated returns model using our usual suspects (Oracle Crystal Ball, Palisade @RISK , Vose ModelRisk and RiskSolver). The objective of the correlated returns model is to take into account the relationship (correlation) of how the selected asset classes move together. Does asset B go up or down when asset A goes up – and by how much? At the end of the day, correlating variables ensures your model will behave correctly and within the realm of the possible.

Copulas Vs. Correlation

Copulas and Rank Order Correlation are two ways to model and/or explain the dependence between 2 or more variables. Historically used in biology and epidemiology, copulas have gained acceptance and prominence in the financial services sector.

In this article we are going to untangle what correlation and copulas are and how they relate to each other. In order to prepare a summary overview, I had to read painfully dry material… but the results is a practical guide to understanding copulas and when you should consider them. I lay no claim to being a stats expert or mathematician… just a risk analysis professional. So my approach to this will be pragmatic. Tools used for the article and demo models are Oracle Crystal Ball 11.1.2.1. and ModelRisk Industrial 4.0

Excel Simulation Show-Down Part 2: Distribution Fitting

 

One of the cool things about professional Monte-Carlo Simulation tools is that they offer the ability to fit data. Fitting enables a modeler to condensate large data sets into representative distributions by estimating the parameters and shape of the data as well as suggest which distributions (using these estimated parameters) replicates the data set best.

Fitting data is a delicate and very math intensive process, especially when you get into larger data sets. As usual, the presence of automation has made us drop our guard on the seriousness of the process and the implications of a poorly executed fitting process/decision. The other consequence of automating distribution fitting is that the importance of sound judgment when validating and selecting fit recommendations (using the Goodness-of-fit statistics) is forsaken for blind trust in the results of a fitting tool.

Now that I have given you the caveat emptor regarding fitting, we are going to see how each tools offers the support for modelers to make the right decisions. For this reason, we have created a series of videos showing comparing how each tool is used to fit historical data to a model / spreadsheet. Our focus will be on :

The goal of this comparison is to see how each tool handles this critical modeling feature.  We have not concerned ourselves with the relative precision of fitting engines because that would lead us down a rabbit hole very quickly – particularly when you want to be empirically fair.

RESEARCH ARTICLES | RISK + CRYSTAL BALL + ANALYTICS