Customer Service: +44 1334 657957

Introduction to statistics

This section introduces some important aspects concerning the calculation and use of table based statistics in QPSMR programs.

Significance testing, confidence limits and probabilities

Question: Is this figure significant? Reply: Compared to what?

When discussing 95% or 99% significance it is important to bear in mind that a figure from our survey (a percentage or a mean score) can only be significantly different when it is compared with some other figure from our survey, or a known value (from another source). The program has many ways of marking a figure as 95% significant but what does this mean?

Let us assume that we have a product rating mean score for two subsets of data. Men have a mean of 1.56 and Women have a mean of 1.34. When compared, QPSMR flags Men as having a significantly higher mean at the 95% level than Women. What this means is that there is only a 1 in 20 chance that a difference this large (0.22) would be obtained if the two subsets had been drawn from the same universe. To put this another way, we can be 95% confident that there is a real difference between Men and Women when rating the product.

The use of the word “significance”, however, in this context is unfortunate because it is not the same as the normal use of the word. On a rating scale of 1 to 5 we might have a mean score difference of 0.01 that is “significant” and a difference of 2.00 that is not “significant”. The word “confidence” is much more descriptive.

So when a figure is marked at 99% it means that it is very unlikely (1 in 100) that there is no difference between the two samples being looked at. The reverse is not true; we cannot say that if a figure is not marked then there is no difference - it may just be that the sample sizes are not large enough for us to be confident. The more interviews we do, the more real differences will be found.

Following many of the tests in the program is a probability figure. This is a way of showing the confidence level of the statistic as a value between 0.0000 and 1.0000, so small figures are significant and large ones are not. A 95% significance will show as a probability of 0.0500 or less, and 99% significance as 0.0100 or less. In general, to convert a probability to a significance level, subtract the probability from 1.0000 and use the first two digits. For example, a probability of 0.1278 becomes 0.8722 which is 87.22% significant.

What is compared with what?

The program will only compare figures that are included in the same break; only columns that appear in the same break will be tested against each other. The majority of the tests in the program compare subsets of the data shown as the columns on a table.

Standard method

As an example we will use six columns: Total, Male, Female, Young, Middle, Old.

The standard method in the program (with format SHG0), is to test whether any particular subset (column) of the data is different from the remainder of the sample. In this way the program will highlight any “interesting” columns for further investigation.

 In our example, Males are tested against Females, Young against Middle and Old together, Middle against Young and Old together, and also Old against Young and Middle together.

The purpose of testing in this way is to highlight columns (breakdowns) which are “different”. This method of testing includes the total sample in every test and is therefore more likely to detect differences than other types of comparison.

Because the “rest of the data” is calculated by subtracting the column being investigated from the total column, this standard method will only work on tables with a total column.

Cells are marked with one asterisk (*) for 95% significance and two (**) for 99%, but this may be changed (formats SMA and SMB).

TIP: If SMA/SMB are set to + (a plus sign), the cells will be marked with + or - depending on whether the proportion is significantly higher (+) or lower (-).

Column identifiers method

This common method for comparison in the program uses labels on each column as an identifier, for example:

Demographics^Area\North (a), Mid (b), South (c), Not stated, Sex\Male (m), Female (f)

 

The word "Demographics" is an over-header above the following columns, in this case all of the columns.

The words "Area" and "Sex" are headers above the following columns.

The identifiers must be included at the end of the individual labels (preferably on a new line) enclosed within parentheses. In this method individual pairs of columns are compared; the range of the comparisons depend upon format SHG.

You will notice that headers (Area and Sex) and an over-header (Demographics) have been used.

With the normal format SHG1 (test within headers) each area is compared with the other two, and the appropriate letter markers placed against the cell, if significant differences are found. For example if “Mid” is found to be different and higher to both of the other areas, with a significance of 90% (with default SLA90) it will be marked with lower case letters “ac” after the value. If “Mid” was found to be different and higher to both of the other areas, with a significance of 95% (with default SLB95) it will be marked with upper case letters “AC” after the value. The “Not stated” column will not be tested because it does not have an identifier.  Males and Females will be compared, so that if Females are different and higher the letter “m” (lower or upper case depending on the significance level) will be placed next to the cell.

If the rare format SHG2 (test with overheaders) was used then all five identified columns would be compared with each other.

IMPORTANT: When marking figures only the higher figure is marked.

Types of test

This section describes the types of testing that can be done:

Distribution Z tests

Where a table has a list of items (for example “Likes”) down the side format SIG can be used to mark cells using a Z test on proportions.

Each row of the table is treated separately and cells are marked depending on whether the percentage is different to the column it is being compared with in the same row.

Mean or average t-tests

Where a table has rows from which a mean score or average is produced, t-tests will be calculated and significant differences marked.

Mean or average F-tests

If format TTF is used an F-test is performed on all of the columns within each group. This test is used to establish whether the group of columns (for example - Area) affects the row mean or average, without looking at all of the individual pairs of columns.

Other tests

For table rows which are assumed to be in order (for example - Small, Medium and Large) but you do not wish to attach score values to the rows, two nonparametric tests can be applied:

  • Kolmogorov-Smirnoff using format KST
  • Mann-Whitney-Wilcoxon using format MWW

For tables with any distribution down the side use Chi-squared using format CHI.

Other software

Data from the program can be output to other software for further statistical tests.