How can I test for significance between expected and observed outcomes?

Hi, I'm working on a project where we are validating a psychometric self-reported tool on personality traits and comparing to other personality measures (self-report and other). Myself and colleagues have identifies which items we expect to correlate on each tool by indicating that a correlation is expected and its direction.

I have run the correlations and now have data to compare what we expected to correlate vs. the actual outcomes.

We would like to run some significance testing to see whether the hits and misses are better than chance. We thought of using a chi-squared, but all the literature I have seen using this method have given actual values for their expected correlations (for example, we expect a correlation of .33 and the observed correlation was .56). We don't have the values for expected outcomes - is there a way we can still carry out a significance test?

I'd be grateful for any advice.

Answer 1

[Scott MacKenzie]

Since there were only two conditions, you can use a t-test (i.e., an anova isn't needed). Here's an example using test random data. Not surprisingly, the difference between the two conditions was not statistically significant, t(18) = 0.395, p > .05.

data = rand(10,2);
[h, p, ci, stats] = ttest2(data(:,1), data(:,2))

h = 0

p = 0.6975

ci = 2×1

-0.1782 0.2608

stats = struct with fields:
    tstat: 0.3950
       df: 18
       sd: 0.2336

Answer 2

[Jeff Miller]

Another alternative: You might consider that you have three classes of items: those for which the predicted correlation was positive, zero, or negative. You have observed correlations for each of those items. So, you could use ANOVA to compare the mean observed correlations for the three classes of items, expected that (e.g.) the mean correlation would be larger for those predicted to have a positive correlation than for those predicted to have a negative correlation.