Mean of vector with values < 1 returns as 1?

Hi everybody,

What I'm dealing with is a vector with a massive amount of values (we're talking up to 20.000 entries). I am trying to find a certain pattern and wanted to try one solution by using the vector's mean.

Now you have to think of the vector as one that has high values for the first 1.000 indexes (y is betweeen 3 and 200), while the rest up to the end is something between 0,2 and 0,4 with occasionnaly having a jump up to 1,2. But that happens only like 3 to 4 times during the entire vector's length.

Now when I want to get the mean of the entrie vector, it gives me something of 0,9961 or something in that direction, which seems fair. I mean I have high values in the beginning and then they're pretty low for the rest. But when I try to get the mean of the vector WITHOUT the high beginning (meaning I take it from 1.001:end) I get a mean of 1.

Not only can that not be true, because it should be lower than the mean of the entrie vector, but it can't be excatly 1 without any decimal digits. Something along the lines of 1.0024 would actually make more sense. I tried to reduce and alter the area of the vector and determine their mean but it's always exactly 1.

How can this be and is there a way to get the actual mean of the part without the high values? I'd hate to have to use Excel or any other programm for that so I'd appreciate it.

Thanks in advace!

[DGM]

Without knowing what the data looks like or what exactly you wrote to isolate that section and take its mean, it's not really possible to be certain.

I'd suggest attaching the vector as a .mat file and posting an example of the code -- or at least note the index range of the segment you're isolating.

[Sokratis Panagiotidis]

@DGM I attached the vector into a struct. You should have both the x- and y-values inside it.

If anything else needed please tell me.

[DGM]

I can't replicate the behavior. How are you calculating the mean? Is the data possibly getting rounded or cast as an integer numeric class somewhere in the process?

load mean_movement.mat

% there are some NaNs in the first part
% so i'm using omitnan option to get mean of whole vector
wholevector = movement.y;
[min(wholevector) max(wholevector)]

ans = 1×2

0.0004 198.5970

mean(wholevector,'omitnan')

ans = 0.3252

% range and mean of latter part
endpart = movement.y(1001:end);
[min(endpart) max(endpart)]

ans = 1×2

0.0004 1.3728

mean(endpart)

ans = 0.0915

plot(endpart)

[John D'Errico]

There are two viable possibilities that I see.

First, you may have simply confused the display format, and do not understand what MATLAB means, when it reports a number as 1.0000. That is, in the default display format, MATLAB rounds the results it shows to 4 digits.

format short
x = 1.000000009
x =
    1.0000
    
x = 0.99999
x =
    1.0000

Do you see that each number is shown to be what apears as exactly 1? In fact, they are not. And they are stored as double precision, so they are as they were originally formed.

format long
x1
x1 =
    1.000000009000000
    
x2
x2 =
    0.999990000000000

It is a common mistake for people to misunderstand that just because MATLAB says something is 1.0000, that it is not in fact exactly 1. In fact, if MATLAB gets the exact number 1, it tells you that. You just need to see it has done that.

format short
round(x1)
ans =
    1

Do you see here, MATLAB did not just return 1.0000, but 1?

x1 == 1
ans =
logical
    0
    
round(x1) == 1
ans =
logical
    1

So it is very possible that you have just misunderstood what the meaning of 1.0000 is when MATLAB reports that. It may just be that you have gotten a result that is close enough to 1, that format short has just rounded the rest of the stuff away in the command window. That does not mean it is truly, exactly 1. In fact, those spare zeros after the decimal point are the hint you need that it is not in fact exactly 1.

But we don't know what you did.

And the other possibility is you have in some way, managed to corrupt the values in your vector, processing them in some unexplained way. This is the second possible reason I can offer to explain what you have. It is often the case that someone THINKS they have gotten some strange result they cannot explain, but in reality, they are working with data that has been corrupted in some way. So I'll test your data:

mean(movement.y(3000:end))
ans =
0.086939

mean(movement.y(5000:end))
ans =
0.089362

mean(movement.y(10000:end))
ans =
0.10033

mean(movement.y(15000:end))
ans =
0.13731

When I take the data you provide, and discard some portion of the beginning of the vector, I see entirely expected results, depending on how much of the initial part I discarded.

Long ago, when I first started doing consulting for mathematics and statistics, I recall taking some data I was provided. I got an entirely wonderful result, pretty much exactly what I would have wanted to see. But I was never able to reproduce what I had gotten. Never able to some way, even on exactly the same data, the magic never worked. (I spent a few days trying to figure out what I did. It was all done in Minitab at the time.) To this day, more than 40 years hence, I think I somehow managed to corrupt my variables in some way that created the result I wanted to see.

However, this taught me a valuable lessson. ALWAYS, KNOW EXACTLY what is in my variables. Plot everything if necessary and possible. Verify that what I just did was correct, that it made sense. And when that is not possible, I am just exceedingly careful in what I do.

Of course, a spreadsheet makes this easy, because you are looking at the result you just computed.

Regardless, we don't know what you did wrong, because the data you show does not support your claims in any way. Sorry. If you want something better, then you need to show EXACTLY what you did. Show that it produces the result you claim to have gotten.

[Adam Danz]

To add this this terrific advice above, when simple stats like this don't make sense to me, I often visualize the data just to get an eye-ball on the bigger picture.

I suggest putting your data into a boxplot or histogram to see if it looks the way you'd expect it to look. Note that boxplots show the median rather than the mean but you could add a yline (for boxplot) or xline (for histogram) to show the mean within the distribution.

[dpb]

"Long ago, when I first started doing consulting for mathematics and statistics, I recall taking some data I was provided. I got an entirely wonderful result, pretty much exactly what I would have wanted to see. But I was never able to reproduce what I had gotten. Never able to some way, even on exactly the same data, the magic never worked. (I spent a few days trying to figure out what I did. ..."

Oh, John -- that story brings back memories!!! I did something very similar years ago with some experimental test data collected at a power plant that I processed and presented to a client...I came down w/ a terrible case of something right after finishing up the test but couldn't postpone the meeting. I spent almost a month trying to recreate those results and never did get there; fortunately still had the original data and the final results, while not identical to what I had presented, at least did show the needed results.

To this day, I also have no idea what I possibly could have done originally...I was using MATLAB (mostly) with some Fortran mex...

[Sokratis Panagiotidis]

@DGM Hey the answer is quite simple and although I appreciate all the extra input from the others, I can say the issue was how I tried to get the mean.

What I did was:

mean(~isnan(movement.y))

instead of

mean(movement.y, 'omitnan')

The rest about the format wasn't the problem, as it would literally give me 1 instead of 1.0000 or how digits one has set.

I still apreciate the effort and the help.

Since I can't mark your answer as helpful could you write a seperate answer not directly written in the comment fields of my question @DGM? I can't mark it otherwise.