"diff" function doesn't work properly with small numbers

For some reason when difference between n and n+1 is too small diff function assumes the solution is 0.

There are +-290 data points on the plot, The precision is 10^(-10), As far as i know Matlab works on 16 or 32 digits so it shouldn't be a problem.

Technically on the plot there should be on no constants, Just increase and decrease of value.

Pomiary=cisnienie300920151701average300
Czas = Pomiary{:, 4};
Temperatura = Pomiary{:, 5}; 
CzasDMY= Czas / 86400  + datenum(1970, 1, 1); 
y = Temperatura;
x = CzasDMY;
ydiff=diff(y,1); 
wieksze = (ydiff > 0);
mniejsze = (ydiff < 0);
gora = y;
dol = y;
gora(~wieksze) = NaN;
dol(~mniejsze) = NaN;

plot(x,y,'b',x, gora, 'r', x, dol, 'g');

grid on;
xlim tight;
xlim("auto");
ylim("auto");
legend("Constant", "Increasing", "Decreasing");
legend("Position", [0.15754,0.1468,0.20438,0.12165]);

[Fangjun Jiang]

If you provide a few example values for y, there might be an reasonalbe explaination for this.

[Sylwester]

36 1023.08766260000
37 1023.03861350000
38 1023.01522350000
39 1023.01522350000
40 1022.96080630000

Used NOT operator on increased and decreased arrays to find places where values in the row were the same, I'm gonna check if i should just move x arrays by +1 or -1.

Update, moving arrays by +-1 didn't work

[dpb]

The reasonable explanation for this is that the difference in temperatures for some cases is less than the resolution of a double.

There are several possibilities of whty this might be, starting with measuring temperature to that precision would be unrealistic. If the vaules were calculated, it's possible they were then written to another text file format first that only had limited precision.

Without the actual input data, it's not possible to say from whence the observed results came, but the problem is not in the diff() function, but in the expectations for it given the actual inputs.

Answer 1

[Fangjun Jiang]

The data value and results make sense. There is no problem using diff() to process your data based on your example data.

%%
format long
y=[36 1023.08766260000
    37 1023.03861350000
    38 1023.01522350000
    39 1023.01522350000
    40 1022.96080630000]

y = 5×2

1.0e+03 * 0.036000000000000 1.023087662600000 0.037000000000000 1.023038613500000 0.038000000000000 1.023015223500000 0.039000000000000 1.023015223500000 0.040000000000000 1.022960806300000

<mw-icon class=""></mw-icon>

<mw-icon class=""></mw-icon>

ydiff=diff(y,1)

ydiff = 4×2

1.000000000000000 -0.049049100000047 1.000000000000000 -0.023389999999949 1.000000000000000 0 1.000000000000000 -0.054417200000103

<mw-icon class=""></mw-icon>

<mw-icon class=""></mw-icon>

wieksze = (ydiff > 0)

wieksze = 4×2 logical array
   1   0
   1   0
   1   0
   1   0

mniejsze = (ydiff < 0)

mniejsze = 4×2 logical array
   0   1
   0   1
   0   0
   0   1

By default, MATLAB uses 64 bits floating-point data to represent a numeric value.

At around value 1023, its relative accuracy is 1e-13, sufficient to represent your data precision 10e-10.

The problem you observed comes from your raw data. Note that y(3,2) and y(4,2) are exactly the same by visual observation.

eps(1023)

ans = 
     1.136868377216160e-13

Check the document for eps(). You will understand the issue better.

doc eps

[Sylwester]

They are meant to be the same, The issue is that for some reason function for marking if value increased/decreased has holes in it and skips points unless difference is high enough despite precision being ok.

I tried using eps() function on "diff" and it failed.

.

Also i changed dataset for this task from working on temperature to pressure so the values should make sense despite variable being named "temperatura"

[Fangjun Jiang]

The length of diff() output is 1 smaller than its input length. Your code didn't seem to consider this.

diff(1:3)

ans = 1×2

1 1

<mw-icon class=""></mw-icon>

<mw-icon class=""></mw-icon>

Answer 2

[dpb]

X=[
36 1023.08766260000
37 1023.03861350000
38 1023.01522350000
39 1023.01522350000
40 1022.96080630000];
dx=diff(X)

dx = 4×2

1.0000 -0.0490 1.0000 -0.0234 1.0000 0 1.0000 -0.0544

<mw-icon class=""></mw-icon>

<mw-icon class=""></mw-icon>

As hypothesized above, some of the temperature values are identical owing to the apparent rounding to seven (7) decimal digits.

You would have to have at least one more decimal place in the above between the 3rd and 4th data values in order for the difference to not be identically zero.

If you're transferring data from one place to another, to avoid this don't use text files but save the whole internal precision by using .mat files or binary formatted transfer if from some external source. Besides being able to retain full precision (note that precision does not necessarily imply accuracy), it's much more efficient in speed and memory/disk space.

As for your comment above about the values that "They are meant to be the same, The issue is that for some reason function for marking if value increased/decreased has holes in it and skips points unless difference is high enough", that makes no sense at all -- the two values are identically the same so how can there be any sense of the value changed that "increased/decreased" implies?

If you're trying to measure an overall change; then diff is entirely the wrong function as it is on a pointwise basis and so will indeed notice when there are any points for which the difference is actually zero.

Looking at your small subsample of data

plot(X(:,1),X(:,2),'*-')

indeed, there is an overall negative trend, but it isn't uniformly decreasing at every point, just overall. If you want indications of trends excluding such points, you'd have to do something like find the inflection points and then (say) the two points on either side and then use the adjusted temperature to compute the change.

Note that you would also have to locate any locations of more than two successive points being the same and then do something over those ranges. Also, in doing something like this you'll run into the issue that @Fangjun Jiang raised about the differenced vector being shorter than the original so the points are offset by one in the addressing.

For the simple example here

ix=find(dx(:,2)==0);                        % locate the zero point     `   
fprintf('%d %15.10f\n',X(ix+[0:1],:).')     % display where are relatively

38 1023.0152235000
39 1023.0152235000

X(:,3)=X(:,2);                              % augment the X array 
X(ix+1,3)=mean(X(ix+[0 2],3));              % replace the unchange with linear interp1
hold on
plot(X(:,1),X(:,3),'rx-')

●

diff(X)

ans = 4×3

1.0000 -0.0490 -0.0490 1.0000 -0.0234 -0.0234 1.0000 0 -0.0272 1.0000 -0.0544 -0.0272

<mw-icon class=""></mw-icon>

<mw-icon class=""></mw-icon>

Now you don't have any zeros in the 3rd column diff().