How to calculate standar deviation and trend in each row of data

I want to ask about my script,

I had a file with different number of column in each row, then calculate the mean, standar deviation and trend of data in each row from column 4 to end. The mean value is work but the std and trend still facing some error.

This is my script

clear;
clc;
format longG;

f = dir('D:\full_Data_Cryosat_c2p0004.txt');
A=1:length(f);
D = f(A).name;

Insufficient number of outputs from right hand side of equal sign to satisfy assignment.

ff=D(:,19:25)

data=readtable(D,'HeaderLines',0);

rata2=mean(data{:,4:end},2,'omitnan');
standar_deviation=std(data{:,4:end},2,'omitnan');
ud=trend(data{:,4:end},2,'omitnan');

ref=data{:,1};
lat=data{:,2};
lon=data{:,3};
fixed=[ref lat lon rata2 std ud]

file_name_data=['data_msl_' ff '.txt']
dlmwrite(file_name_data, fixed, 'delimiter', '\t');

Error using var (line 179)
W must be a vector of nonnegative weights, or a scalar 0 or 1.

Error in std (line 51)
y = sqrt(var(varargin{:}));

Error in script_msl_v2 (line 13)
standar_deviation=std(data{:,4:end},2,'omitnan');

Thanks for helping

Answer 1

[dpb]

The input arguments for std include a flag for the weighting for the divisor and/or the option to compute a weighted standard deviation. Consequetnly, they don't mimic those of mean identically. This makes for a very error-prone user interface, but it was designed in the earliest incarnations of MATLAB and remains that way now for compatibility reasons.

You're far from the first (and certainly won't be the last) to have fallen into the trap, however, ... :)

A section of the help shows the parts of specific interest here and what tripped you up -- the two-argument forms are interpreted as either the normalization flag if it is a single value or the weight vector if an array, but you passed three arguments -- and in a form that doesn't match up to the three-argument form.

>> help std
 std Standard deviation.
    ...
    std normalizes Y by N-1 if N>1, where N is the sample size.  This is
    the sqrt of an unbiased estimator of the variance of the population
    from which X is drawn, as long as X consists of independent,
    identically distributed samples. For N=1, Y is normalized by N.
 
    Y = std(X,1) normalizes by N and produces the square root of the second
    moment of the sample about its mean.  std(X,0) is the same as std(X).
 
    Y = std(X,W) computes the standard deviation using the weight vector W.
    W typically contains either counts or inverse variances.  ...
    ...
 
    Y = std(X,W,DIM) takes the standard deviation along the dimension DIM
    of X.

Hence, you need

standar_deviation=std(data{:,4:end},[],2,'omitnan');

to compute the unbiased estimator over the rows dimension. The [] passes the empty argument placeholder for the weighting vector.

[Steven Lord]

This makes for a very error-prone user interface, but it was designed in the earliest incarnations of MATLAB and remains that way now for compatibility reasons.

I'm not 100% certain but I believe std(X, W) was introduced before MATLAB had 3-dimensional arrays. [I've been here a while but not THAT long.] Rather than trying to squeeze in the DIM input immediately after the data input as std(X, DIM, W) [which would have made the 2-input case ambiguous if you called std(X, 1)] the choice was made to put the DIM input at the end of the arguments list.

[Retno Purwaningsih]

yeahh that's working. but the trend has error like this

Undefined function 'trend' for input arguments of type 'double'.

Error in script_msl_v2 (line 14)

ud=trend(data{:,4:end},[],2,'omitnan');

[dpb]

trend isn't a base MATLAB function; there is one in the OPC Toolbox

https://www.mathworks.com/help/opc/ug/dagroup.trend.html

but that doesn't seem appropriate here.

I suspect you were intending to use detrend instead to remove linear trend from the data?

If so, that would be

ud=detrend(data{:,4:end}.','omitnan').';

where you must explicitly transpose the input data array (and I transposed the result back to keep same orientation) because detrend does NOT support the dimension of the input array over which to operate as an optional input. In this way, it doesn't follow the canonical form, either.

MORAL: You must always consult the actual documentation for each function; particularly many of these long-standing ones that were developed very early in the history of MATLAB because they were written before the use of named parameters was well developed and the very first incarnations followed typical FORTRAN coding practices and these very early routines have not been modified since other than to graft on a few more recent niceties like the 'omitnan' flag on detrend that is quite recent in terms of the overall age of the function's initial incarnation.

https://www.mathworks.com/help/matlab/ref/detrend.html

If, OTOH, you were intending to compute the linear trend of each row in the data, then look at polyfit, but it and none of the other fitting toolsets if you have Curve Fitting or Statistics TB are set up for doing multiple fits on arrays; you have to iterate over the array by row or column.