How to interpolate harmonic data series (without sin / cos equation defined)?

2 views (last 30 days)
Hi, Community
I wanna ask the most crucial in my research and my program. I need to do this to fill data missing in my magnetic data series which re mostly harmonic sinusoidal. Lemme share my magnetic data as plot like this :
I want to interpolate the harmonic signal data missing to be fit with the original data. But i only have data series (without equation, ex. sin eqs or cos eqs) as the only input parameter. In the other equation, like polynomial equation, we can just use polyfit and polyval, but i dont think it would be good if the data series is a harmonic data like that. I already pinned my data (time series with the other variable) here and also, here is my code to read the pinned data file :
[namafile,arah]=uigetfile({'*.txt', 'Text-Files (*.txt)'},'Load Data Magnet Hasil Ekstraksi', 'Multiselect','on');
full_ekstrak = fullfile(arah, namafile);
opts_ekstrak0 = detectImportOptions(full_ekstrak, 'ReadVariableNames',false, 'Delimiter', ' ', 'Whitespace', ' ',...
'ConsecutiveDelimitersRule', 'join', 'EmptyLineRule', 'skip', 'FileType','text');
T_raw = readtable(full_ekstrak,opts_ekstrak0);
T_ekstrak0 = standardizeMissing(T_raw,{'0', '0.00', 'N/A'});
data0 = T_ekstrak0(:,{'Var1','Var2','Var3','Var4','Var5','Var6','Var7','Var8', 'Var9'});
date0 = data0.Var1;
datefix0 = datetime(date0,'Format', 'yyyy-MM-dd');
time0 = data0.Var2;
time0.Format = 'hh:mm:ss';
data0_deklinasi = data0.Var6; %You can use this as variable (Y Plot)
data0_komph = data0.Var3; %You can use this as variable (Y Plot)
data0_kompf = data0.Var4; %You can use this as variable (Y Plot)
data0_kompfobs = data0.Var5; %You can use this as variable (Y Plot)
data0_kompd = data0.Var6; %You can use this as variable (Y Plot)
data0_x = data0.Var7; %You can use this as variable (Y Plot)
data0_y = data0.Var8; %You can use this as variable (Y Plot)
data0_z = data0.Var9; %You can use this as variable (Y Plot)
data0_komph_calc = sqrt((data0_x.^2)+(data0_y.^2)); %You can use this as variable (Y Plot)
data0_kompf_calc = sqrt((data0_x.^2)+(data0_y.^2)+(data0_z.^2)); %You can use this as variable (Y Plot)
data0_declin = atan(data0_y./data0_x); %You can use this as variable (Y Plot)
wak_ekstrak = string(datefix0) +" "+ string(time0); %You can use this as Time Series of X Plot
waktu0 = datetime(wak_ekstrak,'InputFormat','yyyy-MM-dd HH:mm:ss', 'Format', 'yyyy-MM-dd HH:mm:ss');
waktu01 = datetime(wak_ekstrak,'InputFormat','yyyy-MM-dd HH:mm:ss');
So anyone, would you please help me in inding the solution? Iam so grateful, if anyone could help me in this difficult problem, Thank you very much /.\ /.\ /.\
  4 Comments
Tyann Hardyn
Tyann Hardyn on 2 Nov 2021
@KSSV Thanks Sir, it works. Iam trying by using movmedian fillmissing... But it appear not too match as the harmonic sinusoidal original data, only like a trend.... I wonder, whats the best function for it....

Sign in to comment.

Answers (1)

Bjorn Gustavsson
Bjorn Gustavsson on 2 Nov 2021
You should not "inpaint" data this way and then use that to extract SQ-curves or any such averaged quantities. You have the data you have and, saddly, should settle for making do with that, properly taking into account that you have missing data. consider the case where there were a major magnetic storm with an onset 10th of April, recovering just about to 15th(?), if you now use some kind of Fourier-series fit to the current data to fill in the missing periods you would insert completely erroneous data into your data-base. To me this would be a major no-no to do.
What you might do for some very narrow uses of averaged parameters would be to do some linear (or non-linear?) combination fits of nearby observatory-data to the data you have available and then use those data to estimate the data during the time-periods where your data-series have gaps.
HTH
  2 Comments
Bjorn Gustavsson
Bjorn Gustavsson on 3 Nov 2021
It is not very harmonic though, is it? Look at all your transient spikes.
Didn't you read my reply? The point is that this is a bad idea, you still haven't explained for what reason you want the gaps filled in. If you want to calculate trends in your data you should not introduce additional fake data, you will simply have to work with the data you have, and adapt your methods to handle gaps in the data.
Here's one method you could use to all the missing data assign infinite standard deviation such that all your consecutive tools (weighted least-square-fitting etc) will effectively ignore those time-periods and the actual values you fudge-up doesn't matter. You have the data you have.
Here's another method to extract some average B-field variation over your data:
subplot(3,1,1:2)
imagesc(reshape(Mag(:,13),[],30)') % Stored your data in Mag, plotting the last column, adapt
subplot(3,1,1+2)
plot(median(reshape(Mag(:,13),[],30)','omitnan')) % Sensible average
hold on
% Then lets plot some estimates of spread around that average.
plot(median(reshape(Mag(:,13),[],30)','omitnan')+mad(reshape(Mag(:,13),[],30)'))
plot(median(reshape(Mag(:,13),[],30)','omitnan')-mad(reshape(Mag(:,13),[],30)'))
plot(median(reshape(Mag(:,13),[],30)','omitnan')-std(reshape(Mag(:,13),[],30)',0,1,'omitnan'))
plot(median(reshape(Mag(:,13),[],30)','omitnan')+std(reshape(Mag(:,13),[],30)',0,1,'omitnan'))
plot(median(reshape(Mag(:,13),[],30)','omitnan')+std(reshape(Mag(:,13),[],30)',1,1,'omitnan'))
plot(median(reshape(Mag(:,13),[],30)','omitnan')-std(reshape(Mag(:,13),[],30)',1,1,'omitnan'))
The problem with in-painting data from this type of average is that it will bias your results using this data since it assigns this average data in gaps that should/would have other values (it would shift the slope when the gaps are shifted towards an end of the sample-period.) Surely you can find some information about how this type of problem is handled in the geomagnetic litterature.
HTH

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!