How to extract features of videos

Hello.

I want to classify videos after extracting features.

When I use background substraction as extracting features, the result of classification is not good.

Could I get some ideas about feature extraction in videos?

This is code about background substraction. I want to use other methods for feature extraction

%// read the video:
list = dir('*.avi')
% loop through the filenames in the list
for k = 1:length(list)
    reader = VideoReader(list(k).name);
    vid = {};
    while hasFrame(reader)
        vid{end+1} = im2single(readFrame(reader));
    end
    %// simple background estimation using mean:
    bg = csvread('background.csv');
    bg = reshape(bg, 144, 180, 3);
    %// estimate foreground as deviation from estimated background:
    for i=1:25
        fIdx(i) = i; %// do it for frame 1 ~ 60
        fg{i} = sum( abs( vid{fIdx(i)} - bg ), 3 );
        fgh{i} = imresize(fg{i}, 0.2);
        
        fg{i} = reshape(fgh{i},[],1);
    end
    X = cell2mat(fg);

[Kenta]

Another option is the use of pre-trained convolutional neural network.

For example, a video can be classified with CNN and LSTM.

https://jp.mathworks.com/matlabcentral/fileexchange/74402-video-classification-using-lstm-lstm

Answer 1

[Image Analyst]

I extract some features in the attached demo. I compute a dynamic background from the last few frames and subtract it to get the changes, and I also plot the mean R, G, and G values. You could of course compute other things -- whatever you want. If background subtraction is not good for what you want to segment, then you'll have to devise a different segmentation routine. Background subtraction is not appropriate for all situations obviously.

[Kong]

Hello. Thank you so much.

I got this image. Which data is good for classification?

[Image Analyst]

It all depends if the person and/or camera and/or wall is moving. I guess something must be moving otherwise you'd just have a snapshot instead of a video. Do you have the Computer Vision Toolbox? Try the foreground detector.

[Kong]

Yes. I have Computer Vision Toolbox.

When I use the foreground detector, the result is not good like this.

Could you give an idea to solve this?

clear all
close all
%// read the video:
reader = VideoReader('shahar_bend.avi');
vid = {};
while hasFrame(reader)
    vid{end+1} = im2single(readFrame(reader));
end
%// simple background estimation using mean:
bg = mean( cat(4, vid{:}), 4);
%// estimate foreground as deviation from estimated background:
fIdx = 40; %// do it for frame 43
fg1 = sum( abs( vid{fIdx} - bg ), 3 ) > 0.25;
fg2 = imresize(fg1, 0.5);
figure;
subplot(141); imshow( bg ); 
subplot(142); imshow( vid{fIdx} );
subplot(143); imshow( fg2 );

[Image Analyst]

What I would do it to use a higher resolution camera and use less compression, because the images are of horribly low quality - almost unusable.