Does Matlab provide any tools to understand the contents in an image and describe it using words?

Hi,

I am new to neural network and just about to start to build my own.

I am hoping to achieve that with an input of an image, the output would be a few sentences in texts to describe the contents (and even feelings) of this image.

Where should I start? Does Matlab have any developed tools or trained networks that I can just grab and use?

[Bjorn Gustavsson]

@John D'Errico - even your beloved movie-computer would be stumped ("unsure" at least) on what emotion to put on an image from 30 minutes into the Brazil - Germany World-Cup semifinal 2014, surely not even the movie-computer can solve that?

[the cyclist]

Gents, let's not be too hasty and dismissive (not to mention a bit unkind to a new user). The reality is that given a pre-trained set of images that are labeled with emotional content -- which could quite possibly exist somewhere -- one could build a classifier on new images.

@Salad Box, I am not an expert in this, but I would start with looking at examples in the Deep Learning Toolbox, coupled with a keyword search on "sentiment analysis", which is one of the common buzzwords in this type of method. You might get a bit of a start like that, and MATLAB definitely has the computational tools to do this type of analysis.

[John D'Errico]

Yes. All of that is surely possible. But not off the shelf, pretrained.

And emotional content in an image is highly subjective.

[the cyclist]

Building emotional machines: Recognizing image emotions through deep neural networks

Final sentence of that paper: "We also built a database for the emotion estimation with the V-A model and will continue to collect more data. We expect our dataset will be widely used in the field of affective computing."

[Salad Box]

Thank you so much @the cyclist for your advices. Will definitely do what you suggested.

Thank you @John D'Errico@Bjorn Gustavsson for being straightforward. I do appreciate that.

Answer 1

[Matt J]

You could start with this example,

https://www.mathworks.com/help/deeplearning/ug/image-captioning-using-attention.html#ImageCaptioningUsingAttentionExample-20

[Salad Box]

Thank you@Matt J. This is really helpful.

Answer 2

[Steven Lord]

There are examples in various products (mainly using Deep Learning Toolbox and/or Computer Vision Toolbox) to perform image classification to determine which of a set of items in the training data an image represents. See as one example this page that shows how to use a pretrained network (SqueezeNet) with ten lines of code to perform that classification.

But if you're looking for "Computer, identify the object on the viewscreen" a la Star Trek where the object is something completely new, I don't think you're going to be able to achieve that.

[Salad Box]

Thank you for your advice @Steven Lord. They are really helpful.

I am not looking for anything completely new, not at this stage at least. I am just trying to build a basic model and see what I can achieve with that at moment.

Will definitely try examples you mentioned.

Thank you again for the reply.:)