DeepDreamImage for multi input network

I have a network for classifying RGBD data, it is 2 networks concatenated in parallel with a joint loss function and classifier at the end.

One input receives an RGB Image, and the second receives a depth image, based on both images the network classifies.

I want to visualize activations using deep dream, and to receive an Image for each input (a deep dream for RGB, and a deep dream for depth).

I know I can get an activation map for each of the layers but I specifically want a deep dream image.

DeepDreamImage only works with a single input single output networks.

Is there any way to make it work with a double input single output network?

Answer 1

[Malay Agarwal]

Hi Imri,

I understand that you are trying to use the “deepDreamImage” function to visualize a neural network with multiple inputs.

Unfortunately, as of the latest MATLAB release (MATLAB R2023b), the "deepDreamImage" function does not support multi-input networks.

However, Deep Dream is a simple algorithm and can be implemented for your particular use case. A script written in MATLAB R2023b which does that is attached. It contains a function called “deepDreamMultiInput”, which can be used to apply the Deep Dream algorithm on a multi-input network. The implementation is based on this tutorial: https://www.tensorflow.org/tutorials/generative/deepdream.

I have assumed that the RGB image is the first input, and the depth image is the second input in the network. The function takes the following arguments:

• “net” – Pre-trained neural network, represented as a "DAGNetwork" object.
• “rgbPath” – Path to the RGB image Deep Dream should be applied on as a string.
• “depthPath” – Path to the depth image Deep Dream should be applied on as a string.
• “layers” – Names of the layers Deep Dream should focus on as an array of strings.
• “numIterations” – Number of iterations for gradient ascent as an integer. It defaults to 100 iterations.
• “learningRate” – Learning rate for gradient ascent as a real number. It defaults to 0.01.

The function can be used as follows:

[rgbImage, depthImage] = deepDreamMultiInput(net, ...
    "path\to\rgb\image", ...
    "path\to\depth\image", ...
    ["conv_1", "conv_2"]);

The result can then be visualized as follows:

subplot(1, 2, 1);
imshow(rgbImage);
title('RGB');
subplot(1, 2, 2);
imshow(depthImage);
title('Depth');

There are some limitations in the implementation. Particularly, it doesn’t implement the octave improvement that is mentioned in the tutorial linked above. Also, the function only works in MATLAB R2023a or above.

Please let me know if you’d like more details regarding the implementation.

Hope this helps!