Solving the Burgers Equations with Echo state networks

17 views (last 30 days)
Hi,
found this example, which is very interesting for my work.
Can someone give me a hint, how to solve the PDE with an ESN, so change from ordinary neuronal network to reservoir?
Any help is welcome.
Best regards,
Mike

Accepted Answer

David Willingham
David Willingham on 2 Sep 2022
Hi Mike,
Are you able to provide the code for your PDE? How long does it take to train?
  2 Comments
David Willingham
David Willingham on 7 Sep 2022
I followed up with our developers on this. They have stated:
This is a use case we hadn't had a user request before on, so currently don't have an example. Tips for creating an example would include:
  • The ESN reservoir is simply x(n+1) = tanh(Ux(n) + Vu(n))​ for randomly initialized and untrained matrices U,V​, input data u​ and reservoir x​.
  • The ESN output is u(n+1) = Wx(n+1)​ for a trained matrix W​. One of the keys is that W​ is the only dlarray​ to update in training, i.e. the only value on parameters​ in the current example.
  • ESN is a discrete time model, whereas the current PINN example uses time as an input.
  • The PINN loss should work more or less the same - though the time derivative is estimated via finite differences.
To help development prioritise building an example, can you ellaborate on what application the PDE's are being used for?

Sign in to comment.

More Answers (4)

David Willingham
David Willingham on 1 Sep 2022
Hi Mike,
Are you able to ellaborate a little more about the problem you're looking to solve? We currently don't have an example for this, so I'm looking to see what might be the best material / support I can give you.
David

Mike
Mike on 2 Sep 2022
Edited: Mike on 2 Sep 2022
Hi David, Thanks for your answer.
I'am trying to solve different partial differential with neuronal networks.
In some cases it takes a while and I focus on the accurary. I read, that using reservoir networks or ESN have advantages there, they are fast and could improve the accuracy in some application. I wanna give them a try in my analyses...
However I'am not very familiar how to implement them, I thought that the use case of the Burger equation could be a good starting point how to integrate it.
kind regards, Mike

David Willingham
David Willingham on 9 Sep 2022
Hi Mike,
I received an update from our development team on this (attached). Whilst not fully worked through, it should serve as a starting point.
The following is a brief description:
==
Solving PDEs with Echo State Networks
This repo demonstrates how to train an echo-state network to solve a PDE.
It follows this example.
Echo State Networks
An echo-state network is a discrete time recurrent model. Given a sequence x(t) the model computes a reservoir sequence z(t+1) = tanh(U*z(t)+V*x(t)). Then the model output is y(t) = W*z(t). Here the U,V,W are randomly initialized matrices. During training only the W matrix (the output matrix) is trained. This speeds up typical deep learning model training as there are far fewer parameters, and in fact the model z(t) -> y(t) is merely a linear model which can be fit with least squares or ridge regression in typical cases.
Solving PDEs
To solve PDEs with deep learning models we simply add a "soft constraint" into the model loss which is some norm of the PDE residual. This method is typically called Physics-Informed Neural Networks (PINNs).
Technical Details of using ESN to solve PDE
  1. Since the loss involves the PDE term it is not possible to optimize W with simple methods like least squares. This means we need to use gradient-based methods, which lose some of the efficiency gains of typical ESN workflows. In particular we need to re-compute the reservoir dynamics on every iteration of the gradient-based optimization as we need the autodiff system to trace through the reservoir so that it can compute the partial derivatives that define the PDE.
  2. Since ESN is a discrete time model we have to modify this example to be discrete time. In particular this means we have to estimate the partial derivative in time via finite differences.
Future or ToDo
  • This example does not currently use any of the common and important techniques necessary to train an ESN well. For example see A Practical Guide to Appling Echo State Networks
  • This example does not include loss terms to encourage the model to satisfy an initial condition or boundary condition as in the original example. This is simply for brevity, those terms can be added.
  • The example splits the training data x(t) into subsequences x(1:N),x(N+1:N),.... In principle the N could be increased during training, and this may help fine-tune the model to be more accurate for longer time.
  • This example does not use LBFGS to optimize the weights as in this example. It may be important to use LBFGS in PINN workflows according to literature.
  • It may be possible to use the idea of CTESN to extend this to a continuous time model.
==

Mike
Mike on 15 Sep 2022
Hello David, many thanks to you and your team. Excellent work.
I have only one difficulty, I am not quite clear how to take into account the time. If I stick to the example (solve Partial Differential Equ. Using DL), the model function contains x and T. However, with ESN only x. I'am not quite sure how to incooperate time t, or differences of time?
Best regards,
Mike
  2 Comments
Mike
Mike on 19 Sep 2022
Many thanks you for the explanation, Ben, now it has become clearer.

Sign in to comment.

Categories

Find more on Computational Fluid Dynamics (CFD) in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!