1. Machine learning

2. Deep learning

Neural networks (NNs) are powerful function approximators capable of modeling any continuous function. A neural network, parameterized by learnable parameters $\boldsymbol{\theta}$ (typically weights $\boldsymbol{w}$ and biases $\boldsymbol{b}$), learns a function $\hat{y}=f_{NN}(x;\boldsymbol{\theta})$ that approximates the relationship $y = f(x)$. NNs are built using nested linear transformations combined with non-linear activation functions $\sigma$. The simplest form, fully connected neural networks, achieve this through layers of fully connected neurons. The activation $a^i_k$ of each neuron (the $i$th neuron in layer $k$) is computed by applying a non-linear activation function $\sigma$ to a linear combination of the activations from the previous layer.

The most basic NNs: Fully connected NNs achieve this with layers of fully connected neurons, where the activation $a^i_k$ of each neuron (the $i$th neuron of layer $k$) is obtained through linear combinations of the previous layer and the non-linear activation function $\sigma$:

$$a_k^i=\sigma\left(\sum_{j=1}^nw_{kj}^ia_j^{i-1}+b_k^i\right)$$

If more than one layer (excluding input $x$ and output layer $\hat{y}$) is employed, the NN is considered a deep NN, and its training process is thereby deep learning. The evaluation of the NN, i.e., the prediction is referred to as forward propagation. The quality of prediction is determined by a cost function $C(\hat{y})$, which is to be minimized. Its gradients $\nabla_{\boldsymbol{\theta}}C={\nabla_{\boldsymbol{w}}C,\nabla_{\boldsymbol{b}}C}$ with respect to the parameters $\boldsymbol{\theta}$ are obtained with automatic differentiation, specifically referred to as backward propagation in the context of NNs. The gradients are used within a gradient-based optimization to update the parameters $\boldsymbol{\theta}$ and thereby improve the prediction $\hat{y}$.

3. Taxonomy from a methodological perspective

References

[1] Herrmann, L., & Kollmannsberger, S. (2024). Deep learning in computational mechanics: A review. Computational Mechanics.

Other angles

Deep learning for specific applications

Problem oriented perspective