Deep Learning|01 RBF Network

Deep Learning|01 RBF Network

Radial Basis Function Interpolation, RBF Interpolation

Radial basis functions are a class of real-valued functions that depend solely on the radial distance between the input vector and a fixed center point. Their core characteristic is that the function value is independent of the direction of the input and only related to the distance from the input to the center point.

Let $\Phi (x)$ is a RBF , for any input vector $\mathbf{x}$ , and center vectors $\mathbf{c}$ ,we define $\Phi (x)$ uesd Gauss functon :

$$\Phi (\mathbf{x})=e^{-\beta \Vert \mathbf{x}-\mathbf{c}{\Vert }^2}$$

which $\beta$ control the width of Gauss functon .

Note: It is not the only way. In fact, any similar $\delta$ function is available.

Radial Basis Function Interpolation, RBF Interpolation , is essentially an interpolation method that use a linear combination of radial basis functions (RBFs) to
“exactly approximate” the function $f(x)$ at known points.The formular is as follows:

$$f(x)=\sum _{j=1}^n{\lambda }_j{\Phi }_j(\mathbf{x})=\sum _{j=1}^n{\lambda }_j{e}^{-{\beta }_j\Vert \mathbf{x}-{\mathbf{c}}_j{\Vert }^2}$$

This defination is like Maximum Likelihood Estimation。This likely unrigorous？No! Brommehead and Lowe had given the proof of
RBF
. Magically!😊

RBF Network

Radial Basis Function Network (RBF Network) is a type of feedforward neural network with a single hidden layer. Its core feature is that it adopts “radial basis functions (RBFs)” as the activation functions of the hidden layer, and it is specifically used to solve supervised learning tasks such as nonlinear regression, pattern classification, and function approximation. Its key advantages lie in “strong local approximation capability, fast training speed, and stable generalization performance”, making it widely applied in small and medium-sized nonlinear problems.

$$\text{RBF Network}=\sum _{j=1}^n{\lambda }_j{\Phi }_j(\mathbf{x})$$

How to build a simple RBF network?

Aassuming we have series data {(x,y)}i=1n\{\left(\boldsymbol{x},y\right)\}_{i=1}^n{(x,y)}i=1n​ , the training objective of an RBF network is to determine three types of parameters: hidden layer centers {ci}inc\{\boldsymbol{c}_i\}_{i}^{n_c}{ci​}inc​​, hidden layer widths {σi}inσ\{\sigma_i\}_{i}^{n_\sigma}{σi​}inσ​​, and output layer weights {wi}inw\{w_i\}_{i}^{n_w}{wi​}inw​​.

Step1: load data and K-means clustering

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler
import osdef preprocess_data(data):data = data.dropna()if data.dtypes.iloc[-1] == 'object' or len(data.iloc[:, -1].unique()) < 10:X = data.iloc[:, :-1].valueshas_labels = Truelabels = data.iloc[:, -1].valueselse:X = data.valueshas_labels = Falselabels = Nonescaler = StandardScaler()X_scaled = scaler.fit_transform(X)return X_scaled, has_labels, labelsdef perform_kmeans(X, n_clusters=neural_numbers, random_state=42):kmeans = KMeans(n_clusters=n_clusters, random_state=random_state)clusters = kmeans.fit_predict(X)centroids = kmeans.cluster_centers_return clusters, centroidsdef visualize_results(X, original_labels, clusters, centroids, has_original_labels=False):fig, axes = plt.subplots(1, 2, figsize=(16, 8))has_original_labels and original_labels is not None:scatter1 = axes[0].scatter(X[:, 0], X[:, 1], c=original_labels, cmap='viridis', alpha=0.7)axes[0].set_title('oringal data distrbution', fontsize=14)plt.colorbar(scatter1, ax=axes[0], label='oringal lable')scatter2 = axes[1].scatter(X[:, 0], X[:, 1], c=clusters, cmap='viridis', alpha=0.7)axes[1].scatter(centroids[:, 0], centroids[:, 1], s=300, c='red', marker='X', label='clustering center')axes[1].set_title('result of clustering', fontsize=14)plt.colorbar(scatter2, ax=axes[1], label='clustering label')axes[1].legend()for ax in axes:ax.set_xlabel('feature_1（标准化）', fontsize=12)ax.set_ylabel('feature_2（标准化）', fontsize=12)ax.grid(True, linestyle='--', alpha=0.6)plt.tight_layout()if not os.path.exists('results'):os.makedirs('results')plt.savefig('results/clustering_comparison.png', dpi=300, bbox_inches='tight')print("figures save to : results/clustering_comparison.png")plt.show()def main(file_path, n_clusters=3):data = load_data(file_path)if data is None:returnX, has_labels, original_labels = preprocess_data(data)if X.shape[1] > 2:from sklearn.decomposition import PCAprint(f"data dimension is {X.shape[1]}，use PCA to 2-dimension to showase")pca = PCA(n_components=2)X = pca.fit_transform(X)clusters, centroids = perform_kmeans(X, n_clusters)visualize_results(X, original_labels, clusters, centroids, has_labels)if __name__ == "__main__":datapath="data.csv"neural_numbers=3data = pd.read_csv(file_path) # Must .csvmain(datapath, neural_numbers)

**Step2: Calculate the hidden layer widths **$

The widths do not require training and are usually calculated automatically based on the distribution of the centers.Most commonly used，adaptive calculation based on “nearest neighbor center distance”

For each center ${\mathbf{c}}_j$, first find another center ${\mathbf{c}}_k$ that is closest to it ($k\ne j$), and denote this distance as $d_j$:

$$d_j=\underset{\begin{array}{c}k=1,2,...,n\\ k\ne j\end{array}}{\min }\Vert {\mathbf{c}}_j-{\mathbf{c}}_k\Vert$$

(where $d_j$ represents the Euclidean distance from the $j$-th center to its nearest neighboring center)

Then, define ${\beta }_j$ as a quantity related to $d_j$ to ensure that the “effective range” of the RBF matches the distance between adjacent centers:

$${\beta }_j=\frac{1}{2\cdot d_j^2}$$

Step3: Solve for the output layer weights

After the centers {ci}\{\boldsymbol{c}_i\}{ci​} and widths {σi}\{\sigma_i\}{σi​} are determined, the hidden layer output $h_i(\mathbf{x})$ can be regarded as a “fixed nonlinear feature”. At this point, the solution for the output layer weights is transformed into a linear regression problem (gradient descent is not required, and the least squares method can be used directly).

Suppose the training set contains $N$ samples $({\mathbf{x}}_k,y_k)$ (where $y_k$ is the true label), and the network output is$${\hat{y}}_k={\sum }_{i=1}^m{w}_i{h}_i({\mathbf{x}}_k)+w_0$. By minimizing the “squared error between the predicted value and the true value”:

$$\mathcal{L}=\sum _{k=1}^N({\hat{y}}_k-y_k{)}^2$$

This can be converted into the matrix equation $\mathbf{H}\cdot \mathbf{w}=\mathbf{y}$), where:

• $\mathbf{H}$ is an $N\times (m+1)$ “feature matrix”, and each row is$[1,h_1({\mathbf{x}}_k),h_2({\mathbf{x}}_k),...,h_m({\mathbf{x}}_k)]$ (the 1 corresponds to the bias term $w_0$);
• $\mathbf{w}=[w_0,w_1,...,w_m{]}^T$ is the weight vector to be solved;
• $\mathbf{y}=[y_1,y_2,...,y_N{]}^T$ is the true label vector.

The weights are solved using the pseudoinverse matrix (to avoid the problem of matrix singularity):

$$\mathbf{w}=({\mathbf{H}}^T\mathbf{H}{)}^{-1}{\mathbf{H}}^T\mathbf{y}$$

This step is the key to the fast training of the RBF network — linear problems can be solved directly without repeated iterations.

Later, we will construct a Radial Basis Function (RBF) network to accomplish the prediction task.