TP3: Training pipeline sur GPU¶
Dans ce notebook nous allons voir comment procéder à l'entrainement et l'évaluation d'un réseau de neurone sur un problème simple de classification binaire sur GPU.
Dans un premier temps nous verrons comment procéder à l'entrainement sur GPU, puis nous l'utiliserons pour le problème de classification. Le but de ce problème de classification est: à partir des différentes caractéristiques d'une tumeur du sein, prédire si celle-ci est maligne ou bénigne.
In [6]:
import pandas as pd
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim
import matplotlib.pyplot as plt
I) Entrainement sur GPU¶
Voici une photo d'un CPU
et celle d'un GPU
Chaque SMX (Streaming Multiprocessor) n'est pas simplement un "coeur" individuel; il s'agit d'un processeur (multiprocesseur) capable de gérer lui-même des centaines de threads simultanément. Il y a tellement de threads dans ce genre de processeur qu'on les regroupe par ensembles de 32 threads, appelés un "warp".
La carte graphique de cette photo peut gérer jusqu'à 6 warps par multiprocesseur, avec 32 threads chacun, multipliés par 15 multiprocesseurs... soit un total de 2880 threads exécutés simultanément.
Vérifions si un GPU est accessible?
In [14]:
# vérifie si un GPU est disponible
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)
cpu
Nous allons illustrer l'utilisation du GPU sur un exemple du TP précédent.
In [15]:
x = np.linspace(-1, 1, 200)
y0 = np.sin(10*np.pi*x)
noise = 0.1 * np.random.randn(len(x))
y = y0 + noise
In [16]:
plt.plot(x, y0, lw=2, label="ground truth")
plt.plot(x, y, 'o', label="data")
plt.xlabel("x", fontsize=18)
plt.legend(fontsize=12, loc='upper right')
plt.show()
On reprend notre classe de MLP
In [17]:
class MLP(nn.Module):
def __init__(self, layers, activations):
"""
Parameters:
- layers: List of integer, the number of neurons in each layer (including input and output layers).
- activations: List of the activation functions for each layer.
The length of activations should be len(layers) - 2 (one for each hidden layer).
"""
super(MLP, self).__init__()
if len(activations) != len(layers) - 2:
raise ValueError("The number of activation functions must match the number of hidden layers.")
# Build the network
self.layers = nn.Sequential()
for i in range(len(layers) - 1):
self.layers.add_module(f"linear_{i}", nn.Linear(layers[i], layers[i + 1]))
if i < len(activations): # Apply activation to all but the last layer
self.layers.add_module(f"activation_{i}", activations[i])
def forward(self, x):
return self.layers(x)
Les données et le modèle qu'on va entrainer doivent se trouver sur le GPU. Pour les envoyer sur le GPU, on utilise la méthode .to(device).
In [18]:
# mise en forme des données, et transfert sur le GPU
x_data = torch.tensor(x, dtype=torch.float32).reshape(-1,1).to(device)
y_data = torch.tensor(y, dtype=torch.float32).reshape(-1,1).to(device)
In [19]:
layers = [1, 16, 32, 16, 1]
activations = [nn.Tanh(), nn.Tanh(), nn.Tanh()]
In [21]:
model = ML(layers, activations).to(device)
In [22]:
learning_rate = 1e-3
n_epoch = 15000
In [23]:
loss_fct = torch.nn.MSELoss(reduction='sum')
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
In [24]:
for t in range(n_epoch):
y_pred = model(x_data)
loss = loss_fct(y_pred, y_data)
if t%1000 == 0:
print("iteration=%s, loss=%s" % (t,loss.item()))
# On met à zéro les gradients avant de les calculer.
optimizer.zero_grad()
# On calcul les gradients
loss.backward()
# On met à jour les paramètres
optimizer.step()
print("iteration=%s, loss=%s" % (t,loss.item()))
iteration=0, loss=113.5762939453125 iteration=1000, loss=98.29638671875 iteration=2000, loss=63.20646667480469 iteration=3000, loss=25.832937240600586 iteration=4000, loss=14.342706680297852 iteration=5000, loss=7.458376407623291 iteration=6000, loss=2.37371826171875 iteration=7000, loss=1.915839672088623 iteration=8000, loss=1.7865893840789795 iteration=9000, loss=1.7252370119094849 iteration=10000, loss=1.687633991241455 iteration=11000, loss=1.6484202146530151 iteration=12000, loss=1.6147209405899048 iteration=13000, loss=1.5875564813613892 iteration=14000, loss=1.5695767402648926 iteration=14999, loss=1.5539531707763672
Une fois entrainé, on peut faire des opérations avec notre modèle sur le GPU, mais pour afficher des tableaux il faut les rappatrier sur le CPU. Pour cela, on utilise la méthode .cpu().
In [26]:
y_pred_plot = y_pred.detach().cpu()
plt.plot(x, y, "o", label="data")
plt.plot(x, y_pred_plot, "o", label="predictions")
plt.plot(x, y0, lw=2, label="ground truth")
plt.xlabel("x", fontsize=18)
plt.legend(fontsize=12, loc='upper right')
plt.show()
II) Application à un problème simple de classification binaire¶
1) Le jeu de données¶
Commençons par visualiser et mettre en forme le jeu de données.
In [28]:
import pandas as pd
import numpy as np
In [29]:
## sur google colab effectuer executer
## les lignes suivantes et selectionner le jeu de données
## sur votre ordinateur
# from google.colab import files
# data_to_load = files.upload()
In [ ]:
data = pd.read_csv("breast-cancer.csv")
In [30]:
data.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 569 entries, 0 to 568 Data columns (total 32 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 id 569 non-null int64 1 diagnosis 569 non-null object 2 radius_mean 569 non-null float64 3 texture_mean 569 non-null float64 4 perimeter_mean 569 non-null float64 5 area_mean 569 non-null float64 6 smoothness_mean 569 non-null float64 7 compactness_mean 569 non-null float64 8 concavity_mean 569 non-null float64 9 concave points_mean 569 non-null float64 10 symmetry_mean 569 non-null float64 11 fractal_dimension_mean 569 non-null float64 12 radius_se 569 non-null float64 13 texture_se 569 non-null float64 14 perimeter_se 569 non-null float64 15 area_se 569 non-null float64 16 smoothness_se 569 non-null float64 17 compactness_se 569 non-null float64 18 concavity_se 569 non-null float64 19 concave points_se 569 non-null float64 20 symmetry_se 569 non-null float64 21 fractal_dimension_se 569 non-null float64 22 radius_worst 569 non-null float64 23 texture_worst 569 non-null float64 24 perimeter_worst 569 non-null float64 25 area_worst 569 non-null float64 26 smoothness_worst 569 non-null float64 27 compactness_worst 569 non-null float64 28 concavity_worst 569 non-null float64 29 concave points_worst 569 non-null float64 30 symmetry_worst 569 non-null float64 31 fractal_dimension_worst 569 non-null float64 dtypes: float64(30), int64(1), object(1) memory usage: 142.4+ KB
In [31]:
data.head()
Out[31]:
| id | diagnosis | radius_mean | texture_mean | perimeter_mean | area_mean | smoothness_mean | compactness_mean | concavity_mean | concave points_mean | ... | radius_worst | texture_worst | perimeter_worst | area_worst | smoothness_worst | compactness_worst | concavity_worst | concave points_worst | symmetry_worst | fractal_dimension_worst | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 842302 | M | 17.99 | 10.38 | 122.80 | 1001.0 | 0.11840 | 0.27760 | 0.3001 | 0.14710 | ... | 25.38 | 17.33 | 184.60 | 2019.0 | 0.1622 | 0.6656 | 0.7119 | 0.2654 | 0.4601 | 0.11890 |
| 1 | 842517 | M | 20.57 | 17.77 | 132.90 | 1326.0 | 0.08474 | 0.07864 | 0.0869 | 0.07017 | ... | 24.99 | 23.41 | 158.80 | 1956.0 | 0.1238 | 0.1866 | 0.2416 | 0.1860 | 0.2750 | 0.08902 |
| 2 | 84300903 | M | 19.69 | 21.25 | 130.00 | 1203.0 | 0.10960 | 0.15990 | 0.1974 | 0.12790 | ... | 23.57 | 25.53 | 152.50 | 1709.0 | 0.1444 | 0.4245 | 0.4504 | 0.2430 | 0.3613 | 0.08758 |
| 3 | 84348301 | M | 11.42 | 20.38 | 77.58 | 386.1 | 0.14250 | 0.28390 | 0.2414 | 0.10520 | ... | 14.91 | 26.50 | 98.87 | 567.7 | 0.2098 | 0.8663 | 0.6869 | 0.2575 | 0.6638 | 0.17300 |
| 4 | 84358402 | M | 20.29 | 14.34 | 135.10 | 1297.0 | 0.10030 | 0.13280 | 0.1980 | 0.10430 | ... | 22.54 | 16.67 | 152.20 | 1575.0 | 0.1374 | 0.2050 | 0.4000 | 0.1625 | 0.2364 | 0.07678 |
5 rows × 32 columns
In [32]:
data.shape
Out[32]:
(569, 32)
In [33]:
n_data, n_feat = data.shape
n_feat -= 2 # on retire id et diagnosis de inputs.
In [34]:
# vérifie s'il y a des données manquantes
data.isnull().sum()
Out[34]:
id 0 diagnosis 0 radius_mean 0 texture_mean 0 perimeter_mean 0 area_mean 0 smoothness_mean 0 compactness_mean 0 concavity_mean 0 concave points_mean 0 symmetry_mean 0 fractal_dimension_mean 0 radius_se 0 texture_se 0 perimeter_se 0 area_se 0 smoothness_se 0 compactness_se 0 concavity_se 0 concave points_se 0 symmetry_se 0 fractal_dimension_se 0 radius_worst 0 texture_worst 0 perimeter_worst 0 area_worst 0 smoothness_worst 0 compactness_worst 0 concavity_worst 0 concave points_worst 0 symmetry_worst 0 fractal_dimension_worst 0 dtype: int64
In [35]:
data.describe()
Out[35]:
| id | radius_mean | texture_mean | perimeter_mean | area_mean | smoothness_mean | compactness_mean | concavity_mean | concave points_mean | symmetry_mean | ... | radius_worst | texture_worst | perimeter_worst | area_worst | smoothness_worst | compactness_worst | concavity_worst | concave points_worst | symmetry_worst | fractal_dimension_worst | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| count | 5.690000e+02 | 569.000000 | 569.000000 | 569.000000 | 569.000000 | 569.000000 | 569.000000 | 569.000000 | 569.000000 | 569.000000 | ... | 569.000000 | 569.000000 | 569.000000 | 569.000000 | 569.000000 | 569.000000 | 569.000000 | 569.000000 | 569.000000 | 569.000000 |
| mean | 3.037183e+07 | 14.127292 | 19.289649 | 91.969033 | 654.889104 | 0.096360 | 0.104341 | 0.088799 | 0.048919 | 0.181162 | ... | 16.269190 | 25.677223 | 107.261213 | 880.583128 | 0.132369 | 0.254265 | 0.272188 | 0.114606 | 0.290076 | 0.083946 |
| std | 1.250206e+08 | 3.524049 | 4.301036 | 24.298981 | 351.914129 | 0.014064 | 0.052813 | 0.079720 | 0.038803 | 0.027414 | ... | 4.833242 | 6.146258 | 33.602542 | 569.356993 | 0.022832 | 0.157336 | 0.208624 | 0.065732 | 0.061867 | 0.018061 |
| min | 8.670000e+03 | 6.981000 | 9.710000 | 43.790000 | 143.500000 | 0.052630 | 0.019380 | 0.000000 | 0.000000 | 0.106000 | ... | 7.930000 | 12.020000 | 50.410000 | 185.200000 | 0.071170 | 0.027290 | 0.000000 | 0.000000 | 0.156500 | 0.055040 |
| 25% | 8.692180e+05 | 11.700000 | 16.170000 | 75.170000 | 420.300000 | 0.086370 | 0.064920 | 0.029560 | 0.020310 | 0.161900 | ... | 13.010000 | 21.080000 | 84.110000 | 515.300000 | 0.116600 | 0.147200 | 0.114500 | 0.064930 | 0.250400 | 0.071460 |
| 50% | 9.060240e+05 | 13.370000 | 18.840000 | 86.240000 | 551.100000 | 0.095870 | 0.092630 | 0.061540 | 0.033500 | 0.179200 | ... | 14.970000 | 25.410000 | 97.660000 | 686.500000 | 0.131300 | 0.211900 | 0.226700 | 0.099930 | 0.282200 | 0.080040 |
| 75% | 8.813129e+06 | 15.780000 | 21.800000 | 104.100000 | 782.700000 | 0.105300 | 0.130400 | 0.130700 | 0.074000 | 0.195700 | ... | 18.790000 | 29.720000 | 125.400000 | 1084.000000 | 0.146000 | 0.339100 | 0.382900 | 0.161400 | 0.317900 | 0.092080 |
| max | 9.113205e+08 | 28.110000 | 39.280000 | 188.500000 | 2501.000000 | 0.163400 | 0.345400 | 0.426800 | 0.201200 | 0.304000 | ... | 36.040000 | 49.540000 | 251.200000 | 4254.000000 | 0.222600 | 1.058000 | 1.252000 | 0.291000 | 0.663800 | 0.207500 |
8 rows × 31 columns
In [36]:
# on retire id (qui ne sert à rien pour notre tâche)
# diagnosis qui correspond aux sorties de notre classifieur
X = data.drop(columns={'id','diagnosis'}, axis=1)
Y = data['diagnosis'].values
In [37]:
Y
Out[37]:
array(['M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M',
'M', 'M', 'M', 'M', 'M', 'M', 'B', 'B', 'B', 'M', 'M', 'M', 'M',
'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'B', 'M',
'M', 'M', 'M', 'M', 'M', 'M', 'M', 'B', 'M', 'B', 'B', 'B', 'B',
'B', 'M', 'M', 'B', 'M', 'M', 'B', 'B', 'B', 'B', 'M', 'B', 'M',
'M', 'B', 'B', 'B', 'B', 'M', 'B', 'M', 'M', 'B', 'M', 'B', 'M',
'M', 'B', 'B', 'B', 'M', 'M', 'B', 'M', 'M', 'M', 'B', 'B', 'B',
'M', 'B', 'B', 'M', 'M', 'B', 'B', 'B', 'M', 'M', 'B', 'B', 'B',
'B', 'M', 'B', 'B', 'M', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B',
'M', 'M', 'M', 'B', 'M', 'M', 'B', 'B', 'B', 'M', 'M', 'B', 'M',
'B', 'M', 'M', 'B', 'M', 'M', 'B', 'B', 'M', 'B', 'B', 'M', 'B',
'B', 'B', 'B', 'M', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B',
'M', 'B', 'B', 'B', 'B', 'M', 'M', 'B', 'M', 'B', 'B', 'M', 'M',
'B', 'B', 'M', 'M', 'B', 'B', 'B', 'B', 'M', 'B', 'B', 'M', 'M',
'M', 'B', 'M', 'B', 'M', 'B', 'B', 'B', 'M', 'B', 'B', 'M', 'M',
'B', 'M', 'M', 'M', 'M', 'B', 'M', 'M', 'M', 'B', 'M', 'B', 'M',
'B', 'B', 'M', 'B', 'M', 'M', 'M', 'M', 'B', 'B', 'M', 'M', 'B',
'B', 'B', 'M', 'B', 'B', 'B', 'B', 'B', 'M', 'M', 'B', 'B', 'M',
'B', 'B', 'M', 'M', 'B', 'M', 'B', 'B', 'B', 'B', 'M', 'B', 'B',
'B', 'B', 'B', 'M', 'B', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M',
'M', 'M', 'M', 'M', 'M', 'M', 'B', 'B', 'B', 'B', 'B', 'B', 'M',
'B', 'M', 'B', 'B', 'M', 'B', 'B', 'M', 'B', 'M', 'M', 'B', 'B',
'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'M', 'B',
'B', 'M', 'B', 'M', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B',
'B', 'B', 'B', 'B', 'B', 'M', 'B', 'B', 'B', 'M', 'B', 'M', 'B',
'B', 'B', 'B', 'M', 'M', 'M', 'B', 'B', 'B', 'B', 'M', 'B', 'M',
'B', 'M', 'B', 'B', 'B', 'M', 'B', 'B', 'B', 'B', 'B', 'B', 'B',
'M', 'M', 'M', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B',
'B', 'M', 'M', 'B', 'M', 'M', 'M', 'B', 'M', 'M', 'B', 'B', 'B',
'B', 'B', 'M', 'B', 'B', 'B', 'B', 'B', 'M', 'B', 'B', 'B', 'M',
'B', 'B', 'M', 'M', 'B', 'B', 'B', 'B', 'B', 'B', 'M', 'B', 'B',
'B', 'B', 'B', 'B', 'B', 'M', 'B', 'B', 'B', 'B', 'B', 'M', 'B',
'B', 'M', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B',
'B', 'M', 'B', 'M', 'M', 'B', 'M', 'B', 'B', 'B', 'B', 'B', 'M',
'B', 'B', 'M', 'B', 'M', 'B', 'B', 'M', 'B', 'M', 'B', 'B', 'B',
'B', 'B', 'B', 'B', 'B', 'M', 'M', 'B', 'B', 'B', 'B', 'B', 'B',
'M', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'M', 'B',
'B', 'B', 'B', 'B', 'B', 'B', 'M', 'B', 'M', 'B', 'B', 'M', 'B',
'B', 'B', 'B', 'B', 'M', 'M', 'B', 'M', 'B', 'M', 'B', 'B', 'B',
'B', 'B', 'M', 'B', 'B', 'M', 'B', 'M', 'B', 'M', 'M', 'B', 'B',
'B', 'M', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B',
'M', 'B', 'M', 'M', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B',
'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B',
'B', 'B', 'B', 'M', 'M', 'M', 'M', 'M', 'M', 'B'], dtype=object)
Transformons les 'M' (Maligne) et 'B' (Benigne) en 1 et 0 pour un usage mathématique
In [38]:
Y = np.array([1. if Y[i] == 'B' else 0. for i in range(n_data)])
Y
Out[38]:
array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 1., 1., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 1., 1., 1.,
1., 1., 0., 0., 1., 0., 0., 1., 1., 1., 1., 0., 1., 0., 0., 1., 1.,
1., 1., 0., 1., 0., 0., 1., 0., 1., 0., 0., 1., 1., 1., 0., 0., 1.,
0., 0., 0., 1., 1., 1., 0., 1., 1., 0., 0., 1., 1., 1., 0., 0., 1.,
1., 1., 1., 0., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 0., 0.,
0., 1., 0., 0., 1., 1., 1., 0., 0., 1., 0., 1., 0., 0., 1., 0., 0.,
1., 1., 0., 1., 1., 0., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 0., 1., 1., 1., 1., 0., 0., 1., 0., 1., 1., 0., 0., 1.,
1., 0., 0., 1., 1., 1., 1., 0., 1., 1., 0., 0., 0., 1., 0., 1., 0.,
1., 1., 1., 0., 1., 1., 0., 0., 1., 0., 0., 0., 0., 1., 0., 0., 0.,
1., 0., 1., 0., 1., 1., 0., 1., 0., 0., 0., 0., 1., 1., 0., 0., 1.,
1., 1., 0., 1., 1., 1., 1., 1., 0., 0., 1., 1., 0., 1., 1., 0., 0.,
1., 0., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 0., 1., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 1., 1., 1., 1.,
0., 1., 0., 1., 1., 0., 1., 1., 0., 1., 0., 0., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 0., 1., 0., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 0., 1.,
0., 1., 1., 1., 1., 0., 0., 0., 1., 1., 1., 1., 0., 1., 0., 1., 0.,
1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 0., 0., 0., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 0., 0., 1., 0., 0., 0., 1., 0., 0.,
1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 0., 1., 1., 1., 0., 1.,
1., 0., 0., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1.,
0., 1., 1., 1., 1., 1., 0., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 0., 1., 0., 0., 1., 0., 1., 1., 1., 1., 1., 0.,
1., 1., 0., 1., 0., 1., 1., 0., 1., 0., 1., 1., 1., 1., 1., 1., 1.,
1., 0., 0., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 0., 1., 0., 1., 1., 0.,
1., 1., 1., 1., 1., 0., 0., 1., 0., 1., 0., 1., 1., 1., 1., 1., 0.,
1., 1., 0., 1., 0., 1., 0., 0., 1., 1., 1., 0., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 0., 1., 0., 0., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 0., 0., 0., 0., 0., 0., 1.])
Normalisons les inputs (données d'entrées) pour éviter les différences d'échelle en les différentes entrées.
In [39]:
from sklearn.preprocessing import StandardScaler
In [40]:
# standardise chaque colonne des données par (x-mean(x))/std(x)
scaler = StandardScaler()
In [41]:
X = scaler.fit_transform(X)
Maintenant on va couper (de manière aléatoire) le jeu de données en une partie qui va servir à l'entrainenement et l'évaluation des différentes architectures (X_train et Y_train), représentant 80% des données, et une partie test (X_test et Y_test) qui nous donnera un score final (qu'on pourra comparer à celui obtenu avec d'autre classifieur)
In [42]:
from sklearn.model_selection import train_test_split
In [43]:
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, shuffle=True)
In [44]:
print(X.shape, X_train.shape, X_test.shape)
(569, 30) (455, 30) (114, 30)
2) Classe de réseau de neurons¶
Introduisons notre classe de réseau de neurone. Mais cette fois, on va mettre une fonction d'activation finale sigmoid.
In [45]:
class MLP(nn.Module):
def __init__(self, layers, activations):
"""
Parameters:
- layers: List of integer, the number of neurons in each layer (including input and output layers).
- activations: List of the activation functions for each layer.
The length of activations should be len(layers) - 2 (one for each hidden layer).
"""
super().__init__()
if len(activations) != len(layers) - 1:
raise ValueError("The number of activation functions is not correct.")
# Build the network
self.layers = nn.Sequential()
for i in range(len(layers) - 1):
self.layers.add_module(f"linear_{i}", nn.Linear(layers[i], layers[i + 1]))
self.layers.add_module(f"activation_{i}", activations[i])
def forward(self, x):
return self.layers(x)
In [46]:
layers = [n_feat, 20, 20, 1]
activations = [nn.ReLU(), nn.ReLU(), nn.Sigmoid()]
In [47]:
model = MLP(layers, activations)
print(model)
MLP(
(layers): Sequential(
(linear_0): Linear(in_features=30, out_features=20, bias=True)
(activation_0): ReLU()
(linear_1): Linear(in_features=20, out_features=20, bias=True)
(activation_1): ReLU()
(linear_2): Linear(in_features=20, out_features=1, bias=True)
(activation_2): Sigmoid()
)
)
3) Entrainement¶
Définissons la fonction d'entrainement pour un modèle donné.
In [48]:
def train(model, optimizer, loss_fct, input_data, output_data, n_epoch):
"""
Trains a given model on specific data for a defined number of epochs.
Parameters:
- model : neural network model to be trained
- optimizer : optimization method to update the model's weights
- loss_fct : the loss function to measure the error between the model's predictions and the target data
- input_data : features of the tumors
- output_data : tumors status
- n_epochs : number of epochs for training
"""
for t in range(n_epoch):
pred = model(input_data)
loss = loss_fct(pred, output_data)
# On met à zéro les gradients avant de les calculer.
optimizer.zero_grad()
# On calcul les gradients
loss.backward()
# On met à jour les paramètres
optimizer.step()
4) Evaluation¶
Le critère d'évaluation sera ici la précision de classification, c'est à dire la proportion de données correctement classifiées.
In [49]:
from sklearn.metrics import accuracy_score
In [50]:
def evaluate_model(model, X_eval, Y_eval):
"""
Evaluates the accuracy of the model on evaluation data.
Parameters:
- model : neural network model to be trained
- X_eval : inputs for evaluation
- Y_eval : output for evaluation
"""
with torch.no_grad():
pred_eval = model(X_eval)
pred_eval = (pred_eval >= 0.5)
test_acc = accuracy_score(Y_eval.cpu(), pred_eval.cpu()) # move back to CPU for metrics
return test_acc
5) k-fold cross validation¶
Définissons nos boucles d'entrainement et d'évaluation par k-fold cross validation.
In [51]:
from sklearn.model_selection import KFold
In [58]:
def train_model(X_train, Y_train, loss_fct, kf, lr, n_epoch, layers, activations):
"""
Trains and evaluates a given model on specific by k-fold cross validation.
Parameters:
- X_train : features of the tumors for training
- Y_data : tumor status for training
- loss_fct : the loss function to measure the error between the model's predictions and the target data
- kf : indexing for the data splitting
- output_data : tumors status
- epochs : number of epochs for training
- lr : learning rate for the optimization method
- n_epochs : number of epochs for training
- layers : list describing the layer stucture of the model
- activations : activations function in the network
"""
# store the result of each evaluation
fold_results = []
# perfome the training and evalution of the model
for fold, (train_idx, val_idx) in enumerate(kf.split(X_train)):
# print(f"Fold {fold + 1}/{k}")
# split into training and validation for this fold
X_train_fold, X_val_fold = X_train[train_idx], X_train[val_idx]
Y_train_fold, Y_val_fold = Y_train[train_idx], Y_train[val_idx]
# convert to torch tensors and move them to the GPU
X_train_fold_dev = torch.tensor(X_train_fold, dtype=torch.float32).to(device)
Y_train_fold_dev = torch.tensor(Y_train_fold, dtype=torch.float32).reshape(-1,1).to(device)
X_val_fold_dev = torch.tensor(X_val_fold, dtype=torch.float32).to(device)
Y_val_fold_dev = torch.tensor(Y_val_fold, dtype=torch.float32).reshape(-1,1).to(device)
# initialize the model, loss function, and optimizer
model = MLP(layers, activations).to(device)
optimizer = optim.Adam(model.parameters(), lr=lr)
train(model, optimizer, loss_fct, X_train_fold_dev, Y_train_fold_dev, n_epoch)
test_acc = evaluate_model(model, X_val_fold_dev, Y_val_fold_dev)
# print(f"accuracy: {test_acc:.4f}")
fold_results.append(test_acc)
score = np.mean(fold_results)
# print(f"average validation accuracy: {score:.4f}")
return score
In [59]:
# fonction de coût
loss_fct = nn.BCELoss()
In [60]:
# nombre de répétition de l'évaluation du modèle
k = 5
# génère l'indexation pour la division des données
kf = KFold(n_splits=k)
6) Boucle sur les différentes architectures¶
A l'aide des fonctions précédentes, on va comparer différentes architectures. On va comparer les raiseaux de 1 à 5 couches cachées, et chaque couche ayant 10, 20, 30, 40, 50 neurones.
In [61]:
n_epoch = 5000 # pas nécessairement suffisant
learning_rate = 0.001 # pas nécessairement assez petit
In [62]:
n_hidden_layers = range(1, 6)
n_neuron = range(10, 60, 10)
In [64]:
# evaluation du score sur les différentes architecture
scores = np.zeros((len(n_hidden_layers), len(n_neuron)))
for j,L in enumerate(n_hidden_layers):
# define the activation list
activations = [nn.ReLU()]*L
activations.append(nn.Sigmoid())
for l, m in enumerate(n_neuron):
# define the network structure
layers = [m]*L
layers.append(1)
layers.insert(0, n_feat)
score = train_model(X_train, Y_train, loss_fct, kf, learning_rate, n_epoch, layers, activations)
print(f"score: {score:.4f} for {L} hidden layers of {m} neurons")
scores[j,l] = score
score: 0.9692 for 1 hidden layers of 0 neurons score: 0.9692 for 1 hidden layers of 1 neurons score: 0.9648 for 1 hidden layers of 2 neurons score: 0.9692 for 1 hidden layers of 3 neurons score: 0.9670 for 1 hidden layers of 4 neurons score: 0.9692 for 2 hidden layers of 0 neurons score: 0.9736 for 2 hidden layers of 1 neurons score: 0.9648 for 2 hidden layers of 2 neurons score: 0.9582 for 2 hidden layers of 3 neurons score: 0.9626 for 2 hidden layers of 4 neurons score: 0.9736 for 3 hidden layers of 0 neurons score: 0.9582 for 3 hidden layers of 1 neurons score: 0.9582 for 3 hidden layers of 2 neurons score: 0.9604 for 3 hidden layers of 3 neurons score: 0.9626 for 3 hidden layers of 4 neurons score: 0.9670 for 4 hidden layers of 0 neurons score: 0.9692 for 4 hidden layers of 1 neurons score: 0.9648 for 4 hidden layers of 2 neurons score: 0.9582 for 4 hidden layers of 3 neurons score: 0.9582 for 4 hidden layers of 4 neurons score: 0.9670 for 5 hidden layers of 0 neurons score: 0.9604 for 5 hidden layers of 1 neurons score: 0.9582 for 5 hidden layers of 2 neurons score: 0.9604 for 5 hidden layers of 3 neurons score: 0.9582 for 5 hidden layers of 4 neurons
In [65]:
# retourne les indices de la matrice scores où elle a sa plus grande valeur
j,l = np.unravel_index(scores.argmax(), scores.shape)
print(f'Le meilleur réseau à {n_hidden_layers[j]} couches cachées de {n_neuron[l]} neurones')
Le meilleur réseau à 2 couches cachées de 20 neurones
7) Réentrainement de l'architecture qui a obtenue le meilleur score.¶
Une fois la meilleur architecture trouvée, on réentraine un modèle avec l'architecture ayant eu le meilleur score sur toutes les données d'entrainement.
In [75]:
# On redéfinie l'architecture ayant eu le meilleur score
L = n_hidden_layers[j]
m = n_neuron[l]
In [76]:
layers = [m]*L
layers.append(1)
layers.insert(0, n_feat)
In [77]:
activations = [nn.ReLU()]*L
activations.append(nn.Sigmoid())
In [78]:
# On reféfinit un modèle qu'on envoie sur le GPU
model = MLP(layers, activations).to(device)
optimizer = optim.Adam(model.parameters(), lr=learning_rate)
In [79]:
# on envoie les données d'entrainement sur le GPU
X_train_dev = torch.tensor(X_train, dtype=torch.float32).to(device)
Y_train_dev = torch.tensor(Y_train, dtype=torch.float32).reshape(-1,1).to(device)
In [80]:
# On entraine le modèle
train(model, optimizer, loss_fct, X_train_dev, Y_train_dev, n_epoch)
8) Evaluation du score sur les données de test.¶
Il ne reste plus qu'à évaluer ce réseau entrainé sur les données de test pour calculer son score, qu'on pourra comparer à d'autres modèles de classification.
In [83]:
# on envoie les données de test sur le GPU
# pour être évalué par le modèle.
X_test_dev = torch.tensor(X_test, dtype=torch.float32).to(device)
Y_test_dev = torch.tensor(Y_test, dtype=torch.float32).reshape(-1,1).to(device)
In [85]:
# score final
evaluate_model(model, X_test_dev, Y_test_dev)
Out[85]:
0.9736842105263158