7. Simulation studies using sstudy
One important aspect of proposing new machine learning/Statistical estimators and methods is the performance test phrase. With that in mind, we present here a short introduction on package `sstudy <https://sstudy.marcoinacio.com/>`__ that facilitates such procedure.
[1]:
!pip install sstudy
Requirement already satisfied: sstudy in /home/marco/Documents/projects/sstudy (0.0.5)
Requirement already satisfied: peewee in /home/marco/miniforge3/lib/python3.7/site-packages (from sstudy) (3.10.0)
Let us first define the structure of our dataset and create it:
[2]:
from peewee import *
import os
db = SqliteDatabase('results.sqlite3')
class Result(Model):
# Data settings
data_distribution = TextField()
method = TextField()
no_instances = DoubleField()
# Results
score = DoubleField()
elapsed_time = DoubleField()
class Meta:
database = db
Result.create_table()
Now, let’s run the simulations:
[3]:
import numpy as np
import time
from scipy import stats
from sklearn.linear_model import LinearRegression, Lasso
from sstudy import do_simulation_study
no_simulations = 10
to_sample = dict(
data_distribution = [0,1],
no_instances = [100, 1000],
method = ['ols', 'lasso'],
)
def func(
data_distribution,
no_instances,
method,
):
x = stats.norm.rvs(0, 2, size=(no_instances + 10000, 10))
beta = stats.norm.rvs(0, 2, size=(10, 1))
eps = stats.norm.rvs(0, 3, size=(no_instances + 10000, 1))
if data_distribution == 0:
y = np.matmul(x, beta) + eps
if data_distribution == 1:
y = np.matmul(x[:,:5], beta[:5]) + eps
y_train = y[:no_instances]
y_test = y[no_instances:]
x_train = x[:no_instances]
x_test = x[no_instances:]
start_time = time.time()
if method == 'ols':
reg = LinearRegression()
elif method == 'lasso':
reg = Lasso(alpha=0.1)
reg.fit(x_train, y_train)
score = reg.score(x_test, y_test)
elapsed_time = time.time() - start_time
return dict(
score = score,
elapsed_time = elapsed_time,
)
do_simulation_study(to_sample, func, db, Result, max_count=no_simulations)
8 combinations left
Result:
{'score': 0.6139337132523494, 'elapsed_time': 0.02861189842224121}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.8611576272271733, 'elapsed_time': 0.011518239974975586}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.926857889391819, 'elapsed_time': 0.0023622512817382812}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.8548821642738853, 'elapsed_time': 0.009645700454711914}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.8870249256740823, 'elapsed_time': 0.00249481201171875}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.8409022813797401, 'elapsed_time': 0.004044294357299805}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.8448752757733746, 'elapsed_time': 0.0037097930908203125}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.8897067829342962, 'elapsed_time': 0.004401683807373047}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9015128091852608, 'elapsed_time': 0.005560636520385742}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9173762118243225, 'elapsed_time': 0.003095388412475586}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.778717427545883, 'elapsed_time': 0.004297971725463867}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.34999607019544277, 'elapsed_time': 0.0112152099609375}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9339442945166084, 'elapsed_time': 0.004238128662109375}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9491677210715785, 'elapsed_time': 0.012585878372192383}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9550733471904924, 'elapsed_time': 0.009177207946777344}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.8697649290337729, 'elapsed_time': 0.007660865783691406}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9162499804715406, 'elapsed_time': 0.0050165653228759766}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9452732624279011, 'elapsed_time': 0.028743982315063477}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.950441811326444, 'elapsed_time': 0.006459951400756836}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.7668055526121543, 'elapsed_time': 0.016127347946166992}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.8557961743715387, 'elapsed_time': 0.0022199153900146484}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9363305322995981, 'elapsed_time': 0.002021312713623047}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9618996130598892, 'elapsed_time': 0.004746913909912109}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9332619267690534, 'elapsed_time': 0.012091398239135742}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9664944916259364, 'elapsed_time': 0.002245664596557617}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9204026973895163, 'elapsed_time': 0.01087045669555664}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.8751218277903129, 'elapsed_time': 0.0046579837799072266}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.972468591998792, 'elapsed_time': 0.013517379760742188}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.7573226196360956, 'elapsed_time': 0.002424955368041992}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.7424114258603562, 'elapsed_time': 0.016463279724121094}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9171756230714264, 'elapsed_time': 0.009216547012329102}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.8693280456380904, 'elapsed_time': 0.010242938995361328}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9198453720969424, 'elapsed_time': 0.0067365169525146484}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9151432918105386, 'elapsed_time': 0.005576133728027344}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.8823511788043035, 'elapsed_time': 0.003439664840698242}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9076904770665448, 'elapsed_time': 0.014777898788452148}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.7714421603477339, 'elapsed_time': 0.0018000602722167969}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9275495597560977, 'elapsed_time': 0.006983757019042969}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.7656507317306436, 'elapsed_time': 0.00559687614440918}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9478856878095755, 'elapsed_time': 0.007489919662475586}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9277319659081824, 'elapsed_time': 0.005853891372680664}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9553898999855582, 'elapsed_time': 0.011910438537597656}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.8025482879051544, 'elapsed_time': 0.003132343292236328}
Result successfully stored in the database
8 combinations left
7 combinations left
Result:
{'score': 0.8796935556780558, 'elapsed_time': 0.0018222332000732422}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.9490630333532628, 'elapsed_time': 0.022101402282714844}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.9690228203456708, 'elapsed_time': 0.003087759017944336}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.95316118627402, 'elapsed_time': 0.006292104721069336}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.9175738906058983, 'elapsed_time': 0.0032949447631835938}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.9604022502319571, 'elapsed_time': 0.00570225715637207}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.9501598049429956, 'elapsed_time': 0.002569437026977539}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.7774176826009725, 'elapsed_time': 0.008621692657470703}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.9133612124782576, 'elapsed_time': 0.0167999267578125}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.653043736331703, 'elapsed_time': 0.014389753341674805}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.9327320755408688, 'elapsed_time': 0.018154621124267578}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.9016522030811711, 'elapsed_time': 0.006725788116455078}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.9496257758259826, 'elapsed_time': 0.009273290634155273}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.6183744207668223, 'elapsed_time': 0.007914066314697266}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.9663373421756023, 'elapsed_time': 0.008886098861694336}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.593423137125811, 'elapsed_time': 0.004263877868652344}
Result successfully stored in the database
7 combinations left
6 combinations left
Result:
{'score': 0.9158234579741112, 'elapsed_time': 0.015735864639282227}
Result successfully stored in the database
6 combinations left
Result:
{'score': 0.9229704225126353, 'elapsed_time': 0.009942770004272461}
Result successfully stored in the database
6 combinations left
Result:
{'score': 0.9131556194519059, 'elapsed_time': 0.0054168701171875}
Result successfully stored in the database
6 combinations left
Result:
{'score': 0.9095443217072087, 'elapsed_time': 0.004354000091552734}
Result successfully stored in the database
6 combinations left
Result:
{'score': 0.9559537340091908, 'elapsed_time': 0.002058744430541992}
Result successfully stored in the database
6 combinations left
Result:
{'score': 0.9455294877839799, 'elapsed_time': 0.012708187103271484}
Result successfully stored in the database
6 combinations left
Result:
{'score': 0.9145785519039111, 'elapsed_time': 0.002496957778930664}
Result successfully stored in the database
6 combinations left
Result:
{'score': 0.9219038507533641, 'elapsed_time': 0.0185697078704834}
Result successfully stored in the database
6 combinations left
Result:
{'score': 0.9620256020410831, 'elapsed_time': 0.0018486976623535156}
Result successfully stored in the database
6 combinations left
Result:
{'score': 0.9019879200435615, 'elapsed_time': 0.011387109756469727}
Result successfully stored in the database
6 combinations left
Result:
{'score': 0.9386660910357864, 'elapsed_time': 0.002043008804321289}
Result successfully stored in the database
6 combinations left
Result:
{'score': 0.9457011386599795, 'elapsed_time': 0.021100521087646484}
Result successfully stored in the database
6 combinations left
Result:
{'score': 0.9213863243859013, 'elapsed_time': 0.004567861557006836}
Result successfully stored in the database
6 combinations left
5 combinations left
4 combinations left
Result:
{'score': 0.9634731741331573, 'elapsed_time': 0.008003950119018555}
Result successfully stored in the database
4 combinations left
Result:
{'score': 0.9621886490300308, 'elapsed_time': 0.005016803741455078}
Result successfully stored in the database
4 combinations left
Result:
{'score': 0.9584995497741426, 'elapsed_time': 0.0041790008544921875}
Result successfully stored in the database
4 combinations left
Result:
{'score': 0.9402866422513141, 'elapsed_time': 0.001947164535522461}
Result successfully stored in the database
4 combinations left
Result:
{'score': 0.949407023370497, 'elapsed_time': 0.020636796951293945}
Result successfully stored in the database
4 combinations left
Result:
{'score': 0.9469646290312033, 'elapsed_time': 0.0047147274017333984}
Result successfully stored in the database
4 combinations left
Result:
{'score': 0.9076465139324088, 'elapsed_time': 0.004822969436645508}
Result successfully stored in the database
4 combinations left
3 combinations left
2 combinations left
1 combinations left
Result:
{'score': 0.8260911148699304, 'elapsed_time': 0.0027403831481933594}
Result successfully stored in the database
1 combinations left
The good news is that sqlite works though atomic transactions, so either a commit (i.e.: adding a result to the dataset) will happen entirelly or it won’t happen at all.
Therefore, you can kill the simulation study process without fear that the dataset will became corrupted in case you happen to kill while it’s commiting the results.
sstudy works chooses the test cases randomly and independently, therefore, you can spawn multiple simulation study processes that will work independly and can be terminated at any moment.
[4]:
import numpy as np
import pandas as pd
from matplotlib.backends.backend_pdf import PdfPages
import matplotlib.pyplot as plt
df = pd.DataFrame(list(Result.select().dicts()))
del(df['id'])
df.groupby(['data_distribution', 'no_instances', 'method']).mean()
[4]:
| score | elapsed_time | |||
|---|---|---|---|---|
| data_distribution | no_instances | method | ||
| 0 | 100.0 | lasso | 0.933052 | 0.004020 |
| ols | 0.951251 | 0.008777 | ||
| 1000.0 | lasso | 0.943172 | 0.012314 | |
| ols | 0.918918 | 0.006803 | ||
| 1 | 100.0 | lasso | 0.850391 | 0.006652 |
| ols | 0.830170 | 0.005800 | ||
| 1000.0 | lasso | 0.829416 | 0.011706 | |
| ols | 0.823004 | 0.008845 |
[5]:
def mpse(data):
mean = data.mean()
std_error = np.std(data) / np.sqrt(len(data))
return "{0:.3f} ({1:.3f})".format(mean, std_error)
df.groupby(['data_distribution', 'no_instances', 'method']).agg(mpse)
[5]:
| score | elapsed_time | |||
|---|---|---|---|---|
| data_distribution | no_instances | method | ||
| 0 | 100.0 | lasso | 0.933 (0.006) | 0.004 (0.001) |
| ols | 0.951 (0.005) | 0.009 (0.002) | ||
| 1000.0 | lasso | 0.943 (0.006) | 0.012 (0.002) | |
| ols | 0.919 (0.015) | 0.007 (0.002) | ||
| 1 | 100.0 | lasso | 0.850 (0.029) | 0.007 (0.001) |
| ols | 0.830 (0.036) | 0.006 (0.001) | ||
| 1000.0 | lasso | 0.829 (0.032) | 0.012 (0.002) | |
| ols | 0.823 (0.053) | 0.009 (0.002) |
You can also export your table to latex using to_latex pandas method:
[6]:
gdf = df.groupby(['data_distribution', 'no_instances', 'method']).agg(mpse)
print(gdf.to_latex())
# Hint: to enable enable multirow layout (see multirow Latex package)
# try print(gdf.to_latex(multirow=True))
\begin{tabular}{lllll}
\toprule
& & & score & elapsed\_time \\
data\_distribution & no\_instances & method & & \\
\midrule
0 & 100.0 & lasso & 0.933 (0.006) & 0.004 (0.001) \\
& & ols & 0.951 (0.005) & 0.009 (0.002) \\
& 1000.0 & lasso & 0.943 (0.006) & 0.012 (0.002) \\
& & ols & 0.919 (0.015) & 0.007 (0.002) \\
1 & 100.0 & lasso & 0.850 (0.029) & 0.007 (0.001) \\
& & ols & 0.830 (0.036) & 0.006 (0.001) \\
& 1000.0 & lasso & 0.829 (0.032) & 0.012 (0.002) \\
& & ols & 0.823 (0.053) & 0.009 (0.002) \\
\bottomrule
\end{tabular}
7.1. Aditional tools
Additional tools not explored here but available on sstudy package manual at https://sstudy.marcoinacio.com/:
Sample filter
Deleting or updating results
Postgresql database.
Remote SQLite access.
Storage of binary data.
Deterministic results.
Constraints on the dataset.
Real data.