18. Simulation studies using sstudy

One important aspect of proposing new machine learning/Statistical estimators and methods is the performance test phrase. With that in mind, we present here a short introduction on package `sstudy <https://sstudy.marcoinacio.com/>`__ that facilitates such procedure.

[1]:
!pip install sstudy
Requirement already satisfied: sstudy in /home/marco/Documents/projects/sstudy (0.0.5)
Requirement already satisfied: peewee in /home/marco/miniforge3/lib/python3.7/site-packages (from sstudy) (3.10.0)

Let us first define the structure of our dataset and create it:

[2]:
from peewee import *
import os

db = SqliteDatabase('results.sqlite3')

class Result(Model):
    # Data settings
    data_distribution = TextField()
    method = TextField()
    no_instances = DoubleField()

    # Results
    score = DoubleField()
    elapsed_time = DoubleField()

    class Meta:
        database = db

Result.create_table()

Now, let’s run the simulations:

[3]:
import numpy as np
import time
from scipy import stats
from sklearn.linear_model import LinearRegression, Lasso
from sstudy import do_simulation_study

no_simulations = 10

to_sample = dict(
    data_distribution = [0,1],
    no_instances = [100, 1000],
    method = ['ols', 'lasso'],
)

def func(
    data_distribution,
    no_instances,
    method,
    ):

    x = stats.norm.rvs(0, 2, size=(no_instances + 10000, 10))
    beta = stats.norm.rvs(0, 2, size=(10, 1))
    eps = stats.norm.rvs(0, 3, size=(no_instances + 10000, 1))
    if data_distribution == 0:
        y = np.matmul(x, beta) + eps
    if data_distribution == 1:
        y = np.matmul(x[:,:5], beta[:5]) + eps

    y_train = y[:no_instances]
    y_test = y[no_instances:]
    x_train = x[:no_instances]
    x_test = x[no_instances:]

    start_time = time.time()
    if method == 'ols':
        reg = LinearRegression()
    elif method == 'lasso':
        reg = Lasso(alpha=0.1)
    reg.fit(x_train, y_train)
    score = reg.score(x_test, y_test)
    elapsed_time = time.time() - start_time

    return dict(
        score = score,
        elapsed_time = elapsed_time,
    )

do_simulation_study(to_sample, func, db, Result, max_count=no_simulations)

8 combinations left
Result:
{'score': 0.6139337132523494, 'elapsed_time': 0.02861189842224121}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.8611576272271733, 'elapsed_time': 0.011518239974975586}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.926857889391819, 'elapsed_time': 0.0023622512817382812}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.8548821642738853, 'elapsed_time': 0.009645700454711914}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.8870249256740823, 'elapsed_time': 0.00249481201171875}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.8409022813797401, 'elapsed_time': 0.004044294357299805}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.8448752757733746, 'elapsed_time': 0.0037097930908203125}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.8897067829342962, 'elapsed_time': 0.004401683807373047}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9015128091852608, 'elapsed_time': 0.005560636520385742}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9173762118243225, 'elapsed_time': 0.003095388412475586}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.778717427545883, 'elapsed_time': 0.004297971725463867}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.34999607019544277, 'elapsed_time': 0.0112152099609375}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9339442945166084, 'elapsed_time': 0.004238128662109375}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9491677210715785, 'elapsed_time': 0.012585878372192383}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9550733471904924, 'elapsed_time': 0.009177207946777344}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.8697649290337729, 'elapsed_time': 0.007660865783691406}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9162499804715406, 'elapsed_time': 0.0050165653228759766}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9452732624279011, 'elapsed_time': 0.028743982315063477}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.950441811326444, 'elapsed_time': 0.006459951400756836}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.7668055526121543, 'elapsed_time': 0.016127347946166992}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.8557961743715387, 'elapsed_time': 0.0022199153900146484}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9363305322995981, 'elapsed_time': 0.002021312713623047}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9618996130598892, 'elapsed_time': 0.004746913909912109}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9332619267690534, 'elapsed_time': 0.012091398239135742}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9664944916259364, 'elapsed_time': 0.002245664596557617}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9204026973895163, 'elapsed_time': 0.01087045669555664}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.8751218277903129, 'elapsed_time': 0.0046579837799072266}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.972468591998792, 'elapsed_time': 0.013517379760742188}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.7573226196360956, 'elapsed_time': 0.002424955368041992}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.7424114258603562, 'elapsed_time': 0.016463279724121094}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9171756230714264, 'elapsed_time': 0.009216547012329102}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.8693280456380904, 'elapsed_time': 0.010242938995361328}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9198453720969424, 'elapsed_time': 0.0067365169525146484}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9151432918105386, 'elapsed_time': 0.005576133728027344}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.8823511788043035, 'elapsed_time': 0.003439664840698242}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9076904770665448, 'elapsed_time': 0.014777898788452148}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.7714421603477339, 'elapsed_time': 0.0018000602722167969}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9275495597560977, 'elapsed_time': 0.006983757019042969}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.7656507317306436, 'elapsed_time': 0.00559687614440918}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9478856878095755, 'elapsed_time': 0.007489919662475586}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9277319659081824, 'elapsed_time': 0.005853891372680664}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.9553898999855582, 'elapsed_time': 0.011910438537597656}
Result successfully stored in the database
8 combinations left
Result:
{'score': 0.8025482879051544, 'elapsed_time': 0.003132343292236328}
Result successfully stored in the database
8 combinations left
7 combinations left
Result:
{'score': 0.8796935556780558, 'elapsed_time': 0.0018222332000732422}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.9490630333532628, 'elapsed_time': 0.022101402282714844}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.9690228203456708, 'elapsed_time': 0.003087759017944336}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.95316118627402, 'elapsed_time': 0.006292104721069336}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.9175738906058983, 'elapsed_time': 0.0032949447631835938}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.9604022502319571, 'elapsed_time': 0.00570225715637207}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.9501598049429956, 'elapsed_time': 0.002569437026977539}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.7774176826009725, 'elapsed_time': 0.008621692657470703}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.9133612124782576, 'elapsed_time': 0.0167999267578125}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.653043736331703, 'elapsed_time': 0.014389753341674805}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.9327320755408688, 'elapsed_time': 0.018154621124267578}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.9016522030811711, 'elapsed_time': 0.006725788116455078}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.9496257758259826, 'elapsed_time': 0.009273290634155273}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.6183744207668223, 'elapsed_time': 0.007914066314697266}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.9663373421756023, 'elapsed_time': 0.008886098861694336}
Result successfully stored in the database
7 combinations left
Result:
{'score': 0.593423137125811, 'elapsed_time': 0.004263877868652344}
Result successfully stored in the database
7 combinations left
6 combinations left
Result:
{'score': 0.9158234579741112, 'elapsed_time': 0.015735864639282227}
Result successfully stored in the database
6 combinations left
Result:
{'score': 0.9229704225126353, 'elapsed_time': 0.009942770004272461}
Result successfully stored in the database
6 combinations left
Result:
{'score': 0.9131556194519059, 'elapsed_time': 0.0054168701171875}
Result successfully stored in the database
6 combinations left
Result:
{'score': 0.9095443217072087, 'elapsed_time': 0.004354000091552734}
Result successfully stored in the database
6 combinations left
Result:
{'score': 0.9559537340091908, 'elapsed_time': 0.002058744430541992}
Result successfully stored in the database
6 combinations left
Result:
{'score': 0.9455294877839799, 'elapsed_time': 0.012708187103271484}
Result successfully stored in the database
6 combinations left
Result:
{'score': 0.9145785519039111, 'elapsed_time': 0.002496957778930664}
Result successfully stored in the database
6 combinations left
Result:
{'score': 0.9219038507533641, 'elapsed_time': 0.0185697078704834}
Result successfully stored in the database
6 combinations left
Result:
{'score': 0.9620256020410831, 'elapsed_time': 0.0018486976623535156}
Result successfully stored in the database
6 combinations left
Result:
{'score': 0.9019879200435615, 'elapsed_time': 0.011387109756469727}
Result successfully stored in the database
6 combinations left
Result:
{'score': 0.9386660910357864, 'elapsed_time': 0.002043008804321289}
Result successfully stored in the database
6 combinations left
Result:
{'score': 0.9457011386599795, 'elapsed_time': 0.021100521087646484}
Result successfully stored in the database
6 combinations left
Result:
{'score': 0.9213863243859013, 'elapsed_time': 0.004567861557006836}
Result successfully stored in the database
6 combinations left
5 combinations left
4 combinations left
Result:
{'score': 0.9634731741331573, 'elapsed_time': 0.008003950119018555}
Result successfully stored in the database
4 combinations left
Result:
{'score': 0.9621886490300308, 'elapsed_time': 0.005016803741455078}
Result successfully stored in the database
4 combinations left
Result:
{'score': 0.9584995497741426, 'elapsed_time': 0.0041790008544921875}
Result successfully stored in the database
4 combinations left
Result:
{'score': 0.9402866422513141, 'elapsed_time': 0.001947164535522461}
Result successfully stored in the database
4 combinations left
Result:
{'score': 0.949407023370497, 'elapsed_time': 0.020636796951293945}
Result successfully stored in the database
4 combinations left
Result:
{'score': 0.9469646290312033, 'elapsed_time': 0.0047147274017333984}
Result successfully stored in the database
4 combinations left
Result:
{'score': 0.9076465139324088, 'elapsed_time': 0.004822969436645508}
Result successfully stored in the database
4 combinations left
3 combinations left
2 combinations left
1 combinations left
Result:
{'score': 0.8260911148699304, 'elapsed_time': 0.0027403831481933594}
Result successfully stored in the database
1 combinations left

The good news is that sqlite works though atomic transactions, so either a commit (i.e.: adding a result to the dataset) will happen entirelly or it won’t happen at all.

Therefore, you can kill the simulation study process without fear that the dataset will became corrupted in case you happen to kill while it’s commiting the results.

sstudy works chooses the test cases randomly and independently, therefore, you can spawn multiple simulation study processes that will work independly and can be terminated at any moment.

[4]:
import numpy as np
import pandas as pd
from matplotlib.backends.backend_pdf import PdfPages
import matplotlib.pyplot as plt

df = pd.DataFrame(list(Result.select().dicts()))
del(df['id'])
df.groupby(['data_distribution', 'no_instances', 'method']).mean()
[4]:
score elapsed_time
data_distribution no_instances method
0 100.0 lasso 0.933052 0.004020
ols 0.951251 0.008777
1000.0 lasso 0.943172 0.012314
ols 0.918918 0.006803
1 100.0 lasso 0.850391 0.006652
ols 0.830170 0.005800
1000.0 lasso 0.829416 0.011706
ols 0.823004 0.008845
[5]:
def mpse(data):
    mean = data.mean()
    std_error = np.std(data) / np.sqrt(len(data))
    return "{0:.3f} ({1:.3f})".format(mean, std_error)

df.groupby(['data_distribution', 'no_instances', 'method']).agg(mpse)
[5]:
score elapsed_time
data_distribution no_instances method
0 100.0 lasso 0.933 (0.006) 0.004 (0.001)
ols 0.951 (0.005) 0.009 (0.002)
1000.0 lasso 0.943 (0.006) 0.012 (0.002)
ols 0.919 (0.015) 0.007 (0.002)
1 100.0 lasso 0.850 (0.029) 0.007 (0.001)
ols 0.830 (0.036) 0.006 (0.001)
1000.0 lasso 0.829 (0.032) 0.012 (0.002)
ols 0.823 (0.053) 0.009 (0.002)

You can also export your table to latex using to_latex pandas method:

[6]:
gdf = df.groupby(['data_distribution', 'no_instances', 'method']).agg(mpse)
print(gdf.to_latex())
# Hint: to enable enable multirow layout (see multirow Latex package)
# try print(gdf.to_latex(multirow=True))
\begin{tabular}{lllll}
\toprule
  &        &     &          score &   elapsed\_time \\
data\_distribution & no\_instances & method &                &                \\
\midrule
0 & 100.0  & lasso &  0.933 (0.006) &  0.004 (0.001) \\
  &        & ols &  0.951 (0.005) &  0.009 (0.002) \\
  & 1000.0 & lasso &  0.943 (0.006) &  0.012 (0.002) \\
  &        & ols &  0.919 (0.015) &  0.007 (0.002) \\
1 & 100.0  & lasso &  0.850 (0.029) &  0.007 (0.001) \\
  &        & ols &  0.830 (0.036) &  0.006 (0.001) \\
  & 1000.0 & lasso &  0.829 (0.032) &  0.012 (0.002) \\
  &        & ols &  0.823 (0.053) &  0.009 (0.002) \\
\bottomrule
\end{tabular}

18.1. Aditional tools

Additional tools not explored here but available on sstudy package manual at https://sstudy.marcoinacio.com/:

  • Sample filter

  • Deleting or updating results

  • Postgresql database.

  • Remote SQLite access.

  • Storage of binary data.

  • Deterministic results.

  • Constraints on the dataset.

  • Real data.