_images/CreateModels.png

4. Testing

Model testing is a crucial part of the development process. It is important to ensure that the model is working as expected and that it is generalizing well to unseen data.

In order to be able to launch the testing of your model, you need 3 trained models, one for each task (energy, direction and type). Creating a TriModelManager will allow the user to manager the 3 tasks at once.

As usual, ceate a TriModelManager object with the three models you want to test.

MODEL_INDEX_FILE = "/path/to/your/ctlearn_models_index.h5"
energy_model = load_model_from_index("energy_model_nickname", MODEL_INDEX_FILE)
direction_model = load_model_from_index("direction_model_nickname", MODEL_INDEX_FILE)
type_model = load_model_from_index("type_model_nickname", MODEL_INDEX_FILE)
Tri_Model = CTLearnTriModelManager(direction_model=direction_model, energy_model=energy_model, type_model=type_model)

There are two cases if you want to test your model. Either the user doesn’t have the testing DL2 files yet and needs to produce them, or the files exist already.

Testing from scratch

The first step is to set the testing directories for gamma and protons, and their respective coordinates. This step is required only once as the DL1 files will be saved in the manager.

# Add electrons, gamma_diffuse and whatever you like to test your model on, each in a separate DataSample
testing_samples = [
    DataSample(directory='/path/to/your/MC/Gamma_point/20deg/merged/testing/',
            pattern="gamma_point*.h5"),
    DataSample(directory='/path/to/your/MC/Proton_diffuse/20deg/merged/testing/',
            pattern="proton*.h5"),
    ]

Tri_Model.set_testing_data(testing_samples)

Then, the user can launch the training for any of the coordinates of the testing files. One can access the available coordinates by calling get_available_testing_directions(). A few additional settings enable you to choose the output directories, what type of particle to launch, and the cluster configuration. The DL2 MC files will also be stored in the manager with their coordinates.

Note

Tri_Model.get_available_testing_directions() is a very usefule command from which you can directly copy and paste the direction you want to test on.

For DL2 data, the get_available_MC_directions() method is also available. It will return the coordinates of the DL2 files produced by the testing.

# See available testing directions and check the info of the cluster configuration
Tri_Model.plot_zenith_azimuth_ranges()
Tri_Model.get_available_testing_directions()
Tri_Model.cluster_configuration.info()

Launch the testing:

# copy the direction in zenith and azimuth you want to test on:
 Tri_Model.launch_testing(32.059 * u.deg, 248.099 * u.deg,
     output_dirs = ["/path/to/your/Testing_models/gamma_point/"],
     launch_particle_types=[ParticleType.GAMMA_POINT], # Add as much as you want, must provide a directory for each
     config_dir="/path/to/your/configs/", # Where the .sh scripts, configs and logs will be stored
     batch_size=64,
     # Other options are available
     )