Creating the YAML Configuration File

solaris uses a YAML-formatted config file to specify all of the parameters required for data pre-processing, model training, inference, and more. Here, we’ll go through every single element of that file and what it means, as well as how to create your own or modify an existing file for your use.

The elements of the Config file

Top-level arguments

  • model_name: [str] The name of the model being used. This will be cross-referenced against a list of possible options provided by solaris, and if it’s not in that list, the user will be expected to provide the model. Note: currently, using user-provided models requires use of the Python API.

  • model_path: [str] Leave this blank unless you’re using a custom model not native to solaris. solaris will automatically find your model.

  • train: [bool] Should solaris execute model training?

  • infer: [bool] Should solaris execute model inference?

  • pretrained: [bool] Do you wish to use pretrained weights with the model? This must be true if train is false.

  • nn_framework: [str] Which neural network framework are you using? This should either be "torch" or "keras" (more to be added later!)

  • batch_size: [int] What’s the batch size for model training/inference?

Data specs

  • width: [int] The pixel width of the model inputs.

  • height: [int] The pixel height of the model inputs.

  • image_type: [str] One of "normalized" (0-1 range), "zscore", "8bit", "16bit". The data type that the model ingests.

  • rescale: [bool] Should image values be rescaled prior to post-processing?

  • rescale_minima: [str or list] Either "auto" (in which case Solaris automatically determines this) or a value or list of values to set the minimum to.

  • rescale_maxima: [str or list] Either "auto" (in which case Solaris automatically determines this) or a value or list of values to set the maximum to.

  • channels: [int] The number of channels in the input.

  • label_type: [str] currently the only possible value to this argument is "mask".

  • is_categorical: [bool] Currently this argument has no effect. When classification/object detection is added, it will be implemented.

  • mask_channels: [int] The number of channels in the training mask to be used as a training target.

  • val_holdout_frac: [float] The fraction of the training data to hold out for validation. Note that this argument has no effect if validation_data_csv (below) is specified. Otherwise, the a random subset of the samples in the training CSV will be held back for end-of-epoch validation.

  • data_workers: [int] This argument is currently unused.

Data reference files

See Creating reference files to help solaris find your imagery for details on what these files must include.

  • training_data_csv: [str] The path to the training data CSV. See the link above for more details.

  • validation_data_csv: [str] The path to the validation data CSV. See the link above for more details. If you are splitting your training data for validation using val_holdout_frac, ignore this argument.

  • inference_data_csv: [str] The path to the inference data CSV. See the link above for more details.

Training augmentation

Augmentation is critical in training many models, particularly for geospatial data. If you perform data normalization during your augmentation pipeline, you can also specify that here. See XD_XD’s SpaceNet 4 augmentation pipeline for an example of a pipeline.

  • augmentations: [dict] The augmentations to run. The majority of augmentations implemented in albumentations are available here, either using that implementation or a custom version to enable >3-channel imagery ingestion. Pass the name of the augmentation as keys in this dictionary, and kwarg: value pairs as sub-dicts. See the sample linked above if this is unclear.

  • p: [float] The probability that the augmentation pipeline will be applied to images in a batch.

  • shuffle: [bool] Should the order of training images be shuffled as they’re fed into the model? Defaults to true.

Validation augmentation

The same arguments are valid here as for training_augmentation.

Inference augmentation

The same arguments are valid here as for training_augmentation.

Training

This set of parameters define the actual training process.

  • epochs: [int] The number of epochs to train for.

  • steps_per_epoch: [int] The number of batches to train for in each epoch. This is determined automatically if not provided.

  • optimizer: [str] The name of the optimizer to use for training. Options are "Adam", "SGD", "adadelta", "RMSProp", "Adamax", "Nadam" (Keras only), "Adagrad" (Keras only), "SparseAdam" (Torch only), or "ASGD" (Torch only). Pass arguments for these optimizers to opt_args (see below).

  • lr: [float] The learning rate to use (at least at the start of the training process).

  • opt_args: [dict] A dictionary of kwarg: value pairs to pass to the optimizer.

  • loss: [dict] A dictionary of loss function name(s). This allows you to create composite loss functions with ease. If there are any arguments that must be passed to the loss function upon initialization (e.g. the gamma parameter for focal loss), pass them as subdicts here.

  • loss_weights: [dict] A dictionary of loss_name: weight pairs. If provided, the same names must be passed here as were passed in loss. If not provided, the different losses will be weighted equally. Weight values can be ints or floats.

  • metrics: [dict] A dict of training: [list of training metrics], validation: [list of validation metrics]. See the linked example for what this can look like. Note that this only currently has an effect for Keras models.

  • checkpoint_frequency: [int] The frequency at which model checkpoints should be saved.

  • callbacks: [dict] A dict of callback names, whose values are subdicts defining any arguments for the callback. See callbacks for options.

  • model_dest_path: [str] The path to save the final, trained model to.

  • verbose: [bool] Verbose text output during training.

Inference

  • window_step_size_x: [int] If your model takes in an image smaller than your inference chips, this argument will specify how far in the x direction each tile should step. Set to the same value as width if you don’t want any overlap between tiles; if you have overlap, solaris will average predictions across the overlapping areas to give you more robust estimates.

  • window_step_size_y: [int] If your model takes in an image smaller than your inference chips, this argument will specify how far in the y direction each tile should step. Set to the same value as height if you don’t want any overlap between tiles; if you have overlap, solaris will average predictions across the overlapping areas to give you more robust estimates.

How the config file is processed

solaris contains a utility function, solaris.utils.config.parse() which takes one argument: the path to a YAML config file to read in. parse() will check to make sure necessary values are present. This is called automatically by CLI functions that take the config file as an argument, but you can also call it with the Python API to use the config as an argument in solaris.nets functions.