{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating the YAML Configuration File\n",
    "\n",
    "`solaris` uses a YAML-formatted config file to specify all of the parameters required for data pre-processing, model training, inference, and more. Here, we'll go through every single element of that file and what it means, as well as how to create your own or modify an existing file for your use.\n",
    "\n",
    "## Helpful Resources\n",
    "\n",
    "- [Link to the YAML configuration skeleton](https://github.com/CosmiQ/solaris/blob/dev/solaris/nets/configs/config_skeleton.yml)\n",
    "- [Link to a sample YAML config file for XD_XD's SpaceNet 4 solution model](https://github.com/CosmiQ/solaris/blob/dev/solaris/nets/configs/xdxd_spacenet4.yml)\n",
    "\n",
    "## The elements of the Config file\n",
    "\n",
    "#### Top-level arguments\n",
    "\n",
    "- __model\\_name:__ \\[str\\] The name of the model being used. This will be cross-referenced against a list of possible options provided by `solaris`, and if it's not in that list, the user will be expected to provide the model. _Note_: currently, using user-provided models requires use of the Python API.\n",
    "- __model\\_path__: \\[str\\] Leave this blank unless you're using a custom model not native to solaris. solaris will automatically find your model.\n",
    "- __train__: \\[bool\\] Should `solaris` execute model training?\n",
    "- __infer__: \\[bool\\] Should `solaris` execute model inference?\n",
    "- __pretrained__: \\[bool\\] Do you wish to use pretrained weights with the model? This must be `true` if `train` is `false`.\n",
    "- __nn\\_framework__: \\[str\\] Which neural network framework are you using? This should either be `\"torch\"` or `\"keras\"` (more to be added later!)\n",
    "- __batch\\_size__: \\[int\\] What's the batch size for model training/inference?\n",
    "\n",
    "#### Data specs\n",
    "- __width:__ \\[int\\] The pixel width of the model inputs.\n",
    "- __height:__ \\[int\\] The pixel height of the model inputs.\n",
    "- __image_type:__ \\[str\\] One of `\"normalized\"` (0-1 range), `\"zscore\"`, `\"8bit\"`, `\"16bit\"`. The data type that the model ingests.\n",
    "- __rescale:__ \\[bool\\] Should image values be rescaled prior to post-processing?\n",
    "- __rescale\\_minima:__ \\[str or list\\] Either `\"auto\"` (in which case Solaris automatically determines this) or a value or list of values to set the minimum to.\n",
    "- __rescale\\_maxima:__ \\[str or list\\] Either `\"auto\"` (in which case Solaris automatically determines this) or a value or list of values to set the maximum to.\n",
    "- __channels:__ \\[int\\] The number of channels in the input.\n",
    "- __label\\_type:__ \\[str\\] currently the only possible value to this argument is `\"mask\"`.\n",
    "- __is\\_categorical:__ \\[bool\\] Currently this argument has no effect. When classification/object detection is added, it will be implemented.\n",
    "- __mask\\_channels:__ \\[int\\] The number of channels in the training mask to be used as a training target.\n",
    "- __val\\_holdout\\_frac:__ \\[float\\] The fraction of the training data to hold out for validation. Note that this argument has no effect if __validation\\_data\\_csv__ (below) is specified. Otherwise, the a random subset of the samples in the training CSV will be held back for end-of-epoch validation.\n",
    "- __data\\_workers:__ \\[int\\] This argument is currently unused.\n",
    "\n",
    "#### Data reference files\n",
    "\n",
    "See [Creating reference files to help solaris find your imagery](../cli_im_ref) for details on what these files must include.\n",
    "\n",
    "- __training\\_data\\_csv:__ \\[str\\] The path to the training data CSV. See the link above for more details.\n",
    "- __validation\\_data\\_csv:__ \\[str\\] The path to the validation data CSV. See the link above for more details. If you are splitting your training data for validation using __val\\_holdout\\_frac__, ignore this argument.\n",
    "- __inference\\_data\\_csv:__ \\[str\\] The path to the inference data CSV. See the link above for more details.\n",
    "\n",
    "#### Training augmentation\n",
    "\n",
    "Augmentation is critical in training many models, particularly for geospatial data. If you perform data normalization during your augmentation pipeline, you can also specify that here. See [XD_XD's SpaceNet 4 augmentation pipeline](https://github.com/CosmiQ/solaris/blob/dev/solaris/nets/configs/xdxd_spacenet4.yml) for an example of a pipeline.\n",
    "\n",
    "- __augmentations:__ \\[dict\\] The augmentations to run. The majority of augmentations implemented in [albumentations](https://albumentations.readthedocs.io/) are available here, either using that implementation or a custom version to enable >3-channel imagery ingestion. Pass the name of the augmentation as keys in this dictionary, and `kwarg: value` pairs as sub-dicts. See the sample linked above if this is unclear.\n",
    "- __p:__ \\[float\\] The probability that the augmentation pipeline will be applied to images in a batch.\n",
    "- __shuffle:__ \\[bool\\] Should the order of training images be shuffled as they're fed into the model? Defaults to `true`.\n",
    "\n",
    "#### Validation augmentation\n",
    "\n",
    "The same arguments are valid here as for `training_augmentation`.\n",
    "\n",
    "#### Inference augmentation\n",
    "\n",
    "The same arguments are valid here as for `training_augmentation`.\n",
    "\n",
    "#### Training\n",
    "\n",
    "This set of parameters define the actual training process.\n",
    "\n",
    "- __epochs:__ \\[int\\] The number of epochs to train for.\n",
    "- __steps\\_per\\_epoch:__ \\[int\\] The number of batches to train for in each epoch. This is determined automatically if not provided.\n",
    "- __optimizer:__ \\[str\\] The name of the optimizer to use for training. Options are `\"Adam\"`, `\"SGD\"`, `\"adadelta\"`, `\"RMSProp\"`, `\"Adamax\"`, `\"Nadam\"` (Keras only), `\"Adagrad\"` (Keras only), `\"SparseAdam\"` (Torch only), or `\"ASGD\"` (Torch only). Pass arguments for these optimizers to __opt\\_args__ (see below).\n",
    "- __lr:__ \\[float\\] The learning rate to use (at least at the start of the training process).\n",
    "- __opt\\_args:__ \\[dict\\] A dictionary of `kwarg: value` pairs to pass to the optimizer.\n",
    "- __loss:__ \\[dict\\] A dictionary of loss function name(s). This allows you to create composite loss functions with ease. If there are any arguments that must be passed to the loss function upon initialization (e.g. the gamma parameter for focal loss), pass them as subdicts here.\n",
    "- __loss\\_weights:__ \\[dict\\] A dictionary of `loss_name: weight` pairs. If provided, the same names must be passed here as were passed in __loss__. If not provided, the different losses will be weighted equally. Weight values can be ints or floats.\n",
    "- __metrics:__ \\[dict\\] A dict of `training: [list of training metrics], validation: [list of validation metrics]`. See the linked example for what this can look like. Note that this only currently has an effect for Keras models.\n",
    "- __checkpoint\\_frequency:__ \\[int\\] The frequency at which model checkpoints should be saved.\n",
    "- __callbacks:__ \\[dict\\] A dict of callback names, whose values are subdicts defining any arguments for the callback. See [callbacks](../api/nets.rst#module-solaris.nets.callbacks) for options.\n",
    "- __model\\_dest\\_path:__ \\[str\\] The path to save the final, trained model to.\n",
    "- __verbose:__ \\[bool\\] Verbose text output during training.\n",
    "\n",
    "#### Inference\n",
    "\n",
    "- __window\\_step\\_size\\_x:__ \\[int\\] If your model takes in an image smaller than your inference chips, this argument will specify how far in the x direction each tile should step. Set to the same value as __width__ if you don't want any overlap between tiles; if you have overlap, `solaris` will average predictions across the overlapping areas to give you more robust estimates.\n",
    "- __window\\_step\\_size\\_y:__ \\[int\\] If your model takes in an image smaller than your inference chips, this argument will specify how far in the y direction each tile should step. Set to the same value as __height__ if you don't want any overlap between tiles; if you have overlap, `solaris` will average predictions across the overlapping areas to give you more robust estimates.\n",
    "\n",
    "\n",
    "## How the config file is processed\n",
    "\n",
    "`solaris` contains a utility function, [solaris.utils.config.parse()](../../api/utils.rst#solaris.utils.config.parse) which takes one argument: the path to a YAML config file to read in. [parse()](../../api/utils.rst#solaris.utils.config.parse) will check to make sure necessary values are present. This is called automatically by CLI functions that take the config file as an argument, but you can also call it with the Python API to use the config as an argument in [solaris.nets](../../api/nets.rst) functions.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "solaris",
   "language": "python",
   "name": "solaris"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}