{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Scoring model performance with the `solaris` python API\n", "\n", "This tutorial describes how to run evaluation of a proposal (CSV or .geojson) for a single chip against a ground truth (CSV or .geojson) for the same chip.\n", "\n", "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## CSV Eval\n", "\n", "### Steps\n", "1. Imports\n", "2. Load ground truth CSV\n", "3. Load proposal CSV\n", "4. Perform evaluation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "### Imports \n", "\n", "For this test case we will use the `eval` submodule within `solaris`." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "# imports\n", "import os\n", "import solaris as sol\n", "from solaris.data import data_dir\n", "import pandas as pd # just for visualizing the outputs\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "\n", "### Load ground truth CSV\n", "\n", "We will first instantiate an `Evaluator()` object, which is the core class `eval` uses for comparing predicted labels to ground truth labels. `Evaluator()` takes one argument - the path to the CSV or .geojson ground truth label object. It can alternatively accept a pre-loaded `GeoDataFrame` of ground truth label geometries." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Evaluator sample_truth.csv" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ground_truth_path = os.path.join(data_dir, 'sample_truth.csv')\n", "\n", "evaluator = sol.eval.base.Evaluator(ground_truth_path)\n", "evaluator" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "At this point, `evaluator` has the following attributes:\n", "\n", "- `ground_truth_fname`: the filename corresponding to the ground truth data. This is simply `'GeoDataFrame'` if a GDF was passed during instantiation.\n", "\n", "- `ground_truth_GDF`: GeoDataFrame-formatted geometries for the ground truth polygon labels.\n", "\n", "- `ground_truth_GDF_Edit`: A deep copy of `eval_object.ground_truth_GDF` which is edited during the process of matching ground truth label polygons to proposals.\n", "\n", "- `ground_truth_sindex`: The RTree/libspatialindex spatial index for rapid spatial referencing.\n", "\n", "- `proposal_GDF`: An _empty_ GeoDataFrame instantiated to hold proposals later.\n", "\n", "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "### Load proposal CSV\n", "\n", "Next we will load in the proposal CSV file. Note that the `proposalCSV` flag must be set to true for CSV data. If the CSV contains confidence column(s) that indicate confidence in proprosals, the name(s) of the column(s) should be passed as a list of strings with the `conf_field_list` argument; because no such column exists in this case, we will simply pass `conf_field_list=[]`. There are additional arguments available (see [the method documentation](https://cw-eval.readthedocs.io/en/latest/api.html#cw_eval.baseeval.EvalBase.load_proposal)) which can be used for multi-class problems; those will be covered in another recipe. The defaults suffice for single-class problems." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "proposals_path = os.path.join(data_dir, 'sample_preds.csv')\n", "evaluator.load_proposal(proposals_path, proposalCSV=True, conf_field_list=[])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "\n", "### Perform evaluation\n", "\n", "Evaluation iteratively steps through the proposal polygons in `eval_object.proposal_GDF` and determines if any of the polygons in `eval_object.ground_truth_GDF_Edit` have IoU overlap > `miniou` (see [the method documentation](https://cw-eval.readthedocs.io/en/latest/api.html#cw_eval.baseeval.EvalBase.eval_iou)) with that proposed polygon. If one does, that proposal polygon is scored as a true positive. The matched ground truth polygon with the highest IoU (in case multiple had IoU > `miniou`) is removed from `eval_object.ground_truth_GDF_Edit` so it cannot be matched against another proposal. If no ground truth polygon matches with IoU > `miniou`, that proposal polygon is scored as a false positive. After iterating through all proposal polygons, any remaining ground truth polygons in `eval_object.ground_truth_GDF_Edit` are scored as false negatives.\n", "\n", "There are several additional arguments to this method related to multi-class evaluation which will be covered in a later recipe. See [the method documentation](https://cw-eval.readthedocs.io/en/latest/api.html#cw_eval.baseeval.EvalBase.eval_iou) for usage.\n", "\n", "The prediction outputs a `list` of `dict`s for each class evaluated (only one `dict` in this single-class case). The `dict`(s) have the following keys:\n", "\n", "- `'class_id'`: The class being scored in the dict, `'all'` for single-class scoring.\n", "\n", "- `'iou_field'`: The name of the column in `eval_object.proposal_GDF` for the IoU score for this class. See [the method documentation](https://cw-eval.readthedocs.io/en/latest/api.html#cw_eval.baseeval.EvalBase.eval_iou) for more information.\n", "\n", "- `'TruePos'`: The number of polygons in `eval_object.proposal_GDF` that matched a polygon in `eval_object.ground_truth_GDF_Edit`.\n", "\n", "- `'FalsePos'`: The number of polygons in `eval_object.proposal_GDF` that had no match in `eval_object.ground_truth_GDF_Edit`.\n", "\n", "- `'FalseNeg'`: The number of polygons in `eval_object.ground_truth_GDF_Edit` that had no match in `eval_object.proposal_GDF`.\n", "\n", "- `'Precision'`: The [precision statistic](https://en.wikipedia.org/wiki/Precision_and_recall) for IoU between the proposals and the ground truth polygons.\n", "\n", "- `'Recall'`: The [recall statistic](https://en.wikipedia.org/wiki/Precision_and_recall) for IoU between the proposals and the ground truth polygons.\n", "\n", "- `'F1Score'`: Also known as the [SpaceNet Metric](https://medium.com/the-downlinq/the-spacenet-metric-612183cc2ddb), the [F1 score](https://en.wikipedia.org/wiki/F1_score) for IoU between the proposals and the ground truth polygons." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "151it [00:00, 153.44it/s]\n" ] }, { "data": { "text/plain": [ "[{'class_id': 'all',\n", " 'iou_field': 'iou_score_all',\n", " 'TruePos': 151,\n", " 'FalsePos': 0,\n", " 'FalseNeg': 0,\n", " 'Precision': 1.0,\n", " 'Recall': 1.0,\n", " 'F1Score': 1.0}]" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "evaluator.eval_iou(calculate_class_scores=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this case, the score is perfect because the polygons in the ground truth CSV and the proposal CSV are identical. At this point, a new proposal CSV can be loaded (for example, for a new nadir angle at the same chip location) and scoring can be repeated." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## GeoJSON Eval\n", "\n", "The same operation can be completed with .geojson-formatted ground truth and proposal files. See the example below, and see the detailed explanation above for a description of each step's operations." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "28it [00:00, 76.20it/s]\n" ] }, { "data": { "text/plain": [ "[{'class_id': 'all',\n", " 'iou_field': 'iou_score_all',\n", " 'TruePos': 8,\n", " 'FalsePos': 20,\n", " 'FalseNeg': 20,\n", " 'Precision': 0.2857142857142857,\n", " 'Recall': 0.2857142857142857,\n", " 'F1Score': 0.2857142857142857}]" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ground_truth_geojson = os.path.join(data_dir, 'gt.geojson')\n", "proposal_geojson = os.path.join(data_dir, 'pred.geojson')\n", "\n", "evaluator = sol.eval.base.Evaluator(ground_truth_geojson)\n", "evaluator.load_proposal(proposal_geojson, proposalCSV=False, conf_field_list=[])\n", "evaluator.eval_iou(calculate_class_scores=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "(Note that the above comes from a different chip location and different proposal than the CSV example, hence the difference in scores)" ] } ], "metadata": { "kernelspec": { "display_name": "solaris", "language": "python", "name": "solaris" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.7" } }, "nbformat": 4, "nbformat_minor": 4 }