Filter Evaluation Metrics #85
Description
We need to know how well our filters are doing.
- Determine what proportion of data coming from the generic parser typically needs to be rejected.
- Determine the accuracy and false positive/negative rates of the filter pipeline as a whole.
- Determine the accuracy and false positive/negative rates of the local filters as a whole
- Determine the accuracy and false positive/negative rates of the openAI filter. These should be computed in two settings, one with a thusfar unfiltered dataset, and one only using data the local filters passed through.
- Any other metrics you think are useful
Save and track with git any tools developed to accomplish this, as we may want to use them again for strategy-specific evaluations.