diff --git a/config_library/pattern-2/fcc-invoices/README.md b/config_library/pattern-2/fcc-invoices/README.md new file mode 100644 index 000000000..d597790fb --- /dev/null +++ b/config_library/pattern-2/fcc-invoices/README.md @@ -0,0 +1,467 @@ +# FCC Invoice Processing - End-to-End Example + +This directory contains a complete end-to-end example for processing FCC (Federal Communications Commission) political advertising invoices using the IDP accelerator with Stickler-based evaluation. + +## Overview + +This example demonstrates: +1. **Deployment** - Deploy the IDP stack with FCC invoice configuration +2. **Inference** - Run inference on sample FCC invoices +3. **Evaluation** - Evaluate extraction results using Stickler +4. **Review** - Analyze individual and aggregated metrics + +## Directory Contents + +``` +config_library/pattern-2/fcc-invoices/ +├── README.md # This file +├── config.yaml # Base IDP configuration +├── fcc_configured.yaml # Deployed stack configuration +├── stickler_config.json # Stickler evaluation rules +├── bulk_evaluate_fcc_invoices.py # Legacy evaluation script (complex) +├── bulk_evaluate_fcc_invoices_simple.py # Simplified evaluation script (recommended) +├── sample_labels_3.csv # Ground truth for 3 sample documents +└── sr_refactor_labels_5_5_25.csv # Ground truth labels (full dataset) +``` + +## Sample Data + +Sample documents are located in `samples/fcc-invoices/`: +- 3 sample PDF invoices +- `fcc_invoices_sample_3.csv` - Manifest for the 3 samples + +## Prerequisites + +1. **AWS Credentials**: Valid AWS credentials with appropriate permissions +2. **Python Environment**: Python 3.12+ with required packages +3. **IDP CLI**: Installed and configured +4. **Stickler**: Installed with `pip install stickler-eval` +5. **Dependencies**: `pip install pandas` + +## Step 1: Deploy the Stack + +Deploy the IDP stack with FCC invoice configuration: + +```bash +idp-cli deploy \ + --stack-name fcc-inf-test \ + --custom-config config_library/pattern-2/fcc-invoices/config.yaml \ + --region us-west-2 \ + --wait \ + --template-url https://s3.us-west-2.amazonaws.com/bobs-artifacts-us-west-2/idp-wip/idp-main.yaml \ + --admin-email your-email@example.com \ + --pattern pattern-2 +``` + +**What this does:** +- Creates CloudFormation stack with Lambda functions, S3 buckets, and DynamoDB tables +- Configures extraction model (Claude Sonnet 4) +- Sets up OCR with Textract (LAYOUT + TABLES features) +- Deploys with FCC-specific prompts and schema + +**Expected output:** +- Stack creation takes ~5-10 minutes +- Stack status: `CREATE_COMPLETE` + +## Step 2: Run Inference + +Run inference on the sample documents: + +```bash +idp-cli run-inference \ + --stack-name fcc-inf-test \ + --manifest samples/fcc-invoices/fcc_invoices_sample_3.csv \ + --region us-west-2 +``` + +**What this does:** +- Uploads documents to S3 input bucket +- Triggers Lambda processing pipeline +- Performs OCR with Textract +- Extracts structured data using Claude +- Stores results in S3 output bucket + +**Expected output:** +``` +Validating manifest... +✓ Manifest validated successfully +Initializing batch processor for stack: fcc-inf-test +✓ Batch submitted successfully +Batch ID: batch-20251017-140000 +Processing 3 documents... +``` + +**Monitor progress:** +```bash +idp-cli status \ + --stack-name fcc-inf-test \ + --batch-id \ + --region us-west-2 \ + --wait +``` + +## Step 3: Download Results + +Download the inference results locally: + +```bash +idp-cli download-results \ + --stack-name fcc-inf-test \ + --batch-id cli-batch-20251017-190516 \ + --output-dir fcc_results \ + --region us-west-2 +``` + +**Note**: Replace `cli-batch-20251017-190516` with your actual batch ID from the inference step. You can specify any output directory name. + +**What this does:** +- Downloads all result files from S3 +- Creates directory structure: `fcc_results//sections/1/result.json` +- Each result contains extracted fields and metadata + +**Result structure:** +```json +{ + "document_class": { + "type": "FCC-Invoice" + }, + "inference_result": { + "agency": "Agency Name", + "advertiser": "Advertiser Name", + "gross_total": "1,234.56", + "net_amount_due": "1,234.56", + "line_item__description": ["M-F 11a-12p", "M-F 12n-1p"], + "line_item__days": ["MTWTF--", "MTWTF--"], + "line_item__rate": ["100.00", "150.00"], + "line_item__start_date": ["11/01/21", "11/01/21"], + "line_item__end_date": ["11/07/21", "11/07/21"] + } +} +``` + +## Step 4: Run Evaluation + +### Option A: Single Source of Truth (Recommended) + +Use the IDP config directly - no separate Stickler config needed: + +```bash +cd config_library/pattern-2/fcc-invoices + +python bulk_evaluate_from_idp_config.py \ + --results-dir ../../../fcc_results/cli-batch-20251017-190516 \ + --csv-path sample_labels_3.csv \ + --idp-config-path sr_FCC_config.json \ + --output-dir evaluation_output +``` + +**Benefits:** +- Single source of truth - evaluation config comes from IDP config +- Extracts Stickler settings from `x-aws-stickler-*` extensions in JSON Schema +- No need to maintain separate `stickler_config.json` +- Guarantees evaluation matches deployment configuration + +### Option B: Separate Stickler Config + +Use the simplified script with standalone Stickler config: + +```bash +python bulk_evaluate_fcc_invoices_simple.py \ + --results-dir ../../../fcc_results/cli-batch-20251017-190516 \ + --csv-path sample_labels_3.csv \ + --config-path stickler_config.json \ + --output-dir evaluation_output +``` + +**Benefits:** +- 260 lines vs 671 lines (61% less code) +- Easier to understand and modify +- No temporary file overhead +- Direct integration with SticklerEvaluationService + +### Option C: Legacy Script + +Use the original complex script (not recommended): + +```bash +python bulk_evaluate_fcc_invoices.py \ + --results-dir ../../../fcc_results/cli-batch-20251017-190516 \ + --csv-path sample_labels_3.csv \ + --config-path stickler_config.json \ + --output-dir evaluation_output +``` + +**Note**: The `sample_labels_3.csv` contains ground truth for 3 sample documents. For full dataset evaluation, use `sr_refactor_labels_5_5_25.csv`. + +**What evaluation does:** +- Loads ground truth labels from CSV +- Matches documents by doc_id +- Performs doc-by-doc comparison using SticklerEvaluationService +- Saves individual comparison results +- Aggregates metrics across all documents +- Generates comprehensive evaluation report + +**Expected output:** +``` +================================================================================ +BULK FCC INVOICE EVALUATION +================================================================================ + +📊 Loading ground truth from sr_refactor_labels_5_5_25.csv... +✓ Loaded 221 documents with ground truth labels + +📁 Loading inference results from ../../../fcc_results... +✓ Loaded 3 inference results + +🔗 Matching ground truth to inference results... +✓ Matched 3 document pairs + +⚙️ Evaluating 3 documents... +✓ Completed evaluation + Individual results saved to: evaluation_output + +================================================================================ +AGGREGATED EVALUATION RESULTS +================================================================================ + +📊 Processing Summary: + Documents processed: 3 + Errors encountered: 0 + Non-matches found: 23 + +📈 Overall Metrics: + Precision: 0.7341 + Recall: 0.4637 + F1 Score: 0.5684 + Accuracy: 0.3993 + + Confusion Matrix: + TP: 530 | FP: 192 + FN: 613 | TN: 5 + FP1 (False Alarm): 11 + FP2 (Wrong Value): 181 + +📋 Field-Level Metrics (Top 10 by F1 Score): + Field Precision Recall F1 + ---------------------------------------- ---------- ---------- ---------- + line_item__description 0.9236 0.8261 0.8721 + gross_total 1.0000 0.7500 0.8571 + net_amount_due 1.0000 0.7500 0.8571 + line_item__rate 0.8169 0.7117 0.7607 + ... + +💾 Aggregated results saved to evaluation_output/aggregated_metrics.json + +================================================================================ +✅ Evaluation complete! + Individual results: evaluation_output + Aggregated metrics: evaluation_output/aggregated_metrics.json +================================================================================ +``` + +## Step 5: Review Results + +### Individual Document Results + +Each document has a detailed comparison result: + +```bash +cat evaluation_output/0492b95bc342870920c480040bc33513.json | python -m json.tool | less +``` + +**Contains:** +- Field-by-field scores +- Confusion matrix (overall and per-field) +- Non-matches with details +- Similarity scores + +### Aggregated Metrics + +View the overall performance: + +```bash +cat evaluation_output/aggregated_metrics.json | python -m json.tool | less +``` + +**Contains:** +- Overall precision, recall, F1, accuracy +- Per-field performance metrics +- Confusion matrix breakdown +- Non-match summary + +## Understanding the Results + +### Confusion Matrix Metrics + +- **TP (True Positive)**: Correctly extracted field with correct value +- **FP (False Positive)**: Extracted field with incorrect value or shouldn't exist +- **TN (True Negative)**: Correctly didn't extract a field that shouldn't exist +- **FN (False Negative)**: Failed to extract a field that should exist +- **FP1 (False Alarm)**: Extracted a field that shouldn't exist +- **FP2 (Wrong Value)**: Extracted a field with wrong value + +### Derived Metrics + +- **Precision**: TP / (TP + FP) - How many extracted values are correct +- **Recall**: TP / (TP + FN) - How many ground truth values were found +- **F1 Score**: Harmonic mean of precision and recall +- **Accuracy**: (TP + TN) / Total - Overall correctness + +## Stickler Configuration + +The `stickler_config.json` defines validation rules: + +### Simple Fields (Lists) +- `agency`: FuzzyComparator (threshold 0.8) - Allows minor name variations +- `advertiser`: FuzzyComparator (threshold 0.8) +- `gross_total`: ExactComparator (threshold 1.0) - Requires exact match +- `net_amount_due`: ExactComparator (threshold 1.0) + +### Line Item Fields (Lists) +- `line_item__description`: LevenshteinComparator (threshold 0.7) - Allows typos +- `line_item__days`: ExactComparator (threshold 1.0) +- `line_item__rate`: ExactComparator (threshold 1.0) +- `line_item__start_date`: ExactComparator (threshold 1.0) +- `line_item__end_date`: ExactComparator (threshold 1.0) + +**Note**: All fields are configured as lists to match the flat format used by both ground truth and predictions. + +## Data Format + +### Ground Truth (CSV) +The `sr_refactor_labels_5_5_25.csv` contains: +- `doc_id`: Document identifier (without .pdf extension) +- `refactored_labels`: JSON string with ground truth in flat list format + +### Inference Results +Directory structure: `results_dir/{doc_id}.pdf/sections/1/result.json` + +The flat format uses `line_item__` prefix for list fields, where each field is a list of values. + +## Troubleshooting + +### No matched pairs found +- Verify `doc_id` in CSV matches directory names in results +- Check if doc_id has `.pdf` extension mismatch + +### AWS Token Expired +```bash +# Refresh your AWS credentials +aws sso login --profile your-profile +``` + +### Stack not found +```bash +# Verify stack exists +idp-cli list-stacks --region us-west-2 +``` + +### Large matrix warnings +- Normal for documents with many line items (>100) +- Stickler uses Hungarian algorithm for optimal matching +- May be slower but produces accurate results + +## Next Steps + +1. **Scale Up**: Process more documents by creating a larger manifest +2. **Tune Configuration**: Adjust Stickler thresholds based on results +3. **Analyze Errors**: Review non-matches to identify extraction issues +4. **Iterate**: Update prompts or schema based on evaluation findings + +## Update Log + +### October 31, 2024 - Flat Schema Testing + +Testing with updated `sr_FCC_config.json` that uses a flat array structure for all fields. + +**Configuration Changes:** +- Simple fields (agency, advertiser, gross_total, net_amount_due) as single-element arrays +- Line item fields with `line_item__` prefix as parallel arrays +- Stickler evaluation extensions added to JSON Schema +- Assessment and summarization steps disabled for faster processing + +**Test Run Commands:** + +1. Deploy with updated configuration: +```bash +idp-cli deploy \ + --stack-name fcc-inf-test-2 \ + --custom-config config_library/pattern-2/fcc-invoices/sr_FCC_config.json \ + --region us-west-2 \ + --wait \ + --template-url https://s3.us-west-2.amazonaws.com/bobs-artifacts-us-west-2/idp-wip/idp-main.yaml \ + --admin-email sromo@amazon.com \ + --pattern pattern-2 +``` + +2. Run inference: +```bash +idp-cli run-inference \ + --stack-name fcc-inf-test-2 \ + --manifest samples/fcc-invoices/fcc_invoices_sample_3.csv \ + --region us-west-2 +``` + +3. Monitor status: +```bash +idp-cli status \ + --stack-name fcc-inf-test-2 \ + --batch-id cli-batch-20251031-164416 \ + --wait +``` + +4. Download results: +```bash +idp-cli download-results \ + --stack-name fcc-inf-test-2 \ + --batch-id cli-batch-20251031-164416 \ + --output-dir fcc_results-updated-2 \ + --region us-west-2 +``` + +5. Run evaluation with IDP config: +```bash +cd config_library/pattern-2/fcc-invoices + +python bulk_evaluate_from_idp_config.py \ + --results-dir ../../../fcc_results-updated-2/cli-batch-20251031-164416 \ + --csv-path sample_labels_3.csv \ + --idp-config-path sr_FCC_config.json \ + --output-dir evaluation_output-idp-config +``` + +**Evaluation Results:** +``` +📈 Overall Metrics: + Precision: 0.5185 + Recall: 1.0000 + F1 Score: 0.6829 + Accuracy: 0.5185 + + Confusion Matrix: + TP: 14 | FP: 13 + FN: 0 | TN: 0 + FP1: 2 | FP2: 11 + +📋 Field-Level Metrics (Top Fields): + agency F1: 0.8000 + gross_total F1: 0.8000 + net_amount_due F1: 0.8000 + line_item__days F1: 0.8000 + line_item__start_date F1: 0.8000 + line_item__end_date F1: 0.8000 +``` + +**Key Learnings:** +- Multiple deploy/inference cycles were run to iterate on the configuration +- Final batch ID: `cli-batch-20251031-164416` +- Configuration now properly uses `{ATTRIBUTE_NAMES_AND_DESCRIPTIONS}` placeholder for automatic schema injection +- New `bulk_evaluate_from_idp_config.py` extracts Stickler config from `x-aws-stickler-*` extensions +- Single source of truth: IDP config contains both extraction schema and evaluation settings + +## Additional Resources + +- [IDP CLI Documentation](../../README.md) +- [Stickler Documentation](../../../stickler/README.md) +- [Pattern 2 Architecture](../README.md) +- [Evaluation Guide](../../../lib/idp_common_pkg/idp_common/evaluation/README_STICKLER.md) \ No newline at end of file diff --git a/config_library/pattern-2/fcc-invoices/bulk_evaluate_fcc_invoices.py b/config_library/pattern-2/fcc-invoices/bulk_evaluate_fcc_invoices.py new file mode 100755 index 000000000..8f794135e --- /dev/null +++ b/config_library/pattern-2/fcc-invoices/bulk_evaluate_fcc_invoices.py @@ -0,0 +1,671 @@ +#!/usr/bin/env python3 +""" +Bulk evaluation script for FCC invoices using SticklerEvaluationService. + +This script: +1. Loads ground truth from CSV +2. Loads inference results from a directory +3. Matches ground truth to inference results +4. Evaluates using SticklerEvaluationService (IDP framework integration) +5. Produces aggregated evaluation metrics +""" + +import argparse +import json +import sys +from pathlib import Path +from typing import Dict, List, Tuple, Any +import pandas as pd +from collections import defaultdict +import numpy as np + +# Add lib path for idp_common imports +sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "lib" / "idp_common_pkg")) + +from idp_common.evaluation.stickler_service import SticklerEvaluationService +from idp_common.models import Document, Section +from idp_common.evaluation.models import DocumentEvaluationResult + + +def convert_to_json_serializable(obj): + """Convert numpy types to Python native types for JSON serialization.""" + if isinstance(obj, np.bool_): + return bool(obj) + elif isinstance(obj, np.integer): + return int(obj) + elif isinstance(obj, np.floating): + return float(obj) + elif isinstance(obj, np.ndarray): + return obj.tolist() + elif isinstance(obj, dict): + return {k: convert_to_json_serializable(v) for k, v in obj.items()} + elif isinstance(obj, list): + return [convert_to_json_serializable(item) for item in obj] + else: + return obj + + +class BulkEvaluator: + """Handles bulk evaluation of FCC invoice extraction results using SticklerEvaluationService.""" + + def __init__( + self, + results_dir: str, + csv_path: str, + config_path: str, + output_dir: str = "evaluation_output", + doc_id_column: str = "doc_id", + labels_column: str = "refactored_labels" + ): + """ + Initialize the bulk evaluator. + + Args: + results_dir: Directory containing inference results + csv_path: Path to CSV with ground truth labels + config_path: Path to Stickler configuration JSON + output_dir: Directory to save individual comparison results + doc_id_column: Column name for document IDs in CSV + labels_column: Column name for labels in CSV + """ + self.results_dir = Path(results_dir) + self.csv_path = csv_path + self.config_path = config_path + self.output_dir = Path(output_dir) + self.doc_id_column = doc_id_column + self.labels_column = labels_column + + # Create output directory + self.output_dir.mkdir(parents=True, exist_ok=True) + + # Load configuration and initialize SticklerEvaluationService + self.stickler_config = self._load_stickler_config() + + # Initialize SticklerEvaluationService with the loaded config + service_config = { + "stickler_models": { + "fcc_invoice": self.stickler_config + } + } + self.evaluation_service = SticklerEvaluationService(config=service_config) + + # Initialize aggregation state + self.confusion_matrix = { + "overall": defaultdict(int), + "fields": defaultdict(lambda: defaultdict(int)), + } + self.non_matches = [] + self.errors = [] + self.processed_count = 0 + + print(f"✓ Initialized evaluator with SticklerEvaluationService") + print(f" Model: {self.stickler_config.get('model_name', 'fcc_invoice')}") + print(f" Output directory: {self.output_dir}") + + def _load_stickler_config(self) -> Dict[str, Any]: + """Load Stickler configuration from JSON file.""" + with open(self.config_path, 'r') as f: + config = json.load(f) + print(f"✓ Loaded Stickler config from {self.config_path}") + return config + + def load_ground_truth(self) -> pd.DataFrame: + """ + Load ground truth labels from CSV. + + Returns: + DataFrame with doc_id and parsed labels + """ + print(f"\n📊 Loading ground truth from {self.csv_path}...") + df = pd.read_csv(self.csv_path) + + # Filter to rows with valid labels + df = df[df[self.labels_column].notna()].copy() + + print(f"✓ Loaded {len(df)} documents with ground truth labels") + return df + + def load_inference_results(self) -> Dict[str, Tuple[Dict[str, Any], str]]: + """ + Load all inference results from the results directory. + + Returns: + Dictionary mapping doc_id to tuple of (inference_result_data, result_file_path) + """ + print(f"\n📁 Loading inference results from {self.results_dir}...") + results = {} + + # Iterate through document directories + for doc_dir in self.results_dir.iterdir(): + if not doc_dir.is_dir(): + continue + + doc_id = doc_dir.name + result_path = doc_dir / "sections" / "1" / "result.json" + + if not result_path.exists(): + continue + + try: + with open(result_path, 'r') as f: + result_data = json.load(f) + + inference_result = result_data.get("inference_result", {}) + results[doc_id] = (inference_result, str(result_path)) + except Exception as e: + print(f" ⚠️ Error loading {doc_id}: {e}") + continue + + print(f"✓ Loaded {len(results)} inference results") + return results + + def match_ground_truth_to_results( + self, + ground_truth_df: pd.DataFrame, + inference_results: Dict[str, Tuple[Dict[str, Any], str]] + ) -> List[Tuple[str, Dict[str, Any], Dict[str, Any], str]]: + """ + Match ground truth labels to inference results. + + Args: + ground_truth_df: DataFrame with ground truth + inference_results: Dictionary of inference results with file paths + + Returns: + List of tuples (doc_id, expected_results, actual_results, result_file_path) + """ + print(f"\n🔗 Matching ground truth to inference results...") + matched_pairs = [] + + for _, row in ground_truth_df.iterrows(): + doc_id = row[self.doc_id_column] + + # Try to find matching inference result + result_key = None + if doc_id in inference_results: + result_key = doc_id + elif f"{doc_id}.pdf" in inference_results: + result_key = f"{doc_id}.pdf" + else: + doc_id_no_ext = doc_id.replace('.pdf', '') + if doc_id_no_ext in inference_results: + result_key = doc_id_no_ext + + if result_key is None: + continue + + # Parse ground truth labels + try: + labels_json = row[self.labels_column] + if pd.isna(labels_json): + continue + + expected_results = json.loads(labels_json) + actual_results, result_file_path = inference_results[result_key] + + matched_pairs.append((doc_id, expected_results, actual_results, result_file_path)) + except Exception as e: + print(f" ⚠️ Error parsing labels for {doc_id}: {e}") + continue + + print(f"✓ Matched {len(matched_pairs)} document pairs") + return matched_pairs + + def _normalize_to_list_format(self, data: Dict[str, Any]) -> Dict[str, Any]: + """ + Normalize data to list format for all fields. + Ground truth has lists, predictions have strings for simple fields. + + Args: + data: Data to normalize + + Returns: + Data with all values as lists + """ + normalized = {} + + for key, value in data.items(): + if value is None: + # Keep None as empty list for consistency + normalized[key] = [] + elif isinstance(value, list): + # Already a list + normalized[key] = value + elif isinstance(value, str): + # Convert string to single-item list + normalized[key] = [value] + else: + # Other types, wrap in list + normalized[key] = [value] + + return normalized + + def _create_section_from_data(self, doc_id: str, classification: str = "fcc_invoice") -> Section: + """ + Create a Section object for evaluation. + + Args: + doc_id: Document identifier + classification: Document classification + + Returns: + Section object + """ + return Section( + section_id="1", + classification=classification, + confidence=1.0, + page_ids=["1"] + ) + + def _save_ground_truth_to_temp(self, doc_id: str, expected_results: Dict[str, Any]) -> str: + """ + Save ground truth to a temporary file for evaluation. + + Args: + doc_id: Document identifier + expected_results: Ground truth data (will be normalized to list format) + + Returns: + Path to temporary file + """ + temp_dir = self.output_dir / "temp_ground_truth" + temp_dir.mkdir(parents=True, exist_ok=True) + + # Normalize to list format before saving + normalized_results = self._normalize_to_list_format(expected_results) + + temp_file = temp_dir / f"{doc_id}_gt.json" + with open(temp_file, 'w') as f: + json.dump({"inference_result": normalized_results}, f, indent=2) + + return str(temp_file) + + def _save_actual_results_to_temp(self, doc_id: str, actual_results: Dict[str, Any]) -> str: + """ + Save actual results to a temporary file for evaluation. + + Args: + doc_id: Document identifier + actual_results: Actual inference results (will be normalized to list format) + + Returns: + Path to temporary file + """ + temp_dir = self.output_dir / "temp_actual_results" + temp_dir.mkdir(parents=True, exist_ok=True) + + # Normalize to list format before saving + normalized_results = self._normalize_to_list_format(actual_results) + + temp_file = temp_dir / f"{doc_id}_actual.json" + with open(temp_file, 'w') as f: + json.dump({"inference_result": normalized_results}, f, indent=2) + + return str(temp_file) + + def _accumulate_metrics_from_evaluation(self, eval_result: DocumentEvaluationResult): + """ + Accumulate metrics from DocumentEvaluationResult. + + Args: + eval_result: Document evaluation result from SticklerEvaluationService + """ + # Accumulate field-level metrics from section results + for section_result in eval_result.section_results: + for attr_result in section_result.attributes: + field_name = attr_result.name + + # Determine metric contribution based on attribute result + expected = attr_result.expected + actual = attr_result.actual + matched = attr_result.matched + + # Case 1: Expected value is None/empty + if expected is None or (isinstance(expected, str) and not expected.strip()) or (isinstance(expected, list) and len(expected) == 0): + if actual is None or (isinstance(actual, str) and not actual.strip()) or (isinstance(actual, list) and len(actual) == 0): + self.confusion_matrix["fields"][field_name]["tn"] += 1 + self.confusion_matrix["overall"]["tn"] += 1 + else: + self.confusion_matrix["fields"][field_name]["fp"] += 1 + self.confusion_matrix["fields"][field_name]["fp1"] += 1 + self.confusion_matrix["overall"]["fp"] += 1 + self.confusion_matrix["overall"]["fp1"] += 1 + + # Case 2: Expected value exists but actual doesn't + elif actual is None or (isinstance(actual, str) and not actual.strip()) or (isinstance(actual, list) and len(actual) == 0): + self.confusion_matrix["fields"][field_name]["fn"] += 1 + self.confusion_matrix["overall"]["fn"] += 1 + + # Case 3: Both values exist + else: + if matched: + self.confusion_matrix["fields"][field_name]["tp"] += 1 + self.confusion_matrix["overall"]["tp"] += 1 + else: + self.confusion_matrix["fields"][field_name]["fp"] += 1 + self.confusion_matrix["fields"][field_name]["fp2"] += 1 + self.confusion_matrix["overall"]["fp"] += 1 + self.confusion_matrix["overall"]["fp2"] += 1 + + def evaluate_all( + self, + matched_pairs: List[Tuple[str, Dict[str, Any], Dict[str, Any], str]] + ): + """ + Evaluate all matched pairs using SticklerEvaluationService. + + Args: + matched_pairs: List of (doc_id, expected, actual, result_file_path) tuples + """ + print(f"\n⚙️ Evaluating {len(matched_pairs)} documents...") + + for doc_id, expected_results, actual_results, result_file_path in matched_pairs: + try: + # Create a Section object for evaluation + section = self._create_section_from_data(doc_id) + + # Save ground truth and actual results to temporary files (normalized to list format) + gt_file_path = self._save_ground_truth_to_temp(doc_id, expected_results) + actual_file_path = self._save_actual_results_to_temp(doc_id, actual_results) + + # Use SticklerEvaluationService to evaluate + # Note: We're adapting the service to work with local files + eval_result = self.evaluation_service.evaluate_document( + document_id=doc_id, + sections=[section], + expected_results_uri=f"file://{gt_file_path}", + actual_results_uri=f"file://{actual_file_path}" + ) + + # Convert evaluation result to dict for saving + result_dict = { + "doc_id": doc_id, + "evaluation_result": { + "document_id": eval_result.document_id, + "overall_metrics": eval_result.overall_metrics, + "execution_time": eval_result.execution_time, + "section_results": [ + { + "section_id": sr.section_id, + "document_class": sr.document_class, + "metrics": sr.metrics, + "attributes": [ + { + "name": ar.name, + "expected": ar.expected, + "actual": ar.actual, + "matched": ar.matched, + "score": ar.score, + "reason": ar.reason, + "evaluation_method": ar.evaluation_method + } + for ar in sr.attributes + ] + } + for sr in eval_result.section_results + ] + } + } + + # Convert numpy types to JSON-serializable types + result_dict = convert_to_json_serializable(result_dict) + + # Save individual result + result_file = self.output_dir / f"{doc_id}.json" + with open(result_file, 'w') as f: + json.dump(result_dict, f, indent=2) + + # Accumulate metrics for aggregation + self._accumulate_metrics_from_evaluation(eval_result) + + # Collect non-matches (attributes that didn't match) + for section_result in eval_result.section_results: + for attr_result in section_result.attributes: + if not attr_result.matched: + self.non_matches.append({ + "doc_id": doc_id, + "field": attr_result.name, + "expected": attr_result.expected, + "actual": attr_result.actual, + "reason": attr_result.reason + }) + + self.processed_count += 1 + + except Exception as e: + print(f" ✗ Error evaluating {doc_id}: {e}") + import traceback + traceback.print_exc() + + # Save error result + error_file = self.output_dir / f"{doc_id}.error.json" + with open(error_file, 'w') as f: + json.dump({ + "doc_id": doc_id, + "error": str(e), + "error_type": type(e).__name__ + }, f, indent=2) + + # Track error + self.errors.append({ + "doc_id": doc_id, + "error": str(e), + "error_type": type(e).__name__ + }) + + print(f"✓ Completed evaluation") + print(f" Individual results saved to: {self.output_dir}") + + + + def _calculate_metrics(self, cm: Dict[str, int]) -> Dict[str, float]: + """ + Calculate derived metrics from confusion matrix. + + Args: + cm: Confusion matrix with tp, fp, tn, fn counts + + Returns: + Dictionary of calculated metrics + """ + tp = cm.get("tp", 0) + fp = cm.get("fp", 0) + tn = cm.get("tn", 0) + fn = cm.get("fn", 0) + fp1 = cm.get("fp1", 0) + cm.get("fa", 0) # fa is alias for fp1 + fp2 = cm.get("fp2", 0) + cm.get("fd", 0) # fd is alias for fp2 + + total = tp + fp + tn + fn + + # Calculate metrics with zero-division handling + precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0 + recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0 + f1_score = ( + 2 * (precision * recall) / (precision + recall) + if (precision + recall) > 0 + else 0.0 + ) + accuracy = (tp + tn) / total if total > 0 else 0.0 + + return { + "precision": precision, + "recall": recall, + "f1_score": f1_score, + "accuracy": accuracy, + "tp": tp, + "fp": fp, + "tn": tn, + "fn": fn, + "fp1": fp1, + "fp2": fp2, + "total": total + } + + def print_aggregated_results(self): + """Print aggregated metrics summary.""" + print("\n" + "=" * 80) + print("AGGREGATED EVALUATION RESULTS") + print("=" * 80) + + print(f"\n📊 Processing Summary:") + print(f" Documents processed: {self.processed_count}") + print(f" Errors encountered: {len(self.errors)}") + print(f" Non-matches found: {len(self.non_matches)}") + + # Overall metrics + print(f"\n📈 Overall Metrics:") + overall_metrics = self._calculate_metrics(self.confusion_matrix["overall"]) + print(f" Precision: {overall_metrics['precision']:.4f}") + print(f" Recall: {overall_metrics['recall']:.4f}") + print(f" F1 Score: {overall_metrics['f1_score']:.4f}") + print(f" Accuracy: {overall_metrics['accuracy']:.4f}") + + print(f"\n Confusion Matrix:") + print(f" TP: {overall_metrics['tp']:6d} | FP: {overall_metrics['fp']:6d}") + print(f" FN: {overall_metrics['fn']:6d} | TN: {overall_metrics['tn']:6d}") + print(f" FP1 (False Alarm): {overall_metrics['fp1']:6d}") + print(f" FP2 (Wrong Value): {overall_metrics['fp2']:6d}") + + # Field-level metrics + if self.confusion_matrix["fields"]: + print(f"\n📋 Field-Level Metrics (Top 10 by F1 Score):") + field_metrics = [] + for field_path, cm in self.confusion_matrix["fields"].items(): + metrics = self._calculate_metrics(cm) + metrics["field"] = field_path + field_metrics.append(metrics) + + # Sort by F1 score + field_metrics.sort(key=lambda x: x["f1_score"], reverse=True) + + print(f" {'Field':<40} {'Precision':>10} {'Recall':>10} {'F1':>10}") + print(f" {'-'*40} {'-'*10} {'-'*10} {'-'*10}") + for metrics in field_metrics[:10]: + print(f" {metrics['field']:<40} {metrics['precision']:>10.4f} {metrics['recall']:>10.4f} {metrics['f1_score']:>10.4f}") + + print("\n" + "=" * 80) + + def save_aggregated_results(self, output_path: str): + """ + Save aggregated results to JSON file. + + Args: + output_path: Path to save aggregated results + """ + # Calculate metrics for all fields + overall_metrics = self._calculate_metrics(self.confusion_matrix["overall"]) + + field_metrics = {} + for field_path, cm in self.confusion_matrix["fields"].items(): + field_metrics[field_path] = self._calculate_metrics(cm) + + output_data = { + "summary": { + "documents_processed": self.processed_count, + "errors": len(self.errors), + "non_matches": len(self.non_matches) + }, + "overall_metrics": overall_metrics, + "field_metrics": field_metrics, + "non_matches": self.non_matches[:100], # Limit to first 100 for file size + "errors": self.errors + } + + with open(output_path, 'w') as f: + json.dump(output_data, f, indent=2) + + print(f"\n💾 Aggregated results saved to {output_path}") + + def run(self): + """Run the complete bulk evaluation workflow.""" + print("=" * 80) + print("BULK FCC INVOICE EVALUATION") + print("=" * 80) + + # Step 1: Load ground truth + ground_truth_df = self.load_ground_truth() + + # Step 2: Load inference results + inference_results = self.load_inference_results() + + # Step 3: Match ground truth to results + matched_pairs = self.match_ground_truth_to_results( + ground_truth_df, + inference_results + ) + + if not matched_pairs: + print("\n❌ No matched pairs found. Check doc_id column and file naming.") + return + + # Step 4: Evaluate all pairs and save individual results + self.evaluate_all(matched_pairs) + + # Step 5: Print aggregated results + self.print_aggregated_results() + + # Step 6: Save aggregated results + aggregated_path = self.output_dir / "aggregated_metrics.json" + self.save_aggregated_results(str(aggregated_path)) + + print("\n" + "=" * 80) + print(f"✅ Evaluation complete!") + print(f" Individual results: {self.output_dir}") + print(f" Aggregated metrics: {aggregated_path}") + print("=" * 80) + + +def main(): + """Main entry point.""" + parser = argparse.ArgumentParser( + description="Bulk evaluate FCC invoice extraction results using Stickler" + ) + parser.add_argument( + "--results-dir", + required=True, + help="Directory containing inference results" + ) + parser.add_argument( + "--csv-path", + required=True, + help="Path to CSV file with ground truth labels" + ) + parser.add_argument( + "--config-path", + default="config_library/pattern-2/fcc-invoices/stickler_config.json", + help="Path to Stickler configuration JSON" + ) + parser.add_argument( + "--doc-id-column", + default="doc_id", + help="Column name for document IDs in CSV" + ) + parser.add_argument( + "--labels-column", + default="refactored_labels", + help="Column name for labels in CSV" + ) + parser.add_argument( + "--output-dir", + default="evaluation_output", + help="Output directory for individual evaluation results" + ) + + args = parser.parse_args() + + # Create evaluator and run + evaluator = BulkEvaluator( + results_dir=args.results_dir, + csv_path=args.csv_path, + config_path=args.config_path, + output_dir=args.output_dir, + doc_id_column=args.doc_id_column, + labels_column=args.labels_column + ) + + evaluator.run() + + +if __name__ == "__main__": + main() diff --git a/config_library/pattern-2/fcc-invoices/bulk_evaluate_fcc_invoices_simple.py b/config_library/pattern-2/fcc-invoices/bulk_evaluate_fcc_invoices_simple.py new file mode 100644 index 000000000..37258e6a1 --- /dev/null +++ b/config_library/pattern-2/fcc-invoices/bulk_evaluate_fcc_invoices_simple.py @@ -0,0 +1,260 @@ +#!/usr/bin/env python3 +""" +Simple bulk evaluation script for FCC invoices using Stickler. + +This script evaluates FCC invoice extraction results against ground truth +from a CSV file, producing aggregated metrics. +""" + +import argparse +import json +import sys +from pathlib import Path +from typing import Dict, Any +import pandas as pd +import numpy as np +from collections import defaultdict + +# Add lib path for idp_common imports +sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "lib" / "idp_common_pkg")) + +from idp_common.evaluation.stickler_service import SticklerEvaluationService +from idp_common.models import Section + + +def to_json_serializable(obj): + """Convert numpy types to Python native types.""" + if isinstance(obj, (np.bool_, np.integer, np.floating)): + return obj.item() + elif isinstance(obj, np.ndarray): + return obj.tolist() + elif isinstance(obj, dict): + return {k: to_json_serializable(v) for k, v in obj.items()} + elif isinstance(obj, list): + return [to_json_serializable(item) for item in obj] + return obj + + +def load_stickler_config(config_path: str) -> Dict[str, Any]: + """Load Stickler configuration from JSON file.""" + with open(config_path, 'r') as f: + return json.load(f) + + +def normalize_to_list_format(data: Dict[str, Any]) -> Dict[str, Any]: + """Normalize data to list format for all fields.""" + normalized = {} + for key, value in data.items(): + if value is None: + normalized[key] = [] + elif isinstance(value, list): + normalized[key] = value + elif isinstance(value, str): + normalized[key] = [value] + else: + normalized[key] = [value] + return normalized + + +def main(): + parser = argparse.ArgumentParser( + description="Bulk evaluate FCC invoice extraction results" + ) + parser.add_argument("--results-dir", required=True, help="Directory containing inference results") + parser.add_argument("--csv-path", required=True, help="Path to CSV file with ground truth labels") + parser.add_argument("--config-path", required=True, help="Path to Stickler configuration JSON") + parser.add_argument("--doc-id-column", default="doc_id", help="Column name for document IDs") + parser.add_argument("--labels-column", default="refactored_labels", help="Column name for labels") + parser.add_argument("--output-dir", default="evaluation_output", help="Output directory") + + args = parser.parse_args() + + results_dir = Path(args.results_dir) + output_dir = Path(args.output_dir) + output_dir.mkdir(parents=True, exist_ok=True) + + print("=" * 80) + print("BULK FCC INVOICE EVALUATION") + print("=" * 80) + + # Load configuration and initialize service + print(f"\n📋 Loading Stickler config from {args.config_path}...") + stickler_config = load_stickler_config(args.config_path) + + service_config = { + "stickler_models": { + "fcc_invoice": stickler_config + } + } + service = SticklerEvaluationService(config=service_config) + print(f"✓ Service initialized") + + # Load ground truth + print(f"\n📊 Loading ground truth from {args.csv_path}...") + df = pd.read_csv(args.csv_path) + df = df[df[args.labels_column].notna()].copy() + print(f"✓ Loaded {len(df)} documents with ground truth") + + # Load inference results + print(f"\n📁 Loading inference results from {results_dir}...") + inference_results = {} + for doc_dir in results_dir.iterdir(): + if not doc_dir.is_dir(): + continue + result_path = doc_dir / "sections" / "1" / "result.json" + if result_path.exists(): + with open(result_path, 'r') as f: + result_data = json.load(f) + inference_results[doc_dir.name] = result_data.get("inference_result", {}) + print(f"✓ Loaded {len(inference_results)} inference results") + + # Match and evaluate + print(f"\n⚙️ Evaluating documents...") + + # Accumulation state + overall_metrics = defaultdict(int) + field_metrics = defaultdict(lambda: defaultdict(int)) + processed = 0 + errors = [] + + for _, row in df.iterrows(): + doc_id = row[args.doc_id_column] + + # Find matching result + result_key = None + for key in [doc_id, f"{doc_id}.pdf", doc_id.replace('.pdf', '')]: + if key in inference_results: + result_key = key + break + + if not result_key: + continue + + try: + # Parse ground truth and get actual results + expected = json.loads(row[args.labels_column]) + actual = inference_results[result_key] + + # Normalize to list format + expected = normalize_to_list_format(expected) + actual = normalize_to_list_format(actual) + + # Create section and evaluate + section = Section(section_id="1", classification="fcc_invoice", page_ids=["1"]) + result = service.evaluate_section(section, expected, actual) + + # Accumulate metrics from attributes + for attr in result.attributes: + field = attr.name + exp_val = attr.expected + act_val = attr.actual + matched = attr.matched + + # Determine metric type + exp_empty = not exp_val or (isinstance(exp_val, list) and len(exp_val) == 0) + act_empty = not act_val or (isinstance(act_val, list) and len(act_val) == 0) + + if exp_empty and act_empty: + overall_metrics["tn"] += 1 + field_metrics[field]["tn"] += 1 + elif exp_empty and not act_empty: + overall_metrics["fp"] += 1 + overall_metrics["fp1"] += 1 + field_metrics[field]["fp"] += 1 + field_metrics[field]["fp1"] += 1 + elif not exp_empty and act_empty: + overall_metrics["fn"] += 1 + field_metrics[field]["fn"] += 1 + elif matched: + overall_metrics["tp"] += 1 + field_metrics[field]["tp"] += 1 + else: + overall_metrics["fp"] += 1 + overall_metrics["fp2"] += 1 + field_metrics[field]["fp"] += 1 + field_metrics[field]["fp2"] += 1 + + # Save individual result + result_file = output_dir / f"{doc_id}.json" + result_data = { + "doc_id": doc_id, + "metrics": result.metrics, + "attributes": [ + { + "name": a.name, + "expected": a.expected, + "actual": a.actual, + "matched": a.matched, + "score": float(a.score), + "reason": a.reason + } + for a in result.attributes + ] + } + with open(result_file, 'w') as f: + json.dump(to_json_serializable(result_data), f, indent=2) + + processed += 1 + + except Exception as e: + errors.append({"doc_id": doc_id, "error": str(e)}) + print(f" ✗ Error evaluating {doc_id}: {e}") + + print(f"✓ Completed evaluation of {processed} documents") + + # Calculate metrics + def calc_metrics(cm): + tp, fp, tn, fn = cm["tp"], cm["fp"], cm["tn"], cm["fn"] + total = tp + fp + tn + fn + precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0 + recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0 + f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0 + accuracy = (tp + tn) / total if total > 0 else 0.0 + return { + "precision": precision, "recall": recall, "f1_score": f1, "accuracy": accuracy, + "tp": tp, "fp": fp, "tn": tn, "fn": fn, + "fp1": cm["fp1"], "fp2": cm["fp2"], "total": total + } + + overall = calc_metrics(overall_metrics) + fields = {field: calc_metrics(cm) for field, cm in field_metrics.items()} + + # Print results + print("\n" + "=" * 80) + print("AGGREGATED RESULTS") + print("=" * 80) + print(f"\n📊 Summary: {processed} processed, {len(errors)} errors") + print(f"\n📈 Overall Metrics:") + print(f" Precision: {overall['precision']:.4f}") + print(f" Recall: {overall['recall']:.4f}") + print(f" F1 Score: {overall['f1_score']:.4f}") + print(f" Accuracy: {overall['accuracy']:.4f}") + print(f"\n Confusion Matrix:") + print(f" TP: {overall['tp']:6d} | FP: {overall['fp']:6d}") + print(f" FN: {overall['fn']:6d} | TN: {overall['tn']:6d}") + print(f" FP1: {overall['fp1']:6d} | FP2: {overall['fp2']:6d}") + + # Top fields + sorted_fields = sorted(fields.items(), key=lambda x: x[1]["f1_score"], reverse=True) + print(f"\n📋 Field-Level Metrics (Top 10):") + print(f" {'Field':<40} {'Precision':>10} {'Recall':>10} {'F1':>10}") + print(f" {'-'*40} {'-'*10} {'-'*10} {'-'*10}") + for field, metrics in sorted_fields[:10]: + print(f" {field:<40} {metrics['precision']:>10.4f} {metrics['recall']:>10.4f} {metrics['f1_score']:>10.4f}") + + # Save aggregated results + output_file = output_dir / "aggregated_metrics.json" + with open(output_file, 'w') as f: + json.dump({ + "summary": {"documents_processed": processed, "errors": len(errors)}, + "overall_metrics": overall, + "field_metrics": fields, + "errors": errors + }, f, indent=2) + + print(f"\n💾 Results saved to {output_dir}") + print("=" * 80) + + +if __name__ == "__main__": + main() diff --git a/config_library/pattern-2/fcc-invoices/bulk_evaluate_from_idp_config.py b/config_library/pattern-2/fcc-invoices/bulk_evaluate_from_idp_config.py new file mode 100644 index 000000000..d49d795cf --- /dev/null +++ b/config_library/pattern-2/fcc-invoices/bulk_evaluate_from_idp_config.py @@ -0,0 +1,310 @@ +#!/usr/bin/env python3 +""" +Bulk evaluation script that uses IDP configuration directly. + +This script reads the Stickler evaluation configuration from the IDP config file +(sr_FCC_config.json) instead of requiring a separate stickler_config.json. +""" + +import argparse +import json +import sys +from pathlib import Path +from typing import Dict, Any +import pandas as pd +import numpy as np +from collections import defaultdict + +# Add lib path for idp_common imports +sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "lib" / "idp_common_pkg")) + +from idp_common.evaluation.stickler_service import SticklerEvaluationService +from idp_common.models import Section + + +def to_json_serializable(obj): + """Convert numpy types to Python native types.""" + if isinstance(obj, (np.bool_, np.integer, np.floating)): + return obj.item() + elif isinstance(obj, np.ndarray): + return obj.tolist() + elif isinstance(obj, dict): + return {k: to_json_serializable(v) for k, v in obj.items()} + elif isinstance(obj, list): + return [to_json_serializable(item) for item in obj] + return obj + + +def extract_stickler_config_from_idp_config(idp_config: Dict[str, Any]) -> Dict[str, Any]: + """ + Extract Stickler configuration from IDP config JSON Schema. + + Args: + idp_config: Full IDP configuration + + Returns: + Stickler configuration in the format expected by SticklerEvaluationService + """ + # Get the first class definition (assuming single document type) + if "classes" not in idp_config or not idp_config["classes"]: + raise ValueError("No classes found in IDP configuration") + + class_schema = idp_config["classes"][0] + + # Extract model name and threshold + model_name = class_schema.get("x-aws-stickler-model-name", "Document") + match_threshold = class_schema.get("x-aws-stickler-match-threshold", 0.7) + + # Build fields configuration from properties + fields = {} + properties = class_schema.get("properties", {}) + + for field_name, field_schema in properties.items(): + # Extract Stickler extensions + comparator = field_schema.get("x-aws-stickler-comparator") + threshold = field_schema.get("x-aws-stickler-threshold") + weight = field_schema.get("x-aws-stickler-weight", 1.0) + + if comparator: # Only include fields with Stickler configuration + fields[field_name] = { + "type": "list", # All fields are arrays in flat format + "comparator": comparator, + "threshold": threshold, + "weight": weight, + "description": field_schema.get("description", "") + } + + return { + "model_name": model_name, + "match_threshold": match_threshold, + "fields": fields + } + + +def normalize_to_list_format(data: Dict[str, Any]) -> Dict[str, Any]: + """Normalize data to list format for all fields.""" + normalized = {} + for key, value in data.items(): + if value is None: + normalized[key] = [] + elif isinstance(value, list): + normalized[key] = value + elif isinstance(value, str): + normalized[key] = [value] + else: + normalized[key] = [value] + return normalized + + +def main(): + parser = argparse.ArgumentParser( + description="Bulk evaluate using IDP configuration directly" + ) + parser.add_argument("--results-dir", required=True, help="Directory containing inference results") + parser.add_argument("--csv-path", required=True, help="Path to CSV file with ground truth labels") + parser.add_argument("--idp-config-path", required=True, help="Path to IDP configuration JSON (e.g., sr_FCC_config.json)") + parser.add_argument("--doc-id-column", default="doc_id", help="Column name for document IDs") + parser.add_argument("--labels-column", default="refactored_labels", help="Column name for labels") + parser.add_argument("--output-dir", default="evaluation_output", help="Output directory") + + args = parser.parse_args() + + results_dir = Path(args.results_dir) + output_dir = Path(args.output_dir) + output_dir.mkdir(parents=True, exist_ok=True) + + print("=" * 80) + print("BULK FCC INVOICE EVALUATION (IDP Config)") + print("=" * 80) + + # Load IDP configuration + print(f"\n📋 Loading IDP config from {args.idp_config_path}...") + with open(args.idp_config_path, 'r') as f: + idp_config = json.load(f) + + # Extract Stickler configuration from IDP config + print("📋 Extracting Stickler configuration from IDP config...") + stickler_config = extract_stickler_config_from_idp_config(idp_config) + + print(f"✓ Extracted config for model: {stickler_config['model_name']}") + print(f"✓ Found {len(stickler_config['fields'])} fields with Stickler configuration") + + # Initialize SticklerEvaluationService + service_config = { + "stickler_models": { + "fcc_invoice": stickler_config + } + } + service = SticklerEvaluationService(config=service_config) + print(f"✓ Service initialized") + + # Load ground truth + print(f"\n📊 Loading ground truth from {args.csv_path}...") + df = pd.read_csv(args.csv_path) + df = df[df[args.labels_column].notna()].copy() + print(f"✓ Loaded {len(df)} documents with ground truth") + + # Load inference results + print(f"\n📁 Loading inference results from {results_dir}...") + inference_results = {} + for doc_dir in results_dir.iterdir(): + if not doc_dir.is_dir(): + continue + result_path = doc_dir / "sections" / "1" / "result.json" + if result_path.exists(): + with open(result_path, 'r') as f: + result_data = json.load(f) + inference_results[doc_dir.name] = result_data.get("inference_result", {}) + print(f"✓ Loaded {len(inference_results)} inference results") + + # Match and evaluate + print(f"\n⚙️ Evaluating documents...") + + # Accumulation state + overall_metrics = defaultdict(int) + field_metrics = defaultdict(lambda: defaultdict(int)) + processed = 0 + errors = [] + + for _, row in df.iterrows(): + doc_id = row[args.doc_id_column] + + # Find matching result + result_key = None + for key in [doc_id, f"{doc_id}.pdf", doc_id.replace('.pdf', '')]: + if key in inference_results: + result_key = key + break + + if not result_key: + continue + + try: + # Parse ground truth and get actual results + expected = json.loads(row[args.labels_column]) + actual = inference_results[result_key] + + # Normalize to list format + expected = normalize_to_list_format(expected) + actual = normalize_to_list_format(actual) + + # Create section and evaluate + section = Section(section_id="1", classification="fcc_invoice", page_ids=["1"]) + result = service.evaluate_section(section, expected, actual) + + # Accumulate metrics from attributes + for attr in result.attributes: + field = attr.name + exp_val = attr.expected + act_val = attr.actual + matched = attr.matched + + # Determine metric type + exp_empty = not exp_val or (isinstance(exp_val, list) and len(exp_val) == 0) + act_empty = not act_val or (isinstance(act_val, list) and len(act_val) == 0) + + if exp_empty and act_empty: + overall_metrics["tn"] += 1 + field_metrics[field]["tn"] += 1 + elif exp_empty and not act_empty: + overall_metrics["fp"] += 1 + overall_metrics["fp1"] += 1 + field_metrics[field]["fp"] += 1 + field_metrics[field]["fp1"] += 1 + elif not exp_empty and act_empty: + overall_metrics["fn"] += 1 + field_metrics[field]["fn"] += 1 + elif matched: + overall_metrics["tp"] += 1 + field_metrics[field]["tp"] += 1 + else: + overall_metrics["fp"] += 1 + overall_metrics["fp2"] += 1 + field_metrics[field]["fp"] += 1 + field_metrics[field]["fp2"] += 1 + + # Save individual result + result_file = output_dir / f"{doc_id}.json" + result_data = { + "doc_id": doc_id, + "metrics": result.metrics, + "attributes": [ + { + "name": a.name, + "expected": a.expected, + "actual": a.actual, + "matched": a.matched, + "score": float(a.score), + "reason": a.reason + } + for a in result.attributes + ] + } + with open(result_file, 'w') as f: + json.dump(to_json_serializable(result_data), f, indent=2) + + processed += 1 + + except Exception as e: + errors.append({"doc_id": doc_id, "error": str(e)}) + print(f" ✗ Error evaluating {doc_id}: {e}") + + print(f"✓ Completed evaluation of {processed} documents") + + # Calculate metrics + def calc_metrics(cm): + tp, fp, tn, fn = cm["tp"], cm["fp"], cm["tn"], cm["fn"] + total = tp + fp + tn + fn + precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0 + recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0 + f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0 + accuracy = (tp + tn) / total if total > 0 else 0.0 + return { + "precision": precision, "recall": recall, "f1_score": f1, "accuracy": accuracy, + "tp": tp, "fp": fp, "tn": tn, "fn": fn, + "fp1": cm["fp1"], "fp2": cm["fp2"], "total": total + } + + overall = calc_metrics(overall_metrics) + fields = {field: calc_metrics(cm) for field, cm in field_metrics.items()} + + # Print results + print("\n" + "=" * 80) + print("AGGREGATED RESULTS") + print("=" * 80) + print(f"\n📊 Summary: {processed} processed, {len(errors)} errors") + print(f"\n📈 Overall Metrics:") + print(f" Precision: {overall['precision']:.4f}") + print(f" Recall: {overall['recall']:.4f}") + print(f" F1 Score: {overall['f1_score']:.4f}") + print(f" Accuracy: {overall['accuracy']:.4f}") + print(f"\n Confusion Matrix:") + print(f" TP: {overall['tp']:6d} | FP: {overall['fp']:6d}") + print(f" FN: {overall['fn']:6d} | TN: {overall['tn']:6d}") + print(f" FP1: {overall['fp1']:6d} | FP2: {overall['fp2']:6d}") + + # Top fields + sorted_fields = sorted(fields.items(), key=lambda x: x[1]["f1_score"], reverse=True) + print(f"\n📋 Field-Level Metrics (Top 10):") + print(f" {'Field':<40} {'Precision':>10} {'Recall':>10} {'F1':>10}") + print(f" {'-'*40} {'-'*10} {'-'*10} {'-'*10}") + for field, metrics in sorted_fields[:10]: + print(f" {field:<40} {metrics['precision']:>10.4f} {metrics['recall']:>10.4f} {metrics['f1_score']:>10.4f}") + + # Save aggregated results + output_file = output_dir / "aggregated_metrics.json" + with open(output_file, 'w') as f: + json.dump({ + "summary": {"documents_processed": processed, "errors": len(errors)}, + "overall_metrics": overall, + "field_metrics": fields, + "errors": errors, + "stickler_config_used": stickler_config + }, f, indent=2) + + print(f"\n💾 Results saved to {output_dir}") + print("=" * 80) + + +if __name__ == "__main__": + main() diff --git a/config_library/pattern-2/fcc-invoices/config.yaml b/config_library/pattern-2/fcc-invoices/config.yaml new file mode 100644 index 000000000..93796ee93 --- /dev/null +++ b/config_library/pattern-2/fcc-invoices/config.yaml @@ -0,0 +1,131 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: MIT-0 +notes: Configuration for FCC invoice information extraction (no classification) +ocr: + backend: "textract" + features: + - name: LAYOUT + - name: TABLES + +classes: + - name: FCC-Invoice + description: >- + Federal Communications Commission (FCC) political advertising invoice showing broadcast + time purchases, including line items with descriptions, dates, rates, and totals for + political advertising campaigns. + attributes: + - name: agency + description: >- + The advertising agency or media buyer handling the political advertising purchase. + evaluation_method: EXACT + attributeType: simple + + - name: advertiser + description: >- + The political advertiser or campaign purchasing the broadcast time. + evaluation_method: EXACT + attributeType: simple + + - name: gross_total + description: >- + The total gross amount for all line items before any discounts or adjustments. + evaluation_method: NUMERIC_EXACT + attributeType: simple + + - name: net_amount_due + description: >- + The final net amount due after any discounts or adjustments have been applied. + evaluation_method: NUMERIC_EXACT + attributeType: simple + + - name: line_items + listItemTemplate: + itemAttributes: + - name: description + description: >- + The broadcast time slot description, typically showing days of week and time range + (e.g., "M-F 11a-12p" for Monday through Friday 11am to 12pm). + evaluation_method: EXACT + + - name: days + description: >- + The days of the week for this broadcast slot, often in format like "MTWTF--" + where each position represents a day (Monday, Tuesday, Wednesday, Thursday, Friday, + Saturday, Sunday) with dashes for non-broadcast days. + evaluation_method: EXACT + + - name: rate + description: >- + The rate or cost for this specific broadcast time slot, may include commas + for thousands separator. + evaluation_method: NUMERIC_EXACT + + - name: start_date + description: >- + The start date for this line item's broadcast schedule, typically in MM/DD/YY format. + evaluation_method: EXACT + + - name: end_date + description: >- + The end date for this line item's broadcast schedule, typically in MM/DD/YY format. + evaluation_method: EXACT + + itemDescription: >- + Each item represents a specific broadcast time slot purchase with its schedule, + rate, and date range. + + description: >- + List of line items detailing each broadcast time slot purchase, including the time + description, days of week, rate, and date range for the advertising schedule. + evaluation_method: LLM + attributeType: list + +extraction: + model: "us.anthropic.claude-3-7-sonnet-20250219-v1:0" + temperature: 0.0 + top_p: 0.9 + top_k: 5 + max_tokens: 4096 + system_prompt: | + You are an expert at extracting structured information from FCC political advertising invoices. + Extract all requested fields accurately, paying special attention to: + - Line item details including time slots, days, rates, and date ranges + - Monetary amounts (preserve formatting with commas and decimals) + - Date formats (typically MM/DD/YY) + - Agency and advertiser names + + For line items, ensure you capture all rows from any tables showing broadcast schedules. + Days of week are often encoded as 7-character strings where each position represents a day. + + task_prompt: | + Extract the following information from this FCC invoice: + + 1. Agency name + 2. Advertiser name + 3. Gross total amount + 4. Net amount due + 5. All line items with: + - Description (time slot) + - Days (day of week encoding) + - Rate (cost) + - Start date + - End date + + Return the information in the specified JSON schema format. + +classification: + enabled: true + model: "us.anthropic.claude-3-7-sonnet-20250219-v1:0" + temperature: 0.0 + top_p: 0.9 + top_k: 5 + max_tokens: 1024 + system_prompt: "You are a document classification expert. Classify each page according to the provided document types." + task_prompt: "Classify this page as one of the available document types based on its content and structure." + +evaluation: + enabled: true + methods: + - EXACT + - NUMERIC_EXACT + - LLM diff --git a/config_library/pattern-2/fcc-invoices/fcc_configured.yaml b/config_library/pattern-2/fcc-invoices/fcc_configured.yaml new file mode 100644 index 000000000..35e895ef0 --- /dev/null +++ b/config_library/pattern-2/fcc-invoices/fcc_configured.yaml @@ -0,0 +1,576 @@ +classification: + top_p: 0.9 + max_tokens: 1024 + top_k: 5 + task_prompt: |- + + + You are a document classification system. Your task is to analyze a document package containing multiple pages and identify distinct document segments, classifying each segment according to the predefined document types provided below. + + + + + + + {CLASS_NAMES_AND_DESCRIPTIONS} + + + + + + + Key terms used in this task: + + - ordinal_start_page: The one-based beginning page number of a document segment within the document package + + - ordinal_end_page: The one-based ending page number of a document segment within the document package + + - document_type: The document type code detected for a document segment + + - document segment: A continuous range of pages that form a single, complete document + + + + + + + Follow these steps to classify documents: + + 1. Read through the entire document package to understand its contents + + 2. Identify page ranges that form complete, distinct documents + + 3. Match each document segment to ONE of the document types listed in + + 4. CRITICAL: Only use document types explicitly listed in the section + + 5. If a document doesn't clearly match any listed type, assign it to the most similar listed type + + 6. Pay special attention to adjacent documents of the same type - they must be separated into distinct segments + + 7. Record the ordinal_start_page and ordinal_end_page for each identified segment + + 8. Provide appropriate reasons and facts for the predicted document type + + + + + + + Rules for determining document boundaries: + + - Content continuity: Pages with continuing paragraphs, numbered sections, or ongoing narratives belong to the same document + + - Visual consistency: Similar layouts, headers, footers, and styling indicate pages belong together + + - Logical structure: Documents typically have clear beginning, middle, and end sections + + - New document indicators: Title pages, cover sheets, or significantly different subject matter signal a new document + + - Topic coherence: Pages discussing the same subject should be grouped together + + - IMPORTANT: Distinct documents of the same type that are adjacent must be separated into different segments + + + + + + + Return your classification as valid JSON following this exact structure: + + ```json + + { + "segments": [ + { + "ordinal_start_page": 1, + "ordinal_end_page": 3, + "type": "document_type_from_list", + "reason": "facts and reasons to classify as the predicted type", + }, + { + "ordinal_start_page": 4, + "ordinal_end_page": 7, + "type": "document_type_from_list" + "reason": "facts and reasons to classify as the predicted type", + } + ] + } + + ``` + + + + + <> + + + + + {DOCUMENT_TEXT} + + + + + + + Analyze the provided above and: + + 1. Apply the to identify distinct document segments + + 2. Use the to determine where one document ends and another begins + + 3. Classify each segment using ONLY the document types from the list + + 4. Ensure adjacent documents of the same type are separated into distinct segments + + 5. Output your classification in the exact JSON format specified in + + 6. You can get this information from the previous message. Analyze the previous messages to get these instructions. + + + Remember: You must ONLY use document types that appear in the reference data. Do not invent or create new document types. + + + temperature: 0 + model: us.anthropic.claude-sonnet-4-20250514-v1:0 + system_prompt: >- + You are a document classification expert. Classify each page according to + the provided document types. + enabled: true + maxPagesForClassification: ALL + classificationMethod: textbasedHolisticClassification +notes: Configuration for FCC invoice information extraction (no classification) +classes: + - name: FCC-Invoice + description: >- + Federal Communications Commission (FCC) political advertising invoice + showing broadcast time purchases, including line items with descriptions, + dates, rates, and totals for political advertising campaigns. + attributes: + - name: agency + description: >- + The advertising agency or media buyer handling the political + advertising purchase. + evaluation_method: EXACT + attributeType: simple + - name: advertiser + description: The political advertiser or campaign purchasing the broadcast time. + evaluation_method: EXACT + attributeType: simple + - name: gross_total + description: >- + The total gross amount for all line items before any discounts or + adjustments. + evaluation_method: NUMERIC_EXACT + attributeType: simple + - name: net_amount_due + description: >- + The final net amount due after any discounts or adjustments have been + applied. + evaluation_method: NUMERIC_EXACT + attributeType: simple + - name: line_item__description + listItemTemplate: + itemDescription: >- + Each line item description is usually a specification in industry + short-hand of when the advertisement should run. examples include "M + -F 6p -6:30p", "M -F 11a -12p" + description: > + These line item descriptions list include the code of what advertising + spots were bought, in industry short-hand. i.e. 'IA* 11a -12p' + attributeType: list + evaluation_method: '' + - name: line_item__days + listItemTemplate: + itemDescription: >- + Each list item will be a string with shorthand representing the days + of the week the ad should run. I.e. "MTWTF--" + description: >- + The Line item days list contains the days of the week that the ad + should run. + attributeType: list + evaluation_method: '' + - name: line_item__rate + listItemTemplate: + itemDescription: 'Each entry is a unitless float, i.e. "300.00", or "1200.00" ' + description: >- + The line item rate is a list of rates for each of the line items. + These are rates represented in dollars, but without any $ sign. i.e. + attributeType: list + evaluation_method: '' + - name: line_item__start_date + listItemTemplate: + itemDescription: Each start date, formatted, as MM/DD/YY. i.e. "11/07/21" + description: 'The start dates for each line item. ' + attributeType: list + evaluation_method: '' + - name: line_item__end_date + listItemTemplate: + itemDescription: Each end date, formatted, as MM/DD/YY. i.e. "11/07/21" + description: 'The end dates for each line item. ' + attributeType: list + evaluation_method: '' +extraction: + top_p: 0.9 + max_tokens: 4096 + top_k: 5 + task_prompt: | + + + You are an expert in document analysis and information extraction. + You can understand and extract key information from documents classified as type + + {DOCUMENT_CLASS}. + + + + + + + Your task is to take the unstructured text provided and convert it into a well-organized table format using JSON. Identify the main entities, attributes, or categories mentioned in the attributes list below and use them as keys in the JSON object. + Then, extract the relevant information from the text and populate the corresponding values in the JSON object. + + + + + + + Guidelines: + 1. Ensure that the data is accurately represented and properly formatted within + the JSON structure + 2. Include double quotes around all keys and values + 3. Do not make up data - only extract information explicitly found in the + document + 4. Do not use /n for new lines, use a space instead + 5. If a field is not found or if unsure, return null + 6. All dates should be in MM/DD/YYYY format + 7. Do not perform calculations or summations unless totals are explicitly given + 8. If an alias is not found in the document, return null + 9. Guidelines for checkboxes: + 9.A. CAREFULLY examine each checkbox, radio button, and selection field: + - Look for marks like ✓, ✗, x, filled circles (●), darkened areas, or handwritten checks indicating selection + - For checkboxes and multi-select fields, ONLY INCLUDE options that show clear visual evidence of selection + - DO NOT list options that have no visible selection mark + 9.B. For ambiguous or overlapping tick marks: + - If a mark overlaps between two or more checkboxes, determine which option contains the majority of the mark + - Consider a checkbox selected if the mark is primarily inside the check box or over the option text + - When a mark touches multiple options, analyze which option was most likely intended based on position and density. For handwritten checks, the mark typically flows from the selected checkbox outward. + - Carefully analyze visual cues and contextual hints. Think from a human perspective, anticipate natural tendencies, and apply thoughtful reasoning to make the best possible judgment. + 10. Think step by step first and then answer. + + + + If the attributes section below contains a list of attribute names and + descriptions, then output only those attributes, using the provided + descriptions as guidance for finding the correct values. + + + + {ATTRIBUTE_NAMES_AND_DESCRIPTIONS} + + + + + <> + + + + + {DOCUMENT_TEXT} + + + + + + + {DOCUMENT_IMAGE} + + + + + + + Extract key information from the document and return a JSON object with the following key steps: + 1. Carefully analyze the document text to identify the requested attributes + 2. Extract only information explicitly found in the document - never make up data + 3. Format all dates as MM/DD/YYYY and replace newlines with spaces + 4. For checkboxes, only include options with clear visual selection marks + 5. Use null for any fields not found in the document + 6. Ensure the output is properly formatted JSON with quoted keys and values + 7. Think step by step before finalizing your answer + + + + + Next, + 1. Agency name + 2. Advertiser name + 3. Gross total amount + 4. Net amount due + 5. All line items with: + - Description (time slot) + - Days (day of week encoding) + - Rate (cost) + - Start date + - End date + + Return the information in the specified JSON schema format. + temperature: 0 + model: us.anthropic.claude-sonnet-4-20250514-v1:0 + system_prompt: > + You are an expert at extracting structured information from FCC political + advertising invoices. + + Extract all requested fields accurately, paying special attention to: + + - Line item details including time slots, days, rates, and date ranges + + - Monetary amounts (preserve formatting with commas and decimals) + + - Date formats (typically MM/DD/YY) + + - Agency and advertiser names + + + For line items, ensure you capture all rows from any tables showing + broadcast schedules. + + Days of week are often encoded as 7-character strings where each position + represents a day. + agentic: + enabled: false + review_agent: false +evaluation: + enabled: true + methods: + - EXACT + - NUMERIC_EXACT + - LLM + llm_method: + temperature: '0.0' + top_k: '250' +ocr: + features: + - name: LAYOUT + - name: TABLES + backend: textract + image: + dpi: '150' + preprocessing: false +agents: + error_analyzer: + parameters: + time_range_hours_default: '24' + max_log_events: '100' + model_id: us.anthropic.claude-3-7-sonnet-20250219-v1:0 + system_prompt: |2- + + You are an intelligent error analysis agent for the GenAI IDP system. + + Use the analyze_errors tool to investigate issues. ALWAYS format your response with exactly these three sections in this order: + + ## Root Cause + Identify the specific underlying technical reason why the error occurred. Focus on the primary cause, not symptoms. + + ## Recommendations + Provide specific, actionable steps to resolve the issue. Limit to top three recommendations only. + +
+ Evidence + + Format log entries with their source information. For each log entry, show: + **Log Group:** + [full log_group name from tool response] + + **Log Stream:** + [full log_stream name from tool response] + ``` + [ERROR] timestamp message (from events data) + ``` + +
+ + FORMATTING RULES: + - Use the exact three-section structure above + - Make Evidence section collapsible using HTML details tags + - Extract log_group, log_stream, and events data from tool response + - Show complete log group and log stream names without truncation + - Present actual log messages from events array in code blocks + + RECOMMENDATION GUIDELINES: + For code-related issues or system bugs: + - Do not suggest code modifications + - Include error details, timestamps, and context + + For configuration-related issues: + - Direct users to UI configuration panel + - Specify exact configuration section and parameter names + + For operational issues: + - Provide immediate troubleshooting steps + - Include preventive measures + + TIME RANGE PARSING: + - recent/recently: 1 hour + - last week: 168 hours + - last day/yesterday: 24 hours + - No time specified: 24 hours (default) + + SPECIAL CASES: + If analysis_type is "document_not_found": explain document cannot be located, focus on verification steps and processing issues. + + DO NOT include code suggestions, technical summaries, or multiple paragraphs of explanation. Keep responses concise and actionable. + + IMPORTANT: Do not include any search quality reflections, search quality scores, or meta-analysis sections in your response. Only provide the three required sections: Root Cause, Recommendations, and Evidence. +summarization: + top_p: '0.9' + max_tokens: '2048' + top_k: '250' + task_prompt: |- + + + {DOCUMENT_TEXT} + + + + Analyze the provided document () and create a comprehensive summary. + + CRITICAL INSTRUCTION: You MUST return your response as valid JSON with the + EXACT structure shown at the end of these instructions. Do not include any + explanations, notes, or text outside of the JSON structure. + + Create a summary that captures the essential information from the document. + Your summary should: + + 1. Extract key information, main points, and important details + + 2. Maintain the original document's organizational structure where + appropriate + + 3. Preserve important facts, figures, dates, and entities + + 4. Reduce the length while retaining all critical information + + 5. Use markdown formatting for better readability (headings, lists, + emphasis, etc.) + + 6. Cite all relevant facts from the source document using inline citations + in the format [Cite-X, Page-Y] where X is a sequential citation number and Y + is the page number + + 7. Format citations as markdown links that reference the full citation list + at the bottom of the summary + Example: [[Cite-1, Page-3]](#cite-1-page-3) + + 8. At the end of the summary, include a "References" section that lists all + citations with their exact text from the source document in the format: + [Cite-X, Page-Y]: Exact text from the document + + Output Format: + + You MUST return ONLY valid JSON with the following structure and nothing + else: + + ```json + { + "summary": "A comprehensive summary in markdown format with inline citations linked to a references section at the bottom" + } + ``` + + Do not include any text, explanations, or notes outside of this JSON + structure. The JSON must be properly formatted and parseable. + temperature: '0.5' + model: us.amazon.nova-pro-v1:0 + system_prompt: 'do the task. ' + enabled: false +discovery: + output_format: + sample_json: |- + { + "document_class": "Form-1040", + "document_description": "Brief summary of the document", + "groups": [ + { + "name": "PersonalInformation", + "description": "Personal information of Tax payer", + "attributeType": "group", + "groupType": "normal", + "groupAttributes": [ + { + "name": "FirstName", + "dataType": "string", + "description": "First Name of Taxpayer" + }, + { + "name": "Age", + "dataType": "number", + "description": "Age of Taxpayer" + } + ] + } + ] + } + with_ground_truth: + top_p: '0.1' + temperature: '1.0' + user_prompt: >- + This image contains unstructured data. Analyze the data line by line using + the provided ground truth as reference. + {ground_truth_json} + Ground truth reference JSON has the fields we are interested in extracting + from the document/image. Use the ground truth to optimize field + extraction. Match field names, data types, and groupings from the + reference. Image may contain multiple pages, process all pages. Extract + all field names including those without values. Do not change the group + name and field name from ground truth in the extracted data json. Add + field_description field for every field which will contain instruction to + LLM to extract the field data from the image/document. Add data_type field + for every field. Add two fields document_class and document_description. + For document_class generate a short name based on the document content + like W4, I-9, Paystub. For document_description generate a description + about the document in less than 50 words. If the group repeats and follows + table format, add a special field group_type with value "Table" and + description field for the group. Do not extract the values. + model_id: us.amazon.nova-pro-v1:0 + system_prompt: >- + You are an expert in processing forms. Extracting data from images and + documents. Use provided ground truth data as reference to optimize field + extraction and ensure consistency with expected document structure and + field definitions. + max_tokens: '10000' + without_ground_truth: + top_p: '0.1' + temperature: '1.0' + user_prompt: >- + This image contains forms data. Analyze the form line by line. Image may + contains multiple pages, process all the pages. Form may contain multiple + name value pair in one line. Extract all the names in the form including + the name value pair which doesn't have value. Organize them into groups, + extract field_name, data_type and field description. Field_name should be + less than 60 characters, should not have space use '-' instead of space. + field_description is a brief description of the field and the location of + the field like box number or line number in the form and section of the + form. Field_name should be unique within the group. Add two fields + document_class and document_description. For document_class generate a + short name based on the document content like W4, I-9, Paystub. For + document_description generate a description about the document in less + than 50 words. Group the fields based on the section they are grouped in + the form. Group should have attributeType as "group". If the group + repeats, add an additional field groupType and set the value as "Table". + Do not extract the values. Return the extracted data in JSON format. + model_id: us.amazon.nova-pro-v1:0 + system_prompt: >- + You are an expert in processing forms. Extracting data from images and + documents. Analyze forms line by line to identify field names, data types, + and organizational structure. Focus on creating comprehensive blueprints + for document processing without extracting actual values. + max_tokens: '10000' +assessment: + validation_enabled: false + granular: + list_batch_size: '1' + simple_batch_size: '3' + max_workers: '4' + enabled: true + enabled: false diff --git a/config_library/pattern-2/fcc-invoices/fcc_invoice_stickler_schema.json b/config_library/pattern-2/fcc-invoices/fcc_invoice_stickler_schema.json new file mode 100644 index 000000000..1254e1478 --- /dev/null +++ b/config_library/pattern-2/fcc-invoices/fcc_invoice_stickler_schema.json @@ -0,0 +1,107 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "$id": "FCC-Invoice", + "type": "object", + "title": "FCC Invoice", + "description": "Federal Communications Commission (FCC) political advertising invoice showing broadcast time purchases, including line items with descriptions, dates, rates, and totals for political advertising campaigns.", + "x-aws-stickler-model-name": "FCCInvoice", + "x-aws-stickler-match-threshold": 0.7, + "properties": { + "agency": { + "type": "array", + "description": "The advertising agency or media buyer handling the political advertising purchase.", + "items": { + "type": "string" + }, + "x-aws-stickler-comparator": "FuzzyComparator", + "x-aws-stickler-threshold": 0.8, + "x-aws-stickler-weight": 2.0 + }, + "advertiser": { + "type": "array", + "description": "The political advertiser or campaign purchasing the broadcast time.", + "items": { + "type": "string" + }, + "x-aws-stickler-comparator": "FuzzyComparator", + "x-aws-stickler-threshold": 0.8, + "x-aws-stickler-weight": 2.0 + }, + "gross_total": { + "type": "array", + "description": "The total gross amount for all line items before any discounts or adjustments.", + "items": { + "type": "string" + }, + "x-aws-stickler-comparator": "ExactComparator", + "x-aws-stickler-threshold": 1.0, + "x-aws-stickler-weight": 3.0 + }, + "net_amount_due": { + "type": "array", + "description": "The final net amount due after any discounts or adjustments have been applied.", + "items": { + "type": "string" + }, + "x-aws-stickler-comparator": "ExactComparator", + "x-aws-stickler-threshold": 1.0, + "x-aws-stickler-weight": 3.0 + }, + "line_item__description": { + "type": "array", + "description": "Each line item description is usually a specification in industry short-hand of when the advertisement should run. Examples include \"M-F 6p-6:30p\", \"M-F 11a-12p\"", + "items": { + "type": "string" + }, + "x-aws-stickler-comparator": "LevenshteinComparator", + "x-aws-stickler-threshold": 0.7, + "x-aws-stickler-weight": 1.5 + }, + "line_item__days": { + "type": "array", + "description": "Each list item will be a string with shorthand representing the days of the week the ad should run. I.e. \"MTWTF--\"", + "items": { + "type": "string" + }, + "x-aws-stickler-comparator": "ExactComparator", + "x-aws-stickler-threshold": 1.0, + "x-aws-stickler-weight": 1.0 + }, + "line_item__rate": { + "type": "array", + "description": "Each entry is a unitless float, i.e. \"300.00\", or \"1200.00\"", + "items": { + "type": "string" + }, + "x-aws-stickler-comparator": "ExactComparator", + "x-aws-stickler-threshold": 1.0, + "x-aws-stickler-weight": 2.0 + }, + "line_item__start_date": { + "type": "array", + "description": "Each start date, formatted as MM/DD/YY. i.e. \"11/07/21\"", + "items": { + "type": "string" + }, + "x-aws-stickler-comparator": "ExactComparator", + "x-aws-stickler-threshold": 1.0, + "x-aws-stickler-weight": 2.0 + }, + "line_item__end_date": { + "type": "array", + "description": "Each end date, formatted as MM/DD/YY. i.e. \"11/07/21\"", + "items": { + "type": "string" + }, + "x-aws-stickler-comparator": "ExactComparator", + "x-aws-stickler-threshold": 1.0, + "x-aws-stickler-weight": 2.0 + } + }, + "required": [ + "agency", + "advertiser", + "gross_total", + "net_amount_due" + ] +} diff --git a/config_library/pattern-2/fcc-invoices/sample_labels_3.csv b/config_library/pattern-2/fcc-invoices/sample_labels_3.csv new file mode 100644 index 000000000..3fd6fa13d --- /dev/null +++ b/config_library/pattern-2/fcc-invoices/sample_labels_3.csv @@ -0,0 +1,2300 @@ +Unnamed: 0.2,Unnamed: 0,Unnamed: 0.1,document_path,image_files,labels,ocr,original_filename,text,doc_id,refactored_labels +0,0,0,fcc_invoices/files/2dfd91c64c66a5a92b4eb1c46e26f87f.pdf,"[""fcc_invoices/images/2dfd91c64c66a5a92b4eb1c46e26f87f/page_0.png"", ""fcc_invoices/images/2dfd91c64c66a5a92b4eb1c46e26f87f/page_1.png"", ""fcc_invoices/images/2dfd91c64c66a5a92b4eb1c46e26f87f/page_2.png"", ""fcc_invoices/images/2dfd91c64c66a5a92b4eb1c46e26f87f/page_3.png"", ""fcc_invoices/images/2dfd91c64c66a5a92b4eb1c46e26f87f/page_4.png"", ""fcc_invoices/images/2dfd91c64c66a5a92b4eb1c46e26f87f/page_5.png""]","[{""label"": ""Line Item - Description"", ""start"": 5069, ""end"": 5081, ""text"": ""IA* 11a -12p""}, {""label"": ""Line Item - Description"", ""start"": 5082, ""end"": 5095, ""text"": ""M -F 11a -12p""}, {""label"": ""Line Item - Days"", ""start"": 5175, ""end"": 5182, ""text"": ""MTWTF--""}, {""label"": ""Line Item - Rate"", ""start"": 5186, ""end"": 5192, ""text"": ""100.00""}, {""label"": ""Line Item - Start Date"", ""start"": 5206, ""end"": 5214, ""text"": ""11/01/16""}, {""label"": ""Line Item - End Date"", ""start"": 5215, ""end"": 5223, ""text"": ""11/07/16""}, {""label"": ""Line Item - Description"", ""start"": 5224, ""end"": 5236, ""text"": ""M -F 12n -1p""}, {""label"": ""Line Item - Rate"", ""start"": 5268, ""end"": 5274, ""text"": ""150.00""}, {""label"": ""Line Item - Description"", ""start"": 5296, ""end"": 5308, ""text"": ""M -F 12n -1p""}, {""label"": ""Line Item - Days"", ""start"": 5387, ""end"": 5394, ""text"": ""MTWTF--""}, {""label"": ""Line Item - Rate"", ""start"": 5398, ""end"": 5404, ""text"": ""150.00""}, {""label"": ""Line Item - Start Date"", ""start"": 5418, ""end"": 5426, ""text"": ""11/01/16""}, {""label"": ""Line Item - End Date"", ""start"": 5427, ""end"": 5435, ""text"": ""11/07/16""}, {""label"": ""Line Item - Description"", ""start"": 5436, ""end"": 5449, ""text"": ""M -F 4p -430p""}, {""label"": ""Line Item - Rate"", ""start"": 5482, ""end"": 5488, ""text"": ""200.00""}, {""label"": ""Line Item - Description"", ""start"": 5510, ""end"": 5523, ""text"": ""M -F 4p -430p""}, {""label"": ""Line Item - Days"", ""start"": 5603, ""end"": 5610, ""text"": ""MTWTF--""}, {""label"": ""Line Item - Rate"", ""start"": 5614, ""end"": 5620, ""text"": ""200.00""}, {""label"": ""Line Item - Start Date"", ""start"": 5634, ""end"": 5642, ""text"": ""11/01/16""}, {""label"": ""Line Item - End Date"", ""start"": 5643, ""end"": 5651, ""text"": ""11/07/16""}, {""label"": ""Line Item - Description"", ""start"": 5652, ""end"": 5665, ""text"": ""M -F 430p -5p""}, {""label"": ""Line Item - Rate"", ""start"": 5698, ""end"": 5704, ""text"": ""300.00""}, {""label"": ""Line Item - Description"", ""start"": 5726, ""end"": 5739, ""text"": ""M -F 430p -5p""}, {""label"": ""Line Item - Days"", ""start"": 5818, ""end"": 5825, ""text"": ""MTWTF--""}, {""label"": ""Line Item - Rate"", ""start"": 5829, ""end"": 5835, ""text"": ""300.00""}, {""label"": ""Line Item - Start Date"", ""start"": 5849, ""end"": 5857, ""text"": ""11/01/16""}, {""label"": ""Line Item - End Date"", ""start"": 5858, ""end"": 5866, ""text"": ""11/07/16""}, {""label"": ""Line Item - Description"", ""start"": 5867, ""end"": 5881, ""text"": ""M -F 5p -5:30p""}, {""label"": ""Line Item - Rate"", ""start"": 5915, ""end"": 5921, ""text"": ""500.00""}, {""label"": ""Line Item - Description"", ""start"": 5945, ""end"": 5959, ""text"": ""M -F 5p -5:30p""}, {""label"": ""Line Item - Days"", ""start"": 6039, ""end"": 6046, ""text"": ""MTWTF--""}, {""label"": ""Line Item - Rate"", ""start"": 6050, ""end"": 6056, ""text"": ""500.00""}, {""label"": ""Line Item - Start Date"", ""start"": 6070, ""end"": 6078, ""text"": ""11/01/16""}, {""label"": ""Line Item - End Date"", ""start"": 6079, ""end"": 6087, ""text"": ""11/07/16""}, {""label"": ""Line Item - Description"", ""start"": 6088, ""end"": 6099, ""text"": ""M -F 5a -6a""}, {""label"": ""Line Item - Days"", ""start"": 6115, ""end"": 6122, ""text"": ""MTWTF--""}, {""label"": ""Line Item - Rate"", ""start"": 6130, ""end"": 6135, ""text"": ""60.00""}, {""label"": ""Line Item - Description"", ""start"": 6158, ""end"": 6169, ""text"": ""M -F 5a -6a""}, {""label"": ""Line Item - Days"", ""start"": 6249, ""end"": 6256, ""text"": ""MTWTF--""}, {""label"": ""Line Item - Rate"", ""start"": 6260, ""end"": 6265, ""text"": ""60.00""}, {""label"": ""Line Item - Start Date"", ""start"": 6279, ""end"": 6287, ""text"": ""11/01/16""}, {""label"": ""Line Item - End Date"", ""start"": 6288, ""end"": 6296, ""text"": ""11/07/16""}, {""label"": ""Line Item - Description"", ""start"": 6297, ""end"": 6310, ""text"": ""M -F 6p -630p""}, {""label"": ""Line Item - Days"", ""start"": 6329, ""end"": 6336, ""text"": ""MTWTF--""}, {""label"": ""Line Item - Rate"", ""start"": 6344, ""end"": 6350, ""text"": ""900.00""}, {""label"": ""Line Item - Description"", ""start"": 6374, ""end"": 6388, ""text"": ""M -F 6p -6:30p""}, {""label"": ""Line Item - Days"", ""start"": 6468, ""end"": 6475, ""text"": ""MTWTF--""}, {""label"": ""Line Item - Rate"", ""start"": 6479, ""end"": 6485, ""text"": ""900.00""}, {""label"": ""Line Item - Start Date"", ""start"": 6500, ""end"": 6508, ""text"": ""11/01/16""}, {""label"": ""Line Item - End Date"", ""start"": 6509, ""end"": 6517, ""text"": ""11/07/16""}, {""label"": ""Line Item - Description"", ""start"": 6518, ""end"": 6529, ""text"": ""M -F 6a -7a""}, {""label"": ""Line Item - Days"", ""start"": 6545, ""end"": 6552, ""text"": ""MTWTF--""}, {""label"": ""Line Item - Rate"", ""start"": 6560, ""end"": 6566, ""text"": ""300.00""}, {""label"": ""Line Item - Description"", ""start"": 6589, ""end"": 6600, ""text"": ""M -F 6a -7a""}, {""label"": ""Line Item - Days"", ""start"": 6680, ""end"": 6687, ""text"": ""MTWTF--""}, {""label"": ""Line Item - Rate"", ""start"": 6691, ""end"": 6697, ""text"": ""300.00""}, {""label"": ""Line Item - Start Date"", ""start"": 6712, ""end"": 6720, ""text"": ""11/01/16""}, {""label"": ""Line Item - End Date"", ""start"": 6721, ""end"": 6729, ""text"": ""11/07/16""}, {""label"": ""Line Item - Description"", ""start"": 6730, ""end"": 6743, ""text"": ""M -F 630p -7p""}, {""label"": ""Line Item - Days"", ""start"": 6761, ""end"": 6768, ""text"": ""MTWTF--""}, {""label"": ""Line Item - Rate"", ""start"": 6776, ""end"": 6784, ""text"": ""1,200.00""}, {""label"": ""Line Item - Description"", ""start"": 6808, ""end"": 6821, ""text"": ""M -F 630p -7p""}, {""label"": ""Line Item - Days"", ""start"": 6901, ""end"": 6908, ""text"": ""MTWTF--""}, {""label"": ""Line Item - Rate"", ""start"": 6912, ""end"": 6920, ""text"": ""1,200.00""}, {""label"": ""Line Item - Start Date"", ""start"": 6935, ""end"": 6943, ""text"": ""11/01/16""}, {""label"": ""Line Item - End Date"", ""start"": 6944, ""end"": 6952, ""text"": ""11/07/16""}, {""label"": ""Line Item - Description"", ""start"": 6953, ""end"": 6964, ""text"": ""M -F 7a -9a""}, {""label"": ""Line Item - Days"", ""start"": 6980, ""end"": 6987, ""text"": ""MTWTF--""}, {""label"": ""Line Item - Rate"", ""start"": 6995, ""end"": 7001, ""text"": ""500.00""}, {""label"": ""Line Item - Description"", ""start"": 7026, ""end"": 7037, ""text"": ""M -F 7a -9a""}, {""label"": ""Line Item - Days"", ""start"": 7117, ""end"": 7124, ""text"": ""MTWTF--""}, {""label"": ""Line Item - Rate"", ""start"": 7128, ""end"": 7134, ""text"": ""500.00""}, {""label"": ""Line Item - Start Date"", ""start"": 7149, ""end"": 7157, ""text"": ""11/01/16""}, {""label"": ""Line Item - End Date"", ""start"": 7158, ""end"": 7166, ""text"": ""11/07/16""}, {""label"": ""Line Item - Description"", ""start"": 7167, ""end"": 7179, ""text"": ""M -F 9a -10a""}, {""label"": ""Line Item - Days"", ""start"": 7196, ""end"": 7203, ""text"": ""MTWTF--""}, {""label"": ""Line Item - Rate"", ""start"": 7211, ""end"": 7217, ""text"": ""200.00""}, {""label"": ""Line Item - Description"", ""start"": 7240, ""end"": 7252, ""text"": ""M -F 9a -10a""}, {""label"": ""Line Item - Days"", ""start"": 7330, ""end"": 7337, ""text"": ""MTWTF--""}, {""label"": ""Line Item - Rate"", ""start"": 7341, ""end"": 7347, ""text"": ""200.00""}, {""label"": ""Line Item - Start Date"", ""start"": 7364, ""end"": 7372, ""text"": ""11/04/16""}, {""label"": ""Line Item - End Date"", ""start"": 7373, ""end"": 7381, ""text"": ""11/04/16""}, {""label"": ""Line Item - Description"", ""start"": 7382, ""end"": 7391, ""text"": ""Fri 9-10P""}, {""label"": ""Line Item - Rate"", ""start"": 7406, ""end"": 7414, ""text"": ""2,000.00""}, {""label"": ""Line Item - Description"", ""start"": 7438, ""end"": 7447, ""text"": ""Fri 9-10P""}, {""label"": ""Line Item - Days"", ""start"": 7527, ""end"": 7534, ""text"": ""----1--""}, {""label"": ""Line Item - Rate"", ""start"": 7538, ""end"": 7546, ""text"": ""2,000.00""}, {""label"": ""Line Item - Days"", ""start"": 7667, ""end"": 7677, ""text"": ""---- F----""}, {""label"": ""Line Item - Days"", ""start"": 7794, ""end"": 7804, ""text"": ""---- F----""}, {""label"": ""Line Item - Rate"", ""start"": 7810, ""end"": 7818, ""text"": ""2,000.00""}, {""label"": ""Line Item - Start Date"", ""start"": 7856, ""end"": 7864, ""text"": ""11/05/16""}, {""label"": ""Line Item - End Date"", ""start"": 7865, ""end"": 7873, ""text"": ""11/05/16""}, {""label"": ""Line Item - Description"", ""start"": 7874, ""end"": 7886, ""text"": ""Sat 5p -530p""}, {""label"": ""Line Item - Days"", ""start"": 7903, ""end"": 7910, ""text"": ""---- 1-""}, {""label"": ""Line Item - Rate"", ""start"": 7918, ""end"": 7924, ""text"": ""150.00""}, {""label"": ""Line Item - Description"", ""start"": 7937, ""end"": 7949, ""text"": ""Sat 5p -530p""}, {""label"": ""Line Item - Days"", ""start"": 8030, ""end"": 8037, ""text"": ""---- 1-""}, {""label"": ""Line Item - Rate"", ""start"": 8042, ""end"": 8048, ""text"": ""150.00""}, {""label"": ""Line Item - Start Date"", ""start"": 8072, ""end"": 8080, ""text"": ""11/05/16""}, {""label"": ""Line Item - End Date"", ""start"": 8081, ""end"": 8089, ""text"": ""11/05/16""}, {""label"": ""Line Item - Days"", ""start"": 8105, ""end"": 8112, ""text"": ""---- 1-""}, {""label"": ""Line Item - Rate"", ""start"": 8120, ""end"": 8126, ""text"": ""500.00""}, {""label"": ""Line Item - Description"", ""start"": 11801, ""end"": 11811, ""text"": ""M -h oa-ba""}, {""label"": ""Line Item - Description"", ""start"": 11812, ""end"": 11823, ""text"": ""M -F 5a -6a""}, {""label"": ""Line Item - Start Date"", ""start"": 11886, ""end"": 11894, ""text"": ""11/08/16""}, {""label"": ""Line Item - End Date"", ""start"": 11895, ""end"": 11903, ""text"": ""11/14/16""}, {""label"": ""Line Item - Days"", ""start"": 11904, ""end"": 11906, ""text"": ""-1""}, {""label"": ""Line Item - Rate"", ""start"": 11915, ""end"": 11920, ""text"": ""60.00""}, {""label"": ""Line Item - Description"", ""start"": 11953, ""end"": 11965, ""text"": ""M -F 9a -10a""}, {""label"": ""Line Item - Days"", ""start"": 11982, ""end"": 11984, ""text"": ""-1""}, {""label"": ""Line Item - Rate"", ""start"": 11997, ""end"": 12003, ""text"": ""200.00""}, {""label"": ""Line Item - Description"", ""start"": 12016, ""end"": 12028, ""text"": ""M -F 9a -10a""}, {""label"": ""Line Item - Start Date"", ""start"": 12090, ""end"": 12098, ""text"": ""11/08/16""}, {""label"": ""Line Item - End Date"", ""start"": 12099, ""end"": 12107, ""text"": ""11/14/16""}, {""label"": ""Line Item - Days"", ""start"": 12108, ""end"": 12110, ""text"": ""-1""}, {""label"": ""Line Item - Rate"", ""start"": 12119, ""end"": 12125, ""text"": ""200.00""}, {""label"": ""Line Item - Description"", ""start"": 8448, ""end"": 8459, ""text"": ""bat 6.30-/p""}, {""label"": ""Line Item - Description"", ""start"": 8460, ""end"": 8470, ""text"": ""Sat 630-7p""}, {""label"": ""Line Item - Days"", ""start"": 8555, ""end"": 8556, ""text"": ""1""}, {""label"": ""Line Item - Rate"", ""start"": 8561, ""end"": 8567, ""text"": ""500.00""}, {""label"": ""Line Item - Start Date"", ""start"": 8582, ""end"": 8590, ""text"": ""11/05/16""}, {""label"": ""Line Item - End Date"", ""start"": 8591, ""end"": 8599, ""text"": ""11/05/16""}, {""label"": ""Line Item - Description"", ""start"": 8600, ""end"": 8608, ""text"": ""Sat 7-9A""}, {""label"": ""Line Item - Rate"", ""start"": 8636, ""end"": 8642, ""text"": ""300.00""}, {""label"": ""Line Item - Description"", ""start"": 8665, ""end"": 8673, ""text"": ""Sat 7-9A""}, {""label"": ""Line Item - Days"", ""start"": 8757, ""end"": 8758, ""text"": ""1""}, {""label"": ""Line Item - Rate"", ""start"": 8763, ""end"": 8769, ""text"": ""300.00""}, {""label"": ""Line Item - Start Date"", ""start"": 8784, ""end"": 8792, ""text"": ""11/05/16""}, {""label"": ""Line Item - End Date"", ""start"": 8793, ""end"": 8801, ""text"": ""11/05/16""}, {""label"": ""Line Item - Description"", ""start"": 8802, ""end"": 8811, ""text"": ""Sat 9-10P""}, {""label"": ""Line Item - Rate"", ""start"": 8840, ""end"": 8846, ""text"": ""220.00""}, {""label"": ""Line Item - Description"", ""start"": 8869, ""end"": 8878, ""text"": ""Sat 9-10p""}, {""label"": ""Line Item - Days"", ""start"": 8962, ""end"": 8963, ""text"": ""1""}, {""label"": ""Line Item - Rate"", ""start"": 8968, ""end"": 8974, ""text"": ""220.00""}, {""label"": ""Line Item - Start Date"", ""start"": 8989, ""end"": 8997, ""text"": ""11/06/16""}, {""label"": ""Line Item - End Date"", ""start"": 8998, ""end"": 9006, ""text"": ""11/06/16""}, {""label"": ""Line Item - Description"", ""start"": 9007, ""end"": 9023, ""text"": ""Sun 1105p -1205a""}, {""label"": ""Line Item - Rate"", ""start"": 9057, ""end"": 9063, ""text"": ""100.00""}, {""label"": ""Line Item - Description"", ""start"": 9086, ""end"": 9101, ""text"": ""Su 1105p -1205a""}, {""label"": ""Line Item - Days"", ""start"": 9185, ""end"": 9186, ""text"": ""1""}, {""label"": ""Line Item - Rate"", ""start"": 9190, ""end"": 9196, ""text"": ""100.00""}, {""label"": ""Line Item - Start Date"", ""start"": 9211, ""end"": 9219, ""text"": ""11/06/16""}, {""label"": ""Line Item - End Date"", ""start"": 9220, ""end"": 9228, ""text"": ""11/06/16""}, {""label"": ""Line Item - Description"", ""start"": 9229, ""end"": 9242, ""text"": ""Sun 5p -5:30p""}, {""label"": ""Line Item - Rate"", ""start"": 9273, ""end"": 9279, ""text"": ""300.00""}, {""label"": ""Line Item - Description"", ""start"": 9302, ""end"": 9314, ""text"": ""Sun 5p -530p""}, {""label"": ""Line Item - Days"", ""start"": 9398, ""end"": 9399, ""text"": ""1""}, {""label"": ""Line Item - Rate"", ""start"": 9403, ""end"": 9409, ""text"": ""300.00""}, {""label"": ""Line Item - Start Date"", ""start"": 9424, ""end"": 9432, ""text"": ""11/06/16""}, {""label"": ""Line Item - End Date"", ""start"": 9433, ""end"": 9441, ""text"": ""11/06/16""}, {""label"": ""Line Item - Description"", ""start"": 9442, ""end"": 9450, ""text"": ""Sun 8-9a""}, {""label"": ""Line Item - Rate"", ""start"": 9477, ""end"": 9483, ""text"": ""300.00""}, {""label"": ""Line Item - Description"", ""start"": 9506, ""end"": 9514, ""text"": ""Sun 8-9a""}, {""label"": ""Line Item - Days"", ""start"": 9598, ""end"": 9599, ""text"": ""1""}, {""label"": ""Line Item - Rate"", ""start"": 9603, ""end"": 9609, ""text"": ""300.00""}, {""label"": ""Line Item - Start Date"", ""start"": 9626, ""end"": 9634, ""text"": ""11/03/16""}, {""label"": ""Line Item - End Date"", ""start"": 9635, ""end"": 9643, ""text"": ""11/03/16""}, {""label"": ""Line Item - Description"", ""start"": 9644, ""end"": 9653, ""text"": ""Thur 7-8p""}, {""label"": ""Line Item - Rate"", ""start"": 9680, ""end"": 9686, ""text"": ""800.00""}, {""label"": ""Line Item - Description"", ""start"": 9709, ""end"": 9718, ""text"": ""Thur 7-8p""}, {""label"": ""Line Item - Days"", ""start"": 9801, ""end"": 9802, ""text"": ""1""}, {""label"": ""Line Item - Rate"", ""start"": 9809, ""end"": 9815, ""text"": ""800.00""}, {""label"": ""Line Item - Rate"", ""start"": 10061, ""end"": 10067, ""text"": ""800.00""}, {""label"": ""Line Item - Start Date"", ""start"": 10105, ""end"": 10113, ""text"": ""11/01/16""}, {""label"": ""Line Item - End Date"", ""start"": 10114, ""end"": 10122, ""text"": ""11/07/16""}, {""label"": ""Line Item - Description"", ""start"": 10123, ""end"": 10139, ""text"": ""M -Su 10p -1035p""}, {""label"": ""Line Item - Rate"", ""start"": 10175, ""end"": 10183, ""text"": ""1,000.00""}, {""label"": ""Line Item - Description"", ""start"": 10195, ""end"": 10211, ""text"": ""M -Su 10p -1035p""}, {""label"": ""Line Item - Days"", ""start"": 10291, ""end"": 10298, ""text"": ""MTWTFSS""}, {""label"": ""Line Item - Rate"", ""start"": 10302, ""end"": 10310, ""text"": ""1,000.00""}, {""label"": ""Line Item - Start Date"", ""start"": 10325, ""end"": 10333, ""text"": ""11/08/16""}, {""label"": ""Line Item - End Date"", ""start"": 10334, ""end"": 10342, ""text"": ""11/08/16""}, {""label"": ""Line Item - Description"", ""start"": 10343, ""end"": 10356, ""text"": ""M -F 11a -12p""}, {""label"": ""Line Item - Rate"", ""start"": 10389, ""end"": 10395, ""text"": ""100.00""}, {""label"": ""Line Item - Description"", ""start"": 10408, ""end"": 10421, ""text"": ""M -F 11a -12p""}, {""label"": ""Line Item - Days"", ""start"": 10502, ""end"": 10503, ""text"": ""1""}, {""label"": ""Line Item - Rate"", ""start"": 10512, ""end"": 10518, ""text"": ""100.00""}, {""label"": ""Line Item - Start Date"", ""start"": 10533, ""end"": 10541, ""text"": ""11/08/16""}, {""label"": ""Line Item - End Date"", ""start"": 10542, ""end"": 10550, ""text"": ""11/08/16""}, {""label"": ""Line Item - Description"", ""start"": 10551, ""end"": 10563, ""text"": ""M -F 12n -1p""}, {""label"": ""Line Item - Rate"", ""start"": 10595, ""end"": 10601, ""text"": ""150.00""}, {""label"": ""Line Item - Description"", ""start"": 10614, ""end"": 10626, ""text"": ""M -F 12n -1p""}, {""label"": ""Line Item - Days"", ""start"": 10707, ""end"": 10708, ""text"": ""1""}, {""label"": ""Line Item - Rate"", ""start"": 10717, ""end"": 10723, ""text"": ""150.00""}, {""label"": ""Line Item - Start Date"", ""start"": 10738, ""end"": 10746, ""text"": ""11/08/16""}, {""label"": ""Line Item - End Date"", ""start"": 10747, ""end"": 10755, ""text"": ""11/08/16""}, {""label"": ""Line Item - Description"", ""start"": 10756, ""end"": 10769, ""text"": ""M -F 4p -430p""}, {""label"": ""Line Item - Rate"", ""start"": 10802, ""end"": 10808, ""text"": ""200.00""}, {""label"": ""Line Item - Description"", ""start"": 10821, ""end"": 10834, ""text"": ""M -F 4p -430p""}, {""label"": ""Line Item - Days"", ""start"": 10915, ""end"": 10916, ""text"": ""1""}, {""label"": ""Line Item - Rate"", ""start"": 10925, ""end"": 10931, ""text"": ""200.00""}, {""label"": ""Line Item - Start Date"", ""start"": 10946, ""end"": 10954, ""text"": ""11/08/16""}, {""label"": ""Line Item - End Date"", ""start"": 10955, ""end"": 10963, ""text"": ""11/08/16""}, {""label"": ""Line Item - Description"", ""start"": 10964, ""end"": 10977, ""text"": ""M -F 430p -5p""}, {""label"": ""Line Item - Rate"", ""start"": 11010, ""end"": 11016, ""text"": ""300.00""}, {""label"": ""Line Item - Description"", ""start"": 11029, ""end"": 11042, ""text"": ""M -F 430p -5p""}, {""label"": ""Line Item - Days"", ""start"": 11123, ""end"": 11124, ""text"": ""1""}, {""label"": ""Line Item - Rate"", ""start"": 11133, ""end"": 11139, ""text"": ""300.00""}, {""label"": ""Line Item - Start Date"", ""start"": 11154, ""end"": 11162, ""text"": ""11/08/16""}, {""label"": ""Line Item - End Date"", ""start"": 11163, ""end"": 11171, ""text"": ""11/08/16""}, {""label"": ""Line Item - Description"", ""start"": 11172, ""end"": 11186, ""text"": ""M -F 5p -5:30p""}, {""label"": ""Line Item - Rate"", ""start"": 11220, ""end"": 11226, ""text"": ""500.00""}, {""label"": ""Line Item - Description"", ""start"": 11239, ""end"": 11253, ""text"": ""M -F 5p -5:30p""}, {""label"": ""Line Item - Days"", ""start"": 11334, ""end"": 11335, ""text"": ""1""}, {""label"": ""Line Item - Rate"", ""start"": 11344, ""end"": 11350, ""text"": ""500.00""}, {""label"": ""Line Item - Start Date"", ""start"": 11365, ""end"": 11373, ""text"": ""11/08/16""}, {""label"": ""Line Item - End Date"", ""start"": 11374, ""end"": 11382, ""text"": ""11/08/16""}, {""label"": ""Line Item - Rate"", ""start"": 11413, ""end"": 11418, ""text"": ""60.00""}, {""label"": ""Agency"", ""start"": 3178, ""end"": 3198, ""text"": ""Great American Media""}, {""label"": ""Advertiser"", ""start"": 3240, ""end"": 3251, ""text"": ""ISS/DSCC IE""}, {""label"": ""Line Item - Start Date"", ""start"": 3318, ""end"": 3326, ""text"": ""10/31/16""}, {""label"": ""Line Item - End Date"", ""start"": 3328, ""end"": 3336, ""text"": ""11/08/16""}, {""label"": ""Gross Total"", ""start"": 3382, ""end"": 3391, ""text"": ""17,410.00""}, {""label"": ""Net Amount Due"", ""start"": 3665, ""end"": 3674, ""text"": ""14,798.50""}, {""label"": ""Line Item - Start Date"", ""start"": 4253, ""end"": 4261, ""text"": ""11/01/16""}, {""label"": ""Line Item - End Date"", ""start"": 4262, ""end"": 4270, ""text"": ""11/07/16""}, {""label"": ""Line Item - Description"", ""start"": 4271, ""end"": 4282, ""text"": ""M -F 1p -2p""}, {""label"": ""Line Item - Days"", ""start"": 4298, ""end"": 4305, ""text"": ""MTWTF--""}, {""label"": ""Line Item - Description"", ""start"": 4332, ""end"": 4343, ""text"": ""M -F 1p -2p""}, {""label"": ""Line Item - Days"", ""start"": 4423, ""end"": 4430, ""text"": ""MTWTF--""}, {""label"": ""Line Item - Start Date"", ""start"": 4454, ""end"": 4462, ""text"": ""11/01/16""}, {""label"": ""Line Item - End Date"", ""start"": 4463, ""end"": 4471, ""text"": ""11/07/16""}, {""label"": ""Line Item - Description"", ""start"": 4472, ""end"": 4487, ""text"": ""M -F 1035-1135P""}, {""label"": ""Line Item - Days"", ""start"": 4509, ""end"": 4516, ""text"": ""MTWTF--""}, {""label"": ""Line Item - Description"", ""start"": 4542, ""end"": 4559, ""text"": ""M -F 1035p -1135p""}, {""label"": ""Line Item - Days"", ""start"": 4637, ""end"": 4644, ""text"": ""MTWTF--""}, {""label"": ""Line Item - Start Date"", ""start"": 4668, ""end"": 4676, ""text"": ""11/01/16""}, {""label"": ""Line Item - End Date"", ""start"": 4677, ""end"": 4685, ""text"": ""11/07/16""}, {""label"": ""Line Item - Days"", ""start"": 4703, ""end"": 4710, ""text"": ""MTWTF--""}]",fcc_invoices/ocr/2dfd91c64c66a5a92b4eb1c46e26f87f.json.gz,01234878-30e1-46e6-910f-08f09ce50c4d.pdf,"REP HEADLINE# 8250799 TRF# 163314 +$$$ MOD# 0: UNAPPROVED REV #1 $$$ + +ADV # ADV. NAME MSS/DSCC IE + +REP: TEL# 610-293-4100 FAX# 610-225-1191 +CREDIT ADVISORY: AGENCY CREDIT RISK Ell +ORDER WORKSHEET HARRIS REPORT FROM REP SEP27/16 11.22 +**CHANGES** *** WAND -TV *** + +REP.# OFF.# + +AGY # ____ AGY. NAME GREAT AMERICAN MEDIA BUYER NAME THOMAS PINO + +SALESMAN # + +3050 K STREET NW - SUITE 100 SALES PRSN PH- KRISTEN WASKIE + +WASHINGTON, DC 20007 + +ORDER # CONTRACT # 8250799 + +CLASS: NATL. LOCAL REGIONAL + +PRDCT + +DSCC IE + +EST#4789 COMMENTS: (LINE, ORDER, INVOICE) + +FLIGHT DATES NOV1/16 NOV8/16 WK -2 + +CITY TAX STATE TAX + +REP: + +MAKEGOOD M2 -M3 +N/A'S LINES 14,25/ADD LINES 35-36 +TOTAL $17410 +PLEASE CONFIRM THANKS +JAMIE + +CO-OP BILLING NEEDED DATE + +SEP27/16 11.22 + +STA: + +CON CM ***** THIS IS A CASH IN ADVANCE SCHEDULE ***** +DEMOCRATIC SENATORIAL CAMPAIGN COMMITTEE INDEPENDENT EXPENDITURE + +:LINE#:REP :CD: TIME PERIOD : LGTH : SEC : RATE : START : END :SPTS: WEEK : DAYS :TOTL: +:LINE#: : : DATE DATE : /WK: INVT : :SPTS: + +AGENCY ADVERTISER CODE = 49 AGENCY EST# = 4789 +AGENCY PRODUCT CODE = 53 + +14 S 900P -1000P 30 $2,000.00 11/4 11/4 0 FRI 0 + +PROGRAM : DATELINE +CON COM1: DATELINE + +35 A 800P -1000P + +30 $2,000.00 11/4 11/4 1 FRI 1 + +PROGRAM : DATELINE +ORD COM1: M2 PLEASE CORRECT TP FOR DATELINE. +THIS IS A MAKE -GOOD FOR NOV4 ON LINE -14 FOR 1 SPOT/WK +REP HEADLINE# 8250799 TRF# 163314 +$$$ MOD# 0: UNAPPROVED REV #1 $$$ + +REP: TEL# 610-293-4100 FAX# 610-225-1191 +CREDIT ADVISORY: AGENCY CREDIT RISK +ORDER WORKSHEET HARRIS REPORT FROM REP SEP27/16 11.22 +**CHANGES** *** WAND -TV *** + +:LINE#:REP :CD: TIME PERIOD : LGTH SEC : RATE : START : END :SPTS: WEEK : DAYS :TOTL: +:LINE#: : DATE : DATE : /WK: INVT : :SPTS: + +25 S 700P -800P 30 + +PROGRAM : YOU,ME,APOCLYPSE +CON COM1: YOU,ME,APOCLYPSE + +36 A 700P -800P 30 + +$800.00 11/3 + +11/3 0 THU 0 + +$800.00 11/3 11/3 1 THU + +PROGRAM : SUPERSTORE/GOOD PLACE +ORD COM1: M3 PLEASE CORRECT PROGRAM NAME +THIS IS A MAKE -GOOD FOR NOV3 ON LINE -25 FOR 1 SPOT/WK +STATION MAKEGOOD OFFERS: +M2 OK'D BUY#14 MISSED:FRI/900P-1000P NOV4 30S $2,000.00 (SEP27/16) +OFFER:FRI/800P-1000P NOV4 30S $2,000.00 PLS ADVISE. +CMT:M2 PLEASE CORRECT TP FOR DATELINE. + +M3 OK'D BUY#25 MISSED:THU/700P-800P NOV3 30S $800.00 (SEP27/16) +OFFER:THU/700P-800P NOV3 30S $800.00 PLS ADVISE. +CMT:M3 PLEASE CORRECT PROGRAM NAME + +NOV/16 17410.00 + +MARKET TOTALS $56,161 + +SVC- NSI +DEMOS- RA35+* + +CONTRACT TOTAL +TOTAL SPOTS + +WAND 31% WCIA 30% WFHL 0% WRSP 17% WCIX 2% WBUI 3% WICS 17% + +17410.00 +48 + +MOD CODE A -ADD B -BUY TYPE C -CANCELLED DE -DELETE E-EFF DATES L -LENGTH M-MAKEGOOD N -PROGRAM NAME + +P-CLASS,PLAN,SECT Q -PAID PGM R -RATE S -SPOTS PER WEEK T -TIME X -LATE Y -DAYS Z -COMMENTS *-MULTIPLE +Print Date 09/27/16 11:21:24 Page 1 of 4 + +ORDER + +Orders + +Agency + +Order! Rev: + +163314 + +WAND40 + +Alt Order #: + +08250799 + +Product Desc: + +DSCC IE + +Estimate: + +4789 + +NewsCenter 17 + +Flight Dates: 11/01/16 - 11/08/16 Primary AE: + +Telerep Philadelphia + +Original Date / Rev: 09/27/16 / 09/27/16 Sales Office: + +PHIL + +Order Type: GENERAL Sales Region: + +NAT + +Name: + +Buying Contact: + +Billing Contact: + +Advertiser Name: + +Demographic: + +A35+ + +Product Codes: PL Issue +Priority: + +Revenue Codes: + +Bill Plan + +Start Date + +Great American Media + +3050 K Street NE + +Washington, DC 20007 + +ISS/DSCC IE + +P-2 + +AGY, POL, ISS + +End Date + +# Spots + +Gross Amount + +Net Amount + +10/31/16 + +11/08/16 + +Account Executives + +Account Executive + +48 + +$17,410.00 + +Totals + +Month + +Billing Type: Cash + +Billing Calendar: Broadcast + +Billing Cycle: EOM/EOC + +Agency Commission: 15% + +New Business Thru: + +Order Separation: + +Advertiser External ID: + +Agency External ID: + +Unit Code: + +00:30:00 + +General + +# Spots + +Gross Amount + +Net Amount + +Rating + +$14,798.50 November 2016 + +48 $17,410.00 $14,798.50 + +48 $17,410.00 $14,798.50 0.00 + +0.00 + +Totals + +Sales Office + +Sales Region + +Start Date / End Date + +Order % + +Telerep Philadelphia + +Order Share + +Share + +Total + +NewsCenter 17 + +Market + +Competitive Share + +CABLE +UNKWN +WBUI +WCCU +WCIA + +WCIX +WICD +WICS +WRSP + +25% + +$17,410.00 + +100% $69,640.00 + +Share + +Total + +0% +0% +0% +0% +25% +0% +0% +25% +25% + +$0.00 +$0.00 +$0.00 +$0.00 +$17,410.00 +$0.00 +$0.00 +$17,410.00 +$17,410.00 + +Start Of Order - End Of Order + +100% + +Ln Ch Start End Inventory Code Break Start/End Time Days Len Spots Rate Pri Rtg Type Spots Amount + +1 WAND 11/01/16 11/07/16 M -F 1p -2p CM M -F 1p -2p MTWTF-- :30 1 $200.00 P-2 0.00 NM +M -F 1p -2p + +Start Date End Date Weekdays Spots/Week + +Rate + +Rating +Week: 11/01/16 11/07/16 MTWTF-- 1 $200.00 0.00 + +2 WAND 11/01/16 11/07/16 M -F 1035-1135P CM M -F 1035p -1135p MTWTF-- :30 1 $600.00P-2 0.00 NM +M -F 1035p -1135p + +Start Date End Date Weekdays Spots/Week Rate Rating +Week: 11/01/16 11/07/16 MTWTF-- 1 $600.00 0.00 + +3 WAND 11/01/16 11/07/16 CM M -F 11a -12p MTWTF-- :30 3 $100.00 P-2 0.00 NM + +1 $200.00 +$600.00 +3 $300.00 +Print Date: 09/27/16 11:21:24 Page 2 of 4 + +Order / Rev: + +163314 + +Advertiser: + +ISS/DSCC IE + +Alt Order #: + +08250799 + +Product Desc: + +DSCC IE + +NewsCenter 17 + +Flight Dates: 11/01/16 - 11/08/16 Estimate: 4789 + +Ln Ch Start End Inventory Code Break Start/End Time Days Len Spots Rate Pri Rtg Type Spots Amount + +IA* 11a -12p +M -F 11a -12p + +Start Date End Date Weekdays Spots/Week + +Rate + +Rating +Week: 11/01/16 11/07/16 MTWTF-- 3 $100.00 0.00 + +4 WAND 11/01/16 11/07/16 M -F 12n -1p CM M -F 12n -1p MTWTF-- :30 3 $150.00P-2 0.00 NM 3 $450.00 +M -F 12n -1p + +Start Date End Date Weekdays Spots/Week + +Rate + +Rating +Week:11/01/16 11/07/16 MTWTF-- 3 $150.00 0.00 + +5 WAND 11/01/16 11/07/16 M -F 4p -430p CM M -F 4p -430p MTWTF-- :30 3 $200.00P-2 0.00 NM 3 $600.00 +M -F 4p -430p + +Start Date End Date Weekdays Spots/Week + +Rate + +Rating +Week: 11/01/16 11/07/16 MTWTF-- 3 $200.00 0.00 + +6 WAND 11/01/16 11/07/16 M -F 430p -5p CM M -F 430p -5p MTWTF-- :30 3 $300.00P-2 0.00 NM 3 $900.00 +M -F 430p -5p + +Start Date End Date Weekdays Spots/Week + +Rate + +Rating +Week:11/01/16 11/07/16 MTWTF-- 3 $300.00 0.00 + +7 WAND 11/01/16 11/07/16 M -F 5p -5:30p CM M -F 5p -5:30p MTWTF-- :30 3 $500.00P-2 0.00 NM 3 $1,500.00 +M -F 5p -5:30p + +Start Date End Date Weekdays Spots/Week + +Rate + +Rating +Week: 11/01/16 11/07/16 MTWTF-- 3 $500.00 0.00 + +8 WAND 11/01/16 11/07/16 M -F 5a -6a CM M -F 5a -6a MTWTF-- :30 3 $60.00 P-2 0.00 NM 3 $180.00 +M -F 5a -6a + +Start Date End Date Weekdays Spots/Week + +Rate + +Rating +Week: 11/01/16 11/07/16 MTWTF-- 3 $60.00 0.00 + +9 WAND 11/01/16 11/07/16 M -F 6p -630p CM M -F 6p -6:30p MTWTF-- :30 2 $900.00P-2 0.00 NM 2 $1,800.00 +M -F 6p -6:30p + +Start Date End Date Weekdays Spots/Week + +Rate + +Rating +Week: 11/01/16 11/07/16 MTWTF-- 2 $900.00 0.00 + +10 WAND 11/01/16 11/07/16 M -F 6a -7a CM M -F 6a -7a MTWTF-- :30 3 $300.00 P-2 0.00 NM 3 $900.00 +M -F 6a -7a + +Start Date End Date Weekdays Spots/Week + +Rate + +Rating +Week: 11/01/16 11/07/16 MTWTF-- 3 $300.00 0.00 + +11 WAND 11/01/16 11/07/16 M -F 630p -7p CM M -F 630p -7p MTWTF-- :30 1 $1,200.00P-2 0.00 NM 1 $1,200.00 +M -F 630p -7p + +Start Date End Date Weekdays Spots/Week + +Rate + +Rating +Week: 11/01/16 11/07/16 MTWTF-- 1 $1,200.00 0.00 + +12 WAND 11/01/16 11/07/16 M -F 7a -9a CM M -F 7a -9a MTWTF-- :30 2 $500.00 P-2 0.00 NM 2 $1,000.00 +M -F 7a -9a + +Start Date End Date Weekdays Spots/Week + +Rate + +Rating +Week: 11/01/16 11/07/16 MTWTF-- 2 $500.00 0.00 + +13 WAND 11/01/16 11/07/16 M -F 9a -10a CM M -F 9a -10a MTWTF-- :30 3 $200.00 P-2 0.00 NM 3 $600.00 +M -F 9a -10a + +Start Date End Date Weekdays Spots/Week Rate Rating +Week: 11/01/16 11/07/16 MTWTF-- 3 $200.00 0.00 + +N 14 WAND 11/04/16 11/04/16 Fri 9-10P CM F 9-10P 1 $2,000.00P-1 0.00 NM 1 $2,000.00 +Fri 9-10P + +Start Date End Date Weekdays Spots/Week + +Rate + +Rating +Week: 11/04/16 11/10/16 ----1-- 1 $2,000.00 0.00 +Spot Ch Date Range Description Start/End Time Weekdays Length Rate Rtq Type +1 NAND 11/04/16-11/10/16 9-10P F 9-10P ---- F---- :30($27000,00) 0.00 NM +See MG 14.2 +[Wrong Inventory Code-pgm is 2 hrs] +2 NAND 11/04/16-11/10/16 Fri 8-1Op Fri 8-10p ---- F---- :30 $2,000.00 0.00 NM +0 MG for 14.1 11/04 + +15 WAND 11/05/16 11/05/16 Sat 5p -530p CM Sat 5p -530p ---- 1- :30 1 $150.00 P-2 0.00 NM +Sat 5p -530p + +Start Date End Date Weekdays Spots/Week + +Rate + +Rating + +Week: 11/05/16 11/11/16 ---- 1- + +1 $150.00 0.00 + +$150.00 + +16 WAND 11/05/16 11/05/16 + +CM Sat 630-7p ---- 1- :30 1 $500.00 P-2 0.00 NM + +$500.00 +Print Date: 09/27/16 11:21:24 Page 3 of 4 + +Order/ Rev: + +163314 + +Advertiser: + +ISS/DSCC IE + +Alt Order #: + +08250799 + +Product Desc: + +DSCC IE NewsCenter 17 +Flight Dates: 11/01/16 - 11/08/16 Estimate: 4789 + +Ln Ch Start End Inventory Code Break Start/End Time Days Len Spots Rate Pri Rtg Type Spots Amount + +bat 6.30-/p +Sat 630-7p + +Start Date End Date Weekdays Spots/Week + +Rate + +Rating +Week: 11/05/16 11/11/16 ---- 1- 1 $500.00 0.00 + +17 WAND 11/05/16 11/05/16 Sat 7-9A CM Sat 7-9A ---- 1- :30 1 $300.00 P-2 0.00 NM 1 $300.00 +Sat 7-9A + +Start Date End Date Weekdays Spots/Week + +Rate Rating +Week: 11/05/16 11/11/16 ---- 1- 1 $300.00 0.00 + +18 WAND 11/05/16 11/05/16 Sat 9-10P CM Sat 9-10p ---- 1- :30 1 $220.00 P-2 0.00 NM 1 $220.00 +Sat 9-10p + +Start Date End Date Weekdays Spots/Week + +Rate Rating +Week: 11/05/16 11/11/16 ---- 1- 1 $220.00 0.00 + +19 WAND 11/06/16 11/06/16 Sun 1105p -1205a CM Su 1105p -1205a ---- 1 :30 1 $100.00 P-2 0.00 NM 1 $100.00 +Su 1105p -1205a + +Start Date End Date Weekdays Spots/Week + +Rate Rating +Week: 11/06/16 11/12/16 ---- 1 1 $100.00 0.00 + +20 WAND 11/06/16 11/06/16 Sun 5p -5:30p CM Sun 5p -530p ---- 1 :30 1 $300.00 P-2 0.00 NM 1 $300.00 +Sun 5p -530p + +Start Date End Date Weekdays Spots/Week + +Rate Rating +Week: 11/06/16 11/12/16 ---- 1 1 $300.00 0.00 + +21 WAND 11/06/16 11/06/16 Sun 8-9a CM Sun 8-9a ---- 1 :30 1 $300.00 P-2 0.00 NM 1 $300.00 +Sun 8-9a + +Start Date End Date Weekdays Spots/Week + +Rate Rating +Week: 11/06/16 11/12/16 ---- 1 1 $300.00 0.00 + +N 22 WAND 11/03/16 11/03/16 Thur 7-8p CM Th 7-8p ---1--- :30 1 $800.00 P-2 0.00 NM 1 $800.00 +Thur 7-8p + +Start Date End Date Weekdays Spots/Week + +Rate + +Rating +Week: 11/03/16 11/09/16 ---1--- 1 $800.00 0.00 +Spot Ch Date Range Description Start/End Time Weekdays Length + +Rate Rtq Type +1 NAND 11/03/16-11/09/16 Thur 7-8p Th 7-8p ---Th ---- :30 ($800.00} 0.00 NM +See MG 22.2 +[Program Change] +2 NAND 11/03/16-11/03/16 Thur 7-8p SUPERSTORE/G ---- :30 $800.00 0.00 NM +0 MG for 22.1 11/03 + +23 WAND 11/01/16 11/07/16 M -Su 10p -1035p CM M -Su 10p -1035p MTWTFSS :30 1 $1,000.00P-2 0.00 NM +M -Su 10p -1035p + +Start Date End Date Weekdays Spots/Week + +Rate + +Rating +Week: 11/01/16 11/07/16 MTWTFSS 1 $1,000.00 0.00 + +24 WAND 11/08/16 11/08/16 M -F 11a -12p CM M -F 11a -12p -1 ---- :30 1 $100.00 P-2 0.00 NM +M -F 11a -12p + +Start Date End Date Weekdays Spots/Week + +Rate + +Rating +Week: 11/08/16 11/14/16 -1 ---- 1 $100.00 0.00 + +25 WAND 11/08/16 11/08/16 M -F 12n -1p CM M -F 12n -1p -1 ---- :30 1 $150.00 P-2 0.00 NM +M -F 12n -1p + +Start Date End Date Weekdays Spots/Week + +Rate + +Rating +Week: 11/08/16 11/14/16 -1 ---- 1 $150.00 0.00 + +26 WAND 11/08/16 11/08/16 M -F 4p -430p CM M -F 4p -430p -1 ---- :30 1 $200.00 P-2 0.00 NM +M -F 4p -430p + +Start Date End Date Weekdays Spots/Week + +Rate + +Rating +Week: 11/08/16 11/14/16 -1 ---- 1 $200.00 0.00 + +27 WAND 11/08/16 11/08/16 M -F 430p -5p CM M -F 430p -5p -1 ---- :30 1 $300.00 P-2 0.00 NM +M -F 430p -5p + +Start Date End Date Weekdays Spots/Week + +Rate + +Rating +Week: 11/08/16 11/14/16 -1 ---- 1 $300.00 0.00 + +28 WAND 11/08/16 11/08/16 M -F 5p -5:30p CM M -F 5p -5:30p -1 ---- :30 1 $500.00 P-2 0.00 NM +M -F 5p -5:30p + +Start Date End Date Weekdays Spots/Week + +Rate + +Rating +Week: 11/08/16 11/14/16 -1 ---- 1 $500.00 0.00 + +29 WAND 11/08/16 11/08/16 CM M -F 5a -6a -1 ---- :30 1 $60.00 P-2 0.00 NM + +1 $1,000.00 + +1 $100.00 +1 $150.00 +1 $200.00 + +1 $300.00 +1 $500.00 +1 $60.00 +Print Date: 09/27/16 11:21:24 Page 4 of 4 + +Order/ Rev: + +163314 + +Advertiser: + +ISS/DSCC IE +Alt Order #: 08250799 Product Desc: DSCC IE NewsCenter 17 +Flight Dates: 11/01/16 - 11/08/16 Estimate: 4789 + +Ln Ch Start End Inventory Code Break Start/End Time Days Len Spots Rate Pri Rtg Type Spots Amount + +M -h oa-ba +M -F 5a -6a + +Start Date End Date Weekdays Spots/VVeek + +Rate + +Rating +Week: 11/08/16 11/14/16 -1 ---- 1 $60.00 0.00 + +30 WAND 11/08/16 11/08/16 M -F 9a -10a CM M -F 9a -10a -1 ---- :30 1 $200.00 P-2 0.00 NM +M -F 9a -10a + +Start Date End Date Weekdays Spots/Week + +Rate + +Rating +Week: 11/08/16 11/14/16 -1 ---- 1 $200.00 0.00 + +$200.00 + +Totals 48 $17,410.00",2dfd91c64c66a5a92b4eb1c46e26f87f,"{""line_item__description"": [""IA* 11a -12p"", ""M -F 11a -12p"", ""M -F 12n -1p"", ""M -F 12n -1p"", ""M -F 4p -430p"", ""M -F 4p -430p"", ""M -F 430p -5p"", ""M -F 430p -5p"", ""M -F 5p -5:30p"", ""M -F 5p -5:30p"", ""M -F 5a -6a"", ""M -F 5a -6a"", ""M -F 6p -630p"", ""M -F 6p -6:30p"", ""M -F 6a -7a"", ""M -F 6a -7a"", ""M -F 630p -7p"", ""M -F 630p -7p"", ""M -F 7a -9a"", ""M -F 7a -9a"", ""M -F 9a -10a"", ""M -F 9a -10a"", ""Fri 9-10P"", ""Fri 9-10P"", ""Sat 5p -530p"", ""Sat 5p -530p"", ""M -h oa-ba"", ""M -F 5a -6a"", ""M -F 9a -10a"", ""M -F 9a -10a"", ""bat 6.30-/p"", ""Sat 630-7p"", ""Sat 7-9A"", ""Sat 7-9A"", ""Sat 9-10P"", ""Sat 9-10p"", ""Sun 1105p -1205a"", ""Su 1105p -1205a"", ""Sun 5p -5:30p"", ""Sun 5p -530p"", ""Sun 8-9a"", ""Sun 8-9a"", ""Thur 7-8p"", ""Thur 7-8p"", ""M -Su 10p -1035p"", ""M -Su 10p -1035p"", ""M -F 11a -12p"", ""M -F 11a -12p"", ""M -F 12n -1p"", ""M -F 12n -1p"", ""M -F 4p -430p"", ""M -F 4p -430p"", ""M -F 430p -5p"", ""M -F 430p -5p"", ""M -F 5p -5:30p"", ""M -F 5p -5:30p"", ""M -F 1p -2p"", ""M -F 1p -2p"", ""M -F 1035-1135P"", ""M -F 1035p -1135p""], ""line_item__days"": [""MTWTF--"", ""MTWTF--"", ""MTWTF--"", ""MTWTF--"", ""MTWTF--"", ""MTWTF--"", ""MTWTF--"", ""MTWTF--"", ""MTWTF--"", ""MTWTF--"", ""MTWTF--"", ""MTWTF--"", ""MTWTF--"", ""MTWTF--"", ""MTWTF--"", ""MTWTF--"", ""MTWTF--"", ""----1--"", ""---- F----"", ""---- F----"", ""---- 1-"", ""---- 1-"", ""---- 1-"", ""-1"", ""-1"", ""-1"", ""1"", ""1"", ""1"", ""1"", ""1"", ""1"", ""1"", ""MTWTFSS"", ""1"", ""1"", ""1"", ""1"", ""1"", ""MTWTF--"", ""MTWTF--"", ""MTWTF--"", ""MTWTF--"", ""MTWTF--""], ""line_item__rate"": [""100.00"", ""150.00"", ""150.00"", ""200.00"", ""200.00"", ""300.00"", ""300.00"", ""500.00"", ""500.00"", ""60.00"", ""60.00"", ""900.00"", ""900.00"", ""300.00"", ""300.00"", ""1,200.00"", ""1,200.00"", ""500.00"", ""500.00"", ""200.00"", ""200.00"", ""2,000.00"", ""2,000.00"", ""2,000.00"", ""150.00"", ""150.00"", ""500.00"", ""60.00"", ""200.00"", ""200.00"", ""500.00"", ""300.00"", ""300.00"", ""220.00"", ""220.00"", ""100.00"", ""100.00"", ""300.00"", ""300.00"", ""300.00"", ""300.00"", ""800.00"", ""800.00"", ""800.00"", ""1,000.00"", ""1,000.00"", ""100.00"", ""100.00"", ""150.00"", ""150.00"", ""200.00"", ""200.00"", ""300.00"", ""300.00"", ""500.00"", ""500.00"", ""60.00""], ""line_item__start_date"": [""11/01/16"", ""11/01/16"", ""11/01/16"", ""11/01/16"", ""11/01/16"", ""11/01/16"", ""11/01/16"", ""11/01/16"", ""11/01/16"", ""11/01/16"", ""11/04/16"", ""11/05/16"", ""11/05/16"", ""11/08/16"", ""11/08/16"", ""11/05/16"", ""11/05/16"", ""11/06/16"", ""11/06/16"", ""11/06/16"", ""11/03/16"", ""11/01/16"", ""11/08/16"", ""11/08/16"", ""11/08/16"", ""11/08/16"", ""11/08/16"", ""11/08/16"", ""10/31/16"", ""11/01/16"", ""11/01/16"", ""11/01/16""], ""line_item__end_date"": [""11/07/16"", ""11/07/16"", ""11/07/16"", ""11/07/16"", ""11/07/16"", ""11/07/16"", ""11/07/16"", ""11/07/16"", ""11/07/16"", ""11/07/16"", ""11/04/16"", ""11/05/16"", ""11/05/16"", ""11/14/16"", ""11/14/16"", ""11/05/16"", ""11/05/16"", ""11/06/16"", ""11/06/16"", ""11/06/16"", ""11/03/16"", ""11/07/16"", ""11/08/16"", ""11/08/16"", ""11/08/16"", ""11/08/16"", ""11/08/16"", ""11/08/16"", ""11/08/16"", ""11/07/16"", ""11/07/16"", ""11/07/16""], ""agency"": [""Great American Media""], ""advertiser"": [""ISS/DSCC IE""], ""gross_total"": [""17,410.00""], ""net_amount_due"": [""14,798.50""]}" +1,1,1,fcc_invoices/files/e886440f05a096ba3cf66be0dd859474.pdf,"[""fcc_invoices/images/e886440f05a096ba3cf66be0dd859474/page_0.png"", ""fcc_invoices/images/e886440f05a096ba3cf66be0dd859474/page_1.png"", ""fcc_invoices/images/e886440f05a096ba3cf66be0dd859474/page_2.png"", ""fcc_invoices/images/e886440f05a096ba3cf66be0dd859474/page_3.png""]","[{""label"": ""Line Item - Description"", ""start"": 4777, ""end"": 4792, ""text"": ""M -F 12p -1230p""}, {""label"": ""Line Item - Description"", ""start"": 4819, ""end"": 4838, ""text"": ""NFL Reg Ssn Late Gm""}, {""label"": ""Advertiser"", ""start"": 4878, ""end"": 4900, ""text"": ""Senate Leadership Fund""}, {""label"": ""Line Item - Start Date"", ""start"": 5155, ""end"": 5163, ""text"": ""10/03/16""}, {""label"": ""Line Item - End Date"", ""start"": 5167, ""end"": 5175, ""text"": ""10/09/16""}, {""label"": ""Line Item - Days"", ""start"": 5517, ""end"": 5519, ""text"": ""Su""}, {""label"": ""Line Item - Rate"", ""start"": 5555, ""end"": 5563, ""text"": ""8,500.00""}, {""label"": ""Line Item - Description"", ""start"": 5856, ""end"": 5875, ""text"": ""Sunday Prime Hour 1""}, {""label"": ""Line Item - Start Date"", ""start"": 5894, ""end"": 5902, ""text"": ""10/03/16""}, {""label"": ""Line Item - End Date"", ""start"": 5906, ""end"": 5914, ""text"": ""10/09/16""}, {""label"": ""Line Item - Days"", ""start"": 5996, ""end"": 5998, ""text"": ""Su""}, {""label"": ""Line Item - Rate"", ""start"": 6034, ""end"": 6042, ""text"": ""8,500.00""}, {""label"": ""Line Item - Description"", ""start"": 6049, ""end"": 6068, ""text"": ""Sunday Prime Hour 2""}, {""label"": ""Line Item - Days"", ""start"": 6076, ""end"": 6078, ""text"": ""Su""}, {""label"": ""Line Item - Rate"", ""start"": 6114, ""end"": 6122, ""text"": ""8,500.00""}, {""label"": ""Line Item - Description"", ""start"": 6150, ""end"": 6169, ""text"": ""Presidential Debate""}, {""label"": ""Line Item - Description"", ""start"": 6191, ""end"": 6216, ""text"": ""NFL Thursday Night Pre Gm""}, {""label"": ""Line Item - Start Date"", ""start"": 6229, ""end"": 6237, ""text"": ""10/03/16""}, {""label"": ""Line Item - End Date"", ""start"": 6241, ""end"": 6249, ""text"": ""10/09/16""}, {""label"": ""Line Item - Days"", ""start"": 6265, ""end"": 6267, ""text"": ""Th""}, {""label"": ""Line Item - Rate"", ""start"": 6304, ""end"": 6312, ""text"": ""3,500.00""}, {""label"": ""Line Item - Description"", ""start"": 6323, ""end"": 6344, ""text"": ""Saturday Prime Hour 2""}, {""label"": ""Line Item - Start Date"", ""start"": 6364, ""end"": 6372, ""text"": ""10/03/16""}, {""label"": ""Line Item - End Date"", ""start"": 6376, ""end"": 6384, ""text"": ""10/09/16""}, {""label"": ""Line Item - Days"", ""start"": 6395, ""end"": 6397, ""text"": ""Su""}, {""label"": ""Line Item - Rate"", ""start"": 6434, ""end"": 6442, ""text"": ""6,000.00""}, {""label"": ""Line Item - Start Date"", ""start"": 6454, ""end"": 6462, ""text"": ""10/03/16""}, {""label"": ""Line Item - End Date"", ""start"": 6466, ""end"": 6474, ""text"": ""10/09/16""}, {""label"": ""Line Item - Days"", ""start"": 6480, ""end"": 6482, ""text"": ""Sa""}, {""label"": ""Line Item - Rate"", ""start"": 6518, ""end"": 6524, ""text"": ""750.00""}, {""label"": ""Gross Total"", ""start"": 6562, ""end"": 6571, ""text"": ""80,350.00""}, {""label"": ""Line Item - Description"", ""start"": 2501, ""end"": 2514, ""text"": ""Late Run News""}, {""label"": ""Line Item - Description"", ""start"": 2528, ""end"": 2543, ""text"": ""M -F 12p -1230p""}, {""label"": ""Advertiser"", ""start"": 2613, ""end"": 2635, ""text"": ""Senate Leadership Fund""}, {""label"": ""Line Item - Start Date"", ""start"": 3196, ""end"": 3204, ""text"": ""10/03/16""}, {""label"": ""Line Item - End Date"", ""start"": 3208, ""end"": 3216, ""text"": ""10/09/16""}, {""label"": ""Line Item - Days"", ""start"": 3270, ""end"": 3272, ""text"": ""Th""}, {""label"": ""Line Item - Rate"", ""start"": 3309, ""end"": 3317, ""text"": ""1,150.00""}, {""label"": ""Line Item - Start Date"", ""start"": 3339, ""end"": 3347, ""text"": ""10/03/16""}, {""label"": ""Line Item - End Date"", ""start"": 3351, ""end"": 3359, ""text"": ""10/09/16""}, {""label"": ""Line Item - Days"", ""start"": 3381, ""end"": 3383, ""text"": ""Th""}, {""label"": ""Line Item - Rate"", ""start"": 3420, ""end"": 3426, ""text"": ""600.00""}, {""label"": ""Line Item - Description"", ""start"": 3435, ""end"": 3460, ""text"": ""NFL Thursday Night Pre Gm""}, {""label"": ""Line Item - Description"", ""start"": 3485, ""end"": 3503, ""text"": ""NFL Thursday Night""}, {""label"": ""Line Item - Start Date"", ""start"": 3523, ""end"": 3531, ""text"": ""10/03/16""}, {""label"": ""Line Item - End Date"", ""start"": 3535, ""end"": 3543, ""text"": ""10/09/16""}, {""label"": ""Line Item - Days"", ""start"": 3548, ""end"": 3550, ""text"": ""Th""}, {""label"": ""Line Item - Rate"", ""start"": 3585, ""end"": 3593, ""text"": ""5,000.00""}, {""label"": ""Line Item - Start Date"", ""start"": 3618, ""end"": 3626, ""text"": ""10/03/16""}, {""label"": ""Line Item - End Date"", ""start"": 3630, ""end"": 3638, ""text"": ""10/09/16""}, {""label"": ""Line Item - Days"", ""start"": 3650, ""end"": 3652, ""text"": ""Th""}, {""label"": ""Line Item - Rate"", ""start"": 3689, ""end"": 3698, ""text"": ""10,000.00""}, {""label"": ""Line Item - Description"", ""start"": 3709, ""end"": 3724, ""text"": ""M -F 11p -1135p""}, {""label"": ""Line Item - Start Date"", ""start"": 3745, ""end"": 3753, ""text"": ""10/03/16""}, {""label"": ""Line Item - End Date"", ""start"": 3757, ""end"": 3765, ""text"": ""10/09/16""}, {""label"": ""Line Item - Days"", ""start"": 3782, ""end"": 3784, ""text"": ""Tu""}, {""label"": ""Line Item - Rate"", ""start"": 3822, ""end"": 3830, ""text"": ""1,150.00""}, {""label"": ""Line Item - Description"", ""start"": 3841, ""end"": 3856, ""text"": ""M -F 12p -1230p""}, {""label"": ""Line Item - Start Date"", ""start"": 3877, ""end"": 3885, ""text"": ""10/03/16""}, {""label"": ""Line Item - End Date"", ""start"": 3889, ""end"": 3897, ""text"": ""10/09/16""}, {""label"": ""Line Item - Days"", ""start"": 3909, ""end"": 3911, ""text"": ""Tu""}, {""label"": ""Line Item - Rate"", ""start"": 3948, ""end"": 3954, ""text"": ""600.00""}, {""label"": ""Line Item - Description"", ""start"": 3965, ""end"": 3985, ""text"": ""Tuesday Prime Hour 1""}, {""label"": ""Line Item - Start Date"", ""start"": 4004, ""end"": 4012, ""text"": ""10/03/16""}, {""label"": ""Line Item - End Date"", ""start"": 4016, ""end"": 4024, ""text"": ""10/09/16""}, {""label"": ""Line Item - Days"", ""start"": 4036, ""end"": 4038, ""text"": ""Tu""}, {""label"": ""Line Item - Rate"", ""start"": 4074, ""end"": 4082, ""text"": ""5,500.00""}, {""label"": ""Line Item - Days"", ""start"": 4087, ""end"": 4089, ""text"": ""Tu""}, {""label"": ""Line Item - Rate"", ""start"": 4125, ""end"": 4133, ""text"": ""5,500.00""}, {""label"": ""Line Item - Description"", ""start"": 4142, ""end"": 4157, ""text"": ""M -F 11p -1135p""}, {""label"": ""Line Item - Start Date"", ""start"": 4184, ""end"": 4192, ""text"": ""10/03/16""}, {""label"": ""Line Item - End Date"", ""start"": 4196, ""end"": 4204, ""text"": ""10/09/16""}, {""label"": ""Line Item - Days"", ""start"": 4220, ""end"": 4221, ""text"": ""W""}, {""label"": ""Line Item - Rate"", ""start"": 4258, ""end"": 4266, ""text"": ""1,150.00""}, {""label"": ""Line Item - Description"", ""start"": 4276, ""end"": 4291, ""text"": ""M -F 12p -1230p""}, {""label"": ""Line Item - Start Date"", ""start"": 4317, ""end"": 4325, ""text"": ""10/03/16""}, {""label"": ""Line Item - End Date"", ""start"": 4329, ""end"": 4337, ""text"": ""10/09/16""}, {""label"": ""Line Item - Days"", ""start"": 4350, ""end"": 4351, ""text"": ""W""}, {""label"": ""Line Item - Rate"", ""start"": 4388, ""end"": 4394, ""text"": ""600.00""}, {""label"": ""Advertiser"", ""start"": 253, ""end"": 275, ""text"": ""Senate Leadership Fund""}, {""label"": ""Line Item - Description"", ""start"": 561, ""end"": 576, ""text"": ""M -F 11p -1135p""}, {""label"": ""Line Item - Description"", ""start"": 606, ""end"": 621, ""text"": ""M -F 12p -1230p""}, {""label"": ""Line Item - Description"", ""start"": 653, ""end"": 672, ""text"": ""Friday Prime Hour 2""}, {""label"": ""Line Item - Description"", ""start"": 703, ""end"": 718, ""text"": ""M -F 11p -1135p""}, {""label"": ""Line Item - Description"", ""start"": 751, ""end"": 766, ""text"": ""M -F 12p -1230p""}, {""label"": ""Line Item - Description"", ""start"": 790, ""end"": 805, ""text"": ""Big Bang! Kevin""}, {""label"": ""Line Item - Description"", ""start"": 833, ""end"": 854, ""text"": ""Saturday Prime Hour 3""}, {""label"": ""Line Item - Description"", ""start"": 932, ""end"": 953, ""text"": ""Saturday Prime Hour 1""}, {""label"": ""Line Item - Start Date"", ""start"": 1051, ""end"": 1059, ""text"": ""10/03/16""}, {""label"": ""Line Item - End Date"", ""start"": 1063, ""end"": 1071, ""text"": ""10/09/16""}, {""label"": ""Line Item - Days"", ""start"": 1408, ""end"": 1409, ""text"": ""F""}, {""label"": ""Line Item - Start Date"", ""start"": 1437, ""end"": 1445, ""text"": ""10/03/16""}, {""label"": ""Line Item - End Date"", ""start"": 1449, ""end"": 1457, ""text"": ""10/09/16""}, {""label"": ""Line Item - Rate"", ""start"": 1464, ""end"": 1472, ""text"": ""1,150.00""}, {""label"": ""Line Item - Days"", ""start"": 1477, ""end"": 1478, ""text"": ""F""}, {""label"": ""Line Item - Start Date"", ""start"": 1515, ""end"": 1523, ""text"": ""10/03/16""}, {""label"": ""Line Item - End Date"", ""start"": 1527, ""end"": 1535, ""text"": ""10/09/16""}, {""label"": ""Line Item - Days"", ""start"": 1550, ""end"": 1551, ""text"": ""F""}, {""label"": ""Line Item - Rate"", ""start"": 1587, ""end"": 1595, ""text"": ""4,000.00""}, {""label"": ""Line Item - Rate"", ""start"": 1598, ""end"": 1604, ""text"": ""600.00""}, {""label"": ""Line Item - Start Date"", ""start"": 1608, ""end"": 1616, ""text"": ""10/10/16""}, {""label"": ""Line Item - End Date"", ""start"": 1620, ""end"": 1628, ""text"": ""10/16/16""}, {""label"": ""Line Item - Days"", ""start"": 1637, ""end"": 1638, ""text"": ""M""}, {""label"": ""Line Item - Rate"", ""start"": 1675, ""end"": 1683, ""text"": ""1,150.00""}, {""label"": ""Line Item - Start Date"", ""start"": 1688, ""end"": 1696, ""text"": ""10/10/16""}, {""label"": ""Line Item - End Date"", ""start"": 1700, ""end"": 1708, ""text"": ""10/16/16""}, {""label"": ""Line Item - Days"", ""start"": 1715, ""end"": 1716, ""text"": ""M""}, {""label"": ""Line Item - Rate"", ""start"": 1753, ""end"": 1759, ""text"": ""600.00""}, {""label"": ""Line Item - Start Date"", ""start"": 1761, ""end"": 1769, ""text"": ""10/10/16""}, {""label"": ""Line Item - End Date"", ""start"": 1773, ""end"": 1781, ""text"": ""10/16/16""}, {""label"": ""Line Item - Days"", ""start"": 1792, ""end"": 1793, ""text"": ""M""}, {""label"": ""Line Item - Rate"", ""start"": 1829, ""end"": 1837, ""text"": ""3,600.00""}, {""label"": ""Line Item - Start Date"", ""start"": 1841, ""end"": 1849, ""text"": ""10/03/16""}, {""label"": ""Line Item - End Date"", ""start"": 1853, ""end"": 1861, ""text"": ""10/09/16""}, {""label"": ""Line Item - Days"", ""start"": 1866, ""end"": 1868, ""text"": ""Sa""}, {""label"": ""Line Item - Rate"", ""start"": 1940, ""end"": 1948, ""text"": ""1,500.00""}, {""label"": ""Line Item - Start Date"", ""start"": 1988, ""end"": 1996, ""text"": ""10/03/16""}, {""label"": ""Line Item - End Date"", ""start"": 2000, ""end"": 2008, ""text"": ""10/09/16""}, {""label"": ""Line Item - Days"", ""start"": 2022, ""end"": 2024, ""text"": ""Sa""}, {""label"": ""Line Item - Rate"", ""start"": 2062, ""end"": 2068, ""text"": ""750.00""}, {""label"": ""Advertiser"", ""start"": 6754, ""end"": 6776, ""text"": ""Senate Leadership Fund""}, {""label"": ""Agency Commission"", ""start"": 7512, ""end"": 7521, ""text"": ""12,052.50""}, {""label"": ""Net Amount Due"", ""start"": 7523, ""end"": 7532, ""text"": ""68,297.50""}, {""label"": ""Payment Terms"", ""start"": 7548, ""end"": 7555, ""text"": ""30 Days""}]",fcc_invoices/ocr/e886440f05a096ba3cf66be0dd859474.json.gz,00da1104-d80b-438b-a625-557092b8bcb9.pdf,"Remit Address: + +WTTV-CBS +*0 16779 Collections Center Drive +Chicago, IL 60693 +Main: (317) 632-5900 +Billing: (317) 715-2704 + +Billing Address: + +Mentzer Media +Attention: Accounts Payable +210 W Pennsylvania Ave +Ste 250 +Towson, MD 21204 + +INVOICE + +Advertiser + +Senate Leadership Fund + +Product + +SEN LEADERSHIP FUND + +Estimate Number + +3622 + +Station + +WTTV-CBS + +Account Executive + +Justin Votta + +Sales Office + +NSO Philadelphia + +Sales Region + +National + +Billing Calendar + +Broadcast + +Billing Type + +Cash + +Special ____ Handling + +Line Channel Description 1 Time IDay I Date + +2 CBS M -F 11p -1135p 11:00 PM -11:35 PM +CBS +3 CBS M -F 12p -1230p 12:00 PM -12:30 PM + +CBS + +4 CBS Friday Prime Hour 2 9:00 PM -10:00 FM + +CBS + +5 CBS M -F 11p -1135p 11:00 PM -11:35 PM + +CBS + +Et CBS M -F 12p -1230p 12p -1230p + +CBS + +7 CBS Big Bang! Kevin 8:00 PM -9:00 PM + +CBS + +CBS Saturday Prime Hour 3 10:00 PM -11:00 PM + +CBS + +9 CBS Sa 1205a -1235a 12:05 AM -12:35 AM +CBS +i0 CBS Saturday Prime Hour 1 8:00 PM -9:00 PM + +CBS + +11 CBS Saturday Prime Hour 2 9:00 PM -10:00 PM + +Length 'Air Time + +Ad -ID + +10/03/16 to 10/09/16 lx + +----1--- + +Page 1 of 4 + +Invoice # + +WC161 00116 + +invoice Date + +10/16/16 + +Invoice Month + +October 2016 + +Invoice Period + +09/26/16 - 10/10/16 + +Order # + +373338 + +Alt Order # + +08358687 + +Deal # + +Order Flight + +10/04/16 - 10/10/16 + +IDB # + +1021 + +Advertiser Code + +275 + +Product Code + +311 + +Agency Ref + +Advertiser Ref + +'Rate + +Reconciliation + +Ref # + +F :30 11:33 PM sLFTv30IN09H + +10/03/16 to 10/09/16 + +lx + +$1,150.00 + +1 + +F 10/07/16 :30 12:21 PM SLFIV3OINO9H + +10/03/16 to 10/09/16 + +lx + +----1-- + +F 10/07/16 :30 9:36 PM sLFTv30IN09H $4,000.00 + +$600.00 1 + +10/10/16 to 10/16/16 + +lx + +1 + +M 10/10/16 :30 11:13 PM SLFTV3DIN09H $1,150.00 + +1 + +10/10/16 to 10/16/16 lx 1 + +M 10/10/16 :30 12:12 PM SLFTV30IN09H $600.00 + +10/10/16 to 10/16/16 1 x 1 + +1 + +M 10/10/16 :30 8:48 PM SLFTV30IN09H $3,600.00 1 + +10/03/16 to 10/09/16 lx + +Sa 10/08/16 :30 10:54 PM SLFTV30IN09H + +10/03/16 to 10/09/16 + +lx + +---- 1 + +$1,500.00 1 + +Sa 10/08/16 :00 $250440 Credited 1 + +10/03/16 to 10/09/16 + +lx + +---- 1 + +Sa 10/08/16 :30 9:17 PM siLFrv3Oir409H $750.00 1 +DT + +10/03/16 to 10/09/16 1x + +We warrant that the actual broadcast information shown on this invoice was taken from the program log. +Remit Address: + +WTTV-CBS +16779 Collections Center Drive +Chicago, IL 60693 +Main: (317) 632-5900 +Billing: (317) 715-2704 + +Billing Address: + +Mentzer Media +Attention: Accounts Payable +210 W Pennsylvania Ave +Ste 250 +Towson, MD 21204 + +Line + +Channel 'Description + +11 CBS Saturday Prime Hour 2 +CBS + +15 CBS Late Run News + +CBS + +16 CBS M -F 12p -1230p + +Time fDay Flats + +9:00 PM -10:00 PM + +11p -1135p + +INVOICE + +Advertiser Senate Leadership Fund + +Product + +Estimate Number1 3622 + +SEN LEADERSHIP FUND + +Station + +Account Executive + +Sales Office + +Sales Region + +WTTV-CBS +Justin Votta +NSO Philadelphia + +National + +Billing Calendar + +Broadcast + +Billing Type + +Cash + +Special Handling + +Page 2 of 4 + +Invoice # + +WC161 00116 + +Invoice Date + +10/16/16 + +Invoice Month + +October 2016 + +Invoice Period + +09/26/16 - 10/10/16 + +Order # + +373338 + +Alt Order # + +08358687 + +Deal # + +Order Flight + +10/04/16 - 10/10/16 + +IDB # + +1021 + +Advertiser Code + +275 + +Product Code + +311 + +Agency Ref + +Advertiser Ref + +Length Air Time Ad -ID + +Sa 10/08/16 :00 + +10/03/16 to 10/09/16 + +1x---1--- + +'Rate 'Reconciliation + +$7-50X0 Credited + +Th 10/06/16 :30 12:20 AM SLFFV3OINO9H $1,150.00 + +12:00 PM -12:30 PM + +10/03/16 to 10/09/16 lx + +Ref # + +1 + +1 + +CBS Th 10/06/16 :30 12:12 PM SLED/30INO9H $600.00 + +17 CBS NFL Thursday Night Pre Gm 730p -815p + +CBS + +18 CBS NFL Thursday Night 8:15 PM -11:15 PM + +10/03/16 to 10/09/16 lx + +Th 10/06/16 :30 7:29 PM JGIN161330H $5,000.00 +Time swap + +--.-1--- + +1 + +10/03/16 to 10/09/16 lx + +1 + +CBS Th 10/06/16 :30 11:45 PM SLFTV3OINO9H $10,000.00 1 + +19 CBS M -F 11p -1135p 11:00 PM -11:35 PM + +10/03/16 to 10/09/16 lx -1 ---- + +CBS Tu 10/04/16 :30 11:22 PM + +SLFTV3OINO8H $1,150.00 1 + +20 CBS M -F 12p -1230p 12:00 PM -12:30 PM + +10/03/16 to 10/09/16 lx -1 + +CBS Tu 10/04/16 :30 12:12 PM SLFTV3OINO8H $600.00 1 + +21 CBS Tuesday Prime Hour 1 8:00 PM -9:00 PM + +10/03/16 to 10/09/16 2x -2 + +CBS Tu 10/04/16 :30 8:31 PM SLFTV3OINO8H $5,500.00 +CBS Tu 10/04/16 :30 8:44 PM SLFTV3OINO8H $5,500.00 + +22 CBS M -F 11p -1135p 11:00 PM -11:35 PM + +1 + +2 + +10/03/16 to 10/09/16 lx --1---- +CBS W 10/05/16 :30 11:30 PM SLFTV3OINO8H $1,150.00 1 +23 CBS M -F 12p -1230p 12:00 PM -12:30 PM + +CBS + +10/03/16 to 10/09/16 lx --1---- + +W +10/05/16 :30 12:24 PM SLFTv3OINO8H $600.00 +We warrant that the actual broadcast information shown on this invoice was taken from the program log. +Remit Address: + +WTTV-C BS +16779 Collections Center Drive +Chicago, IL 60693 +Main: (317) 632-5900 +Billing: (317) 715-2704 + +Billing Address: + +Mentzer Media +Attention: Accounts Payable +210 W Pennsylvania Ave +Ste 250 +Towson, MD 21204 + +Line i Channel i Description + +Time I Day + +23 CBS M -F 12p -1230p 12:00 PM -12:30 PM +24 CBS NFL Reg Ssn Late Gm 4:00 PM -7:30 PM + +INVOICE + +Advertiser + +Senate Leadership Fund + +Product + +SEN LEADERSHIP FUND + +Estimate Number + +3622 + +Station + +Account Executive + +WTTV-CBS + +Justin \/otta + +Sales Office + +NSO Philadelphia + +Sales Region + +Nationai + +Billing Calendar + +Broadcast + +Billing Type + +Cash + +Special Handling + +Date + +Length + +Air Time + +10/03/16 to 10/09/16 lx + +1 + +Page 3 of 4 + +Invoice # + +WC161 00116 + +Invoice Date + +10/16/16 + +Invoice Month + +October 2016 + +invoice Period + +09/26/16 - 10/10/16 + +Order # + +373338 + +Alt Order # + +08358687 + +Deal # + +Order Flight + +10/04/16 - 10/10/16 + +IDB # + +1021 + +Advertiser Code + +275 + +Product Code + +311 + +Agency Ref + +Advertiser Ref + +Ad -ID + +[Rate + +Reconciliation + +Ref # + +CBS Su 10/09/16 :30 6:37 PM SLFTV3OINO9H $8,500.00 1 + +25 CBS Sunday Prime Hour 1 7:30 PM -8:30 PM + +10/03/16 to 10/09/16 2x ---- 2 +CBS Su 10/09/16 :00 $5700040 Credited 1 + +CBS Su 10/09/16 :00 + +26 CBS Sunday Prime Hour 2 8:30 PM -9:30 PM + +10/03/16 to 10/09/16 lx + +1 + +____ INI•1=1•111=1.k. ____ + +$5;000,00 Credited 2 + +CBS Su 10/09/16 :00 + +27 CBS Sunday Prime Hour 1 7:30 PM -8:30 PM + +10/03/16 to 10/09/16 2x ---- 2 + +$5,0004)0 Credited 1 + +CBS Su 10/09/16 :00 $8,500,08 See MG 27.3 2 +CBS Su 10/09/16 :30 8:24 PM SLFTV3OINO9H $8,500.00 1 +CBS Sunday Prime Hour 2 8p -9p Su 10/09/16 :30 8:42 PM SLFTV3OINO9H $8,500.00 MG for 27.2 10/09 3 +28 CBS Presidential Debate 9p -11p + +CBS + +29 CBS NFL Thursday Night Pre Gm 730p -815p + +10/03/16 to 10/09/16 lx ---1--- +CBS Th 10/06/16 :30 7:54 PM sLF-rv3OINO9H $3,500.00 1 + +30 CBS Saturday Prime Hour 2 9:00 PM -10:00 PM + +10/03/16 to 10/09/16 lx ---- S +Su 10/09/16 :30 11:05 PM SLFT/30INO9H $6,000.00 1 +DT + +CBS + +10/03/16 to 10/09/16 1 x + +Sa 10/08/16 :30 10:25 PM SLFTV3OINON $750.00 1 +DT + +Aired Spots + +24 + +Gross Total + +$80,350.00 + +We warrant that the actual broadcast Information shown on this invoice was taken from the program log. +Remit Address: INVOICE +MTV -CRS +16779 Collections Center Drive I Advertiser I Senate Leadership Fund + +Chicago, IL 60693 + +f Product I SEN LEADERSHIP FUND + +Main: (317) 632-5900 + +Estimate Number L3622 + +Billing: (317) 715-2704 + +Billing Address: + +Mentzer Media +Attention: Accounts Payable +210 W Pennsylvania Ave +Ste 250 +Towson, MD 21204 + +Station + +WTTV-CBS + +Account Executive + +Justin Votta + +Sales Office + +NSO Philadelphia + +Sales Region + +National + +Billing Calendar + +Broadcast + +Billing Type + +Cash + +Special Handling + +Agency Commission + +Net Amount Due + +Page 4 of 4 + +Invoice # + +WC161 00116 + +Invoice Date + +10/16/16 + +Invoice Month + +October 2016 + +Invoice Period + +09/26/16 - 10/10/16 + +Order # + +373338 + +Alt Order # + +08358687 + +Deal # + +Order Flight + +10/04/16 - 10/10/16 + +IDB # + +1021 + +Advertiser Code + +275 + +Product Code + +311 + +Agency Ref + +Advertiser Ref + +$12,052.50 +$68,297.50 + +Payment Terms 30 Days + +,Ne warrant that the actual broadcast information shown on this invoice was taken from the program log.",e886440f05a096ba3cf66be0dd859474,"{""line_item__description"": [""M -F 12p -1230p"", ""NFL Reg Ssn Late Gm"", ""Sunday Prime Hour 1"", ""Sunday Prime Hour 2"", ""Presidential Debate"", ""NFL Thursday Night Pre Gm"", ""Saturday Prime Hour 2"", ""Late Run News"", ""M -F 12p -1230p"", ""NFL Thursday Night Pre Gm"", ""NFL Thursday Night"", ""M -F 11p -1135p"", ""M -F 12p -1230p"", ""Tuesday Prime Hour 1"", ""M -F 11p -1135p"", ""M -F 12p -1230p"", ""M -F 11p -1135p"", ""M -F 12p -1230p"", ""Friday Prime Hour 2"", ""M -F 11p -1135p"", ""M -F 12p -1230p"", ""Big Bang! Kevin"", ""Saturday Prime Hour 3"", ""Saturday Prime Hour 1""], ""advertiser"": [""Senate Leadership Fund"", ""Senate Leadership Fund"", ""Senate Leadership Fund"", ""Senate Leadership Fund""], ""line_item__start_date"": [""10/03/16"", ""10/03/16"", ""10/03/16"", ""10/03/16"", ""10/03/16"", ""10/03/16"", ""10/03/16"", ""10/03/16"", ""10/03/16"", ""10/03/16"", ""10/03/16"", ""10/03/16"", ""10/03/16"", ""10/03/16"", ""10/03/16"", ""10/03/16"", ""10/03/16"", ""10/10/16"", ""10/10/16"", ""10/10/16"", ""10/03/16"", ""10/03/16""], ""line_item__end_date"": [""10/09/16"", ""10/09/16"", ""10/09/16"", ""10/09/16"", ""10/09/16"", ""10/09/16"", ""10/09/16"", ""10/09/16"", ""10/09/16"", ""10/09/16"", ""10/09/16"", ""10/09/16"", ""10/09/16"", ""10/09/16"", ""10/09/16"", ""10/09/16"", ""10/09/16"", ""10/16/16"", ""10/16/16"", ""10/16/16"", ""10/09/16"", ""10/09/16""], ""line_item__days"": [""Su"", ""Su"", ""Su"", ""Th"", ""Su"", ""Sa"", ""Th"", ""Th"", ""Th"", ""Th"", ""Tu"", ""Tu"", ""Tu"", ""Tu"", ""W"", ""W"", ""F"", ""F"", ""F"", ""M"", ""M"", ""M"", ""Sa"", ""Sa""], ""line_item__rate"": [""8,500.00"", ""8,500.00"", ""8,500.00"", ""3,500.00"", ""6,000.00"", ""750.00"", ""1,150.00"", ""600.00"", ""5,000.00"", ""10,000.00"", ""1,150.00"", ""600.00"", ""5,500.00"", ""5,500.00"", ""1,150.00"", ""600.00"", ""1,150.00"", ""4,000.00"", ""600.00"", ""1,150.00"", ""600.00"", ""3,600.00"", ""1,500.00"", ""750.00""], ""gross_total"": [""80,350.00""], ""agency_commission"": [""12,052.50""], ""net_amount_due"": [""68,297.50""], ""payment_terms"": [""30 Days""]}" +2,2,2,fcc_invoices/files/34d4c874dd87752d683e4763e9333559.pdf,"[""fcc_invoices/images/34d4c874dd87752d683e4763e9333559/page_0.png"", ""fcc_invoices/images/34d4c874dd87752d683e4763e9333559/page_1.png"", ""fcc_invoices/images/34d4c874dd87752d683e4763e9333559/page_2.png"", ""fcc_invoices/images/34d4c874dd87752d683e4763e9333559/page_3.png"", ""fcc_invoices/images/34d4c874dd87752d683e4763e9333559/page_4.png"", ""fcc_invoices/images/34d4c874dd87752d683e4763e9333559/page_5.png"", ""fcc_invoices/images/34d4c874dd87752d683e4763e9333559/page_6.png"", ""fcc_invoices/images/34d4c874dd87752d683e4763e9333559/page_7.png"", ""fcc_invoices/images/34d4c874dd87752d683e4763e9333559/page_8.png"", ""fcc_invoices/images/34d4c874dd87752d683e4763e9333559/page_9.png"", ""fcc_invoices/images/34d4c874dd87752d683e4763e9333559/page_10.png"", ""fcc_invoices/images/34d4c874dd87752d683e4763e9333559/page_11.png"", ""fcc_invoices/images/34d4c874dd87752d683e4763e9333559/page_12.png"", ""fcc_invoices/images/34d4c874dd87752d683e4763e9333559/page_13.png""]","[{""label"": ""Advertiser"", ""start"": 182, ""end"": 203, ""text"": ""POLI/M HEINZ/D/CON/AZ""}, {""label"": ""Agency"", ""start"": 239, ""end"": 255, ""text"": ""BUYING TIME, LLC""}, {""label"": ""Gross Total"", ""start"": 581, ""end"": 587, ""text"": ""24,980""}, {""label"": ""Payment Terms"", ""start"": 688, ""end"": 703, ""text"": ""CASH IN ADVANCE""}, {""label"": ""Advertiser"", ""start"": 720, ""end"": 737, ""text"": ""HEINZ FOR ARIZONA""}, {""label"": ""Line Item - Rate"", ""start"": 955, ""end"": 961, ""text"": ""105.00""}, {""label"": ""Line Item - Description"", ""start"": 974, ""end"": 982, ""text"": ""THE TALK""}, {""label"": ""Line Item - Rate"", ""start"": 1021, ""end"": 1027, ""text"": ""375.00""}, {""label"": ""Line Item - Description"", ""start"": 1039, ""end"": 1052, ""text"": ""NEWS 13 AT 10""}, {""label"": ""Line Item - Rate"", ""start"": 1097, ""end"": 1103, ""text"": ""275.00""}, {""label"": ""Line Item - Description"", ""start"": 1115, ""end"": 1129, ""text"": ""PRICE IS RIGHT""}, {""label"": ""Line Item - Start Date"", ""start"": 1156, ""end"": 1160, ""text"": ""9/23""}, {""label"": ""Line Item - End Date"", ""start"": 1161, ""end"": 1165, ""text"": ""9/23""}, {""label"": ""Line Item - Days"", ""start"": 1168, ""end"": 1171, ""text"": ""FRI""}, {""label"": ""Line Item - Start Date"", ""start"": 1174, ""end"": 1178, ""text"": ""9/23""}, {""label"": ""Line Item - End Date"", ""start"": 1179, ""end"": 1183, ""text"": ""9/23""}, {""label"": ""Line Item - Days"", ""start"": 1186, ""end"": 1189, ""text"": ""FRI""}, {""label"": ""Line Item - Start Date"", ""start"": 1192, ""end"": 1196, ""text"": ""9/23""}, {""label"": ""Line Item - End Date"", ""start"": 1197, ""end"": 1201, ""text"": ""9/23""}, {""label"": ""Line Item - Days"", ""start"": 1204, ""end"": 1207, ""text"": ""FRI""}, {""label"": ""Line Item - Description"", ""start"": 12003, ""end"": 12020, ""text"": ""KOLD AM NEWS 6-7A""}, {""label"": ""Line Item - Rate"", ""start"": 12069, ""end"": 12075, ""text"": ""185.00""}, {""label"": ""Line Item - Start Date"", ""start"": 12076, ""end"": 12080, ""text"": ""9/20""}, {""label"": ""Line Item - End Date"", ""start"": 12081, ""end"": 12085, ""text"": ""9/20""}, {""label"": ""Line Item - Days"", ""start"": 12088, ""end"": 12091, ""text"": ""TUE""}, {""label"": ""Line Item - Rate"", ""start"": 12098, ""end"": 12104, ""text"": ""325.00""}, {""label"": ""Line Item - Start Date"", ""start"": 12105, ""end"": 12109, ""text"": ""9/20""}, {""label"": ""Line Item - End Date"", ""start"": 12110, ""end"": 12114, ""text"": ""9/20""}, {""label"": ""Line Item - Days"", ""start"": 12117, ""end"": 12120, ""text"": ""TUE""}, {""label"": ""Line Item - Description"", ""start"": 12134, ""end"": 12155, ""text"": ""ENTERTAINMENT TONIGHT""}, {""label"": ""Line Item - Description"", ""start"": 12214, ""end"": 12230, ""text"": ""CBS THIS MORNING""}, {""label"": ""Line Item - Description"", ""start"": 12284, ""end"": 12288, ""text"": ""BULL""}, {""label"": ""Line Item - Description"", ""start"": 12331, ""end"": 12347, ""text"": ""NCIS NEW ORLEANS""}, {""label"": ""Line Item - Description"", ""start"": 12401, ""end"": 12418, ""text"": ""RIGHT THIS MINUTE""}, {""label"": ""Line Item - Description"", ""start"": 12474, ""end"": 12491, ""text"": ""RIGHT THIS MINUTE""}, {""label"": ""Line Item - Description"", ""start"": 12545, ""end"": 12553, ""text"": ""THE TALK""}, {""label"": ""Line Item - Description"", ""start"": 12599, ""end"": 12612, ""text"": ""NEWS 13 AT 10""}, {""label"": ""Line Item - Rate"", ""start"": 12642, ""end"": 12648, ""text"": ""125.00""}, {""label"": ""Line Item - Start Date"", ""start"": 12649, ""end"": 12653, ""text"": ""9/20""}, {""label"": ""Line Item - End Date"", ""start"": 12654, ""end"": 12658, ""text"": ""9/20""}, {""label"": ""Line Item - Days"", ""start"": 12661, ""end"": 12664, ""text"": ""TUE""}, {""label"": ""Line Item - Rate"", ""start"": 12671, ""end"": 12680, ""text"": ""1',100.00""}, {""label"": ""Line Item - Start Date"", ""start"": 12681, ""end"": 12685, ""text"": ""9/20""}, {""label"": ""Line Item - End Date"", ""start"": 12686, ""end"": 12690, ""text"": ""9/20""}, {""label"": ""Line Item - Days"", ""start"": 12693, ""end"": 12696, ""text"": ""TUE""}, {""label"": ""Line Item - Rate"", ""start"": 12702, ""end"": 12710, ""text"": ""1,100.00""}, {""label"": ""Line Item - Start Date"", ""start"": 12711, ""end"": 12715, ""text"": ""9/20""}, {""label"": ""Line Item - End Date"", ""start"": 12716, ""end"": 12720, ""text"": ""9/20""}, {""label"": ""Line Item - Days"", ""start"": 12723, ""end"": 12726, ""text"": ""TUE""}, {""label"": ""Line Item - Rate"", ""start"": 12733, ""end"": 12738, ""text"": ""60.00""}, {""label"": ""Line Item - Start Date"", ""start"": 12739, ""end"": 12743, ""text"": ""9/20""}, {""label"": ""Line Item - End Date"", ""start"": 12744, ""end"": 12748, ""text"": ""9/20""}, {""label"": ""Line Item - Days"", ""start"": 12751, ""end"": 12754, ""text"": ""TUE""}, {""label"": ""Line Item - Rate"", ""start"": 12762, ""end"": 12767, ""text"": ""60.00""}, {""label"": ""Line Item - Start Date"", ""start"": 12768, ""end"": 12772, ""text"": ""9/20""}, {""label"": ""Line Item - End Date"", ""start"": 12773, ""end"": 12777, ""text"": ""9/20""}, {""label"": ""Line Item - Days"", ""start"": 12780, ""end"": 12783, ""text"": ""TUE""}, {""label"": ""Line Item - Rate"", ""start"": 12790, ""end"": 12796, ""text"": ""105.00""}, {""label"": ""Line Item - Start Date"", ""start"": 12797, ""end"": 12801, ""text"": ""9/21""}, {""label"": ""Line Item - End Date"", ""start"": 12802, ""end"": 12806, ""text"": ""9/21""}, {""label"": ""Line Item - Days"", ""start"": 12809, ""end"": 12812, ""text"": ""WED""}, {""label"": ""Line Item - Rate"", ""start"": 12820, ""end"": 12826, ""text"": ""375.00""}, {""label"": ""Line Item - Start Date"", ""start"": 12827, ""end"": 12831, ""text"": ""9/21""}, {""label"": ""Line Item - End Date"", ""start"": 12832, ""end"": 12836, ""text"": ""9/21""}, {""label"": ""Line Item - Days"", ""start"": 12839, ""end"": 12842, ""text"": ""WED""}, {""label"": ""Line Item - Rate"", ""start"": 15505, ""end"": 15510, ""text"": ""60.00""}, {""label"": ""Line Item - Start Date"", ""start"": 15511, ""end"": 15515, ""text"": ""9/21""}, {""label"": ""Line Item - End Date"", ""start"": 15516, ""end"": 15520, ""text"": ""9/21""}, {""label"": ""Line Item - Days"", ""start"": 15523, ""end"": 15526, ""text"": ""WED""}, {""label"": ""Line Item - Description"", ""start"": 15540, ""end"": 15557, ""text"": ""RIGHT THIS MINUTE""}, {""label"": ""Gross Total"", ""start"": 15782, ""end"": 15790, ""text"": ""24980.00""}, {""label"": ""Line Item - Rate"", ""start"": 14325, ""end"": 14331, ""text"": ""315.00""}, {""label"": ""Line Item - Description"", ""start"": 14343, ""end"": 14355, ""text"": ""NEWS 13 AT 5""}, {""label"": ""Line Item - Rate"", ""start"": 14398, ""end"": 14404, ""text"": ""100.00""}, {""label"": ""Line Item - Description"", ""start"": 14416, ""end"": 14433, ""text"": ""KOLD AM NEWS 5-6A""}, {""label"": ""Line Item - Rate"", ""start"": 14481, ""end"": 14487, ""text"": ""290.00""}, {""label"": ""Line Item - Description"", ""start"": 14499, ""end"": 14513, ""text"": ""NEWS 13 AT 6PM""}, {""label"": ""Line Item - Rate"", ""start"": 14558, ""end"": 14564, ""text"": ""185.00""}, {""label"": ""Line Item - Description"", ""start"": 14576, ""end"": 14593, ""text"": ""KOLD AM NEWS 6-7A""}, {""label"": ""Line Item - Rate"", ""start"": 14641, ""end"": 14647, ""text"": ""325.00""}, {""label"": ""Line Item - Description"", ""start"": 14659, ""end"": 14680, ""text"": ""ENTERTAINMENT TONIGHT""}, {""label"": ""Line Item - Rate"", ""start"": 14732, ""end"": 14738, ""text"": ""125_00""}, {""label"": ""Line Item - Description"", ""start"": 14750, ""end"": 14766, ""text"": ""CBS THIS MORNING""}, {""label"": ""Line Item - Rate"", ""start"": 14813, ""end"": 14821, ""text"": ""1,100.00""}, {""label"": ""Line Item - Description"", ""start"": 14833, ""end"": 14847, ""text"": ""CRIMINAL MINDS""}, {""label"": ""Line Item - Rate"", ""start"": 14893, ""end"": 14899, ""text"": ""675.00""}, {""label"": ""Line Item - Description"", ""start"": 14911, ""end"": 14921, ""text"": ""CODE BLACK""}, {""label"": ""Line Item - Rate"", ""start"": 14963, ""end"": 14968, ""text"": ""60.00""}, {""label"": ""Line Item - Description"", ""start"": 14980, ""end"": 14997, ""text"": ""RIGHT THIS MINUTE""}, {""label"": ""Line Item - Start Date"", ""start"": 15027, ""end"": 15031, ""text"": ""9/21""}, {""label"": ""Line Item - End Date"", ""start"": 15032, ""end"": 15036, ""text"": ""9/21""}, {""label"": ""Line Item - Days"", ""start"": 15039, ""end"": 15042, ""text"": ""WED""}, {""label"": ""Line Item - Start Date"", ""start"": 15045, ""end"": 15049, ""text"": ""9/21""}, {""label"": ""Line Item - End Date"", ""start"": 15050, ""end"": 15054, ""text"": ""9/21""}, {""label"": ""Line Item - Days"", ""start"": 15057, ""end"": 15060, ""text"": ""WED""}, {""label"": ""Line Item - Start Date"", ""start"": 15063, ""end"": 15067, ""text"": ""9/21""}, {""label"": ""Line Item - End Date"", ""start"": 15068, ""end"": 15072, ""text"": ""9/21""}, {""label"": ""Line Item - Days"", ""start"": 15075, ""end"": 15078, ""text"": ""WED""}, {""label"": ""Line Item - Start Date"", ""start"": 15082, ""end"": 15086, ""text"": ""9/21""}, {""label"": ""Line Item - End Date"", ""start"": 15087, ""end"": 15091, ""text"": ""9/21""}, {""label"": ""Line Item - Days"", ""start"": 15094, ""end"": 15097, ""text"": ""WED""}, {""label"": ""Line Item - Start Date"", ""start"": 15100, ""end"": 15104, ""text"": ""9/21""}, {""label"": ""Line Item - End Date"", ""start"": 15105, ""end"": 15109, ""text"": ""9/21""}, {""label"": ""Line Item - Days"", ""start"": 15112, ""end"": 15115, ""text"": ""WED""}, {""label"": ""Line Item - Start Date"", ""start"": 15118, ""end"": 15122, ""text"": ""9/21""}, {""label"": ""Line Item - End Date"", ""start"": 15123, ""end"": 15127, ""text"": ""9/21""}, {""label"": ""Line Item - Days"", ""start"": 15130, ""end"": 15133, ""text"": ""WED""}, {""label"": ""Line Item - Start Date"", ""start"": 15137, ""end"": 15141, ""text"": ""9/21""}, {""label"": ""Line Item - End Date"", ""start"": 15142, ""end"": 15146, ""text"": ""9/21""}, {""label"": ""Line Item - Days"", ""start"": 15149, ""end"": 15152, ""text"": ""WED""}, {""label"": ""Line Item - Start Date"", ""start"": 15155, ""end"": 15159, ""text"": ""9/21""}, {""label"": ""Line Item - End Date"", ""start"": 15160, ""end"": 15164, ""text"": ""9/21""}, {""label"": ""Line Item - Days"", ""start"": 15167, ""end"": 15170, ""text"": ""WED""}, {""label"": ""Line Item - Start Date"", ""start"": 15173, ""end"": 15177, ""text"": ""9/21""}, {""label"": ""Line Item - End Date"", ""start"": 15178, ""end"": 15182, ""text"": ""9/21""}, {""label"": ""Line Item - Days"", ""start"": 15185, ""end"": 15188, ""text"": ""WED""}, {""label"": ""Line Item - Description"", ""start"": 13156, ""end"": 13170, ""text"": ""PRICE IS RIGHT""}, {""label"": ""Line Item - Rate"", ""start"": 13218, ""end"": 13224, ""text"": ""275.00""}, {""label"": ""Line Item - Start Date"", ""start"": 13225, ""end"": 13229, ""text"": ""9/21""}, {""label"": ""Line Item - End Date"", ""start"": 13230, ""end"": 13234, ""text"": ""9/21""}, {""label"": ""Line Item - Days"", ""start"": 13237, ""end"": 13240, ""text"": ""WED""}, {""label"": ""Line Item - Rate"", ""start"": 13248, ""end"": 13254, ""text"": ""165.00""}, {""label"": ""Line Item - Start Date"", ""start"": 13255, ""end"": 13259, ""text"": ""9/21""}, {""label"": ""Line Item - End Date"", ""start"": 13260, ""end"": 13264, ""text"": ""9/21""}, {""label"": ""Line Item - Days"", ""start"": 13267, ""end"": 13270, ""text"": ""WED""}, {""label"": ""Line Item - Description"", ""start"": 13284, ""end"": 13303, ""text"": ""LATE SHOW - COLBERT""}, {""label"": ""Line Item - Rate"", ""start"": 13355, ""end"": 13360, ""text"": ""90.00""}, {""label"": ""Line Item - Start Date"", ""start"": 13361, ""end"": 13365, ""text"": ""9/21""}, {""label"": ""Line Item - End Date"", ""start"": 13366, ""end"": 13370, ""text"": ""9/21""}, {""label"": ""Line Item - Days"", ""start"": 13373, ""end"": 13376, ""text"": ""WED""}, {""label"": ""Line Item - Description"", ""start"": 13390, ""end"": 13404, ""text"": ""LATE LATE SHOW""}, {""label"": ""Line Item - Rate"", ""start"": 13450, ""end"": 13456, ""text"": ""125.00""}, {""label"": ""Line Item - Start Date"", ""start"": 13457, ""end"": 13461, ""text"": ""9/21""}, {""label"": ""Line Item - End Date"", ""start"": 13462, ""end"": 13466, ""text"": ""9/21""}, {""label"": ""Line Item - Days"", ""start"": 13469, ""end"": 13472, ""text"": ""WED""}, {""label"": ""Line Item - Description"", ""start"": 13486, ""end"": 13502, ""text"": ""YOUNG & RESTLESS""}, {""label"": ""Line Item - Rate"", ""start"": 13550, ""end"": 13556, ""text"": ""125.00""}, {""label"": ""Line Item - Start Date"", ""start"": 13557, ""end"": 13561, ""text"": ""9/21""}, {""label"": ""Line Item - End Date"", ""start"": 13562, ""end"": 13566, ""text"": ""9/21""}, {""label"": ""Line Item - Days"", ""start"": 13569, ""end"": 13572, ""text"": ""WED""}, {""label"": ""Line Item - Description"", ""start"": 13586, ""end"": 13601, ""text"": ""NEWS 13 AT NOON""}, {""label"": ""Line Item - Rate"", ""start"": 13646, ""end"": 13652, ""text"": ""105.00""}, {""label"": ""Line Item - Start Date"", ""start"": 13653, ""end"": 13657, ""text"": ""9/21""}, {""label"": ""Line Item - End Date"", ""start"": 13658, ""end"": 13662, ""text"": ""9/21""}, {""label"": ""Line Item - Days"", ""start"": 13665, ""end"": 13668, ""text"": ""WED""}, {""label"": ""Line Item - Description"", ""start"": 13682, ""end"": 13699, ""text"": ""LET'S MAKE A DEAL""}, {""label"": ""Line Item - Rate"", ""start"": 13747, ""end"": 13752, ""text"": ""60.00""}, {""label"": ""Line Item - Start Date"", ""start"": 13753, ""end"": 13757, ""text"": ""9/21""}, {""label"": ""Line Item - End Date"", ""start"": 13758, ""end"": 13762, ""text"": ""9/21""}, {""label"": ""Line Item - Days"", ""start"": 13765, ""end"": 13768, ""text"": ""WED""}, {""label"": ""Line Item - Description"", ""start"": 13782, ""end"": 13797, ""text"": ""CRIMEWTCH DAILY""}, {""label"": ""Line Item - Rate"", ""start"": 13843, ""end"": 13849, ""text"": ""125.00""}, {""label"": ""Line Item - Start Date"", ""start"": 13850, ""end"": 13854, ""text"": ""9/21""}, {""label"": ""Line Item - End Date"", ""start"": 13855, ""end"": 13859, ""text"": ""9/21""}, {""label"": ""Line Item - Days"", ""start"": 13862, ""end"": 13865, ""text"": ""WED""}, {""label"": ""Line Item - Description"", ""start"": 13879, ""end"": 13893, ""text"": ""NEWS 13 AT 4PM""}, {""label"": ""Line Item - Rate"", ""start"": 13939, ""end"": 13944, ""text"": ""40.00""}, {""label"": ""Line Item - Start Date"", ""start"": 13945, ""end"": 13949, ""text"": ""9/21""}, {""label"": ""Line Item - End Date"", ""start"": 13950, ""end"": 13954, ""text"": ""9/21""}, {""label"": ""Line Item - Days"", ""start"": 13957, ""end"": 13960, ""text"": ""WED""}, {""label"": ""Line Item - Description"", ""start"": 13974, ""end"": 13992, ""text"": ""KOLD AM NEWS 430AM""}, {""label"": ""Line Item - Rate"", ""start"": 10841, ""end"": 10847, ""text"": ""135.00""}, {""label"": ""Line Item - Start Date"", ""start"": 10848, ""end"": 10852, ""text"": ""9/20""}, {""label"": ""Line Item - End Date"", ""start"": 10853, ""end"": 10857, ""text"": ""9/20""}, {""label"": ""Line Item - Days"", ""start"": 10860, ""end"": 10863, ""text"": ""TUE""}, {""label"": ""Line Item - Description"", ""start"": 10877, ""end"": 10893, ""text"": ""BOLD & BEAUTIFUL""}, {""label"": ""Line Item - Rate"", ""start"": 10941, ""end"": 10947, ""text"": ""125.00""}, {""label"": ""Line Item - Start Date"", ""start"": 10948, ""end"": 10952, ""text"": ""9/20""}, {""label"": ""Line Item - End Date"", ""start"": 10953, ""end"": 10957, ""text"": ""9/20""}, {""label"": ""Line Item - Days"", ""start"": 10960, ""end"": 10963, ""text"": ""TUE""}, {""label"": ""Line Item - Description"", ""start"": 10977, ""end"": 10992, ""text"": ""NEWS 13 AT NOON""}, {""label"": ""Line Item - Rate"", ""start"": 11037, ""end"": 11043, ""text"": ""105.00""}, {""label"": ""Line Item - Start Date"", ""start"": 11044, ""end"": 11048, ""text"": ""9/20""}, {""label"": ""Line Item - End Date"", ""start"": 11049, ""end"": 11053, ""text"": ""9/20""}, {""label"": ""Line Item - Days"", ""start"": 11056, ""end"": 11059, ""text"": ""TUE""}, {""label"": ""Line Item - Description"", ""start"": 11073, ""end"": 11090, ""text"": ""LET'S MAKE A DEAL""}, {""label"": ""Line Item - Rate"", ""start"": 11137, ""end"": 11142, ""text"": ""60.00""}, {""label"": ""Line Item - Start Date"", ""start"": 11143, ""end"": 11147, ""text"": ""9/20""}, {""label"": ""Line Item - End Date"", ""start"": 11148, ""end"": 11152, ""text"": ""9/20""}, {""label"": ""Line Item - Days"", ""start"": 11155, ""end"": 11158, ""text"": ""TUE""}, {""label"": ""Line Item - Description"", ""start"": 11170, ""end"": 11185, ""text"": ""CRIMEWTCH DAILY""}, {""label"": ""Line Item - Rate"", ""start"": 11230, ""end"": 11236, ""text"": ""125.00""}, {""label"": ""Line Item - Start Date"", ""start"": 11237, ""end"": 11241, ""text"": ""9/20""}, {""label"": ""Line Item - End Date"", ""start"": 11242, ""end"": 11246, ""text"": ""9/20""}, {""label"": ""Line Item - Days"", ""start"": 11249, ""end"": 11252, ""text"": ""TUE""}, {""label"": ""Line Item - Description"", ""start"": 11266, ""end"": 11280, ""text"": ""NEWS 13 AT 4PM""}, {""label"": ""Line Item - Rate"", ""start"": 11324, ""end"": 11329, ""text"": ""40.00""}, {""label"": ""Line Item - Start Date"", ""start"": 11330, ""end"": 11334, ""text"": ""9/20""}, {""label"": ""Line Item - End Date"", ""start"": 11335, ""end"": 11339, ""text"": ""9/20""}, {""label"": ""Line Item - Days"", ""start"": 11342, ""end"": 11345, ""text"": ""TUE""}, {""label"": ""Line Item - Description"", ""start"": 11359, ""end"": 11377, ""text"": ""KOLD AM NEWS 430AM""}, {""label"": ""Line Item - Rate"", ""start"": 11425, ""end"": 11431, ""text"": ""315.00""}, {""label"": ""Line Item - Start Date"", ""start"": 11432, ""end"": 11436, ""text"": ""9/20""}, {""label"": ""Line Item - End Date"", ""start"": 11437, ""end"": 11441, ""text"": ""9/20""}, {""label"": ""Line Item - Days"", ""start"": 11444, ""end"": 11447, ""text"": ""TUE""}, {""label"": ""Line Item - Description"", ""start"": 11461, ""end"": 11473, ""text"": ""NEWS 13 AT 5""}, {""label"": ""Line Item - Rate"", ""start"": 11515, ""end"": 11521, ""text"": ""100.00""}, {""label"": ""Line Item - Start Date"", ""start"": 11522, ""end"": 11526, ""text"": ""9/20""}, {""label"": ""Line Item - End Date"", ""start"": 11527, ""end"": 11531, ""text"": ""9/20""}, {""label"": ""Line Item - Days"", ""start"": 11534, ""end"": 11537, ""text"": ""TUE""}, {""label"": ""Line Item - Description"", ""start"": 11551, ""end"": 11568, ""text"": ""KOLD AM NEWS 5-6A""}, {""label"": ""Line Item - Rate"", ""start"": 11616, ""end"": 11622, ""text"": ""290.00""}, {""label"": ""Line Item - Start Date"", ""start"": 11623, ""end"": 11627, ""text"": ""9/20""}, {""label"": ""Line Item - End Date"", ""start"": 11628, ""end"": 11632, ""text"": ""9/20""}, {""label"": ""Line Item - Days"", ""start"": 11635, ""end"": 11638, ""text"": ""TUE""}, {""label"": ""Line Item - Description"", ""start"": 11652, ""end"": 11666, ""text"": ""NEWS 13 AT 6PM""}, {""label"": ""Line Item - Description"", ""start"": 9676, ""end"": 9692, ""text"": ""CBS THIS MORNING""}, {""label"": ""Line Item - Description"", ""start"": 9746, ""end"": 9763, ""text"": ""RIGHT THIS MINUTE""}, {""label"": ""Line Item - Description"", ""start"": 9819, ""end"": 9836, ""text"": ""RIGHT THIS MINUTE""}, {""label"": ""Line Item - Description"", ""start"": 9891, ""end"": 9899, ""text"": ""THE TALK""}, {""label"": ""Line Item - Description"", ""start"": 9947, ""end"": 9960, ""text"": ""NEWS 13 AT 10""}, {""label"": ""Line Item - Description"", ""start"": 10013, ""end"": 10027, ""text"": ""PRICE IS RIGHT""}, {""label"": ""Line Item - Rate"", ""start"": 10075, ""end"": 10081, ""text"": ""125.00""}, {""label"": ""Line Item - Start Date"", ""start"": 10082, ""end"": 10086, ""text"": ""9/22""}, {""label"": ""Line Item - End Date"", ""start"": 10087, ""end"": 10091, ""text"": ""9/22""}, {""label"": ""Line Item - Days"", ""start"": 10094, ""end"": 10097, ""text"": ""THU""}, {""label"": ""Line Item - Rate"", ""start"": 10104, ""end"": 10109, ""text"": ""60.00""}, {""label"": ""Line Item - Start Date"", ""start"": 10110, ""end"": 10114, ""text"": ""9/22""}, {""label"": ""Line Item - End Date"", ""start"": 10115, ""end"": 10119, ""text"": ""9/22""}, {""label"": ""Line Item - Days"", ""start"": 10122, ""end"": 10125, ""text"": ""THU""}, {""label"": ""Line Item - Rate"", ""start"": 10133, ""end"": 10138, ""text"": ""60.00""}, {""label"": ""Line Item - Start Date"", ""start"": 10139, ""end"": 10143, ""text"": ""9/22""}, {""label"": ""Line Item - End Date"", ""start"": 10144, ""end"": 10148, ""text"": ""9/22""}, {""label"": ""Line Item - Days"", ""start"": 10151, ""end"": 10154, ""text"": ""THU""}, {""label"": ""Line Item - Rate"", ""start"": 10161, ""end"": 10167, ""text"": ""105.00""}, {""label"": ""Line Item - Start Date"", ""start"": 10168, ""end"": 10172, ""text"": ""9/20""}, {""label"": ""Line Item - End Date"", ""start"": 10173, ""end"": 10177, ""text"": ""9/20""}, {""label"": ""Line Item - Days"", ""start"": 10180, ""end"": 10183, ""text"": ""TUE""}, {""label"": ""Line Item - Rate"", ""start"": 10191, ""end"": 10197, ""text"": ""375.00""}, {""label"": ""Line Item - Start Date"", ""start"": 10198, ""end"": 10202, ""text"": ""9/20""}, {""label"": ""Line Item - End Date"", ""start"": 10203, ""end"": 10207, ""text"": ""9/20""}, {""label"": ""Line Item - Days"", ""start"": 10210, ""end"": 10213, ""text"": ""TUE""}, {""label"": ""Line Item - Rate"", ""start"": 10220, ""end"": 10226, ""text"": ""275.00""}, {""label"": ""Line Item - Start Date"", ""start"": 10227, ""end"": 10231, ""text"": ""9/20""}, {""label"": ""Line Item - End Date"", ""start"": 10232, ""end"": 10236, ""text"": ""9/20""}, {""label"": ""Line Item - Days"", ""start"": 10239, ""end"": 10242, ""text"": ""TUE""}, {""label"": ""Line Item - Description"", ""start"": 10260, ""end"": 10279, ""text"": ""LATE SHOW - COLBERT""}, {""label"": ""Line Item - Description"", ""start"": 10338, ""end"": 10352, ""text"": ""LATE LATE SHOW""}, {""label"": ""Line Item - Description"", ""start"": 10405, ""end"": 10421, ""text"": ""YOUNG & RESTLESS""}, {""label"": ""Line Item - Rate"", ""start"": 10455, ""end"": 10461, ""text"": ""165.00""}, {""label"": ""Line Item - Start Date"", ""start"": 10462, ""end"": 10466, ""text"": ""9/20""}, {""label"": ""Line Item - End Date"", ""start"": 10467, ""end"": 10471, ""text"": ""9/20""}, {""label"": ""Line Item - Days"", ""start"": 10474, ""end"": 10477, ""text"": ""TUE""}, {""label"": ""Line Item - Rate"", ""start"": 10481, ""end"": 10486, ""text"": ""90.00""}, {""label"": ""Line Item - Start Date"", ""start"": 10487, ""end"": 10491, ""text"": ""9/20""}, {""label"": ""Line Item - End Date"", ""start"": 10492, ""end"": 10496, ""text"": ""9/20""}, {""label"": ""Line Item - Days"", ""start"": 10499, ""end"": 10502, ""text"": ""TUE""}, {""label"": ""Line Item - Rate"", ""start"": 10510, ""end"": 10516, ""text"": ""125.00""}, {""label"": ""Line Item - Start Date"", ""start"": 10517, ""end"": 10521, ""text"": ""9/20""}, {""label"": ""Line Item - End Date"", ""start"": 10522, ""end"": 10526, ""text"": ""9/20""}, {""label"": ""Line Item - Days"", ""start"": 10529, ""end"": 10532, ""text"": ""TUE""}, {""label"": ""Line Item - Rate"", ""start"": 8497, ""end"": 8503, ""text"": ""105.00""}, {""label"": ""Line Item - Description"", ""start"": 8515, ""end"": 8532, ""text"": ""LET'S MAKE A DEAL""}, {""label"": ""Line Item - Rate"", ""start"": 8579, ""end"": 8584, ""text"": ""60.00""}, {""label"": ""Line Item - Description"", ""start"": 8596, ""end"": 8611, ""text"": ""CRIMEWTCH DAILY""}, {""label"": ""Line Item - Rate"", ""start"": 8656, ""end"": 8662, ""text"": ""125.00""}, {""label"": ""Line Item - Description"", ""start"": 8674, ""end"": 8688, ""text"": ""NEWS 13 AT 4PM""}, {""label"": ""Line Item - Rate"", ""start"": 8732, ""end"": 8737, ""text"": ""40.00""}, {""label"": ""Line Item - Description"", ""start"": 8749, ""end"": 8767, ""text"": ""KOLD AM NEWS 430AM""}, {""label"": ""Line Item - Rate"", ""start"": 8815, ""end"": 8821, ""text"": ""315.00""}, {""label"": ""Line Item - Description"", ""start"": 8833, ""end"": 8845, ""text"": ""NEWS 13 AT 5""}, {""label"": ""Line Item - Rate"", ""start"": 8887, ""end"": 8893, ""text"": ""100.00""}, {""label"": ""Line Item - Description"", ""start"": 8905, ""end"": 8922, ""text"": ""KOLD AM NEWS 5-6A""}, {""label"": ""Line Item - Rate"", ""start"": 8969, ""end"": 8975, ""text"": ""290.00""}, {""label"": ""Line Item - Description"", ""start"": 8987, ""end"": 9001, ""text"": ""NEWS 13 AT 6PM""}, {""label"": ""Line Item - Rate"", ""start"": 9045, ""end"": 9051, ""text"": ""185.00""}, {""label"": ""Line Item - Description"", ""start"": 9063, ""end"": 9080, ""text"": ""KOLD AM NEWS 6-7A""}, {""label"": ""Line Item - Rate"", ""start"": 9128, ""end"": 9134, ""text"": ""325.00""}, {""label"": ""Line Item - Description"", ""start"": 9146, ""end"": 9167, ""text"": ""ENTERTAINMENT TONIGHT""}, {""label"": ""Line Item - Start Date"", ""start"": 9201, ""end"": 9205, ""text"": ""9/22""}, {""label"": ""Line Item - End Date"", ""start"": 9206, ""end"": 9210, ""text"": ""9/22""}, {""label"": ""Line Item - Days"", ""start"": 9213, ""end"": 9216, ""text"": ""THU""}, {""label"": ""Line Item - Start Date"", ""start"": 9219, ""end"": 9223, ""text"": ""9/22""}, {""label"": ""Line Item - End Date"", ""start"": 9224, ""end"": 9228, ""text"": ""9/22""}, {""label"": ""Line Item - Days"", ""start"": 9231, ""end"": 9234, ""text"": ""THU""}, {""label"": ""Line Item - Start Date"", ""start"": 9237, ""end"": 9241, ""text"": ""9/22""}, {""label"": ""Line Item - End Date"", ""start"": 9242, ""end"": 9246, ""text"": ""9/22""}, {""label"": ""Line Item - Days"", ""start"": 9249, ""end"": 9252, ""text"": ""THU""}, {""label"": ""Line Item - Start Date"", ""start"": 9256, ""end"": 9260, ""text"": ""9/22""}, {""label"": ""Line Item - End Date"", ""start"": 9261, ""end"": 9265, ""text"": ""9/22""}, {""label"": ""Line Item - Days"", ""start"": 9268, ""end"": 9271, ""text"": ""THU""}, {""label"": ""Line Item - Start Date"", ""start"": 9274, ""end"": 9278, ""text"": ""9/22""}, {""label"": ""Line Item - End Date"", ""start"": 9279, ""end"": 9283, ""text"": ""9/22""}, {""label"": ""Line Item - Days"", ""start"": 9286, ""end"": 9289, ""text"": ""THU""}, {""label"": ""Line Item - Start Date"", ""start"": 9292, ""end"": 9296, ""text"": ""9/22""}, {""label"": ""Line Item - End Date"", ""start"": 9297, ""end"": 9301, ""text"": ""9/22""}, {""label"": ""Line Item - Days"", ""start"": 9304, ""end"": 9307, ""text"": ""THU""}, {""label"": ""Line Item - Start Date"", ""start"": 9311, ""end"": 9315, ""text"": ""9/22""}, {""label"": ""Line Item - End Date"", ""start"": 9316, ""end"": 9320, ""text"": ""9/22""}, {""label"": ""Line Item - Days"", ""start"": 9323, ""end"": 9326, ""text"": ""THU""}, {""label"": ""Line Item - Start Date"", ""start"": 9329, ""end"": 9333, ""text"": ""9/22""}, {""label"": ""Line Item - End Date"", ""start"": 9334, ""end"": 9338, ""text"": ""9/22""}, {""label"": ""Line Item - Days"", ""start"": 9341, ""end"": 9344, ""text"": ""THU""}, {""label"": ""Line Item - Start Date"", ""start"": 9347, ""end"": 9351, ""text"": ""9/22""}, {""label"": ""Line Item - End Date"", ""start"": 9352, ""end"": 9356, ""text"": ""9/22""}, {""label"": ""Line Item - Days"", ""start"": 9359, ""end"": 9362, ""text"": ""THU""}, {""label"": ""Line Item - Description"", ""start"": 7347, ""end"": 7354, ""text"": ""NCIS LA""}, {""label"": ""Line Item - Description"", ""start"": 7399, ""end"": 7407, ""text"": ""THE TALK""}, {""label"": ""Line Item - Description"", ""start"": 7455, ""end"": 7468, ""text"": ""NEWS 13 AT 10""}, {""label"": ""Line Item - Description"", ""start"": 7521, ""end"": 7535, ""text"": ""PRICE IS RIGHT""}, {""label"": ""Line Item - Rate"", ""start"": 7579, ""end"": 7585, ""text"": ""675_00""}, {""label"": ""Line Item - Rate"", ""start"": 7587, ""end"": 7593, ""text"": ""105.00""}, {""label"": ""Line Item - Rate"", ""start"": 7595, ""end"": 7601, ""text"": ""375.00""}, {""label"": ""Line Item - Rate"", ""start"": 7604, ""end"": 7610, ""text"": ""275.00""}, {""label"": ""Line Item - Rate"", ""start"": 7632, ""end"": 7638, ""text"": ""165.00""}, {""label"": ""Line Item - Description"", ""start"": 7651, ""end"": 7670, ""text"": ""LATE SHOW - COLBERT""}, {""label"": ""Line Item - Rate"", ""start"": 7721, ""end"": 7726, ""text"": ""90.00""}, {""label"": ""Line Item - Description"", ""start"": 7738, ""end"": 7752, ""text"": ""LATE LATE SHOW""}, {""label"": ""Line Item - Rate"", ""start"": 7798, ""end"": 7804, ""text"": ""125.00""}, {""label"": ""Line Item - Description"", ""start"": 7816, ""end"": 7832, ""text"": ""YOUNG & RESTLESS""}, {""label"": ""Line Item - Rate"", ""start"": 7879, ""end"": 7885, ""text"": ""135.00""}, {""label"": ""Line Item - Description"", ""start"": 7897, ""end"": 7913, ""text"": ""BOLD & BEAUTIFUL""}, {""label"": ""Line Item - Rate"", ""start"": 7962, ""end"": 7968, ""text"": ""125.00""}, {""label"": ""Line Item - Description"", ""start"": 7980, ""end"": 7995, ""text"": ""NEWS 13 AT NOON""}, {""label"": ""Line Item - Start Date"", ""start"": 8023, ""end"": 8027, ""text"": ""9/25""}, {""label"": ""Line Item - End Date"", ""start"": 8028, ""end"": 8032, ""text"": ""9/25""}, {""label"": ""Line Item - Days"", ""start"": 8035, ""end"": 8038, ""text"": ""SUN""}, {""label"": ""Line Item - Start Date"", ""start"": 8041, ""end"": 8045, ""text"": ""9/22""}, {""label"": ""Line Item - End Date"", ""start"": 8046, ""end"": 8050, ""text"": ""9/22""}, {""label"": ""Line Item - Days"", ""start"": 8053, ""end"": 8056, ""text"": ""THU""}, {""label"": ""Line Item - Start Date"", ""start"": 8059, ""end"": 8063, ""text"": ""9/22""}, {""label"": ""Line Item - End Date"", ""start"": 8064, ""end"": 8068, ""text"": ""9/22""}, {""label"": ""Line Item - Days"", ""start"": 8071, ""end"": 8074, ""text"": ""THU""}, {""label"": ""Line Item - Start Date"", ""start"": 8078, ""end"": 8082, ""text"": ""9/22""}, {""label"": ""Line Item - End Date"", ""start"": 8083, ""end"": 8087, ""text"": ""9/22""}, {""label"": ""Line Item - Days"", ""start"": 8090, ""end"": 8093, ""text"": ""THU""}, {""label"": ""Line Item - Start Date"", ""start"": 8096, ""end"": 8100, ""text"": ""9/22""}, {""label"": ""Line Item - End Date"", ""start"": 8101, ""end"": 8105, ""text"": ""9/22""}, {""label"": ""Line Item - Days"", ""start"": 8108, ""end"": 8111, ""text"": ""THU""}, {""label"": ""Line Item - Start Date"", ""start"": 8114, ""end"": 8118, ""text"": ""9/22""}, {""label"": ""Line Item - End Date"", ""start"": 8119, ""end"": 8123, ""text"": ""9/22""}, {""label"": ""Line Item - Days"", ""start"": 8126, ""end"": 8129, ""text"": ""THU""}, {""label"": ""Line Item - Start Date"", ""start"": 8133, ""end"": 8137, ""text"": ""9/22""}, {""label"": ""Line Item - End Date"", ""start"": 8138, ""end"": 8142, ""text"": ""9/22""}, {""label"": ""Line Item - Days"", ""start"": 8145, ""end"": 8148, ""text"": ""THU""}, {""label"": ""Line Item - Start Date"", ""start"": 8151, ""end"": 8155, ""text"": ""9/22""}, {""label"": ""Line Item - End Date"", ""start"": 8156, ""end"": 8160, ""text"": ""9/22""}, {""label"": ""Line Item - Days"", ""start"": 8163, ""end"": 8166, ""text"": ""THU""}, {""label"": ""Line Item - Start Date"", ""start"": 8169, ""end"": 8173, ""text"": ""9/22""}, {""label"": ""Line Item - End Date"", ""start"": 8174, ""end"": 8178, ""text"": ""9/22""}, {""label"": ""Line Item - Days"", ""start"": 8181, ""end"": 8184, ""text"": ""THU""}, {""label"": ""Line Item - Description"", ""start"": 6180, ""end"": 6199, ""text"": ""PRESIDENTIAL DEBATE""}, {""label"": ""Line Item - Description"", ""start"": 6257, ""end"": 6265, ""text"": ""SCORPION""}, {""label"": ""Line Item - Rate"", ""start"": 6287, ""end"": 6293, ""text"": ""500.00""}, {""label"": ""Line Item - Start Date"", ""start"": 6294, ""end"": 6298, ""text"": ""9/26""}, {""label"": ""Line Item - End Date"", ""start"": 6300, ""end"": 6304, ""text"": ""9/26""}, {""label"": ""Line Item - Days"", ""start"": 6307, ""end"": 6310, ""text"": ""MON""}, {""label"": ""Line Item - Rate"", ""start"": 6318, ""end"": 6324, ""text"": ""775.00""}, {""label"": ""Line Item - Start Date"", ""start"": 6325, ""end"": 6329, ""text"": ""9/26""}, {""label"": ""Line Item - End Date"", ""start"": 6330, ""end"": 6334, ""text"": ""9/26""}, {""label"": ""Line Item - Days"", ""start"": 6337, ""end"": 6340, ""text"": ""MON""}, {""label"": ""Line Item - Rate"", ""start"": 6362, ""end"": 6367, ""text"": ""60.00""}, {""label"": ""Line Item - Description"", ""start"": 6379, ""end"": 6397, ""text"": ""FIX IT & FINISH IT""}, {""label"": ""Line Item - Rate"", ""start"": 6447, ""end"": 6453, ""text"": ""225.00""}, {""label"": ""Line Item - Description"", ""start"": 6465, ""end"": 6484, ""text"": ""NEWS 13 SAT AT 10PM""}, {""label"": ""Line Item - Rate"", ""start"": 6533, ""end"": 6539, ""text"": ""125.00""}, {""label"": ""Line Item - Description"", ""start"": 6551, ""end"": 6571, ""text"": ""NEWS 13 SAT AT 530PM""}, {""label"": ""Line Item - Rate"", ""start"": 6622, ""end"": 6628, ""text"": ""325.00""}, {""label"": ""Line Item - Description"", ""start"": 6640, ""end"": 6648, ""text"": ""48 HOURS""}, {""label"": ""Line Item - Rate"", ""start"": 6688, ""end"": 6694, ""text"": ""300.00""}, {""label"": ""Line Item - Description"", ""start"": 6706, ""end"": 6724, ""text"": ""NEWS 13 SUN AT 10P""}, {""label"": ""Line Item - Rate"", ""start"": 6772, ""end"": 6778, ""text"": ""200.00""}, {""label"": ""Line Item - Description"", ""start"": 6790, ""end"": 6809, ""text"": ""NEWS 13 SUN AT 530P""}, {""label"": ""Line Item - Rate"", ""start"": 6859, ""end"": 6865, ""text"": ""525.00""}, {""label"": ""Line Item - Description"", ""start"": 6876, ""end"": 6886, ""text"": ""60 MINUTES""}, {""label"": ""Line Item - Start Date"", ""start"": 6909, ""end"": 6913, ""text"": ""9/26""}, {""label"": ""Line Item - End Date"", ""start"": 6914, ""end"": 6918, ""text"": ""9/26""}, {""label"": ""Line Item - Days"", ""start"": 6921, ""end"": 6924, ""text"": ""MON""}, {""label"": ""Line Item - Start Date"", ""start"": 6928, ""end"": 6932, ""text"": ""9/24""}, {""label"": ""Line Item - End Date"", ""start"": 6933, ""end"": 6937, ""text"": ""9/24""}, {""label"": ""Line Item - Days"", ""start"": 6940, ""end"": 6943, ""text"": ""SAT""}, {""label"": ""Line Item - Start Date"", ""start"": 6946, ""end"": 6950, ""text"": ""9/24""}, {""label"": ""Line Item - End Date"", ""start"": 6951, ""end"": 6955, ""text"": ""9/24""}, {""label"": ""Line Item - Days"", ""start"": 6958, ""end"": 6961, ""text"": ""SAT""}, {""label"": ""Line Item - Start Date"", ""start"": 6964, ""end"": 6968, ""text"": ""9/24""}, {""label"": ""Line Item - End Date"", ""start"": 6969, ""end"": 6973, ""text"": ""9/24""}, {""label"": ""Line Item - Days"", ""start"": 6976, ""end"": 6979, ""text"": ""SAT""}, {""label"": ""Line Item - Start Date"", ""start"": 6983, ""end"": 6987, ""text"": ""9/25""}, {""label"": ""Line Item - End Date"", ""start"": 6988, ""end"": 6992, ""text"": ""9/25""}, {""label"": ""Line Item - Days"", ""start"": 6995, ""end"": 6998, ""text"": ""SUN""}, {""label"": ""Line Item - Start Date"", ""start"": 7001, ""end"": 7005, ""text"": ""9/25""}, {""label"": ""Line Item - End Date"", ""start"": 7006, ""end"": 7010, ""text"": ""9/25""}, {""label"": ""Line Item - Days"", ""start"": 7013, ""end"": 7016, ""text"": ""SUN""}, {""label"": ""Line Item - Start Date"", ""start"": 7019, ""end"": 7023, ""text"": ""9/25""}, {""label"": ""Line Item - End Date"", ""start"": 7024, ""end"": 7028, ""text"": ""9/25""}, {""label"": ""Line Item - Days"", ""start"": 7031, ""end"": 7034, ""text"": ""SUN""}, {""label"": ""Line Item - Rate"", ""start"": 4992, ""end"": 4998, ""text"": ""125.00""}, {""label"": ""Line Item - Description"", ""start"": 5010, ""end"": 5024, ""text"": ""NEWS 13 AT 4PM""}, {""label"": ""Line Item - Rate"", ""start"": 5068, ""end"": 5073, ""text"": ""40.00""}, {""label"": ""Line Item - Description"", ""start"": 5085, ""end"": 5103, ""text"": ""KOLD AM NEWS 430AM""}, {""label"": ""Line Item - Rate"", ""start"": 5151, ""end"": 5157, ""text"": ""315.00""}, {""label"": ""Line Item - Description"", ""start"": 5169, ""end"": 5181, ""text"": ""NEWS 13 AT 5""}, {""label"": ""Line Item - Rate"", ""start"": 5223, ""end"": 5229, ""text"": ""100.00""}, {""label"": ""Line Item - Description"", ""start"": 5241, ""end"": 5258, ""text"": ""KOLD AM NEWS 5-6A""}, {""label"": ""Line Item - Rate"", ""start"": 5305, ""end"": 5311, ""text"": ""290.00""}, {""label"": ""Line Item - Description"", ""start"": 5323, ""end"": 5337, ""text"": ""NEWS 13 AT 6PM""}, {""label"": ""Line Item - Rate"", ""start"": 5381, ""end"": 5387, ""text"": ""185.00""}, {""label"": ""Line Item - Description"", ""start"": 5399, ""end"": 5416, ""text"": ""KOLD AM NEWS 6-7A""}, {""label"": ""Line Item - Rate"", ""start"": 5463, ""end"": 5469, ""text"": ""325.00""}, {""label"": ""Line Item - Description"", ""start"": 5481, ""end"": 5502, ""text"": ""ENTERTAINMENT TONIGHT""}, {""label"": ""Line Item - Rate"", ""start"": 5553, ""end"": 5559, ""text"": ""500.00""}, {""label"": ""Line Item - Description"", ""start"": 5571, ""end"": 5590, ""text"": ""PRESIDENTIAL DEBATE""}, {""label"": ""Line Item - Rate"", ""start"": 5640, ""end"": 5646, ""text"": ""125.00""}, {""label"": ""Line Item - Description"", ""start"": 5658, ""end"": 5674, ""text"": ""CBS THIS MORNING""}, {""label"": ""Line Item - Start Date"", ""start"": 5703, ""end"": 5707, ""text"": ""9/26""}, {""label"": ""Line Item - End Date"", ""start"": 5708, ""end"": 5712, ""text"": ""9/26""}, {""label"": ""Line Item - Days"", ""start"": 5715, ""end"": 5718, ""text"": ""MON""}, {""label"": ""Line Item - Start Date"", ""start"": 5721, ""end"": 5725, ""text"": ""9/26""}, {""label"": ""Line Item - End Date"", ""start"": 5726, ""end"": 5730, ""text"": ""9/26""}, {""label"": ""Line Item - Days"", ""start"": 5733, ""end"": 5736, ""text"": ""MON""}, {""label"": ""Line Item - Start Date"", ""start"": 5739, ""end"": 5743, ""text"": ""9/26""}, {""label"": ""Line Item - End Date"", ""start"": 5744, ""end"": 5748, ""text"": ""9/26""}, {""label"": ""Line Item - Days"", ""start"": 5751, ""end"": 5754, ""text"": ""MON""}, {""label"": ""Line Item - Start Date"", ""start"": 5758, ""end"": 5762, ""text"": ""9/26""}, {""label"": ""Line Item - End Date"", ""start"": 5763, ""end"": 5767, ""text"": ""9/26""}, {""label"": ""Line Item - Days"", ""start"": 5770, ""end"": 5773, ""text"": ""MON""}, {""label"": ""Line Item - Start Date"", ""start"": 5776, ""end"": 5780, ""text"": ""9/26""}, {""label"": ""Line Item - End Date"", ""start"": 5781, ""end"": 5785, ""text"": ""9/26""}, {""label"": ""Line Item - Days"", ""start"": 5788, ""end"": 5791, ""text"": ""MON""}, {""label"": ""Line Item - Start Date"", ""start"": 5794, ""end"": 5798, ""text"": ""9/26""}, {""label"": ""Line Item - End Date"", ""start"": 5799, ""end"": 5803, ""text"": ""9/26""}, {""label"": ""Line Item - Days"", ""start"": 5806, ""end"": 5809, ""text"": ""MON""}, {""label"": ""Line Item - Start Date"", ""start"": 5813, ""end"": 5817, ""text"": ""9/26""}, {""label"": ""Line Item - End Date"", ""start"": 5818, ""end"": 5822, ""text"": ""9/26""}, {""label"": ""Line Item - Days"", ""start"": 5825, ""end"": 5828, ""text"": ""MON""}, {""label"": ""Line Item - Start Date"", ""start"": 5831, ""end"": 5835, ""text"": ""9/26""}, {""label"": ""Line Item - End Date"", ""start"": 5836, ""end"": 5840, ""text"": ""9/26""}, {""label"": ""Line Item - Days"", ""start"": 5843, ""end"": 5846, ""text"": ""MON""}, {""label"": ""Line Item - Start Date"", ""start"": 5849, ""end"": 5853, ""text"": ""9/26""}, {""label"": ""Line Item - End Date"", ""start"": 5854, ""end"": 5858, ""text"": ""9/26""}, {""label"": ""Line Item - Days"", ""start"": 5861, ""end"": 5864, ""text"": ""MON""}, {""label"": ""Line Item - Rate"", ""start"": 3838, ""end"": 3843, ""text"": ""60.00""}, {""label"": ""Line Item - Description"", ""start"": 3855, ""end"": 3872, ""text"": ""RIGHT THIS MINUTE""}, {""label"": ""Line Item - Rate"", ""start"": 3919, ""end"": 3925, ""text"": ""105.00""}, {""label"": ""Line Item - Description"", ""start"": 3937, ""end"": 3945, ""text"": ""THE TALK""}, {""label"": ""Line Item - Rate"", ""start"": 3985, ""end"": 3991, ""text"": ""375.00""}, {""label"": ""Line Item - Description"", ""start"": 4003, ""end"": 4016, ""text"": ""NEWS 13 AT 10""}, {""label"": ""Line Item - Rate"", ""start"": 4061, ""end"": 4067, ""text"": ""275.00""}, {""label"": ""Line Item - Description"", ""start"": 4079, ""end"": 4093, ""text"": ""PRICE IS RIGHT""}, {""label"": ""Line Item - Rate"", ""start"": 4139, ""end"": 4145, ""text"": ""165.00""}, {""label"": ""Line Item - Description"", ""start"": 4157, ""end"": 4176, ""text"": ""LATE SHOW - COLBERT""}, {""label"": ""Line Item - Rate"", ""start"": 4227, ""end"": 4233, ""text"": ""125.00""}, {""label"": ""Line Item - Description"", ""start"": 4245, ""end"": 4261, ""text"": ""YOUNG & RESTLESS""}, {""label"": ""Line Item - Rate"", ""start"": 4308, ""end"": 4314, ""text"": ""135.00""}, {""label"": ""Line Item - Description"", ""start"": 4326, ""end"": 4342, ""text"": ""BOLD & BEAUTIFUL""}, {""label"": ""Line Item - Rate"", ""start"": 4390, ""end"": 4396, ""text"": ""125.00""}, {""label"": ""Line Item - Description"", ""start"": 4408, ""end"": 4423, ""text"": ""NEWS 13 AT NOON""}, {""label"": ""Line Item - Rate"", ""start"": 4468, ""end"": 4474, ""text"": ""105.00""}, {""label"": ""Line Item - Description"", ""start"": 4486, ""end"": 4503, ""text"": ""LET'S MAKE A DEAL""}, {""label"": ""Line Item - Start Date"", ""start"": 4533, ""end"": 4537, ""text"": ""9/23""}, {""label"": ""Line Item - End Date"", ""start"": 4538, ""end"": 4542, ""text"": ""9/23""}, {""label"": ""Line Item - Days"", ""start"": 4545, ""end"": 4548, ""text"": ""FRI""}, {""label"": ""Line Item - Start Date"", ""start"": 4551, ""end"": 4555, ""text"": ""9/26""}, {""label"": ""Line Item - End Date"", ""start"": 4556, ""end"": 4560, ""text"": ""9/26""}, {""label"": ""Line Item - Days"", ""start"": 4563, ""end"": 4566, ""text"": ""MON""}, {""label"": ""Line Item - Start Date"", ""start"": 4569, ""end"": 4573, ""text"": ""9/26""}, {""label"": ""Line Item - End Date"", ""start"": 4574, ""end"": 4578, ""text"": ""9/26""}, {""label"": ""Line Item - Days"", ""start"": 4581, ""end"": 4584, ""text"": ""MON""}, {""label"": ""Line Item - Start Date"", ""start"": 4587, ""end"": 4591, ""text"": ""9/26""}, {""label"": ""Line Item - End Date"", ""start"": 4592, ""end"": 4596, ""text"": ""9/26""}, {""label"": ""Line Item - Days"", ""start"": 4599, ""end"": 4602, ""text"": ""MON""}, {""label"": ""Line Item - Start Date"", ""start"": 4606, ""end"": 4610, ""text"": ""9/26""}, {""label"": ""Line Item - End Date"", ""start"": 4611, ""end"": 4615, ""text"": ""9/26""}, {""label"": ""Line Item - Days"", ""start"": 4618, ""end"": 4621, ""text"": ""MON""}, {""label"": ""Line Item - Start Date"", ""start"": 4622, ""end"": 4626, ""text"": ""9/26""}, {""label"": ""Line Item - End Date"", ""start"": 4627, ""end"": 4631, ""text"": ""9/26""}, {""label"": ""Line Item - Days"", ""start"": 4634, ""end"": 4637, ""text"": ""MON""}, {""label"": ""Line Item - Start Date"", ""start"": 4640, ""end"": 4644, ""text"": ""9/26""}, {""label"": ""Line Item - End Date"", ""start"": 4645, ""end"": 4649, ""text"": ""9/26""}, {""label"": ""Line Item - Days"", ""start"": 4652, ""end"": 4655, ""text"": ""MON""}, {""label"": ""Line Item - Start Date"", ""start"": 4658, ""end"": 4662, ""text"": ""9/26""}, {""label"": ""Line Item - End Date"", ""start"": 4663, ""end"": 4667, ""text"": ""9/26""}, {""label"": ""Line Item - Days"", ""start"": 4670, ""end"": 4673, ""text"": ""MON""}, {""label"": ""Line Item - Start Date"", ""start"": 4676, ""end"": 4680, ""text"": ""9/26""}, {""label"": ""Line Item - End Date"", ""start"": 4681, ""end"": 4685, ""text"": ""9/26""}, {""label"": ""Line Item - Days"", ""start"": 4688, ""end"": 4691, ""text"": ""MON""}, {""label"": ""Line Item - Rate"", ""start"": 2689, ""end"": 2695, ""text"": ""315.00""}, {""label"": ""Line Item - Description"", ""start"": 2707, ""end"": 2719, ""text"": ""NEWS 13 AT 5""}, {""label"": ""Line Item - Rate"", ""start"": 2761, ""end"": 2767, ""text"": ""100.00""}, {""label"": ""Line Item - Description"", ""start"": 2779, ""end"": 2796, ""text"": ""KOLD AM NEWS 5-6A""}, {""label"": ""Line Item - Rate"", ""start"": 2843, ""end"": 2849, ""text"": ""190.00""}, {""label"": ""Line Item - Description"", ""start"": 2861, ""end"": 2875, ""text"": ""NEWS 13 AT 6PM""}, {""label"": ""Line Item - Rate"", ""start"": 2919, ""end"": 2925, ""text"": ""185.00""}, {""label"": ""Line Item - Description"", ""start"": 2937, ""end"": 2954, ""text"": ""KOLD AM NEWS 6-7A""}, {""label"": ""Line Item - Rate"", ""start"": 3001, ""end"": 3007, ""text"": ""325.00""}, {""label"": ""Line Item - Description"", ""start"": 3019, ""end"": 3040, ""text"": ""ENTERTAINMENT TONIGHT""}, {""label"": ""Line Item - Rate"", ""start"": 3091, ""end"": 3097, ""text"": ""375.00""}, {""label"": ""Line Item - Description"", ""start"": 3109, ""end"": 3117, ""text"": ""MACGYVER""}, {""label"": ""Line Item - Rate"", ""start"": 3155, ""end"": 3161, ""text"": ""125.00""}, {""label"": ""Line Item - Description"", ""start"": 3173, ""end"": 3189, ""text"": ""CBS THIS MORNING""}, {""label"": ""Line Item - Rate"", ""start"": 3236, ""end"": 3244, ""text"": ""1,000.00""}, {""label"": ""Line Item - Description"", ""start"": 3256, ""end"": 3267, ""text"": ""BLUE BLOODS""}, {""label"": ""Line Item - Rate"", ""start"": 3309, ""end"": 3314, ""text"": ""60.00""}, {""label"": ""Line Item - Description"", ""start"": 3326, ""end"": 3343, ""text"": ""RIGHT THIS MINUTE""}, {""label"": ""Line Item - Start Date"", ""start"": 3373, ""end"": 3377, ""text"": ""9/23""}, {""label"": ""Line Item - End Date"", ""start"": 3378, ""end"": 3382, ""text"": ""9/23""}, {""label"": ""Line Item - Days"", ""start"": 3385, ""end"": 3388, ""text"": ""FRI""}, {""label"": ""Line Item - Start Date"", ""start"": 3391, ""end"": 3395, ""text"": ""9/23""}, {""label"": ""Line Item - End Date"", ""start"": 3396, ""end"": 3400, ""text"": ""9/23""}, {""label"": ""Line Item - Days"", ""start"": 3403, ""end"": 3406, ""text"": ""FRI""}, {""label"": ""Line Item - Start Date"", ""start"": 3409, ""end"": 3413, ""text"": ""9/23""}, {""label"": ""Line Item - End Date"", ""start"": 3414, ""end"": 3418, ""text"": ""9/23""}, {""label"": ""Line Item - Days"", ""start"": 3421, ""end"": 3424, ""text"": ""FRI""}, {""label"": ""Line Item - Start Date"", ""start"": 3428, ""end"": 3432, ""text"": ""9/23""}, {""label"": ""Line Item - End Date"", ""start"": 3433, ""end"": 3437, ""text"": ""9/23""}, {""label"": ""Line Item - Days"", ""start"": 3440, ""end"": 3443, ""text"": ""FRI""}, {""label"": ""Line Item - Start Date"", ""start"": 3446, ""end"": 3450, ""text"": ""9/23""}, {""label"": ""Line Item - End Date"", ""start"": 3451, ""end"": 3455, ""text"": ""9/23""}, {""label"": ""Line Item - Days"", ""start"": 3458, ""end"": 3461, ""text"": ""FRI""}, {""label"": ""Line Item - Start Date"", ""start"": 3464, ""end"": 3468, ""text"": ""9/23""}, {""label"": ""Line Item - End Date"", ""start"": 3469, ""end"": 3473, ""text"": ""9/23""}, {""label"": ""Line Item - Days"", ""start"": 3476, ""end"": 3479, ""text"": ""FRI""}, {""label"": ""Line Item - Start Date"", ""start"": 3483, ""end"": 3487, ""text"": ""9/23""}, {""label"": ""Line Item - End Date"", ""start"": 3488, ""end"": 3492, ""text"": ""9/23""}, {""label"": ""Line Item - Days"", ""start"": 3495, ""end"": 3498, ""text"": ""FRI""}, {""label"": ""Line Item - Start Date"", ""start"": 3501, ""end"": 3505, ""text"": ""9/23""}, {""label"": ""Line Item - End Date"", ""start"": 3506, ""end"": 3510, ""text"": ""9/23""}, {""label"": ""Line Item - Days"", ""start"": 3513, ""end"": 3516, ""text"": ""FRI""}, {""label"": ""Line Item - Start Date"", ""start"": 3519, ""end"": 3523, ""text"": ""9/23""}, {""label"": ""Line Item - End Date"", ""start"": 3524, ""end"": 3528, ""text"": ""9/23""}, {""label"": ""Line Item - Days"", ""start"": 3531, ""end"": 3534, ""text"": ""FRI""}, {""label"": ""Line Item - Rate"", ""start"": 1517, ""end"": 1523, ""text"": ""165.00""}, {""label"": ""Line Item - Start Date"", ""start"": 1524, ""end"": 1528, ""text"": ""9/23""}, {""label"": ""Line Item - End Date"", ""start"": 1529, ""end"": 1533, ""text"": ""9/23""}, {""label"": ""Line Item - Days"", ""start"": 1536, ""end"": 1539, ""text"": ""FRI""}, {""label"": ""Line Item - Description"", ""start"": 1553, ""end"": 1572, ""text"": ""LATE SHOW - COLBERT""}, {""label"": ""Line Item - Rate"", ""start"": 1623, ""end"": 1628, ""text"": ""90.00""}, {""label"": ""Line Item - Description"", ""start"": 1640, ""end"": 1654, ""text"": ""LATE LATE SHOW""}, {""label"": ""Line Item - Rate"", ""start"": 1699, ""end"": 1705, ""text"": ""125.00""}, {""label"": ""Line Item - Description"", ""start"": 1717, ""end"": 1733, ""text"": ""YOUNG & RESTLESS""}, {""label"": ""Line Item - Rate"", ""start"": 1779, ""end"": 1785, ""text"": ""135.00""}, {""label"": ""Line Item - Description"", ""start"": 1797, ""end"": 1813, ""text"": ""BOLD & BEAUTIFUL""}, {""label"": ""Line Item - Rate"", ""start"": 1860, ""end"": 1866, ""text"": ""125.00""}, {""label"": ""Line Item - Description"", ""start"": 1878, ""end"": 1893, ""text"": ""NEWS 13 AT NOON""}, {""label"": ""Line Item - Rate"", ""start"": 1937, ""end"": 1943, ""text"": ""105.00""}, {""label"": ""Line Item - Description"", ""start"": 1955, ""end"": 1972, ""text"": ""LET'S MAKE A DEAL""}, {""label"": ""Line Item - Rate"", ""start"": 2019, ""end"": 2024, ""text"": ""60.00""}, {""label"": ""Line Item - Description"", ""start"": 2036, ""end"": 2051, ""text"": ""CRIMEWTCH DAILY""}, {""label"": ""Line Item - Rate"", ""start"": 2096, ""end"": 2102, ""text"": ""125.00""}, {""label"": ""Line Item - Description"", ""start"": 2114, ""end"": 2128, ""text"": ""NEWS 13 AT 4PM""}, {""label"": ""Line Item - Rate"", ""start"": 2173, ""end"": 2178, ""text"": ""40.00""}, {""label"": ""Line Item - Description"", ""start"": 2190, ""end"": 2208, ""text"": ""KOLD AM NEWS 430AM""}, {""label"": ""Line Item - Start Date"", ""start"": 2239, ""end"": 2243, ""text"": ""9/23""}, {""label"": ""Line Item - End Date"", ""start"": 2244, ""end"": 2248, ""text"": ""9/23""}, {""label"": ""Line Item - Days"", ""start"": 2251, ""end"": 2254, ""text"": ""FRI""}, {""label"": ""Line Item - Start Date"", ""start"": 2258, ""end"": 2262, ""text"": ""9/23""}, {""label"": ""Line Item - End Date"", ""start"": 2263, ""end"": 2267, ""text"": ""9/23""}, {""label"": ""Line Item - Days"", ""start"": 2270, ""end"": 2273, ""text"": ""FRI""}, {""label"": ""Line Item - Start Date"", ""start"": 2277, ""end"": 2281, ""text"": ""9/23""}, {""label"": ""Line Item - End Date"", ""start"": 2282, ""end"": 2286, ""text"": ""9/23""}, {""label"": ""Line Item - Days"", ""start"": 2289, ""end"": 2292, ""text"": ""FRI""}, {""label"": ""Line Item - Start Date"", ""start"": 2295, ""end"": 2299, ""text"": ""9/23""}, {""label"": ""Line Item - End Date"", ""start"": 2300, ""end"": 2304, ""text"": ""9/23""}, {""label"": ""Line Item - Days"", ""start"": 2307, ""end"": 2310, ""text"": ""FRI""}, {""label"": ""Line Item - Start Date"", ""start"": 2313, ""end"": 2317, ""text"": ""9/23""}, {""label"": ""Line Item - End Date"", ""start"": 2318, ""end"": 2322, ""text"": ""9/23""}, {""label"": ""Line Item - Days"", ""start"": 2325, ""end"": 2328, ""text"": ""FRI""}, {""label"": ""Line Item - Start Date"", ""start"": 2332, ""end"": 2336, ""text"": ""9/23""}, {""label"": ""Line Item - End Date"", ""start"": 2337, ""end"": 2341, ""text"": ""9/23""}, {""label"": ""Line Item - Days"", ""start"": 2344, ""end"": 2347, ""text"": ""FRI""}, {""label"": ""Line Item - Start Date"", ""start"": 2350, ""end"": 2354, ""text"": ""9/23""}, {""label"": ""Line Item - End Date"", ""start"": 2355, ""end"": 2359, ""text"": ""9/23""}, {""label"": ""Line Item - Days"", ""start"": 2362, ""end"": 2365, ""text"": ""FRI""}, {""label"": ""Line Item - Start Date"", ""start"": 2368, ""end"": 2372, ""text"": ""9/23""}, {""label"": ""Line Item - End Date"", ""start"": 2373, ""end"": 2377, ""text"": ""9/23""}, {""label"": ""Line Item - Days"", ""start"": 2380, ""end"": 2383, ""text"": ""FRI""}]",fcc_invoices/ocr/34d4c874dd87752d683e4763e9333559.json.gz,00c7b3f3-369b-4e99-868e-f2cb09c12e49.pdf,"REP: TEL# 703-528-9967 +REP HEADLINE# 8372114 ORDER WORKSHEET +*** ORIGINAL REV#0 *** + +FAX# 703-516-9680 +HARRIS REPORT FROM REP SEP16/16 17.31 +*** KOLD-TV *** + +ADV # + +AGY # + +ADV. NAME POLI/M HEINZ/D/CON/AZ REP.# + +OFF.# SALESMAN # + +AGY_ NAME BUYING TIME, LLC BUYER NAME KATHRYN WELSH + +650 MASSACHUSETTS AVE.N.W. #210 SALES PRSN WA- RYAN GORDON (H) + +WASHINGTON, DC 20001 + +ORDER # CONTRACT # 8372114 + +CLASS: NATL. LOCAL REGIONAL + +PRDCT + +HEINZ FOR ARIZONA + +EST#4996 COMMENTS: (LINE, ORDER, INVOICE) + +FLIGHT DATES SEP20/16 SEP26/16 WK -1 + +CITY TAX STATE TAX + +REP: + +CON CM + +NEW ORDER +TTL $24,980 8 112X +PLS CFM +THX, ALEX FOR FRANK + +*** + +CO-OP BILLING NEEDED ____ DATE SEP16/16 17.31 + +* THIS IS A CASH IN ADVANCE SCHEDULE ***** + +HEINZ FOR ARIZONA + +:LINE#:REP :CD: TIME PERIOD : LGTH : SEC : RATE START END :SPTS: WEEK : DAYS :TOTL: +:LINE#: DATE DATE : /WK: INVT : :SPTS: + +AGENCY ADVERTISER CODE = 782 AGENCY EST# = 4996 +AGENCY PRODUCT CODE = 792 + +1 100P -200P 30 $105.00 + +PROGRAM •: THE TALK +CON COM1: THE TALK +2 1000P -1035P 30 $375.00 + +PROGRAM : NEWS 13 AT 10 +CON COM1: NEWS 13 AT 10 + +3 1000A -1100A 30 $275.00 + +PROGRAM : PRICE IS RIGHT +CON COM1: PRICE IS RIGHT + +9/23 9/23 1 FRI 1 +9/23 9/23 1 FRI 1 +9/23 9/23 1 FRI 1 +REP: TEL# 703-528-9967 +REP HEADLINE# 8372114 ORDER WORKSHEET +*** ORIGINAL REV00 *** + +FAX# 703-516-9680 +HARRIS REPORT FROM REP SEP16/16 17.31 +*** KOLD-TV *** + +:LINE#:REP :CD: TIME PERIOD : LGTH : SEC : RATE START END :SPTS: WEEK : DAYS :TOTL: + +:LINE*: : : DATE DATE : /WK: INVT : :SPTS: + +4 + +1035P -1137P 30 $165.00 9/23 9/23 1 FRI 1 + +PROGRAM : LATE SHOW - COLBERT +CON COM1: LATE SHOW - COLBERT + +5 1137P -1237A 30 $90.00 + +PROGRAM : LATE LATE SHOW +CON COM1: LATE LATE SHOW +6 1100A -1200N 30 $125.00 + +PROGRAM : YOUNG & RESTLESS +CON COM1: YOUNG & RESTLESS +7 1230P -100P 30 $135.00 + +PROGRAM : BOLD & BEAUTIFUL +CON COM1: BOLD & BEAUTIFUL +8 1200N -1230P 30 $125.00 + +PROGRAM : NEWS 13 AT NOON +CON COM1: NEWS 13 AT NOON +9 200P -300P 30 $105.00 + +PROGRAM : LET'S MAKE A DEAL +CON COM1: LET'S MAKE A DEAL +10 300P -400P 30 $60.00 + +PROGRAM : CRIMEWTCH DAILY +CON COM1: CRIMEWTCH DAILY +11 400P -500P 30 $125.00 + +PROGRAM : NEWS 13 AT 4PM +CON COM1: NEWS 13 AT 4PM + +12 430A -459A 30 $40.00 + +PROGRAM : KOLD AM NEWS 430AM +CON COM1: KOLD AM NEWS 430AM + +9/23 9/23 1 FRI 1 + +9/23 9/23 1 FRI 1 + +9/23 9/23 1 FRI 1 +9/23 9/23 1 FRI 1 +9/23 9/23 1 FRI 1 + +9/23 9/23 1 FRI 1 +9/23 9/23 1 FRI 1 +9/23 9/23 1 FRI 1 +REP: TEL# 703-528-9967 +REP HEADLINE# 8372114 ORDER WORKSHEET +*** ORIGINAL REV#0 *** + +FAX# 703-516-9680 +HARRIS REPORT FROM REP SEP16/16 17.31 +*** KOLD-TV *** + +:LINE#:REP :CD: TIME PERIOD : LGTH : SEC : RATE START END :SPTS: WEEK : DAYS :TOTL: + +:LINE*: : : DATE DATE : /WK: INVT :SPTS: + +13 500P -530P 30 $315.00 + +PROGRAM : NEWS 13 AT 5 +CON COM1: NEWS 13 AT 5 +14 500A -600A 30 $100.00 + +PROGRAM : KOLD AM NEWS 5-6A +CON COM1: KOLD AM NEWS 5-6A +15 600P -630P 30 $190.00 + +PROGRAM : NEWS 13 AT 6PM +CON COM1: NEWS 13 AT 6PM +16 600A -700A 30 $185.00 + +PROGRAM : KOLD AM NEWS 6-7A +CON COM1: KOLD AM NEWS 6-7A +17 630P -700P 30 $325.00 + +PROGRAM : ENTERTAINMENT TONIGHT +CON COM1: ENTERTAINMENT TONIGHT +18 700P -800P 30 $375.00 + +PROGRAM : MACGYVER +CON COM1: MACGYVER +19 700A -900A 30 $125.00 + +PROGRAM : CBS THIS MORNING +CON COM1: CBS THIS MORNING +20 900P -1000P 30 $1,000.00 + +PROGRAM : BLUE BLOODS +CON COM1: BLUE BLOODS + +21 900A -930A 30 $60.00 + +PROGRAM : RIGHT THIS MINUTE +CON COM1: RIGHT THIS MINUTE + +9/23 9/23 1 FRI 1 +9/23 9/23 1 FRI 1 +9/23 9/23 1 FRI 1 + +9/23 9/23 1 FRI 1 +9/23 9/23 1 FRI 1 +9/23 9/23 1 FRI 1 + +9/23 9/23 1 FRI 1 +9/23 9/23 1 FRI 1 +9/23 9/23 1 FRI 1 +REP: TEL# 703-528-9967 +REP HEADLINE# 8372114 ORDER WORKSHEET +*** ORIGINAL REV*0 *** + +FAX# 703-516-9680 +HARRIS REPORT FROM REP SEP16/16 17.31 +*** KOLD-TV *** + +:LINE#:REP :CD: TIME PERIOD : LGTH : SEC : RATE START END :SPTS: WEEK : DAYS :TOTL: +:LINE*: : DATE DATE : /WK: INVT :SPTS: + +22 930A -1000A 30 $60.00 + +PROGRAM : RIGHT THIS MINUTE +CON COM1: RIGHT THIS MINUTE +23 100P -200P 30 $105.00 + +PROGRAM : THE TALK +CON COM1: THE TALK +24 1000P -1035P 30 $375.00 + +PROGRAM : NEWS 13 AT 10 +CON COM1: NEWS 13 AT 10 +25 1000A -1100A 30 $275.00 + +PROGRAM : PRICE IS RIGHT +CON COM1: PRICE IS RIGHT +26 1035P -1137P 30 $165.00 + +PROGRAM : LATE SHOW - COLBERT +CON COM1: LATE SHOW - COLBERT +27 1100A -1200N 30 $125.00 + +PROGRAM : YOUNG & RESTLESS +CON COM1: YOUNG & RESTLESS +28 1230P -100P 30 $135.00 + +PROGRAM : BOLD & BEAUTIFUL +CON COM1: BOLD & BEAUTIFUL +29 1200N -1230P 30 $125.00 + +PROGRAM : NEWS 13 AT NOON +CON COM1: NEWS 13 AT NOON + +30 2009-3009 30 $105.00 + +PROGRAM : LET'S MAKE A DEAL +CON COM1: LET'S MAKE A DEAL + +9/23 9/23 1 FRI 1 +9/26 9/26 1 MON 1 +9/26 9/26 1 MON 1 +9/26 9/26 1 MON 1 + +9/26 9/26 1 MON +9/26 9/26 1 MON 1 +9/26 9/26 1 MON 1 +9/26 9/26 1 MON 1 +9/26 9/26 1 MON 1 +REP: TEL# 703-528-9967 +REP HEADLINE# 8372114 ORDER WORKSHEET +*** ORIGINAL REV#0 *** + +FAX# 703-516-9680 +HARRIS REPORT FROM REP SEP16/16 17.31 +*** KOLD-TV *** + +:LINE#:REP :CD: TIME PERIOD : LGTH : SEC : RATE START END :SPTS: WEEK : DAYS :TOTL: +:LINE#: DATE DATE : /WK: INVT :SPTS: + +31 400P -500P 30 $125.00 + +PROGRAM : NEWS 13 AT 4PM +CON COM1: NEWS 13 AT 4PM +32 430A -459A 30 $40.00 + +PROGRAM : KOLD AM NEWS 430AM +CON COM1: KOLD AM NEWS 430AM +33 500P -530P 30 $315.00 + +PROGRAM : NEWS 13 AT 5 +CON COM1: NEWS 13 AT 5 +34 500A -600A 30 $100.00 + +PROGRAM : KOLD AM NEWS 5-6A +CON COM1: KOLD AM NEWS 5-6A +35 600P -630P 30 $290.00 + +PROGRAM : NEWS 13 AT 6PM +CON COM1: NEWS 13 AT 6PM +36 600A -700A 30 $185.00 + +PROGRAM : KOLD AM NEWS 6-7A +CON COM1: KOLD AM NEWS 6-7A +37 630P -700P 30 $325.00 + +PROGRAM : ENTERTAINMENT TONIGHT +CON COM1: ENTERTAINMENT TONIGHT +38 700P -900P 30 $500.00 + +PROGRAM : PRESIDENTIAL DEBATE +CON COM1: PRESIDENTIAL DEBATE + +39 700A -900A 30 $125.00 + +PROGRAM : CBS THIS MORNING +CON COM1: CBS THIS MORNING + +9/26 9/26 1 MON 1 +9/26 9/26 1 MON 1 +9/26 9/26 1 MON 1 + +9/26 9/26 1 MON 1 +9/26 9/26 1 MON 1 +9/26 9/26 1 MON 1 + +9/26 9/26 1 MON 1 +9/26 9/26 1 MON 1 +9/26 9/26 1 MON 1 +REP: TEL# 703-528-9967 +REP HEADLINE# 8372114 ORDER WORKSHEET +*** ORIGINAL REV#0 *** + +FAX# 703-516-9680 +HARRIS REPORT FROM REP SEP16/16 17.31 +*** KOLD-TV *** + +:LINE#:REP :CD: TIME PERIOD : LGTH : SEC : RATE START END :SPTS: WEEK : DAYS :TOTL: +:LINE#: : : DATE DATE : /WK: INVT :SPTS: + +40 700P -900P + +30 + +PROGRAM : PRESIDENTIAL DEBATE +CON COM1: PRESIDENTIAL DEBATE + +41 900P -1000P + +PROGRAM : SCORPION +CON COM1: SCORPION + +$500.00 9/26 + +9/26 1 MON 1 + +30 $775.00 9/26 9/26 1 MON 1 + +42 900A -930A 30 $60.00 + +PROGRAM : FIX IT & FINISH IT +CON COM1: FIX IT & FINISH IT +43 1000P -1035P 30 $225.00 + +PROGRAM : NEWS 13 SAT AT 10PM +CON COM1: NEWS 13 SAT AT 10PM +44 530P -600P 30 $125.00 + +PROGRAM : NEWS 13 SAT AT 530PM +CON COM1: NEWS 13 SAT AT 530PM +45 900P -1000P 30 $325.00 + +PROGRAM : 48 HOURS +CON COM1: 48 HOURS +46 1000P -1035P 30 $300.00 + +PROGRAM : NEWS 13 SUN AT 10P +CON COM1: NEWS 13 SUN AT 109 +47 530P -600P 30 $200.00 + +PROGRAM : NEWS 13 SUN AT 530P +CON COM1: NEWS 13 SUN AT 530P + +48 600P -700P 30 $525.00 +PROGRAM : 60 MINUTES +CON COM1: 60 MINUTES + +9/26 9/26 1 MON 1 + +9/24 9/24 1 SAT 1 +9/24 9/24 1 SAT 1 +9/24 9/24 1 SAT 1 + +9/25 9/25 1 SUN 1 +9/25 9/25 1 SUN 1 +9/25 9/25 1 SUN 1 +REP: TEL# 703-528-9967 +REP HEADLINE# 8372114 ORDER WORKSHEET +*** ORIGINAL REV#0 *** + +FAX# 703-516-9680 +HARRIS REPORT FROM REP SEP16/16 17.31 +*** KOLD-TV *** + +:LINE#:REP :CD: TIME PERIOD : LGTH : SEC : RATE START END :SPTS: WEEK : DAYS :TOTL: +:LINE#: : DATE : DATE : /WK: INVT :SPTS: + +49 900P -1000P + +PROGRAM : NCIS LA +CON COM1: NCIS LA + +50 100P -200P + +PROGRAM : THE TALK +CON COM1: THE TALK + +51 1000P -1035P + +PROGRAM : NEWS 13 AT 10 +CON COM1: NEWS 13 AT 10 + +52 1000A -1100A + +PROGRAM : PRICE IS RIGHT +CON COM1: PRICE IS RIGHT + +30 + +30 + +30 + +30 + +$675_00 +$105.00 +$375.00 + +$275.00 + +53 1035P -1137P 30 $165.00. + +PROGRAM : LATE SHOW - COLBERT +CON COM1: LATE SHOW - COLBERT +54 1137P -1237A 30 $90.00 + +PROGRAM : LATE LATE SHOW +CON COM1: LATE LATE SHOW +55 1100A -1200N 30 $125.00 + +PROGRAM : YOUNG & RESTLESS +CON COM1: YOUNG & RESTLESS +56 1230P -100P 30 $135.00 + +PROGRAM : BOLD & BEAUTIFUL +CON COM1: BOLD & BEAUTIFUL + +57 1200N -1230P 30 $125.00 + +PROGRAM : NEWS 13 AT NOON +CON COM1: NEWS 13 AT NOON + +9/25 9/25 1 SUN 1 +9/22 9/22 1 THU 1 +9/22 9/22 1 THU 1 + +9/22 9/22 1 THU 1 +9/22 9/22 1 THU 1 +9/22 9/22 1 THU 1 + +9/22 9/22 1 THU 1 +9/22 9/22 1 THU 1 +9/22 9/22 1 THU 1 +REP: TEL# 703-528-9967 +REP HEADLINE# 8372114 ORDER WORKSHEET +*** ORIGINAL REV#0 *** + +FAX# 703-516-9680 +HARRIS REPORT FROM REP SEP16/16 17.31 +*** KOLD-TV *** + +:LINE4:REP :CD: TIME PERIOD : LGTH : SEC : RATE START : END :SPTS: WEEK : DAYS :TOTL: +. :LINE#: : DATE . : DATE : /WK: INVT : :SPTS: + +58 200P -300P 30 $105.00 + +PROGRAM : LET'S MAKE A DEAL +CON COM1: LET'S MAKE A DEAL +59 300P -400P 30 $60.00 + +PROGRAM : CRIMEWTCH DAILY +CON COM1: CRIMEWTCH DAILY +60 400P -500P 30 $125.00 + +PROGRAM : NEWS 13 AT 4PM +CON COM1: NEWS 13 AT 4PM +61 430A -459A 30 $40.00 + +PROGRAM : KOLD AM NEWS 430AM +CON COM1: KOLD AM NEWS 430AM +62 500P -530P 30 $315.00 + +PROGRAM : NEWS 13 AT 5 +CON COM1: NEWS 13 AT 5 +63 500A -600A 30 $100.00 + +PROGRAM : KOLD AM NEWS 5-6A +CON COM1: KOLD AM NEWS 5-6A +64 600P -630P 30 $290.00 + +PROGRAM : NEWS 13 AT 6PM +CON COM1: NEWS 13 AT 6PM +65 600A -700A 30 $185.00 + +PROGRAM : KOLD AM NEWS 6-7A +CON COM1: KOLD AM NEWS 6-7A + +66 630P -700P 30 $325.00 + +PROGRAM : ENTERTAINMENT TONIGHT +CON COM1: ENTERTAINMENT TONIGHT + +9/22 9/22 1 THU 1 +9/22 9/22 1 THU 1 +9/22 9/22 1 THU 1 + +9/22 9/22 1 THU 1 +9/22 9/22 1 THU 1 +9/22 9/22 1 THU 1 + +9/22 9/22 1 THU 1 +9/22 9/22 1 THU 1 +9/22 9/22 1 THU 1 +REP: TEL# 703-528-9967 +REP HEADLINE# 8372114 ORDER WORKSHEET +*** ORIGINAL REV#0 *** + +FAX# 703-516-9680 +HARRIS REPORT FROM REP SEP16/16 17.31 +*** KOLD-TV *** + +:LINE#:REP :CD: TIME PERIOD : LGTH : SEC : RATE START +:LINE#: DATE + +END :SPTS: WEEK : + +DATE : /WK: INVT : + +DAYS :TOTL: + +:SPTS: + +67 700A -900A + +PROGRAM : CBS THIS MORNING +CON COM1: CBS THIS MORNING + +68 900A -930A + +PROGRAM : RIGHT THIS MINUTE +CON COM1: RIGHT THIS MINUTE + +69 930A -1000A + +PROGRAM : RIGHT THIS MINUTE +CON COM1: RIGHT THIS MINUTE + +70 100P -200P + +PROGRAM : THE TALK +CON COM1: THE TALK + +71 1000P -1035P + +PROGRAM : NEWS 13 AT 10 +CON COM1: NEWS 13 AT 10 + +72 1000A -1100A + +PROGRAM : PRICE IS RIGHT +CON COM1: PRICE IS RIGHT + +73 1035P-:11372 + +30 $125.00 9/22 9/22 1 THU 1 +30 $60.00 9/22 9/22 1 THU 1 + +30 $60.00 9/22 9/22 1 THU 1 +30 $105.00 9/20 9/20 1 TUE 1 + +30 $375.00 9/20 9/20 1 TUE 1 +30 $275.00 9/20 9/20 1 TUE 1 + +30 + +PROGRAM : LATE SHOW - COLBERT +CON COM1: LATE SHOW - COLBERT + +74 1137P -1237A + +PROGRAM : LATE LATE SHOW +CON COM1: LATE LATE SHOW + +75 1100A -1200N +PROGRAM : YOUNG & RESTLESS +CON COM1: YOUNG & RESTLESS + +30 + +$165.00 9/20 9/20 1 TUE 1 +$90.00 9/20 9/20 1 TUE 1 + +30 $125.00 9/20 9/20 1 TUE 1 +REP: TEL# 703-528-9967 +REP HEADLINE# 8372114 ORDER WORKSHEET +*** ORIGINAL REV#0 *** + +FAX# 703-516-9680 +HARRIS REPORT FROM REP SEP16/16 17.31 +*** KOLD-TV *** + +:LINE#:REP :CD: TIME PERIOD : LGTH : SEC : RATE START END :SPTS: WEEK : DAYS :TOTL: + +:LINE*: : DATE : DATE : /WK: INVT : :SPTS: + +76 1230P -100P 30 $135.00 9/20 9/20 1 TUE 1 + +PROGRAM : BOLD & BEAUTIFUL +CON COM1: BOLD & BEAUTIFUL +77 1200N -1230P 30 $125.00 9/20 9/20 1 TUE 1 + +PROGRAM : NEWS 13 AT NOON +CON COM1: NEWS 13 AT NOON +78 200P -300P 30 $105.00 9/20 9/20 1 TUE 1 + +PROGRAM : LET'S MAKE A DEAL +CON COM1: LET'S MAKE A DEAL +79 300P -400P 30 $60.00 9/20 9/20 1 TUE + +PROGRAM : CRIMEWTCH DAILY +CON COM1: CRIMEWTCH DAILY +80 400P -500P 30 $125.00 9/20 9/20 1 TUE 1 + +PROGRAM : NEWS 13 AT 4PM +CON COM1: NEWS 13 AT 4PM +81 430A -459A 30 $40.00 9/20 9/20 1 TUE 1 + +PROGRAM : KOLD AM NEWS 430AM +CON COM1: KOLD AM NEWS 430AM +82 500P -530P 30 $315.00 9/20 9/20 1 TUE 1 + +PROGRAM : NEWS 13 AT 5 +CON COM1: NEWS 13 AT 5 +83 500A -600A 30 $100.00 9/20 9/20 1 TUE 1 + +PROGRAM : KOLD AM NEWS 5-6A +CON COM1: KOLD AM NEWS 5-6A + +84 600P -630P 30 $290.00 9/20 9/20 1 TUE 1 + +PROGRAM : NEWS 13 AT 6PM +CON COM1: NEWS 13 AT 6PM +REP: TEL# 703-528-9967 +REP HEADLINE* 8372114 ORDER WORKSHEET +*** ORIGINAL REV*0 *** + +FAX# 703-516-9680 +HARRIS REPORT FROM REP SEP16/16 17.31 +*** KOLD-TV *** + +:LINE4:REP :CD: TIME PERIOD : LGTH : SEC : RATE START END :SPTS: WEEK : DAYS :TOTL: +:LINE4: : DATE : DATE : /WK: INVT : :SPTS: + +85 600A -700A + +PROGRAM : KOLD AM NEWS 6-7A +CON COM1: KOLD AM NEWS 6-7A + +86 630P -700P + +30 $185.00 9/20 9/20 1 TUE 1 +30 $325.00 9/20 9/20 1 TUE 1 + +PROGRAM : ENTERTAINMENT TONIGHT +CON COM1: ENTERTAINMENT TONIGHT + +87 700A -900A + +PROGRAM : CBS THIS MORNING +CON COM1: CBS THIS MORNING + +88 800P -900P + +PROGRAM : BULL +CON COM1: BULL + +89 900P -1000P + +PROGRAM : NCIS NEW ORLEANS +CON COM1: NCIS NEW ORLEANS + +90 900A -930A + +PROGRAM : RIGHT THIS MINUTE +CON COM1: RIGHT THIS MINUTE + +91 930A -1000A + +PROGRAM : RIGHT THIS MINUTE +CON COM1: RIGHT THIS MINUTE + +92 100P-2009 + +PROGRAM : THE TALK +CON COM1: THE TALK +93 1000P -1035P +PROGRAM : NEWS 13 AT 10 +CON COM1: NEWS 13 AT 10 + +30 $125.00 9/20 9/20 1 TUE 1 +30 $1',100.00 9/20 9/20 1 TUE + +30 $1,100.00 9/20 9/20 1 TUE 1 +30 $60.00 9/20 9/20 1 TUE 1 + +30 $60.00 9/20 9/20 1 TUE 1 +30 $105.00 9/21 9/21 1 WED 1 + +30 $375.00 9/21 9/21 1 WED 1 +REP: TEL# 703-528-9967 +REP HEADLINE# 8372114 ORDER WORKSHEET +*** ORIGINAL REV#0 *** + +FAX# 703-516-9680 +HARRIS REPORT FROM REP SEP16/16 17.31 +*** KOLD-TV *** + +:LINE#:REP :CD: TIME PERIOD : LGTH : SEC : RATE START END :SPTS: WEEK : DAYS :TOTL: +:LINE*: : DATE : DATE : /WK: INVT :SPTS: + +94 1000A -1100A + +PROGRAM : PRICE IS RIGHT +CON COM1: PRICE IS RIGHT + +95 1035P -1137P + +30 $275.00 9/21 9/21 1 WED 1 + +30 $165.00 9/21 9/21 1 WED 1 + +PROGRAM : LATE SHOW - COLBERT +CON COM1: LATE SHOW - COLBERT + +96 1137P -1237A 30 $90.00 9/21 9/21 1 WED 1 + +PROGRAM : LATE LATE SHOW +CON COM1: LATE LATE SHOW +97 1100A -1200N 30 $125.00 9/21 9/21 1 WED 1 + +PROGRAM : YOUNG & RESTLESS +CON COM1: YOUNG & RESTLESS +98 1200N -1230P 30 $125.00 9/21 9/21 1 WED 1 + +PROGRAM : NEWS 13 AT NOON +CON COM1: NEWS 13 AT NOON +99 200P -300P 30 $105.00 9/21 9/21 1 WED 1 + +PROGRAM : LET'S MAKE A DEAL +CON COM1: LET'S MAKE A DEAL +100 300P -400P 30 $60.00 9/21 9/21 1 WED 1 + +PROGRAM : CRIMEWTCH DAILY +CON COM1: CRIMEWTCH DAILY +101 400P -500P 30 $125.00 9/21 9/21 1 WED 1 + +PROGRAM : NEWS 13 AT 4PM +CON COM1: NEWS 13 AT 4PM + +102 430A -459A 30 $40.00 9/21 9/21 1 WED 1 + +PROGRAM : KOLD AM NEWS 430AM +CON COM1: KOLD AM NEWS 430AM +REP: TEL# 703-528-9967 +REP HEADLINE* 8372114 ORDER WORKSHEET +*** ORIGINAL REV#0 *** + +FAX# 703-516-9680 +HARRIS REPORT FROM REP SEP16/16 17.31 +*** KOLD-TV *** + +:LINE#:REP :CD: TIME PERIOD : LGTH : SEC : RATE START END :SPTS: WEEK : DAYS :TOTL: +:LINE#: : DATE DATE : /WK: INVT : :SPTS: + +103 500P -530P 30 $315.00 + +PROGRAM : NEWS 13 AT 5 +CON COM1: NEWS 13 AT 5 +104 500A -600A 30 $100.00 + +PROGRAM : KOLD AM NEWS 5-6A +CON COM1: KOLD AM NEWS 5-6A +105 600P -630P 30 $290.00 + +PROGRAM : NEWS 13 AT 6PM +CON COM1: NEWS 13 AT 6PM +106 600A -700A 30 $185.00 + +PROGRAM : KOLD AM NEWS 6-7A +CON COM1: KOLD AM NEWS 6-7A +107 630P -700P 30 $325.00 + +PROGRAM : ENTERTAINMENT TONIGHT +CON COM1: ENTERTAINMENT TONIGHT +108 700A -900A 30 $125_00 + +PROGRAM : CBS THIS MORNING +CON COM1: CBS THIS MORNING +109 800P -900P 30 $1,100.00 + +PROGRAM : CRIMINAL MINDS +CON COM1: CRIMINAL MINDS +110 900P -1000P 30 $675.00 + +PROGRAM : CODE BLACK +CON COM1: CODE BLACK + +111 900A -930A 30 $60.00 + +PROGRAM : RIGHT THIS MINUTE +CON COM1: RIGHT THIS MINUTE + +9/21 9/21 1 WED 1 +9/21 9/21 1 WED 1 +9/21 9/21 1 WED 1 + +9/21 9/21 1 WED 1 +9/21 9/21 1 WED 1 +9/21 9/21 1 WED 1 + +9/21 9/21 1 WED 1 +9/21 9/21 1 WED 1 +9/21 9/21 1 WED 1 +REP: TEL# 703-528-9967 +REP HEADLINE* 8372114 ORDER WORKSHEET +*** ORIGINAL REV#0 *** + +FAX# 703-516-9680 +HARRIS REPORT FROM REP SEP16/16 17.31 +*** KOLD-TV *** + +:LINE#:REP :CD: TIME PERIOD : LGTH : SEC : RATE : START END :SPTS: WEEK : DAYS :TOTL: +:LINE*: : : : : DATE : DATE : /WK: INVT : :SPTS: + +112 930A -1000A 30 $60.00 9/21 9/21 1 WED 1 + +PROGRAM : RIGHT THIS MINUTE +CON COM1: RIGHT THIS MINUTE + +SEP/16 20230.00 OCT/16 $4,750.00 CONTRACT TOTAL +TOTAL SPOTS + +MARKET TOTALS $90,991 + +ACCURATE SHARES + +SVC- NSI +DEMOS- RA35-1-* + +KOLD 27% KGUN 27% KVOA 32% KMSB 7% KTTU 7% CABL 0% KWBA 0% +FOLD 0% + +24980.00 +112",34d4c874dd87752d683e4763e9333559,"{""advertiser"": [""POLI/M HEINZ/D/CON/AZ"", ""HEINZ FOR ARIZONA""], ""agency"": [""BUYING TIME, LLC""], ""gross_total"": [""24,980"", ""24980.00""], ""payment_terms"": [""CASH IN ADVANCE""], ""line_item__rate"": [""105.00"", ""375.00"", ""275.00"", ""185.00"", ""325.00"", ""125.00"", ""1',100.00"", ""1,100.00"", ""60.00"", ""60.00"", ""105.00"", ""375.00"", ""60.00"", ""315.00"", ""100.00"", ""290.00"", ""185.00"", ""325.00"", ""125_00"", ""1,100.00"", ""675.00"", ""60.00"", ""275.00"", ""165.00"", ""90.00"", ""125.00"", ""125.00"", ""105.00"", ""60.00"", ""125.00"", ""40.00"", ""135.00"", ""125.00"", ""105.00"", ""60.00"", ""125.00"", ""40.00"", ""315.00"", ""100.00"", ""290.00"", ""125.00"", ""60.00"", ""60.00"", ""105.00"", ""375.00"", ""275.00"", ""165.00"", ""90.00"", ""125.00"", ""105.00"", ""60.00"", ""125.00"", ""40.00"", ""315.00"", ""100.00"", ""290.00"", ""185.00"", ""325.00"", ""675_00"", ""105.00"", ""375.00"", ""275.00"", ""165.00"", ""90.00"", ""125.00"", ""135.00"", ""125.00"", ""500.00"", ""775.00"", ""60.00"", ""225.00"", ""125.00"", ""325.00"", ""300.00"", ""200.00"", ""525.00"", ""125.00"", ""40.00"", ""315.00"", ""100.00"", ""290.00"", ""185.00"", ""325.00"", ""500.00"", ""125.00"", ""60.00"", ""105.00"", ""375.00"", ""275.00"", ""165.00"", ""125.00"", ""135.00"", ""125.00"", ""105.00"", ""315.00"", ""100.00"", ""190.00"", ""185.00"", ""325.00"", ""375.00"", ""125.00"", ""1,000.00"", ""60.00"", ""165.00"", ""90.00"", ""125.00"", ""135.00"", ""125.00"", ""105.00"", ""60.00"", ""125.00"", ""40.00""], ""line_item__description"": [""THE TALK"", ""NEWS 13 AT 10"", ""PRICE IS RIGHT"", ""KOLD AM NEWS 6-7A"", ""ENTERTAINMENT TONIGHT"", ""CBS THIS MORNING"", ""BULL"", ""NCIS NEW ORLEANS"", ""RIGHT THIS MINUTE"", ""RIGHT THIS MINUTE"", ""THE TALK"", ""NEWS 13 AT 10"", ""RIGHT THIS MINUTE"", ""NEWS 13 AT 5"", ""KOLD AM NEWS 5-6A"", ""NEWS 13 AT 6PM"", ""KOLD AM NEWS 6-7A"", ""ENTERTAINMENT TONIGHT"", ""CBS THIS MORNING"", ""CRIMINAL MINDS"", ""CODE BLACK"", ""RIGHT THIS MINUTE"", ""PRICE IS RIGHT"", ""LATE SHOW - COLBERT"", ""LATE LATE SHOW"", ""YOUNG & RESTLESS"", ""NEWS 13 AT NOON"", ""LET'S MAKE A DEAL"", ""CRIMEWTCH DAILY"", ""NEWS 13 AT 4PM"", ""KOLD AM NEWS 430AM"", ""BOLD & BEAUTIFUL"", ""NEWS 13 AT NOON"", ""LET'S MAKE A DEAL"", ""CRIMEWTCH DAILY"", ""NEWS 13 AT 4PM"", ""KOLD AM NEWS 430AM"", ""NEWS 13 AT 5"", ""KOLD AM NEWS 5-6A"", ""NEWS 13 AT 6PM"", ""CBS THIS MORNING"", ""RIGHT THIS MINUTE"", ""RIGHT THIS MINUTE"", ""THE TALK"", ""NEWS 13 AT 10"", ""PRICE IS RIGHT"", ""LATE SHOW - COLBERT"", ""LATE LATE SHOW"", ""YOUNG & RESTLESS"", ""LET'S MAKE A DEAL"", ""CRIMEWTCH DAILY"", ""NEWS 13 AT 4PM"", ""KOLD AM NEWS 430AM"", ""NEWS 13 AT 5"", ""KOLD AM NEWS 5-6A"", ""NEWS 13 AT 6PM"", ""KOLD AM NEWS 6-7A"", ""ENTERTAINMENT TONIGHT"", ""NCIS LA"", ""THE TALK"", ""NEWS 13 AT 10"", ""PRICE IS RIGHT"", ""LATE SHOW - COLBERT"", ""LATE LATE SHOW"", ""YOUNG & RESTLESS"", ""BOLD & BEAUTIFUL"", ""NEWS 13 AT NOON"", ""PRESIDENTIAL DEBATE"", ""SCORPION"", ""FIX IT & FINISH IT"", ""NEWS 13 SAT AT 10PM"", ""NEWS 13 SAT AT 530PM"", ""48 HOURS"", ""NEWS 13 SUN AT 10P"", ""NEWS 13 SUN AT 530P"", ""60 MINUTES"", ""NEWS 13 AT 4PM"", ""KOLD AM NEWS 430AM"", ""NEWS 13 AT 5"", ""KOLD AM NEWS 5-6A"", ""NEWS 13 AT 6PM"", ""KOLD AM NEWS 6-7A"", ""ENTERTAINMENT TONIGHT"", ""PRESIDENTIAL DEBATE"", ""CBS THIS MORNING"", ""RIGHT THIS MINUTE"", ""THE TALK"", ""NEWS 13 AT 10"", ""PRICE IS RIGHT"", ""LATE SHOW - COLBERT"", ""YOUNG & RESTLESS"", ""BOLD & BEAUTIFUL"", ""NEWS 13 AT NOON"", ""LET'S MAKE A DEAL"", ""NEWS 13 AT 5"", ""KOLD AM NEWS 5-6A"", ""NEWS 13 AT 6PM"", ""KOLD AM NEWS 6-7A"", ""ENTERTAINMENT TONIGHT"", ""MACGYVER"", ""CBS THIS MORNING"", ""BLUE BLOODS"", ""RIGHT THIS MINUTE"", ""LATE SHOW - COLBERT"", ""LATE LATE SHOW"", ""YOUNG & RESTLESS"", ""BOLD & BEAUTIFUL"", ""NEWS 13 AT NOON"", ""LET'S MAKE A DEAL"", ""CRIMEWTCH DAILY"", ""NEWS 13 AT 4PM"", ""KOLD AM NEWS 430AM""], ""line_item__start_date"": [""9/23"", ""9/23"", ""9/23"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/22"", ""9/22"", ""9/22"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/22"", ""9/22"", ""9/22"", ""9/22"", ""9/22"", ""9/22"", ""9/22"", ""9/22"", ""9/22"", ""9/25"", ""9/22"", ""9/22"", ""9/22"", ""9/22"", ""9/22"", ""9/22"", ""9/22"", ""9/22"", ""9/26"", ""9/26"", ""9/26"", ""9/24"", ""9/24"", ""9/24"", ""9/25"", ""9/25"", ""9/25"", ""9/26"", ""9/26"", ""9/26"", ""9/26"", ""9/26"", ""9/26"", ""9/26"", ""9/26"", ""9/26"", ""9/23"", ""9/26"", ""9/26"", ""9/26"", ""9/26"", ""9/26"", ""9/26"", ""9/26"", ""9/26"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23""], ""line_item__end_date"": [""9/23"", ""9/23"", ""9/23"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/21"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/22"", ""9/22"", ""9/22"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/20"", ""9/22"", ""9/22"", ""9/22"", ""9/22"", ""9/22"", ""9/22"", ""9/22"", ""9/22"", ""9/22"", ""9/25"", ""9/22"", ""9/22"", ""9/22"", ""9/22"", ""9/22"", ""9/22"", ""9/22"", ""9/22"", ""9/26"", ""9/26"", ""9/26"", ""9/24"", ""9/24"", ""9/24"", ""9/25"", ""9/25"", ""9/25"", ""9/26"", ""9/26"", ""9/26"", ""9/26"", ""9/26"", ""9/26"", ""9/26"", ""9/26"", ""9/26"", ""9/23"", ""9/26"", ""9/26"", ""9/26"", ""9/26"", ""9/26"", ""9/26"", ""9/26"", ""9/26"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23"", ""9/23""], ""line_item__days"": [""FRI"", ""FRI"", ""FRI"", ""TUE"", ""TUE"", ""TUE"", ""TUE"", ""TUE"", ""TUE"", ""TUE"", ""WED"", ""WED"", ""WED"", ""WED"", ""WED"", ""WED"", ""WED"", ""WED"", ""WED"", ""WED"", ""WED"", ""WED"", ""WED"", ""WED"", ""WED"", ""WED"", ""WED"", ""WED"", ""WED"", ""WED"", ""WED"", ""TUE"", ""TUE"", ""TUE"", ""TUE"", ""TUE"", ""TUE"", ""TUE"", ""TUE"", ""TUE"", ""THU"", ""THU"", ""THU"", ""TUE"", ""TUE"", ""TUE"", ""TUE"", ""TUE"", ""TUE"", ""THU"", ""THU"", ""THU"", ""THU"", ""THU"", ""THU"", ""THU"", ""THU"", ""THU"", ""SUN"", ""THU"", ""THU"", ""THU"", ""THU"", ""THU"", ""THU"", ""THU"", ""THU"", ""MON"", ""MON"", ""MON"", ""SAT"", ""SAT"", ""SAT"", ""SUN"", ""SUN"", ""SUN"", ""MON"", ""MON"", ""MON"", ""MON"", ""MON"", ""MON"", ""MON"", ""MON"", ""MON"", ""FRI"", ""MON"", ""MON"", ""MON"", ""MON"", ""MON"", ""MON"", ""MON"", ""MON"", ""FRI"", ""FRI"", ""FRI"", ""FRI"", ""FRI"", ""FRI"", ""FRI"", ""FRI"", ""FRI"", ""FRI"", ""FRI"", ""FRI"", ""FRI"", ""FRI"", ""FRI"", ""FRI"", ""FRI"", ""FRI""]}" diff --git a/config_library/pattern-2/fcc-invoices/sr_FCC_config.json b/config_library/pattern-2/fcc-invoices/sr_FCC_config.json new file mode 100644 index 000000000..6514f1cd2 --- /dev/null +++ b/config_library/pattern-2/fcc-invoices/sr_FCC_config.json @@ -0,0 +1,848 @@ +{ + "notes": "Configuration for FCC invoice information extraction (no classification)", + "ocr": { + "backend": "textract", + "model_id": "us.anthropic.claude-3-7-sonnet-20250219-v1:0", + "system_prompt": "You are an expert OCR system. Extract all text from the provided image accurately, preserving layout where possible.", + "task_prompt": "Extract all text from this document image. Preserve the layout, including paragraphs, tables, and formatting.", + "features": [ + { + "name": "LAYOUT" + }, + { + "name": "TABLES" + } + ], + "max_workers": 20, + "image": { + "target_width": null, + "target_height": null, + "dpi": 150, + "preprocessing": null + } + }, + "classification": { + "model": "us.anthropic.claude-3-7-sonnet-20250219-v1:0", + "system_prompt": "You are a document classification expert. Classify each page according to the provided document types.", + "task_prompt": "Classify this page as one of the available document types based on its content and structure.", + "temperature": 0, + "top_p": 0.9, + "top_k": 5, + "max_tokens": 1024, + "maxPagesForClassification": 0, + "classificationMethod": "textbasedHolisticClassification", + "image": { + "target_width": null, + "target_height": null, + "dpi": null, + "preprocessing": null + } + }, + "extraction": { + "model": "us.anthropic.claude-3-7-sonnet-20250219-v1:0", + "system_prompt": "You are an expert at extracting structured information from FCC political advertising invoices.\nExtract all requested fields accurately, paying special attention to:\n- Line item details including time slots, days, rates, and date ranges\n- Monetary amounts (preserve formatting with commas and decimals)\n- Date formats (typically MM/DD/YY)\n- Agency and advertiser names\n\nFor line items, ensure you capture all rows from any tables showing broadcast schedules.\nDays of week are often encoded as 7-character strings where each position represents a day.\n\nIMPORTANT: Return data in a FLAT structure where:\n- Simple fields (agency, advertiser, gross_total, net_amount_due) are arrays with single values\n- Line item fields use the prefix 'line_item__' and are parallel arrays (same length)\n- Each line item field is an array where index N corresponds to the Nth line item\n", + "task_prompt": "Extract the following attributes from this FCC invoice. Return data in a FLAT array format where:\n- Simple fields are arrays with single values\n- Line item fields use the 'line_item__' prefix and are parallel arrays (same length)\n- Each line item field array index N corresponds to the Nth line item\n\n\n\n{ATTRIBUTE_NAMES_AND_DESCRIPTIONS}\n\n\n\n<>\n\n\n\n{DOCUMENT_TEXT}\n\n\n\n\n\n{DOCUMENT_IMAGE}\n\n\n\n\n\nExtract the attributes listed above and return them in the exact JSON schema format specified. Ensure all line_item__ arrays have the same length.\n\n\n", + "temperature": 0, + "top_p": 0.9, + "top_k": 5, + "max_tokens": 4096, + "image": { + "target_width": null, + "target_height": null, + "dpi": null, + "preprocessing": null + }, + "agentic": { + "enabled": false, + "review_agent": false + }, + "custom_prompt_lambda_arn": null + }, + "assessment": { + "enabled": false, + "model": "us.amazon.nova-lite-v1:0", + "system_prompt": "You are a document analysis assessment expert. Your role is to evaluate the confidence and accuracy of data extraction results by analyzing them against source documents.\nProvide accurate confidence scores for each assessment. When bounding boxes are requested, provide precise coordinate locations where information appears in the document.", + "task_prompt": " You are an expert document analysis assessment system. Your task is to evaluate the confidence of extraction results for a document of class {DOCUMENT_CLASS} and provide precise spatial localization for each field. \n Analyze the extraction results against the source document and provide confidence assessments AND bounding box coordinates for each extracted attribute. Consider factors such as: 1. Text clarity and OCR quality in the source regions 2. Alignment between extracted values and document content 3. Presence of clear evidence supporting the extraction 4. Potential ambiguity or uncertainty in the source material 5. Completeness and accuracy of the extracted information 6. Precise spatial location of each field in the document \n For each attribute, provide: - A confidence score between 0.0 and 1.0 where:\n - 1.0 = Very high confidence, clear and unambiguous evidence\n - 0.8-0.9 = High confidence, strong evidence with minor uncertainty\n - 0.6-0.7 = Medium confidence, reasonable evidence but some ambiguity\n - 0.4-0.5 = Low confidence, weak or unclear evidence\n - 0.0-0.3 = Very low confidence, little to no supporting evidence\n- A clear explanation of the confidence reasoning - Precise spatial coordinates where the field appears in the document\nGuidelines: - Base assessments on actual document content and OCR quality - Consider both text-based evidence and visual/layout clues - Account for OCR confidence scores when provided - Be objective and specific in reasoning - If an extraction appears incorrect, score accordingly with explanation - Provide tight, accurate bounding boxes around the actual text \n For each field, provide bounding box coordinates: - bbox: [x1, y1, x2, y2] coordinates in normalized 0-1000 scale - page: Page number where the field appears (starting from 1)\nCoordinate system: - Use normalized scale 0-1000 for both x and y axes - x1, y1 = top-left corner of bounding box - x2, y2 = bottom-right corner of bounding box - Ensure x2 > x1 and y2 > y1 - Make bounding boxes tight around the actual text content - If a field spans multiple lines, create a bounding box that encompasses all relevant text \n Analyze the extraction results against the source document and provide confidence assessments with spatial localization. Return a JSON object with the following structure based on the attribute type:\nFor SIMPLE attributes: {\n \"simple_attribute_name\": {\n \"confidence\": 0.85,\n \"bbox\": [100, 200, 300, 250],\n \"page\": 1\n }\n}\nFor GROUP attributes (nested object structure): {\n \"group_attribute_name\": {\n \"sub_attribute_1\": {\n \"confidence\": 0.90,\n \"bbox\": [150, 300, 250, 320],\n \"page\": 1\n },\n \"sub_attribute_2\": {\n \"confidence\": 0.75,\n \"bbox\": [150, 325, 280, 345],\n \"page\": 1\n }\n }\n}\nFor LIST attributes (array of assessed items): {\n \"list_attribute_name\": [\n {\n \"item_attribute_1\": {\n \"confidence\": 0.95,\n \"bbox\": [100, 400, 200, 420],\n \"page\": 1\n },\n \"item_attribute_2\": {\n \"confidence\": 0.88,\n \"bbox\": [250, 400, 350, 420],\n \"page\": 1\n }\n },\n {\n \"item_attribute_1\": {\n \"confidence\": 0.92,\n \"bbox\": [100, 425, 200, 445],\n \"page\": 1\n },\n \"item_attribute_2\": {\n \"confidence\": 0.70,\n \"bbox\": [250, 425, 350, 445],\n \"page\": 1\n }\n }\n ]\n}\nIMPORTANT: - For LIST attributes like \"Transactions\", assess EACH individual item in the list separately with individual bounding boxes - Each transaction should be assessed as a separate object in the array with its own spatial coordinates - Do NOT provide aggregate assessments for list items - assess each one individually with precise locations - Include assessments AND bounding boxes for ALL attributes present in the extraction results - Match the exact structure of the extracted data - Provide page numbers for all bounding boxes (starting from 1) \n<>\n {DOCUMENT_IMAGE} \n {OCR_TEXT_CONFIDENCE} \n<>\n {ATTRIBUTE_NAMES_AND_DESCRIPTIONS} \n {EXTRACTION_RESULTS} ", + "temperature": 0, + "top_p": 0.1, + "top_k": 5, + "max_tokens": 10000, + "default_confidence_threshold": 0.8, + "validation_enabled": false, + "image": { + "target_width": null, + "target_height": null, + "dpi": null, + "preprocessing": null + }, + "granular": { + "enabled": true, + "list_batch_size": 1, + "simple_batch_size": 3, + "max_workers": 20 + } + }, + "summarization": { + "enabled": false, + "model": "us.anthropic.claude-3-7-sonnet-20250219-v1:0", + "system_prompt": "You are a document summarization expert who can analyze and summarize documents from various domains including medical, financial, legal, and general business documents. Your task is to create a summary that captures the key information, main points, and important details from the document. Your output must be in valid JSON format. \\nSummarization Style: Balanced\\\\nCreate a balanced summary that provides a moderate level of detail. Include the main points and key supporting information, while maintaining the document's overall structure. Aim for a comprehensive yet concise summary.\\n Your output MUST be in valid JSON format with markdown content. You MUST strictly adhere to the output format specified in the instructions.", + "task_prompt": "\n{DOCUMENT_TEXT}\n\nAnalyze the provided document () and create a comprehensive summary.\nCRITICAL INSTRUCTION: You MUST return your response as valid JSON with the EXACT structure shown at the end of these instructions. Do not include any explanations, notes, or text outside of the JSON structure.\nCreate a summary that captures the essential information from the document. Your summary should:\n1. Extract key information, main points, and important details\n2. Maintain the original document's organizational structure where appropriate\n3. Preserve important facts, figures, dates, and entities\n4. Reduce the length while retaining all critical information\n5. Use markdown formatting for better readability (headings, lists, emphasis, etc.)\n6. Cite all relevant facts from the source document using inline citations in the format [Cite-X, Page-Y] where X is a sequential citation number and Y is the page number\n7. Format citations as markdown links that reference the full citation list at the bottom of the summary\n Example: [[Cite-1, Page-3]](#cite-1-page-3)\n\n8. At the end of the summary, include a \"References\" section that lists all citations with their exact text from the source document in the format:\n [Cite-X, Page-Y]: Exact text from the document\n\nOutput Format:\nYou MUST return ONLY valid JSON with the following structure and nothing else:\n```json {\n \"summary\": \"A comprehensive summary in markdown format with inline citations linked to a references section at the bottom\"\n} ```\nDo not include any text, explanations, or notes outside of this JSON structure. The JSON must be properly formatted and parseable.", + "temperature": 0, + "top_p": 0.1, + "top_k": 5, + "max_tokens": 4096 + }, + "criteria_validation": { + "model": "us.anthropic.claude-3-5-sonnet-20240620-v1:0", + "system_prompt": "", + "task_prompt": "", + "temperature": 0, + "top_p": 0.01, + "top_k": 20, + "max_tokens": 4096, + "semaphore": 3, + "max_chunk_size": 180000, + "token_size": 4, + "overlap_percentage": 10, + "response_prefix": "" + }, + "agents": { + "error_analyzer": { + "model_id": "us.anthropic.claude-sonnet-4-20250514-v1:0", + "error_patterns": [ + "ERROR", + "CRITICAL", + "FATAL", + "Exception", + "Traceback", + "Failed", + "Timeout", + "AccessDenied", + "ThrottlingException" + ], + "system_prompt": "You are an intelligent error analysis agent for the GenAI IDP system.\n\nUse the analyze_errors tool to investigate issues. ALWAYS format your response with exactly these three sections in this order:\n\n## Root Cause\nIdentify the specific underlying technical reason why the error occurred. Focus on the primary cause, not symptoms.\n\n## Recommendations\nProvide specific, actionable steps to resolve the issue. Limit to top three recommendations only.\n\n
\nEvidence\n\nFormat log entries with their source information. For each log entry, show:\n**Log Group:** \n[full log_group name from tool response]\n\n**Log Stream:** \n[full log_stream name from tool response]\n```\n[ERROR] timestamp message (from events data)\n```\n\n
\n\nFORMATTING RULES:\n- Use the exact three-section structure above\n- Make Evidence section collapsible using HTML details tags\n- Extract log_group, log_stream, and events data from tool response\n- Show complete log group and log stream names without truncation\n- Present actual log messages from events array in code blocks\n\nANALYSIS GUIDELINES:\n- If has_performance_issues is false, focus on application logic errors\n- Use service timeline to rule out infrastructure bottlenecks\n- Service response times help eliminate timeout-related causes\n- For application errors use CloudWatch error messages for recommendations\n\nROOT CAUSE DETERMINATION:\n- Start with Step Function failure details (most specific)\n- Validate with CloudWatch error logs (most detailed)\n- Use X-Ray to categorize as infrastructure vs. application issue\n- DynamoDB provides supporting timeline context only\n\nRECOMMENDATION GUIDELINES:\nFor code-related issues or system bugs:\n- Do not suggest code modifications\n- Include error details, timestamps, and context\n\nFor configuration-related issues:\n- Direct users to UI configuration panel\n- Specify exact configuration section and parameter names\n\nFor operational issues:\n- Provide immediate troubleshooting steps\n- Include preventive measures\n\nTIME RANGE PARSING:\n- recent/recently: 1 hour\n- last week: 168 hours \n- last day/yesterday: 24 hours\n- No time specified: 24 hours (default)\n\nSPECIAL CASES:\nIf analysis_type is \"document_not_found\": explain document cannot be located, focus on verification steps and processing issues.\n\nDO NOT include code suggestions, technical summaries, or multiple paragraphs of explanation. Keep responses concise and actionable.\n\nIMPORTANT: Do not include any search quality reflections, search quality scores, or meta-analysis sections in your response. Only provide the three required sections: Root Cause, Recommendations, and Evidence.", + "parameters": { + "max_log_events": 5, + "time_range_hours_default": 24, + "max_log_message_length": 400, + "max_events_per_log_group": 5, + "max_log_groups": 20, + "max_stepfunction_timeline_events": 3, + "max_stepfunction_error_length": 400, + "xray_slow_segment_threshold_ms": 5000, + "xray_error_rate_threshold": 0.05, + "xray_response_time_threshold_ms": 10000 + } + }, + "chat_companion": { + "model_id": "us.anthropic.claude-sonnet-4-20250514-v1:0:1m", + "error_patterns": [ + "ERROR", + "CRITICAL", + "FATAL", + "Exception", + "Traceback", + "Failed", + "Timeout", + "AccessDenied", + "ThrottlingException" + ], + "system_prompt": "\n You are an intelligent error analysis agent for the GenAI IDP system with access to specialized diagnostic tools.\n\n GENERAL TROUBLESHOOTING WORKFLOW:\n 1. Identify document status from DynamoDB\n 2. Find any errors reported during Step Function execution\n 3. Collect relevant logs from CloudWatch\n 4. Identify any performance issues from X-Ray traces\n 5. Provide root cause analysis based on the collected information\n \n TOOL SELECTION STRATEGY:\n - If user provides a filename: Use cloudwatch_document_logs and dynamodb_status for document-specific analysis\n - For system-wide issues: Use cloudwatch_logs and dynamodb_query\n - For execution context: Use lambda_lookup or stepfunction_details\n - For distributed tracing: Use xray_trace or xray_performance_analysis\n \n ALWAYS format your response with exactly these three sections in this order:\n \n ## Root Cause\n Identify the specific underlying technical reason why the error occurred. Focus on the primary cause, not symptoms.\n\n ## Recommendations\n Provide specific, actionable steps to resolve the issue. Limit to top three recommendations only.\n\n
\n Evidence\n \n Format evidence with source information. Include relevant data from tool responses:\n \n **For CloudWatch logs:**\n **Log Group:** [full log_group name]\n **Log Stream:** [full log_stream name]\n ```\n [ERROR] timestamp message\n ```\n \n **For other sources (DynamoDB, Step Functions, X-Ray):**\n **Source:** [service name and resource]\n ```\n Relevant data from tool response\n ```\n\n
\n\n FORMATTING RULES:\n - Use the exact three-section structure above\n - Make Evidence section collapsible using HTML details tags\n - Include relevant data from all tool responses (CloudWatch, DynamoDB, Step Functions, X-Ray)\n - For CloudWatch: Show complete log group and log stream names without truncation\n - Present evidence data in code blocks with appropriate source labels\n \n ANALYSIS GUIDELINES:\n - Use multiple tools for comprehensive analysis when needed\n - Start with document-specific tools for targeted queries\n - Use system-wide tools for pattern analysis\n - Combine DynamoDB status with CloudWatch logs for complete picture\n - Leverage X-Ray for distributed system issues\n \n ROOT CAUSE DETERMINATION:\n 1. Document Status: Check dynamodb_status first\n 2. Execution Details: Use stepfunction_details for workflow failures\n 3. Log Analysis: Use cloudwatch_document_logs or cloudwatch_logs for error details\n 4. Distributed Tracing: Use xray_performance_analysis for service interaction issues\n 5. Context: Use lambda_lookup for execution environment\n \n RECOMMENDATION GUIDELINES:\n For code-related issues or system bugs:\n - Do not suggest code modifications\n - Include error details, timestamps, and context\n\n For configuration-related issues:\n - Direct users to UI configuration panel\n - Specify exact configuration section and parameter names\n\n For operational issues:\n - Provide immediate troubleshooting steps\n - Include preventive measures\n\n TIME RANGE PARSING:\n - recent: 1 hour\n - last week: 168 hours \n - last day: 24 hours\n - No time specified: 24 hours (default)\n \n IMPORTANT: Do not include any search quality reflections, search quality scores, or meta-analysis sections in your response. Only provide the three required sections: Root Cause, Recommendations, and Evidence.", + "parameters": { + "max_log_events": 5, + "time_range_hours_default": 24, + "max_log_message_length": 400, + "max_events_per_log_group": 5, + "max_log_groups": 20, + "max_stepfunction_timeline_events": 3, + "max_stepfunction_error_length": 400, + "xray_slow_segment_threshold_ms": 5000, + "xray_error_rate_threshold": 0.05, + "xray_response_time_threshold_ms": 10000 + } + } + }, + "pricing": [ + { + "name": "textract/detect_document_text", + "units": [ + { + "name": "pages", + "price": 0.0015 + } + ] + }, + { + "name": "textract/analyze_document-Layout", + "units": [ + { + "name": "pages", + "price": 0.004 + } + ] + }, + { + "name": "textract/analyze_document-Signatures", + "units": [ + { + "name": "pages", + "price": 0.0035 + } + ] + }, + { + "name": "textract/analyze_document-Forms", + "units": [ + { + "name": "pages", + "price": 0.05 + } + ] + }, + { + "name": "textract/analyze_document-Tables", + "units": [ + { + "name": "pages", + "price": 0.015 + } + ] + }, + { + "name": "textract/analyze_document-Tables+Forms", + "units": [ + { + "name": "pages", + "price": 0.065 + } + ] + }, + { + "name": "bedrock/us.amazon.nova-lite-v1:0", + "units": [ + { + "name": "inputTokens", + "price": 6e-8 + }, + { + "name": "outputTokens", + "price": 2.4e-7 + }, + { + "name": "cacheReadInputTokens", + "price": 1.5e-8 + }, + { + "name": "cacheWriteInputTokens", + "price": 6e-8 + } + ] + }, + { + "name": "bedrock/us.amazon.nova-pro-v1:0", + "units": [ + { + "name": "inputTokens", + "price": 8e-7 + }, + { + "name": "outputTokens", + "price": 0.0000032 + }, + { + "name": "cacheReadInputTokens", + "price": 2e-7 + }, + { + "name": "cacheWriteInputTokens", + "price": 8e-7 + } + ] + }, + { + "name": "bedrock/us.amazon.nova-premier-v1:0", + "units": [ + { + "name": "inputTokens", + "price": 0.0000025 + }, + { + "name": "outputTokens", + "price": 0.0000125 + } + ] + }, + { + "name": "bedrock/us.anthropic.claude-3-haiku-20240307-v1:0", + "units": [ + { + "name": "inputTokens", + "price": 2.5e-7 + }, + { + "name": "outputTokens", + "price": 0.00000125 + } + ] + }, + { + "name": "bedrock/us.anthropic.claude-3-5-haiku-20241022-v1:0", + "units": [ + { + "name": "inputTokens", + "price": 8e-7 + }, + { + "name": "outputTokens", + "price": 0.000004 + }, + { + "name": "cacheReadInputTokens", + "price": 8e-8 + }, + { + "name": "cacheWriteInputTokens", + "price": 0.000001 + } + ] + }, + { + "name": "bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0", + "units": [ + { + "name": "inputTokens", + "price": 0.0000011 + }, + { + "name": "outputTokens", + "price": 0.0000055 + }, + { + "name": "cacheReadInputTokens", + "price": 1.1e-7 + }, + { + "name": "cacheWriteInputTokens", + "price": 0.0000014 + } + ] + }, + { + "name": "bedrock/us.anthropic.claude-3-5-sonnet-20241022-v2:0", + "units": [ + { + "name": "inputTokens", + "price": 0.000003 + }, + { + "name": "outputTokens", + "price": 0.000015 + }, + { + "name": "cacheReadInputTokens", + "price": 3e-7 + }, + { + "name": "cacheWriteInputTokens", + "price": 0.00000375 + } + ] + }, + { + "name": "bedrock/us.anthropic.claude-3-7-sonnet-20250219-v1:0", + "units": [ + { + "name": "inputTokens", + "price": 0.000003 + }, + { + "name": "outputTokens", + "price": 0.000015 + }, + { + "name": "cacheReadInputTokens", + "price": 3e-7 + }, + { + "name": "cacheWriteInputTokens", + "price": 0.00000375 + } + ] + }, + { + "name": "bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0", + "units": [ + { + "name": "inputTokens", + "price": 0.000003 + }, + { + "name": "outputTokens", + "price": 0.000015 + }, + { + "name": "cacheReadInputTokens", + "price": 3e-7 + }, + { + "name": "cacheWriteInputTokens", + "price": 0.00000375 + } + ] + }, + { + "name": "bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0:1m", + "units": [ + { + "name": "inputTokens", + "price": 0.000006 + }, + { + "name": "outputTokens", + "price": 0.0000225 + }, + { + "name": "cacheReadInputTokens", + "price": 6e-7 + }, + { + "name": "cacheWriteInputTokens", + "price": 0.0000075 + } + ] + }, + { + "name": "bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0", + "units": [ + { + "name": "inputTokens", + "price": 0.0000033 + }, + { + "name": "outputTokens", + "price": 0.0000165 + }, + { + "name": "cacheReadInputTokens", + "price": 3.3e-7 + }, + { + "name": "cacheWriteInputTokens", + "price": 0.000004125 + } + ] + }, + { + "name": "bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0:1m", + "units": [ + { + "name": "inputTokens", + "price": 0.0000066 + }, + { + "name": "outputTokens", + "price": 0.00002475 + }, + { + "name": "cacheReadInputTokens", + "price": 6.6e-7 + }, + { + "name": "cacheWriteInputTokens", + "price": 0.00000825 + } + ] + }, + { + "name": "bedrock/us.anthropic.claude-opus-4-20250514-v1:0", + "units": [ + { + "name": "inputTokens", + "price": 0.000015 + }, + { + "name": "outputTokens", + "price": 0.000075 + }, + { + "name": "cacheReadInputTokens", + "price": 0.0000015 + }, + { + "name": "cacheWriteInputTokens", + "price": 0.00001875 + } + ] + }, + { + "name": "bedrock/us.anthropic.claude-opus-4-1-20250805-v1:0", + "units": [ + { + "name": "inputTokens", + "price": 0.000015 + }, + { + "name": "outputTokens", + "price": 0.000075 + }, + { + "name": "cacheReadInputTokens", + "price": 0.0000015 + }, + { + "name": "cacheWriteInputTokens", + "price": 0.00001875 + } + ] + }, + { + "name": "bedrock/eu.amazon.nova-lite-v1:0", + "units": [ + { + "name": "inputTokens", + "price": 7.8e-8 + }, + { + "name": "outputTokens", + "price": 3.1e-7 + }, + { + "name": "cacheReadInputTokens", + "price": 1.9e-8 + }, + { + "name": "cacheWriteInputTokens", + "price": 7.8e-8 + } + ] + }, + { + "name": "bedrock/eu.amazon.nova-pro-v1:0", + "units": [ + { + "name": "inputTokens", + "price": 0.000001 + }, + { + "name": "outputTokens", + "price": 0.0000042 + }, + { + "name": "cacheReadInputTokens", + "price": 2.6e-7 + }, + { + "name": "cacheWriteInputTokens", + "price": 0.000001 + } + ] + }, + { + "name": "bedrock/eu.anthropic.claude-3-haiku-20240307-v1:0", + "units": [ + { + "name": "inputTokens", + "price": 2.5e-7 + }, + { + "name": "outputTokens", + "price": 0.00000125 + } + ] + }, + { + "name": "bedrock/eu.anthropic.claude-haiku-4-5-20251001-v1:0", + "units": [ + { + "name": "inputTokens", + "price": 0.0000011 + }, + { + "name": "outputTokens", + "price": 0.0000055 + }, + { + "name": "cacheReadInputTokens", + "price": 1.1e-7 + }, + { + "name": "cacheWriteInputTokens", + "price": 0.0000014 + } + ] + }, + { + "name": "bedrock/eu.anthropic.claude-3-5-sonnet-20241022-v2:0", + "units": [ + { + "name": "inputTokens", + "price": 0.000003 + }, + { + "name": "outputTokens", + "price": 0.000015 + }, + { + "name": "cacheReadInputTokens", + "price": 3e-7 + }, + { + "name": "cacheWriteInputTokens", + "price": 0.00000375 + } + ] + }, + { + "name": "bedrock/eu.anthropic.claude-3-7-sonnet-20250219-v1:0", + "units": [ + { + "name": "inputTokens", + "price": 0.000003 + }, + { + "name": "outputTokens", + "price": 0.000015 + }, + { + "name": "cacheReadInputTokens", + "price": 3e-7 + }, + { + "name": "cacheWriteInputTokens", + "price": 0.00000375 + } + ] + }, + { + "name": "bedrock/eu.anthropic.claude-sonnet-4-20250514-v1:0", + "units": [ + { + "name": "inputTokens", + "price": 0.000003 + }, + { + "name": "outputTokens", + "price": 0.000015 + }, + { + "name": "cacheReadInputTokens", + "price": 3e-7 + }, + { + "name": "cacheWriteInputTokens", + "price": 0.00000375 + } + ] + }, + { + "name": "bedrock/eu.anthropic.claude-sonnet-4-5-20250929-v1:0", + "units": [ + { + "name": "inputTokens", + "price": 0.0000033 + }, + { + "name": "outputTokens", + "price": 0.0000165 + }, + { + "name": "cacheReadInputTokens", + "price": 3.3e-7 + }, + { + "name": "cacheWriteInputTokens", + "price": 0.000004125 + } + ] + }, + { + "name": "bedrock/eu.anthropic.claude-sonnet-4-5-20250929-v1:0:1m", + "units": [ + { + "name": "inputTokens", + "price": 0.0000066 + }, + { + "name": "outputTokens", + "price": 0.00002475 + }, + { + "name": "cacheReadInputTokens", + "price": 6.6e-7 + }, + { + "name": "cacheWriteInputTokens", + "price": 0.00000825 + } + ] + }, + { + "name": "lambda/requests", + "units": [ + { + "name": "invocations", + "price": 2e-7 + } + ] + }, + { + "name": "lambda/duration", + "units": [ + { + "name": "gb_seconds", + "price": 0.0000166667 + } + ] + } + ], + "classes": [ + { + "description": "Federal Communications Commission (FCC) political advertising invoice showing broadcast time purchases, including line items with descriptions, dates, rates, and totals for political advertising campaigns.", + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "x-aws-idp-document-type": "FCC-Invoice", + "x-aws-stickler-model-name": "FCCInvoice", + "x-aws-stickler-match-threshold": 0.7, + "properties": { + "agency": { + "type": "array", + "description": "The advertising agency or media buyer handling the political advertising purchase.", + "items": { + "type": "string" + }, + "x-aws-idp-evaluation-method": "EXACT", + "x-aws-stickler-comparator": "FuzzyComparator", + "x-aws-stickler-threshold": 0.8, + "x-aws-stickler-weight": 2.0 + }, + "advertiser": { + "type": "array", + "description": "The political advertiser or campaign purchasing the broadcast time.", + "items": { + "type": "string" + }, + "x-aws-idp-evaluation-method": "EXACT", + "x-aws-stickler-comparator": "FuzzyComparator", + "x-aws-stickler-threshold": 0.8, + "x-aws-stickler-weight": 2.0 + }, + "gross_total": { + "type": "array", + "description": "The total gross amount for all line items before any discounts or adjustments.", + "items": { + "type": "string" + }, + "x-aws-idp-evaluation-method": "NUMERIC_EXACT", + "x-aws-stickler-comparator": "ExactComparator", + "x-aws-stickler-threshold": 1.0, + "x-aws-stickler-weight": 3.0 + }, + "net_amount_due": { + "type": "array", + "description": "The final net amount due after any discounts or adjustments have been applied.", + "items": { + "type": "string" + }, + "x-aws-idp-evaluation-method": "NUMERIC_EXACT", + "x-aws-stickler-comparator": "ExactComparator", + "x-aws-stickler-threshold": 1.0, + "x-aws-stickler-weight": 3.0 + }, + "line_item__description": { + "type": "array", + "description": "List of broadcast time slot descriptions (e.g., 'M-F 11a-12p' for Monday through Friday 11am to 12pm).", + "items": { + "type": "string" + }, + "x-aws-idp-evaluation-method": "EXACT", + "x-aws-stickler-comparator": "LevenshteinComparator", + "x-aws-stickler-threshold": 0.7, + "x-aws-stickler-weight": 1.5 + }, + "line_item__days": { + "type": "array", + "description": "List of days of the week for each broadcast slot (e.g., 'MTWTF--' where each position represents a day).", + "items": { + "type": "string" + }, + "x-aws-idp-evaluation-method": "EXACT", + "x-aws-stickler-comparator": "ExactComparator", + "x-aws-stickler-threshold": 1.0, + "x-aws-stickler-weight": 1.0 + }, + "line_item__rate": { + "type": "array", + "description": "List of rates or costs for each broadcast time slot (may include commas for thousands separator).", + "items": { + "type": "string" + }, + "x-aws-idp-evaluation-method": "NUMERIC_EXACT", + "x-aws-stickler-comparator": "ExactComparator", + "x-aws-stickler-threshold": 1.0, + "x-aws-stickler-weight": 2.0 + }, + "line_item__start_date": { + "type": "array", + "description": "List of start dates for each line item's broadcast schedule (typically in MM/DD/YY format).", + "items": { + "type": "string" + }, + "x-aws-idp-evaluation-method": "EXACT", + "x-aws-stickler-comparator": "ExactComparator", + "x-aws-stickler-threshold": 1.0, + "x-aws-stickler-weight": 2.0 + }, + "line_item__end_date": { + "type": "array", + "description": "List of end dates for each line item's broadcast schedule (typically in MM/DD/YY format).", + "items": { + "type": "string" + }, + "x-aws-idp-evaluation-method": "EXACT", + "x-aws-stickler-comparator": "ExactComparator", + "x-aws-stickler-threshold": 1.0, + "x-aws-stickler-weight": 2.0 + } + }, + "$id": "FCC-Invoice" + } + ], + "discovery": { + "without_ground_truth": { + "model_id": "us.amazon.nova-pro-v1:0", + "system_prompt": "You are an expert in processing forms. Extracting data from images and documents. Analyze forms line by line to identify field names, data types, and organizational structure. Focus on creating comprehensive blueprints for document processing without extracting actual values.", + "temperature": 1, + "top_p": 0.1, + "max_tokens": 10000, + "user_prompt": "This image contains forms data. Analyze the form line by line. Image may contains multiple pages, process all the pages. Form may contain multiple name value pair in one line. Extract all the names in the form including the name value pair which doesn't have value. Organize them into groups, extract field_name, data_type and field description Field_name should be less than 60 characters, should not have space use '-' instead of space. field_description is a brief description of the field and the location of the field like box number or line number in the form and section of the form. Field_name should be unique within the group. Add two fields document_class and document_description. For document_class generate a short name based on the document content like W4, I-9, Paystub. For document_description generate a description about the document in less than 50 words. \nGroup the fields based on the section they are grouped in the form. Group should have attributeType as \"group\". If the group repeats and follows table format, update the attributeType as \"list\". Do not extract the values. Return the extracted data in JSON format. Format the extracted data using the below JSON format: Format the extracted groups and fields using the below JSON format:" + }, + "with_ground_truth": { + "model_id": "us.amazon.nova-pro-v1:0", + "system_prompt": "You are an expert in processing forms. Extracting data from images and documents. Use provided ground truth data as reference to optimize field extraction and ensure consistency with expected document structure and field definitions.", + "temperature": 1, + "top_p": 0.1, + "max_tokens": 10000, + "user_prompt": "This image contains unstructured data. Analyze the data line by line using the provided ground truth as reference. {ground_truth_json} Ground truth reference JSON has the fields we are interested in extracting from the document/image. Use the ground truth to optimize field extraction. Match field names, data types, and groupings from the reference. Image may contain multiple pages, process all pages. Extract all field names including those without values. Do not change the group name and field name from ground truth in the extracted data json. Add field_description field for every field which will contain instruction to LLM to extract the field data from the image/document. Add data_type field for every field. Add two fields document_class and document_description. For document_class generate a short name based on the document content like W4, I-9, Paystub. For document_description generate a description about the document in less than 50 words. If the group repeats and follows table format, update the attributeType as \"list\". Do not extract the values. Format the extracted data using the below JSON format: Format the extracted groups and fields using the below JSON format:" + } + }, + "evaluation": { + "enabled": true, + "llm_method": { + "top_p": 0.1, + "max_tokens": 4096, + "top_k": 5, + "task_prompt": "I need to evaluate attribute extraction for a document of class: {DOCUMENT_CLASS}.\n\nFor the attribute named \"{ATTRIBUTE_NAME}\" described as \"{ATTRIBUTE_DESCRIPTION}\":\n- Expected value: {EXPECTED_VALUE}\n- Actual value: {ACTUAL_VALUE}\n\nDo these values match in meaning, taking into account formatting differences, word order, abbreviations, and semantic equivalence?\nProvide your assessment as a JSON with three fields:\n- \"match\": boolean (true if they match, false if not)\n- \"score\": number between 0 and 1 representing the confidence/similarity score\n- \"reason\": brief explanation of your decision\n\nRespond ONLY with the JSON and nothing else. Here's the exact format:\n{\n \"match\": true or false,\n \"score\": 0.0 to 1.0,\n \"reason\": \"Your explanation here\"\n}", + "temperature": 0, + "model": "us.anthropic.claude-3-haiku-20240307-v1:0", + "system_prompt": "You are an evaluator that helps determine if the predicted and expected values match for document attribute extraction. You will consider the context and meaning rather than just exact string matching." + } + }, + "summary": null, + "criteria_types": null, + "request_bucket": null, + "request_history_prefix": null, + "criteria_bucket": null, + "output_bucket": null, + "textract_page_tracker": null, + "cost_report_bucket": null +} \ No newline at end of file diff --git a/config_library/pattern-2/fcc-invoices/stickler_config.json b/config_library/pattern-2/fcc-invoices/stickler_config.json new file mode 100644 index 000000000..2121a0ffb --- /dev/null +++ b/config_library/pattern-2/fcc-invoices/stickler_config.json @@ -0,0 +1,87 @@ +{ + "model_name": "FCCInvoice", + "match_threshold": 0.7, + "fields": { + "agency": { + "type": "list", + "comparator": "FuzzyComparator", + "threshold": 0.8, + "weight": 2.0, + "required": false, + "default": [], + "description": "The advertising agency or media buyer handling the political advertising purchase" + }, + "advertiser": { + "type": "list", + "comparator": "FuzzyComparator", + "threshold": 0.8, + "weight": 2.0, + "required": false, + "default": [], + "description": "The political advertiser or campaign purchasing the broadcast time" + }, + "gross_total": { + "type": "list", + "comparator": "ExactComparator", + "threshold": 1.0, + "weight": 3.0, + "required": false, + "default": [], + "description": "The total gross amount for all line items before any discounts or adjustments (stored as string with commas)" + }, + "net_amount_due": { + "type": "list", + "comparator": "ExactComparator", + "threshold": 1.0, + "weight": 3.0, + "required": false, + "default": [], + "description": "The final net amount due after any discounts or adjustments have been applied (stored as string with commas)" + }, + "line_item__description": { + "type": "list", + "comparator": "LevenshteinComparator", + "threshold": 0.7, + "weight": 1.5, + "required": false, + "default": [], + "description": "List of broadcast time slot descriptions (e.g., 'M-F 11a-12p' for Monday through Friday 11am to 12pm)" + }, + "line_item__days": { + "type": "list", + "comparator": "ExactComparator", + "threshold": 1.0, + "weight": 1.0, + "required": false, + "default": [], + "description": "List of days of the week for each broadcast slot (e.g., 'MTWTF--' where each position represents a day)" + }, + "line_item__rate": { + "type": "list", + "comparator": "ExactComparator", + "threshold": 1.0, + "weight": 2.0, + "required": false, + "default": [], + "description": "List of rates or costs for each broadcast time slot (may include commas for thousands separator)" + }, + "line_item__start_date": { + "type": "list", + "comparator": "ExactComparator", + "threshold": 1.0, + "weight": 2.0, + "required": false, + "default": [], + "description": "List of start dates for each line item's broadcast schedule (typically in MM/DD/YY format)" + }, + "line_item__end_date": { + "type": "list", + "comparator": "ExactComparator", + "threshold": 1.0, + "weight": 2.0, + "required": false, + "default": [], + "description": "List of end dates for each line item's broadcast schedule (typically in MM/DD/YY format)" + } + } +} diff --git a/evaluation_output-idp-config/aggregated_metrics.json b/evaluation_output-idp-config/aggregated_metrics.json new file mode 100644 index 000000000..730abcaf3 --- /dev/null +++ b/evaluation_output-idp-config/aggregated_metrics.json @@ -0,0 +1,208 @@ +{ + "summary": { + "documents_processed": 3, + "errors": 0 + }, + "overall_metrics": { + "precision": 0.5185185185185185, + "recall": 1.0, + "f1_score": 0.6829268292682926, + "accuracy": 0.5185185185185185, + "tp": 14, + "fp": 13, + "tn": 0, + "fn": 0, + "fp1": 2, + "fp2": 11, + "total": 27 + }, + "field_metrics": { + "agency": { + "precision": 0.6666666666666666, + "recall": 1.0, + "f1_score": 0.8, + "accuracy": 0.6666666666666666, + "tp": 2, + "fp": 1, + "tn": 0, + "fn": 0, + "fp1": 1, + "fp2": 0, + "total": 3 + }, + "advertiser": { + "precision": 0.3333333333333333, + "recall": 1.0, + "f1_score": 0.5, + "accuracy": 0.3333333333333333, + "tp": 1, + "fp": 2, + "tn": 0, + "fn": 0, + "fp1": 0, + "fp2": 2, + "total": 3 + }, + "gross_total": { + "precision": 0.6666666666666666, + "recall": 1.0, + "f1_score": 0.8, + "accuracy": 0.6666666666666666, + "tp": 2, + "fp": 1, + "tn": 0, + "fn": 0, + "fp1": 0, + "fp2": 1, + "total": 3 + }, + "net_amount_due": { + "precision": 0.6666666666666666, + "recall": 1.0, + "f1_score": 0.8, + "accuracy": 0.6666666666666666, + "tp": 2, + "fp": 1, + "tn": 0, + "fn": 0, + "fp1": 1, + "fp2": 0, + "total": 3 + }, + "line_item__description": { + "precision": 0.0, + "recall": 0.0, + "f1_score": 0.0, + "accuracy": 0.0, + "tp": 0, + "fp": 3, + "tn": 0, + "fn": 0, + "fp1": 0, + "fp2": 3, + "total": 3 + }, + "line_item__days": { + "precision": 0.6666666666666666, + "recall": 1.0, + "f1_score": 0.8, + "accuracy": 0.6666666666666666, + "tp": 2, + "fp": 1, + "tn": 0, + "fn": 0, + "fp1": 0, + "fp2": 1, + "total": 3 + }, + "line_item__rate": { + "precision": 0.3333333333333333, + "recall": 1.0, + "f1_score": 0.5, + "accuracy": 0.3333333333333333, + "tp": 1, + "fp": 2, + "tn": 0, + "fn": 0, + "fp1": 0, + "fp2": 2, + "total": 3 + }, + "line_item__start_date": { + "precision": 0.6666666666666666, + "recall": 1.0, + "f1_score": 0.8, + "accuracy": 0.6666666666666666, + "tp": 2, + "fp": 1, + "tn": 0, + "fn": 0, + "fp1": 0, + "fp2": 1, + "total": 3 + }, + "line_item__end_date": { + "precision": 0.6666666666666666, + "recall": 1.0, + "f1_score": 0.8, + "accuracy": 0.6666666666666666, + "tp": 2, + "fp": 1, + "tn": 0, + "fn": 0, + "fp1": 0, + "fp2": 1, + "total": 3 + } + }, + "errors": [], + "stickler_config_used": { + "model_name": "FCCInvoice", + "match_threshold": 0.7, + "fields": { + "agency": { + "type": "list", + "comparator": "FuzzyComparator", + "threshold": 0.8, + "weight": 2.0, + "description": "The advertising agency or media buyer handling the political advertising purchase." + }, + "advertiser": { + "type": "list", + "comparator": "FuzzyComparator", + "threshold": 0.8, + "weight": 2.0, + "description": "The political advertiser or campaign purchasing the broadcast time." + }, + "gross_total": { + "type": "list", + "comparator": "ExactComparator", + "threshold": 1.0, + "weight": 3.0, + "description": "The total gross amount for all line items before any discounts or adjustments." + }, + "net_amount_due": { + "type": "list", + "comparator": "ExactComparator", + "threshold": 1.0, + "weight": 3.0, + "description": "The final net amount due after any discounts or adjustments have been applied." + }, + "line_item__description": { + "type": "list", + "comparator": "LevenshteinComparator", + "threshold": 0.7, + "weight": 1.5, + "description": "List of broadcast time slot descriptions (e.g., 'M-F 11a-12p' for Monday through Friday 11am to 12pm)." + }, + "line_item__days": { + "type": "list", + "comparator": "ExactComparator", + "threshold": 1.0, + "weight": 1.0, + "description": "List of days of the week for each broadcast slot (e.g., 'MTWTF--' where each position represents a day)." + }, + "line_item__rate": { + "type": "list", + "comparator": "ExactComparator", + "threshold": 1.0, + "weight": 2.0, + "description": "List of rates or costs for each broadcast time slot (may include commas for thousands separator)." + }, + "line_item__start_date": { + "type": "list", + "comparator": "ExactComparator", + "threshold": 1.0, + "weight": 2.0, + "description": "List of start dates for each line item's broadcast schedule (typically in MM/DD/YY format)." + }, + "line_item__end_date": { + "type": "list", + "comparator": "ExactComparator", + "threshold": 1.0, + "weight": 2.0, + "description": "List of end dates for each line item's broadcast schedule (typically in MM/DD/YY format)." + } + } + } +} \ No newline at end of file diff --git a/idp_cli/idp_cli/cli.py b/idp_cli/idp_cli/cli.py index 33ced577c..dead508f5 100644 --- a/idp_cli/idp_cli/cli.py +++ b/idp_cli/idp_cli/cli.py @@ -7,7 +7,9 @@ Command-line tool for batch document processing with the IDP Accelerator. """ +import fnmatch import logging +import os import sys import time from typing import Optional @@ -1084,8 +1086,6 @@ def generate_manifest( prefix = uri_parts[1] if len(uri_parts) > 1 else "" # List S3 objects - import fnmatch - import boto3 s3 = boto3.client("s3", region_name=region) @@ -1134,8 +1134,6 @@ def generate_manifest( f"[bold blue]Matching baselines from: {baseline_dir}[/bold blue]" ) - import os - baseline_path = os.path.abspath(baseline_dir) # Scan for baseline subdirectories diff --git a/lib/idp_common_pkg/examples/test_stickler_with_fcc_data.py b/lib/idp_common_pkg/examples/test_stickler_with_fcc_data.py new file mode 100644 index 000000000..527fedabf --- /dev/null +++ b/lib/idp_common_pkg/examples/test_stickler_with_fcc_data.py @@ -0,0 +1,260 @@ +#!/usr/bin/env python3 +""" +Test SticklerEvaluationService with actual FCC invoice data. + +This script demonstrates using SticklerEvaluationService to evaluate +real FCC invoice extraction results against ground truth labels. +""" + +import json +import os +import pandas as pd +from pathlib import Path +from idp_common.evaluation import SticklerEvaluationService +from idp_common.models import Section + + +def load_ground_truth_from_csv(csv_path: str, doc_id: str): + """ + Load ground truth labels from the refactored labels CSV. + + Args: + csv_path: Path to the CSV file with refactored_labels column + doc_id: Document ID to look up + + Returns: + Dictionary of ground truth labels + """ + df = pd.read_csv(csv_path) + + # Find the row for this document + row = df[df['doc_id'] == doc_id] + + if row.empty: + print(f"Warning: No ground truth found for doc_id: {doc_id}") + return None + + # Parse the refactored_labels JSON + labels_json = row['refactored_labels'].values[0] + + if pd.isna(labels_json): + print(f"Warning: refactored_labels is empty for doc_id: {doc_id}") + return None + + try: + labels = json.loads(labels_json) + return labels + except json.JSONDecodeError as e: + print(f"Error parsing refactored_labels JSON: {e}") + return None + + +def create_fcc_stickler_config(): + """ + Create a Stickler configuration for FCC invoices. + + Returns: + Configuration dictionary for SticklerEvaluationService + """ + config = { + "stickler_models": { + "fcc-invoice": { + "model_name": "FCCInvoice", + "match_threshold": 0.7, + "fields": { + "agency": { + "type": "str", + "comparator": "FuzzyComparator", + "threshold": 0.8, + "weight": 2.0, + }, + "advertiser": { + "type": "str", + "comparator": "FuzzyComparator", + "threshold": 0.8, + "weight": 2.0, + }, + "gross_total": { + "type": "str", # Stored as string with commas + "comparator": "ExactComparator", + "threshold": 1.0, + "weight": 3.0, + }, + "net_amount_due": { + "type": "str", # Stored as string with commas + "comparator": "ExactComparator", + "threshold": 1.0, + "weight": 3.0, + }, + "line_item__description": { + "type": "list", + "comparator": "LevenshteinComparator", + "threshold": 0.7, + "weight": 1.5, + }, + "line_item__days": { + "type": "list", + "comparator": "ExactComparator", + "threshold": 1.0, + "weight": 1.0, + }, + "line_item__rate": { + "type": "list", + "comparator": "ExactComparator", + "threshold": 1.0, + "weight": 2.0, + }, + "line_item__start_date": { + "type": "list", + "comparator": "ExactComparator", + "threshold": 1.0, + "weight": 2.0, + }, + "line_item__end_date": { + "type": "list", + "comparator": "ExactComparator", + "threshold": 1.0, + "weight": 2.0, + }, + }, + } + } + } + + return config + + +def main(): + """Run the FCC invoice evaluation test.""" + + print("=" * 80) + print("SticklerEvaluationService - FCC Invoice Data Test") + print("=" * 80) + + # Paths + csv_path = "sr_refactor_labels_5_5_25.csv" + data_dir = "tmp_data/cli-batch-20251017-154358" + + # Check if paths exist + if not os.path.exists(csv_path): + print(f"Error: CSV file not found: {csv_path}") + return + + if not os.path.exists(data_dir): + print(f"Error: Data directory not found: {data_dir}") + return + + # Create Stickler configuration + print("\n1. Creating Stickler configuration for FCC invoices...") + config = create_fcc_stickler_config() + print(" ✓ Configuration created") + + # Initialize service + print("\n2. Initializing SticklerEvaluationService...") + service = SticklerEvaluationService(config=config) + print(f" ✓ Service initialized with models: {list(service.stickler_models.keys())}") + + # Find a sample document to test + doc_dirs = [d for d in os.listdir(data_dir) if os.path.isdir(os.path.join(data_dir, d))] + + if not doc_dirs: + print("Error: No document directories found") + return + + # Use the first document + sample_doc = doc_dirs[0] + doc_path = os.path.join(data_dir, sample_doc) + + print(f"\n3. Testing with document: {sample_doc}") + + # Load the extraction result + result_path = os.path.join(doc_path, "sections/1/result.json") + + if not os.path.exists(result_path): + print(f"Error: Result file not found: {result_path}") + return + + with open(result_path, 'r') as f: + result_data = json.load(f) + + actual_results = result_data.get('inference_result', {}) + doc_class = result_data.get('document_class', {}).get('type', 'unknown') + + print(f" Document class: {doc_class}") + print(f" Extracted fields: {list(actual_results.keys())}") + + # Load ground truth from CSV + # Extract doc_id from filename (remove .pdf extension) + doc_id_from_filename = sample_doc.replace('.pdf', '') + + print(f"\n4. Loading ground truth for doc_id: {doc_id_from_filename}") + ground_truth = load_ground_truth_from_csv(csv_path, doc_id_from_filename) + + if ground_truth is None: + print(" Warning: No ground truth available, using actual results as expected") + print(" This will show perfect matches (for demonstration purposes)") + expected_results = actual_results + else: + expected_results = ground_truth + print(f" ✓ Ground truth loaded with {len(expected_results)} fields") + + # Create a section + section = Section( + section_id="section1", + classification="fcc-invoice", + page_ids=["page1"] + ) + + # Evaluate + print("\n5. Evaluating extraction results...") + try: + result = service.evaluate_section( + section=section, + expected_results=expected_results, + actual_results=actual_results + ) + + print(" ✓ Evaluation completed") + + # Display results + print("\n6. Evaluation Results") + print("-" * 80) + print(f"Section ID: {result.section_id}") + print(f"Document Class: {result.document_class}") + + if result.metrics: + print(f"\nMetrics:") + for metric_name, metric_value in result.metrics.items(): + print(f" {metric_name:25} {metric_value:.4f}") + + if result.attributes: + print(f"\nAttribute Results ({len(result.attributes)} attributes):") + print(f"{'Attribute':<30} {'Match':<8} {'Score':<8}") + print("-" * 50) + + matched_count = 0 + for attr in result.attributes[:20]: # Show first 20 + match_symbol = "✓" if attr.matched else "✗" + if attr.matched: + matched_count += 1 + print(f"{attr.name:<30} {match_symbol:<8} {attr.score:<8.3f}") + + if len(result.attributes) > 20: + print(f"... and {len(result.attributes) - 20} more attributes") + + print(f"\nSummary: {matched_count}/{len(result.attributes)} attributes matched") + else: + print("\nNo attributes evaluated (model may not be configured for this class)") + + except Exception as e: + print(f" ✗ Error during evaluation: {str(e)}") + import traceback + traceback.print_exc() + + print("\n" + "=" * 80) + print("Test completed!") + print("=" * 80) + + +if __name__ == "__main__": + main() diff --git a/lib/idp_common_pkg/idp_common/evaluation/README_STICKLER.md b/lib/idp_common_pkg/idp_common/evaluation/README_STICKLER.md new file mode 100644 index 000000000..abb5a4414 --- /dev/null +++ b/lib/idp_common_pkg/idp_common/evaluation/README_STICKLER.md @@ -0,0 +1,233 @@ +# SticklerEvaluationService + +The `SticklerEvaluationService` is an evaluation service that uses the [Stickler library](https://github.com/awslabs/stickler) for advanced validation rules and custom evaluation criteria. It provides the same interface as the standard `EvaluationService` but leverages Stickler's powerful comparison capabilities. + +## Features + +- **JSON Configuration**: Define evaluation models using JSON configuration +- **Advanced Comparators**: Use Stickler's built-in comparators (Exact, Fuzzy, Numeric, Levenshtein, etc.) +- **Nested Structure Support**: Handle complex nested data structures +- **List Matching**: Automatic Hungarian algorithm for optimal list matching +- **Flexible Validation**: Custom validation rules per document class + +## Installation + +The Stickler library must be installed to use this service: + +```bash +pip install stickler-eval +``` + +Or install from the local stickler directory: + +```bash +pip install -e ./stickler +``` + +## Configuration + +The service is configured using a JSON structure that defines Stickler models for each document class: + +```python +config = { + "stickler_models": { + "invoice": { + "model_name": "Invoice", + "match_threshold": 0.8, + "fields": { + "invoice_number": { + "type": "str", + "comparator": "ExactComparator", + "threshold": 1.0, + "weight": 3.0 + }, + "total_amount": { + "type": "float", + "comparator": "NumericComparator", + "comparator_config": {"tolerance": 0.01}, + "threshold": 0.95, + "weight": 2.0 + }, + "customer": { + "type": "structured_model", + "threshold": 0.8, + "weight": 2.0, + "fields": { + "name": { + "type": "str", + "comparator": "LevenshteinComparator", + "threshold": 0.8, + "weight": 1.0 + }, + "address": { + "type": "str", + "comparator": "FuzzyComparator", + "threshold": 0.7, + "weight": 0.8 + } + } + }, + "line_items": { + "type": "list_structured_model", + "weight": 2.0, + "match_threshold": 0.7, + "fields": { + "product": { + "type": "str", + "comparator": "FuzzyComparator", + "threshold": 0.8, + "weight": 1.0 + }, + "quantity": { + "type": "int", + "comparator": "NumericComparator", + "threshold": 1.0, + "weight": 0.8 + }, + "price": { + "type": "float", + "comparator": "NumericComparator", + "threshold": 0.95, + "weight": 1.2 + } + } + } + } + } + } +} +``` + +## Usage + +### Basic Usage + +```python +from idp_common.evaluation import SticklerEvaluationService +from idp_common.models import Section + +# Initialize the service with configuration +service = SticklerEvaluationService(region="us-east-1", config=config) + +# Create a section to evaluate +section = Section( + section_id="section1", + classification="invoice", + page_ids=["page1"] +) + +# Expected and actual extraction results +expected_results = { + "invoice_number": "INV-2024-001", + "total_amount": 1247.50, + "customer": { + "name": "Acme Corporation", + "address": "123 Business St, Suite 100" + }, + "line_items": [ + {"product": "Widget A", "quantity": 5, "price": 29.99}, + {"product": "Widget B", "quantity": 10, "price": 12.99} + ] +} + +actual_results = { + "invoice_number": "INV-2024-001", + "total_amount": 1247.48, + "customer": { + "name": "ACME Corp", + "address": "123 Business Street, Ste 100" + }, + "line_items": [ + {"product": "Widget B", "quantity": 10, "price": 12.99}, # Reordered + {"product": "Widget A", "quantity": 5, "price": 29.99} + ] +} + +# Evaluate the section +result = service.evaluate_section( + section=section, + expected_results=expected_results, + actual_results=actual_results +) + +# Access results +print(f"Section: {result.section_id}") +print(f"Metrics: {result.metrics}") +for attr in result.attributes: + print(f" {attr.name}: matched={attr.matched}, score={attr.score:.3f}") +``` + +### Document-Level Evaluation + +```python +from idp_common.models import Document, Section + +# Evaluate a complete document +document_result = service.evaluate_document( + document_id="doc123", + sections=[section1, section2], + expected_results_uri="s3://bucket/ground-truth/doc123.json", + actual_results_uri="s3://bucket/inference/doc123.json" +) + +# Access overall metrics +print(f"Overall Precision: {document_result.overall_metrics['precision']:.3f}") +print(f"Overall Recall: {document_result.overall_metrics['recall']:.3f}") +print(f"Overall F1 Score: {document_result.overall_metrics['f1_score']:.3f}") +``` + +## Available Comparators + +Stickler provides several built-in comparators: + +- **ExactComparator**: Exact string matching (case-sensitive) +- **LevenshteinComparator**: Edit distance-based string comparison +- **FuzzyComparator**: Fuzzy string matching using rapidfuzz +- **NumericComparator**: Numeric comparison with optional tolerance +- **SemanticComparator**: Semantic similarity (requires embeddings) + +## Field Types + +- **str**: String fields +- **int**: Integer fields +- **float**: Floating-point fields +- **bool**: Boolean fields +- **structured_model**: Nested object (group of fields) +- **list_structured_model**: List of objects (uses Hungarian matching) + +## Comparison with EvaluationService + +| Feature | EvaluationService | SticklerEvaluationService | +|---------|-------------------|---------------------------| +| Configuration | YAML-based | JSON-based (Stickler format) | +| Comparators | Built-in (EXACT, FUZZY, LLM, etc.) | Stickler comparators | +| List Matching | Hungarian with custom comparators | Stickler's Hungarian implementation | +| Nested Structures | Flattened with dot notation | Native nested support | +| Weights | Not supported | Field-level weights | +| Thresholds | Per-method thresholds | Per-field thresholds | + +## Benefits + +1. **Flexible Configuration**: Define validation rules in JSON without code changes +2. **Advanced Matching**: Leverage Stickler's sophisticated comparison algorithms +3. **Business-Weighted Scoring**: Assign importance weights to critical fields +4. **List Reordering**: Automatic optimal matching for lists regardless of order +5. **Nested Support**: Natural handling of complex nested structures + +## Example Configuration Files + +See the `config_library/pattern-2/fcc-invoices/` directory for example Stickler configurations. + +## Testing + +Run the unit tests: + +```bash +pytest lib/idp_common_pkg/tests/unit/evaluation/test_stickler_service.py -v +``` + +## References + +- [Stickler Documentation](https://github.com/awslabs/stickler) +- [Stickler Dynamic Model Creation](../../stickler/docs/StructuredModel_Dynamic_Creation.md) +- [IDP Evaluation Documentation](./README.md) diff --git a/lib/idp_common_pkg/idp_common/evaluation/__init__.py b/lib/idp_common_pkg/idp_common/evaluation/__init__.py index 9283bdb31..470f146cc 100644 --- a/lib/idp_common_pkg/idp_common/evaluation/__init__.py +++ b/lib/idp_common_pkg/idp_common/evaluation/__init__.py @@ -23,6 +23,7 @@ SectionEvaluationResult, ) from idp_common.evaluation.service import EvaluationService +from idp_common.evaluation.stickler_service import SticklerEvaluationService __all__ = [ "EvaluationMethod", @@ -31,6 +32,7 @@ "SectionEvaluationResult", "DocumentEvaluationResult", "EvaluationService", + "SticklerEvaluationService", "compare_values", "compare_exact", "compare_numeric", diff --git a/lib/idp_common_pkg/idp_common/evaluation/stickler_service.py b/lib/idp_common_pkg/idp_common/evaluation/stickler_service.py new file mode 100644 index 000000000..74408e83b --- /dev/null +++ b/lib/idp_common_pkg/idp_common/evaluation/stickler_service.py @@ -0,0 +1,441 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: MIT-0 + +""" +Stickler-based evaluation service for document extraction results. + +This module provides a service for evaluating extraction results using the +Stickler library for advanced validation rules and custom evaluation criteria. +""" + +import json +import logging +import time +from typing import Any, Dict, List, Optional, Tuple + +from idp_common import s3 +from idp_common.evaluation.metrics import calculate_metrics +from idp_common.evaluation.models import ( + AttributeEvaluationResult, + DocumentEvaluationResult, + SectionEvaluationResult, +) +from idp_common.models import Document, Section + +try: + from stickler.structured_object_evaluator.models.structured_model import ( + StructuredModel, + ) + + STICKLER_AVAILABLE = True +except ImportError: + STICKLER_AVAILABLE = False + StructuredModel = None + +logger = logging.getLogger(__name__) + + +class SticklerEvaluationService: + """Evaluation service using Stickler library for custom validation rules.""" + + def __init__(self, region: str = None, config: Dict[str, Any] = None): + """ + Initialize the stickler evaluation service. + + Args: + region: AWS region (for S3 access) + config: Configuration dictionary containing stickler evaluation settings + + Raises: + ImportError: If stickler library is not installed + """ + if not STICKLER_AVAILABLE: + raise ImportError( + "Stickler library is not installed. " + "Install it with: pip install stickler-eval" + ) + + self.config = config or {} + self.region = region + self.stickler_models = {} # Cache for dynamically created Stickler models + + # Load Stickler model configurations from config + self._load_stickler_models() + + logger.info("Initialized SticklerEvaluationService") + + def _load_stickler_models(self) -> None: + """Load Stickler model configurations from the config.""" + stickler_config = self.config.get("stickler_models", {}) + + for class_name, model_config in stickler_config.items(): + try: + # Create dynamic Stickler model from JSON configuration + model_class = StructuredModel.model_from_json(model_config) + self.stickler_models[class_name.lower()] = { + "model_class": model_class, + "config": model_config, + } + logger.info(f"Loaded Stickler model for class: {class_name}") + except Exception as e: + logger.error( + f"Error loading Stickler model for {class_name}: {str(e)}" + ) + + def _load_extraction_results(self, uri: str) -> Dict[str, Any]: + """ + Load extraction results from S3 or local file. + + Args: + uri: S3 URI or file:// URI to the extraction results + + Returns: + Dictionary of extraction results + """ + try: + # Handle local file paths (file:// URIs) + if uri.startswith("file://"): + file_path = uri.replace("file://", "") + with open(file_path, 'r') as f: + content = json.load(f) + else: + # Handle S3 URIs + content = s3.get_json_content(uri) + + # Check if results are wrapped in inference_result key + if isinstance(content, dict) and "inference_result" in content: + return content["inference_result"] + else: + return content + except Exception as e: + logger.error(f"Error loading extraction results from {uri}: {str(e)}") + return {} + + def _flatten_comparison_result( + self, comparison_result: Dict[str, Any], prefix: str = "" + ) -> List[Tuple[str, float, Any]]: + """ + Flatten nested comparison results from Stickler into attribute-level scores. + + Args: + comparison_result: Result from Stickler's compare_with method + prefix: Prefix for nested field names + + Returns: + List of tuples (field_name, score, details) + """ + flattened = [] + + # Extract field scores + field_scores = comparison_result.get("field_scores", {}) + for field_name, score in field_scores.items(): + full_name = f"{prefix}.{field_name}" if prefix else field_name + flattened.append((full_name, score, None)) + + # Extract nested scores + nested_scores = comparison_result.get("nested_scores", {}) + for field_name, nested_result in nested_scores.items(): + full_name = f"{prefix}.{field_name}" if prefix else field_name + if isinstance(nested_result, dict): + # Recursively flatten nested structures + nested_flattened = self._flatten_comparison_result( + nested_result, full_name + ) + flattened.extend(nested_flattened) + + # Extract list scores + list_scores = comparison_result.get("list_scores", {}) + for field_name, list_result in list_scores.items(): + full_name = f"{prefix}.{field_name}" if prefix else field_name + if isinstance(list_result, dict): + # For lists, use the overall list score + list_score = list_result.get("overall_score", 0.0) + flattened.append((full_name, list_score, list_result)) + + return flattened + + def _create_attribute_result( + self, + attr_name: str, + expected_value: Any, + actual_value: Any, + score: float, + details: Any = None, + ) -> AttributeEvaluationResult: + """ + Create an AttributeEvaluationResult from Stickler comparison. + + Args: + attr_name: Attribute name + expected_value: Expected value + actual_value: Actual value + score: Similarity score from Stickler (0.0 to 1.0) + details: Additional details from Stickler comparison + + Returns: + AttributeEvaluationResult + """ + # Determine if values match based on score + # Use a threshold of 0.8 for matching (can be made configurable) + matched = score >= 0.8 + + # Generate reason based on score + if score == 1.0: + reason = "Exact match according to Stickler validation" + elif score >= 0.8: + reason = f"Values match with similarity score {score:.3f}" + else: + reason = f"Values do not match (similarity score: {score:.3f})" + + # Add list matching details if available + if details and isinstance(details, dict): + matched_pairs = details.get("matched_pairs", []) + unmatched_expected = details.get("unmatched_expected", []) + unmatched_actual = details.get("unmatched_actual", []) + + if matched_pairs or unmatched_expected or unmatched_actual: + reason += f" | Matched: {len(matched_pairs)}, " + reason += f"Missing: {len(unmatched_expected)}, " + reason += f"Extra: {len(unmatched_actual)}" + + return AttributeEvaluationResult( + name=attr_name, + expected=expected_value, + actual=actual_value, + matched=matched, + score=score, + reason=reason, + evaluation_method="STICKLER", + ) + + def evaluate_section( + self, + section: Section, + expected_results: Dict[str, Any], + actual_results: Dict[str, Any], + ) -> SectionEvaluationResult: + """ + Evaluate extraction results for a document section using Stickler. + + Args: + section: Document section + expected_results: Expected extraction results + actual_results: Actual extraction results + + Returns: + Evaluation results for the section + """ + class_name = section.classification.lower() + + logger.debug( + f"Evaluating Section {section.section_id} - class: {class_name}" + ) + + # Check if we have a Stickler model for this class + if class_name not in self.stickler_models: + logger.warning( + f"No Stickler model configured for class: {class_name}. " + f"Available models: {list(self.stickler_models.keys())}" + ) + # Return empty result if no model configured + return SectionEvaluationResult( + section_id=section.section_id, + document_class=section.classification, + attributes=[], + metrics={}, + ) + + model_info = self.stickler_models[class_name] + model_class = model_info["model_class"] + + try: + # Create Stickler model instances from the data + expected_instance = model_class(**expected_results) + actual_instance = model_class(**actual_results) + + # Perform comparison using Stickler + comparison_result = expected_instance.compare_with(actual_instance) + + # Flatten the comparison results to attribute-level scores + flattened_results = self._flatten_comparison_result(comparison_result) + + # Create AttributeEvaluationResult for each field + attribute_results = [] + for attr_name, score, details in flattened_results: + # Get the actual values for this attribute + expected_value = self._get_nested_value(expected_results, attr_name) + actual_value = self._get_nested_value(actual_results, attr_name) + + attr_result = self._create_attribute_result( + attr_name, expected_value, actual_value, score, details + ) + attribute_results.append(attr_result) + + # Calculate metrics from attribute results + metrics = self._calculate_section_metrics(attribute_results) + + return SectionEvaluationResult( + section_id=section.section_id, + document_class=section.classification, + attributes=attribute_results, + metrics=metrics, + ) + + except Exception as e: + logger.error( + f"Error evaluating section {section.section_id} with Stickler: {str(e)}" + ) + # Return empty result on error + return SectionEvaluationResult( + section_id=section.section_id, + document_class=section.classification, + attributes=[], + metrics={}, + ) + + def _get_nested_value(self, data: Dict[str, Any], path: str) -> Any: + """ + Get a nested value from a dictionary using dot notation. + + Args: + data: Dictionary to extract value from + path: Dot-separated path (e.g., "customer.name") + + Returns: + Value at the path, or None if not found + """ + keys = path.split(".") + value = data + + for key in keys: + if isinstance(value, dict): + value = value.get(key) + if value is None: + return None + else: + return None + + return value + + def _calculate_section_metrics( + self, attribute_results: List[AttributeEvaluationResult] + ) -> Dict[str, float]: + """ + Calculate metrics for a section from attribute results. + + Args: + attribute_results: List of attribute evaluation results + + Returns: + Dictionary of metrics + """ + if not attribute_results: + return {} + + # Count true/false positives/negatives + tp = fp = fn = tn = fp1 = fp2 = 0 + + for attr_result in attribute_results: + expected = attr_result.expected + actual = attr_result.actual + matched = attr_result.matched + + # Case 1: Expected value is None/empty + if expected is None or (isinstance(expected, str) and not expected.strip()): + if actual is None or (isinstance(actual, str) and not actual.strip()): + tn += 1 # Correctly didn't predict a value + else: + fp += 1 # Incorrectly predicted a value + fp1 += 1 + + # Case 2: Expected value exists but actual doesn't + elif actual is None or (isinstance(actual, str) and not actual.strip()): + fn += 1 # Missing prediction + + # Case 3: Both values exist + else: + if matched: + tp += 1 # Correct prediction + else: + fp += 1 # Incorrect prediction + fp2 += 1 + + # Calculate metrics using the common metrics function + metrics = calculate_metrics(tn, fp, fn, tp, fp1, fp2) + + return metrics + + def evaluate_document( + self, + document_id: str, + sections: List[Section], + expected_results_uri: str, + actual_results_uri: str, + ) -> DocumentEvaluationResult: + """ + Evaluate a document using Stickler validation rules. + + Args: + document_id: Document identifier + sections: List of document sections + expected_results_uri: S3 URI to expected results (ground truth) + actual_results_uri: S3 URI to actual extraction results + + Returns: + Document evaluation result with Stickler validation + """ + start_time = time.time() + + # Load expected and actual results + expected_results = self._load_extraction_results(expected_results_uri) + actual_results = self._load_extraction_results(actual_results_uri) + + section_results = [] + + for section in sections: + section_result = self.evaluate_section( + section=section, + expected_results=expected_results, + actual_results=actual_results, + ) + section_results.append(section_result) + + # Calculate overall metrics from section results + overall_metrics = self._calculate_overall_metrics(section_results) + + execution_time = time.time() - start_time + + return DocumentEvaluationResult( + document_id=document_id, + section_results=section_results, + overall_metrics=overall_metrics, + execution_time=execution_time, + ) + + def _calculate_overall_metrics( + self, section_results: List[SectionEvaluationResult] + ) -> Dict[str, float]: + """ + Calculate overall metrics from section results. + + Args: + section_results: List of section evaluation results + + Returns: + Dictionary of overall metrics + """ + if not section_results: + return {} + + # Aggregate attribute results across all sections to recalculate metrics + all_attributes = [] + for section_result in section_results: + all_attributes.extend(section_result.attributes) + + # Recalculate metrics from all attributes + if all_attributes: + overall_metrics = self._calculate_section_metrics(all_attributes) + else: + overall_metrics = {} + + return overall_metrics diff --git a/lib/idp_common_pkg/tests/unit/evaluation/test_stickler_service.py b/lib/idp_common_pkg/tests/unit/evaluation/test_stickler_service.py new file mode 100644 index 000000000..8b917467a --- /dev/null +++ b/lib/idp_common_pkg/tests/unit/evaluation/test_stickler_service.py @@ -0,0 +1,257 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: MIT-0 + +""" +Unit tests for SticklerEvaluationService. +""" + +import pytest +from unittest.mock import MagicMock, patch + +from idp_common.evaluation.stickler_service import ( + SticklerEvaluationService, + STICKLER_AVAILABLE, +) +from idp_common.evaluation.models import ( + AttributeEvaluationResult, + SectionEvaluationResult, +) +from idp_common.models import Section + + +@pytest.mark.skipif(not STICKLER_AVAILABLE, reason="Stickler library not installed") +class TestSticklerEvaluationService: + """Test cases for SticklerEvaluationService.""" + + def test_initialization(self): + """Test service initialization.""" + config = { + "stickler_models": { + "invoice": { + "model_name": "Invoice", + "fields": { + "invoice_number": { + "type": "str", + "comparator": "ExactComparator", + }, + "total": { + "type": "float", + "comparator": "NumericComparator", + }, + }, + } + } + } + + service = SticklerEvaluationService(config=config) + assert service is not None + assert "invoice" in service.stickler_models + + def test_flatten_comparison_result(self): + """Test flattening of nested comparison results.""" + config = {} + service = SticklerEvaluationService(config=config) + + comparison_result = { + "overall_score": 0.95, + "field_scores": {"name": 1.0, "age": 0.9}, + "nested_scores": { + "address": { + "overall_score": 0.85, + "field_scores": {"street": 0.8, "city": 0.9}, + } + }, + } + + flattened = service._flatten_comparison_result(comparison_result) + + # Should have 4 entries: name, age, address.street, address.city + assert len(flattened) >= 2 + field_names = [f[0] for f in flattened] + assert "name" in field_names + assert "age" in field_names + + def test_get_nested_value(self): + """Test getting nested values from dictionary.""" + config = {} + service = SticklerEvaluationService(config=config) + + data = {"customer": {"name": "John Doe", "address": {"city": "New York"}}} + + # Test simple nested access + assert service._get_nested_value(data, "customer.name") == "John Doe" + + # Test deeper nesting + assert service._get_nested_value(data, "customer.address.city") == "New York" + + # Test non-existent path + assert service._get_nested_value(data, "customer.phone") is None + + def test_create_attribute_result(self): + """Test creation of AttributeEvaluationResult.""" + config = {} + service = SticklerEvaluationService(config=config) + + # Test exact match + result = service._create_attribute_result( + "invoice_number", "INV-001", "INV-001", 1.0 + ) + + assert isinstance(result, AttributeEvaluationResult) + assert result.name == "invoice_number" + assert result.matched is True + assert result.score == 1.0 + assert result.evaluation_method == "STICKLER" + + # Test partial match + result = service._create_attribute_result( + "customer_name", "John Smith", "Jon Smith", 0.85 + ) + + assert result.matched is True # Above 0.8 threshold + assert result.score == 0.85 + + # Test no match + result = service._create_attribute_result( + "total", "100.00", "200.00", 0.5 + ) + + assert result.matched is False # Below 0.8 threshold + assert result.score == 0.5 + + def test_calculate_section_metrics(self): + """Test calculation of section metrics.""" + config = {} + service = SticklerEvaluationService(config=config) + + # Create sample attribute results + attribute_results = [ + AttributeEvaluationResult( + name="field1", + expected="value1", + actual="value1", + matched=True, + score=1.0, + evaluation_method="STICKLER", + ), + AttributeEvaluationResult( + name="field2", + expected="value2", + actual="value2_different", + matched=False, + score=0.5, + evaluation_method="STICKLER", + ), + AttributeEvaluationResult( + name="field3", + expected=None, + actual=None, + matched=True, + score=1.0, + evaluation_method="STICKLER", + ), + ] + + metrics = service._calculate_section_metrics(attribute_results) + + # Check that metrics are returned (calculate_metrics returns precision, recall, etc.) + assert "precision" in metrics + assert "recall" in metrics + assert "f1_score" in metrics + assert "accuracy" in metrics + assert "false_alarm_rate" in metrics + assert "false_discovery_rate" in metrics + + # Verify metrics are reasonable + assert 0.0 <= metrics["precision"] <= 1.0 + assert 0.0 <= metrics["recall"] <= 1.0 + assert 0.0 <= metrics["f1_score"] <= 1.0 + + @patch("idp_common.evaluation.stickler_service.s3") + def test_evaluate_section_no_model(self, mock_s3): + """Test section evaluation when no Stickler model is configured.""" + config = {} + service = SticklerEvaluationService(config=config) + + section = Section( + section_id="section1", + classification="unknown_class", + page_ids=["page1"], + ) + + result = service.evaluate_section( + section=section, expected_results={}, actual_results={} + ) + + assert isinstance(result, SectionEvaluationResult) + assert result.section_id == "section1" + assert len(result.attributes) == 0 + + def test_calculate_overall_metrics(self): + """Test calculation of overall metrics from section results.""" + config = {} + service = SticklerEvaluationService(config=config) + + # Create sample section results with attributes + section_results = [ + SectionEvaluationResult( + section_id="section1", + document_class="invoice", + attributes=[ + AttributeEvaluationResult( + name="field1", + expected="value1", + actual="value1", + matched=True, + score=1.0, + evaluation_method="STICKLER", + ), + AttributeEvaluationResult( + name="field2", + expected="value2", + actual="value2_diff", + matched=False, + score=0.5, + evaluation_method="STICKLER", + ), + ], + metrics={}, + ), + SectionEvaluationResult( + section_id="section2", + document_class="invoice", + attributes=[ + AttributeEvaluationResult( + name="field3", + expected="value3", + actual="value3", + matched=True, + score=1.0, + evaluation_method="STICKLER", + ), + ], + metrics={}, + ), + ] + + overall_metrics = service._calculate_overall_metrics(section_results) + + # Check that metrics are returned + assert "precision" in overall_metrics + assert "recall" in overall_metrics + assert "f1_score" in overall_metrics + assert "accuracy" in overall_metrics + + # Verify metrics are reasonable + assert 0.0 <= overall_metrics["precision"] <= 1.0 + assert 0.0 <= overall_metrics["recall"] <= 1.0 + assert 0.0 <= overall_metrics["f1_score"] <= 1.0 + + +def test_stickler_not_available(): + """Test that appropriate error is raised when Stickler is not available.""" + with patch( + "idp_common.evaluation.stickler_service.STICKLER_AVAILABLE", False + ): + with pytest.raises(ImportError, match="Stickler library is not installed"): + SticklerEvaluationService()