Marketplace Research Suite

🕷️ scrape.py - Marketplace Data Scraper

Extract comprehensive service data from the UK Digital Marketplace with live categories and keyword extraction.

• Live marketplace categories (18 official categories via marketplace-data-extractor)
• Keyword extraction with Light English stemmer (88.9% Elasticsearch alignment)
• Automatic retry logic for failed URLs with exponential backoff
• Resume capability with discovery URL caching
• Multiple output formats (JSON + CSV from single run)
• Parallel processing with configurable worker count

# Quick test extraction
python tools/scrape.py --max-services 100

# Full marketplace with CSV export
python tools/scrape.py --max-workers 16 --csv-output marketplace_data.csv

# Resume interrupted scraping
python tools/scrape.py --resume --output data/run_20241212_143052/services.json

🎯 competitors.py - FAISS-Accelerated Competitor Analysis

Enterprise-grade competitor analysis with pure FAISS cosine similarity search and comprehensive intelligence.

• FAISS-Accelerated Search: 10-100x faster than linear search
• Dataset Coverage: Auto-detects latest data (44K+ services)
• Analysis Speed: ~1-2 seconds analysis (after index build)
• Live Category Extraction: Real marketplace categories
• Synonym Expansion: BI ↔ business intelligence, CRM ↔ customer relationship management
• Possessive Handling: Normalizes "company's" → "company"

# Standard FAISS analysis (TF-IDF cosine similarity)
python tools/competitors.py --url "https://www.applytosupply.digitalmarketplace.service.gov.uk/g-cloud/services/123456"

# High-quality semantic embeddings analysis (slower but better similarity)
python tools/competitors.py --service-id 123456 --use-embeddings --max-competitors 20

# Force rebuild FAISS index for fresh vectorization
python tools/competitors.py --service-id 123456 --rebuild-index

# JSON-only analysis (faster for batch processing)
python tools/competitors.py --service-id 123456 --no-report

📊 generate_report.py - Report Generator

Generate reports from competitor analysis JSON data with strategic recommendations.

• Fast report generation from cached analysis data
• Batch processing for multiple analyses
• Template-based generation with consistent formatting
• Independent operation - no re-analysis needed

# Generate report from analysis JSON
python tools/generate_report.py --json outputs/123456/competitor_analysis_20250616_094736.json

# Batch generate reports for multiple analyses
python tools/generate_report.py --batch "outputs/*/competitor_*.json"

🚀 enhanced_competitors.py - Batch Competitor Analysis

Advanced competitor analysis with batch processing capabilities for large-scale analysis.

• Batch processing: Analyze hundreds of services from discovered URLs file
• Parallel execution: Concurrent processing with configurable workers
• Organized outputs: Individual directories per service ID
• Comprehensive reporting: Batch summary with success rates and performance metrics
• Error resilience: Continues processing if individual services fail
• URL parsing: Automatically extracts service IDs from marketplace URLs

# Batch processing from discovered URLs file (44K+ services)
python tools/enhanced_competitors.py --batch-file data/run_20250612_115548/discovered_urls.txt --batch-limit 100

# Batch processing from comma-separated IDs
python tools/enhanced_competitors.py --batch-ids "123456,789012,345678" --max-competitors 15

# Custom output directory with no markdown reports (faster)
python tools/enhanced_competitors.py --batch-file urls.txt --batch-output-dir outputs/my_batch --no-report

🌐 competitor_graph.py - Supplier Network Visualization

Generate interactive supplier competitive network graphs with full-page visualization.

• Supplier-level aggregation: Service-level similarities rolled up to supplier relationships
• Interactive HTML: Full-page, zoomable, draggable network visualization
• Filtering options: Edge weight/count thresholds, node degree filtering
• Physics control: Static or dynamic layouts with optimized stability
• Guaranteed inclusion: Force specific suppliers into filtered visualizations
• TF-IDF similarity: Cosine similarity based on service descriptions and metadata

# Interactive HTML visualization with filtering
python tools/competitor_graph.py --dataset data.json --html-output network.html \
    --min-weight 2.0 --min-count 3 --max-html-nodes 200

# Static layout with specific suppliers included
python tools/competitor_graph.py --dataset data.json --html-output network.html \
    --static --include "advice cloud" "ibm" --max-html-nodes 100

📈 trigram_analyzer.py - Tri-gram Analysis (Recommended)

Analyze 3-word phrase patterns across marketplace services with enhanced performance and quality.

• Enhanced performance: 2.6x faster + 9.1% better quality vs bi-grams
• Comprehensive extraction from service descriptions
• Lot-based grouping (cloud-software, cloud-support, cloud-hosting)
• Statistical analysis with frequency counts and percentages
• Multiple output formats (JSON summary, Markdown report, optional CSV)
• Domain-specific terminology analysis by government lot categories

# Analyze top 100 tri-grams per lot type (2.6x faster + 9.1% better quality)
python tools/trigram_analyzer.py --top-n 100

# Include CSV exports for each lot/category group
python tools/trigram_analyzer.py --top-n 100 --csv

# Use specific dataset
python tools/trigram_analyzer.py --dataset data/latest.json --output results

📊 bigram_analyzer.py - Bi-gram Analysis (Classic)

Analyze 2-word phrase patterns for comparison with modular preprocessing.

• Classic bi-gram analysis with modular preprocessing
• Same lot-based grouping and statistical analysis as tri-gram analyzer
• Direct comparison capability with tri-gram analysis results
• Multiple output formats (JSON summary, Markdown report, optional CSV)
• Domain-aware text preprocessing for consistent quality

# Classic bi-gram analysis with modular preprocessing
python tools/bigram_analyzer.py --top-n 100 --csv

🔑 exclusive_keyword_finder.py - Exclusive Keyword Finder

Identifies the smallest combination of keywords that are unique to a specific service, allowing you to find it exclusively in the marketplace search.

• Inverted Index Search - Builds a comprehensive map of where every keyword appears across all 44,000+ services.
• Combinatorial Analysis - Finds the smallest set of keywords that, when searched together, return only the target service.
• Search-Optimized - Processes text in a way that mirrors the marketplace search, ensuring the found keywords will work as expected.
• Actionable Output - Provides a list of keyword sets ready to be used in the Digital Marketplace search bar.

# Find exclusive keywords for a specific service
python tools/exclusive_keyword_finder.py --service-id "https://www.applytosupply.digitalmarketplace.service.gov.uk/g-cloud/services/926923877591843"

# Increase the search depth for harder-to-find services
python tools/exclusive_keyword_finder.py --service-id 123456 --max-combo-size 7

🔠 acronym_analyzer.py - UK Government Acronyms

Identify and expand UK government acronyms in marketplace data or custom text.

• 4,400+ official acronyms from the Acronym Buster dataset
• Confidence scoring with department filtering
• Search and expand acronyms in text
• Dataset-wide analysis with optional markdown reports

# Analyze entire dataset
python tools/acronym_analyzer.py

# Analyze specific text
python tools/acronym_analyzer.py --text "HMRC and DfE work with BEIS"

🚀 FAISS-Accelerated Search

Vector similarity search with approximate nearest neighbor algorithms for 10-100x performance improvements over linear search.

🏷️ Live Category Extraction

Real marketplace categories from Digital Marketplace APIs, not algorithmic inference, with lot-aware processing.

🔤 Domain-Aware Text Processing

Advanced text processing with synonym expansion (BI ↔ business intelligence), possessive handling, and marketplace terminology preservation.

🎯 FAISS Cosine Similarity

Pure FAISS cosine similarity search using TF-IDF or semantic embeddings with approximate nearest neighbor algorithms for optimal performance.

📊 Research-Grade Accuracy

88.9% alignment with actual Digital Marketplace search algorithms through comprehensive testing and validation.

🛡️ Enterprise Reliability

Robust error handling, automatic retry logic, resume capability, and graceful degradation for production use.