​
Introduction

In the rapidly evolving landscape of AI and Large Language Models (LLMs), a new challenge has emerged for brands and content creators: ensuring their web content is optimized for machine consumption. Traditional SEO focused on search engines; now we need Generative Engine Optimization (GEO) to ensure LLMs can accurately understand and represent our content.

GEO Evaluator is a sophisticated Python-based tool that analyzes websites through the lens of an LLM, evaluating how well content is structured and presented for machine understanding. This post explores the technical architecture and innovative approaches that make this analysis possible.

​
The Core Problem

LLMs process web content differently than traditional search engines. While search engines primarily index keywords and backlinks, LLMs attempt to understand context, relationships, and meaning. This fundamental difference requires a new approach to content optimization:

  1. Semantic Structure: LLMs benefit from clear HTML semantics that indicate content hierarchy and relationships
  2. Content Efficiency: Token limits make content-to-markup ratios critical
  3. Machine Readability: Structured data and standardized formats (like llms.txt) provide explicit signals
  4. Accessibility: Alt text and clear language benefit both humans and machines

​
Technical Architecture

​
The Pipeline Pattern

GEO Evaluator implements a sophisticated pipeline architecture that separates concerns while maintaining flexibility:

class GEOAnalysisPipeline:
    def __init__(self, config: Dict[str, Any]):
        self.crawler = WebCrawler(config)
        self.scoring_engine = ScoringEngine(config)
        self.analyzers = {}  # Pluggable analyzer system

This design allows for:

  • Modular Analysis: Each aspect of GEO optimization is handled by a specialized analyzer
  • Configurable Weights: Different use cases can prioritize different optimization aspects
  • Extensibility: New analyzers can be added without modifying the core pipeline

​
Intelligent Web Crawling

The crawler component (src/crawler.py) goes beyond simple page fetching. It implements several sophisticated features:

​
1. Respectful Crawling

def _should_crawl_url(self, url: str) -> bool:
    # Check robots.txt compliance
    if self.robots_parser:
        if not self.robots_parser.can_fetch(self.user_agent, url):
            return False
    
    # Check excluded paths and file types
    # Implements intelligent filtering

The crawler respects website policies while efficiently discovering content through multiple methods:

  • Sitemap parsing (XML and text formats)
  • Robots.txt compliance
  • Configurable delays and timeouts
  • Smart retry logic with exponential backoff

​
2. Content Extraction

The extraction process uses BeautifulSoup to parse HTML but goes beyond simple text extraction:

def _extract_page_data(self, url: str, html_content: str, depth: int) -> Dict[str, Any]:
    # Prioritize semantic content areas
    for selector in ['main', 'article', '.content', '#content']:
        main_elem = soup.select_one(selector)
        if main_elem:
            main_content = main_elem.get_text()
            break

This approach recognizes that not all content is equal - main content areas are prioritized over navigation and sidebars.

​
The Scoring Engine

The heart of GEO Evaluator is its weighted scoring system (src/scoring_engine.py). This implements a sophisticated multi-factor analysis:

​
1. Category-Based Analysis

Each category represents a different aspect of LLM optimization:

self.weights = {
    'structural_html': 0.25,      # Semantic markup quality
    'content_organization': 0.30,  # Readability and structure
    'token_efficiency': 0.20,      # Content-to-markup ratio
    'llm_technical': 0.15,         # LLM-specific features
    'accessibility': 0.10          # Alt text and clarity
}

​
2. Intelligent Scoring Algorithms

The scoring system uses threshold-based calculations that reflect real-world optimization targets:

def _calculate_percentage_score(self, value: float, excellent_threshold: float, 
                              good_threshold: float) -> float:
    if value >= excellent_threshold:
        # Non-linear scaling for excellence
        normalized = (value - excellent_threshold) / (1.0 - excellent_threshold)
        return 0.8 + (normalized * 0.2)

This non-linear scoring ensures that:

  • Small improvements at high optimization levels are rewarded
  • There’s always room for improvement
  • Scores align with human interpretation of quality

​
Placeholder Analysis System

Currently, the tool implements a clever placeholder analysis system that provides meaningful results even before specialized analyzers are built:

​
1. Structural HTML Analysis

# Check for semantic elements
semantic_elements = ['<main', '<article', '<section', '<header', '<footer', '<nav']
if any(elem in html.lower() for elem in semantic_elements):
    semantic_pages += 1

This simple but effective approach:

  • Detects presence of semantic HTML5 elements
  • Calculates ratios across all pages
  • Generates actionable recommendations

​
2. LLM Technical Features Detection

One particularly interesting implementation is the llms.txt detection:

# Check if llms.txt exists at the root
llms_txt_url = base_url.rstrip('/') + '/llms.txt'
try:
    response = requests.head(llms_txt_url, timeout=5)
    has_llms_txt = response.status_code == 200
except Exception:
    has_llms_txt = False

This showcases the tool’s focus on LLM-specific optimizations that traditional SEO tools miss.

​
Key Technical Innovations

​
1. Affected Pages Tracking

Unlike many analysis tools that provide generic recommendations, GEO Evaluator tracks exactly which pages are affected by each issue:

recommendations.append({
    'title': 'Expand Short Content',
    'description': f'{short_content_pages} pages have very short content (<100 words)',
    'priority': 'medium',
    'affected_pages': short_content_urls  # Actual page list
})

​
2. Multi-Format Output Strategy

The tool generates multiple output formats simultaneously:

  • JSON: Complete machine-readable results
  • HTML: Human-friendly reports
  • Dashboard Data: Specialized format for visualization

This is achieved through a clean separation of data generation and formatting:

def _generate_outputs(self) -> Dict[str, Any]:
    results = {
        'metadata': self._generate_metadata(),
        'overall_score': self._format_overall_score(),
        'analysis_summary': self._generate_analysis_summary(),
        'recommendations': self.final_scores.recommendations,
        'page_scores': self._generate_page_scores()
    }

​
3. Benchmark Integration

The scoring engine includes sophisticated benchmarking that places results in context:

def get_benchmarks(self, overall_score: OverallScore) -> Dict[str, Any]:
    # Calculate percentile based on industry estimates
    if score >= industry_benchmarks['leader_threshold']:
        percentile = 99
    elif score >= industry_benchmarks['top_decile']:
        percentile = 90 + interpolation_calculation

This provides users with context about their optimization level relative to industry standards.

​
Performance Optimizations

​
1. Streaming Content Size Checks

To avoid downloading massive files, the crawler implements streaming downloads with size checks:

response = self.session.get(url, stream=True)
content = b''
for chunk in response.iter_content(chunk_size=8192):
    content += chunk
    if len(content) > self.max_file_size:
        return None  # Abort large downloads

​
2. Efficient URL Discovery

The sitemap parser implements recursive parsing for sitemap index files while avoiding infinite loops:

# Handle sitemap index files
for sitemap in root.findall('.//{...}sitemap'):
    sub_urls = self._fetch_and_parse_sitemap(loc_elem.text)
    urls.extend(sub_urls)

​
3. Memory-Efficient HTML Storage

While the tool stores raw HTML for detailed analysis, it implements size limits and efficient storage patterns to manage memory usage.

​
Future Architecture Considerations

The current implementation provides a solid foundation for future enhancements:

​
1. Pluggable Analyzer System

The architecture supports adding specialized analyzers:

# Future analyzer interface
class BaseAnalyzer(ABC):
    @abstractmethod
    def analyze(self, pages: List[Dict[str, Any]]) -> AnalysisResult:
        pass

​
2. Machine Learning Integration

The scoring engine’s design allows for ML model integration:

  • Feature extraction from current analysis
  • Training on human-validated scores
  • Adaptive threshold adjustment

​
3. Real-time Monitoring

The pipeline architecture supports streaming analysis:

  • WebSocket connections for live updates
  • Incremental scoring updates
  • Change detection and alerts

​
Conclusion

GEO Evaluator represents a sophisticated approach to a new problem in web optimization. By combining intelligent crawling, multi-factor analysis, and actionable recommendations, it provides a technical foundation for ensuring content is optimized for the AI-powered future.

The tool’s architecture demonstrates several best practices:

  • Separation of Concerns: Clear boundaries between crawling, analysis, and scoring
  • Configurability: Extensive configuration options without code changes
  • Extensibility: Easy to add new analysis categories or output formats
  • Performance: Efficient handling of large websites
  • User Focus: Actionable recommendations with specific page references

As LLMs become increasingly important for brand representation and content discovery, tools like GEO Evaluator will be essential for maintaining visibility and accuracy in AI-generated responses. The technical patterns established here provide a blueprint for the next generation of web optimization tools.