Skip to main content

​
Overview

Intent Crawler is a Python tool that crawls websites and interprets and analyzes the intents conveyed to LLMs and AI agents on your exisiting website. Part of the Airbais AI Tools Suite, Intent Crawler extracts content, discovers intents dynamically using multiple ML techniques, and provides results and recommendations on an interactive dashboard. This let’s you review what your site is saying to AI and identify what needs adjusting.
Intent Crawler Dashboard

​
Key Features

Intelligent Web Crawling

Respectful crawling with robots.txt compliance, automatic sitemap discovery, and configurable rate limiting

Advanced Intent Discovery

User-focused analysis (default), plus LDA topic modeling, sentence embeddings, and clustering

Modern Dashboard

A web dashboard with all the results from your evaluation. Suppors light/dark mode and responsive layout

Structured Exports

Outputs in llmstxt format and JSON for in depth LLM evaluation

​
Getting Started

​
Installation

1

Clone the repository

git clone https://github.com/Airbais/intent-tools.git
cd intentcrawler
2

Install dependencies

pip install -r requirements.txt
python -m spacy download en_core_web_sm
3

Run your first analysis

python intentcrawler.py https://example.com --dashboard

​
Quick Examples

  • Basic Usage
  • With Dashboard
  • View Past Results
  • List All Results
# Analyze a website
python intentcrawler.py https://example.com

​
Configuration

Customize the tool’s behavior through config.yaml:
Control how the tool crawls websites:
crawler:
  max_pages: 1000          # Maximum pages to crawl
  rate_limit: 2            # Seconds between requests
  respect_robots: true     # Follow robots.txt
  use_sitemap: true        # Auto-discover sitemaps
Configure the ML-powered intent discovery:
intents:
  extraction_method: 'user_intent' # Default: user-focused analysis
  # Alternative: 'dynamic', 'enhanced', 'original'
  use_embeddings: true             # Sentence transformers
  use_lda: true                    # Topic modeling
  lda_topics: 10                   # Number of topics
  embeddings_model: 'sentence-transformers/all-MiniLM-L6-v2'
  fallback_keywords: true          # Keyword fallback
  similarity_threshold: 0.7        # Intent merging
  custom_keywords:                 # Domain-specific
    product_discovery: ['products', 'catalog', 'browse']
    support: ['help', 'support', 'faq']
    integration: ['api', 'developer', 'sdk']
Configure how results are stored:
output:
  base_directory: 'results'
  date_format: '%Y-%m-%d'
  keep_past_results: 7      # Days to keep (-1 for all)
  overwrite_today: true

​
How It Works

​
Intent Discovery Process

1

Content Extraction

Crawls website pages and extracts clean, structured content
2

Text Preprocessing

Removes noise, normalizes text, and prepares for analysis
3

Feature Extraction

  • TF-IDF: Identifies important keywords
  • Embeddings: Captures semantic meaning
  • N-grams: Detects meaningful phrases
4

Intent Clustering

  • LDA: Discovers latent topics
  • DBSCAN: Groups semantically similar content
  • Keywords: Matches known patterns
5

Intent Merging

Combines similar intents based on configurable similarity threshold
6

Naming & Scoring

Automatically generates descriptive intent names and confidence scores

​
ML Techniques Explained

LDA Topic Modeling

Discovers latent topics across all content with configurable topic counts

Embedding Clustering

Uses sentence transformers and DBSCAN for semantic understanding

Keyword Fallback

Configurable keywords ensure baseline intent detection

​
Output Structure

Results are organized by date for easy historical tracking: 
results/
├── 2024-06-26/              # Today's results
│   ├── llmstxt/
│   │   ├── llms.txt         # Main llmstxt file
│   │   └── pages/           # Individual page summaries
│   ├── intent-report.json   # Detailed intent analysis
│   ├── dashboard-data.json  # Dashboard visualization data
│   ├── intent-summary.md    # Human-readable summary
│   └── llm-export.json      # Structured for LLM tools
└── 2024-06-25/              # Yesterday's results
    └── ...

​
Example Output

{
  "discovered_intents": [
    {
      "primary_intent": "learn_integration",
      "confidence": 0.85,
      "keywords": ["api", "integration", "connect"],
      "representative_phrases": [
        "integrate with your application",
        "api documentation and guides"
      ],
      "page_count": 23,
      "extraction_method": "lda"
    }
  ]
}

​
Dashboard Features

Two dashboard options: Local tool-specific dashboard and Master multi-tool dashboard

​
Modern Design System

Airbais Design

Professional orange/gray color scheme with Inter font family

Light/Dark Mode

Toggle themes with persistent user preferences

Responsive Layout

Works perfectly on desktop and mobile devices

Fast Performance

Optimized loading and smooth interactions
  • Overview Stats
  • Interactive Charts
  • Dashboard Options
  • Export Features
  • Total Pages: Number of pages analyzed
  • Discovered Intents: Count of unique user intents
  • Site Sections: Structural breakdown of the website
  • Confidence Indicators: Visual quality scores

​
Command Line Reference

​
url
string
required
The website URL to analyze
​
--config
string
Path to custom configuration file
​
--output
string
Override default output directory
​
--log-level
string
Set logging level: DEBUG, INFO, WARNING, ERROR
​
--dashboard
flag
Launch dashboard after analysis completes
​
--dashboard-only
flag
View existing results without running analysis
​
--dashboard-date
string
View results from specific date (YYYY-MM-DD)
​
--list-results
flag
List all available result dates

​
Performance Guidelines

Processing time increases with site size and enabled ML features
All features work well with default settings
Consider reducing LDA topics for faster processing
May need to disable embeddings or increase rate limiting

​
Troubleshooting

  • Check minimum cluster size in config
  • Ensure content has sufficient text
  • Try enabling fallback keywords
  • Reduce number of LDA topics
  • Disable embedding extraction
  • Increase rate limit delay
  • Check if results exist in date folder
  • Verify dashboard-data.json is present
  • Check console for port conflicts

​
Requirements

System Requirements

  • Python 3.8+
  • See requirements.txt for full dependency list
  • Optional: GPU for faster embeddings processing

​
AI Tools Suite Integration

IntentCrawler is part of the larger Airbais AI Tools Suite with centralized dashboard

Master Dashboard

Centralized view of all AI tool results at ../dashboard/

Auto-Discovery

New tools are automatically detected and integrated

Standard Format

JSON output compatible with other suite tools

Consistent Design

Shared Airbais design system across all tools

​
Master Dashboard Benefits

  • Multi-Tool View: See results from all AI tools in one interface
  • Tool Selection: Dropdown to choose between different analysis tools
  • Date Selection: Browse historical results across all tools
  • Future-Ready: Architecture designed for easy tool addition

​
Future Roadmap

1

Multi-language Support

Expand beyond English content analysis
2

Real-time Tracking

Monitor intent changes over time with the master dashboard
3

A/B Testing

Compare intents across different site versions
4

Suite Expansion

Add sentiment analysis, performance monitoring, and SEO tools
5

API Access

Programmatic access to all suite tools through unified API

​
Contributing

We welcome contributions in these key areas:

Algorithms

Additional clustering algorithms and ML techniques

Visualization

Enhanced dashboard features and data visualization

Performance

Optimization for large-scale websites

Integrations

CMS plugins and third-party tool connections
⌘I