🔍 PastHunter

Advanced Host Discovery & Historical URL Analysis Plugin for Caido

PastHunter is a powerful Caido plugin that extracts host addresses from your HTTP traffic and integrates with the Wayback Machine to discover historical URLs for comprehensive reconnaissance.

✨ Features

• 🎯 Real-time Host Discovery: Automatically extracts domains and subdomains from Caido HTTP traffic
• 🕰️ Wayback Machine Integration: Fetches up to 10,000 historical URLs per domain from the Internet Archive
• 📋 Smart Organization:
  • Scrollable host list with clean interface
  • Domain-by-domain wayback results with navigation
  • Responsive design that fits any screen
• 🔄 Bulk Operations: Copy all URLs from a domain with one click
• 🎨 Modern UI: Colorful, intuitive interface with dark/light mode support
• ⚡ Performance Optimized: Efficient scrolling containers that stay within screen boundaries

🚀 Installation

1. Download the latest release from the Releases page
2. In Caido, go to Settings > Plugins
3. Click Install Plugin and select the downloaded file
4. The plugin will appear in your sidebar as "PastHunter"

📖 Usage

Host Discovery

1. Click "Initialize from Recent Requests" to scan your Caido traffic
2. The plugin will extract all unique hosts from your HTTP requests
3. Browse through discovered hosts using the scrollable interface

Historical URL Analysis

1. Click the "Wayback" button next to any discovered host
2. PastHunter will query the Internet Archive for historical URLs
3. Navigate between domains using the Previous/Next buttons
4. Use "Copy All URLs" to copy all historical URLs for a domain

Features Overview

• Copy Individual Hosts: Click "Copy" next to any host
• Copy All URLs: Bulk copy all historical URLs for a domain
• Scrollable Results: No pagination needed - smooth scrolling through all results
• Responsive Design: Containers automatically limit height to fit your screen

How It Works

Host Tracking

The plugin runs in the background and:

• 🎯 Tracks the hostname of every HTTP request made through Caido
• 📝 Stores discovered hosts with timestamps and request counts
• 🔄 Maintains the list across plugin sessions
• 📊 Shows statistics about discovered hosts and total requests tracked

Wayback Machine Integration

When you click the Wayback button for a domain:

• 🌐 Queries the Internet Archive CDX API: https://web.archive.org/cdx/search/cdx
• 📊 Uses parameters: url=domain/*&matchType=domain&output=json&fl=timestamp,original,statuscode&collapse=urlkey&limit=100
• ⏱️ Uses Caido's HTTP module (import { fetch } from "caido:http") for external requests
• 📋 Displays up to 10 results in the UI with clickable links and timestamps
• 🔍 Full results available in browser console for detailed analysis
• ⚡ 30-second timeout with proper error handling

Technical Details

Backend Implementation

The backend (packages/backend/src/index.ts) includes:

• Host Validation: RFC-compliant domain validation
• Content Parsing: Advanced regex patterns for host extraction
• Multi-Source Analysis: Extracts from:
  • Sitemap entries
  • HTTP request URLs and headers
  • HTTP response headers and bodies
• Error Handling: Graceful handling of invalid URLs and data

Frontend Implementation

The frontend (packages/frontend/src/views/App.vue) features:

• Vue 3 Composition API: Modern reactive framework
• PrimeVue Components: Professional UI components
• Real-time Statistics: Live update of extraction progress
• Responsive Design: Works across different screen sizes
• Toast Notifications: User feedback for actions

Host Detection Patterns

The plugin uses multiple regex patterns to identify hosts:

1. Standard Domain Pattern: Matches https?://domain.com formats
2. Subdomain Pattern: Matches multi-level subdomains
3. Header Extraction: Extracts from Host, Referer, Location headers
4. Content Parsing: Finds embedded URLs in request/response bodies

Development

Project Structure

├── packages/
│   ├── backend/          # Backend plugin logic
│   │   └── src/
│   │       └── index.ts  # Main backend implementation
│   └── frontend/         # Frontend Vue.js application
│       └── src/
│           ├── index.ts  # Frontend entry point
│           └── views/
│               └── App.vue # Main UI component
├── caido.config.ts       # Plugin configuration
└── package.json          # Project dependencies

Available Scripts

• npm run build - Build the plugin package
• npm run watch - Watch mode for development
• npm run lint - Run ESLint
• npm run typecheck - Run TypeScript type checking

API Integration

The plugin uses a simple, real-time tracking approach:

Core Tracking Logic:

// Track hosts from every HTTP request
const onRequest = (sdk: SDK, request: any): void => {
  const url = request.getUrl();
  const urlObj = new URL(url);
  const hostname = urlObj.hostname;
  
  // Store in persistent map
  trackHost(hostname, urlObj.pathname);
};

Initialization from History:

// Populate from recent requests on startup
const query = sdk.requests.query().first(500);
const results = await query.execute();

for (const item of results.items) {
  const url = item.request.getUrl();
  const urlObj = new URL(url);
  trackHost(urlObj.hostname, urlObj.pathname);
}

What We Track:

• ✅ Request URL hostnames
• ✅ Host headers (if different)
• ✅ Request timestamps and counts
• ✅ Unique paths per host

What We DON'T Track (keeps it clean):

• ❌ Response body content
• ❌ JavaScript embedded URLs
• ❌ CSS/image references
• ❌ Third-party tracking domains from content

Technology Stack

• TypeScript - Type-safe JavaScript
• Vue.js 3 - Progressive frontend framework
• PrimeVue - Rich UI component library
• pnpm - Fast, disk space efficient package manager
• Caido SDK - Plugin development framework

Contributing

1. Fork the repository
2. Create a feature branch
3. Implement your changes
4. Run tests and linting
5. Submit a pull request

Security Considerations

• All extracted data is processed locally within Caido
• No external network requests are made
• Domain validation prevents injection attacks
• Proper error handling prevents crashes

License

This project is licensed under the MIT License.