Scaling SEO Across Large Sites

E-commerce site:
- Optimize product page template (1 template)
- 10,000 product pages benefit automatically
- Update once, improvements across all

Website structure:
├── Category 1
│   ├── Page 1
│   ├── Page 2
│   └── Page 3
├── Category 2
└── Category 3

Optimize each category page once
(improves child pages through hierarchy)

WordPress + Yoast:
- Auto-generate meta descriptions
- Title tag suggestions
- Readability checks
- Schema markup generation

Custom CMS:
- API for meta tags
- Automatic URL generation
- Bulk optimization

Formula: [Primary keyword] - [Category] | [Brand]

Example:
- Product: Samsung TV
- Category: Electronics
- Brand: MyStore

Title: "Samsung TV - Electronics | MyStore"
(Applied to 5,000 product pages automatically)

// CMS script
for (page in productPages) {
  page.title = `${page.product} - ${page.category} | MyStore`
}

Formula: [Product name] - [Key feature]. [CTA]. [Brand]

Example:
"Samsung 55\" 4K TV - Ultra HD display with smart TV features. 
Shop now at MyStore with free shipping."

Applied to all products automatically

All pages:
- H1: Product/page name
- H2: Key benefits (auto-generated from attributes)
- H2: Specifications
- H2: Customer reviews
- H2: FAQ

// Generated for all products
{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "[PRODUCT_NAME]",
  "description": "[AUTO_GENERATED]",
  "price": "[PRICE]",
  "aggregateRating": {
    "ratingValue": "[RATING]",
    "reviewCount": "[COUNT]"
  }
}

Priority 1: Content pages, category pages (crawl daily)
Priority 2: Archive pages, older content (crawl weekly)
Priority 3: Pagination, filters (crawl monthly)
Don't crawl: Admin pages, duplicate content, thin content

User-agent: *

# High priority - crawl frequently
Allow: /products/
Allow: /blog/

# Low priority - crawl less
Crawl-delay: 5  # Slow down crawl
User-agent: Googlebot
Crawl-delay: 0  # But not for Google

# Don't crawl
Disallow: /admin/
Disallow: /cart/
Disallow: /*?sort= # Filter parameters

Issues that waste crawl budget:
- Duplicate content (consolidate or noindex)
- Broken links (fix or remove)
- Redirect chains (simplify)
- Paginated archives (limit or noindex)
- URL parameters (manage in GSC)

Fix these → Free up budget for important pages

Check monthly:
- Pages crawled/day (trend)
- Bytes downloaded (efficiency)
- Response time (performance)

Target:
- Steady crawl rate for important pages
- Declining crawl of unimportant pages
- Fast response times

Total pages: 500,000
Indexed: 450,000 (90%) ✓ Good
Excluded: 40,000 (8%) (OK - maybe thin content)
Errors: 10,000 (2%) (FIX - issues)

Monitor aggregate metrics:
- % pages passing LCP
- % pages passing FID/INP
- % pages passing CLS

Target: 75%+ passing all

Product pages: 50% of search traffic
Category pages: 30%
Blog content: 15%
Other: 5%

Optimize highest traffic templates first

# Monitor crawl stats programmatically
from google.ads.googleads.client import GoogleAdsClient

# Get crawl stats
# Alert if crawl budget drops 20%+
# Alert if crawl errors increase
# Alert if core web vitals degrade