SEO Technical
Technical SEO audit across 9 categories: crawlability, indexability, security, URL structure, mobile, Core Web Vitals, structured data, JavaScript rendering, and IndexNow protocol. Use when user says "technical SEO", "crawl issues", "robots.txt", "Core Web Vitals", "site speed", or "security headers".
MCP get_skill({ skillId: "technical-seo-audit-1d169105" })Use this skill with your agent
Create a free account and connect via MCP
# Technical SEO Audit ## Categories ### 1. Crawlability - robots.txt: exists, valid, not blocking important resources - XML sitemap: exists, referenced in robots.txt, valid format - Noindex tags: intentional vs accidental - Crawl depth: important pages within 3 clicks of homepage - JavaScript rendering: check if critical content requires JS execution - Crawl budget: for large sites (>10k pages), efficiency matters #### AI Crawler Management As of 2025-2026, AI companies actively crawl the web to train models and power AI search. Managing these crawlers via robots.txt is a critical technical SEO consideration. **Known AI crawlers:** | Crawler | Company | robots.txt token | Purpose | |---------|---------|-----------------|---------| | GPTBot | OpenAI | `GPTBot` | Model training | | ChatGPT-User | OpenAI | `ChatGPT-User` | Real-time browsing | | ClaudeBot | Anthropic | `ClaudeBot` | Model training | | PerplexityBot | Perplexity | `PerplexityBot` | Search index + training | | Bytespider | ByteDance | `Bytespider` | Model training | | Google-Extended | Google | `Google-Extended` | Gemini training (NOT search) | | CCBot | Common Crawl | `CCBot` | Open dataset | **Key distinctions:** - Blocking `Google-Extended` prevents Gemini training use but does NOT affect Google Search indexing or AI Overviews (those use `Googlebot`) - Blocking `GPTBot` prevents OpenAI training but does NOT prevent ChatGPT from citing your content via browsing (`ChatGPT-User`) - ~3-5% of websites now use AI-specific robots.txt rules **Example, selective AI crawler blocking:** ``` # Allow search indexing, block AI training crawlers User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Bytespider Disallow: / # Allow all other crawlers (including Googlebot for search) User-agent: * Allow: / ``` **Recommendation:** Consider your AI visibility strategy before blocking. Being cited by AI systems drives brand awareness and referral traffic. Cross-reference the `seo-geo` skill for full AI visibility optimization. ### 2. Indexability - Canonical tags: self-referencing, no conflicts with noindex - Duplicate content: near-duplicates, parameter URLs, www vs non-www - Thin content: pages below minimum word counts per type - Pagination: rel=next/prev or load-more pattern - Hreflang: correct for multi-language/multi-region sites - Index bloat: unnecessary pages consuming crawl budget ### 3. Security - HTTPS: enforced, valid SSL certificate, no mixed content - Security headers: - Content-Security-Policy (CSP) - Strict-Transport-Security (HSTS) - X-Frame-Options - X-Content-Type-Options - Referrer-Policy - HSTS preload: check preload list inclusion for high-security sites ### 4. URL Structure - Clean URLs: descriptive, hyphenated, no query parameters for content - Hierarchy: logical folder structure reflecting site architecture - Redirects: no chains (max 1 hop), 301 for permanent moves - URL length: flag >100 characters - Trailing slashes: consistent usage ### 5. Mobile Optimization - Responsive design: viewport meta tag, responsive CSS - Touch targets: minimum 48x48px with 8px spacing - Font size: minimum 16px base - No horizontal scroll - Mobile-first indexing: Google indexes mobile version. **Mobile-first indexing is 100% complete as of July 5, 2024.** Google now crawls and indexes ALL websites exclusively with the mobile Googlebot user-agent. ### 6. Core Web Vitals - **LCP** (Largest Contentful Paint): target <2.5s - **INP** (Interaction to Next Paint): target <200ms - INP replaced FID on March 12, 2024. FID was fully removed from all Chrome tools (CrUX API, PageSpeed Insights, Lighthouse) on September 9, 2024. Do NOT reference FID anywhere. - **CLS** (Cumulative Layout Shift): target <0.1 - Evaluation uses 75th percentile of real user data - Use PageSpeed Insights API or CrUX data if MCP available ### 7. Structured Data - Detection: JSON-LD (preferred), Microdata, RDFa - Validation against Google's supported types - See seo-schema skill for full analysis ### 8. JavaScript Rendering - Check if content visible in initial HTML vs requires JS - Identify client-side rendered (CSR) vs server-side rendered (SSR) - Flag SPA frameworks (React, Vue, Angular) that may cause indexing issues - Verify dynamic rendering setup if applicable #### JavaScript SEO: Canonical & Indexing Guidance (December 2025) Google updated its JavaScript SEO documentation in December 2025 with critical clarifications: 1. **Canonical conflicts:** If a canonical tag in raw HTML differs from one injected by JavaScript, Google may use EITHER one. Ensure canonical tags are identical between server-rendered HTML and JS-rendered output. 2. **noindex with JavaScript:** If raw HTML contains `<meta name="robots" content="noindex">` but JavaScript removes it, Google MAY still honor the noindex from raw HTML. Serve correct robots directives in the initial HTML response. 3. **Non-200 status codes:** Google does NOT render JavaScript on pages returning non-200 HTTP status codes. Any content or meta tags injected via JS on error pages will be invisible to Googlebot. 4. **Structured data in JavaScript:** Product, Article, and other structured data injected via JS may face delayed processing. For time-sensitive structured data (especially e-commerce Product markup), include it in the initial server-rendered HTML. **Best practice:** Serve critical SEO elements (canonical, meta robots, structured data, title, meta description) in the initial server-rendered HTML rather than relying on JavaScript injection. ### 9. IndexNow Protocol - Check if site supports IndexNow for Bing, Yandex, Naver - Supported by search engines other than Google - Recommend implementation for faster indexing on non-Google engines ## Agent-Friendly Pages (forward-looking) AI agents (not just AI summarizers) increasingly read sites through three channels: vision models on screenshots, raw HTML/DOM, and the **accessibility tree** (the cleanest signal). Audit criteria — semantic HTML (real `<button>` and `<a>`, not `<div onclick>`), label associations, interactive target sizing, layout stability across templates, `cursor: pointer` correctness — live in `references/agent-friendly-pages.md`. ### Audit command ```bash # Render with Playwright + capture accessibility tree, then score python3 scripts/agent_ux_check.py https://example.com --json ``` The scanner outputs an Agent-UX score (0-100) plus itemized issues: - HTML findings: real buttons / anchors, `<div onclick>` widgets, semantic landmarks, inputs without `<label for>`, inputs without ARIA labels - Accessibility tree findings: total nodes, interactive nodes, unnamed interactive elements, `role="generic"` ratio The accessibility-tree snapshot uses Playwright's `page.accessibility.snapshot(interesting_only=False)`. To capture the tree without scoring, use `python3 scripts/render_page.py <url> --a11y-tree --json`. Surface findings as **opportunities**, not failures. The standards (WebMCP, agent UX heuristics) are early — don't gate audits on a sub-100 score. ## Output ### Technical Score: XX/100 ### Category Breakdown | Category | Status | Score | |----------|--------|-------| | Crawlability | pass/warn/fail | XX/100 | | Indexability | pass/warn/fail | XX/100 | | Security | pass/warn/fail | XX/100 | | URL Structure | pass/warn/fail | XX/100 | | Mobile | pass/warn/fail | XX/100 | | Core Web Vitals | pass/warn/fail | XX/100 | | Structured Data | pass/warn/fail | XX/100 | | JS Rendering | pass/warn/fail | XX/100 | | IndexNow | pass/warn/fail | XX/100 | ### Critical Issues (fix immediately) ### High Priority (fix within 1 week) ### Medium Priority (fix within 1 month) ### Low Priority (backlog) ## DataForSEO Integration (Optional) If DataForSEO MCP tools are available, use `on_page_instant_pages` for real page analysis (status codes, page timing, broken links, on-page checks), `on_page_lighthouse` for Lighthouse audits (performance, accessibility, SEO scores), and `domain_analytics_technologies_domain_technologies` for technology stack detection. ## Google API Integration (Optional) If Google API credentials are configured, use `python3 scripts/pagespeed_check.py <url> --json` for real PSI + CrUX field data (replaces lab-only CWV estimates), `python3 scripts/crux_history.py <url> --json` for 25-week CWV trends, and `python3 scripts/gsc_inspect.py <url> --json` for real indexation status per URL. ## Error Handling | Scenario | Action | |----------|--------| | URL unreachable | Report connection error with status code. Suggest verifying URL, checking DNS resolution, and confirming the site is publicly accessible. | | robots.txt not found | Note that no robots.txt was detected at the root domain. Recommend creating one with appropriate directives. Continue audit on remaining categories. | | HTTPS not configured | Flag as a critical issue. Report whether HTTP is served without redirect, mixed content exists, or SSL certificate is missing/expired. | | Core Web Vitals data unavailable | Note that CrUX data is not available (common for low-traffic sites). Suggest using Lighthouse lab data as a proxy and recommend increasing traffic before re-testing. |
Related Skills
More skills in Business, Marketing & Sales
Ab Testing
When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "should I test this," "which version is better," "test two versions," "statistical significance," "how long should I run this test," "growth experiments," "experiment velocity," "experiment backlog," "ICE score," "experimentation program," or "experiment playbook." Use this whenever someone is comparing two approaches and wants to measure which performs better, or when they want to build a systematic experimentation practice. For tracking implementation, see analytics. For page-level conversion optimization, see cro.
Ab Test Setup
When the user wants to plan, design, or implement an A/B test or experiment. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "conversion experiment," "statistical significance," or "test this." For tracking implementation, see analytics-tracking.
Ab Test Setup
Ab Test Setup linked from Corey Haines marketing skills, with the upstream skill instructions available on GitHub.
Ab Test Store Listing
When the user wants to A/B test App Store product page elements to improve conversion rate. Also use when the user mentions "A/B test", "product page optimization", "test my screenshots", "test my icon", "conversion rate optimization", "CPP", or "custom product pages". For screenshot design, see screenshot-optimization. For metadata optimization, see metadata-optimization.
Account Research
Research a company or person and get actionable sales intel. Works standalone with web search, supercharged when you connect enrichment tools or your CRM. Trigger with "research [company]", "look up [person]", "intel on [prospect]", "who is [name] at [company]", or "tell me about [company]".
Account Research
Research a company using Common Room data. Triggers on 'research [company]', 'tell me about [domain]', 'pull up signals for [account]', 'what's going on with [company]', or any account-level question.
Explore Other Categories
Skills from other categories with shared topics
Image
When the user wants to create, generate, edit, or optimize images for marketing — blog heroes, social graphics, product mockups, profile banners, listing visuals, or brand assets. Also use when the user mentions 'AI image generation,' 'generate an image,' 'create a graphic,' 'product mockup,' 'hero image,' 'social media graphic,' 'banner image,' 'cover photo,' 'profile banner,' 'listing screenshot,' 'Flux,' 'Flux Kontext,' 'Midjourney,' 'DALL-E,' 'GPT Image,' 'ChatGPT Images,' 'Ideogram,' 'Gemini image,' 'Nano Banana,' 'Recraft,' 'Stable Diffusion,' 'Canva,' 'Figma,' 'image optimization,' 'compress images,' 'WebP,' or 'OG image.' Use this for general-purpose marketing image creation and optimization. For paid ad image creative and platform-specific ad specs, see ad-creative. For video production, see video.
Kb Article
Draft a knowledge base article from a resolved issue or common question. Use when a ticket resolution is worth documenting for self-service, the same question keeps coming up, a workaround needs to be published, or a known issue should be communicated to customers.
Agent Protocol
Inter-agent communication protocol for C-suite agent teams. Defines invocation syntax, loop prevention, isolation rules, and response formats. Use when C-suite agents need to query each other, coordinate cross-functional analysis, or run board meetings with multiple agent roles.