Back to Skills

Index Fix

Diagnose and fix non-indexed pages using GSC and Bing Webmaster data. Finds exactly why each page isn't indexed and applies the fix.

By Houseofmvps
11413Updated 4 weeks agoJavaScriptMIT

Skill Content

# Index Fix — Get Every Page Indexed

Diagnose why pages aren't indexed in Google and Bing, fix the root causes, and resubmit for indexing. This skill uses real data from GSC URL Inspection API and Bing Webmaster Tools — no guessing.

**Goal: 100% index coverage for pages that SHOULD be indexed. Not every URL belongs in the index — staging pages, admin panels, thin pagination, and intentionally private content should stay excluded.**

## Phase 1: Credential Check

Verify data sources:
1. **GSC** (required): `ULTRASHIP_GSC_CREDENTIALS` or `ULTRASHIP_GSC_ACCESS_TOKEN`
2. **Bing** (recommended): `ULTRASHIP_BING_KEY`

If GSC is not configured, show setup guide and stop — GSC is required for URL inspection.

## Phase 2: Index Coverage Overview

Get the current index state:
```bash
node ${CLAUDE_PLUGIN_ROOT}/tools/index-doctor.mjs coverage <site-url>
```

This shows:
- Total pages submitted via sitemaps
- Total pages indexed
- Index rate percentage
- Health status (HEALTHY/WARNING/CRITICAL)

## Phase 3: Cross-Engine Comparison

If Bing key is available:
```bash
node ${CLAUDE_PLUGIN_ROOT}/tools/index-doctor.mjs compare <site-url> <sitemap-url>
```

Compare Google vs Bing indexing to find:
- Pages indexed by Google but not Bing (submit to Bing)
- Pages indexed by Bing but not Google (investigate Google issues)
- Overall gap analysis

## Phase 4: Diagnose Non-Indexed Pages

Run the full diagnosis:
```bash
node ${CLAUDE_PLUGIN_ROOT}/tools/index-doctor.mjs diagnose <site-url> <sitemap-url>
```

This inspects up to 50 URLs via GSC URL Inspection API and reports:
- Which pages are indexed vs not
- The specific reason each page isn't indexed
- Severity rating (critical/high/medium/low)
- Exact fix for each issue

### Common Non-Indexing Reasons and Fixes:

**Blocked by robots.txt** (critical):
- **FIRST: Determine if the block is intentional.** Common legitimate reasons to block:
  - Admin/dashboard pages, staging environments, internal tools
  - User-generated content pages, search result pages, faceted navigation
  - API endpoints, health check routes
- If the block is intentional, do NOT remove it — remove the page from the sitemap instead
- If the block is unintentional, find the Disallow rule and modify it using the Edit tool
- Verify the fix: the page should be crawlable

**Noindex tag** (critical):
- **FIRST: Determine WHY noindex was added.** Common legitimate reasons:
  - Staging/preview pages, paginated archives, tag/category pages
  - Thank-you/confirmation pages with sensitive info
  - Legal/compliance requirements, duplicate content by design
  - Pages that should only be accessible via direct link
- **Ask the user before removing noindex** — removing it from staging or admin pages can expose sensitive content to search engines
- If removal is confirmed appropriate, find `<meta name="robots" content="noindex">` in the page HTML
- Remove the noindex directive using the Edit tool
- Check for X-Robots-Tag header in server config

**Soft 404** (high):
- Google thinks the page looks empty or error-like
- Add substantial unique content (minimum 300 words)
- Ensure the page returns HTTP 200 with real content
- Remove any empty state or placeholder content

**Crawled but not indexed** (high):
- Google found the page but decided not to index it
- Usually means content is too thin, duplicate, or low quality
- Add more unique, valuable content
- Build 3+ internal links pointing to this page
- Differentiate from similar pages on the site

**Discovered but not crawled** (medium):
- Google knows the URL exists but hasn't visited it yet
- Build internal links from high-traffic pages
- Submit directly for indexing
- Usually resolves within 1-2 weeks

**Redirect** (medium):
- Page redirects to another URL
- Update sitemap and internal links to use the final destination URL
- Remove the redirecting URL from sitemap

**Canonical mismatch** (medium):
- Google chose a different canonical URL
- Either fix the canonical tag to point to this URL, or accept Google's choice
- Remove from sitemap if it's truly a duplicate

**Server error** (critical):
- Page returns 5xx error when Google crawls it
- Fix the server error (check logs)
- Ensure the page loads correctly under load

**404 Not Found** (high):
- Page doesn't exist anymore
- Remove from sitemap
- Set up 301 redirect to relevant page

## Phase 5: Apply Fixes

**⚠️ SAFETY FIRST: Before applying ANY fix, verify the block/exclusion wasn't intentional. Ask the user if unsure. Removing noindex from staging pages or robots.txt Disallow from admin paths can expose sensitive content.**

For each diagnosed issue, apply the fix:

**robots.txt fixes:**
- Read the current robots.txt
- **Check if the Disallow rule protects staging, admin, or private pages** — if so, remove the page from the sitemap instead of unblocking it
- Only edit rules that are clearly unintentional blocks on public content
- Ensure important public paths are not blocked

**noindex fixes:**
- **Verify with the user that the page should be public** before removing noindex
- Search HTML files for noindex tags
- Remove them using Edit tool only after confirming they're not protecting sensitive content

**Content quality fixes:**
- Identify thin pages (<300 words)
- Flag for content expansion
- Cannot auto-generate content, but can restructure and add schema

**Sitemap cleanup:**
- Remove 404 and redirect URLs from sitemap
- Regenerate if needed:
```bash
node ${CLAUDE_PLUGIN_ROOT}/tools/sitemap-generator.mjs <dir> <base-url>
```

**Internal linking:**
- Find high-traffic pages (from GSC query data)
- Add contextual links from high-traffic pages to non-indexed pages
- Use descriptive anchor text with target keywords

## Phase 6: Resubmit for Indexing

**Only resubmit pages where you've made SUBSTANTIAL fixes** (added content, removed blocking directives, fixed server errors). Do NOT resubmit unchanged pages — this wastes API quota and can flag your account for spam.

After fixes, submit to both search engines:

**Google:**
```bash
# Resubmit sitemap (only after sitemap changes)
node ${CLAUDE_PLUGIN_ROOT}/tools/gsc-client.mjs submit-sitemap <site-url> <sitemap-url>
```

**Bing:**
```bash
# Submit sitemap (only after sitemap changes)
node ${CLAUDE_PLUGIN_ROOT}/tools/bing-webmaster.mjs submit-sitemap <site-url> <sitemap-url>

# Batch submit specific fixed URLs for fast indexing (max 500/day — only URLs you actually fixed)
node ${CLAUDE_PLUGIN_ROOT}/tools/bing-webmaster.mjs submit-url-batch <site-url> <url1> <url2> ...
```

## Phase 7: Auto-Fix and Resubmit

Run the auto-fix command which diagnoses AND submits non-indexed URLs:
```bash
node ${CLAUDE_PLUGIN_ROOT}/tools/index-doctor.mjs fix <site-url> <sitemap-url>
```

This automatically:
1. Inspects all sitemap URLs
2. Identifies non-indexed pages
3. Submits fixable URLs to Bing for re-indexing
4. Returns a prioritized fix plan

## Phase 8: Verification Plan

After applying fixes:
1. Re-run coverage check to compare before/after
2. Set a reminder to re-check in 1-2 weeks (Google re-crawl takes time)
3. Monitor GSC for new indexing issues

```bash
node ${CLAUDE_PLUGIN_ROOT}/tools/index-doctor.mjs coverage <site-url>
```

## Phase 9: Prevention

Advise on preventing future indexing issues:
- Always test new pages for noindex tags before deploying
- Keep sitemap updated (auto-generate on deploy)
- Monitor robots.txt changes
- Ensure all pages have unique, substantial content
- Build internal links to every new page
- Submit sitemap to both GSC and Bing after every deploy

## Key Principles

1. **Diagnose before fixing** — understand WHY each page isn't indexed
2. **Verify intent before removing blocks** — noindex and robots.txt Disallow rules are often intentional (staging, admin, privacy, legal). Ask before removing.
3. **Fix the root cause** — don't just resubmit, fix the underlying issue
4. **Both engines matter** — Bing powers ChatGPT Search, DuckDuckGo, Yahoo
5. **Only submit pages you've actually fixed** — resubmitting unchanged pages wastes quota and can trigger spam flags
6. **Prevention > cure** — set up processes to catch issues before they happen

## Rollback Plan

If fixes cause unexpected issues (pages appearing in search that shouldn't, traffic drops):
1. **Re-add noindex** to any pages that were incorrectly unblocked
2. **Restore robots.txt** Disallow rules that were removed in error
3. **Remove pages from sitemap** that shouldn't be indexed
4. Re-run `coverage` to verify the rollback took effect
5. Google re-crawl may take 1-2 weeks — request re-crawl via GSC for urgent fixes

How to use

  1. Copy the skill content above
  2. Create a .claude/skills directory in your project
  3. Save as .claude/skills/ultraship-index-fix.md
  4. Use /ultraship-index-fix in Claude Code to invoke this skill

README

View on GitHub
<div align="center"> <img src="assets/hero-banner.jpg" alt="Ultraship — Claude Code Plugin" width="100%"/>

Claude Code plugin. 43 expert-level skills for building, shipping, and scaling production software. 37 audit tools (accessibility, vibe-coding security, AI evals, pentest, code quality, bundle size, SEO + AI Readiness check) plus a blocking ship-gate close the loop before deploy. A built-in Currency Guard keeps Claude on current docs, not stale training data.

npm version npm downloads npm total GitHub stars License: MIT CI Sponsor

Follow @kaileskkhumar LinkedIn houseofmvps.com kailxlabs.co

Built by Kaileskkhumar, founder of HouseofMVPs and Kailxlabs

</div> 
0 dependencies · 274 tests · Node.js ESM · MIT

Install

# Claude Code plugin
claude plugin marketplace add Houseofmvps/ultraship
claude plugin install ultraship

# Or standalone via npx
npx ultraship ship .
npx ultraship seo .
npx ultraship security .

How It Works

flowchart LR
    U["You type a<br/>slash command"] --> S["Skill<br/>(markdown instructions)"]
    S --> A["Agent<br/>(dispatched worker)"]
    S --> T["Tools<br/>(Node.js scripts)"]
    A --> T
    T --> O["JSON Results"]
    O --> R["Scorecard / Report /<br/>Actionable Fixes"]

    style U fill:#f59e0b,stroke:#d97706,color:#000
    style S fill:#8b5cf6,stroke:#7c3aed,color:#fff
    style A fill:#3b82f6,stroke:#2563eb,color:#fff
    style T fill:#10b981,stroke:#059669,color:#000
    style R fill:#ef4444,stroke:#dc2626,color:#fff
flowchart TD
    subgraph Lifecycle["Full Lifecycle Coverage"]
        direction LR
        I["Idea<br/>/brainstorm"] --> B["Build<br/>/sprint"]
        B --> AU["Audit<br/>/ship /seo /secure"]
        AU --> D["Ship<br/>/deploy"]
        D --> L["Launch<br/>/launch /compete"]
        L --> G["Grow<br/>/grow /cost"]
        G --> RE["Rescue<br/>/rescue /canary"]
    end

    style I fill:#8b5cf6,stroke:#7c3aed,color:#fff
    style B fill:#3b82f6,stroke:#2563eb,color:#fff
    style AU fill:#f59e0b,stroke:#d97706,color:#000
    style D fill:#10b981,stroke:#059669,color:#000
    style L fill:#06b6d4,stroke:#0891b2,color:#000
    style G fill:#84cc16,stroke:#65a30d,color:#000
    style RE fill:#ef4444,stroke:#dc2626,color:#fff

What /ship Does

/ship runs 6 tools in parallel and outputs a scorecard:

flowchart LR
    SHIP["/ship"] --> SEO["seo-scanner<br/>63 rules"]
    SHIP --> A11Y["a11y-scanner<br/>WCAG 2.2"]
    SHIP --> SEC["secret-scanner<br/>+ npm audit"]
    SHIP --> CODE["code-profiler<br/>N+1, leaks, ReDoS"]
    SHIP --> BUNDLE["bundle-tracker<br/>JS/CSS/images"]
    SHIP --> ENV["env-validator<br/>+ migration-checker"]

    SEO --> SC["Scorecard<br/>READY TO SHIP"]
    A11Y --> SC
    SEC --> SC
    CODE --> SC
    BUNDLE --> SC
    ENV --> SC

    style SHIP fill:#f59e0b,stroke:#d97706,color:#000
    style SC fill:#10b981,stroke:#059669,color:#000
    style SEO fill:#3b82f6,stroke:#2563eb,color:#fff
    style SEC fill:#3b82f6,stroke:#2563eb,color:#fff
    style CODE fill:#3b82f6,stroke:#2563eb,color:#fff
    style BUNDLE fill:#3b82f6,stroke:#2563eb,color:#fff
    style ENV fill:#3b82f6,stroke:#2563eb,color:#fff
+===========================================+
|      U L T R A S H I P   S C O R E       |
+===========================================+
|  SEO + AI Vis.  92/100  ############-    |
|  Security        95/100  ############-    |
|  Code Quality    88/100  ###########--    |
|  Bundle Size     97/100  ############-    |
+===========================================+
|   OVERALL         90/100                  |
|   STATUS          READY TO SHIP           |
+===========================================+
<details> <summary>Demo</summary> <img src="assets/demo.gif" alt="Ultraship — SEO audit, secret scanning, scorecard" width="100%"/> </details>

Tools (40)

Each tool is a standalone Node.js script (node tools/<name>.mjs). JSON output. Exit 0 always. No build step.

Auditing

ToolWhat it checks
seo-scanner63 rules: 39 SEO (meta tags, canonicals, headings, OG tags, structured data, sitemap, cross-page duplicate/orphan detection), 20 GEO (AI bot access in robots.txt, snippet restrictions, llms.txt, structured data for AI extraction), 4 AEO (FAQPage/HowTo/speakable schema)
a11y-scannerWCAG 2.2 A/AA static checks: missing alt text, unlabeled form controls, icon-only buttons, missing lang/title/main, heading order, positive tabindex, zoom disabled, duplicate ids, broken aria references. Zero false positives.
ship-gateBlocking quality gate — scores all auditors (shared math with /ship), compares to .ultraship/ship-gate.json thresholds, hard-fails on leaked secrets / critical findings, exits 1 on fail. Generates a pre-push hook + GitHub Actions workflow.
secret-scannerAWS keys, Stripe keys, JWT secrets, database URLs, private keys. Redacts values in output.
vibe-security-scannerVibe-Coding Security Sentinel — context secret-scanner misses: server-only secrets behind a NEXT_PUBLIC_/VITE_ prefix, a decoded Supabase service_role key exposed to the client, service_role in a "use client" file, Supabase tables with no RLS. Zero false positives.
eval-scannerLocates every LLM call site (Anthropic, OpenAI, Gemini, Mistral, Cohere, Ollama, Vercel AI SDK, LangChain) by provider + model id, detects the test runner and whether an eval suite exists. Flags AI features shipping with no evals. Seeds /evals. Zero false positives.
code-profilerN+1 queries, sync I/O in handlers, unbounded queries, missing indexes, memory leaks, sequential awaits, ReDoS risk
bundle-trackerJS/CSS/image sizes in build output. Detects heavy deps (moment→dayjs, lodash→native). History for before/after. Monorepo-aware.
dep-doctorUnused dependencies via import graph analysis (not just grep). Dead wrapper files. Outdated packages.
content-scorerFlesch-Kincaid readability, keyword density, thin content detection, GEO heading analysis
lighthouse-runnerLighthouse via headless Chrome. Core Web Vitals, render-blocking resources, diagnostics.

Validation

ToolWhat it checks
health-checkHTTP status, response time, SSL certificate (issuer, expiry), 6 security headers
env-validatorCompares .env.example against actual .env. Catches missing/empty/placeholder vars.
migration-checkerPending DB migrations for Drizzle, Prisma, Knex
og-validatorOpen Graph tags, image reachability, size validation
redirect-checkerRedirect chains, loops, mixed HTTP/HTTPS. Sitemap-based bulk check.
api-smoke-testHit API endpoints, check status codes, response times, CORS headers

Generators

ToolWhat it creates
sitemap-generatorsitemap.xml from HTML files and routes
robots-generatorAI-friendly robots.txt (allows GPTBot, PerplexityBot, ClaudeBot)
llms-txt-generatorllms.txt for AI assistant discoverability
structured-data-generatorJSON-LD schema markup

Competitive & Launch

ToolWhat it does
compete-analyzerCompares two URLs: tech stack, SEO score, security headers, response time. ASCII comparison card.
launch-prepReads project, generates PH/Twitter/LinkedIn/HN copy, 14-item checklist, press kit
demo-prepFinds console.logs, TODOs, placeholder text, missing favicons. Scores demo readiness.

Operations

ToolWhat it does
incident-commanderHealth check + git culprit analysis + error patterns + rollback commands + post-mortem template
growth-trackerUptime, git velocity, SEO trajectory, dep health. Stores snapshots for week-over-week comparison.
cost-trackerLog AI token usage per feature/model. Built-in pricing for Claude, GPT-4o, Gemini. Daily trends.
pentest-scannerAutomated penetration testing: XSS, SQLi, SSTI, command injection, path traversal, CORS, JWT, GraphQL introspection, prototype pollution, race conditions, request smuggling. Zero false positives, every finding has proof-of-concept.
canary-monitorPost-deploy canary monitoring: HTTP status, response time, error patterns, baseline regression detection. Auto-saves baselines for future comparison.
retro-analyzerSprint retrospective: git velocity, commit patterns (features vs fixes), test health, hot files, shipping cadence. Generates insights and recommendations.
learnings-managerProject learnings CRUD: save, search, list, prune, export. Structured knowledge that compounds across sessions.

Project Analysis

ToolWhat it does
onboard-generatorAuto-generates developer guide: stack, directory tree, routes, schema, env vars, Mermaid diagram
architecture-mapper4 Mermaid diagrams: system overview, route tree, DB ER, data flow. Circular dependency + orphan detection.
pattern-analyzerAnalyzes testing, error handling, TypeScript usage, CI/CD, git practices. Cross-repo comparison.
audit-historySaves/compares audit scores over time

Integrations (optional)

ToolWhat it does
gsc-clientGoogle Search Console: submit sitemaps, inspect URLs, query rankings (requires ULTRASHIP_GSC_CREDENTIALS)
bing-webmasterBing Webmaster: submit sitemaps/URLs, IndexNow instant push, keyword research, backlinks, site-scan, URL inspection (requires ULTRASHIP_BING_KEY). Powers ChatGPT Search + Microsoft Copilot.
ga4-clientGoogle Analytics 4: overview, top-pages, landing-pages, traffic-sources, conversions, user-journey, devices, realtime, ai-traffic (ChatGPT/Perplexity/Copilot tracking), organic (search-only). --organic flag.
keyword-intelligence12-command keyword engine: analyze, quick-wins, cannibalization, content-gaps, intent-map, trending, high-intent, page-keywords, content-decay, difficulty, anomalies (CTR anomalies), cross-reference (GSC↔GA4). --brand flag for non-brand filtering.
index-doctorIndex diagnosis: inspect URLs via GSC URL Inspection API, diagnose 15+ coverage states, auto-fix and submit to Bing.

…

View source on GitHub