| name | reverse-proxy-cms-indexing |
| description | Fix SEO and indexing for a CMS (Odoo, or any web app) served behind a Host-rewriting reverse proxy (nginx or Cloudflare), where the app sees a different internal host than the public domain and therefore emits the wrong robots.txt, sitemap, or canonical URLs. Use this skill whenever a site is not getting indexed, robots.txt says Disallow when it should not, the sitemap or canonical tags point at an internal host, an internal tenant host is itself a crawlable duplicate, or page/template edits do not render live, ESPECIALLY on the everjust.app / customdomain.ai Odoo platform. Covers the domain-mismatch robots+noindex bug, nginx sub_filter canonicalization, duplicate-host noindex, the compiled-template render cache, and a safe backup-validate-rollback config-edit process. |
| license | Complete terms in LICENSE.txt |
SEO/Indexing for a CMS Behind a Host-Rewriting Reverse Proxy
When nginx (or Cloudflare) serves a public domain but rewrites the Host header to an internal host before the app sees it — a common multi-tenant pattern, and exactly how everjust.app tenants + customdomain.ai work (nginx rewrites customdomain.ai -> connectdomain.everjust.app so Odoo's dbfilter matches) — a whole class of SEO bugs appears, because the app builds URLs from the host it sees, not the public one. This skill is the tested playbook.
Symptom -> cause -> fix
1. robots.txt says Disallow: / (whole site blocked from AI + search)
On Odoo, the website.robots template emits Disallow: / inside <t t-if="website.domain and not website._is_indexable_url(url_root)">. Because the proxy rewrites the Host, url_root is the internal host, _is_indexable_url returns False, and the Disallow fires — silently blocking every crawler and LLM fetcher even though pages return 200.
Fix: a DB inherit view on website.robots that sets that t-if to False (xpath, position="attributes"), so robots always renders allow-all. This mirrors the same domain-mismatch patch usually already applied to the page-level <meta noindex> (website.layout); the robots template is a commonly-missed second spot.
2. Sitemap / canonical / robots Sitemap: line emit the INTERNAL host
Canonical <link> and og:url usually already use website.get_base_url() (correct if website.domain is set). But the sitemap <loc>s and the robots Sitemap: directive use url_root (the rewritten internal host), which the app can't fix because it never sees the public host. Do NOT bother flipping website.domain — verify it first; it is often already correct.
Fix at the proxy (the only layer that knows the real host): add locations on the public server block that sub_filter the internal host to the public one:
location ~ ^/sitemap.*\.xml$ {
proxy_pass http://app_upstream;
proxy_set_header Host internal.host; # so dbfilter still matches
proxy_set_header Accept-Encoding ""; # so sub_filter sees uncompressed body
sub_filter_once off;
sub_filter_types application/xml text/xml;
sub_filter 'internal.host' 'public.domain';
}
location = /robots.txt {
proxy_pass http://app_upstream;
proxy_set_header Host internal.host;
proxy_set_header Accept-Encoding "";
sub_filter_once off; sub_filter_types text/plain;
sub_filter 'internal.host' 'public.domain';
}
3. The internal host is a public, crawlable DUPLICATE of the site
On a *.everjust.app wildcard, the internal host (e.g. connectdomain.everjust.app) is itself publicly reachable and crawlable — a full duplicate of the public domain. Canonical tags mitigate it, but it splits crawl budget and can get the internal host indexed.
Fix: a dedicated nginx server block for that exact host (a specific server_name beats the ~^(?<tenant>.+)\.everjust\.app$ regex wildcard, so only that host is affected) that adds add_header X-Robots-Tag "noindex, nofollow" always; and otherwise proxies identically. Use X-Robots-Tag, NOT a redirect — the internal host is web.base.url, so backend/OAuth/email links must keep resolving.
The render-cache gotcha (this wastes hours if unknown)
Editing an Odoo view/template from odoo shell does NOT show up live — the running workers serve a cached compiled template, and env.registry.clear_cache() + bumping the orm_signaling_templates/orm_signaling_registry sequences do NOT propagate cross-process. What actually works:
- A brand-new page/view renders immediately (nothing is cached yet). No restart needed.
- Editing an existing compiled template needs either
docker restart <odoo-container> (a ~30-60s maintenance blip; reload is unreliable due to a stale mounted-file inode), OR — better — writing the view via XML-RPC or the MCP website_edit_page tools, which run in a web worker and invalidate the cache immediately (no restart). Prefer the in-worker write path.
- DATA/content changes (a record field,
ir.attachment, a website setting like custom_code_head) render immediately. Only VIEW/TEMPLATE changes hit the cache.
Corollary: to inject site-wide CSS without a restart, embed it in content (a Html field with sanitize=False) rather than a template <style> inherit that will not render until reload.
Safe process for editing shared production config (nginx)
This config is multi-tenant — a mistake takes down every domain. Always:
- Back up the live file first (
sudo cp everjust.conf everjust.conf.bak-<ts>). It usually needs sudo.
- Make additive edits (new
server/location blocks — they never collide with other edits).
docker exec <nginx> nginx -t to validate.- If OK,
docker restart <nginx>; if not, restore the backup and do NOT restart (the running config stays good until restart).
- Persist to git so a deploy does not revert it: the box's git metadata is often stale/unreliable (the working file is the real live config), so commit to
origin/master via an isolated git worktree off origin/master (never the shared dev checkout), as an additive change, then push.
everjust/customdomain specifics
- Box:
deployment-odoo-1 (Odoo, all tenants) + deployment-nginx-1 (nginx). Config: deployment/nginx/everjust.conf, git-tracked, mounted into the nginx container.
- Verify a fix across ALL tenant domains after an nginx restart (everjust.app, tcstartupweek.com, connectdomain.app, the customdomain, careers.*), not just the one you changed.
Related skills
For the content/citation side of discoverability, see generative-engine-optimization. For agent/MCP discovery, see mcp-server-discoverability.
