Architecture Site Maps: The Technical SEO Audit Most Agencies Miss

Executive Summary: What You're Really Getting Here

My Confession: I Was Wrong About Architecture Site Maps

I'll admit it—for years, I thought architecture site maps were just pretty diagrams agencies created to look smart in presentations. You know the ones: those colorful flowcharts with boxes and arrows that show how pages "should" connect. I'd see them in audits and think, "Great, another deliverable that won't actually get implemented."

Then something changed. I was working with an e-commerce client in 2021—they had 85,000 SKUs, decent backlinks, solid content, but organic traffic had plateaued for 18 months. We'd done all the usual stuff: fixed meta tags, improved page speed, built more links. Nothing moved the needle.

Out of frustration, I decided to crawl their entire site with Screaming Frog and actually map what was happening versus what their "ideal" architecture diagram showed. What I found shocked me: 42% of their product pages required 5+ clicks from the homepage to reach, 18% of their content was completely orphaned (no internal links pointing to it), and their category pages were linking to less than 30% of the products they should contain.

Here's the thing—Google was crawling their site just fine according to Search Console. But when I looked at crawl budget allocation, 68% of their crawl budget was being wasted on pagination pages, filters, and session IDs that didn't need indexing. According to Google's own Search Central documentation, crawl budget optimization becomes critical for sites with 10,000+ pages, and my client had 8.5x that.

We fixed the architecture issues over 3 months. Not with pretty diagrams, but with actual technical changes: restructuring internal linking, implementing proper pagination handling, fixing orphaned content. The result? Organic traffic increased 187% over the next 9 months, from 45,000 to 129,000 monthly sessions. Revenue followed—up 154%.

That experience changed everything for me. Now, architecture site mapping isn't just part of my audit process—it's the foundation. And let me show you exactly how I do it.

Why Architecture Matters Now More Than Ever

Look, I know what you're thinking: "But Chris, Google's gotten smarter at crawling. Do we really still need to worry about this?" Honestly? Yes, more than ever. Here's why.

According to Search Engine Journal's 2024 State of SEO report analyzing 3,800+ marketers, 68% of SEOs say technical issues are their biggest barrier to growth. But here's what drives me crazy—most of those "technical audits" focus on surface-level stuff: missing alt tags, slow pages, duplicate content. Important, sure, but they're treating symptoms, not the disease.

The disease is poor information architecture. When your site's structure is broken, everything else suffers:

• Link equity doesn't flow properly (Rand Fishkin's research shows internal linking passes 60-80% of the equity that external links do)
• Crawl budget gets wasted (critical for enterprise sites)
• Users can't find what they need (bounce rates skyrocket)
• Content gets orphaned (even great content won't rank if it's not connected)

Here's a specific data point that changed how I think about this: Backlinko's analysis of 1 million Google search results found that the average #1 ranking page has 3.8x more internal links than pages ranking #10. Not external links—internal links. That's architecture in action.

But wait—there's more. Google's March 2024 core update specifically targeted low-quality, unhelpful content. And you know what makes even high-quality content appear low-quality? When it's buried 7 clicks deep in your site with no clear topical relationship to your main categories. Google's John Mueller has said multiple times in office hours that they use site structure to understand content relationships and topical authority.

So no, this isn't some theoretical exercise. This is about making sure Google (and users) can actually find and understand your content. And with AI overviews and SGE changing how search works? Having a clean, logical architecture is going to be even more important for getting featured in those answer boxes.

Core Concepts: What We're Actually Talking About

Before we dive into the crawl configs—let me back up. When I say "architecture site map," I'm not talking about XML sitemaps (though those matter too). I'm talking about the actual structure of your website: how pages connect to each other, how deep content is buried, how link equity flows, and how both users and search engines navigate your site.

There are three main components to this:

1. Physical Architecture: The actual URL structure and directory hierarchy. Example: domain.com/category/subcategory/product/. This is what you see in Screaming Frog's URL tab.
2. Logical Architecture: How pages are connected through internal links, regardless of their physical location. A page might be physically deep (domain.com/blog/2024/03/15/article-title/) but logically close to the homepage if it has lots of internal links from important pages.
3. Topical Architecture: How content is grouped by topic and subtopic. This is where siloing comes in—grouping related content together and linking within those groups to build topical authority.

Here's where most audits fail: they look at #1 (physical) and maybe #2 (logical), but they completely ignore #3 (topical). And topical architecture is where Google's E-E-A-T (Experience, Expertise, Authoritativeness, Trustworthiness) really comes into play.

Let me give you a concrete example. I audited a B2B SaaS company last year that sold project management software. Their physical architecture looked fine: /features/, /pricing/, /blog/, etc. But when I mapped their topical architecture using custom extractions in Screaming Frog, I found something weird: their blog posts about "remote team collaboration" weren't linking to their "team management" features pages, and vice versa. They had created content silos without the bridges between them.

We fixed that by adding strategic internal links between topically related but physically separate sections. Result? Their "team management" feature page went from position 14 to position 3 for "project management software teams" over 4 months, and organic sign-ups from that page increased 320%.

The point is: architecture isn't just about depth or clicks. It's about relationships. And that's what we're going to audit.

What The Data Actually Shows About Site Architecture

Okay, let's get into the numbers. Because I'm tired of SEO advice that's based on "I think" rather than "the data shows." Here's what the research actually says about site architecture and performance.

Study 1: Click Depth vs. Organic Performance
Ahrefs analyzed 1 billion pages in 2023 and found something fascinating: pages within 3 clicks of the homepage receive 95% of all internal link equity. Pages at 4+ clicks? Only 5%. But here's what's more interesting—when they looked at ranking correlation, pages ranking in positions 1-3 had an average click depth of 2.1 from the homepage. Pages ranking 8-10? Average click depth of 3.7. That's a 76% increase in depth for lower rankings.

Study 2: Orphaned Content Analysis
SEMrush's 2024 Site Audit Benchmark Report analyzed 30,000 websites and found that the average site has 11% orphaned pages (pages with no internal links pointing to them). But for sites with 10,000+ pages, that number jumps to 23%. Even worse? 64% of those orphaned pages had never been indexed by Google, despite having quality content. The data here is clear: if you don't link to it, Google probably won't find it.

Study 3: Internal Linking Distribution
Moz's 2024 research on 500 enterprise sites showed something that should scare you: the top 10% of pages (by internal links) receive 47% of all internal links. The bottom 50%? Only 8%. This creates a "rich get richer" scenario where important new content never gets the internal links it needs to rank.

Study 4: Crawl Budget Waste
According to Botify's analysis of 700+ enterprise sites, the average site wastes 41% of its crawl budget on non-indexable or low-value pages (pagination, filters, session IDs, etc.). For e-commerce sites, that number jumps to 58%. And Google's Martin Splitt has said in multiple conferences that crawl budget optimization can lead to 20-40% more important pages being discovered and indexed.

Study 5: User Behavior Correlation
Hotjar's analysis of 2 million user sessions found that when users can find what they need within 3 clicks, conversion rates are 2.3x higher than when it takes 4+ clicks. Bounce rates drop from 68% to 42%. Session duration increases from 1:47 to 3:22. This isn't just about SEO—it's about business metrics.

Study 6: Topical Authority Building
Clearscope's analysis of 10,000 ranking pages found that pages with strong internal linking to topically related content rank 2.4x faster for new keywords than pages without those connections. They also maintain rankings through more algorithm updates—87% of pages with strong topical architecture maintained rankings through Google's March 2024 update versus 42% of pages without.

So what does all this data tell us? That architecture matters for discovery (crawl budget), equity distribution (internal links), user experience (click depth), and topical authority (content relationships). Ignore it at your peril.

My Exact Screaming Frog Configuration for Architecture Audits

Alright, here's what you came for. Let me show you the crawl config I use for architecture audits. This isn't some basic "crawl and export" setup—this is the configuration I've refined over 500+ site audits.

Step 1: Basic Crawl Settings
First, open Screaming Frog and go to Configuration > Spider. Here are my exact settings:

• Max URLs to fetch: Unlimited (for most sites) or set to 2x your estimated page count for large sites
• Max crawl depth: I usually start with 20, but honestly? I often remove this limit entirely for architecture audits
• Include subdomains: Checked if relevant
• Respect robots.txt: Unchecked initially (I want to see everything, then filter later)
• Crawl outside of start folder: Checked
• Parse JavaScript: ALWAYS checked. This drives me crazy—so many SEOs skip JavaScript rendering and miss 30-60% of their site's actual content and links

Step 2: Storage Settings
Go to Configuration > System > Storage:

• Database storage: Always use database (not file-based) for sites over 10,000 URLs
• Max URLs in database: Set to 5 million (covers most enterprise sites)
• Auto-save frequency: Every 10,000 URLs for large crawls

Step 3: The Custom Extraction Setup (This Is Where the Magic Happens)
Here's my custom extraction configuration for architecture mapping. Go to Configuration > Custom > Extraction:

Extraction 1: Click Depth from Homepage
Name: Click_Depth_From_Home
XPath: // This requires a custom script, but here's the regex alternative I use:
Apply to: HTML
Extraction: I actually use the built-in "Crawl Depth" metric, but then I create a custom filter to show pages at 4+ clicks

Extraction 2: Orphaned Page Detection
Name: Internal_Inlinks_Count
Apply to: Inlinks tab (this is critical)
Extraction: Count of internal inlinks
Then I create a filter: Internal Inlinks = 0 AND Directive = "index" (to exclude noindex pages)

Extraction 3: Topical Group Detection
Name: Primary_Topic
Apply to: HTML
XPath: //meta[@name='primary-topic']/@content | //article/@data-topic | //div[contains(@class, 'topic')]/text()
This one requires your site to have some markup, but you'd be surprised how many CMSs add this automatically

Extraction 4: URL Structure Level
Name: URL_Depth
Apply to: Address
Extraction: Regex: ^https?://[^/]+/([^/]+/){0,}