Thirty Thousand Part Numbers: Crawl Budget, Discovery and the Indexing Hub
A parts catalogue is written for one reader: the planner who types an exact part number at seven in the morning and needs the right page back. If that page has never been crawled, the catalogue does not exist for him. This article deals with the stage before ranking: how a bot rations its visits, how a sitemap steers them, and where the Indexing Hub fits.
Most Edmonton suppliers can state their catalogue size to the line: 38,000 stock numbers, or 1,200 live units plus a rental fleet turning over monthly and a service-area tree from Nisku to Fort Saskatchewan. No such figure exists for how many of those pages an engine has seen.
The gap between the two counts is where quote requests go missing, and no dashboard reports it. Traffic looks flat, rankings look unchanged, and the 6,000 supersession pages published in March are absent from the market until the autumn.
A published part number is not a findable part number
Publishing and being found are two events, weeks apart on a large catalogue, with three mechanisms in between. An engine has to obtain the address from somewhere. A bot has to spend a request on it and receive something usable. The index has to accept the result. One word covers all three, which is why the argument in the meeting is usually about the wrong stage.
- Discovery is an information failure. Nothing links to the page. It sits behind a lookup form that posts a query, or three clicks deep in a filter tree. Rejection is not the issue; the address was never communicated.
- Crawling is a resource failure. Requests to your domain are rationed daily, and the ration can go entirely on cross-reference permutations resolving to a dozen products.
- Retention is a judgement. A page carrying a stock number, a weight and nothing else resembles the 4,000 around it, so it is fetched, assessed and left out.
- The fixes do not substitute for one another. Resubmission cannot rescue a page that was fetched and declined; a rewrite cannot help one no bot has asked for.
Take a Nisku distributor that loads 6,400 supersession pages and finds 1,900 indexed four months later. Two readings of one number: if the log holds no bot request for the absent 4,500, the failure is discovery and a queue will move them. If all 4,500 were served cleanly and dropped anyway, another round of submissions changes nothing.
Crawl budget is spent, not granted
Crawl budget is not a quota you can request, buy or negotiate. It emerges from two things an engine measures continuously: how quickly and reliably your server responds, and what the domain appears to be worth. Serve a run of 500s under Monday morning load and the rate stays depressed for weeks afterwards, since a bot throttles itself before it will overload a host.
What makes this awkward is where the budget lands. Most of it goes on addresses no one chose to publish — machine output, not editorial decisions — and there are ten of those per page a person wrote.
| Source of the drain | How it appears on an Edmonton catalogue | Order of magnitude | Where the fix lives |
|---|---|---|---|
| Faceted parts navigation | Bore, thread, pressure rating, material and brand filters, all serving a 200 status | Tens of thousands of addresses out of a few thousand stock numbers | Disallow rules, plus canonical tags on every filter state |
| Cross-reference permutations | Competitor number to your number, generated both ways, sometimes three deep | Multiplies the catalogue four or five times | One canonical page per part, references pointing at it |
| Sold and returned fleet units | Machines long gone from the yard whose listings still resolve | Accumulates every month the yard trades | A disposal rule — gone means 410 or a redirect |
| Slow responses under catalogue load | Parts search timing out while a bot walks the filter tree | Holds the rate down long after the load passes | Caching and headroom sized for automated traffic |
Work it through for a mid-sized distributor: eight thousand genuine part pages, five filter dimensions and a two-way cross-reference table exceed 200,000 fetchable addresses. A line added on Tuesday sits behind thread-and-material permutations, no error is raised anywhere, and it surfaces around the time the quoting season ends.
Four shapes of Edmonton site that hit this wall
One trait links the firms that hit this first: page count follows inventory and territory, not an editorial calendar. No one approved 38,000 pages. The ERP produced them.
Parts catalogues with supersessions
Every stock number gets a page, plus the numbers it replaces and those replacing it. The catalogue grows in three directions.
- Superseded numbers carry the demand
- New numbers indexed months late
- Cross-references duplicate the tree
Dealer stock and rental fleets
Units arrive, go out on rent, return and sell. Commercial value and page lifespan run in opposite directions here.
- Sold before the page is crawled
- Sold listings never retired
- Thin serial-specific pages
Capability and material pages
Tolerances, plate thicknesses, alloys, weld procedures and certifications, each a page a specifier searches by name.
- Low volume, high intent
- Templated variants read as duplicates
- Certifications buried under PDFs
Service areas across the region
Field service sold across a wide catchment produces a page per community, differing only in a place name.
- Thirty near-identical pages per service
- Multiplied by every service line
- Absorbs budget, earns nothing
The same reversal runs through all four. This week's arrivals and this quarter's supersessions carry the value and expire fastest, yet they wait behind pages that never expire and never generate an enquiry. That is the quiet bottleneck.
It also distorts measurement: a monthly traffic comparison tells you little when the page set itself moved. The honest question is narrower — of the part numbers published in a month, how many were crawled within thirty days.
A sitemap is a shipping manifest, not paperwork
For most companies the sitemap is a file a plugin emits and nobody has opened since launch. Where stock changes weekly it is the one mechanism for declaring what moved, and how it is built determines whether you can interpret the reply.
Feed the Hub an uploaded file or an address and it walks the tree three levels down: a top index, section indexes underneath, address files at the bottom. One job follows as many as 1,000 sitemaps, so splitting the tree is implementable at catalogue scale.
A sitemap tree that a parts catalogue can live with
Divide it by rate of change, so that a section falling behind is legible from the counters alone.
- The top file is one index and nothing else. One address given to the engine and the Hub, listing nothing but the section indexes.
- Section indexes track turnover. Stock numbers, live fleet, capability pages, territory pages and the French tree each get their own, since each moves at its own speed.
- Address files stay small. Sized so a parse error is confined to one file you can still open and read.
- Dates must carry information. Where lastmod is stamped by the nightly build regardless of edits, the value stops being read at all.
- List only what a visitor can reach. Redirects, dead numbers, noindexed pages and anything canonicalised elsewhere come out. A manifest describing crates that never went on the truck misleads the receiver.
Pages that rarely change
Capabilities, certifications, contact, category tree. Resubmitted only after a real edit.
Part numbers and supersessions
Built from the ERP export: one canonical page per part, cross-references excluded.
Units currently on the lot
Rebuilt nightly. A unit leaves the file the day it sells or goes on long-term rental.
Service-area pages
Small and slow-moving, kept apart so weak performance cannot hide in the catalogue counts.
Built like that, the file stops being documentation and starts producing readings. Nine hundred submitted against 240 found in the stock-number index, beside 40 of 44 in the capability index, identifies the failing section.
The four ceilings inside the Indexing Hub
The Hub occupies one tab of the panel, next to campaign automation, Search Console analytics and rank tracking; its remit stops where ranking begins. The walkthrough of the rebuilt Semalt panel covers how one account and a shared set of site tags span all of them — which a dealer group with six domains notices immediately. Four figures define what the Hub accepts.
Daily rate, batch ceiling, parse depth, parallel jobs
Build the submission calendar for a 40,000-page catalogue around these and nothing surprises you.
- The tracker moves 1,000 URLs in a day. The allowance belongs to the account and expires unused. Skipping the weekend does not produce a 3,000-URL Monday.
- A batch may carry 10,000 URLs. This governs how much you hand over at once, not how fast it moves: a full batch is ten days of processing.
- Parsing follows three levels of nesting. Supply the file or the address; indexes inside indexes are traversed, to a limit of 1,000 sitemaps per job.
- Two jobs run together; twenty may wait. There is no widening it, so a catalogue migration has to be ordered in advance rather than thrown at the Hub.
Delivery runs over the IndexNow API, which announces to GoogleBot and BingBot that something at an address is new or changed. That is the whole protocol: it compresses the wait before the first fetch, and everything after the fetch is outside its remit.
How to read a finished batch honestly
The Hub keeps a per-URL record: which bot visited, with a timestamp, the status returned and the error detail where there was one. Live counters for submitted, found and failed run beside it — the only part of the process producing evidence rather than an impression.
| What the record shows | What it actually means | What to do next |
|---|---|---|
| Submitted, no bot visit logged | Notification sent, no fetch yet. Normal for days, not weeks | Wait a cycle, then check internal linking and crawl activity |
| Visited, 200 returned, not indexed | Fetched and set aside. A judgement was made | Stop submitting. Look at cross-reference duplication and at how thin the page is |
| Visited, 404 or 410 | A retired unit or discontinued number went in | Fix the export that builds the batch |
| Visited, 5xx or timeout | The server failed under crawl load | Infrastructure. Resubmitting under the same condition worsens it |
| High found count, no change in enquiries | The wrong pages were indexed | Check in Search Console which pages draw technical queries |
The last row catches people out. Twelve thousand indexed pages is not an outcome; a planner searching an exact number, landing on the right page and sending a quote request is. The Search Console analytics that sit beside it close the loop: the Hub confirms a page was reached, the analytics say whether it attracts the queries you quote against.
Where a submission queue helps, and where it cannot
Submission is the last step, not the first. Run it before the site is in order and all you get is a well-documented account of your own duplication.
- Cut the noise first. Close off filter states, collapse cross-references onto one canonical page per part, retire sold units. The largest gain is here.
- Then fix the manifest. Rebuild the tree to list only live canonical addresses with honest timestamps, split by rate of change.
- Then submit, in priority order. At 1,000 a day the sequence is a decision. New supersessions and current stock go before the archive.
- Then verify against demand. Check which indexed pages receive queries carrying part numbers, capacities or material specs. The rest is inventory.
How much of this you run yourself is a separate question from whether it gets done. AutoSEO at 149 USD a month per domain runs keyword discovery, backlink building and on-site suggestions alongside the Hub; FullSEO at 500 USD adds manual keyword selection, backlink placement against a target domain rating, and a human review step before on-site changes go live. Where a wrong canonical costs a product line, that review step earns its price. Stream, the assistant attached to My SEO takes URL and keyword lists in bulk.
Questions that come up first
How long should it take before a submitted page is found?
There is no guaranteed interval, and anyone quoting one is guessing. Establish your own baseline: submit a batch, record how many were visited at fourteen days and how many at thirty, then repeat. After three batches a slow one shows up as an exception rather than a suspicion.
We have 38,000 SKUs. Should all of them be submitted?
Rarely. Split by commercial value first: current stock numbers, active supersessions and anything with demand behind it go in the queue. Discontinued numbers with no replacement stay published without consuming budget. At 1,000 a day the full catalogue is thirty-eight days, most of them spent on pages nobody searches for.
Do we need the sitemap if we are submitting URLs directly?
Yes, for a different reason. Direct submission is a one-off notice about one address. The sitemap is the standing declaration of what the site holds, consulted on the engine's own schedule — and the only structure showing which section lags.
Our fleet listings sell before they are indexed. Is that solvable?
Partly. Submit units the day they are listed rather than in a monthly batch, and rebuild the fleet sitemap nightly so sold units leave it at once. That will not beat a two-week sales cycle every time. For fast-moving stock the category and capability pages usually matter more, since they last long enough to accumulate authority.
Does submitting the same URL repeatedly speed anything up?
No, and it spends budget a never-visited page could have used. If a URL was fetched and not retained, the constraint is the page. Resubmitting is re-sending a delivery note for a crate that was inspected and rejected.
The closing calculation, and a first week
Put the numbers together for a realistic Edmonton distributor: 38,000 part pages, 1,400 fleet listings, 340 capability pages and 210 service-area pages, or roughly 40,000 addresses. At 1,000 a day that is forty days of unbroken budget, before anything new is published.
Now apply a priority filter. Stock numbers with demand behind them, active supersessions, live fleet units, every capability page and the service areas that generate enquiries come to about 6,500 addresses — seven days rather than forty. The other 33,500 are archive: published, linked and not worth a slot until something changes.
What to do before submitting anything
Order matters more than volume: three of the five happen before the first batch goes out.
- Count the addresses your software generates. Crawl your own site and compare the total with the CMS count. The difference is what filters and cross-references produce unsupervised.
- Write the retirement rule. Decide what a sold unit and a discontinued number become — 410, 301 or replacement — and automate it in the job that publishes new ones.
- Rebuild the sitemap tree. One index, section indexes split by rate of change, address files small enough to read.
- Submit the priority set, not the catalogue. A week of budget on the pages that precede a quote request; hold the rest.
- Read the batch at fourteen and thirty days. Record submitted, found and absent. That table is your baseline, and beats any benchmark.
None of this is glamorous; it is closer to inventory control than to marketing, which is why it gets skipped where it matters most. A specifier searching an exact part number is the highest-intent visitor a distributor will ever get, and that visit depends on whether a crawler was told the page existed.
To see your own numbers rather than these, open an account on the Semalt dashboard and register the domain: within a day the Hub is reporting submitted, found and failed counts against your live catalogue, and that first set of counters usually ends the argument about which mechanism is at fault.
