An invite-only app has a discovery problem: nobody can see it. Our answer was to put the nights on the open web first. Every night gets a page. Every city gets a page, with pages underneath it for what people type: tonight, this weekend, free, run clubs, comedy. On 4 October that was 1,254 addresses in one sitemap, each with structured data a search engine can read.
What a page promises
- One address per night, forever. Other spellings redirect to it.
- The facts first, in our words, then the host's link. We are not the host and the page says so.
- A page is offered to search only when it has something to show. A kind page needs three nights. A night that is over asks not to be indexed. A city with nothing yet says so plainly.
- Every number is read live from the database. We do not pad.
Then a crawler found two faults
A site audit came back with 2,562 addresses and an ugly list: 131 timed out, 2,409 slow, 214 orphans, 195 pages linking to broken pages. I did not believe it, so I crawled our own site the way the auditor does, eight pages at a time. 1,265 of 1,335 pages answered with an error.
Both of our web servers had been killed for running out of memory. Underneath that was a smaller mistake with large consequences: the container image gave its files to one user and ran the server as another, so the server had never been able to write its page cache to disk. Every page was rendered from scratch on every visit, the cache lived in memory only, and a crawler reading a thousand pages pushed both servers over their 512 MiB limit. The auditor's slow pages, timeouts, orphans and broken links were one outage, seen from outside.
What changed
- The image owns its cache. The servers get 1 GiB, with the runtime told where the ceiling is so it tidies up before the kernel kills it.
- One address may make 30 requests a second. Beyond that it is told to come back later, instead of taking the site down for everyone.
- Every title fits a search result. Every description is 110 to 155 characters, facts first, cut at a word. Every page has a share picture.
- A city page now links every upcoming night, month by month, so no night in the sitemap is without a link from the site.
The honest part
None of this makes a search engine want to crawl us. On the day I write this, Google indexes 3 of our 14 hub pages and has never fetched the other 11, because a two-week-old domain with no inbound links is not worth its time yet. What the fixes do is remove every reason to skip us once it arrives. What brings it are links from sites it already trusts, requests by hand for the pages that matter, and weeks. This post is one of those links.
Next: THE LINE, and why we open a city by invitation instead of a launch.