Why I don’t scrape, use an API or rely on AI

Every so often someone asks why I don't just scrape another database, hook into an API, or get AI to fill the site in over a weekend. It's a fair question. It would be a lot faster. This post explains why I don't.

Everything is entered by hand

Every title, release, company and credit on Omni Games is added manually. That's slow. I expect to keep adding games for the next two or three years, and the counts on the site reflect how far I've got, not how big each platform's library really is.

That's a trade I'm happy with.

Other sites did the work

Every games database is built from hours of someone else's effort. Scraping that and republishing it here would make Omni Games a copy of other sites rather than a site of its own. Most of them don't allow it anyway, and I don't want this project to depend on breaking anyone's terms.

I'm not tied to one source

Because nothing is pulled in automatically, I can use whatever resource is best for the job. That might be a box or manual, an official site, a magazine, a publisher's catalogue, or several databases checked against each other. Some sources are strong on one platform and weak on another. If a better source turns up, I can start using it straight away without rewriting an import.

A scraper or API locks you into one source's coverage, one source's mistakes and one source's way of doing things.

Their data doesn't fit my structure

Omni Games tracks things at release level. Each regional release, re-release, demo, DLC and update is its own record, linked back to one game group. Most outside sources don't model things that way. Pulling their data in would mean either flattening my structure to match theirs, or cleaning up every imported row by hand. If I'm going to check every row anyway, I'd rather enter it properly the first time.

I know where every entry came from

When something on the site is wrong, it's my mistake and I can fix it. With scraped data, an error from the source spreads into the catalogue and nobody can tell where it came from. Entering it myself means I've looked at every record at least once.

APIs change and disappear

APIs get rate-limited, become paid, change format or shut down. If the site relied on one, a decision made elsewhere could break it overnight. Data entered by hand stays here no matter what other sites do.

What about AI?

AI is only as good as the data it's been trained on. For games, that data largely comes from the same databases, wikis and forums I'd be scraping, mistakes included. It also fills in gaps with answers that sound right but aren't: release dates, product codes, regional titles and publishers that never existed. Those are hard to spot, because they look exactly like real data.

Using AI to generate entries would bring every problem with scraping, plus some new ones, and remove any way of tracing where an error came from. Nothing in the catalogue is generated by AI.

What about the DATs?

The dump data does come from outside: No-Intro, Redump and the other DAT groups. That isn't scraping. These are files the groups publish for people to use. I import them as they're released (the weekly DAT update posts cover each run), and they're kept separate from the hand-entered game data. The work is in linking each dump to the right release, and that part is done by hand too.

The downside

It's slow, and there are gaps. Some platforms are well covered, others barely started. If you spot something missing or wrong, community notes or r/OmniGamesInfo are the best way to let me know.