Scraping at scale: what 19k stars taught me about async APIs

2026-07-26 · #python #async #fastapi #scraping

I wrote the first version of Douyin_TikTok_Download_API to solve one annoyance: I wanted a Douyin video without the watermark baked into the corner. One script, one link, one clean MP4.

That script is now a FastAPI service with a PyPI package, a hosted demo, and more GitHub stars than I ever planned to maintain. Along the way it taught me more about async Python, anti-bot signatures, and the tax of a popular repo than any tutorial did.

Why async, and why it actually matters here

Scraping is almost pure I/O. You send a request, then you wait. You send another, then you wait again. The CPU is idle the whole time. A synchronous scraper handling ten links does them one after another — wait, wait, wait, ten times over.

# synchronous: ten round-trips, back to back
for url in links:
    data = fetch(url)   # blocks until the network answers
    results.append(parse(data))

The same work with asyncio fires all ten requests, then collects the answers as they land. The wall-clock time collapses from the sum of the waits to the longest single wait.

# asynchronous: ten round-trips in flight at once
async def scrape(links):
    return await asyncio.gather(*(fetch(url) for url in links))

That is the whole reason the project is built async top to bottom. FastAPI gives you async request handlers for free, httpx gives you an async client, and the batch-parsing endpoint — hand it a list of links, get back a list of results — is just asyncio.gather wearing a REST interface.

The lesson that took longest to sink in: async does nothing for CPU-bound work. Parsing a giant response, decoding, hashing — that still blocks the event loop. Anything heavy goes in a thread or a process pool, or it quietly ruins your throughput for every other in-flight request.

The real hard part isn’t the API. It’s the signature.

Anyone can write a GET. The reason a scraper for a platform like Douyin is a project and not a one-liner is that the endpoints expect a signed request. Miss the signature, or get it slightly wrong, and you get an empty body or a soft ban instead of the data.

Those signatures are generated by obfuscated JavaScript running in the page. So the work is never really “call the API.” It is:

read the minified JS that builds the signature parameter,
understand the inputs (URL, timestamp, a rolling bit of state),
reimplement that function in Python so every request is signed the same way the browser would sign it.

And then they change it. That is the actual job of maintaining a scraper: it is not code, it is a standing subscription to someone else’s cat-and- mouse game. The endpoints move, the signature algorithm gets a new twist, a field gets renamed. A scraper is never done; it is only currently working.

What a popular repo actually costs

Nobody warns you that stars come with an inbox.

Every platform change is now your outage. When the signature algorithm shifts, hundreds of people find out at the same time, and they all open the same issue.
“It doesn’t work” is most of the queue. Half of debugging a public scraper is asking for the input link, because the failure is almost always one specific URL shape you never handled.
Rate limiting is a feature you owe your users, not an afterthought. A hosted demo with no throttle is a free proxy for abuse and a fast way to get your IP range blocked.

None of that is glamorous. All of it is the reason the project survived long enough to be useful.

Takeaways, if you’re building one of these

Go async only where you’re waiting on the network. Push CPU work off the event loop or it will stall every concurrent request.
Treat the signature as the product. The endpoint is trivial; the thing that keeps working is your reimplementation of their signing logic. Isolate it so you can swap it fast when it changes.
Design the batch endpoint first. One link is a demo. A list of links, parsed concurrently, is the thing people actually deploy.
Ship rate limiting on day one. Not for politeness — for survival.

The version I keep coming back to: a scraper’s uptime is not a function of your code quality. It is a function of how fast you can re-read someone else’s obfuscated JavaScript at 2 a.m. Build for that, not for the happy path.