Seven Chinese platforms share one collection interface
MediaCrawler gathers public information from Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, and Zhihu. Each platform is documented as supporting keyword search, individual post collection, nested comments, creator pages, saved login state, proxies, and comment word clouds. Results can go to CSV, JSON, JSONL, Excel, SQLite, or MySQL. This is useful breadth for research that would otherwise require several unrelated scripts.
The project is Chinese-first. Its main README and configuration comments are Chinese, while README_en.md provides an English setup and feature translation. English-speaking users can get through installation, but diagnosing a platform-specific failure may still lead into Chinese issue reports, comments, and code labels. That matters because social crawlers need regular repair when login flows, signatures, and response shapes change.
Browser state replaces signature reverse engineering
Instead of reproducing every private signing algorithm in standalone code, MediaCrawler keeps a real browser context logged in and evaluates JavaScript there to obtain request parameters. The current Chinese README recommends connecting to an existing Chrome 144 or newer session through the Chrome DevTools Protocol. Users enable remote debugging at 127.0.0.1:9222, start the crawler, and accept Chrome's connection prompt within 60 seconds.
That approach reuses cookies, extensions, and a familiar browser profile. It also joins collection to an interactive user session. QR login may be required, expired cookies need attention, and a platform can still classify automated behavior. Open issue 915 reports Xiaohongshu warning a Windows 10 user about AI-like activity while CDP mode was enabled. The report does not establish whether the crawler, VPN, account, or another signal triggered the warning. It does show that using a real browser is not immunity from platform defenses.
What happened when we ran it
Our fresh Node 22 sandbox installed 269 npm packages in 26 seconds, consuming 233 MB on disk. Npm audit reported 0 known vulnerabilities across critical, high, moderate, and low categories. The commit d6f7c5b checkout contained 329 files, about 32,223 lines of source, and occupied 25.5 MB. We found 1 CI workflow and a tests directory, but no Dockerfile.
There was no npm build script or target, so our harness skipped the build. There was also no npm test script or target, so it skipped tests. Those are findings about the measured Node package surface, not proof that the Python crawler runs correctly. We did not log into any of the 7 supported services, install its Python environment, or collect user data in the unprivileged sandbox with no secrets. A successful dependency install cannot stand in for an authenticated crawl.
Python, Node, Chrome, and a login all enter the setup
The recommended crawler path uses uv to install Python packages. Node.js 16 or newer is also required, and standard Playwright mode adds a browser-driver download. The documented CDP route can skip that driver install by attaching to local Chrome. Commands select a platform, login method, and search or detail mode; comment collection is disabled by default and must be enabled in config/base_config.py.
The optional WebUI introduces two processes during development: a Uvicorn API on port 8080 and Vite on port 5173. Building the frontend lets the API serve its static files from one address. This is convenient for manual runs, but the README does not describe an unattended production service, a container image, or credential rotation. Proxy settings, crawl limits, data retention, and account access are left to the operator.
The license blocks the obvious commercial use
The repository does not use a standard open-source license. Its Non-Commercial Learning License 1.1 grants use, copying, modification, and merging for non-commercial learning. It prohibits commercial use without written permission and also bars large-scale crawling or activity that disrupts platform operations. The README repeats the learning-only and non-commercial limits, then points readers to a paid Pro offering with multi-account and resume support.
That restriction changes the buying decision more than the 63,851 GitHub stars do. A company cannot treat this as a free ingestion component merely because the code is public. Researchers must still consider target-site terms, privacy, copyright, and local law. The repository's disclaimer does not grant permission from Xiaohongshu, Douyin, or any person whose content appears in a collected dataset.
Open reports show where platform drift appears
Issue 843 reports incomplete Kuaishou comment collection and a REST API V2 error during a configured 500-post crawl. Issue 954 says Kuaishou detail mode did not download media even with the media setting enabled. Issue 866 reports a 406 response while collecting Xiaohongshu comment replies. These reports do not prove every user is affected, but each concerns a capability listed in the feature table.
GitHub recorded the last push on August 14, 2026, while issue 915 received activity on August 27. The repository had 191 combined issues and pull requests, and GitHub returned no latest release. Code and discussion are active, yet there is no release tag to pin as the latest supported checkpoint. For a permitted research job, record the exact commit, collect a small sample first, and verify missing comments or media before trusting totals.