A practical shortcut with a narrow legal lane
MediaCrawler solves the tedious part of collecting data from China's largest social platforms. Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, and Zhihu each have different login flows, request formats, signatures, and response shapes. This project wraps them behind one Python entry point and a shared configuration system. It can search by keyword, fetch specific posts, visit creator pages, collect nested comments, download media, and generate comment word clouds. Results can go to CSV, JSON, JSONL, Excel, SQLite, or MySQL.
The method is more pragmatic than elegant. MediaCrawler uses Playwright and, in its recommended mode, connects to a Chrome session that already holds your login state. JavaScript executed inside that browser obtains signed request parameters, so the collector does not have to reproduce every private signing algorithm. That makes the code accessible to a learner and keeps real authentication behavior in a real browser. It also means this is never just an anonymous HTTP scraper. Your browser profile, platform account, cookies, and interaction with anti-bot systems are part of the system.
Before considering the feature list, read the license. The repository does not use a standard permissive open-source license. Its Non-Commercial Learning License 1.1 permits use, copying, modification, and merging for noncommercial learning, while prohibiting commercial use without written consent and barring large-scale crawling or disruption. That is suitable for education and some research. It disqualifies the free repository from most commercial products, agencies, and internal business data pipelines.
Setup is clear, but not one step
The recommended dependency path is good. Install uv, install Node.js 16 or newer, clone the project, and run uv sync. The current Chinese README recommends CDP mode with Chrome 144 or newer. You enable remote debugging from Chrome's inspection page, verify that the local debugging server is listening, then accept Chrome's connection prompt when the crawler starts. Standard Playwright mode remains available and requires installing its browser driver.
Running a job is simple once that state exists. The CLI selects a platform, login method, and crawl type. Keyword lists, post IDs, output limits, and feature switches live in config/base_config.py. That last detail matters: comment collection is off by default, and configuration comments are mainly Chinese. A first-time user can easily get a smaller dataset than expected without noticing the switch.
The Web UI improves discoverability by exposing platforms, crawl types, logs, configuration, and data previews. During development it is not a single service. You run a FastAPI backend on one port, install the frontend packages, and run Vite on another. A production-style local build can be served by the API after an npm build. This is reasonable for contributors, but heavier than the friendly screenshots suggest.
Wide coverage does not guarantee complete output
MediaCrawler's strongest feature is breadth. A shared project covering seven platforms is far more convenient than maintaining seven unrelated scripts. Login-state caching, proxy-pool support, creator collection, nested replies, and several storage targets make it useful for exploratory studies. The browser approach can also survive changes that would break a hand-copied signature implementation.
Platform behavior is still a moving target. An August 2026 Bilibili report demonstrates a particularly dangerous failure: ordinary comments are saved, the process ends normally, but a pinned comment and its nested replies are omitted because the client reads the regular replies field and not the separate top field. A matching fix pull request is active, which is encouraging, but the lesson applies across connectors. Successful completion is not a completeness check.
Other open reports describe incomplete Kuaishou comment collection after a REST API error, Xiaohongshu replies returning 406, and Weibo detail-mode comment requests receiving an error page. These reports do not prove every installation fails, and platform defenses can depend on accounts, networks, and timing. They do prove that production-style acceptance tests are necessary. Keep a small set of posts whose visible counts and pinned threads are known, then compare collected records after every update.
Account safety deserves equal weight. CDP mode is promoted as a way to reuse Chrome state and reduce platform risk, not as a guarantee of invisibility. One open report shows Xiaohongshu warning a user that AI operation was detected while CDP mode was enabled. Anyone using a valued personal or business account should treat that as a real operational risk.
Documentation and project health
The main Chinese README is detailed and includes prerequisites, CDP steps, CLI examples, Web UI commands, output options, the paid Pro comparison, and a long legal disclaimer. English and Spanish versions exist, which is better than leaving international users to machine translation. The English file is not fully synchronized with the newer Chinese instructions, however. It describes older browser-install guidance where the Chinese README specifies current Chrome and remote-debugging requirements.
The repository was pushed on August 5, 2026, and issues and pull requests were still being updated on August 8. Its GitHub count of 187 combines issues and pull requests, so it is an active work queue rather than a count of confirmed defects. GitHub reports no latest release, but current commits and issue activity show ongoing work. With more than 60,000 stars, the project has major visibility, though popularity does not settle license fit or connector accuracy.
The buying decision
Choose MediaCrawler for learning, source study, or supervised noncommercial research where its ready-made platform adapters save weeks of initial work. Budget time for browser setup, Chinese configuration, account management, and record-level checks. Do not choose it for a commercial collector, an unattended compliance-sensitive pipeline, or a study where missing replies could invalidate the conclusion. For those cases, a permissively licensed framework such as Scrapy or Crawlee gives you less immediate platform coverage but a much sounder foundation.