Script Planner Assistant▸ 
AI Apps 
AI News 
MediaCrawler: Open-Source Multi-Platform Social Media Data Scraper
MediaCrawler: Open-Source Multi-Platform Social Media Data ScraperIntroduction
MediaCrawler is an open-source social media crawler tool mainly used to automatically scrape public content from mainstream social platforms, such as Xiaohongshu, Douyin, Kuaishou, Weibo, and Bilibili. The content it can scrape includes:
Posts, videos, and image-text posts published by users.
Video files (MP4) and images (JPG/PNG).
Text content, such as titles, descriptions, and comments.
User information, such as nickname, ID, and follower count.
MediaCrawler Core Capabilities
Two crawling methods. You can search by keyword, or precisely target posts or video IDs, and even capture first-level and second-level comments.
Login state management. Supports QR code or Cookie login, and can store the login state so you don't have to scan the code repeatedly.
Anti-scraping protection. Uses Playwright to simulate a real browser, automatically runs JS to obtain signature parameters, eliminating the need for complex JS reverse engineering. Also adds an IP proxy pool and handles slider CAPTCHA.
Data output. Supports CSV, JSON, SQLite, MySQL, and other formats, making it convenient for later analysis or database storage.
Visualization plugin. One click to generate a comment word cloud and quickly spot trending topics.
Resumable crawling and multi-account (Pro version). The Pro version allows switching between multiple accounts, supports Linux daemon processes and Docker deployment, and the code is modular, making it suitable for large-scale or enterprise use, and easy for secondary development.
MediaCrawler Technical Principles
Uses Playwright to launch a real browser, retains login states like Cookie and LocalStorage after login.
Runs the platform's own JS code directly in the browser to obtain request signature parameters, then sends them to the backend API. This eliminates the need to reverse-engineer encryption algorithms, saving a lot of effort.
MediaCrawler Use Cases
Content operations or competitor research. Batch scrape competitors' videos and comments, generate word clouds, and identify trending topics.
Data analysis and sentiment analysis. Store comment data for use in sentiment models and to observe topic changes.
Market monitoring. Watch the real-time response of marketing campaigns on Weibo and Douyin.
Academic public opinion research. Use public social data to assist in public opinion or communication studies analysis.
MediaCrawler Quick Start
1. Clone the repository
git clone https://github.com/NanmiCoder/MediaCrawler.git cd MediaCrawler
2. Install dependencies
python -m venv venv && source venv/bin/activate # Windows3;indows use Scripts\activate pip install -r requirements.txt playwright install
3. Scan QR code to log in and start scraping
python main.py --platform xhs --lt qrcode --type search
Enter keywords as prompted, and data is saved to the data/ directory by default.
Legal and Compliance Reminder
The author has always stated that it can only be used for learning and research, not for commercial use, and large-scale illegal scraping is prohibited. By using it, you agree to the disclaimer.
If you need stronger concurrency, enterprise-level maintenance, or a desktop video downloader, you can check out the author's paid version, MediaCrawlerPro.
In summary, MediaCrawler uses browser automation combined with a no-JS-reverse-engineering approach, significantly lowering the barrier to Chinese social media data collection, making it suitable for operations staff, data analysts, and researchers to quickly test ideas.
Open-source address:https://github.com/NanmiCoder/MediaCrawler
