Zhihu

Overview

The Zhihu Daily Spider-Man project focuses on creating a web scraper using a sophisticated technology stack. Built with Node.js, Vue.js, and MongoDB, this project delivers an efficient and streamlined way to fetch, analyze, and display news content from the Zhihu Daily platform. The combination of backend and frontend technologies ensures a responsive user experience while managing data effectively.

By utilizing Express for web services and MongoDB for data storage, the project allows for an organized structure that is easy to manage and scale. The frontend is designed as a single-page application using Vue.js, providing users with fast and interactive access to the latest news and historical data.

Features

• Node.js Backend: Built on Node.js with Express, ensuring a robust server-side environment for handling requests efficiently.
• MongoDB Storage: Utilizes MongoDB to store scraped data, allowing for quick data retrieval and management.
• Vue.js Frontend: A single-page application using Vue 2.0, providing a smooth and dynamic user experience.
• Custom Webpack Configuration: Separate webpack configurations for building frontend code, enhancing organization and minimizing conflicts.
• Scheduled Tasks: Supports regular scraping through scheduled tasks, enabling auto-updates of news content daily.
• Data Analysis with Node-Jieba: Implements Chinese word segmentation to analyze tags in the scraped content for better data search and categorization.
• User Authentication: Configurable user authentication mechanism using API keys to secure access to the scraper’s functionalities.
• Comprehensive API Access: Offers various endpoints to fetch latest news, historical data, article details, and user interactions such as comments and likes.