scraperaiscraperai/scraperai
breaks web scraping into reusable skill modules so an Agent can pick a scraping strategy by page type and output structu…
No ratings yet
Rate this skill
Stars486
Forks67
Watchers486
Open Issues4
InstallOne-line install
UnconfirmedWhether this repository follows the Agent Skills spec (that is, whether it contains a SKILL.md) has not been verified yet. Until that is settled, clone it and use what you need:
git clone https://github.com/scraperai/scraperai.git
OverviewOverview
Key Features
breaks web scraping into reusable skill modules so an Agent can pick a scraping strategy by page type and output structured data. Key capabilities include: 1. Catalog listing-page scraping: identifies repeating elements on listing pages and batch-extracts fields for every record in one pass; 2. Details detail-page parsing: locates and extracts specified fields from a single record's detail page to fill gaps left by the listing page; 3. Captcha handling: triggers a handling flow when a page is blocked by a captcha so the scraping task does not stall; 4. xpath targeting: uses XPath expressions to pinpoint the exact elements to scrape on pages with unstable structure; 5. scroll loading: automatically scrolls infinite-scroll pages and keeps collecting newly loaded content; 6. urls link discovery: bulk-extracts and organizes target URLs from a page as an entry list for later scraping.
Use Cases
web data collection tasks that need batch collection from listing pages, field completion from detail pages, continued scraping on login or captcha pages, complete scraping of scroll-loaded pages, and link discovery before batch scraping.
Target Roles
Data Analyst / Data Engineer, AI / Algorithm Engineer, Backend Developer, Test / QA Engineer.
TopicsTopics
RatingUser ratings
Nobody has rated this skill yet. Yours would be the first on the site — your vote becomes the initial average.
RepositoryRepository
| Skill ID | 1879 |
|---|---|
| Data source | GitHub |
| Repository | scraperai/scraperai |
| Category | AI & Agent Skills / Data Collection & Web Parsing |
| Primary language | HTML |
| License | GNU General Public License v3.0 |
| Default branch | main |
| GitHub repository ID | 710781843 |
| Node ID | R_kgDOKl2rkw |
| Rank | #1294 (position within the source ranking) |
| Site-wide popularity rank | #1471 (computed live from Stars, across all 5833 entries) |
GrowthStar / Fork growth
One snapshot per day, tracking how Stars and Forks changed
- Now486
- Previous point · 09-27 484
- vs previous point +2
- 2 data points 14 days
- Now67
- Previous point · 09-27 67
- vs previous point ±0
- 2 data points 14 days
TimelineTimeline
Repository created2023-10-27 12:31:462023-10-27
Data last updated2026-10-26 12:11:5822 days ago
Last commit2025-09-18 19:02:562025-09-18
CommitLatest commit
Version [latest] committed by iakov-kaiumov. Update Category: Bug Fixes & Stability Improvements. Commit details: Merge pull request #6 from AnnaLisachenko/main. This update resolves known runtime errors and enhances system reliability.