IT and Computers

Tips, tricks and blogs on Computers

Scrapy is an extremely efficient framework to write web spiders.  With just 30 lines of codes, you can write a crawler that can go through each link in your website and report each link , with HTTP status and the link from where it is linked. I use this spider to detect broken links on …

Continue reading Crawl your site and find all broken links using Scrapy