Hi, I have read few comments under the post, there are great suggestions also your questions regarding task are on the point. But I believe handling this with a script might be not easy. If I were you, I would use Apache Nutch or similar open source software/library.I have used Nutch for my thesis for similar task that I had to scrap a lot of blog pages and the other pages they were referencing. You can configure... Source: over 1 year ago

I've never used it, but I was on a project where we considered Apache Nutch: https://nutch.apache.org/. Source: over 1 year ago

Point and click web browser plugin GUI: https://webscraper.io/. Source: about 1 year ago

In my 5+ years of experience as the scraper guy in the office, paying for these services could take a lot of money. So automated scraping might be your option. If you need help, tap me. Or you could use webscraper.io for easier nocode approach to it if you wanna do it yourself. Source: over 1 year ago

I don't know what corpus linguistic analysis is, but you can scrape the articles off of their website and analyse it in whichever software you're comfortable with. If you're not familiar with a programming language, you can use a GUI scraper like this one. Source: over 1 year ago

I'm looking into VPNs that have rotating IPs with time-set features. Didnt find any yet that I can try for free first. For the scraping Im using a free chrome browser extension from https://webscraper.io/. Source: over 1 year ago

For text only dbs a even a scraper addon would do. Try something like webscraper.io, it takes a bit of fucking around to get it working but it's foolproof. Source: over 1 year ago