Quote:
Originally Posted by Klen
By scrapping , you mean downloading videos?
|
Not exclusively that, but yes. The issue is if you make too many page requests in too short a period of time cloudflare throws that bot verification page up instead of the target page. When you are downloading videos, it actually works a bit better because you are throttled by the fact you have to download the video - but you still run into issues in general. Between cloudflare and the fact that most sites now have content on CDNs and have speed limits set on those CDNs to around 150-200k, downloading videos in bulk from the big tube sites is (disregarding any legal aspects of it) technologically inviable.