Avoiding Web Scraping Detection

[anjalisoni3655]

Problem addressed:

• some websites do not approve of web scraping.
• These sites will implement ways to detect if you are using a web scraping tool such as Beautiful Soup. For example, a website can detect if a large number of requests were made in a short amount of time.

Solution added:

• In order to potentially avoid detection, we can randomize our request rate to closely mimic human interaction.
• randomly choosing a value from a list and waiting that selected amount of time before the loop begins again. - This will slow down our code but will help us avoid detection.