Q17
What is the role of the robots.txt file in relation to web scraping?
A.
It blocks all web scraping activities
B.
It allows all web scraping activities
C.
It provides guidelines for web scraping
AnswerD.
It allows scraping any content
Answer: Option C
Solution
Answer: Option C
Solution:
The robots.txt file provides guidelines on which parts of a website can be scraped.