Q8
What is the robots.txt file used for?
A.
Blocking all web scraping activities
B.
Allowing all web scraping activities
C.
Providing guidelines for web scraping
AnswerD.
Controlling browser rendering
Answer: Option C
Solution
Answer: Option C
Solution:
The robots.txt file provides guidelines for web crawlers, including what can be scraped.