A small, tested, no-frills parser of robots.txt files in Swift.
Currently used in
Full docs can be found here
let exampleRobotsFileLocation = URL(string: "https://www.bbc.co.uk/robots.txt")!
let robotsData = try! Data(contentsOf: exampleRobotsFileLocation)
let robotsText = String(data: robotsData, encoding: .utf8)!
let check = CanProceed.parse(robotsText)
let shouldProceed = check.agent("ChatGPT-User", canProceedTo: "/news"))
guard shouldProceed else {
complain()
return
}
...
check.sitemaps // contains any sitemaps
check.crawlDelay // provides any crawl delay setting
check.sitemps // contains any XML sitemaps
Copyright (c) 2024 Paul Tsochantaris. Licensed under the MIT License, see LICENSE for details.