NaijaPressBot
The crawler that reads public news feeds and article pages for NaijaPress, a Nigerian news API.
How to recognise it
Requests from NaijaPressBot send this user agent:
NaijaPressBot/1.0 (+https://naijapress.com/bot)
They come from the IP address 161.97.107.137.
What it fetches
- RSS and Atom feeds of the publishers NaijaPress follows, typically every 30 to 60 minutes per feed. Unchanged feeds are skipped using standard caching headers.
- Article pages linked from those feeds, once per article, to read the story text and the article photo address that the page declares for link previews.
- robots.txt, about once a day per site.
It does not download images, follow links beyond the articles in a feed, submit forms or sign in.
How often
- At most one article-page request to a site every 10 seconds, or slower if your robots.txt sets a
Crawl-delay(honoured up to 60 seconds). - If a site answers with a bot challenge or HTTP 401, 403 or 429, NaijaPressBot stops requesting its article pages for six hours.
- Feeds that keep failing are checked less and less often, down to once a day.
How NaijaPress uses your content
Article text is used to produce metadata for the NaijaPress API, such as the category, the people and organisations mentioned, a short attributed summary and an article photo link. It is not kept afterwards unless the publisher has agreed. API results always credit and link to the original article on your website. Full text is not redistributed unless a publisher has licensed it.
How to control it
NaijaPressBot follows robots.txt. To stop it entirely, add:
User-agent: NaijaPressBot Disallow: /
To slow it down, for example to one request every 30 seconds:
User-agent: NaijaPressBot Crawl-delay: 30
Changes are picked up within a day.
Contact
To ask us to stop, change how often we visit, correct how your publication appears, or discuss licensing, email [email protected].