ListCentralBot

ListCentral lists car dealerships' vehicles on Facebook Marketplace. To keep those listings in step with what is actually on a dealership's lot, ListCentralBot reads the dealership's inventory from the dealership's own website. That way new arrivals get listed, price changes are carried over and sold vehicles come down.

It visits a dealership's website when ListCentral updates that dealership's inventory, usually once a day. It reads vehicle listings only.

Every request ListCentralBot makes carries this User-Agent:

ListCentralBot/1.0 (+https://listcentral.ai/listcentralbot)

Pages

• robots.txt, first, every time.
• The homepage: for the dealership's name, address and phone number, and for the links to its inventory.
• The pages that list the vehicles for sale, and the pages after them.
• The inventory data those pages load, when they load it separately from the page.
• A vehicle's own page, when the list leaves out its price, mileage or model.
• The sitemap's vehicle pages, when the inventory pages show no vehicles without JavaScript.

Information

It keeps what a vehicle listing shows:

• VIN, year, make, model and trim
• Price and mileage
• Colors, stock number and specifications
• The address of the vehicle's page and the links to its photos
• The dealership's public name, address and phone number

What it doesn't do

• It doesn't read pages your robots.txt disallows.
• It doesn't run JavaScript.
• It doesn't fill in or submit forms, log in, or create accounts.
• It doesn't collect anything about your site's visitors or customers.

• One request at a time. It never asks your site for more than one page at once, and waits at least 3 seconds between requests. If your Crawl-delay asks for longer, it waits longer.
• Only what changed. It re-reads a vehicle's page only when the vehicle is new, when its price or mileage has changed, or when 14 days have passed since it last read it.
• It takes no for an answer. If your site refuses it (a 403, a 429 or a bot challenge), it stops asking for the rest of that visit. It doesn't retry the request, and it doesn't try to get around the block: no disguised User-Agent, no browser and no other IP addresses.
• Plain requests. It fetches pages the way a feed reader would, without loading images, scripts or stylesheets.

ListCentralBot follows your robots.txt (RFC 9309). It obeys rules written for ListCentralBot, or your User-agent: * rules if none name it.

User-agent: ListCentralBot
Allow: /
Crawl-delay: 10

User-agent: ListCentralBot
Disallow: /directory/

User-agent: ListCentralBot
Disallow: /

A missing robots.txt (404) means everything is allowed. A robots.txt your server fails to serve (a 5xx error) means stay out entirely, until it is served again.

If a firewall, CDN or bot-management service in front of your site blocks ListCentralBot and you want it to reach your inventory, you can allow it by its User-Agent: ListCentralBot. You can allow it by IP address once its addresses are listed above. Robots.txt still applies once it gets through.

If you run a dealer website platform and want to talk about how ListCentralBot reaches the dealerships on it, email us.

To report a problem, ask it to slow down or stay away, or ask a question about it, email us. Include your domain, and the time of the requests if it's about specific ones.