# Center on Budget and Policy Priorities (CBPP)

> The Center on Budget and Policy Priorities is a nonpartisan research and policy
> institute that analyzes federal and state budget and tax policies and their
> effects on people with low and moderate incomes. This file states our terms for
> automated and AI agent access to www.cbpp.org, and points crawlers at the
> complete, low-cost index of our content.

## Permitted use

- Retrieval in response to a specific user question is permitted, including
  quoting and summarizing with attribution and a link to the source page.
- Bulk collection of this site for general-purpose model training, fine-tuning,
  or dataset redistribution is **not** permitted.
- This matches the machine-readable signal in our robots.txt:
  `Content-Signal: search=yes, ai-train=no, ai-input=yes`.

## Do not request

- `/search`, `/es/search`, and any URL containing `cbpp_search_redirect` — these
  are user-facing search interfaces with unbounded query-string combinations.
  They are expensive to serve, contain no unique content, and are disallowed for
  all user agents in robots.txt. Requests to them may be blocked or rate limited.
- Faceted and paged listing variants such as `/blog?...` and `/es/blog?...`.
- `/cms/`, `/admin/`, and `/user/` paths.

## Finding our content

There is no need to crawl the site for links. Every published page is listed in
our sitemap:

- [Sitemap](https://www.cbpp.org/sitemap.xml): complete index of all current
  content, English and Spanish (`/es/`), including reports, blog posts, press
  releases, topic pages, and staff profiles.
- [Pre-2005 content archive](https://www.cbpp.org/pre-2005-content-archive):
  older material that predates our current content management system and is not
  in the sitemap. This is the only part of the site not covered above.

## Crawl behavior

- Please honor the `crawl-delay: 10` in robots.txt.
- Send a descriptive User-Agent with a URL or contact address.
- Respect `304 Not Modified` and `Last-Modified`; our published content
  changes infrequently and repeated full re-crawls are not necessary.
- Questions about access: communications@cbpp.org