Skip to main content
Data Brokers searches an index of listings that data brokers advertise for sale on marketplace sites. Each listing is one offer: a record set, an account or a batch of personal or corporate data, with its site, seller, price and listing date. Search for acme and you get every indexed listing that names Acme. The search is one synchronous GET. There is no job to start and nothing to poll. Results come back a page at a time, and you fetch the next page with a cursor. DarknetSearch crawls each broker site on its own schedule and stores every listing it finds. Marketplaces that only answer a per-query search, such as Russian Market, cannot be crawled in bulk and are not in this index. Search them live with Live Data Broker Search. Availability: every plan with a license in force. Cost: no metered allowance. Each successful call counts as one call toward your API Calls allowance. When your organization’s license is not active, the three listing endpoints return 403:

How it differs from other searches

Search the listings

Call data_broker_search with your term in q.
  • items: the listings on this page. What comes back describes each field.
  • total: the number of listings that match. It is counted on the first page only and is null on every later page.
  • next_cursor: pass it as cursor to get the next page. null means this is the last page.
  • sort_by, sort_dir: the order the page was sorted in.
  • applied_filters: the filters the search applied, including q and how it was matched.

Write the query

q takes up to 500 characters. How a term matches depends on whether the query uses an operator. Plain words. A query with no operators matches each word at the start of a word, case-insensitively. Every word must match. Only the first 10 words are used. The words are looked up in the listing’s name, seller, data type, country, state, city, numeric listing ID and detail fields.
  • acme matches Acme Corp, acme.com, mail.acme.com and acme-login.com.
  • acme does not match inside a word, as in MyAcme.
  • Punctuation is matched as typed. acme.com matches only text that spells out acme.com.
Search the organization’s name, not its domain. acme finds listings that spell out acme.com as well as listings that only say Acme Corp. acme.com finds only the first kind.
Operators. Add any of these and the query switches to whole-word matching: In this mode a word matches only a whole word, unless you add *. Words are split at ., _, /, :, @ and -, so acme.com matches acme followed by com. A query whose operators do not form a valid expression, such as acme OR, returns 400 with {"detail": "Invalid search syntax: acme OR"}. You can leave out q and search with filters alone. If a search cannot finish within 28 seconds, the API returns 400 and asks you to narrow it. Add a filter or a more specific term.

Narrow with filters

Every filter is a query parameter on the same call. Filters combine with q and with each other, so a listing must match all of them. Exact filters compare the stored text, and sites write the same value in different ways. One site stores US, another USA or United States. Take the values from data_broker_filters rather than guessing them.

Filter on detail fields

Each site records its own fields in details, such as a website, an employer or a date of birth. details_filter takes a JSON object that maps a field name to a condition:
Field names may contain only letters, digits, _ and -. A listing must meet every condition. Find the fields and sample values in details_fields from data_broker_filters.

Sort and page

sort_by takes relevance, scraped_at, listing_date, price, seller, country or name. sort_dir takes asc or desc, and defaults to desc.
  • With q, the default is relevance. Without q, it is scraped_at, newest first. relevance without q also sorts by scraped_at.
  • relevance is always descending. For plain words, listings whose subject matches come first: the term is in the name, data type or seller, or in a subject field such as a website, domain, employer, category or bank name. Listings that mention the term elsewhere come next. Listings where the term matches a hosting provider or ISP, but not the subject, come last. Within each group, the most recently indexed listing comes first. For a query with operators, relevance is the full-text match rank.
  • Empty values sort last when you sort by any other field.
Results come back 50 at a time. Set limit from 1 to 200.
1

Request the first page

Send q, your filters, sort_by, sort_dir and limit. Read total here: later pages return null.
2

Request the next page

Send the same parameters again, plus cursor set to the next_cursor you received.
3

Stop at the last page

Stop when next_cursor is null.
A cursor belongs to the sort it was issued for. Change sort_by and the old cursor returns 400. Changing sort_dir does the same, except for relevance with q, which is always descending. With relevance, switching q between plain words and operators also returns 400. Start again from the first page. Keep q and the filters the same on every page, or the pages will not line up.

What comes back

Each site is crawled again on its own schedule, typically every 1 to 72 hours. A listing stays in the index after it leaves the site, and is_available is never set to false. Treat scraped_at as the freshness signal. To collect only listings added since your last run, pass the time of that run as scraped_since and sort by scraped_at:

Discover filter values

data_broker_filters returns the values you can filter on. Pass site_id or data_type to scope the counts to one site or one data type.
The counts describe the index, not the results of a search. They are precomputed, so they can lag new listings. Counts for details_fields come from a sample and are approximate.

List the indexed sites

data_broker_sites returns the broker sites in the index. It takes no parameters.
Each site has:
  • id: pass it as site_id to search or count one site.
  • name, display_name, url: the site’s identity.
  • data_types: the kinds of data the site sells.
  • status: the crawler’s state for the site: active, paused, onboarding, healing, failed or disabled.
  • last_healthy_at: the last time the crawler reached the site.
  • ai_description: a short description of the site.
Listings stay searchable whatever the site’s status.

Errors

Handle API errors explains how to branch on each status.

Endpoints