acme and you get every indexed listing that names Acme.
The search is one synchronous GET. There is no job to start and nothing to poll. Results come back a page at a time, and you fetch the next page with a cursor.
DarknetSearch crawls each broker site on its own schedule and stores every listing it finds. Marketplaces that only answer a per-query search, such as Russian Market, cannot be crawled in bulk and are not in this index. Search them live with Live Data Broker Search.
Availability: every plan with a license in force.
Cost: no metered allowance. Each successful call counts as one call toward your API Calls allowance.
When your organization’s license is not active, the three listing endpoints return 403:
How it differs from other searches
Search the listings
Calldata_broker_search with your term in q.
items: the listings on this page. What comes back describes each field.total: the number of listings that match. It is counted on the first page only and isnullon every later page.next_cursor: pass it ascursorto get the next page.nullmeans this is the last page.sort_by,sort_dir: the order the page was sorted in.applied_filters: the filters the search applied, includingqand how it was matched.
Write the query
q takes up to 500 characters. How a term matches depends on whether the query uses an operator.
Plain words. A query with no operators matches each word at the start of a word, case-insensitively. Every word must match. Only the first 10 words are used. The words are looked up in the listing’s name, seller, data type, country, state, city, numeric listing ID and detail fields.
acmematchesAcme Corp,acme.com,mail.acme.comandacme-login.com.acmedoes not match inside a word, as inMyAcme.- Punctuation is matched as typed.
acme.commatches only text that spells outacme.com.
In this mode a word matches only a whole word, unless you add
*. Words are split at ., _, /, :, @ and -, so acme.com matches acme followed by com.
A query whose operators do not form a valid expression, such as acme OR, returns 400 with {"detail": "Invalid search syntax: acme OR"}.
You can leave out q and search with filters alone. If a search cannot finish within 28 seconds, the API returns 400 and asks you to narrow it. Add a filter or a more specific term.
Narrow with filters
Every filter is a query parameter on the same call. Filters combine withq and with each other, so a listing must match all of them.
Exact filters compare the stored text, and sites write the same value in different ways. One site stores
US, another USA or United States. Take the values from data_broker_filters rather than guessing them.
Filter on detail fields
Each site records its own fields indetails, such as a website, an employer or a date of birth. details_filter takes a JSON object that maps a field name to a condition:
Field names may contain only letters, digits,
_ and -. A listing must meet every condition. Find the fields and sample values in details_fields from data_broker_filters.
Sort and page
sort_by takes relevance, scraped_at, listing_date, price, seller, country or name. sort_dir takes asc or desc, and defaults to desc.
- With
q, the default isrelevance. Withoutq, it isscraped_at, newest first.relevancewithoutqalso sorts byscraped_at. relevanceis always descending. For plain words, listings whose subject matches come first: the term is in the name, data type or seller, or in a subject field such as a website, domain, employer, category or bank name. Listings that mention the term elsewhere come next. Listings where the term matches a hosting provider or ISP, but not the subject, come last. Within each group, the most recently indexed listing comes first. For a query with operators, relevance is the full-text match rank.- Empty values sort last when you sort by any other field.
limit from 1 to 200.
1
Request the first page
Send
q, your filters, sort_by, sort_dir and limit. Read total here: later pages return null.2
Request the next page
Send the same parameters again, plus
cursor set to the next_cursor you received.3
Stop at the last page
Stop when
next_cursor is null.sort_by and the old cursor returns 400. Changing sort_dir does the same, except for relevance with q, which is always descending. With relevance, switching q between plain words and operators also returns 400. Start again from the first page. Keep q and the filters the same on every page, or the pages will not line up.
What comes back
Each site is crawled again on its own schedule, typically every 1 to 72 hours. A listing stays in the index after it leaves the site, and
is_available is never set to false. Treat scraped_at as the freshness signal.
To collect only listings added since your last run, pass the time of that run as scraped_since and sort by scraped_at:
Discover filter values
data_broker_filters returns the values you can filter on. Pass site_id or data_type to scope the counts to one site or one data type.
The counts describe the index, not the results of a search. They are precomputed, so they can lag new listings. Counts for
details_fields come from a sample and are approximate.
List the indexed sites
data_broker_sites returns the broker sites in the index. It takes no parameters.
id: pass it assite_idto search or count one site.name,display_name,url: the site’s identity.data_types: the kinds of data the site sells.status: the crawler’s state for the site:active,paused,onboarding,healing,failedordisabled.last_healthy_at: the last time the crawler reached the site.ai_description: a short description of the site.
status.
Errors
Handle API errors explains how to branch on each status.