Skip to main content
Open Cloud Storage searches an index of publicly listable cloud storage buckets on Amazon S3, Google Cloud Storage and Azure. You give it a term such as acme. It returns the files whose name matches, with a direct link, the bucket that holds each file, its size and when it was last modified. In the API reference these endpoints sit under the Bucket Monitoring tag. In the dashboard the source is Search by Source → Open Cloud Storage. The flow is asynchronous: start a search, poll it until it finishes, then read the files.

Availability and cost

Availability: the Cloud Storage Exposure plan feature. Not every plan includes it. Manage → Buy more in the dashboard lists the features your plan includes. Plans and quotas explains how plan features work. Cost: no metered allowance. Calls count toward your organization’s per-second rate limit, and every 2xx response to an API-key request uses one unit of your monthly API Calls allowance. A non-2xx response is refunded. Without the feature, or when your license is not active, the four search endpoints (create_, get_, list_ and delete_bucket_search_request) return 403:
Cloud Storage Exposure has no allowance and no capacity slots, so these endpoints never return the allowance 429 or the capacity 409. A 429 with "message": "Rate limit exceeded" comes from the rate limit or the API Calls allowance. Rate limits explains both.

What a search matches

  • The term is matched against file names. Bare words must all appear. A word with a - prefix excludes files that contain it: acme invoice -draft finds files whose name contains acme and invoice but not draft. Set fullpath to match the whole file path instead of the name. Set regexp to treat the term as a regular expression.
  • Domain terms are shortened first. The search drops the top-level domain and the www., http://, https://, ftp:// and mailto: prefixes, so acme.com is searched as acme. If the shortened term reaches a fifth page of results, the search runs again with the term you sent and keeps those files instead. search_term in the search response shows the term the search used.
  • A search collects at most 5,000 files. It reads up to 5 pages of 1,000 files from the index. When the search reaches a fifth page, has_more_result_than_limit is true and more files may match than the search kept.
  • Submit filters decide what fills the cap. extensions, stopextensions, size_from, size_to, last_modified_from, last_modified_to, bucket and excluded_buckets narrow the files before they are collected. order and direction decide which files come first. The read filters on list_bucket_search_result only slice the files a search already kept.
  • A search runs once. It does not refresh. Start a new search to pick up files indexed since, or create a monitor.
1

Start the search

create_bucket_search_request takes a JSON body. Only search_term is required.
The call returns 201 with the search:
Keep the id. is_monitoring, alert_cycle, is_email, is_reporting and report_id are scheduling fields that a search you start here leaves at their defaults.
2

Poll until it finishes

Fetch the search with get_bucket_search_request until status is finished.
status takes two values:
  • in_progress: still running.
  • finished: done. Read the files.
There is no failed state. If the index cannot be reached, the search still ends finished, with no files. Poll every 10 seconds or slower: each successful poll uses an API call.When the search finishes, check has_more_result_than_limit and search_term. true means the search stopped at 5,000 files. A search_term that differs from what you sent means the shortened domain term was used.
3

Read the files

list_bucket_search_result returns the files of one search. Pass the search id as searchrequest.
file_category groups files by extension:An unknown file_category returns no files rather than an error.

What comes back

The envelope:
  • count: files that match the read filters you sent, across all pages.
  • total_count: every file the search kept, ignoring search, file_category and size_range.
  • available_categories: the categories present among all the search’s files. Use it to offer only filters that return data.
  • next / previous: null at either end. The URLs point at the internal search service, not at client-api.leak.center, so don’t call them. Raise page until next is null.
Each file:

Manage searches and files

Watch the term continuously

A search runs once. To re-run a term on a schedule and receive only new files, create a monitor on the bucket source. See Create a monitoring rule. A monitor on this source needs the same Cloud Storage Exposure plan feature. Quick search also runs this source for domain, IP address, host name and file name searches. On a plan without the feature, quick search reports this source as not_entitled.

Errors

Errors lists every error the API returns.

Endpoints