1061 Llms 65cf5660

Apify Documentation

The entire content of Apify documentation is available in a single Markdown file at https://docs.apify.com/llms-full.txt

Apify API

• Apify API
• Apify API: The Apify API (version 2) provides programmatic access to the Apify platform.
  • Abort build: :::caution deprecated This endpoint has been deprecated and may be replaced or removed in future versions of the API.
  • Get default build: Clients Python JavaScriptGet the default build for an Actor.
  • Get build: :::caution deprecated API endpoints related to build of the Actor were moved under new namespace actor-builds.
  • Get list of builds: Clients Python JavaScriptGets the list of builds of a specific Actor.
  • Build Actor: Clients Python JavaScriptBuilds an Actor.
  • Delete Actor: Clients JavaScriptDeletes an Actor.
  • Get Actor: Clients Python JavaScriptGets an object that contains all the details about a specific Actor.
  • Get OpenAPI definition: Get the OpenAPI definition for Actor builds.
  • Update Actor: Clients Python JavaScriptUpdates settings of an Actor using values specified by an Actor object passed as JSON in the POST payload.
  • Abort run: :::caution deprecated This endpoint has been deprecated and may be replaced or removed in future versions of the API.
  • Get run: :::caution deprecated This endpoint has been deprecated and may be replaced or removed in future versions of the API.
  • Metamorph run: :::caution deprecated This endpoint has been deprecated and may be replaced or removed in future versions of the API.
  • Resurrect run: [DEPRECATED] API endpoints related to run of the Actor were moved under new namespace actor-runs.Resurrects a finished Actor run and returns an object that contains all the details about the resurrected run.
  • Without input: Runs a specific Actor and returns its output.
  • Run Actor synchronously without input and get dataset items: Runs a specific Actor and returns its dataset items.
  • Run Actor synchronously with input and get dataset items: Runs a specific Actor and returns its dataset items.
  • Run Actor synchronously with input and return output: Runs a specific Actor and returns its output.
  • Get list of runs: Clients Python JavaScriptGets the list of runs of a specific Actor.
  • Get last run: This is not a single endpoint, but an entire group of endpoints that lets you to retrieve and manage the last run of given Actor or any of its default storages.
  • Run Actor: Clients Python JavaScriptRuns an Actor and immediately returns without waiting for the run to finish.
  • Delete version: Deletes a specific version of Actor's source code.
  • Delete environment variable: Deletes a specific environment variable.
  • Get environment variable: Clients PythonGets a EnvVar object that contains all the details about a specific environment variable of an Actor.
  • Update environment variable: Clients PythonUpdates Actor environment variable using values specified by a EnvVar object passed as JSON in the POST payload.
  • Get list of environment variables: Clients PythonGets the list of environment variables for a specific version of an Actor.
  • Create environment variable: Clients PythonCreates an environment variable of an Actor using values specified in a EnvVar object passed as JSON in the POST payload.
  • Get version: Clients PythonGets a Version object that contains all the details about a specific version of an Actor.
  • Update version: Clients PythonUpdates Actor version using values specified by a Version object passed as JSON in the POST payload.
  • Get list of versions: Clients PythonGets the list of versions of a specific Actor.
  • Create version: Clients PythonCreates a version of an Actor using values specified in a Version object passed as JSON in the POST payload.
  • Get list of webhooks: Gets the list of webhooks of a specific Actor.
  • Abort build: Clients Python JavaScriptAborts an Actor build and returns an object that contains all the details about the build.
  • Delete build: Clients JavaScriptDelete the build.
  • Get build: Clients Python JavaScriptGets an object that contains all the details about a specific build of an Actor.
  • Get log: Check out Logs for full reference.
  • Get OpenAPI definition: Clients Python JavaScriptGet the OpenAPI definition for Actor builds.
  • Actor builds - Introduction: The API endpoints described in this section enable you to manage, and delete Apify Actor builds.
  • Get user builds list: Gets a list of all builds for a user.
  • Abort run: Clients Python JavaScriptAborts an Actor run and returns an object that contains all the details about the run.
  • Delete run: Clients JavaScriptDelete the run.
  • Get run: This is not a single endpoint, but an entire group of endpoints that lets you retrieve the run or any of its default storages.
  • Metamorph run: Clients Python JavaScriptTransforms an Actor run into a run of another Actor with a new input.
  • Update status message: You can set a single status message on your run that will be displayed in the Apify Console UI.
  • Reboot run: Clients Python JavaScriptReboots an Actor run and returns an object that contains all the details about the rebooted run.
  • Actor runs - Introduction: The API endpoints described in this section enable you to manage, and delete Apify Actor runs.
  • Get user runs list: Gets a list of all runs for a user.
  • Delete task: Clients JavaScriptDelete the task specified through the actorTaskId parameter.
  • Get task: Clients Python JavaScriptGet an object that contains all the details about a task.
  • Get task input: Clients Python JavaScriptReturns the input of a given task.
  • Update task input: Clients Python JavaScriptUpdates the input of a task using values specified by an object passed as JSON in the PUT payload.
  • Update task: Clients Python JavaScriptUpdate settings of a task using values specified by an object passed as JSON in the POST payload.
  • Run task synchronously: Run a specific task and return its output.
  • Run task synchronously and get dataset items: Run a specific task and return its dataset items.
  • Run task synchronously and get dataset items: Runs an Actor task and synchronously returns its dataset items.
  • Run task synchronously: Runs an Actor task and synchronously returns its output.
  • Get list of task runs: Get a list of runs of a specific task.
  • Get last run: This is not a single endpoint, but an entire group of endpoints that lets you to retrieve and manage the last run of given actor task or any of its default storages.
  • Run task: Clients Python JavaScriptRuns an Actor task and immediately returns without waiting for the run to finish.
  • Get list of webhooks: Gets the list of webhooks of a specific Actor task.
  • Actor tasks - Introduction: The API endpoints described in this section enable you to create, manage, delete, and run Apify Actor tasks.
  • Get list of tasks: Clients Python JavaScriptGets the complete list of tasks that a user has created or used.
  • Create task: Clients Python JavaScriptCreate a new task with settings specified by the object passed as JSON in the POST payload.
  • Actors - Introduction: The API endpoints in this section allow you to manage Apify Actors.
  • Actor builds - Introduction: The API endpoints in this section allow you to manage your Apify Actors builds.
  • Actor runs - Introduction: The API endpoints in this section allow you to manage your Apify Actors runs.
  • Actor versions - Introduction: The API endpoints in this section allow you to manage your Apify Actors versions.
  • Webhook collection - Introduction: The API endpoint in this section allows you to get a list of webhooks of a specific Actor.
  • Get list of Actors: Clients Python JavaScriptGets the list of all Actors that the user created or used.
  • Create Actor: Clients Python JavaScriptCreates a new Actor with settings specified in an Actor object passed as JSON in the POST payload.
  • Delete dataset: Clients JavaScriptDeletes a specific dataset.
  • Get dataset: Clients Python JavaScriptReturns dataset object for given dataset ID.
  • Get items: Clients Python JavaScriptReturns data stored in the dataset in a desired format.
  • Store items: Clients Python JavaScriptAppends an item or an array of items to the end of the dataset.
  • Update dataset: Clients Python JavaScriptUpdates a dataset's name using a value specified by a JSON object passed in the PUT payload.
  • Get dataset statistics: Returns statistics for given dataset.
  • Get list of datasets: Clients Python JavaScriptLists all of a user's datasets.
  • Create dataset: Clients Python JavaScriptCreates a dataset and returns its object.
  • Getting started with Apify API: The Apify API provides programmatic access to the Apify platform.
  • Delete store: Clients JavaScriptDeletes a key-value store.
  • Get store: Clients Python JavaScriptGets an object that contains all the details about a specific key-value store.
  • Get list of keys: Clients Python JavaScriptReturns a list of objects describing keys of a given key-value store, as well as some information about the values (e.g.
  • Update store: Clients Python JavaScriptUpdates a key-value store's name using a value specified by a JSON object passed in the PUT payload.
  • Delete record: Clients Python JavaScriptRemoves a record specified by a key from the key-value store.
  • Get record: Clients Python JavaScriptGets a value stored in the key-value store under a specific key.
  • Check if a record exists: Clients Python JavaScriptCheck if a value is stored in the key-value store under a specific key.
  • Store record: Clients Python JavaScriptStores a value under a specific key to the key-value store.
  • Get list of key-value stores: Clients Python JavaScriptGets the list of key-value stores owned by the user.
  • Create key-value store: Clients Python JavaScriptCreates a key-value store and returns its object.
  • Get log: Clients Python JavaScriptRetrieves logs for a specific Actor build or run.
  • Logs - Introduction: The API endpoints described in this section are used the download the logs generated by Actor builds and runs.
  • Charge events in run: Clients Python JavaScriptCharge for events in the run of your pay per event Actor.
  • Resurrect run: Clients Python JavaScriptResurrects a finished Actor run and returns an object that contains all the details about the resurrected run.
  • Delete request queue: Clients JavaScriptDeletes given queue.
  • Get request queue: Clients Python JavaScriptReturns queue object for given queue ID.
  • Get head: Clients Python JavaScriptReturns given number of first requests from the queue.
  • Get head and lock: Clients Python JavaScriptReturns the given number of first requests from the queue and locks them for the given time.
  • Update request queue: Clients Python JavaScriptUpdates a request queue's name using a value specified by a JSON object passed in the PUT payload.
  • Delete request: Clients JavaScriptDeletes given request from queue.
  • Get request: Clients Python JavaScriptReturns request from queue.
  • Delete request lock: Clients Python JavaScriptDeletes a request lock.
  • Prolong request lock: Clients Python JavaScriptProlongs request lock.
  • Update request: Clients Python JavaScriptUpdates a request in a queue.
  • Delete requests: Clients Python JavaScriptBatch-deletes given requests from the queue.
  • Add requests: Clients Python JavaScriptAdds requests to the queue in batch.
  • List requests: Clients Python JavaScriptReturns a list of requests.
  • Add request: Clients Python JavaScriptAdds request to the queue.
  • Unlock requests: Clients Python JavaScriptUnlocks requests in the queue that are currently locked by the client.
  • Get list of request queues: Clients Python JavaScriptLists all of a user's request queues.
  • Create request queue: Clients Python JavaScriptCreates a request queue and returns its object.
  • Delete schedule: Clients JavaScriptDeletes a schedule.
  • Get schedule: Clients Python JavaScriptGets the schedule object with all details.
  • Get schedule log: Clients Python JavaScriptGets the schedule log as a JSON array containing information about up to a 1000 invocations of the schedule.
  • Update schedule: Clients Python JavaScriptUpdates a schedule using values specified by a schedule object passed as JSON in the POST payload.
  • Schedules - Introduction: This section describes API endpoints for managing schedules.
  • Get list of schedules: Clients Python JavaScriptGets the list of schedules that the user created.
  • Create schedule: Clients Python JavaScriptCreates a new schedule with settings provided by the schedule object passed as JSON in the payload.
  • Datasets - Introduction: This section describes API endpoints to manage Datasets.
  • Key-value stores - Introduction: This section describes API endpoints to manage Key-value stores.
  • Request queues - Introduction: This section describes API endpoints to create, manage, and delete request queues.
  • Requests- Introduction: This section describes API endpoints to create, manage, and delete requests within request queues.
  • Requests locks - Introduction: This section describes API endpoints to create, manage, and delete request locks within request queues.
  • Store - Introduction: Apify Store is home to thousands of public Actors available to the Apify community.
  • Get list of Actors in store: Gets the list of public Actors in Apify Store.
  • Get public user data: Returns public information about a specific user account, similar to what can be seen on public profile pages (e.g.
  • Users - Introduction: The API endpoints described in this section return information about user accounts.
  • Get private user data: Returns information about the current user account, including both public and private information.
  • Get limits: Returns a complete summary of your account's limits.
  • Update limits: Updates the account's limits manageable on your account's Limits page.
  • Get monthly usage: Returns a complete summary of your usage for the current usage cycle, an overall sum, as well as a daily breakdown of usage.
  • Delete webhook: Clients JavaScriptDeletes a webhook.
  • Get webhook dispatch: Clients Python JavaScriptGets webhook dispatch object with all details.
  • Get list of webhook dispatches: Clients Python JavaScriptGets the list of webhook dispatches that the user have.
  • Get webhook: Clients Python JavaScriptGets webhook object with all details.
  • Update webhook: Clients Python JavaScriptUpdates a webhook using values specified by a webhook object passed as JSON in the POST payload.
  • Test webhook: Clients Python JavaScriptTests a webhook.
  • Get collection: Clients PythonGets a given webhook's list of dispatches.
  • Get list of webhooks: Clients Python JavaScriptGets the list of webhooks that the user created.
  • Create webhook: Clients Python JavaScriptCreates a new webhook with settings provided by the webhook object passed as JSON in the payload.
  • Webhook dispatches - Introduction: This section describes API endpoints to get webhook dispatches.
  • Webhooks - Introduction: This section describes API endpoints to manage webhooks.

open-source

• Apify open source

sdk

• Apify SDK

search

• Search the documentation

Apify academy

• Web Scraping Academy: Learn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.
  • Actor marketing playbook: Learn how to optimize and monetize your Actors on Apify Store by sharing them with other platform users. --- Apify Store is a marketplace featuring thousands of ready-made automation tools called Actors.
    • Actor description & SEO description: Learn about Actor description and meta description.
    • Actors and emojis: Using emojis in Actors is a science on its own.
    • How to create an Actor README: Learn how to write a comprehensive README to help users better navigate, understand and run public Actors in Apify Store. --- ## What's a README in the Apify sense?
    • Importance of Actor URL: **Actor URL (or technical name, as we call it), is the page URL of the Actor shown on the web.
    • Name your Actor: **Apify's standards for Actor naming.
    • Emails to Actor users: **Getting users is one thing, but keeping them is another.
    • Handle Actor issues: **Once you publish your Actor in Apify Store, it opens the door to new users, feedback, and… issue reports.
    • Your Apify Store bio: ## Your Apify Store bio and Store “README” To help our community showcase their talents and projects, we introduced public profile pages for developers.
    • Actor bundles: Learn what an Actor bundle is, explore existing examples, and discover how to promote them. --- ## What is an Actor bundle?
    • How to create a great input schema: Optimizing your input schema.
    • Affiliates: The Apify Affiliate Program offers you a way to earn recurring commissions while helping others discover automation and web scraping solutions.
    • Blogs and blog resources: **Blogs remain a powerful tool for promoting your Actors and establishing authority in the field.
    • Marketing checklist: You're a developer, not a marketer.
    • Parasite SEO: **Do you want to attract more users to your Actors?
    • Product Hunt: Product Hunt is one of the best platforms for introducing new tools, especially in the tech community.
    • SEO: SEO means optimizing your content to rank high for your target queries in search engines such as Google, Bing, etc.
    • Social media: **Social media is a powerful way to connect with your Actor users and potential users.
    • Video tutorials: **Videos and live streams are powerful tools for connecting with users and potential users, especially when promoting your Actors.
    • Webinars: Webinars and live streams are a fantastic way to connect with your audience, showcase your Actor's capabilities, and gather feedback from users.
    • How Actor monetization works: **You can turn your web scrapers into a source of income by publishing them on Apify Store.
    • How Apify Store works: **Out of thousands of Actors on Apify Store marketplace, most of them were created by developers just like you.
    • How to build Actors: At Apify, we try to make building web scraping and automation straightforward.
  • Wrap open-source as an Actor: Apify is a cloud platform with a marketplace of 6,000+ web scraping and automation tools called Actors.
  • Advanced web scraping: In the Web scraping basics for JavaScript devs course, we have learned the necessary basics required to create a scraper.
    • Crawling sitemaps: In the previous lesson, we learned what is the utility (and dangers) of crawling sitemaps.
    • Scraping websites with search: # Scraping websites with search In this lesson, we will start with a simpler example of scraping HTML based websites with limited pagination.
    • Sitemaps vs search: The core crawling problem comes to down to ensuring that we reliably find all detail pages on the target website or inside its categories.
    • Tips and tricks for robustness: **Learn how to make your automated processes more effective.
    • AI agent tutorial: **In this section of the Apify Academy, we show you how to build an AI agent with the CrewAI Python framework.
  • Anti-scraping protections: # Anti-scraping protections {#anti-scraping-protections} Understand the various anti-scraping measures different sites use to prevent bots from accessing them, and how to appear more human to fix these issues. --- If at any point in time you've strayed away from the Academy's demo content, and into the Wild West by writing some scrapers of your own, you may have been hit with anti-scraping measures.
    • Anti-scraping mitigation: # Anti-scraping mitigation {#anti-scraping-mitigation} After learning about the various different anti-scraping techniques websites use, learn how to mitigate them with a few different techniques. --- In the techniques section of this course, you learned about multiple methods websites use to prevent bots from accessing their content.
      • Bypassing Cloudflare browser check: # Bypassing Cloudflare browser check {#cloudflare-challenge} Learn how to bypass Cloudflare browser challenge with Crawlee. --- If you find yourself stuck, there are a few strategies that you can employ.
      • Generating fingerprints: # Generating fingerprints {#generating-fingerprints} Learn how to use two super handy npm libraries to generate fingerprints and inject them into a Playwright or Puppeteer page. --- In Crawlee, you can use FingerprintOptions on a crawler to automatically generate fingerprints.
      • Proxies: # Proxies {#about-proxies} Learn all about proxies, how they work, and how they can be leveraged in a scraper to avoid blocking and other anti-scraping tactics. --- A proxy server provides a gateway between users and the internet, to be more specific in our case - between the crawler and the target website.
      • Using proxies: # Using proxies {#using-proxies} Learn how to use and automagically rotate proxies in your scrapers by using Crawlee, and a bit about how to obtain pools of proxies. --- In the Web scraping basics for JavaScript devs course, we learned about the power of Crawlee, and how it can streamline the development process of web crawlers.
    • Anti-scraping techniques: # Anti-scraping techniques {#anti-scraping-techniques} Understand the various common (and obscure) anti-scraping techniques used by websites to prevent bots from accessing their content. --- In this section, we'll be discussing some of the most common (as well as some obscure) anti-scraping techniques used by websites to detect and block/limit bots from accessing their content.
      • Browser challenges: # Browser challenges {#fingerprinting} > Learn how to navigate browser challenges like Cloudflare's to effectively scrape data from protected websites.
      • Captchas: # Captchas {#captchas} Learn about the reasons a bot might be presented a captcha, the best ways to avoid captchas in the first place, and how to programmatically solve them. --- In general, a website will present a user (or scraper) a captcha for 2 main reasons: 1.
      • Fingerprinting: # Fingerprinting {#fingerprinting} Understand browser fingerprinting, an advanced technique used by browsers to track user data and even block bots from accessing them. --- Browser fingerprinting is a method that some websites use to collect information about a browser's type and version, as well as the operating system being used, any active plugins, the time zone and language of the machine, the screen resolution, and various other active settings.
      • Firewalls: # Firewalls {#firewalls} Understand what a web-application firewall is, how they work, and the various common techniques for avoiding them altogether. --- A web-application firewall (or WAF) is a tool for website admins which allows them to set various access rules for their visitors.
      • Geolocation: # Geolocation {#geolocation} Learn about the geolocation techniques to determine where requests are coming from, and a bit about how to avoid being blocked based on geolocation. --- Geolocation is yet another way websites can detect and block access or show limited data.
      • Rate-limiting: # Rate-limiting {#rate-limiting} Learn about rate-limiting, a common tactic used by websites to avoid a large and non-human rate of requests coming from a single IP address. --- When crawling a website, a web scraping bot will typically send many more requests from a single IP address than a human user could generate over the same period.
  • Using Apify API: # Using Apify API A collection of various tutorials explaining how to interact with the Apify platform programmatically using its API. --- This section explains how you can run Apify Actors using Apify's API, retrieve their results, and integrate them into your own product and workflows.
  • API scraping: # API scraping Learn all about how the professionals scrape various types of APIs with various configurations, parameters, and requirements. --- API scraping is locating a website's API endpoints, and fetching the desired data directly from their API, as opposed to parsing the data from their rendered HTML pages.
    • General API scraping: # General API scraping {#general-api-scraping} Learn the benefits and drawbacks of API scraping, how to locate an API, how to utilize its features, and how to work around common roadblocks. --- This section will teach you everything you should know about API scraping before moving into the next sections in the API Scraping module.
      • Dealing with headers, cookies, and tokens: # Dealing with headers, cookies, and tokens {#challenges} Learn about how some APIs require certain cookies, headers, and/or tokens to be present in a request in order for data to be received. --- Unfortunately, most APIs will require a valid cookie to be included in the cookie field within a request's headers in order to be authorized.
      • Handling pagination: # Handling pagination {#handling-pagination} Learn about the three most popular API pagination techniques and how to handle each of them when scraping an API with pagination. --- When scraping large APIs, you'll quickly realize that most APIs limit the number of results it responds back with.
      • Locating API endpoints: # Locating API endpoints {#locating-endpoints} Learn how to effectively locate a website's API endpoints, and learn how to use them to get the data you want faster and more reliably. --- In order to retrieve a website's API endpoints, as well as other data about them, the Network tab within Chrome's (or another browser's) DevTools can be used.
    • GraphQL scraping: # GraphQL scraping {#graphql-scraping} **Dig into the topic of scraping APIs which use the latest and greatest API technology - GraphQL.
      • Custom queries: # Custom queries {#custom-queries} Learn how to write custom GraphQL queries, how to pass input values into GraphQL requests as variables, and how to retrieve and output the data from a scraper. --- Sometimes, the queries found in the Network tab aren't good enough for your use case.
      • [Introspection](https://docs.apify.com/academy/api-scraping/graphql-scrapin

…(truncated)