scrapling

v0.2.3

Scrapling as an iii worker — scrapling::* functions run HTTP / anti-bot / browser fetches, CSS/XPath/regex/adaptive extraction, persistent sessions, and a declarative crawl over the iii bus.

Published 3mo agoanti-bot browser crawler extraction http scraping

Deprecated

This worker is deprecated in favor of browser. Please migrate to the browser worker.

Use browser instead.

Install

bash

iii trigger compose::add worker=scrapling@0.2.3

Run in your project with the iii engine running. Exact versions are immutable and digest-pinned.

Functions21

scrapling::crawl

Function

BFS-crawl from start_urls (follow same-domain links), extract per page, stream items.

Request

allowed_domainsstring[]
only follow links on these hosts
concurrencyinteger
css_selectorstring
scope the render to this CSS subtree (e.g. a page's content div)
download_delaynumber
seconds to wait between crawl rounds
fetcherstringenum http, stealthy, dynamic
formatstringenum markdown, text
render page body to this format
impersonatestring
include_htmlboolean
main_content_onlyboolean
strip nav/scripts/hidden before rendering
max_depthinteger
max_pagesinteger
same_domainboolean
follow only same-host links (default true)
selectorsobject[]
- allboolean
  return every match as a list
- attrstring
  extract this attribute instead of text
- cssstring
- htmlboolean
  extract inner HTML instead of text
- namestringrequired
- regexstring
- xpathstring
start_urlsstring[]
stream_namestring
stream to emit items on (default scrapling::crawl)
urlstring
single start URL (alternative to start_urls)

Response

itemsobject[]
a small sample of streamed items
Empty object
statsobject
- crawledinteger
- errorsinteger
- itemsinteger
- stoppedstring
streamobject
read the full item stream via stream::on with this name + group_id
- group_idstring
- namestring

scrapling::css

Function

One CSS query over HTML; first-or-all; `attr` pulls an attribute else text.

Request

adaptiveboolean
relocate elements after a site change via saved identities
adaptive_domainstring
page URL/domain that keys saved identities
attrstring
auto_saveboolean
save matched identities (defaults on when adaptive)
firstboolean
htmlstringrequired
identifierstring
stable key for the saved element
querystringrequired

Response

resultstring[]

scrapling::describe

Function

Describe the first css/xpath match: attrs, generated selectors, class list, DOM context.

Request

htmlstringrequired
kindstringenum css, xpath
querystringrequired

Response

elementobject
- attrsobject
- childreninteger
- classesstring[]
- cssstring
- full_cssstring
- full_xpathstring
- htmlstring
- parent_tagstring
- siblingsinteger
- tagstring
- textstring
- xpathstring
foundboolean

scrapling::dynamic-fetch

Function

Playwright/Chromium fetch: JS render, waits, XHR capture, CDP; extraction + bulk.

Request

block_adsboolean
blocked_domainsstring[]
capture_xhrstring
cdp_urlstring
cookiesobject
css_selectorstring
scope the render to this CSS subtree (e.g. a page's content div)
disable_resourcesboolean
dns_over_httpsboolean
extra_flagsstring[]
extra_headersobject
formatstringenum markdown, text
render page body to this format
google_searchboolean
headlessboolean
include_htmlboolean
load_domboolean
localestring
main_content_onlyboolean
strip nav/scripts/hidden before rendering
max_pagesinteger
network_idleboolean
proxystring
real_chromeboolean
retriesinteger
retry_delaynumber
selectorsobject[]
- allboolean
  return every match as a list
- attrstring
  extract this attribute instead of text
- cssstring
- htmlboolean
  extract inner HTML instead of text
- namestringrequired
- regexstring
- xpathstring
timeoutnumber
milliseconds (browser fetcher)
timezone_idstring
urlstring
urlsstring[]
useragentstring
waitnumber
extra ms to wait after load
wait_selectorstring
wait_selector_statestringenum attached, detached, visible, hidden

Response

captured_xhrobject[]
Empty object
contentstring
markdown/text render when `format` requested
cookiesobject
encodingstring
errorstring
extractedobject
formatstring
headersobject
htmlstring
resultsobject[]
Empty object
statusinteger
urlstring

scrapling::extract

Function

Parse HTML with a selector list (css/xpath/regex, text/attr/html, all-or-first).

Request

adaptiveboolean
relocate elements after a site change via saved identities
adaptive_domainstring
page URL/domain that keys saved identities
auto_saveboolean
save matched identities (defaults on when adaptive)
htmlstringrequired
selectorsobject[]required
- allboolean
  return every match as a list
- attrstring
  extract this attribute instead of text
- cssstring
- htmlboolean
  extract inner HTML instead of text
- namestringrequired
- regexstring
- xpathstring

Response

extractedobject

scrapling::fetch

Function

Fast HTTP fetch, TLS impersonation: get/post/put/delete, inline extraction, bulk `urls`.

Request

cookiesobject
css_selectorstring
scope the render to this CSS subtree (e.g. a page's content div)
dataobject
follow_redirectsboolean
formatstringenum markdown, text
render page body to this format
headersobject
http3boolean
impersonatestring
TLS/UA fingerprint, e.g. 'chrome'
include_htmlboolean
jsonobject
main_content_onlyboolean
strip nav/scripts/hidden before rendering
max_redirectsinteger
methodstringenum get, post, put, delete
paramsobject
proxiesobject
per-scheme proxies, e.g. {"https": "http://..."}
proxystring
proxy_authstring[]
[user, password]
retriesinteger
retry_delaynumber
selectorsobject[]
- allboolean
  return every match as a list
- attrstring
  extract this attribute instead of text
- cssstring
- htmlboolean
  extract inner HTML instead of text
- namestringrequired
- regexstring
- xpathstring
stealthy_headersboolean
timeoutnumber
seconds (HTTP fetcher)
urlstring
urlsstring[]
verifyboolean

Response

captured_xhrobject[]
Empty object
contentstring
markdown/text render when `format` requested
cookiesobject
encodingstring
errorstring
extractedobject
formatstring
headersobject
htmlstring
resultsobject[]
Empty object
statusinteger
urlstring

scrapling::find

Function

Find elements by tag/attribute filters (+ optional text regex); BeautifulSoup-style.

Request

attrsobject
attribute filters, e.g. {"class": "card"}
firstboolean
htmlstringrequired
limitinteger
tagstring
tag name or list of tag names
text_regexstring
keep only elements whose text matches this regex

Response

countinteger
itemsobject[]
- attrsobject
- cssstring
- htmlstring
- tagstring
- textstring
- xpathstring

scrapling::find-by-regex

Function

Find elements whose visible text matches a regex pattern.

Request

case_sensitiveboolean
clean_matchboolean
firstboolean
htmlstringrequired
limitinteger
patternstringrequired

Response

countinteger
itemsobject[]
- attrsobject
- cssstring
- htmlstring
- tagstring
- textstring
- xpathstring

scrapling::find-by-text

Function

Find elements whose visible text matches a string (exact or `partial`).

Request

case_sensitiveboolean
clean_matchboolean
ignore surrounding/collapsing whitespace
firstboolean
htmlstringrequired
limitinteger
partialboolean
match elements that contain the text
textstringrequired

Response

countinteger
itemsobject[]
- attrsobject
- cssstring
- htmlstring
- tagstring
- textstring
- xpathstring

scrapling::find-similar

Function

Structural auto-match: given one example element, return it plus similar elements.

Request

anchorstringrequired
CSS selector to one example element
htmlstringrequired
match_textboolean
selectorsobject[]
- allboolean
  return every match as a list
- attrstring
  extract this attribute instead of text
- cssstring
- htmlboolean
  extract inner HTML instead of text
- namestringrequired
- regexstring
- xpathstring
similarity_thresholdnumber

Response

countinteger
itemsobject[]
Empty object

scrapling::inject-guidance

Function

Internal pre_generate hook: appends scrapling usage guidance to the agent system prompt. Bound to harness::hook::pre-generate at worker startup; not called directly.

Request

generateobject
- system_promptstring
  system prompt assembled so far

Response

mutationsobjectrequired
- system_promptstring
  full replacement prompt (base + guidance)

scrapling::on-registry-changed

Function

Internal: on engine registry changes, bind the scrapling inject-guidance pre-generate hook once the harness has registered its trigger type. Not called directly.

Request

valuenull

Response

okbooleanrequired

scrapling::regex

Function

Run a regex over the visible text of provided HTML; `first` returns the first match, else all.

Request

firstboolean
htmlstringrequired
patternstringrequired

Response

resultstring[]

scrapling::screenshot

Function

Capture a page screenshot as image content blocks via a browser fetcher (dynamic or stealthy).

Request

fetcherstringenum dynamic, stealthy
formatstringenum png, jpeg
full_pageboolean
headlessboolean
network_idleboolean
proxystring
timeoutnumber
urlstringrequired
wait_selectorstring

Response

contentobject[]
image blocks (one per tile, width<=1024/height<=1536) + a text caption
- datastring
  base64 image bytes (image blocks)
- mimestring
- textstring
- typestringrequiredenum image, text
mimestring
urlstring

scrapling::session-close

Function

Close a session and free its browser/connection.

Request

session_idstringrequired

Response

closedboolean

scrapling::session-fetch

Function

Fetch a URL on an open session (reuses its cookies/browser); same page/extraction output.

Request

css_selectorstring
scope the render to this CSS subtree (e.g. a page's content div)
dataobject
formatstringenum markdown, text
render page body to this format
headersobject
include_htmlboolean
jsonobject
main_content_onlyboolean
strip nav/scripts/hidden before rendering
methodstringenum get, post, put, delete
paramsobject
selectorsobject[]
- allboolean
  return every match as a list
- attrstring
  extract this attribute instead of text
- cssstring
- htmlboolean
  extract inner HTML instead of text
- namestringrequired
- regexstring
- xpathstring
session_idstringrequired
urlstringrequired
wait_selectorstring

Response

captured_xhrobject[]
Empty object
contentstring
markdown/text render when `format` requested
cookiesobject
encodingstring
errorstring
extractedobject
formatstring
headersobject
htmlstring
resultsobject[]
Empty object
statusinteger
urlstring

scrapling::session-list

Function

List open sessions with their type and idle time.

Request

typestringenum http, dynamic, stealthy
filter by type

Response

sessionsobject[]
- created_atnumber
- idle_snumber
- last_usednumber
- session_idstring
- typestring

scrapling::session-open

Function

Open a persistent HTTP/browser session; returns a session_id that reuses cookies + state.

Request

capture_xhrstring
regex; capture matching XHRs (browser sessions)
headersobject
headlessboolean
impersonatestring
proxiesobject
proxystring
real_chromeboolean
solve_cloudflareboolean
timeoutnumber
typestringenum http, dynamic, stealthy
session engine
useragentstring

Response

session_idstring
typestring

scrapling::stealthy-fetch

Function

Camoufox stealth browser: solves Cloudflare, hardens WebRTC/canvas; extraction + bulk.

Request

allow_webglboolean
block_adsboolean
block_webrtcboolean
blocked_domainsstring[]
capture_xhrstring
cookiesobject
css_selectorstring
scope the render to this CSS subtree (e.g. a page's content div)
disable_resourcesboolean
dns_over_httpsboolean
extra_flagsstring[]
extra_headersobject
formatstringenum markdown, text
render page body to this format
google_searchboolean
headlessboolean
hide_canvasboolean
include_htmlboolean
load_domboolean
localestring
main_content_onlyboolean
strip nav/scripts/hidden before rendering
max_pagesinteger
network_idleboolean
proxystring
retriesinteger
retry_delaynumber
selectorsobject[]
- allboolean
  return every match as a list
- attrstring
  extract this attribute instead of text
- cssstring
- htmlboolean
  extract inner HTML instead of text
- namestringrequired
- regexstring
- xpathstring
solve_cloudflareboolean
timeoutnumber
milliseconds (browser fetcher)
timezone_idstring
urlstring
urlsstring[]
useragentstring
waitnumber
extra ms to wait after load
wait_selectorstring
wait_selector_statestringenum attached, detached, visible, hidden

Response

captured_xhrobject[]
Empty object
contentstring
markdown/text render when `format` requested
cookiesobject
encodingstring
errorstring
extractedobject
formatstring
headersobject
htmlstring
resultsobject[]
Empty object
statusinteger
urlstring

scrapling::to-markdown

Function

Convert HTML to compact Markdown (or text/html); optional CSS scope + main-content clean.

Request

css_selectorstring
convert only the subtree matching this CSS selector
formatstringenum markdown, text, html
htmlstringrequired
main_content_onlyboolean
strip nav/scripts/hidden nodes first

Response

contentstring
formatstring

scrapling::xpath

Function

One XPath query over HTML; first-or-all; `attr` pulls an attribute else text.

Request

adaptiveboolean
relocate elements after a site change via saved identities
adaptive_domainstring
page URL/domain that keys saved identities
attrstring
auto_saveboolean
save matched identities (defaults on when adaptive)
firstboolean
htmlstringrequired
identifierstring
stable key for the saved element
querystringrequired

Response

resultstring[]

Triggers0

No triggers registered

Install

Functions21

Triggers0

Search the registry