shub-workflow

Use when building, updating, fixing, or understanding a shub-workflow issuer — a fundamental Scrapy Cloud data-pipeline component that reads a massive input (finished spider jobs or batch files), processes/dedups/transforms the items, and writes a massive batch output, chaining into the post-crawl processing pipeline up to delivery. Built on shub_workflow.issuer (IssuerScript / IssuerScriptWithFileSystemInput / IssuerScriptWithSCJobInput) over BaseLoopScript. Use for consumers, deduplicators, filters, balancers, reducers, and issuer-based delivery scripts — and when migrating an old delivery (BaseDeliverScript) to an issuer.

Use for help using shub_workflow's scanjobs tool — scanning ScrapyCloud jobs to extract and plot data from stats, logs, items or spider args. Covers building a scanjobs command line (the stat/log/item/spider-arg patterns, the postscript post-processor -c, the --plot mini-language, time windows, and output modes) and the predefined "programs" shortcut: the PROGRAMS dict in a project's scripts/scanjobs.py (a subclass of shub_workflow.utils.scanjobs.ScanJobs), invoked as `scanjobs.py -g <program> -v key:val`, including {var} placeholders and {{ }} escaping. Applies to any project whose scripts/scanjobs.py subclasses shub_workflow's ScanJobs.

Use when building, updating, fixing, or understanding a shub-workflow monitor — a script that gives a broad cross-job view of a workflow: scanning the spider/script jobs in a time window, aggregating their stats (and log-derived stats), computing ratios, emitting reports, and raising threshold alerts (Slack / Sentry). Built on shub_workflow.utils.monitor.BaseMonitor (a BaseScript, not a loop manager) plus the AlertSenderMixin / SlackMixin / SentryMixin alert layer. Use for any monitor.py that subclasses BaseMonitor (directly or via a project base mixin).

Use when writing, fixing, or updating a Python script built on the shub_workflow.script base classes (BaseScript / BaseLoopScript / BaseLoopScriptAsyncMixin / ArgumentParserScript) — i.e. any script that operates on Scrapy Cloud: scheduling spiders or scripts, scanning/querying SC jobs, aggregating stats, or running as a crawl manager, monitor, scheduler, consumer/deliverer, or an ad-hoc CLI that talks to SC — whether it runs ON Scrapy Cloud or locally against a project. When asked to create a new "script", first confirm it is a Scrapy Cloud script (see below), since these base classes are the right tool precisely when the script deploys to or operates on Scrapy Cloud.

Use when reading, writing, listing, copying, moving, or deleting files through shub-workflow's cloud-agnostic filesystem layer — the `shub_workflow.utils.futils` module and its `FSHelper` class (the recommended one; `S3Helper` is its deprecated backward-compatibility base) — i.e. any code that touches `s3://`, `gs://`, or local paths the same way, needs AWS role-assumed (STS) credentials, or must list/download deliveries in a bucket. Covers the prefix-based dispatch, module functions vs. the helper class, credential/role handling, the three listing variants (`list_folder` vs `list_path` vs recursive), `op_kwargs`/ACLs, and the install extras. Reach for it whenever you see `from shub_workflow.utils.futils import ...` or an `FSHelper`.

Use when building, updating, fixing, or understanding a shub-workflow graph manager — a script that runs an arbitrary DAG of tasks (spiders and scripts, including crawl managers and deliver scripts) on Scrapy Cloud with dependencies, built on shub_workflow.graph (GraphManager + Task / SpiderTask) and WorkFlowManager. Covers configure_workflow(), Task vs SpiderTask, dependency linking (add_next_task / add_wait_for), the on_finish/retry routing, resources, parallelization, and the start directives. Use for any graph-manager / flow-manager script that subclasses GraphManager (directly or via a project base mixin).

Use when building, updating, fixing, or understanding a shub-workflow crawl manager — a script that schedules spider jobs on Scrapy Cloud and reacts to their outcomes, built on shub_workflow.crawl (CrawlManager / PeriodicCrawlManager / GeneratorCrawlManager / AsyncSchedulerCrawlManagerMixin) and WorkFlowManager. Covers choosing the base class, the set_parameters_gen() generator pattern, the outcome/retry/throttling hooks, async concurrent scheduling, and the name/flow-id/loop-mode rules. Use for any crawl-manager script in any project that subclasses these classes (directly or via a project base mixin).