| name | ubq-upstream |
| description | Use when querying third-party upstream data through the ubq library's 'upstream' provider: get the latest upstream release version of a package and resolve its upstream project location. Covers the static data-file model (packages.toml), the source types (gnome/gitlab/sourceforge/gnu/kde/gstreamer/pypi/cpan/http/ftp), why series/pocket/archive are ignored, no-auth access, and the fact that bugs and merge requests are NOT supported. Load the central 'ubq' skill for the shared API and models. |
| version | 1.0.0 |
| author | Lena Voytek |
| metadata | {"tags":["ubq","upstream","ubuntu","versions","packages","releases","data-file","third-party"],"related_skills":["ubq","ubq-launchpad","ubq-github","ubq-snapcraft"]} |
ubq — Upstream Provider (provider_name="upstream")
Overview
The Upstream provider serves third-party upstream metadata for packages —
the version a project ships at its own release site, independent of any
Ubuntu/Debian archive. It answers "what's the latest upstream release of this
package, and where does that project live?" It is read-only and supports
just two capabilities: version and package. It has no bug or
merge-request capability — calling those raises ValueError.
Read the central ubq skill first for the QueryService API, auth modes,
and model shapes. This skill covers Upstream-specific arguments, the data-file
model, and field semantics.
Capabilities: version ✅ · package ✅ · bug ❌ · merge_request ❌
(the same capability shape as Snapcraft).
Unlike the other providers, upstream does not call a live service API for
its package list. It is driven by a generated static data file shipped inside
the library (src/ubq/providers/upstream/data/packages.toml). Each entry names
a package, a source type, a download/listing URL, and a regex for extracting
the version. Package resolution is offline (pure data-file lookup); version
resolution fetches that one URL over HTTP/FTP and scrapes the highest version.
Scope. The data file deliberately excludes sources that other tools/
providers already cover: it does not track GitHub- or Launchpad-hosted
projects, nor sources followed via the Debian or Ubuntu archives. If a package
isn't in the file (or was classified out of scope), every call returns None.
For a deb package's archive version use ubq-launchpad; for a snap use
ubq-snapcraft; for a GitHub project's issues/PRs use ubq-github.
When to Use
- Find the latest upstream release version of a package as published at its
own project site (GNOME download server, a GitLab tags page, SourceForge,
GNU, KDE, GStreamer, PyPI, CPAN, or a plain HTTP/FTP directory listing).
- Resolve a package's upstream project location (its project page or the
stable download URL).
- Diff "what upstream shipped" against "what Ubuntu/Debian shipped" — pair an
upstream
get_version with a Launchpad get_version (see example below).
Don't use for: archive/distribution versions (use ubq-launchpad), snap
channel versions (use ubq-snapcraft), GitHub/Launchpad-hosted sources (not in
scope — the data file omits them), or bugs / merge-pull requests (Upstream
supports neither).
Authentication
The upstream provider needs no credentials — anonymous is the only mode.
login() builds a trivial session (no token, no keyring, no network), so it is
completely safe for servers, CI, containers, and automated agents.
from ubq import QueryService
service = QueryService()
service.login(provider_name="upstream")
Passing ProviderCredentials is accepted for interface uniformity but is
ignored — there is nothing to authenticate against. Don't supply a token; there
is no private upstream data and no rate-limited authenticated tier here.
The Data-File Model (packages.toml)
The provider is backed by src/ubq/providers/upstream/data/packages.toml, a
generated table with one section per package:
[accountsservice]
type = "gitlab"
stable_url = "https://gitlab.freedesktop.org/accountsservice/accountsservice/tags"
stable_regex = "tags/(?:accountsservice[-_])?v?([0-9][0-9.]*)"
tags = []
flavors = ["ubuntu"]
version_fixups = ["underscore_to_dot_if_full", "dash_to_dot_if_full"]
Fields (see data_file.py / UpstreamSource):
type — source classification. Must be one of the ten valid types:
gnome, gitlab, sourceforge, gnu, kde, gstreamer, pypi, cpan,
http, ftp. (All ten currently share one DirectoryHandler; the distinct
types are kept for provenance and future divergence.)stable_url (required) — the listing/page/directory URL to fetch. Its
final path segment may itself be a version regex (e.g.
.../files/aa-lib/([0-9.]+)); the handler then descends into the
highest-versioned matching subdirectory before scraping.stable_regex (required) — regex whose group 1 is the version, run
against every line of the fetched body; the dpkg-highest match wins.version_fixups — ordered rewrite ops applied to each raw match (e.g.
underscore_to_dot_if_full, dash_to_dot_if_full, gnome_prerelease,
sub:PATTERN=>REPL, prepend:X, append:X, drop_if_equal:X,
libreoffice_triplet, mozjs_dots, first_two_dotted). These normalize
upstream's formatting toward a Debian-comparable version.unstable_url / unstable_regex — optional development-series variants.
Not used by the current get_version path (which always reads the stable
pair); present in the data model only.project_url — optional canonical project page; used by get_package.tags / flavors — provenance metadata (e.g. flavors=["kubuntu"]);
not part of the returned records.
load_sources() validates every entry at load time: an unknown/out-of-scope
type, a missing stable_url/stable_regex, or an un-compilable regex raises
ValueError. The package list is loaded lazily on first query and cached on the
provider instance.
Versions
get_version(package_name, archive, series, pocket, provider_name) returns the
latest upstream release found at the package's stable source.
service.login(provider_name="upstream")
v = service.get_version(
package_name="accountsservice",
archive="",
series="",
pocket=None,
provider_name="upstream",
)
if v:
print(v.package_name, v.version_string)
Behavior and gotchas:
series and pocket are ignored — upstream sources have no notion of an
Ubuntu series or pocket. They are accepted for the shared QueryService
signature and echoed straight back onto the returned VersionRecord
(whatever you pass in series/pocket reappears there unchanged).archive is also ignored for lookup and merely stored on the record.
There is no distribution/base concept here, so archive="" is the idiomatic
value (the bundled comparison script passes ""). Don't pass "ubuntu" /
"debian" expecting it to mean anything.- The result carries only the version:
version_string is populated;
changelog, created_at, and released_at are always None for
upstream (the provider sets changelog=None and never fills the dates).
- Fetch mechanics: the handler resolves any trailing version-regex path
segment (descending to the newest matching subdir), fetches the URL over
HTTP/FTP (
User-Agent: ubq/1.0), applies version_fixups to each match, and
returns the dpkg-highest version (a pure-Python dpkg comparator, no
python-apt).
- Returns
None if the package isn't in the data file or if the fetch
yielded no matching version. A network failure that isn't a timeout (DNS,
connection refused, HTTP error) is swallowed and also surfaces as None — so
a None from get_version can mean either "unknown package" or "couldn't
reach/parse the source right now". Distinguish them with get_package
(offline): if get_package returns a record but get_version returns None,
the package is known and the fetch/scrape came up empty.
- A genuine timeout raises
ubq.RequestTimeoutError (default 30s), not
None. Raise QueryService(timeout=...) for slow mirrors.
Diff upstream vs. the Ubuntu archive (pair with the Launchpad provider):
from ubq import QueryService
service = QueryService()
service.login(provider_name="upstream")
service.login(provider_name="launchpad")
upstream = service.get_version("accountsservice", archive="", series="",
pocket=None, provider_name="upstream")
archived = service.get_version("accountsservice", archive="ubuntu",
series="resolute", pocket="Release",
provider_name="launchpad")
print("upstream:", upstream.version_string if upstream else "n/a")
print("ubuntu: ", archived.version_string if archived else "n/a")
Packages
get_package(package_name, archive, provider_name) resolves a package's
upstream project location. This is a pure data-file lookup — no network.
pkg = service.get_package(package_name="accountsservice", archive="",
provider_name="upstream")
print(pkg.name, pkg.package_url)
package_url is the source's project_url if present, otherwise its
stable_url. Because the fallback is stable_url, the URL may be a raw
download/listing target and can even contain a trailing version-regex segment
(e.g. .../files/aa-lib/([0-9.]+)) — it is not guaranteed to be a
clean, browsable project home. Treat package_url as "where this version
came from," not necessarily a landing page.archive is ignored for lookup and only echoed onto the record; pass "".- Returns
None if the package isn't in the data file.
Example Programs (source repo scripts/)
The ubq source repository ships upstream-related scripts. They're references,
not a runtime dependency — your code calls the library directly.
check_upstream_versions_table.py — the canonical end-to-end usage: logs in
to upstream, iterates a package table, and calls
service.get_version(package_name=..., archive="", series="", pocket=None, provider_name="upstream") for each. Copy its call shape.
Flags: [--url URL] [--package PKG ...] [--limit N] [--verbose].convert_upstream_packages.py — the generator that builds
data/packages.toml from the legacy ubuntu-desktop-versions package map
(classifies each source by URL host, drops out-of-scope
github/launchpad/debian/ubuntu entries, emits typed TOML). Run this to
regenerate/refresh the data file; it is not part of normal querying.
Common Pitfalls
- Passing
series/pocket/archive and expecting them to filter. All
three are ignored by the upstream provider. series/pocket are echoed
back onto the record; archive is stored only. Pass series="",
pocket=None, archive="".
- Reading
None as "no such package". get_version also returns None
on a non-timeout fetch/parse failure. Use get_package (offline) to confirm
whether the package is actually in the data file before concluding it's
unknown.
- Expecting a changelog or dates. Upstream
VersionRecords carry only
version_string; changelog, created_at, and released_at are always
None.
- Assuming
package_url is a tidy project homepage. It's project_url
when set, else the raw stable_url (which may include a version-regex path
segment). Don't render it blindly as a canonical link.
- Calling unsupported capabilities.
get_bug/search_bugs/submit_bug
and the merge-request methods raise ValueError: Provider 'upstream' does not support ... queries. Branch on supports("upstream", ...) first.
- Looking for a GitHub/Launchpad-hosted package here. Out of scope by
design — the data file omits them. Use
ubq-github / ubq-launchpad.
- Trying to authenticate. There are no upstream credentials; supplying a
token does nothing. Just
login(provider_name="upstream").
- Slow/unreachable mirror. A live version fetch can hit a slow SourceForge/
FTP mirror and raise
ubq.RequestTimeoutError (default 30s). Raise
QueryService(timeout=...) when scanning many packages.
Verification Checklist
