new-plugin

# New SkyWalking Python Plugin Generate a complete instrumentation plugin for the SkyWalking Python agent. Ask the user for: 1. **Library name** (e.g., `httpx`, `pymemcache`, `clickhouse-driver`) 2. **Plugin type**: web framework (Entry spans), HTTP client (Exit spans), database (Exit spans), cache (Exit spans), message queue (Entry+Exit spans), RPC (Entry+Exit spans) 3. **Library versions to test** (e.g., `['1.0', '2.0']`) 4. **Minimum Python version** (default `>=3.7`) 5. **External service needed for tests?** (e.g., Redis, MySQL, Kafka — or none for HTTP client plugins) If the user provides a library name without details, research the library to determine the appropriate type and instrumentation points. ## Files to Generate ### 1. Plugin Module: `skywalking/plugins/sw_<name>.py` Use the Apache 2.0 license header (copy from any existing plugin file). Follow these patterns based on plugin type: **For HTTP Exit plugins** (like `sw_requests.py`): ```python from skywalking import Layer, Component, config from skywalking.trace.context import get_context, NoopContext from skywalking.trace.span import NoopSpan from skywalking.trace.tags import TagHttpMethod, TagHttpURL, TagHttpStatusCode link_vector = ['<documentation URL>'] support_matrix = { '<pip-package-name>': { '>=3.13': ['<major>.*'], # use .* wildcard for latest patch (e.g., '4.*') '>=3.10': ['<older_minor>.*', '<major>.*'], } } note = """""" def install(): from <library> import <Class> _original = <Class>.<method> def _sw_method(this, *args, **kwargs): # Parse URL/peer from args span = NoopSpan(NoopContext()) if config.ignore_http_method_check(method) \ else get_context().new_exit_span(op=path, peer=netloc, component=Component.<Name>) with span: carrier = span.inject() span.layer = Layer.Http # Inject carrier into outgoing headers for item in carrier: headers[item.key] = item.val span.tag(TagHttpMethod(method)) span.tag(TagHttpURL(url)) res = _original(this, *args, **kwargs) span.tag(TagHttpStatusCode(res.status_code)) if res.status_code >= 400: span.error_occurred = True return res <Class>.<method> = _sw_method ``` **For Web Entry plugins** (like `sw_flask.py`): ```python from skywalking import Layer, Component, config from skywalking.trace.carrier import Carrier from skywalking.trace.context import get_context, NoopContext from skywalking.trace.span import NoopSpan from skywalking.trace.tags import TagHttpMethod, TagHttpURL, TagHttpStatusCode def install(): from <framework> import <App> _original = <App>.<handler_method> def _sw_handler(this, *args, **kwargs): req = <get_request> carrier = Carrier() for item in carrier: if item.key.capitalize() in req.headers: item.val = req.headers[item.key.capitalize()] span = NoopSpan(NoopContext()) if config.ignore_http_method_check(req.method) \ else get_context().new_entry_span(op=req.path, carrier=carrier, inherit=Component.General) with span: span.layer = Layer.Http span.component = Component.<Name> span.peer = f"{remote_addr}:{remote_port}" span.tag(TagHttpMethod(req.method)) span.tag(TagHttpURL(req.url)) resp = _original(this, *args, **kwargs) span.tag(TagHttpStatusCode(resp.status_code)) if resp.status_code >= 400: span.error_occurred = True return resp <App>.<handler_method> = _sw_handler ``` **For Database/Cache Exit plugins** (like `sw_redis.py`): ```python from skywalking import Layer, Component from skywalking.trace.context import get_context from skywalking.trace.tags import TagDbType, TagDbInstance, TagDbStatement # or TagCacheType, TagCacheOp, etc. def install(): from <library> import <Connection> _original = <Connection>.<method> def _sw_method(this, *args, **kwargs): peer = f'{this.host}:{this.port}' with get_context().new_exit_span(op='<DB>/<operation>', peer=peer, component=Component.<Name>) as span: span.layer = Layer.Database # or Layer.Cache span.tag(TagDbType('<DB type>')) span.tag(TagDbStatement(query)) res = _original(this, *args, **kwargs) return res <Connection>.<method> = _sw_method ``` **For MQ plugins** (producer Exit + consumer Entry): - Producer: new_exit_span, inject carrier into message headers, Layer.MQ - Consumer: new_entry_span, extract carrier from message headers, Layer.MQ **For async plugins**: Use `async def` wrappers, `await` the original call. **For C extension libraries**: Use `wrapt.ObjectProxy` pattern (see `sw_psycopg2.py`). ### 2. Component Enum: `skywalking/__init__.py` Add a new entry to the `Component` enum. Check the last Python-specific ID (7000+) and increment. Example: ```python NewLib = 7020 # Next available ID ``` If the component already exists in the SkyWalking ecosystem (check existing enum), reuse that ID. ### 3. Test Directory: `tests/plugin/{data|http|web}/sw_<name>/` Choose the subdirectory: - `web/` for web frameworks - `http/` for HTTP clients - `data/` for databases, caches, message queues Create these files: #### `__init__.py` Empty file with license header. #### `test_<name>.py` ```python from typing import Callable import pytest import requests from skywalking.plugins.sw_<name> import support_matrix from tests.orchestrator import get_test_vector from tests.plugin.base import TestPluginBase @pytest.fixture def prepare(): return lambda *_: requests.get('http://0.0.0.0:9090/<endpoint>', timeout=5) class TestPlugin(TestPluginBase): @pytest.mark.parametrize('version', get_test_vector(lib_name='<name>', support_matrix=support_matrix)) def test_plugin(self, docker_compose, version): self.validate() ``` #### `docker-compose.yml` ```yaml version: '2.1' services: collector: extends: service: collector file: ../../docker-compose.base.yml # Add external service if needed (database, cache, MQ broker) # <service_name>: # image: <image> # ports: [<port>:<port>] # healthcheck: ... # networks: [beyond] provider: extends: service: agent file: ../../docker-compose.base.yml ports: - 9091:9091 volumes: - .:/app command: ['bash', '-c', 'pip install flask && pip install -r /app/requirements.txt && sw-python run python3 /app/services/provider.py'] depends_on: collector: condition: service_healthy healthcheck: test: ["CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/9091"] interval: 5s timeout: 60s retries: 120 environment: SW_AGENT_NAME: provider SW_AGENT_LOGGING_LEVEL: DEBUG consumer: extends: service: agent file: ../../docker-compose.base.yml ports: - 9090:9090 volumes: - .:/app command: ['bash', '-c', 'pip install flask && pip install -r /app/requirements.txt && sw-python run python3 /app/services/consumer.py'] depends_on: collector: condition: service_healthy provider: condition: service_healthy environment: SW_AGENT_NAME: consumer SW_AGENT_LOGGING_LEVEL: DEBUG networks: beyond: ``` Key points: - Consumer listens on port 9090, provider on 9091 - Consumer calls provider to create cross-process trace - Use `sw-python run python3` to start with agent instrumentation - The consumer uses Flask to expose an HTTP endpoint for the test's `prepare()` fixture to hit - The provider uses the target library (e.g., makes a Redis call, DB query) - Both install the target library via `pip install -r /app/requirements.txt` #### `services/provider.py` Flask app on port 9091 that uses the target library. Example for a data plugin: ```python if __name__ == '__main__': from flask import Flask, jsonify app = Flask(__name__) @app.route('/endpoint', methods=['POST', 'GET']) def handler(): import <target_library> # Make a call using the target library # e.g., client = redis.StrictRedis(host='redis'); client.get('key') return jsonify({'status': 'ok'}) app.run(host='0.0.0.0', port=9091) ``` #### `services/consumer.py` Flask app on port 9090 that calls the provider: ```python import requests if __name__ == '__main__': from flask import Flask, jsonify app = Flask(__name__) @app.route('/endpoint', methods=['POST', 'GET']) def handler(): res = requests.post('http://provider:9091/endpoint', timeout=5) return jsonify(res.json()) app.run(host='0.0.0.0', port=9090) ``` #### `services/__init__.py` Empty file with license header. #### `expected.data.yml` Define expected spans. Structure depends on plugin type: For a **data/cache Exit** plugin (consumer -> provider -> external service): ```yaml segmentItems: - serviceName: provider segmentSize: 1 segments: - segmentId: not null spans: - operationName: <DB>/<operation> parentSpanId: 0 spanId: 1 spanLayer: Database # or Cache startTime: gt 0 endTime: gt 0 componentId: <ID> spanType: Exit peer: <service>:<port> skipAnalysis: false tags: - key: db.type value: <type> - key: db.statement value: <query> - operationName: /endpoint parentSpanId: -1 spanId: 0 spanLayer: Http startTime: gt 0 endTime: gt 0 componentId: 7001 spanType: Entry peer: not null skipAnalysis: false tags: - key: http.method value: POST - key: http.url value: http://provider:9091/endpoint - key: http.status_code value: '200' refs: - parentEndpoint: /endpoint networkAddress: 'provider:9091' refType: CrossProcess parentSpanId: 1 parentTraceSegmentId: not null parentServiceInstance: not null parentService: consumer traceId: not null - serviceName: consumer segmentSize: 1 segments: - segmentId: not null spans: - operationName: /endpoint parentSpanId: 0 spanId: 1 spanLayer: Http startTime: gt 0 endTime: gt 0 componentId: 7002 spanType: Exit peer: provider:9091 skipAnalysis: false tags: - key: http.method value: POST - key: http.url value: http://provider:9091/endpoint - key: http.status_code value: '200' - operationName: /endpoint parentSpanId: -1 spanId: 0 spanLayer: Http startTime: gt 0 endTime: gt 0 componentId: 7001 spanType: Entry peer: not null skipAnalysis: false tags: - key: http.method value: GET - key: http.url value: http://0.0.0.0:9090/endpoint - key: http.status_code value: '200' ``` Important notes for expected data: - Spans within a segment are ordered **child-first** (highest spanId first) - Entry spans have `parentSpanId: -1` - Exit spans reference the entry span as parent - Cross-process refs link consumer exit span to provider entry span - `componentId` must match the Component enum value exactly - Consumer's exit span uses Requests component (7002) since it calls via `requests.post()` - Consumer's entry span uses Flask component (7001) since it receives via Flask ### 4. Update `pyproject.toml` Add the library to `[tool.poetry.group.plugins.dependencies]`: ```bash poetry add <library> --group plugins ``` ### 5. Regenerate Docs ```bash make doc-gen ``` ### 6. Lint Check ```bash make lint ``` ## Post-Generation Reminders After generating all files, remind the user: