From c9d0e30e42975be6964e0d79da4f4f1f4beaadda Mon Sep 17 00:00:00 2001 From: Athanasius Date: Sun, 30 Jan 2022 12:55:33 +0000 Subject: [PATCH] schemas/README: Flesh out this in its new 'stub' form The remaining extant text has also now been moved into `docs/Developers.md`. --- schemas/README-EDDN-schemas.md | 386 ++------------------------------- 1 file changed, 16 insertions(+), 370 deletions(-) diff --git a/schemas/README-EDDN-schemas.md b/schemas/README-EDDN-schemas.md index 8ebb1d4..d0d4022 100644 --- a/schemas/README-EDDN-schemas.md +++ b/schemas/README-EDDN-schemas.md @@ -1,377 +1,23 @@ # EDDN Schemas Documentation -### Sending data -Messages sent to EDDN **MUST**: +EDDN message Schemas are [JSON](https://www.json.org/json-en.html) files +conforming to 'draft 04' of the [JSON Schema](https://json-schema.org/) +specification. -- Use the URL: `https://eddn.edcd.io:4430/upload/`. Note the use of - TLS-encrypted HTTPS. A plain HTTP request will elicit a `400 Bad - Request` response. -- Use the HTTP 1.1 protocol. HTTP/2 is not supported at this time. -- Use a **POST** request, with the body containing the EDDN message. No - query parameters in the URL are supported or necessary. +## Canonical location of Schema files -The body of an EDDN message is a JSON object in UTF-8 encoding. If you do not -compress this body then you MUST set a `Content-Type` header of -`applicaton/json`. +For the EDDN Live service you should always be checking +[the live version of the schemas, and their READMEs](https://github.com/EDCD/EDDN/tree/live/schemas). +Any other version of the Schemas is not guaranteed to be synchronized with +those actually running on the Live service. -For historical reasons URL form-encoded data *is* supported, **but this is -deprecated and no new software should attempt this method**. We -purposefully do not further document the exact format for this. +## Documentation of Schema files -You *MAY* use gzip compression on the body of the message, but it is not -required. If you do compress the body then you **MUST* send a `Content-Type` -header of `gzip` instead of `application/json`. +The Schema files themselves are considered to be the canonical definition of +the required, and allowed, contents of the relevant EDDN message. There +**SHOULD** be an accompanying README file, e.g. for `commodity-v3.0.json` there +is also a `commodity-README.md` file in the project root `schemas/` directory. -You should be prepared to handle all scenarios where sending of a message -fails: - -1. Connection refused. -2. Connection timed out. -3. Other possible responses as documented in - [Server responses](#server-responses). - -Carefully consider whether you should queue a 'failed' message for later -retry. In particular, you should ensure that one 'bad' message does not -block other messages from being successfully sent. - -You **MUST** wait some reasonable time (minimum 1 minute) before retrying -any failed message. - -You **MUST NOT** retry any message that received a HTTP `400` or `426` code. -An exception can be made if, **and only if**, *you have manually verified that -you have fixed the issues with it (i.e. updated the Schema/version to a -currently supported one and adjusted the data to fit that Schema/version).* - -You **MAY** retry a message that initially received a `413` response (in -the hopes that the EDDN service admins decided to increase the maximum -allowed request size), but should not do so too quickly or in perpetuity. - -In general: - -- No data is better than bad data. -- *Delayed* good data is better than degrading the EDDN service for others. - -### Format of uploaded messages -Each message is a JSON object in UTF-8 encoding containing the following -key+value pairs: - -1. `$schemaRef` - Which Schema (including version) this message is for. -2. `header` - Object containing mandatory information about the upload; - 1. `uploaderID` - a unique ID for the player uploading this data. - Don't worry about privacy, the EDDN service will hash this with a key - that is regularly changed so no-one knows who an uploader is in-game. - 2. `softwareName` - an identifier for the software performing the upload. - 3. `softwareVersion` - The version of that software being used. - - Listeners MAY make decisions about whether to utilise the data in any - message based on the combination of `softwareName` and `softwareVersion`. - - **DO not** add `gatewaytimestamp` yourself. The EDDN Gateway will add - this and will overwrite any that you provide, so don't bother. -4. `message` - Object containing the data for this message. Consult the - relevant README file within this documentation, e.g. - [codexentry-README.md](./codexentry-README.md). There are some general - guidelines [below](#contents-of-message). - -For example, a shipyard message, version 2, might look like: - -```JSON -{ - "$schemaRef": "https://eddn.edcd.io/schemas/shipyard/2", - "header": { - "uploaderID": "Bill", - "softwareName": "My excellent app", - "softwareVersion": "0.0.1" - }, - "message": { - "systemName": "Munfayl", - "stationName": "Samson", - "marketId": 128023552, - "horizons": true, - "timestamp": "2019-01-08T06:39:43Z", - "ships": [ - "anaconda", - "dolphin", - "eagle", - "ferdelance", - "hauler", - "krait_light", - "krait_mkii", - "mamba", - "python", - "sidewinder" - ] - } -} -``` - -### Contents of `message` -Every message MUST comply with the Schema its `$schemaRef` value cites. Each -Schema file should have a matching `-README.md` file in the project -root `schemas/` directory. Always consult this so that you're aware of any -Schema-specific requirements. - -The Schema file, `-v.json`, is considered the authority on -the format of messages for that Schema. If anything in the accompanying -documentation is in disagreement with this then please -[open an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose) -so that we can investigate and correct, or clarify, as necessary. - -Apart from short time windows during deployment of a new version the live -EDDN service should always be using -[the Schemas as present in the live branch](https://github.com/EDCD/EDDN/tree/live/schemas). -So, be sure you're checking the live versions and not, e.g. those in the -`master` or other branches. - -1. Each `message` object must have, at bare minimum: - - 1. `timestamp` - string date and time in ISO8601 format. Whilst this - technically allows for any timezone to be cited you SHOULD provide this in - UTC, aka 'Zulu Time' as in the example above. You MUST ensure that you are - doing this properly. Do not claim 'Z' whilst actually using a local time - that is offset from UTC. - - If you are only utilising Journal-sourced data then simply using the - value from there should be sufficient as the PC game client is meant to - always be correctly citing UTC for this. Indeed it has been observed, - in the Odyssey 4.0.0.1002 client, that with the Windows clock behind UTC - by 21 seconds both the in-game UI clock *and* the Journal event - timestamps are still properly UTC to the nearest second. - - Listeners MAY make decisions on accepting data based on this time stamp, - i.e. "too old". - 2. Where the data is sourced from a Journal event please do preserve the - `event` key and value. Yes, where we use an event-specific Schema this - might seem redundant, but it might aid an EDDN listener in streamlining - their code, and it does no harm. - - Any new Schema based on Journal data **MUST** make `event` a required - property of the `message` dictionary. - 3. At least one other key/value pair representing the data. In general there - will be much more than this. Consult the - [Schemas and their documentation](./). - 4. Include - [horizons and odyssey flags](#horizons-and-odyssey-flags) where the - Schema requires them. - -2. Because the first versions of some Schemas were defined when only the CAPI - data was available, before Journal files existed, many of the key names chosen - in the Schemas are based on the equivalent in CAPI data, not Journal events. - This means you MUST rename many of the keys from Journal events to match the - Schemas. - -3. EDDN is intended to transport generic data not specific to any particular Cmdr - and to reflect only the data that every player would see in-game in station - services or the local map. To that end: - 1. Uploading applications MUST ensure that messages do not contain any - Cmdr-specific data (other than "uploaderID", the "horizons" flag, and - the "odyssey" flag). - 2. Uploading applications MUST remove any data where the name of the - relevant key has a `_Localised` suffix. - - The individual Schemas will instruct you on various elisions (removals) to - be made to comply with this. - -Some of these requirements are also enforced by the Schemas, and some things -the Schemas enforce might not be explicitly called out here. So, **do** -check what you're sending against the relevant Schema(s) when making any -changes to your code. - -It is also advisable to Watch this repository on GitHub so that you are aware -of any changes to Schemas. - -#### `horizons` and `odyssey` flags - -Where the Schema allows for them, add the `horizons` and `odyssey` -keys with boolean values. `null` is not allowed in the values, so **if -you cannot determine a value do not include that key at all**. - -The only source of these is the `LoadGame` event from journals. It's present -both in the PC local files and the CAPI journal data. If you're -composing a shipyard or outfitting message from CAPI data then it is -possible to synthesise the `horizons` flag. For now consult the function -`capi_is_horizons()` in -[EDMarketConnector:plugins/eddn.py](https://github.com/EDCD/EDMarketConnector/blob/stable/plugins/eddn.py) -for a method to achieve this. - -As of 2022-01-29 the following was observed for the `LoadGame` events as -present in CAPI-sourced Journal files (which were confirmed to match the -PC-local files for these events): - -- PC Odyssey Client, game version `4.0.0.1100`: - ```json - { "timestamp":"2022-01-29T16:17:02Z", "event":"LoadGame", "FID":"", "Commander":"", "Horizons":true, "Odyssey":true,... - ``` -- PC Horizons Client, game version `3.8.0.404`, no `Odyssey` key was - present: - ```json - { "timestamp":"2022-01-29T16:15:07Z", "event":"LoadGame", "FID":"", "Commander":"", "Horizons":true,... - ``` - -- PC 'base' Client, game version `3.8.0.404`, no `Odyssey` key was - present: - ```json - { "timestamp":"2022-01-29T16:11:54Z", "event":"LoadGame", "FID":"", "Commander":"", "Horizons":false,... - ``` - -Do not attempt to use the value(s) from a `Fileheader` event as the semantics -are different. With clients 3.8.0.404 and 4.0.0.1100 the following was observed: - -| Game Client | Fileheader | LoadGame | -| ---------------: | ---------------- | ------------------------------: | -| Base | "Odyssey":false | "Horizons":false | -| Horizons | "Odyssey":false | "Horizons":true | -| Odyssey | "Odyssey":true | "Horizons":true, "Odyssey":true | - -NB: The 'Base' client appears to simply be the Horizons client with any -Horizons-only features disabled. - -- In the `Fileheader` event it's indicating whether it's an Odyssey game client. -- In the `LoadGame` it's indicating whether Horizons and/or Odyssey features are - active, but in the non-Odyssey game client case you only get the Horizons - boolean. - -### Server responses -There are three possible sources of HTTP responses when sending an upload -to EDDN. - -1. The reverse proxy that initially accepts the request. -2. The python `bottle` module that the Gateway uses to process the - forwarded requests. This might object to a message before the actual - EDDN code gets to process it at all. -3. The actual EDDN Gateway code. - -Once a message has cleared the EDDN Gateway then there is no mechanism for any -further issue (such as a message being detected as a duplicate in the -Monitor downstream of the Gateway) to be reported back to the sender. - -To state the obvious, if there are no issues with a request then an HTTP -200 response will be received by the sender. The body of the response -should be the string `OK`. - -#### Reverse Proxy responses -In addition to generic "you typoed the URL" and other such "you just didn't -make a valid request" responses you might experience the following: - -1. `408` - `Request Timed Out` - the sender took too long to make/complete - its request and the reverse proxy rejected it as a result. -2. `503` - `Service Unavailable` - the EDDN Gateway process is either not - running, or not responding. -3. `400` - `Bad Request` - if you attempt to use plain HTTP, rather than - HTTPS. - -#### bottle responses -1. `413` - `Payload Too Large` - `bottle` enforces a maximum request size - and the request exceeds that. As of 2022-01-07 the limit is 1MiB, - which is versus the compressed size of the body, if compression is - used. Thus compression *will* allow for sending approximately 10x - larger messages. - To verify the current limit check for the line that looks like: - - ``` - bottle.BaseRequest.MEMFILE_MAX = 1024 * 1024 # 1MiB, default is/was 100KiB - ``` - - in - [src/eddn/Gateway.py](https://github.com/EDCD/EDDN/blob/live/src/eddn/Gateway.py), - as added in - [commit 0e80c76cb564771465f61825e694227dcc3be312](https://github.com/EDCD/EDDN/commit/0e80c76cb564771465f61825e694227dcc3be312). - -#### EDDN Gateway responses -For all failures the response body will contain text that begins `FAIL: `. Currently two different HTTP status codes are utilised: - -1. `400` - `Bad Request` - This indicates something wrong with the request - body. Possibly due to a format issue (compression, form encoding), or - the actual content of the EDDN message: - 1. `FAIL: zlib.error: ` - A failure to decompress a message that - claimed to be compressed. - - 2. `FAIL: Malformed Upload: ` - the message appeared to be - form-encoded, but either the format was bad or there was no `data` - key. - - 3. `FAIL: JSON parsing: ` - the - message couldn't be parsed as valid JSON. e.g. - - ``` - FAIL: JSON parsing: Expecting property name enclosed in double quotes: line 1 column 2 (char 1) - ``` - - 4. `FAIL: Schema Validation: ` - the message failed to validate - against the cited Schema. e.g. - - ``` - FAIL: Schema Validation: [] - ``` - - The exact detail will be very much dependent on both the Schema the - message cited and the contents of the message that failed to pass the - validation. - - In particular, if the message contains a key that is tagged 'disallowed' in - the Schema, then the response will look like: - - ``` - FAIL: Schema Validation: "[]" - ``` - This is due to the use of a JSON schema stanza that says "don't allow - any valid type for the value of this key" to trigger the error for such - disallowed keys. - - Note how the **value** for the disallowed key is cited, not the key *name* - itself. This is a limitation of how the `jsonschema` python module - reports errors, and we are - [hoping to improve this](https://github.com/EDCD/EDDN/issues/163). - -2. `426` - `Upgrade Required` - This indicates that the cited Schema, or - version thereof, is outdated. The body of the response will be: - - ``` - FAIL: Oudated Schema: The schema you have used is no longer supported. Please check for an updated version of your application. - ``` - The wording here is aimed at users of applications that send messages - over EDDN. If you're the developer of such an application then - obviously you need to update your code to use a currently supported - Schema and version thereof. - - -There shouldn't be any other variants of a 'FAIL' message. If you find -any then please -[open an issue on GitHub](https://github.com/EDCD/EDDN/issues/new) -with as much detail as possible so that we can update this documentation. - -## Receiving messages - -EDDN provides a continuous stream of information from uploaders. To use this -data you'll need to connect to the stream using ZeroMQ (a library is probably -available for your language of choice). - -The URL for the live Relay is: - - tcp://eddn.edcd.io:9500 - -Depending on the programming language and library used, you might need -to explicitly subscribe to an empty topic, `''`, in order to receive -anything. - -Unfortunately at this time we're using an old version of -ZeroMQ which does not support server-side subscription filtering. So by -all means specify a more specific topic filter if it suits your needs, -but realise that the dropping of not-matching messages will happen at -your end, not the server. - -Once you've connected to that you will receive messages. To access the -data you will first need to zlib-decompress each message. Then you will -have a textual JSON object as per the Schemas. - -In general, check the guidance for [Uploading messages](#uploading-messages) -for the expected format of the messages. - -Consumers can utilise the `$schemaRef` value to determine which Schema a -particular message is for. There is no need to validate the messages -against the Schemas yourself, as that is performed on the EDDN Gateway. -Messages that do not pass the schema validation there are not forwarded to -receivers. - -There is [example code](https://github.com/EDCD/EDDN/tree/master/examples) -available for a variety of programming languages to help you get started. +For more general documentation that all developers wanting to either Upload +messages or Listen to the stream of messages from the Relay, please consult +[the Developer documentation](../docs/Developers.md).