third-party-mirrors/EDDN
1
0
Fork 0
mirror of https://github.com/EDCD/EDDN.git synced 2025-04-20 18:47:39 +03:00
Code Issues Actions Packages Projects Releases Wiki Activity

schemas/README: Flesh out this in its new 'stub' form

Browse Source 
The remaining extant text has also now been moved into `docs/Developers.md`.
This commit is contained in:
Athanasius 2022-01-30 12:55:33 +00:00
parent 3e69b7b630
commit c9d0e30e42
No known key found for this signature in database
GPG Key ID: AE3E527847057C7D
1 changed files with 16 additions and 370 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

386
schemas/README-EDDN-schemas.md
View File

@ -1,377 +1,23 @@
# EDDN Schemas Documentation
### Sending data
Messages sent to EDDN **MUST**:
EDDN message Schemas are [JSON](https://www.json.org/json-en.html) files
conforming to 'draft 04' of the [JSON Schema](https://json-schema.org/)
specification.
- Use the URL: `https://eddn.edcd.io:4430/upload/`. Note the use of
TLS-encrypted HTTPS. A plain HTTP request will elicit a `400 Bad
Request` response.
- Use the HTTP 1.1 protocol. HTTP/2 is not supported at this time.
- Use a **POST** request, with the body containing the EDDN message. No
query parameters in the URL are supported or necessary.
## Canonical location of Schema files
The body of an EDDN message is a JSON object in UTF-8 encoding. If you do not
compress this body then you MUST set a `Content-Type` header of
`applicaton/json`.
For the EDDN Live service you should always be checking
[the live version of the schemas, and their READMEs](https://github.com/EDCD/EDDN/tree/live/schemas).
Any other version of the Schemas is not guaranteed to be synchronized with
those actually running on the Live service.
For historical reasons URL form-encoded data *is* supported, **but this is
deprecated and no new software should attempt this method**. We
purposefully do not further document the exact format for this.
## Documentation of Schema files
You *MAY* use gzip compression on the body of the message, but it is not
required. If you do compress the body then you **MUST* send a `Content-Type`
header of `gzip` instead of `application/json`.
The Schema files themselves are considered to be the canonical definition of
the required, and allowed, contents of the relevant EDDN message. There
**SHOULD** be an accompanying README file, e.g. for `commodity-v3.0.json` there
is also a `commodity-README.md` file in the project root `schemas/` directory.
You should be prepared to handle all scenarios where sending of a message
fails:
1. Connection refused.
2. Connection timed out.
3. Other possible responses as documented in
[Server responses](#server-responses).
Carefully consider whether you should queue a 'failed' message for later
retry. In particular, you should ensure that one 'bad' message does not
block other messages from being successfully sent.
You **MUST** wait some reasonable time (minimum 1 minute) before retrying
any failed message.
You **MUST NOT** retry any message that received a HTTP `400` or `426` code.
An exception can be made if, **and only if**, *you have manually verified that
you have fixed the issues with it (i.e. updated the Schema/version to a
currently supported one and adjusted the data to fit that Schema/version).*
You **MAY** retry a message that initially received a `413` response (in
the hopes that the EDDN service admins decided to increase the maximum
allowed request size), but should not do so too quickly or in perpetuity.
In general:
- No data is better than bad data.
- *Delayed* good data is better than degrading the EDDN service for others.
### Format of uploaded messages
Each message is a JSON object in UTF-8 encoding containing the following
key+value pairs:
1. `$schemaRef` - Which Schema (including version) this message is for.
2. `header` - Object containing mandatory information about the upload;
1. `uploaderID` - a unique ID for the player uploading this data.
Don't worry about privacy, the EDDN service will hash this with a key
that is regularly changed so no-one knows who an uploader is in-game.
2. `softwareName` - an identifier for the software performing the upload.
3. `softwareVersion` - The version of that software being used.
Listeners MAY make decisions about whether to utilise the data in any
message based on the combination of `softwareName` and `softwareVersion`.
**DO not** add `gatewaytimestamp` yourself. The EDDN Gateway will add
this and will overwrite any that you provide, so don't bother.
4. `message` - Object containing the data for this message. Consult the
relevant README file within this documentation, e.g.
[codexentry-README.md](./codexentry-README.md). There are some general
guidelines [below](#contents-of-message).
For example, a shipyard message, version 2, might look like:
```JSON
{
"$schemaRef": "https://eddn.edcd.io/schemas/shipyard/2",
"header": {
"uploaderID": "Bill",
"softwareName": "My excellent app",
"softwareVersion": "0.0.1"
},
"message": {
"systemName": "Munfayl",
"stationName": "Samson",
"marketId": 128023552,
"horizons": true,
"timestamp": "2019-01-08T06:39:43Z",
"ships": [
"anaconda",
"dolphin",
"eagle",
"ferdelance",
"hauler",
"krait_light",
"krait_mkii",
"mamba",
"python",
"sidewinder"
]
}
}
```
### Contents of `message`
Every message MUST comply with the Schema its `$schemaRef` value cites. Each
Schema file should have a matching `<schema>-README.md` file in the project
root `schemas/` directory. Always consult this so that you're aware of any
Schema-specific requirements.
The Schema file, `<schema>-v<version>.json`, is considered the authority on
the format of messages for that Schema. If anything in the accompanying
documentation is in disagreement with this then please
[open an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose)
so that we can investigate and correct, or clarify, as necessary.
Apart from short time windows during deployment of a new version the live
EDDN service should always be using
[the Schemas as present in the live branch](https://github.com/EDCD/EDDN/tree/live/schemas).
So, be sure you're checking the live versions and not, e.g. those in the
`master` or other branches.
1. Each `message` object must have, at bare minimum:
1. `timestamp` - string date and time in ISO8601 format. Whilst this
technically allows for any timezone to be cited you SHOULD provide this in
UTC, aka 'Zulu Time' as in the example above. You MUST ensure that you are
doing this properly. Do not claim 'Z' whilst actually using a local time
that is offset from UTC.
If you are only utilising Journal-sourced data then simply using the
value from there should be sufficient as the PC game client is meant to
always be correctly citing UTC for this. Indeed it has been observed,
in the Odyssey 4.0.0.1002 client, that with the Windows clock behind UTC
by 21 seconds both the in-game UI clock *and* the Journal event
timestamps are still properly UTC to the nearest second.
Listeners MAY make decisions on accepting data based on this time stamp,
i.e. "too old".
2. Where the data is sourced from a Journal event please do preserve the
`event` key and value. Yes, where we use an event-specific Schema this
might seem redundant, but it might aid an EDDN listener in streamlining
their code, and it does no harm.
Any new Schema based on Journal data **MUST** make `event` a required
property of the `message` dictionary.
3. At least one other key/value pair representing the data. In general there
will be much more than this. Consult the
[Schemas and their documentation](./).
4. Include
[horizons and odyssey flags](#horizons-and-odyssey-flags) where the
Schema requires them.
2. Because the first versions of some Schemas were defined when only the CAPI
data was available, before Journal files existed, many of the key names chosen
in the Schemas are based on the equivalent in CAPI data, not Journal events.
This means you MUST rename many of the keys from Journal events to match the
Schemas.
3. EDDN is intended to transport generic data not specific to any particular Cmdr
and to reflect only the data that every player would see in-game in station
services or the local map. To that end:
1. Uploading applications MUST ensure that messages do not contain any
Cmdr-specific data (other than "uploaderID", the "horizons" flag, and
the "odyssey" flag).
2. Uploading applications MUST remove any data where the name of the
relevant key has a `_Localised` suffix.
The individual Schemas will instruct you on various elisions (removals) to
be made to comply with this.
Some of these requirements are also enforced by the Schemas, and some things
the Schemas enforce might not be explicitly called out here. So, **do**
check what you're sending against the relevant Schema(s) when making any
changes to your code.
It is also advisable to Watch this repository on GitHub so that you are aware
of any changes to Schemas.
#### `horizons` and `odyssey` flags
Where the Schema allows for them, add the `horizons` and `odyssey`
keys with boolean values. `null` is not allowed in the values, so **if
you cannot determine a value do not include that key at all**.
The only source of these is the `LoadGame` event from journals. It's present
both in the PC local files and the CAPI journal data. If you're
composing a shipyard or outfitting message from CAPI data then it is
possible to synthesise the `horizons` flag. For now consult the function
`capi_is_horizons()` in
[EDMarketConnector:plugins/eddn.py](https://github.com/EDCD/EDMarketConnector/blob/stable/plugins/eddn.py)
for a method to achieve this.
As of 2022-01-29 the following was observed for the `LoadGame` events as
present in CAPI-sourced Journal files (which were confirmed to match the
PC-local files for these events):
- PC Odyssey Client, game version `4.0.0.1100`:
```json
{ "timestamp":"2022-01-29T16:17:02Z", "event":"LoadGame", "FID":"<elided>", "Commander":"<elided>", "Horizons":true, "Odyssey":true,...
```
- PC Horizons Client, game version `3.8.0.404`, no `Odyssey` key was
present:
```json
{ "timestamp":"2022-01-29T16:15:07Z", "event":"LoadGame", "FID":"<elided>", "Commander":"<elided>", "Horizons":true,...
```
- PC 'base' Client, game version `3.8.0.404`, no `Odyssey` key was
present:
```json
{ "timestamp":"2022-01-29T16:11:54Z", "event":"LoadGame", "FID":"<elided>", "Commander":"<elided>", "Horizons":false,...
```
Do not attempt to use the value(s) from a `Fileheader` event as the semantics
are different. With clients 3.8.0.404 and 4.0.0.1100 the following was observed:
| Game Client | Fileheader | LoadGame |
| ---------------: | ---------------- | ------------------------------: |
| Base | "Odyssey":false | "Horizons":false |
| Horizons | "Odyssey":false | "Horizons":true |
| Odyssey | "Odyssey":true | "Horizons":true, "Odyssey":true |
NB: The 'Base' client appears to simply be the Horizons client with any
Horizons-only features disabled.
- In the `Fileheader` event it's indicating whether it's an Odyssey game client.
- In the `LoadGame` it's indicating whether Horizons and/or Odyssey features are
active, but in the non-Odyssey game client case you only get the Horizons
boolean.
### Server responses
There are three possible sources of HTTP responses when sending an upload
to EDDN.
1. The reverse proxy that initially accepts the request.
2. The python `bottle` module that the Gateway uses to process the
forwarded requests. This might object to a message before the actual
EDDN code gets to process it at all.
3. The actual EDDN Gateway code.
Once a message has cleared the EDDN Gateway then there is no mechanism for any
further issue (such as a message being detected as a duplicate in the
Monitor downstream of the Gateway) to be reported back to the sender.
To state the obvious, if there are no issues with a request then an HTTP
200 response will be received by the sender. The body of the response
should be the string `OK`.
#### Reverse Proxy responses
In addition to generic "you typoed the URL" and other such "you just didn't
make a valid request" responses you might experience the following:
1. `408` - `Request Timed Out` - the sender took too long to make/complete
its request and the reverse proxy rejected it as a result.
2. `503` - `Service Unavailable` - the EDDN Gateway process is either not
running, or not responding.
3. `400` - `Bad Request` - if you attempt to use plain HTTP, rather than
HTTPS.
#### bottle responses
1. `413` - `Payload Too Large` - `bottle` enforces a maximum request size
and the request exceeds that. As of 2022-01-07 the limit is 1MiB,
which is versus the compressed size of the body, if compression is
used. Thus compression *will* allow for sending approximately 10x
larger messages.
To verify the current limit check for the line that looks like:
```
bottle.BaseRequest.MEMFILE_MAX = 1024 * 1024 # 1MiB, default is/was 100KiB
```
in
[src/eddn/Gateway.py](https://github.com/EDCD/EDDN/blob/live/src/eddn/Gateway.py),
as added in
[commit 0e80c76cb564771465f61825e694227dcc3be312](https://github.com/EDCD/EDDN/commit/0e80c76cb564771465f61825e694227dcc3be312).
#### EDDN Gateway responses
For all failures the response body will contain text that begins `FAIL: `. Currently two different HTTP status codes are utilised:
1. `400` - `Bad Request` - This indicates something wrong with the request
body. Possibly due to a format issue (compression, form encoding), or
the actual content of the EDDN message:
1. `FAIL: zlib.error: <detail>` - A failure to decompress a message that
claimed to be compressed.
2. `FAIL: Malformed Upload: <detail>` - the message appeared to be
form-encoded, but either the format was bad or there was no `data`
key.
3. `FAIL: JSON parsing: <detail>` - the
message couldn't be parsed as valid JSON. e.g.
```
FAIL: JSON parsing: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
```
4. `FAIL: Schema Validation: <detail>` - the message failed to validate
against the cited Schema. e.g.
```
FAIL: Schema Validation: [<ValidationError: "'StarPos' is a required property">]
```
The exact detail will be very much dependent on both the Schema the
message cited and the contents of the message that failed to pass the
validation.
In particular, if the message contains a key that is tagged 'disallowed' in
the Schema, then the response will look like:
```
FAIL: Schema Validation: "[<ValidationError: "{'type': ['array', 'boolean', 'integer', 'number', 'null', 'object', 'string']} is not allowed for 'BadVALUE'">]"
```
This is due to the use of a JSON schema stanza that says "don't allow
any valid type for the value of this key" to trigger the error for such
disallowed keys.
Note how the **value** for the disallowed key is cited, not the key *name*
itself. This is a limitation of how the `jsonschema` python module
reports errors, and we are
[hoping to improve this](https://github.com/EDCD/EDDN/issues/163).
2. `426` - `Upgrade Required` - This indicates that the cited Schema, or
version thereof, is outdated. The body of the response will be:
```
FAIL: Oudated Schema: The schema you have used is no longer supported. Please check for an updated version of your application.
```
The wording here is aimed at users of applications that send messages
over EDDN. If you're the developer of such an application then
obviously you need to update your code to use a currently supported
Schema and version thereof.
There shouldn't be any other variants of a 'FAIL' message. If you find
any then please
[open an issue on GitHub](https://github.com/EDCD/EDDN/issues/new)
with as much detail as possible so that we can update this documentation.
## Receiving messages
EDDN provides a continuous stream of information from uploaders. To use this
data you'll need to connect to the stream using ZeroMQ (a library is probably
available for your language of choice).
The URL for the live Relay is:
tcp://eddn.edcd.io:9500
Depending on the programming language and library used, you might need
to explicitly subscribe to an empty topic, `''`, in order to receive
anything.
Unfortunately at this time we're using an old version of
ZeroMQ which does not support server-side subscription filtering. So by
all means specify a more specific topic filter if it suits your needs,
but realise that the dropping of not-matching messages will happen at
your end, not the server.
Once you've connected to that you will receive messages. To access the
data you will first need to zlib-decompress each message. Then you will
have a textual JSON object as per the Schemas.
In general, check the guidance for [Uploading messages](#uploading-messages)
for the expected format of the messages.
Consumers can utilise the `$schemaRef` value to determine which Schema a
particular message is for. There is no need to validate the messages
against the Schemas yourself, as that is performed on the EDDN Gateway.
Messages that do not pass the schema validation there are not forwarded to
receivers.
There is [example code](https://github.com/EDCD/EDDN/tree/master/examples)
available for a variety of programming languages to help you get started.
For more general documentation that all developers wanting to either Upload
messages or Listen to the stream of messages from the Relay, please consult
[the Developer documentation](../docs/Developers.md).