diff --git a/schemas/README-EDDN-schemas.md b/schemas/README-EDDN-schemas.md new file mode 100644 index 0000000..550a56e --- /dev/null +++ b/schemas/README-EDDN-schemas.md @@ -0,0 +1,208 @@ +# EDDN Schemas Documentation + +## Introduction + +EDDN is a +[zermoq](https://zeromq.org/) service to allow players of the game +[Elite Dangerous](https://www.elitedangerous.com/), published +by [Frontier Developments](https://www.frontier.co.uk/), to upload game data so +that interested listeners can receive a copy. + +EDDN accepts HTTP POST uploads in a defined format representing this game data +and then passes it on to any interested listeners. + +--- +## Sources + +There are two sources of game data, both provided by the publisher of the game, +Frontier Developerments. They are both explicitly approved for use by +third-party software. + +### Journal Files + +On the PC version of the game, "Journal files" are written during any game +session. These are in newline-delimited JSON format, with each line +representing a single JSON object. Frontier Developments publishes +documentation for the various events in their +[Player Tools & API Discussions](https://forums.frontier.co.uk/forums/elite-api-and-tools/) +forum. + +In general the documentation is made available in a file named something like: + + Journal_Manual-v + +as both a MicroSoft word `.doc` file, or a `.pdf` file. Historically the +use of `_` versus `-` in those filenames has varied. + +Consult the latest of these for documentation on individual events. +However, be aware that sometimes the documentation is in error, possibly due to +not having been updated after a game client change. + +### Companion API (CAPI) data + +Frontier Developments provides an API to retrieve certain game data, even +without the game running. Historically this was for use by its short-lived +iOS "Companion" app, and was only intended to be used by that app. There was no +public documentation, or even admission of its existence. + +Eventually, after some enterprising players had snooped the connections and +figured out the login method and endpoints, Frontier Developments +[allowed general use of this](https://forums.frontier.co.uk/threads/open-letter-to-frontier-developments.218658/page-19#post-3371472) +. + +Originally the API authentication required being supplied with the email and +password as used to login to the game (but at least this was over HTTPS). + +In late 2018 Frontier switched the authentication to using an oAuth2 flow, +meaning players no longer need to supply their email and password to +third-party sites and clients. + +As of October 2021 there has still never been any official documentation about +the available endpoints and how they work. There is some +[third-party documentation](https://github.com/Athanasius/fd-api/blob/main/docs/README.md) +by Athanasius. + +When using the Companion API please be aware that the server that supplies this +data sometimes lags behind the game - usually by a few seconds, sometimes by +minutes. You MUST check in the data from the CAPI that the Cmdr is +docked (`["commander"]["docked"]` is `True`) and that the station and +system (`["lastStarport"]["name"]` and `["lastSystem"]["name"]`) match those +reported from the Journal before using the data for the commodity, outfitting +and shipyard schemas. + +--- + +## Uploading messages + +### Send only live data to the live schemas +You MUST **NOT** send information from any non-live (e.g. alpha or beta) +version of the game to the main schemas on this URL. + +You MAY send such to this URL so long as you append `/test` to the `$schemaRef` +value, e.g. + + "$schemaRef": "https://eddn.edcd.io/schemas/shipyard/2/test", + +You MUST also utilise these test forms of the schemas when first testing your +code. There might also be a beta.eddn.edcd.io, or dev.eddn.edcd.io, service +available from time to time as necessary, e.g. for testing new schemas or +changes to existing ones. + +### Sending data +To upload market data to EDDN, you'll need to make a POST request to the URL: + +* https://eddn.edcd.io:4430/upload/ + +The body of this is a JSON object, so you SHOULD set a `Content-Type` header of +`applicaton/json`, and NOT any of: + +* `application/x-www-form-urlencoded` +* `multipart/form-data` +* `text/plain` + +### Format of uploaded messages +Each message is a JSON object in utf-8 encoding containing the following +key+value pairs: + +1. `$schemaRef` - Which schema (including version) this message is for. +2. `header` - Object containing mandatory information about the upload; + 1. `uploaderID` - a unique ID for the player uploading this data. + Don't worry about privacy, the EDDN service will hash this with a key + that is regularly changed so no-one knows who an uploader is in-game. + 2. `softwareName` - an identifier for the software performing the upload. + 3. `softwareVersion` - The version of that software being used. + + Listeners MAY make decisions about whether to utilise the data in any + message based on the combination of `softwareName` and `softwareVersion`. + + **DO not** add `gatewaytimestamp` yourself. The EDDN Gateway will add + this and will overwrite any that you provide, so don't bother. +4. `message` - Object containing the data for this message. Consult the + relevant README file within this documentation, e.g. + [codexentry-README.md](./codexentry-README.md). There are some general + guidelines [below](#contents-of-message). + +For example, a shipyard message, version 2, might look like: + +```JSON +{ + "$schemaRef": "https://eddn.edcd.io/schemas/shipyard/2", + "header": { + "uploaderID": "Bill", + "softwareName": "My excellent app", + "softwareVersion": "0.0.1" + }, + "message": { + "systemName": "Munfayl", + "stationName": "Samson", + "marketId": 128023552, + "horizons": true, + "timestamp": "2019-01-08T06:39:43Z", + "ships": [ + "anaconda", + "dolphin", + "eagle", + "ferdelance", + "hauler", + "krait_light", + "krait_mkii", + "mamba", + "python", + "sidewinder" + ] + } +} +``` + +### Contents of `message` + +Each `message` object must have, at bare minimum: + +1. `timestamp` - string date and time in ISO8601 format. Whilst that + technically allows for any timezone to be cited you SHOULD provide this in + UTC, aka 'Zulu Time' as in the example above. You MUST ensure that you are + doing this properly. Do not claim 'Z' whilst actually using a local time + that is offset from UTC. + + Listeners MAY make decisions on accepting data based on this time stamp, + i.e. "too old". +2. One other key/value pair representing the data. In general there will be + much more than this. Again, consult the + [schemas and their documentation](./). + +EDDN is intended to transport generic data not specific to any particular Cmdr +and to reflect the data that a player would see in-game in station services or +the local map. To that end, uploading applications MUST ensure that messages do +not contain any Cmdr-specific data (other than "uploaderID" and the "horizons" +flag). In practice as of E:D 3.3 this means: + +* commodity: Skip commodities with `"categoryname": "NonMarketable"` (i.e. + Limpets - not purchasable in station market) or `"legality":` *non-empty + string* (not normally traded at this station market). +* outfitting: Skip items whose availability depends on the Cmdr's status rather + than on the station. Namely: + - Items that aren't weapons/utilities (`Hpt_*`), standard/internal + modules (`Int_*`) or armour (`*_Armour_*`) (i.e. bobbleheads, decals, + paintjobs and shipkits). + - Items that have a non-null `"sku"` property, unless + it's `"ELITE_HORIZONS_V_PLANETARY_LANDINGS"` (i.e. PowerPlay and tech + broker items). + - The `"Int_PlanetApproachSuite"` module (for historical reasons). +* shipyard: *Include* ships listed in the `"unavailable_list"` property (i.e. + available at this station, but not to this Cmdr). +* journal: Strip out `"..._Localised"` properties throughout the data + structure. +* journal/Docked: Strip out `"Wanted"`, `"ActiveFine"`, `"CockpitBreach"` + properties +* journal/FSDJump: Strip out `"Wanted"`, `"BoostUsed"`, `"FuelLevel"` + , `"FuelUsed"` and `"JumpDist"` properties. +* journal/Location: Strip out `"Wanted"`, `"Latitude"` and `"Longitude"` + properties. +* journal/Location and journal/FSDJump: strip out `"HappiestSystem"` + , `"HomeSystem"`, `"MyReputation"` and `"SquadronFaction"` properties within + the list of `"Factions"`. + +Some of these requirements are also enforced by the schemas, and some things +the schemas enforce might not be explicitly called out here, so **do** +check what you're sending against the schema when implementing sending new +events. diff --git a/schemas/codexentry-README.md b/schemas/codexentry-README.md new file mode 100644 index 0000000..69b6e4a --- /dev/null +++ b/schemas/codexentry-README.md @@ -0,0 +1,118 @@ +# EDDN CodexEntry Schema + +## Introduction +Here we document how to take data from an ED `CodexEntry` Journal Event and +properly structure it for sending to EDDN. + +Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general +documentation for a schema such as this. + +## Senders +The primary data source for this schema is the ED Journal event `CodexEntry`. + +### Elisions +You MUST remove any key where the key name ends in +`_Localised`. + +You MUST remove the two keys `IsNewEntry` and `NewTraitsDiscovered`. + +### Augmentations +#### StarPos +You MUST **add** a `StarPos` key with value of type `array` containing the +galaxy co-ordinates of the system. You will need to have obtained these +from prior event(s) upon the player arriving, or logging into, the system. + +e.g. if the system is `Alpha Centauri`: +```json + "StarPos": [3.03125, -0.09375, 3.15625] +``` + +#### BodyID and BodyName +You SHOULD attempt to track the BodyName and BodyID where the player is +and add keys/values for these. + +You MUST track `BodyName` both from Status.json *and* also from some +[Journal](./README-EDDN-schemas.md#journal-files) +events in order to cross-check it before using the `BodyID` from +[Journal](./README-EDDN-schemas.md#journal-files) events. + +The following is correct as of game version 4.0.0.801 (Odyssey initial +release, Update 7, plus one patch). + +1. Record `journal_body_name` and `journal_body_id` from the + `BodyName` and `BodyID` values in `ApproachBody` events. + + This will occur when the player flies below Orbital Cruise altitude + around a body. +2. Also record these from `Location` events to cover logging in already there. +3. Unset both `journal_body_name` and `journal_body_id` on `LeaveBody` and + `FSDJump` events. + Do NOT do so for `SupercruiseEntry`, because a player can enter supercruise + below max Orbital Cruise altitude and then come back down without a new + `ApproachBody` event occurring. +4. If Status.json has `BodyName` present, record that as `status_body_name`. + + This key and its value will be present whenever the player comes close + enough to a body for the Orbital Cruise/Glide HUD elements to appear. + It will disappear again when they fly back above that altitude, or jump + away. +5. If Status.json does **not** have `BodyName` then clear `status_body_name`. +6. For a `CodexEntry` event: + 1. Check that `status_body_name` is set. If it is not, exit. + 1. Set the EDDN `codexentry` schema message `BodyName` to this value. + 2. Check if it matches the `journal_body_name` value, and + ONLY if they match, set `BodyID` in the EDDN `codexentry` + schema message to the value of `journal_body_id`. + + If `status_body_name` is not set then you MUST NOT include `BodyName` or + `BodyID` keys/values in the EDDN message. + + If `status_body_name` is set, but does not match with + `journal_body_name` then you MUST NOT include a `BodyID` key+value in the + EDDN message. + + For emphasis, in both of these cases you MUST NOT include the keys with a + `null`, `''`, or otherwise 'empty' value. Do not include the key(s) at all. + +One possible issue is binary bodies where you might get an `ApproachBody` for +one before descending towards the other, without an additional `ApproachBody` +to correct things. + +An example of this is `Baliscii 7 a` and `Baliscii 7 b`. Approaching one +and going below Orbital Cruise altitude will set `journal_body_name` and +`journal_body_id` to it, but you can then turn and approach the other +without a new `ApproachBody` event, but `status_body_name` will change to +the other when you are close enough. + +In this case due to `status_body_name` and `journal_body_name` not matching +the `codexentry` message MUST be sent **without** `BodyID`, but SHOULD be +sent with the `status_body_name` value on the `BodyName` key. + +e.g. for `Bestia A 2 a` +```json + "BodyName": "Bestia A 2 a", + "BodyID": 15, +``` + +If you cannot properly obtain the values for `BodyName` or `BodyID` then +you MUST NOT include them. + +## Receivers + +As per ['BodyID and BodyName'](#bodyid-and-bodyname) above be aware that +you are not guaranteed to receive these values for any given event. Some +codex entries will be in space, and thus they aren't even relevant. In +other cases it may not have been possible to properly determine both of them. + +So you might receive any of: + +1. Neither `BodyName` nor `BodyID` present in the message, not even the + key names. This SHOULD indicate a codex entry object which is not on a + body surface. +2. `BodyName` key present with a value, but no `BodyID` key. This SHOULD + indicate a codex entry object which is on a body surface, but probably + where there is a close-orbiting binary companion which has confused things. +3. Both `BodyName` and `BodyID` keys present, with values. This SHOULD + indicate a codex entry object which is on a body surface. + +Adjust your local processing accordingly. \ No newline at end of file diff --git a/schemas/codexentry-v1.0.json b/schemas/codexentry-v1.0.json index 98a4637..4a4727b 100644 --- a/schemas/codexentry-v1.0.json +++ b/schemas/codexentry-v1.0.json @@ -1,6 +1,7 @@ { "$schema" : "http://json-schema.org/draft-04/schema#", "id" : "https://eddn.edcd.io/schemas/codexentry/1#", + "description" : "EDDN schema for CodexEntry Journal events. Full documentation at https://github.com/EDCD/EDDN/tree/master/schemas/codexentry-README.md", "type" : "object", "additionalProperties" : false, "required": [ "$schemaRef", "header", "message" ], @@ -104,6 +105,12 @@ "minLength" : 1 } }, + "BodyID": { + "type" : "integer" + }, + "BodyName": { + "type" : "string" + }, "IsNewEntry": { "$ref" : "#/definitions/disallowed", "description" : "Contains personal data" diff --git a/schemas/scanbarycentre-v1.0.json b/schemas/scanbarycentre-v1.0.json index 960d50d..15a8985 100644 --- a/schemas/scanbarycentre-v1.0.json +++ b/schemas/scanbarycentre-v1.0.json @@ -35,7 +35,7 @@ "additionalProperties" : false, "required" : [ "timestamp", "event", "StarSystem", "StarPos", "SystemAddress", "BodyID" ], "properties" : { - "timestamp":{ + "timestamp": { "type" : "string", "format" : "date-time" }, diff --git a/src/eddn/core/Validator.py b/src/eddn/core/Validator.py index a24b32c..fae15af 100644 --- a/src/eddn/core/Validator.py +++ b/src/eddn/core/Validator.py @@ -13,8 +13,12 @@ class Validator(object): def addSchemaResource(self, schemaRef, schema): if schemaRef in self.schemas.keys(): raise Exception("Attempted to redefine schema for " + schemaRef) - schema = simplejson.loads(schema) - self.schemas[schemaRef] = schema + try: + schema = simplejson.loads(schema) + self.schemas[schemaRef] = schema + + except simplejson.errors.JSONDecodeError as e: + raise Exception('SCHEMA: Failed to load: ' + schemaRef) def validate(self, json_object): results = ValidationResults()