diff --git a/docs/Developers.md b/docs/Developers.md index 6170e6b..ef846ef 100644 --- a/docs/Developers.md +++ b/docs/Developers.md @@ -301,6 +301,48 @@ Horizons-only features disabled. active, but in the non-Odyssey game client case you only get the Horizons boolean. +#### Other data Augmentations +Some schemas mandate that extra data be added, beyond what is in the source +data, to aid Listeners. + +This is usually related to specifying which system an event took place in, and +usually means ensuring there is the full set of: + +1. `StarSystem` - the name of the system. +2. `SystemAddress` - the game's unique numerical identifier for the system. +3. `StarPos` - The system's co-ordinates. + +Whilst it can be argued that any Listener should see preceding event(s) that +give any missing information where at least the system name or `SystemAddress` +is already in the event data, this might not always be true. So Senders MUST +add this data where required. It helps to fill out basic system information +(name, SystemAddress and co-ordinates). + +However, there is a known game bug that can result in it stopping writing to +the game journal, and some observed behaviour implies that it might then later +resume writing to that file, but with events missing. This means any Sender +that blindly assumes it knows the current system/location and uses that for +these Augmentations might send erroneous data. + +1. **Senders MUST cross-check available event data with prior 'location' + event(s) to be sure the correct extra data is being added.** +2. **Listeners SHOULD realise that any data added as an Augmentation might be + in error.** + +For Senders, if the source data only has `SystemAddress` then you MUST check +that it matches that from the prior `Location`, `FSDJump` or `CarrierJump` +event before adding `StarSystem` and `StarPos` data to a message. Drop the +message entirely if it does not match. Apply similar logic if it is only +the system's name that is already present in data. Do not blindly add +`SystemAddress` or `StarPos`. Likewise, do not blindly add `StarPos` if the +other data is already in the source, without cross-checking the system name +and `SystemAddress`. + +Listeners might be able to apply their own cross-check on received messages, +and use any mismatch with respect to what they already know to make a decision +whether to trust the augmented data. Flagging it for manual review is probably +wise. + ### Server responses There are three possible sources of HTTP responses when sending an upload to EDDN. @@ -435,7 +477,11 @@ data you will first need to zlib-decompress each message. Then you will have a textual JSON object as per the Schemas. In general, check the guidance for [Uploading messages](#uploading-messages) -for the expected format of the messages. +for the expected format of the messages. **Pay particular attention to any +schema-specific Augmentations**. Whilst Senders MUST make every effort to +ensure such data is correct it is possible that bugs in either their code, or +the game itself, could mean it is incorrect. Listeners use such data at +their own risk. Consumers can utilise the `$schemaRef` value to determine which Schema a particular message is for. There is no need to validate the messages diff --git a/schemas/TEMPLATES/journalevent-README.md b/schemas/TEMPLATES/journalevent-README.md index f85fc86..890d782 100644 --- a/schemas/TEMPLATES/journalevent-README.md +++ b/schemas/TEMPLATES/journalevent-README.md @@ -106,10 +106,20 @@ value is what the name would have been in the source Journal data. Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags) in the Developers' documentation. +#### StarSystem +You MUST add a `StarSystem` string containing the name of the system from the +last `FSDJump`, `CarrierJump`, or `Location` event. + +**You MUST apply a location cross-check, as per +[Other data augmentations](../docs/Developers.md#other-data-augmentations).** + #### StarPos You MUST add a `StarPos` array containing the system co-ordinates from the last `FSDJump`, `CarrierJump`, or `Location` event. +**You MUST apply a location cross-check, as per +[Other data augmentations](../docs/Developers.md#other-data-augmentations).** + ## Listeners The advice above for [Senders](#senders), combined with the actual Schema file *should* provide all the information you need to process these events. diff --git a/schemas/approachsettlement-README.md b/schemas/approachsettlement-README.md index 13bfcd7..93c8237 100644 --- a/schemas/approachsettlement-README.md +++ b/schemas/approachsettlement-README.md @@ -64,6 +64,12 @@ You MUST add a StarSystem key/value pair representing the name of the system this event occurred in. Source this from either Location, FSDJump or CarrierJump as appropriate. +**You MUST apply a location cross-check, as per +[Other data augmentations](../docs/Developers.md#other-data-augmentations).** + #### StarPos You MUST add a `StarPos` array containing the system co-ordinates from the last `FSDJump`, `CarrierJump`, or `Location` event. + +**You MUST apply a location cross-check, as per +[Other data augmentations](../docs/Developers.md#other-data-augmentations).** diff --git a/schemas/codexentry-README.md b/schemas/codexentry-README.md index 1c675d3..beb6555 100644 --- a/schemas/codexentry-README.md +++ b/schemas/codexentry-README.md @@ -27,6 +27,9 @@ in the Developers' documentation. You MUST add a `StarPos` array containing the system co-ordinates from the last `FSDJump`, `CarrierJump`, or `Location` event. +**You MUST apply a location cross-check, as per +[Other data augmentations](../docs/Developers.md#other-data-augmentations).** + #### BodyID and BodyName You SHOULD attempt to track the BodyName and BodyID where the player is and add keys/values for these. diff --git a/schemas/fssallbodiesfound-README.md b/schemas/fssallbodiesfound-README.md index 350a318..e535b32 100644 --- a/schemas/fssallbodiesfound-README.md +++ b/schemas/fssallbodiesfound-README.md @@ -27,3 +27,6 @@ in the Developers' documentation. #### StarPos You MUST add a `StarPos` array containing the system co-ordinates from the last `FSDJump`, `CarrierJump`, or `Location` event. + +**You MUST apply a location cross-check, as per +[Other data augmentations](../docs/Developers.md#other-data-augmentations).** diff --git a/schemas/fssbodysignals-README.md b/schemas/fssbodysignals-README.md new file mode 100644 index 0000000..10122a3 --- /dev/null +++ b/schemas/fssbodysignals-README.md @@ -0,0 +1,49 @@ +# EDDN FSSAllBodiesFound Schema + +## Introduction +Here we document how to take data from an ED `FSSBodySignals` Journal +Event and properly structure it for sending to EDDN. + +Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general +documentation for a schema such as this. + +If you find any discrepancies between what this document says and what is +defined in the relevant Schema file, then you should, in the first instance, +assume that it is the Schema file that is correct. +**PLEASE open +[an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose) +to report any such anomalies you find so that we can check and resolve the +discrepancy.** + +## Senders +The primary data source for this schema is the ED Journal event +`FSSBodySignals`. + +### Augmentations +#### horizons and odyssey flags +Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags) +in the Developers' documentation. + +#### StarSystem +You MUST add a `StarSystem` string containing the name of the system from the +last `FSDJump`, `CarrierJump`, or `Location` event. + +**You MUST apply a location cross-check, as per +[Other data augmentations](../docs/Developers.md#other-data-augmentations).** + +#### StarPos +You MUST add a `StarPos` array containing the system co-ordinates from the +last `FSDJump`, `CarrierJump`, or `Location` event. + +**You MUST apply a location cross-check, as per +[Other data augmentations](../docs/Developers.md#other-data-augmentations).** + +#### Remove _Localised key/values +All keys whose name ends with `_Localised`, i.e. the `Type_Localised` +key/values in Signals. + +#### Examples: + +```json +{ "timestamp":"2022-05-18T00:10:57Z", "event":"FSSBodySignals", "BodyName":"Phoi Auwsy ZY-Z d132 7 a", "BodyID":37, "SystemAddress":4546986398603, "Signals":[ { "Type":"$SAA_SignalType_Geological;", "Type_Localised":"Geological", "Count":2 } ] } +``` diff --git a/schemas/fssbodysignals-v1.0.json b/schemas/fssbodysignals-v1.0.json new file mode 100644 index 0000000..68ee08e --- /dev/null +++ b/schemas/fssbodysignals-v1.0.json @@ -0,0 +1,96 @@ +{ + "$schema" : "http://json-schema.org/draft-04/schema#", + "id" : "https://eddn.edcd.io/schemas/fssbodysignals/1#", + "type" : "object", + "additionalProperties" : false, + "required": [ "$schemaRef", "header", "message" ], + "properties": { + "$schemaRef": { + "type" : "string" + }, + "header": { + "type" : "object", + "additionalProperties" : true, + "required" : [ "uploaderID", "softwareName", "softwareVersion" ], + "properties" : { + "uploaderID": { + "type" : "string" + }, + "softwareName": { + "type" : "string" + }, + "softwareVersion": { + "type" : "string" + }, + "gatewayTimestamp": { + "type" : "string", + "format" : "date-time", + "description" : "Timestamp upon receipt at the gateway. If present, this property will be overwritten by the gateway; submitters are not intended to populate this property." + } + } + }, + "message": { + "type" : "object", + "description" : "Contains all properties from the listed events in the client's journal, minus the Localised strings and the properties marked below as 'disallowed'", + "additionalProperties" : false, + "required" : [ "timestamp", "event", "StarSystem", "StarPos", "SystemAddress", "BodyID", "Signals" ], + "properties" : { + "timestamp": { + "type" : "string", + "format" : "date-time" + }, + "event" : { + "enum" : [ "FSSBodySignals" ] + }, + "horizons": { + "type" : "boolean", + "description" : "Whether the sending Cmdr has a Horizons pass." + }, + "odyssey": { + "type" : "boolean", + "description" : "Whether the sending Cmdr has an Odyssey expansion." + }, + "StarSystem": { + "type" : "string", + "minLength" : 1 + }, + "StarPos": { + "type" : "array", + "items" : { "type": "number" }, + "minItems" : 3, + "maxItems" : 3, + "description" : "Must be added by the sender if not present in the journal event" + }, + "SystemAddress": { + "type" : "integer" + }, + "BodyID" : { + "type" : "integer" + }, + "BodyName": { + "type" : "string", + "minLength" : 1 + }, + "Signals": { + "type" : "array", + "items" : { + "type" : "object", + "additionalProperties" : false, + "required" : [ "Type", "Count" ], + "properties" : { + "Type" : { + "type" : "string" + }, + "Count" : { + "type" : "integer" + } + } + } + } + } + } + }, + "definitions": { + "disallowed" : { "not" : { "type": [ "array", "boolean", "integer", "number", "null", "object", "string" ] } } + } +} diff --git a/schemas/fssdiscoveryscan-README.md b/schemas/fssdiscoveryscan-README.md index ba5a5eb..f7c8757 100644 --- a/schemas/fssdiscoveryscan-README.md +++ b/schemas/fssdiscoveryscan-README.md @@ -27,3 +27,6 @@ in the Developers' documentation. #### StarPos You MUST add a `StarPos` array containing the system co-ordinates from the last `FSDJump`, `CarrierJump`, or `Location` event. + +**You MUST apply a location cross-check, as per +[Other data augmentations](../docs/Developers.md#other-data-augmentations).** diff --git a/schemas/journal-README.md b/schemas/journal-README.md index 5dd7cea..c2aa572 100644 --- a/schemas/journal-README.md +++ b/schemas/journal-README.md @@ -72,15 +72,20 @@ You SHOULD add this key/value pair, using the value from the `LoadGame` event. You SHOULD add this key/value pair, using the value from the `LoadGame` event. #### StarSystem -You MUST add a `StarSystem` key/value pair representing the name of the -system this event occurred in. Source this from either `Location`, -`FSDJump` or `CarrierJump` as appropriate. +If not already present, you MUST add a `StarSystem` string containing the +name of the system from the last `FSDJump`, `CarrierJump`, or `Location` event. -#### SystemAddress -You MUST add a `SystemAddress` key/value pair representing the numerical ID -of the system this event occurred in. Source this from either `Location`, -`FSDJump` or `CarrierJump` as appropriate. +**You MUST apply a location cross-check, as per +[Other data augmentations](../docs/Developers.md#other-data-augmentations).** + +This should only apply to `SAASignalsFound` events. #### StarPos -You MUST add a `StarPos` array containing the system co-ordinates from the -last `FSDJump`, `CarrierJump`, or `Location` event. +If not already present, you MUST add a `StarPos` array containing the +system co-ordinates from the last `FSDJump`, `CarrierJump`, or `Location` +event. + +**You MUST apply a location cross-check, as per +[Other data augmentations](../docs/Developers.md#other-data-augmentations).** + +This should only apply to `Docked`, `Scan` and `SAASignalsFound` events. diff --git a/schemas/navbeaconscan-README.md b/schemas/navbeaconscan-README.md index 39167a2..09ecdef 100644 --- a/schemas/navbeaconscan-README.md +++ b/schemas/navbeaconscan-README.md @@ -29,6 +29,12 @@ You MUST add a `StarSystem` key/value pair representing the name of the system this event occurred in. Source this from either `Location`, `FSDJump` or `CarrierJump` as appropriate. +**You MUST apply a location cross-check, as per +[Other data augmentations](../docs/Developers.md#other-data-augmentations).** + #### StarPos You MUST add a `StarPos` array containing the system co-ordinates from the last `FSDJump`, `CarrierJump`, or `Location` event. + +**You MUST apply a location cross-check, as per +[Other data augmentations](../docs/Developers.md#other-data-augmentations).** diff --git a/schemas/scanbarycentre-README.md b/schemas/scanbarycentre-README.md index d7b5f01..89c1ce6 100644 --- a/schemas/scanbarycentre-README.md +++ b/schemas/scanbarycentre-README.md @@ -30,3 +30,6 @@ in the Developers' documentation. #### StarPos You MUST add a `StarPos` array containing the system co-ordinates from the last `FSDJump`, `CarrierJump`, or `Location` event. + +**You MUST apply a location cross-check, as per +[Other data augmentations](../docs/Developers.md#other-data-augmentations).** diff --git a/scripts/eddn-report-log-errors b/scripts/eddn-report-log-errors index 83ebf63..35e7806 100755 --- a/scripts/eddn-report-log-errors +++ b/scripts/eddn-report-log-errors @@ -62,14 +62,36 @@ def process_file(input_file: str) -> None: # print(matches.group('sender_ip')) # print('') - software_version = semantic_version.Version.coerce(matches.group('software_version')) + try: + software_version = semantic_version.Version.coerce(matches.group('software_version')) + + except ValueError as e: + print(f"Error parsing sofwareVersion for:\n{matches.group('software_version')}\n{line}\n") + next + ################################################################### # Issues we know about and HAVE already alerted their # developers to. ################################################################### if matches.group('software_name') == 'EDDiscovery': # https://github.com/EDDiscovery/EDDiscovery/releases/latest - if software_version >= semantic_version.Version.coerce('12.1.7.0'): + if software_version >= semantic_version.Version.coerce('15.0.0.0'): + if matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/approachsettlement/1': + if matches.group('err_msg') == 'Failed Validation "[]"': + # + pass + + elif matches.group('err_msg') == 'Failed Validation "[]"': + # + pass + + else: + print(line) + + else: + print(line) + + elif software_version >= semantic_version.Version.coerce('12.1.7.0'): if matches.group('schema_ref') in ( 'https://eddn.edcd.io/schemas/shipyard/2', 'https://eddn.edcd.io/schemas/outfitting/2', @@ -238,6 +260,16 @@ def process_file(input_file: str) -> None: else: print(line) + elif matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/shipyard/2': + # + if matches.group('err_msg').startswith( + 'Failed Validation "[]"' + ): + pass + + else: + print(line) + else: print(line) diff --git a/src/eddn/conf/Settings.py b/src/eddn/conf/Settings.py index 5ca2a6f..c09e2ad 100644 --- a/src/eddn/conf/Settings.py +++ b/src/eddn/conf/Settings.py @@ -59,25 +59,28 @@ class _Settings(object): "https://eddn.edcd.io/schemas/journal/1" : "schemas/journal-v1.0.json", "https://eddn.edcd.io/schemas/journal/1/test" : "schemas/journal-v1.0.json", - "https://eddn.edcd.io/schemas/scanbarycentre/1" : "schemas/scanbarycentre-v1.0.json", - "https://eddn.edcd.io/schemas/scanbarycentre/1/test" : "schemas/scanbarycentre-v1.0.json", + "https://eddn.edcd.io/schemas/scanbarycentre/1" : "schemas/scanbarycentre-v1.0.json", + "https://eddn.edcd.io/schemas/scanbarycentre/1/test" : "schemas/scanbarycentre-v1.0.json", - "https://eddn.edcd.io/schemas/fssdiscoveryscan/1" : "schemas/fssdiscoveryscan-v1.0.json", - "https://eddn.edcd.io/schemas/fssdiscoveryscan/1/test" : "schemas/fssdiscoveryscan-v1.0.json", + "https://eddn.edcd.io/schemas/fssdiscoveryscan/1" : "schemas/fssdiscoveryscan-v1.0.json", + "https://eddn.edcd.io/schemas/fssdiscoveryscan/1/test" : "schemas/fssdiscoveryscan-v1.0.json", - "https://eddn.edcd.io/schemas/codexentry/1" : "schemas/codexentry-v1.0.json", - "https://eddn.edcd.io/schemas/codexentry/1/test" : "schemas/codexentry-v1.0.json", + "https://eddn.edcd.io/schemas/codexentry/1" : "schemas/codexentry-v1.0.json", + "https://eddn.edcd.io/schemas/codexentry/1/test" : "schemas/codexentry-v1.0.json", - "https://eddn.edcd.io/schemas/navbeaconscan/1" : "schemas/navbeaconscan-v1.0.json", - "https://eddn.edcd.io/schemas/navbeaconscan/1/test" : "schemas/navbeaconscan-v1.0.json", + "https://eddn.edcd.io/schemas/navbeaconscan/1" : "schemas/navbeaconscan-v1.0.json", + "https://eddn.edcd.io/schemas/navbeaconscan/1/test" : "schemas/navbeaconscan-v1.0.json", - "https://eddn.edcd.io/schemas/navroute/1" : "schemas/navroute-v1.0.json", - "https://eddn.edcd.io/schemas/navroute/1/test" : "schemas/navroute-v1.0.json", + "https://eddn.edcd.io/schemas/navroute/1" : "schemas/navroute-v1.0.json", + "https://eddn.edcd.io/schemas/navroute/1/test" : "schemas/navroute-v1.0.json", - "https://eddn.edcd.io/schemas/approachsettlement/1" : "schemas/approachsettlement-v1.0.json", - "https://eddn.edcd.io/schemas/approachsettlement/1/test" : "schemas/approachsettlement-v1.0.json", - "https://eddn.edcd.io/schemas/fssallbodiesfound/1" : "schemas/fssallbodiesfound-v1.0.json", - "https://eddn.edcd.io/schemas/fssallbodiesfound/1/test" : "schemas/fssallbodiesfound-v1.0.json", + "https://eddn.edcd.io/schemas/approachsettlement/1" : "schemas/approachsettlement-v1.0.json", + "https://eddn.edcd.io/schemas/approachsettlement/1/test" : "schemas/approachsettlement-v1.0.json", + "https://eddn.edcd.io/schemas/fssallbodiesfound/1" : "schemas/fssallbodiesfound-v1.0.json", + "https://eddn.edcd.io/schemas/fssallbodiesfound/1/test" : "schemas/fssallbodiesfound-v1.0.json", + + "https://eddn.edcd.io/schemas/fssbodysignals/1" : "schemas/fssbodysignals-v1.0.json", + "https://eddn.edcd.io/schemas/fssbodysignals/1/test" : "schemas/fssbodysignals-v1.0.json", } GATEWAY_OUTDATED_SCHEMAS = [