Merge branch 'master' into live

docs/Developers.md

@@ -175,6 +175,8 @@ For example, a shipyard message, version 2, might look like:
   "$schemaRef": "https://eddn.edcd.io/schemas/shipyard/2",
   "header": {
     "uploaderID": "Bill",
+    "gameversion": "4.0.0.1451", 
+    "gamebuild": "r286916/r0 ",
     "softwareName": "My excellent app",
     "softwareVersion": "0.0.1"
   },
@@ -183,7 +185,8 @@ For example, a shipyard message, version 2, might look like:
     "stationName": "Samson",
     "marketId": 128023552,
     "horizons": true,
-    "timestamp": "2019-01-08T06:39:43Z",
+    "odyssey": true,  
+    "timestamp": "2022-09-27T06:39:43Z",
     "ships": [
       "anaconda",
       "dolphin",
@@ -200,6 +203,72 @@ For example, a shipyard message, version 2, might look like:
 }
 ```
 
+---
+
+### Contents of `header`
+You **MUST** send the `header` component of the message.
+
+#### uploaderID
+The EDDN Relay will obfuscate the `uploaderID` value to prevent long-term
+tracking of individual players.  Please **DO** send a sensible
+`uploaderID` value, preferably simply the relevant in-game Commander name.
+
+#### softwareName
+You **MUST** set a unique, and self-consistent, value of `softwareName` so
+that you can be easily identified should any issues be found with the messages
+you send.
+
+#### softwareVersion
+You **MUST** set a pertinent value for `softwareVersion`.  We would recommend
+using [Semantic Versionining](https://semver.org/#semantic-versioning-specification-semver)
+in your project.
+
+Listeners MAY make decisions on whether to accept data, or to treat it
+differently, based on this.  As such you **MUST** increment your version
+number if you make any changes to the content of messages your software sends
+to EDDN.
+
+#### `gameversions` and `gamebuild`
+To ensure that Listeners can make decisions on how to handle data based on
+the client and feature set it came from there are two mandatory fields in
+the headers of EDDN messages.  NB: Initially the *schemas* do not actually
+make these mandatory, **but all Senders should make every effort to include
+them ASAP**.
+
+Where present in the data source the `gameversion` value **MUST** come from
+the field of that name in the data source, i.e. from either `Fileheader` or
+`LoadGame` as outlined below.
+
+For `gamebuild` you **MUST** use the value of the `build` field in the data
+source.
+
+1. If you are using Journal files directly then you **MUST** use the value
+  from the`Fileheader` event.
+2. If you are using the CAPI `/journal` endpoint to retrieve and process
+  Journal events then:
+   1. You will not have `Fileheader` available.
+   2. If the field is present in the `LoadGame` event, as in 4.0 clients,
+     use its value.
+   3. If `LoadGame` does not have the field, as with 3.8 Horizons
+     clients (up to at least `3.8.0.407`), you **SHOULD** set the value to
+     `"CAPI-journal"`. If, for reasons of code architecture, you are unable to
+     determine that data was CAPI-sourced then you MAY set it to `""` instead.
+3. If you are sourcing data from other CAPI endpoints, i.e. for commodity,
+  shipyard or outfitting messages, then you **SHOULD** set the values
+  appropriately as per the CAPI endpoint the data came from:
+   1. If it's a commodity message, then use `"CAPI-market"`.
+   2. If it's a shipyard message, then use `"CAPI-shipyard"`.
+   3. If it's an oufitting message, then also use `"CAPI-shipyard"`.
+
+   Again, if your code architecture doesn't allow for signalling that the data
+  source was CAPI, then you MAY set it to `""` instead.
+
+For emphasis, **if you cannot set a data-source value, or an appropriate
+`"CAPI-..."` value then you **MUST** still send the field with an empty string
+value.
+
+---
+
 ### Contents of `message`
 Every message MUST comply with the Schema its `$schemaRef` value cites.  Each
 Schema file should have a matching `<schema>-README.md` file in the
@@ -285,7 +354,7 @@ PC-local files for these events):
     { "timestamp":"2022-09-27T11:28:53Z", "event":"LoadGame", "FID":"<elided>", "Commander":"<elided>", "Horizons":true, ...
     ```
 
-- PC 'base' Client, game version `i3.8.0.407`, no `Odyssey` key was
+- PC 'base' Client, game version `3.8.0.407`, no `Odyssey` key was
   present:
     ```json
     { "timestamp":"2022-09-27T11:31:32Z", "event":"LoadGame", "FID":"<elided>", "Commander":"<elided>", "Horizons":false, ...

schemas/approachsettlement-README.md

@@ -58,6 +58,10 @@ Examples:
 Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
 in the Developers' documentation.
 
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
 #### StarSystem
 
 You MUST add a StarSystem key/value pair representing the name of the system

schemas/approachsettlement-v1.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

schemas/codexentry-README.md

@@ -23,6 +23,10 @@ The primary data source for this schema is the ED Journal event `CodexEntry`.
 Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
 in the Developers' documentation.
 
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
 #### StarPos
 You MUST add a `StarPos` array containing the system co-ordinates from the
 last `FSDJump`, `CarrierJump`, or `Location` event.

schemas/codexentry-v1.0.json

@@ -17,6 +17,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

schemas/commodity-README.md

@@ -60,6 +60,10 @@ Remove not only the `Category_Localised` key:values, but also the
 Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
 in the Developers' documentation.
 
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
 ### Using CAPI data
 It is *not* recommended to use CAPI data as the source as it's fraught with 
 additional issues.  EDMarketConnector does so in order to facilitate 

schemas/commodity-v3.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "Fileheader->gameversion, else LoadGame->gameversion, else 'CAPI-market', else ''."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "Fileheader->build, else LoadGame->build, else 'CAPI-market', else ''."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

schemas/fcmaterials_capi-README.md

@@ -57,6 +57,10 @@ You **MUST NOT** set them otherwise, as e.g. the player could be active in
 the game on another computer, using a different game mode and the CAPI data
 will be for that game mode.
 
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
 ## Listeners
 The advice above for [Senders](#senders), combined with the actual Schema file
 *should* provide all the information you need to process these events.

schemas/fcmaterials_capi-v1.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "Value of 'CAPI-market' if possible, else empty string."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "Value of 'CAPI-market' if possible, else empty string."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

schemas/fcmaterials_journal-README.md

@@ -33,6 +33,10 @@ of any augmentations, as noted below.
 Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
 in the Developers' documentation.
 
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
 ## Listeners
 The advice above for [Senders](#senders), combined with the actual Schema file
 *should* provide all the information you need to process these events.

schemas/fcmaterials_journal-v1.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

schemas/fssallbodiesfound-README.md

@@ -24,6 +24,10 @@ The primary data source for this schema is the ED Journal event
 Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
 in the Developers' documentation.
 
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
 #### StarPos
 You MUST add a `StarPos` array containing the system co-ordinates from the 
 last `FSDJump`, `CarrierJump`, or `Location` event.

schemas/fssallbodiesfound-v1.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

schemas/fssbodysignals-README.md

@@ -24,6 +24,10 @@ The primary data source for this schema is the ED Journal event
 Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
 in the Developers' documentation.
 
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
 #### StarSystem
 You MUST add a `StarSystem` string containing the name of the system from the 
 last `FSDJump`, `CarrierJump`, or `Location` event.

schemas/fssbodysignals-v1.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

schemas/fssdiscoveryscan-README.md

@@ -24,6 +24,10 @@ The primary data source for this schema is the ED Journal event
 Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
 in the Developers' documentation.
 
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
 #### StarPos
 You MUST add a `StarPos` array containing the system co-ordinates from the 
 last `FSDJump`, `CarrierJump`, or `Location` event.

schemas/fssdiscoveryscan-v1.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

schemas/fsssignaldiscovered-README.md

@@ -87,6 +87,10 @@ You SHOULD add this key/value pair, using the value from the `LoadGame` event.
 #### odyssey flag
 You SHOULD add this key/value pair, using the value from the `LoadGame` event.
 
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
 #### StarSystem
 You **MUST** add a `StarSystem` string containing the system name from the last
 tracked location.  You **MUST** cross-check each `FSSSignalDiscovered`

schemas/fsssignaldiscovered-v1.0.json

@@ -17,6 +17,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

schemas/journal-README.md

@@ -65,11 +65,19 @@ The following keys+values should be removed from `Location` event data:
 - `SquadronFaction` from within the list of `Factions`.
 
 ### Augmentations
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
 #### horizons flag
-You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+You **MUST** add this key/value pair, using the value from the `LoadGame` event.
+
+Note caveats in [docs/Developers.md](../docs/Developers.md).
 
 #### odyssey flag
-You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+You **MUST** add this key/value pair, using the value from the `LoadGame` event.
+
+Note caveats in [docs/Developers.md](../docs/Developers.md).
 
 #### StarSystem
 If not already present, you MUST add a `StarSystem` string containing the

schemas/journal-v1.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

schemas/navbeaconscan-README.md

@@ -24,6 +24,10 @@ The primary data source for this schema is the ED Journal event
 Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
 in the Developers' documentation.
 
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
 #### StarSystem
 You MUST add a `StarSystem` key/value pair representing the name of the
 system this event occurred in.  Source this from either `Location`,

schemas/navbeaconscan-v1.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

schemas/navroute-README.md

@@ -31,3 +31,7 @@ separate file.
 Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
 in the Developers' documentation.
 
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+

schemas/navroute-v1.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

schemas/outfitting-README.md

@@ -53,3 +53,8 @@ station. Namely:
 #### horizons and odyssey flags
 Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
 in the Developers' documentation.
+
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+

schemas/outfitting-v2.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "Fileheader->gameversion, else LoadGame->gameversion, else 'CAPI-shipyard', else ''."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "Fileheader->build, else LoadGame->build, else 'CAPI-shipyard', else ''."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

schemas/scanbarycentre-README.md

@@ -27,6 +27,10 @@ senders SHOULD include any defined in the schema if it's in the source data.
 Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
 in the Developers' documentation.
 
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
 #### StarPos
 You MUST add a `StarPos` array containing the system co-ordinates from the
 last `FSDJump`, `CarrierJump`, or `Location` event.

schemas/scanbarycentre-v1.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

schemas/shipyard-README.md

@@ -40,3 +40,8 @@ value is what the name would have been in the source Journal data.
 #### horizons and odyssey flags
 Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
 in the Developers' documentation.
+
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+

schemas/shipyard-v2.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "Fileheader->gameversion, else LoadGame->gameversion, else 'CAPI-shipyard', else ''."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "Fileheader->build, else LoadGame->build, else 'CAPI-shipyard', else ''."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

scripts/eddn-report-log-errors

@@ -75,24 +75,24 @@ def process_file(input_file: str) -> None:
         ###################################################################
         if matches.group('software_name') == 'EDDiscovery':
           # https://github.com/EDDiscovery/EDDiscovery/releases/latest
-          if software_version >= semantic_version.Version.coerce('15.1.2.0'):
+          if software_version >= semantic_version.Version.coerce('15.1.4.0'):
-            # if matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/outfitting/2':
+            if matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/outfitting/2':
-              # err_msg = matches.group('err_msg')
+              err_msg = matches.group('err_msg')
-              # if (
+              if (
-                # err_msg.startswith('Failed Validation "[<ValidationError: "u\'\\\\u0131nt_') and
+                err_msg.startswith('Failed Validation "[<ValidationError: "\'paintjob_') and
-                # err_msg.find('\' does not match \'(^Hpt_|^hpt_|^Int_|^int_|_Armour_|_armour_)\'">]') != -1
+                err_msg.find('\' does not match \'(^Hpt_|^hpt_|^Int_|^int_|_Armour_|_armour_)\'">]') != -1
-              # ):
+              ):
-                # <https://github.com/EDDiscovery/EDDiscovery/issues/3304>
+                # <https://github.com/EDDiscovery/EDDiscovery/issues/3328>
-                # pass
+                pass
 
-              # else: 
+              else: 
-                # print(line)
+                print(line)
 
-            # else:
+            else:
-            print(line)
+              print(line)
 
         elif matches.group('software_name') == 'EDDLite':
-          # https://github.com/EDDiscovery/EDDLite/releases/tag/latest
+          # https://github.com/EDDiscovery/EDDLite/releases/latest
           if software_version >= semantic_version.Version.coerce('2.3.0'):
             print(line)
 
@@ -103,7 +103,7 @@ def process_file(input_file: str) -> None:
 
         elif matches.group('software_name').startswith('E:D Market Connector'):
           # https://github.com/EDCD/EDMarketConnector/releases/latest
-          if software_version >= semantic_version.Version.coerce('5.4.1'):
+          if software_version >= semantic_version.Version.coerce('5.5.0'):
             if matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/journal/1':
               if matches.group('err_msg').startswith(
                 'Failed Validation "[<ValidationError: "{\'type\': [\'array\', \'boolean\', \'integer\', \'number\', \'null\', \'object\', \'string\']} is not allowed for'