Update fsssignaldiscovered-v1.0.json

Added SpawningPower property.

README.md

@@ -65,14 +65,11 @@ then you're probably looking for one of the sites listed below.  NB: These are
 listed in name-alphabetical order and no particular ranking or endorsement is
 intended.
 
-- [EDDB](https://eddb.io/) - a website which tries to act as a database of all
- the data available in the game. In general EDDB tries to help finding 
- stuff which players are looking for.
 - [EDSM](https://www.edsm.net/) - originally focused on being a 'Star Map', 
   but has since expanded its functionality.  Of particular interest to 
   in-game explorers.
-- [Inara](https://inara.cz/) - a popular alternative to EDDB, with a lot of 
-  its own unique functionality.
+- [Inara](https://inara.cz/) - a popular alternative to the now defunct EDDB, 
+  with a lot of its own unique functionality.
 - [Spansh](https://www.spansh.co.uk/plotter) - originally this had one tool,
   a 'Neutron Star' route plotter, but has since expanded into offering many 
   other route plotting tools and general data searching.
@@ -91,10 +88,6 @@ use:
     large, but is currently the only source of an "all known bodies" dump.
     Pay attention to the 'Generated' "time ago" column.
 
-- [EDDB dumps](https://eddb.io/api) represent a snapshot of the data EDDB uses.
-    NB: There has been no "bodies" data for years now, EDDB itself stopped
-    updating or adding to this.
-
 - [EDSM nightly dumps](https://www.edsm.net/en/nightly-dumps) represent a
     snapshot of the data EDSM uses.  NB: there's only a "last 7 days" bodies
     dump as the full data proved too large to dump in a timely manner.
@@ -171,5 +164,5 @@ Hosting is currently provided by the
 
 ### Contacting the EDDN team
 
-* [EDCD Discord](https://discord.gg/XBsdCq9) - **Use the `#eddn` channel**.
+* [EDCD Discord - #eddn channel](https://discord.gg/DdqJ8nWVGc)
 * [E:D forum thread](https://forums.frontier.co.uk/threads/elite-dangerous-data-network-eddn.585701/#post-9400060)

contrib/eddn-logs-archive

@@ -78,7 +78,7 @@ for service in gateway monitor relay ;
 do
   log "Service: ${service}"
   log "         Expiring old logs..."
-  find . -name "${service}.log.*.gz" -a -atime +${MAX_LOGFILE_AGE} -exec rm -fv {} \;
+  find . -name "${service}.log.*.gz" -a -atime +${MAX_LOGFILE_AGE} -exec rm -f {} \;
   log "         DONE"
 
   log "         Checking if current logfile needs archiving..."
@@ -106,7 +106,7 @@ do
     fi
 
     # Now compress the newly archived log
-    gzip -9v "${ARCHIVED_NAME}"
+    gzip -9 "${ARCHIVED_NAME}"
     log "         DONE"
   else
     log "         No"

contrib/monitor/schemas.html

@@ -241,7 +241,7 @@
                     // Loop results
                     $.each(schemasCount, function(schema, hits){
                         // IF TEST CONTINUE
-                        if(schema.substr(schema.length - 4) == 'test')
+                        if(schema.slice(-4) == 'test')
                             return;
 
                         var slug = makeSlug(schema);

docs/Developers.md

@@ -1,7 +1,7 @@
 ## Introduction
 
 EDDN is a
-[zermoq](https://zeromq.org/) service which allows players of the game
+[zeromq](https://zeromq.org/) service which allows players of the game
 [Elite Dangerous](https://www.elitedangerous.com/), published
 by [Frontier Developments](https://www.frontier.co.uk/), to upload game data so
 that interested listeners can receive a copy.
@@ -15,7 +15,7 @@ representing this game data and then passes it on to any interested listeners.
 ## Sources
 
 There are two sources of game data, both provided by the publisher of the game,
-Frontier Developerments.  They are both explicitly approved for use by
+Frontier Developments.  They are both explicitly approved for use by
 third-party software.
 
 ### Journal Files
@@ -127,13 +127,12 @@ The body of an EDDN message is a JSON object in UTF-8 encoding.  If you do not
 compress this body then you MUST set a `Content-Type` header of
 `applicaton/json`.
 
-For historical reasons URL form-encoded data *is* supported, **but this is
-deprecated and no new software should attempt this method**.  We
-purposefully do not further document the exact format for this.
-
 You *MAY* use gzip compression on the body of the message, but it is not
-required.  If you do compress the body then you **MUST* send a `Content-Type`
-header of `gzip` instead of `application/json`.
+required.  If you do compress the body then you **MUST* send a
+`Content-Encoding` header of `gzip`.
+
+**Due to issues when messages are compressed, form-encoded data is NO LONGER
+SUPPORTED as of 2022-06-16.**
 
 You should be prepared to handle all scenarios where sending of a message
 fails:
@@ -176,6 +175,8 @@ For example, a shipyard message, version 2, might look like:
   "$schemaRef": "https://eddn.edcd.io/schemas/shipyard/2",
   "header": {
     "uploaderID": "Bill",
+    "gameversion": "4.0.0.1451", 
+    "gamebuild": "r286916/r0 ",
     "softwareName": "My excellent app",
     "softwareVersion": "0.0.1"
   },
@@ -184,7 +185,8 @@ For example, a shipyard message, version 2, might look like:
     "stationName": "Samson",
     "marketId": 128023552,
     "horizons": true,
-    "timestamp": "2019-01-08T06:39:43Z",
+    "odyssey": true,  
+    "timestamp": "2022-09-27T06:39:43Z",
     "ships": [
       "anaconda",
       "dolphin",
@@ -201,6 +203,111 @@ For example, a shipyard message, version 2, might look like:
 }
 ```
 
+---
+
+### Contents of `header`
+You **MUST** send the `header` component of the message.
+
+#### uploaderID
+The EDDN Relay will obfuscate the `uploaderID` value to prevent long-term
+tracking of individual players.  Please **DO** send a sensible
+`uploaderID` value, preferably simply the relevant in-game Commander name.
+
+#### softwareName
+You **MUST** set a unique, and self-consistent, value of `softwareName` so
+that you can be easily identified should any issues be found with the messages
+you send.
+
+#### softwareVersion
+You **MUST** set a pertinent value for `softwareVersion`.  We would recommend
+using [Semantic Versionining](https://semver.org/#semantic-versioning-specification-semver)
+in your project.
+
+Listeners MAY make decisions on whether to accept data, or to treat it
+differently, based on this.  As such you **MUST** increment your version
+number if you make any changes to the content of messages your software sends
+to EDDN.
+
+#### `gameversions` and `gamebuild`
+To ensure that Listeners can make decisions on how to handle data based on
+the client and feature set it came from there are two mandatory fields in
+the headers of EDDN messages.  NB: Initially the *schemas* do not actually
+make these mandatory, **but all Senders MUST make every effort to include
+them ASAP**.
+
+Where present in the data source the `gameversion` value **MUST** come from
+the field of that name in the data source, i.e. from either `Fileheader` or
+`LoadGame` as outlined below.  
+
+1. If you are using Journal files directly then you **MUST** use the value
+  from the`Fileheader` event.  Values from `LoadGame` are also acceptable, but
+  there are some events before that which might be relevant to EDDN messages,
+  especially as the ordering of 'startup'/login events can change with game
+  patches.
+2. If you are using the CAPI `/journal` endpoint to retrieve and process
+  Journal events then:
+   1. You will not have `Fileheader` available.
+   2. If the field is present in the `LoadGame` event, as in 4.0 clients,
+     use its value.
+   3. If `LoadGame` does not have the field, as with 3.8 Horizons
+     clients (up to at least `3.8.0.1400`), you **SHOULD** set the value to
+     `"CAPI-Legacy-journal"`. If, for reasons of code architecture, you are
+     unable to determine that data was CAPI-sourced then you MAY set it to
+     `""` instead, **but this is only a fallback and you should endeavour to
+     send a more useful value**.
+3. If you are sourcing data from other CAPI endpoints, i.e. for commodity,
+  shipyard, outfitting or fcmaterials_capi messages, then you **SHOULD** set
+  the values appropriately as per the CAPI host and endpoint the data came
+  from.  You **MUST NOT** use the value from any Journal you are also reading 
+  for the user.  The general form is `"CAPI-(Live|Legacy)-<endpoint>"`.
+   1. If it's a commodity message from the Live CAPI host, then use
+     `"CAPI-Live-market"`.  If it was from the Legacy CAPI host then use
+     `"CAPI-Legacy-market"`.
+   2. If it's a shipyard message, then use `"CAPI-Live-shipyard"` or
+     `"CAPI-Legacy-shipyard"`.
+   3. If it's an oufitting message, then also use `"CAPI-Live-shipyard"` or
+     `"CAPI-Legacy-shipyard"`.
+   4. If it's an fcmaterials_capi message, then use `"CAPI-Live-market`" or
+     `"CAPI-Legacy-market"`, as the data comes from that endpoint.
+
+   Again, if your code architecture doesn't allow for signalling that the data
+  source was CAPI, then you MAY set it to `""` instead, **but this is only a
+  fallback and you should endeavour to send a more useful value**.
+
+For `gamebuild` you **MUST** use the value of the `build` field in the data
+source, following the same logic as for `gameversion` above, else send as `""`.
+An alternative to the latter is to mirror the `gameversion` value, **but only
+where that has not come from a Journal value**, i.e. `CAPI-...` strings.
+Do **NOT** strip any leading or trailing white space, pass it through as-is.
+
+For emphasis, *if you cannot set a data-source value, or an appropriate
+`"CAPI-..."` value then you **MUST** still send the field with an empty string
+value.*
+
+Examples of valid values:
+
+| Data Source |     Data Type | Game Galaxy | gameversion         | gamebuild           |
+|------------:|--------------:|:-----------:|:----------------------|:----------------------|
+|     Journal |       Journal |    Live     | `"4.0.0.1475"`          | `"r289563/r0 "`         |
+|     Journal |       Journal |   Legacy    | `"3.8.0.1400"`          | `"r289583/r0 "`         |
+| CAPI |    `/journal` |  Live[^1]   | `"4.0.0.1475"`          | `"r289563/r0 "`         |
+| CAPI |    `/journal` | Legacy[^2]  | `"CAPI-Legacy-journal"` | `""`                    |
+| CAPI |    `/journal` | Legacy[^2]  | `"CAPI-Legacy-journal"` | `"CAPI-Legacy-journal"` |
+| CAPI | `/market`[^3] |    Live     | `"CAPI-Live-market"`    | `""`                    |
+| CAPI | `/market`[^3] |    Live     | `"CAPI-Live-market"`    | `"CAPI-Live-market"`    |
+| CAPI | `/market`[^3] |   Legacy    | `"CAPI-Legacy-market"`    | `""`                    |
+| CAPI | `/market`[^3] |   Legacy    | `"CAPI-Legacy-market"`    | `"CAPI-Legacy-market"`    |
+[^1]: Live is a 4.0 client and has `gameversion` in `LoadGame` which is present
+  in CAPI-sourced journals.
+
+[^2]: Legacy is a 3.8 client and last tested still doesn't have `gameversion`
+  in `LoadGame`, and CAPI-sourced journals don't have `Fileheader`.
+
+[^3]: And similarly for other CAPI endpoints, e.g. `shipyard`.
+
+
+---
+
 ### Contents of `message`
 Every message MUST comply with the Schema its `$schemaRef` value cites.  Each
 Schema file should have a matching `<schema>-README.md` file in the
@@ -252,10 +359,14 @@ so that you are aware of any changes to Schemas.
 
 #### `horizons` and `odyssey` flags
 
-Where the Schema allows for them, `horizons` and `odyssey` keys SHOULD be
+Where the Schema allows for them, `horizons` and `odyssey` keys **MUST** be
 added with appropriate boolean values.  `null` is not allowed in the values,
 so **if you cannot determine a value do not include that key at all**.
 
+To emphasise that, *in the case where there is no `Odyssey` boolean in the
+`LoadGame` event* **DO NOT INCLUDE IT IN THE EDDN MESSAGE**.  No, not with a
+`false` value.  **DO NOT INCLUDE IT**.
+
 The only source of these is the `LoadGame` event from journals.  It's present
 both in the PC local files and the CAPI journal data.  If you're
 composing a shipyard or outfitting message from CAPI data then it is
@@ -264,42 +375,90 @@ possible to synthesise the `horizons` flag.  For now consult the function
 [EDMarketConnector:plugins/eddn.py](https://github.com/EDCD/EDMarketConnector/blob/stable/plugins/eddn.py)
 for a method to achieve this.
 
-As of 2022-01-29 the following was observed for the `LoadGame` events as
+As of 2022-09-27 the following was observed for the `LoadGame` events as
 present in CAPI-sourced Journal files (which were confirmed to match the
 PC-local files for these events):
 
-- PC Odyssey Client, game version `4.0.0.1100`:
+- PC Odyssey Client, game version `4.0.0.1450`:
     ```json
-    { "timestamp":"2022-01-29T16:17:02Z", "event":"LoadGame", "FID":"<elided>", "Commander":"<elided>", "Horizons":true, "Odyssey":true,...
+    { "timestamp":"2022-09-27T09:47:35Z", "event":"LoadGame", "FID":"<elided>", "Commander":"<elided>", "Horizons":true, "Odyssey":true, ...
     ```
-- PC Horizons Client, game version `3.8.0.404`, no `Odyssey` key was
+- PC Horizons 4.0 Client, game version `4.0.0.1450`:
+    ```json
+    { "timestamp":"2022-09-27T11:25:45Z", "event":"LoadGame", "FID":"<elided>", "Commander":"<elided>", "Horizons":true, "Odyssey":false, ...
+    ```
+- PC Horizons Client, game version `3.8.0.407`, no `Odyssey` key was
   present:
     ```json
-    { "timestamp":"2022-01-29T16:15:07Z", "event":"LoadGame", "FID":"<elided>", "Commander":"<elided>", "Horizons":true,...
+    { "timestamp":"2022-09-27T11:28:53Z", "event":"LoadGame", "FID":"<elided>", "Commander":"<elided>", "Horizons":true, ...
     ```
 
-- PC 'base' Client, game version `3.8.0.404`, no `Odyssey` key was
+- PC 'base' Client, game version `3.8.0.407`, no `Odyssey` key was
   present:
     ```json
-    { "timestamp":"2022-01-29T16:11:54Z", "event":"LoadGame", "FID":"<elided>", "Commander":"<elided>", "Horizons":false,...
+    { "timestamp":"2022-09-27T11:31:32Z", "event":"LoadGame", "FID":"<elided>", "Commander":"<elided>", "Horizons":false, ...
     ```
 
 Do not attempt to use the value(s) from a `Fileheader` event as the semantics
-are different.  With clients 3.8.0.404 and 4.0.0.1100 the following was observed:
+are different.  With clients 3.8.0.407 and 4.0.0.1450 the following was observed:
 
-| Game Client      | Fileheader       | LoadGame                        |
-| ---------------: | ---------------- | ------------------------------: |
-| Base             | "Odyssey":false  | "Horizons":false                |
-| Horizons         | "Odyssey":false  | "Horizons":true                 |
-| Odyssey          | "Odyssey":true   | "Horizons":true, "Odyssey":true |
+| Game Client      | Fileheader      | LoadGame                         |
+| ---------------: |-----------------|---------------------------------:|
+| Base             | "Odyssey":false |                 "Horizons":false |
+| Horizons 3.8     | "Odyssey":false |                  "Horizons":true |
+| Horizons 4.0     | "Odyssey":true  | "Horizons":true, "Odyssey":false |
+| Odyssey          | "Odyssey":true  |  "Horizons":true, "Odyssey":true |
 
 NB: The 'Base' client appears to simply be the Horizons client with any
 Horizons-only features disabled.
 
-- In the `Fileheader` event it's indicating whether it's an Odyssey game client.
-- In the `LoadGame` it's indicating whether Horizons and/or Odyssey features are
-  active, but in the non-Odyssey game client case you only get the Horizons
-  boolean.
+- In the `Fileheader` event the `Odyssey` flag is indicating whether it's a
+  `4.0` game client.
+- In the `LoadGame` event the `Horizons` and `Odyssey` flags indicate if those
+  features are active, but in the `3.8` game client case you only get the
+  `Horizons` boolean.
+
+#### Other data Augmentations
+Some schemas mandate that extra data be added, beyond what is in the source
+data, to aid Listeners.
+
+This is usually related to specifying which system an event took place in, and
+usually means ensuring there is the full set of:
+
+1. `StarSystem` - the name of the system.
+2. `SystemAddress` - the game's unique numerical identifier for the system.
+3. `StarPos` - The system's co-ordinates.
+
+Whilst it can be argued that any Listener should see preceding event(s) that
+give any missing information where at least the system name or `SystemAddress`
+is already in the event data, this might not always be true.  So Senders MUST
+add this data where required.  It helps to fill out basic system information
+(name, SystemAddress and co-ordinates).
+
+However, there is a known game bug that can result in it stopping writing to
+the game journal, and some observed behaviour implies that it might then later
+resume writing to that file, but with events missing.  This means any Sender
+that blindly assumes it knows the current system/location and uses that for
+these Augmentations might send erroneous data.
+
+1. **Senders MUST cross-check available event data with prior 'location'
+  event(s) to be sure the correct extra data is being added.**
+2. **Listeners SHOULD realise that any data added as an Augmentation might be
+  in error.**
+
+For Senders, if the source data only has `SystemAddress` then you MUST check
+that it matches that from the prior `Location`, `FSDJump` or `CarrierJump`
+event before adding `StarSystem` and `StarPos` data to a message.  Drop the
+message entirely if it does not match.  Apply similar logic if it is only
+the system's name that is already present in data.  Do not blindly add
+`SystemAddress` or `StarPos`.  Likewise, do not blindly add `StarPos` if the
+other data is already in the source, without cross-checking the system name
+and `SystemAddress`.
+
+Listeners might be able to apply their own cross-check on received messages,
+and use any mismatch with respect to what they already know to make a decision
+whether to trust the augmented data.  Flagging it for manual review is probably
+wise.
 
 ### Server responses
 There are three possible sources of HTTP responses when sending an upload
@@ -435,7 +594,11 @@ data you will first need to zlib-decompress each message.  Then you will
 have a textual JSON object as per the Schemas.
 
 In general, check the guidance for [Uploading messages](#uploading-messages)
-for the expected format of the messages.
+for the expected format of the messages.  **Pay particular attention to any
+schema-specific Augmentations**.  Whilst Senders MUST make every effort to
+ensure such data is correct it is possible that bugs in either their code, or
+the game itself, could mean it is incorrect.  Listeners use such data at
+their own risk.
 
 Consumers can utilise the `$schemaRef` value to determine which Schema a
 particular message is for.  There is no need to validate the messages

examples/Java/SimpleSubscribe/pom.xml

@@ -0,0 +1,72 @@
+<?xml version="1.0" encoding="UTF-8"?>
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+  <modelVersion>4.0.0</modelVersion>
+
+  <groupId>com.example</groupId>
+  <artifactId>SimpleJavaEDDNSubscribe</artifactId>
+  <version>1.0-SNAPSHOT</version>
+
+  <name>SimpleJavaEDDNSubscribe</name>
+
+  <properties>
+    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+    <maven.compiler.source>1.11</maven.compiler.source>
+    <maven.compiler.target>1.11</maven.compiler.target>
+  </properties>
+
+  <dependencies>
+    <dependency> 
+      <groupId>org.zeromq</groupId>
+      <artifactId>jeromq</artifactId>
+      <version>0.5.2</version>
+    </dependency>
+  </dependencies>
+
+  <build>
+    <pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
+      <plugins>
+        <!-- clean lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#clean_Lifecycle -->
+        <plugin>
+          <artifactId>maven-clean-plugin</artifactId>
+          <version>3.1.0</version>
+        </plugin>
+        <!-- default lifecycle, jar packaging: see https://maven.apache.org/ref/current/maven-core/default-bindings.html#Plugin_bindings_for_jar_packaging -->
+        <plugin>
+          <artifactId>maven-resources-plugin</artifactId>
+          <version>3.0.2</version>
+        </plugin>
+        <plugin>
+          <artifactId>maven-compiler-plugin</artifactId>
+          <version>3.8.0</version>
+        </plugin>
+        <plugin>
+          <artifactId>maven-surefire-plugin</artifactId>
+          <version>2.22.1</version>
+        </plugin>
+        <plugin>
+          <artifactId>maven-jar-plugin</artifactId>
+          <version>3.0.2</version>
+        </plugin>
+        <plugin>
+          <artifactId>maven-install-plugin</artifactId>
+          <version>2.5.2</version>
+        </plugin>
+        <plugin>
+          <artifactId>maven-deploy-plugin</artifactId>
+          <version>2.8.2</version>
+        </plugin>
+        <!-- site lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#site_Lifecycle -->
+        <plugin>
+          <artifactId>maven-site-plugin</artifactId>
+          <version>3.7.1</version>
+        </plugin>
+        <plugin>
+          <artifactId>maven-project-info-reports-plugin</artifactId>
+          <version>3.0.0</version>
+        </plugin>
+      </plugins>
+    </pluginManagement>
+  </build>
+</project>

examples/Java/SimpleSubscribe/src/main/java/org/eddn/examples/SimpleJavaEDDNSubscribe.java

@@ -0,0 +1,53 @@
+package org.eddn.examples;
+
+import java.util.zip.Inflater;
+
+import org.zeromq.SocketType;
+import org.zeromq.ZMQ;
+import org.zeromq.ZContext;
+
+public class SimpleJavaEDDNSubscribe {
+    private static final int MAX_MESSAGE_SIZE_KB = 200;
+    private static final String EDDN_SERVER = "tcp://eddn.edcd.io:9500";
+    private static ZContext context;
+
+    public static void main(String[] args) throws Exception {
+        context = new ZContext();    
+        String jsonMessage = getOneMessage();
+        System.out.println(jsonMessage);
+        context.close();
+    }
+
+    private static String getOneMessage() throws Exception {
+        byte[] deflatedMessage = receiveOneDeflatedMessage();
+        return inflateMessage(deflatedMessage);
+    }
+
+    private static byte[] receiveOneDeflatedMessage() {
+        ZMQ.Socket socket = getEDDNSubscriptionSocket();    
+        byte[] deflatedMessage = socket.recv();
+        return deflatedMessage;
+    }
+
+    private static ZMQ.Socket getEDDNSubscriptionSocket() {
+        ZMQ.Socket socket = context.createSocket(SocketType.SUB);
+        socket.connect(EDDN_SERVER);
+        
+        // need to subscribe to the empty topic to receive anything
+        socket.subscribe("");
+        return socket;
+    }
+
+    public static String inflateMessage(byte[] bytes) throws Exception {
+
+        Inflater decompresser = new Inflater();
+        decompresser.setInput(bytes);
+        byte[] result = new byte[MAX_MESSAGE_SIZE_KB * 1024];
+        int resultLength = decompresser.inflate(result);
+        decompresser.end();
+
+        // Decode the bytes into a String
+        String outputString = new String(result, 0, resultLength, "UTF-8");
+        return outputString;
+    }
+}

schemas/README-EDDN-schemas.md

@@ -15,7 +15,7 @@ those actually running on the Live service.
 
 The Schema files themselves are considered to be the canonical definition of
 the required, and allowed, contents of the relevant EDDN message.  There
-**SHOULD** be an accompanying README file, e.g. for `commodity-v3.0.json` there
+**MUST** be an accompanying README file, e.g. for `commodity-v3.0.json` there
 is also a `commodity-README.md` file in the project root `schemas/` directory.
 
 For more general documentation that all developers wanting to either Upload
@@ -29,7 +29,16 @@ It is best to base any new Schema file on
 contents all Schemas specify a top-level JSON Object with the data:
 
 1. `$schemaRef` - Which Schema (including version) this message is for.
-2. `header` - Object containing mandatory information about the upload;
+2. `$id` - The canonical URL for this schema once it is in live service.
+   1. Remember to have the version as in `journal/1` not `journal-v1.0`.
+   2. Do **NOT** end this with a `#` empty fragment.  This is
+  [documented](https://json-schema.org/draft/2020-12/json-schema-core.html#section-8.2.1)
+  as unnecessary.
+   3. Where there are two separate schemas for the same kind of data, but one
+     is for Journal-sourced, and the other for CAPI-sourced, you should have
+     the "filename" of the schema end with `_<source>`, e.g.
+     `fcmaterials_journal/1` and `fcmaterials_capi/1`.
+3. `header` - Object containing mandatory information about the upload;
     1. `uploaderID` - a unique ID for the player uploading this data.  
        Don't worry about privacy, the EDDN service will hash this with a key
        that is regularly changed so no-one knows who an uploader is in-game.
@@ -45,15 +54,31 @@ contents all Schemas specify a top-level JSON Object with the data:
    relevant README file within this documentation, e.g.
    [codexentry-README.md](./codexentry-README.md).
 
+Whilst currently (2022-11-23) the `gameversion` and `gamebuild` fields in the
+message `header` aren't yet mandatory in the sense of what's in the schema
+definitions, all Senders are **strongly encouraged** to send them.
+
+Indeed, the per-schema documentation, for every schema now states:
+
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
 ### General EDDN message outline
 
 Each `message` object must have, at bare minimum:
 
-1. `timestamp` - string date and time in ISO8601 format. Whilst this
-   technically allows for any timezone to be cited you SHOULD provide this in
-   UTC, aka 'Zulu Time' as in the example above. You MUST ensure that you are
-   doing this properly. Do not claim 'Z' whilst actually using a local time
-   that is offset from UTC.
+1. `timestamp` - string date and time in ISO8601 format. 
+    1. Whilst this technically allows for any timezone to be cited you SHOULD
+      provide this in UTC, aka 'Zulu Time' as in the example above.
+      You MUST ensure that you are doing this properly. 
+      Do not claim 'Z' whilst actually using a local time that is offset from
+      UTC.
+    2. Historically we had never been explicit about if Senders should include
+      sub-second resolution in the timestamps, or if Listeners should be
+      prepared to accept such.  As of 2022-06-24 we are explicitly stating that
+      Senders **MAY** include sub-second resolution, and Listeners **MUST**
+      be prepared to accept such.
 
    If you are only utilising Journal-sourced data then simply using the
    value from there should be sufficient as the PC game client is meant to

schemas/TEMPLATES/journalevent-README.md

@@ -106,10 +106,20 @@ value is what the name would have been in the source Journal data.
 Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
 in the Developers' documentation.
 
+#### StarSystem
+You MUST add a `StarSystem` string containing the name of the system from the 
+last `FSDJump`, `CarrierJump`, or `Location` event.
+
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**
+
 #### StarPos
 You MUST add a `StarPos` array containing the system co-ordinates from the 
 last `FSDJump`, `CarrierJump`, or `Location` event.
 
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**
+
 ## Listeners
 The advice above for [Senders](#senders), combined with the actual Schema file
 *should* provide all the information you need to process these events.

schemas/TEMPLATES/journalevent-v1.0.json

@@ -39,7 +39,7 @@
 #    as a result disallow the key.
 {
     "$schema"               : "http://json-schema.org/draft-04/schema#",
-    "id"                    : "https://eddn.edcd.io/schemas/newjournalevent/1#",
+    "id"                    : "https://eddn.edcd.io/schemas/newjournalevent/1",
     "type"                  : "object",
     "additionalProperties"  : false,
     "required": [ "$schemaRef", "header", "message" ],

schemas/approachsettlement-README.md

@@ -19,17 +19,61 @@ discrepancy.**
 The primary data source for this schema is the ED Journal event 
 `ApproachSettlement`.
 
+### MarketID
+Whilst the `MarketID` property is not in the required list **YOU MUST
+ABSOLUTELY SEND THIS WHEN IT IS PRESENT IN THE SOURCE DATA**.
+
+The only reason it is optional is that there are `ApproachSettlement`
+Journal events for things like visitor beacons that do not have a market, and
+thus no MarketID.
+
+Examples:
+
+```json
+{
+    "timestamp":"2022-02-18T14:33:35Z",
+    "event":"ApproachSettlement",
+    "Name":"Battlegroup's Disappearance",
+    "SystemAddress":1109989017963,
+    "BodyID":8,
+    "BodyName":"Alioth 1 a",
+    "Latitude":59.972752,
+    "Longitude":-84.506294
+},
+{
+    "timestamp": "2022-02-18T15:02:04Z",
+    "event": "ApproachSettlement",
+    "Name": "$Ancient:#index=1;",
+    "Name_Localised": "Ancient Ruins (1)",
+    "SystemAddress": 3515254557027,
+    "BodyID": 13,
+    "BodyName": "Synuefe XR-H d11-102 1 b",
+    "Latitude": -46.576923,
+    "Longitude": 133.985107
+},
+```
+
 ### Augmentations
 #### horizons and odyssey flags
 Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
 in the Developers' documentation.
 
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
 #### StarSystem
 
 You MUST add a StarSystem key/value pair representing the name of the system
 this event occurred in. Source this from either Location, FSDJump or
 CarrierJump as appropriate.
 
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**
+
 #### StarPos
 You MUST add a `StarPos` array containing the system co-ordinates from the
 last `FSDJump`, `CarrierJump`, or `Location` event.
+
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**

schemas/approachsettlement-v1.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },
@@ -33,7 +41,7 @@
             "type"                  : "object",
             "description"           : "Contains all properties from the listed events in the client's journal minus the Localised strings and the properties marked below as 'disallowed'",
             "additionalProperties"  : false,
-            "required"              : [ "timestamp", "event", "StarSystem", "StarPos", "SystemAddress", "Name", "MarketID", "BodyID", "BodyName", "Latitude", "Longitude" ],
+            "required"              : [ "timestamp", "event", "StarSystem", "StarPos", "SystemAddress", "Name", "BodyID", "BodyName", "Latitude", "Longitude" ],
             "properties"            : {
                 "timestamp": {
                     "type"          : "string",
@@ -62,6 +70,49 @@
                     "maxItems"      : 3,
                     "description"   : "Must be added by the sender"
                 },
+                "StationGovernment": {
+                    "type"          : "string"
+                },
+                "StationAllegiance": {
+                    "type"          : "string"
+                },
+                "StationEconomies": {
+                    "type"          : "array",
+                    "items"         : {
+                       "type"      : "object",
+                        "additionalProperties"  : false,
+                        "properties": {
+                            "Name": {
+                                "type"          : "string"
+                            },
+                            "Proportion": {
+                                "type"          : "number"
+                            }
+                        },
+                       "patternProperties"     : {
+                           "_Localised$"       : { "$ref" : "#/definitions/disallowed" }
+                       }
+                    }
+                },
+                "StationFaction": {
+                   "type"      : "object",
+                   "additionalProperties"  : false,
+                   "properties": {
+                       "Name": {
+                           "type"          : "string"
+                       },
+                       "FactionState": {
+                           "type"          : "string"
+                       }
+                   }
+                },	
+                "StationServices": {
+                    "type"          : "array",
+                    "items"         : { "type": "string" }
+                },
+                "StationEconomy": {
+                    "type"          : "string"
+                },	
                 "SystemAddress": {
                     "type"          : "integer"
                 },

schemas/codexentry-README.md

@@ -23,10 +23,17 @@ The primary data source for this schema is the ED Journal event `CodexEntry`.
 Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
 in the Developers' documentation.
 
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
 #### StarPos
 You MUST add a `StarPos` array containing the system co-ordinates from the
 last `FSDJump`, `CarrierJump`, or `Location` event.
 
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**
+
 #### BodyID and BodyName
 You SHOULD attempt to track the BodyName and BodyID where the player is 
 and add keys/values for these.

schemas/codexentry-v1.0.json

@@ -17,6 +17,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

schemas/commodity-README.md

@@ -51,6 +51,11 @@ In the list of commodites:
   Limpets - not purchasable in station market) or a *non-empty*`"legality":` 
   string (not normally traded at this station market).
 
+If the data is sourced from the journal folder:
+- Remove the `$` prefix and `_name;` suffix from the `Name` field.
+- As the Journal Market.json doesn't contain `economies` or `prohibited` data, 
+  leave these entirely out of the message.  You **MUST NOT** send empty lists.
+
 #### Item Category
 Remove not only the `Category_Localised` key:values, but also the
 `Category` key:value pair from each Item.
@@ -60,6 +65,10 @@ Remove not only the `Category_Localised` key:values, but also the
 Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
 in the Developers' documentation.
 
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
 ### Using CAPI data
 It is *not* recommended to use CAPI data as the source as it's fraught with 
 additional issues.  EDMarketConnector does so in order to facilitate 

schemas/commodity-v3.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "Fileheader->gameversion, else LoadGame->gameversion, else 'CAPI-market', else ''."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "Fileheader->build, else LoadGame->build, else 'CAPI-market', else ''."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },
@@ -36,13 +44,22 @@
             "properties"            : {
                 "systemName": {
                     "type"      : "string",
+                    "renamed"   : "StarSystem",
                     "minLength" : 1
                 },
                 "stationName": {
                     "type"      : "string",
-                    "renamed"   : "StarSystem",
+                    "renamed"   : "StationName",
                     "minLength" : 1
                 },                
+                "stationType": {
+                    "type"      : "string",
+                    "renamed"   : "StationType"
+                },
+                "carrierDockingAccess": {
+                    "type"      : "string",
+                    "renamed"   : "CarrierDockingAccess"
+                },                
                 "marketId": {
                     "type"      : "integer",
                     "renamed"   : "MarketID"
@@ -79,7 +96,7 @@
                             },
                             "buyPrice": {
                                 "type"          : "integer",
-                                "renaamed"      : "BuyPrice",
+                                "renamed"       : "BuyPrice",
                                 "description"   : "Price to buy from the market"
                             },
                             "stock": {
@@ -152,10 +169,6 @@
                         "type"          : "string",
                         "minLength"     : 1
                     }
-                },
-                "StationType": {
-                    "$ref"          : "#/definitions/disallowed",
-                    "description"   : "Not present in CAPI data, so removed from Journal-sourced data"
                 }
             }
         }

schemas/dockingdenied-README.md

@@ -0,0 +1,42 @@
+# EDDN DockingDenied Schema
+
+## Introduction
+Here we document how to take data from an ED `` Journal 
+Event and properly structure it for sending to EDDN.
+
+Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
+documentation for a schema such as this.
+
+If you find any discrepancies between what this document says and what is
+defined in the relevant Schema file, then you should, in the first instance,
+assume that it is the Schema file that is correct.
+**PLEASE open
+[an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose)
+to report any such anomalies you find so that we can check and resolve the
+discrepancy.**
+
+## Senders
+The primary data source for this schema is the ED Journal event 
+`DockingDenied`.
+
+Examples:
+
+```json
+{
+    "timestamp":"2022-06-10T10:09:41Z",
+    "event":"DockingDenied",
+    "Reason":"RestrictedAccess",
+    "MarketID":3706117376,
+    "StationName":"V7G-T1G",
+    "StationType":"FleetCarrier"
+}
+```
+
+### Augmentations
+#### horizons and odyssey flags
+Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
+in the Developers' documentation.
+
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.

schemas/dockingdenied-v1.0.json

@@ -0,0 +1,82 @@
+{
+    "$schema"               : "http://json-schema.org/draft-04/schema#",
+    "id"                    : "https://eddn.edcd.io/schemas/dockingdenied/1#",
+    "type"                  : "object",
+    "additionalProperties"  : false,
+    "required": [ "$schemaRef", "header", "message" ],
+    "properties": {
+        "$schemaRef": {
+            "type"                  : "string"
+        },
+        "header": {
+            "type"                  : "object",
+            "additionalProperties"  : true,
+            "required"              : [ "uploaderID", "softwareName", "softwareVersion" ],
+            "properties"            : {
+                "uploaderID": {
+                    "type"          : "string"
+                },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
+                "softwareName": {
+                    "type"          : "string"
+                },
+                "softwareVersion": {
+                    "type"          : "string"
+                },
+                "gatewayTimestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time",
+                    "description"   : "Timestamp upon receipt at the gateway. If present, this property will be overwritten by the gateway; submitters are not intended to populate this property."
+                }
+            }
+        },
+        "message": {
+            "type"                  : "object",
+            "description"           : "Contains all properties from the listed events in the client's journal minus the Localised strings and the properties marked below as 'disallowed'",
+            "additionalProperties"  : false,
+            "required"              : [ "timestamp", "event", "MarketID", "StationName", "Reason" ],
+            "properties"            : {
+                "timestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time"
+                },
+                "event" : {
+                    "enum"          : [ "DockingDenied" ]
+                },
+                "horizons": {
+                    "type"          : "boolean",
+                    "description"   : "Whether the sending Cmdr has a Horizons pass."
+                },
+                "odyssey": {
+                    "type"      : "boolean",
+                    "description" : "Whether the sending Cmdr has an Odyssey expansion."
+                },
+                "MarketID": {
+                    "type"          : "integer"
+                },		
+                "StationName": {
+                    "type"          : "string",
+                    "description"   : "Name of station"
+                },
+                "StationType": {
+                    "type"          : "string",
+                    "description"   : "Type of station"
+                },
+                "Reason": {
+                    "type"          : "string",
+                    "description"   : "Reason docking was denied"
+                }
+            }
+        }
+    },
+    "definitions": {
+        "disallowed" : { "not" : { "type": [ "array", "boolean", "integer", "number", "null", "object", "string" ] } }
+    }
+}

schemas/dockinggranted-README.md

@@ -0,0 +1,42 @@
+# EDDN DockingGranted Schema
+
+## Introduction
+Here we document how to take data from an ED `` Journal 
+Event and properly structure it for sending to EDDN.
+
+Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
+documentation for a schema such as this.
+
+If you find any discrepancies between what this document says and what is
+defined in the relevant Schema file, then you should, in the first instance,
+assume that it is the Schema file that is correct.
+**PLEASE open
+[an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose)
+to report any such anomalies you find so that we can check and resolve the
+discrepancy.**
+
+## Senders
+The primary data source for this schema is the ED Journal event 
+`DockingGranted`.
+
+Examples:
+
+```json
+{
+    "timestamp":"2023-10-01T14:56:34Z",
+    "event":"DockingGranted",
+    "LandingPad":41,
+    "MarketID":3227312896,
+    "StationName":"Evans Horizons",
+    "StationType":"Coriolis"
+}
+```
+
+### Augmentations
+#### horizons and odyssey flags
+Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
+in the Developers' documentation.
+
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.

schemas/dockinggranted-v1.0.json

@@ -0,0 +1,82 @@
+{
+    "$schema"               : "http://json-schema.org/draft-04/schema#",
+    "id"                    : "https://eddn.edcd.io/schemas/dockinggranted/1#",
+    "type"                  : "object",
+    "additionalProperties"  : false,
+    "required": [ "$schemaRef", "header", "message" ],
+    "properties": {
+        "$schemaRef": {
+            "type"                  : "string"
+        },
+        "header": {
+            "type"                  : "object",
+            "additionalProperties"  : true,
+            "required"              : [ "uploaderID", "softwareName", "softwareVersion" ],
+            "properties"            : {
+                "uploaderID": {
+                    "type"          : "string"
+                },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
+                "softwareName": {
+                    "type"          : "string"
+                },
+                "softwareVersion": {
+                    "type"          : "string"
+                },
+                "gatewayTimestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time",
+                    "description"   : "Timestamp upon receipt at the gateway. If present, this property will be overwritten by the gateway; submitters are not intended to populate this property."
+                }
+            }
+        },
+        "message": {
+            "type"                  : "object",
+            "description"           : "Contains all properties from the listed events in the client's journal minus the Localised strings and the properties marked below as 'disallowed'",
+            "additionalProperties"  : false,
+            "required"              : [ "timestamp", "event", "MarketID", "StationName" ],
+            "properties"            : {
+                "timestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time"
+                },
+                "event" : {
+                    "enum"          : [ "DockingGranted" ]
+                },
+                "horizons": {
+                    "type"          : "boolean",
+                    "description"   : "Whether the sending Cmdr has a Horizons pass."
+                },
+                "odyssey": {
+                    "type"      : "boolean",
+                    "description" : "Whether the sending Cmdr has an Odyssey expansion."
+                },
+                "MarketID": {
+                    "type"          : "integer"
+                },		
+                "StationName": {
+                    "type"          : "string",
+                    "description"   : "Name of station"
+                },
+                "StationType": {
+                    "type"          : "string",
+                    "description"   : "Type of station"
+                },
+                "LandingPad": {
+                    "type"          : "integer",
+                    "description"   : "Pad number Cmdr was granted landing to"
+                }
+            }
+        }
+    },
+    "definitions": {
+        "disallowed" : { "not" : { "type": [ "array", "boolean", "integer", "number", "null", "object", "string" ] } }
+    }
+}

schemas/fcmaterials_capi-README.md

@@ -0,0 +1,82 @@
+# EDDN FCMaterials Schema
+
+## Introduction
+This is the documentation for how to take data from an the ED CAPI `/market`
+endpoint and properly structure it for sending to EDDN.
+
+Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
+documentation for a schema such as this.
+
+If you find any discrepancies between what this document says and what is
+defined in the relevant Schema file, then you should, in the first instance,
+assume that it is the Schema file that is correct.
+**PLEASE open
+[an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose)
+to report any such anomalies you find so that we can check and resolve the
+discrepancy.**
+
+## Senders
+The data source for this schema is the `/market` ED CAPI endpoint.  You only
+want selected parts of the full data returned for this schema.
+
+You **MUST NOT** construct the message by starting with the entirety of the
+CAPI data and then removing everything but what you need.  That risks Frontier
+adding more data to the endpoint and your `fcmaterials_capi` messages being
+rejected as invalid.  Instead, construct the message content by setting just
+the data that is necessary.
+
+Your `message` object **MUST**:
+1. Have an `"event":"FCMaterials"` member to aid Listeners who pass this
+   through a "usually from the Journal" code path.
+2. Set a `"MarketID"` key with the value from `"id"` in the CAPI data.
+3. Set a `"CarrierID"` key with the value from the `"name"` in the CAPI data.
+4. Set the `"Items"` key's contents directly from the `/market` -> `orders`
+   -> `onfootmicroresources` CAPI data.
+5. Remove any data where the key is `"locName"` from the `"Items"` data.
+
+You **MUST NOT**:
+1. Attempt to set a `"CarrierName"` from any source, the `CarrierID` is
+   sufficient.
+
+### Example algorithm
+
+1. Make a CAPI `/market` query.
+2. Set `event`, `MarketID` and `CarrierID` as outlined above.
+3. Set `Items` value to the data in `orders.onfootmicroresources`.
+4. Process the contents of `Items`, removing any data with a key of `locName`.
+
+### Augmentations
+#### horizons and odyssey flags
+Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
+in the Developers' documentation.
+
+You **SHOULD** set these flags from the Journal `FileHeader` data if you are
+reasonably sure you have a live game session against which you are performing
+CAPI queries.
+You **MUST NOT** set them otherwise, as e.g. the player could be active in
+the game on another computer, using a different game mode and the CAPI data
+will be for that game mode.
+
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
+## Listeners
+The advice above for [Senders](#senders), combined with the actual Schema file
+*should* provide all the information you need to process these events.
+
+Do note that the data source for this is the CAPI, and as such the data is not
+the same as for the `fcmaterials_journal` schema:
+
+1. There is no good source of `CarrierName` in CAPI `/market` endpoint data, so 
+   that is not included.
+2. The `sales` and `purchases` values do **not** contain the same form of data.
+3. The `sales` member of `Items` will be `[]` if there are no sales orders, but
+   when there are orders:
+    1. It will be an object/dictionary.
+    2. The keys are the commodity ID.
+    3. The value of that key is the rest of the data for that sales order.
+4. The `purchases` value:
+    1. Is always an array, unlike `sales`.
+    2. As a consequence does **not** provide the commodity id at all, only
+      the name.

schemas/fcmaterials_capi-v1.0.json

@@ -0,0 +1,152 @@
+{
+    "$schema"               : "http://json-schema.org/draft-04/schema#",
+    "id"                    : "https://eddn.edcd.io/schemas/fcmaterials_capi/1",
+    "type"                  : "object",
+    "additionalProperties"  : false,
+    "required": [ "$schemaRef", "header", "message" ],
+    "properties": {
+        "$schemaRef": {
+            "type"                  : "string"
+        },
+        "header": {
+            "type"                  : "object",
+            "additionalProperties"  : true,
+            "required"              : [ "uploaderID", "softwareName", "softwareVersion" ],
+            "properties"            : {
+                "uploaderID": {
+                    "type"          : "string"
+                },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "Value of 'CAPI-market' if possible, else empty string."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "Value of 'CAPI-market' if possible, else empty string."
+                },
+                "softwareName": {
+                    "type"          : "string"
+                },
+                "softwareVersion": {
+                    "type"          : "string"
+                },
+                "gatewayTimestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time",
+                    "description"   : "Timestamp upon receipt at the gateway. If present, this property will be overwritten by the gateway; submitters are not intended to populate this property."
+                }
+            }
+        },
+        "message": {
+            "type"                  : "object",
+            "description"           : "Contains all properties from the listed events in the client's journal minus the Localised strings and the properties marked below as 'disallowed'",
+            "additionalProperties"  : false,
+            "required"              : [ "timestamp", "event", "MarketID", "CarrierID", "Items" ],
+            "properties"            : {
+                "timestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time"
+                },
+                "event" : {
+                    "enum"          : [ "FCMaterials" ]
+                },
+                "horizons": {
+                    "type"          : "boolean",
+                    "description"   : "Boolean value copied from the Journal LoadGame event, when it is present there."
+                },
+                "odyssey": {
+                    "type"          : "boolean",
+                    "description"   : "Boolean value copied from the Journal LoadGame event, when it is present there."
+                },
+
+                "MarketID": {
+                    "type"          : "integer"
+                },
+
+                "CarrierID": {
+                    "type"          : "string",
+                    "minLength"     : 1
+                },
+
+                "Items": {
+                    "properties": {
+                        "purchases": {
+                            "type": "array",
+                            "items": {
+                                "type": "object",
+                                "required": [ "name", "price", "outstanding", "total" ],
+                                "additionalProperties": false,
+                                "properties": {
+                                    "name": {
+                                        "type": "string",
+                                        "minLength": 1
+                                    },
+                                    "locName": {
+                                        "$ref": "#/definitions/disallowed"
+                                    },
+                                    "outstanding": {
+                                        "type": "integer"
+                                    },
+                                    "price": {
+                                        "type": "integer"
+                                    },
+                                    "total": {
+                                        "type": "integer"
+                                    }
+                                }
+                            }
+                        },
+                        "sales": {
+                            "anyOf": [
+                                {
+                                    "type": "array",
+                                    "$comment": "If there are no items then sales is an empty array",
+                                    "minItems": 0,
+                                    "maxItems": 0
+                                },
+                                {
+                                    "type": "object",
+                                    "$comment": "If there ARE items then sales is an object, *NOT* an array",
+                                    "patternProperties": {
+                                        "^[0-9]+$": {
+                                            "type"      : "object",
+                                            "required"  : [ "id", "name", "price", "stock" ],
+                                            "additionalProperties": false,
+                                            "properties": {
+                                                "id"    : {
+                                                    "type"  : "integer"
+                                                },
+                                                "name"  : {
+                                                    "type"  : "string",
+                                                    "minLength": 1
+                                                },
+                                                "locName"   : {
+                                                    "$ref": "#/definitions/disallowed"
+                                                },
+                                                "price" : {
+                                                    "type": "integer"
+                                                },
+                                                "stock": {
+                                                    "type": "integer"
+                                                }
+                                            }
+                                        }
+                                    }
+                                }
+                            ]
+                        }
+                    }
+                }
+            }
+        }
+    },
+    "definitions": {
+        "disallowed": {
+            "not" : {
+                "type": [
+                    "array", "boolean", "integer", "number", "null", "object", "string"
+                ]
+            }
+        }
+    }
+}

schemas/fcmaterials_journal-README.md

@@ -0,0 +1,42 @@
+# EDDN FCMaterials Schema
+
+## Introduction
+This is the documentation for how to take data from an ED `FCMaterials.json`
+file and properly structure it for sending to EDDN.
+
+Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
+documentation for a schema such as this.
+
+If you find any discrepancies between what this document says and what is
+defined in the relevant Schema file, then you should, in the first instance,
+assume that it is the Schema file that is correct.
+**PLEASE open
+[an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose)
+to report any such anomalies you find so that we can check and resolve the
+discrepancy.**
+
+## Senders
+The data source for this schema is the file `FCMaterials.json`.  That it has
+been freshly written is signalled by the ED Journal event `FCMaterials`.
+**NB: This schema is not, currently, for sending CAPI `/market`-sourced data
+about these materials.**
+
+So, monitor the Journal as normal, and when you see a `FCMaterials` event open
+the `FCMaterials.json` file for reading, read it, and close it again.  Use the
+data you got from reading this file, not merely the Journal event.
+
+Your `message` should primarily be the contents of this file, with the addition
+of any augmentations, as noted below.
+
+### Augmentations
+#### horizons and odyssey flags
+Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
+in the Developers' documentation.
+
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
+## Listeners
+The advice above for [Senders](#senders), combined with the actual Schema file
+*should* provide all the information you need to process these events.

schemas/fcmaterials_journal-v1.0.json

@@ -0,0 +1,110 @@
+{
+    "$schema"               : "http://json-schema.org/draft-04/schema#",
+    "id"                    : "https://eddn.edcd.io/schemas/fcmaterials_journal/1",
+    "type"                  : "object",
+    "additionalProperties"  : false,
+    "required": [ "$schemaRef", "header", "message" ],
+    "properties": {
+        "$schemaRef": {
+            "type"                  : "string"
+        },
+        "header": {
+            "type"                  : "object",
+            "additionalProperties"  : true,
+            "required"              : [ "uploaderID", "softwareName", "softwareVersion" ],
+            "properties"            : {
+                "uploaderID": {
+                    "type"          : "string"
+                },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
+                "softwareName": {
+                    "type"          : "string"
+                },
+                "softwareVersion": {
+                    "type"          : "string"
+                },
+                "gatewayTimestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time",
+                    "description"   : "Timestamp upon receipt at the gateway. If present, this property will be overwritten by the gateway; submitters are not intended to populate this property."
+                }
+            }
+        },
+        "message": {
+            "type"                  : "object",
+            "description"           : "Contains all properties from the listed events in the client's journal minus the Localised strings and the properties marked below as 'disallowed'",
+            "additionalProperties"  : false,
+            "required"              : [ "timestamp", "event", "MarketID", "CarrierName", "CarrierID", "Items" ],
+            "properties"            : {
+                "timestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time"
+                },
+                "event" : {
+                    "enum"          : [ "FCMaterials" ]
+                },
+                "horizons": {
+                    "type"          : "boolean",
+                    "description"   : "Boolean value copied from the Journal LoadGame event, when it is present there."
+                },
+                "odyssey": {
+                    "type"          : "boolean",
+                    "description"   : "Boolean value copied from the Journal LoadGame event, when it is present there."
+                },
+
+                "MarketID": {
+                    "type"          : "integer"
+                },
+
+                "CarrierName": {
+                    "type"          : "string",
+                    "minLength"     : 1
+                },
+
+                "CarrierID": {
+                    "type"          : "string",
+                    "minLength"     : 1
+                },
+
+                "Items": {
+                    "type"          : "array",
+                    "items": {
+                        "type"          : "object",
+                        "required"      : [ "id", "Name", "Price", "Stock", "Demand" ],
+                        "properties"    : {
+                            "id" : {
+                                "type"  : "integer"
+                            },
+                            "Name": {
+                                "type"      : "string",
+                                "minLength" : 1
+                            },
+                            "Price": {
+                                "type"      : "integer"
+                            },
+                            "Stock": {
+                                "type"      : "integer"
+                            },
+                            "Demand": {
+                                "type"      : "integer"
+                            }
+                        },
+                        "patternProperties": {
+                            "_Localised$"       : { "$ref" : "#/definitions/disallowed" }
+                        }
+                    }
+                }
+            }
+        }
+    },
+    "definitions": {
+        "disallowed" : { "not" : { "type": [ "array", "boolean", "integer", "number", "null", "object", "string" ] } }
+    }
+}

schemas/fssallbodiesfound-README.md

@@ -24,6 +24,13 @@ The primary data source for this schema is the ED Journal event
 Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
 in the Developers' documentation.
 
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
 #### StarPos
 You MUST add a `StarPos` array containing the system co-ordinates from the 
 last `FSDJump`, `CarrierJump`, or `Location` event.
+
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**

schemas/fssallbodiesfound-v1.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

schemas/fssbodysignals-README.md

@@ -0,0 +1,53 @@
+# EDDN FSSBodySignals Schema
+
+## Introduction
+Here we document how to take data from an ED `FSSBodySignals` Journal 
+Event and properly structure it for sending to EDDN.
+
+Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
+documentation for a schema such as this.
+
+If you find any discrepancies between what this document says and what is
+defined in the relevant Schema file, then you should, in the first instance,
+assume that it is the Schema file that is correct.
+**PLEASE open
+[an issue on GitHub](https://github.com/EDCD/EDDN/issues/new/choose)
+to report any such anomalies you find so that we can check and resolve the
+discrepancy.**
+
+## Senders
+The primary data source for this schema is the ED Journal event 
+`FSSBodySignals`.
+
+### Augmentations
+#### horizons and odyssey flags
+Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
+in the Developers' documentation.
+
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
+#### StarSystem
+You MUST add a `StarSystem` string containing the name of the system from the 
+last `FSDJump`, `CarrierJump`, or `Location` event.
+
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**
+
+#### StarPos
+You MUST add a `StarPos` array containing the system co-ordinates from the 
+last `FSDJump`, `CarrierJump`, or `Location` event.
+
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**
+
+#### Remove _Localised key/values
+All keys whose name ends with `_Localised`, i.e. the `Type_Localised`
+key/values in Signals.
+
+#### Examples:
+
+```json
+{ "timestamp":"2022-05-18T00:10:57Z", "event":"FSSBodySignals", "BodyName":"Phoi Auwsy ZY-Z d132 7 a", "BodyID":37, "SystemAddress":4546986398603, "Signals":[ { "Type":"$SAA_SignalType_Geological;", "Type_Localised":"Geological", "Count":2 } ] }
+```

schemas/fssbodysignals-v1.0.json

@@ -0,0 +1,104 @@
+{
+    "$schema"               : "http://json-schema.org/draft-04/schema#",
+    "id"                    : "https://eddn.edcd.io/schemas/fssbodysignals/1#",
+    "type"                  : "object",
+    "additionalProperties"  : false,
+    "required": [ "$schemaRef", "header", "message" ],
+    "properties": {
+        "$schemaRef": {
+            "type"                  : "string"
+        },
+        "header": {
+            "type"                  : "object",
+            "additionalProperties"  : true,
+            "required"              : [ "uploaderID", "softwareName", "softwareVersion" ],
+            "properties"            : {
+                "uploaderID": {
+                    "type"          : "string"
+                },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
+                "softwareName": {
+                    "type"          : "string"
+                },
+                "softwareVersion": {
+                    "type"          : "string"
+                },
+                "gatewayTimestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time",
+                    "description"   : "Timestamp upon receipt at the gateway. If present, this property will be overwritten by the gateway; submitters are not intended to populate this property."
+                }
+            }
+        },
+        "message": {
+            "type"                  : "object",
+            "description"           : "Contains all properties from the listed events in the client's journal, minus the Localised strings and the properties marked below as 'disallowed'",
+            "additionalProperties"  : false,
+            "required"              : [ "timestamp", "event", "StarSystem", "StarPos", "SystemAddress", "BodyID", "Signals" ],
+            "properties"            : {
+                "timestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time"
+                },
+                "event" : {
+                    "enum"          : [ "FSSBodySignals" ]
+                },
+                "horizons": {
+                    "type"          : "boolean",
+                    "description"   : "Whether the sending Cmdr has a Horizons pass."
+                },
+                "odyssey": {
+                    "type"      : "boolean",
+                    "description" : "Whether the sending Cmdr has an Odyssey expansion."
+                },
+                "StarSystem": {
+                    "type"          : "string",
+                    "minLength"     : 1
+                },
+                "StarPos": {
+                    "type"          : "array",
+                    "items"         : { "type": "number" },
+                    "minItems"      : 3,
+                    "maxItems"      : 3,
+                    "description"   : "Must be added by the sender if not present in the journal event"
+                },
+                "SystemAddress": {
+                    "type"          : "integer"
+                },
+                "BodyID"             : {
+                    "type"          : "integer"
+                },
+                "BodyName": {
+                    "type"          : "string",
+                    "minLength"     : 1
+                },
+                "Signals": {
+                    "type"          : "array",
+                    "items"         : {
+                        "type"          : "object",
+                        "additionalProperties"  : false,
+                        "required"              : [ "Type", "Count" ],
+                        "properties"            : {
+                            "Type"             : {
+                                "type"          : "string"
+                            },
+                            "Count"             : {
+                                "type"          : "integer"
+                            }
+                        }
+                    }
+                }
+            }
+        }
+    },
+    "definitions": {
+        "disallowed" : { "not" : { "type": [ "array", "boolean", "integer", "number", "null", "object", "string" ] } }
+    }
+}

schemas/fssdiscoveryscan-README.md

@@ -24,6 +24,13 @@ The primary data source for this schema is the ED Journal event
 Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
 in the Developers' documentation.
 
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
 #### StarPos
 You MUST add a `StarPos` array containing the system co-ordinates from the 
 last `FSDJump`, `CarrierJump`, or `Location` event.
+
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**

schemas/fssdiscoveryscan-v1.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

schemas/fsssignaldiscovered-README.md

@@ -0,0 +1,191 @@
+# EDDN FSSSignalDiscovered Schema
+
+## Introduction
+Here we document how to take data from an ED `FSSSignalDiscovered` Journal 
+Event and properly structure it for sending to EDDN.
+
+Please consult [EDDN Schemas README](./README-EDDN-schemas.md) for general
+documentation for a schema such as this.
+
+## Senders
+The only data source for this schema is the ED Journal event 
+`FSSSignalDiscovered`.
+
+### Batching
+You MUST coalesce contiguous runs of `FSSSignalDiscovered` events into a
+single `signals` array in the message. Minimum size of `signals` is 1 item.
+
+Do not make a request for every single event other than where they occur
+singly (such as when a player utilises the FSS to zoom into USS individually,
+if there is a different following event).
+
+Suggested algorithm for batching:
+
+1. You will need to track the current location from `Location`, `FSDJump` and
+  `CarrierJump` events.  This is in order to add the top-level augmentation
+  of `StarSystem` (system name) and `StarPos`.  You will need to record:
+    1. `SystemAddress` - for cross-checking.
+    2. `StarSystem` - name of the star system.
+    3. `StarPos` - the galactic co-ordinates of the system.
+2. If the event is `FSSSignalDiscovered`, store it to the temporal list.
+3. If the event is any other, then:
+    1. check if it is `Location`, `FSDJump` or `CarrierJump` - if so you should
+      use this new location in the message for the augmentations.
+    2. If it is not one of those events then you should use the tracked
+      location from the prior such event for the augmentations.
+   
+    Now construct the full `fsssignaldiscovered` schema message using the
+    tracked location and the stored list of events.  *You **MUST** check that
+    the `SystemAddress` for each `FSSSignalDiscovered` event matches the
+    tracked location.*  If there is a mis-match then drop that event.
+4. Use the `timestamp` of the first signal in the batch as the top-level
+   `timestamp` in the `message` object.
+
+Point 3i/ii above are because in the current (3.8.0.406) Horizons client the
+`FSSSignalDiscovered` events arrive after `Location`/`FSDJump`/`CarrierJump`,
+but in the current (4.0.0.1302) Odyssey client they arrive before such events.
+
+Thus, in Horizons you use the last-tracked location, but in Odyssey you use
+the "just arrived" location.
+
+Manually FSS-scanned USS type signals will come in one by one, possibly with
+other events between them (such as `Music` due to zooming in/out in FSS).
+There is no need to attempt batching these together if separated by other
+events, even though you'll be using the `timestamp` of the first on the
+message, despite the actual time-line being dependent on how quickly the
+player scans them.
+
+This batching is more concerned with not causing an EDDN message per event
+upon entry into a system.
+
+### Elisions
+Remove the `event` key/pair from each member of the `signals` array.  Including
+this would be redundant as by definition we're sending `FSSSignalDiscovered`
+events on this schema.
+
+You MUST remove the following key/value pairs from the data:
+
+  - `TimeRemaining` key/value pair (will be present on USS).  This has a slight
+    PII nature and is also very ephemeral.
+
+You MUST drop the whole `FSSSignalDiscovered` event if `USSType` key
+has `$USS_Type_MissionTarget;` value.  Only the Cmdr with the mission has any
+use of these.  There's not even a statistical use.
+
+Because of the location cross-check the `SystemAddress` is in the top-level
+`message` object, and thus you **MUST** remove such from each signal in the
+array.
+
+Do **NOT** remove the `timestamp` from each signal in the array.  Whilst these
+should be identical for a "just logged in or arrived in system" set of signals,
+this is not true of manually FSS scanned USS signals.
+
+### Augmentations
+#### horizons flag
+You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+
+#### odyssey flag
+You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
+#### StarSystem
+You **MUST** add a `StarSystem` string containing the system name from the last
+tracked location.  You **MUST** cross-check each `FSSSignalDiscovered`
+->`SystemAddress` value to ensure it matches.  If it does not, you **MUST**
+drop the event.
+
+#### StarPos
+You **MUST** add a `StarPos` array containing the system co-ordinates from the
+tracked location.  You **MUST** cross-check each `FSSSignalDiscovered`
+->`SystemAddress` value to ensure it matches.  If it does not, you **MUST**
+drop the event.
+
+## Receivers
+### Augmentations are 'SHOULD', not 'MUST'
+Receivers should remember that  `horizons` and  `odyssey` augmentations
+are optional key/value pairs.  You **SHOULD NOT** rely on them being present
+in any given event.
+
+### Duplicate messages from 'busy' systems
+When a system is particularly full of signals, such as when many Fleet Carriers
+are present, it has been observed that the game repeats the identical
+sequence of `FSSSignalDiscovered` events.  So you might receive what looks like
+a duplicate message, other than the timestamp (if the timestamp is the same
+then the EDDN Relay should drop the duplicate).
+
+## Examples
+This is a few example of messages that passes current `FSSSignalDiscovered` schema.
+1. A message without `horizons` or `odyssey` augmentations.
+```json
+{
+   "$schemaRef":"https://eddn.edcd.io/schemas/fsssignaldiscovered/1",
+   "header":{
+      "gatewayTimestamp":"2021-11-06T22:48:43.483147Z",
+      "softwareName":"a software",
+      "softwareVersion":"a version",
+      "uploaderID":"an uploader"
+   },
+   "message":{
+      "timestamp":"2021-11-06T22:48:42Z",
+      "event":"FSSSignalDiscovered",
+      "SystemAddress":1774711381,
+      "signals":[
+         {
+            "timestamp":"2021-11-06T22:48:42Z",
+            "SignalName":"EXPLORER-CLASS X2X-74M",
+            "IsStation":true
+         }
+      ],
+      "StarSystem":"HR 1185",
+      "StarPos": [
+          -64.66, -148.94, -330.41
+      ]
+   }
+}
+```
+
+2. A message with `horizons`, `odyssey`, `systemName`, `StarPos` fields which says it sent from Odyssey.
+```json
+{
+   "$schemaRef":"https://eddn.edcd.io/schemas/fsssignaldiscovered/1",
+   "header":{
+      "gatewayTimestamp":"2021-11-06T22:48:43.483147Z",
+      "softwareName":"a software",
+      "softwareVersion":"a version",
+      "uploaderID":"an uploader"
+   },
+   "message":{
+      "timestamp":"2021-11-06T22:48:42Z",
+      "event":"FSSSignalDiscovered",
+      "SystemAddress":1350507186531,
+      "signals":[
+         {
+            "timestamp":"2021-11-06T22:48:42Z",
+            "event":"FSSSignalDiscovered",
+            "SignalName":"EXPLORER-CLASS X2X-74M",
+            "IsStation":true
+         },
+         {
+            "timestamp":"2021-11-06T22:48:42Z",
+            "event":"FSSSignalDiscovered", 
+            "SignalName":"$USS_NonHumanSignalSource;", 
+            "USSType":"$USS_Type_NonHuman;",
+            "SpawningState":"$FactionState_None;", 
+            "SpawningFaction":"$faction_none;",
+            "ThreatLevel":5
+            }
+      ],
+      "StarPos": [
+          8.1875,
+          124.21875,
+          -38.5
+      ],
+      "StarSystem": "HIP 56186",
+      "horizons": true,
+      "odyssey": true
+   }
+}
+```

schemas/fsssignaldiscovered-v1.0.json

@@ -0,0 +1,120 @@
+{
+    "$schema"               : "http://json-schema.org/draft-04/schema#",
+    "id"                    : "https://eddn.edcd.io/schemas/fsssignaldiscovered/1#",
+    "description"           : "EDDN schema for FSSSignalDiscovered Journal events.  Full documentation at https://github.com/EDCD/EDDN/tree/master/schemas/fsssignaldiscovered-README.md",
+    "type"                  : "object",
+    "additionalProperties"  : false,
+    "required": [ "$schemaRef", "header", "message" ],
+    "properties": {
+        "$schemaRef": {
+            "type"                  : "string"
+        },
+        "header": {
+            "type"                  : "object",
+            "additionalProperties"  : true,
+            "required"              : [ "uploaderID", "softwareName", "softwareVersion" ],
+            "properties"            : {
+                "uploaderID": {
+                    "type"          : "string"
+                },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
+                "softwareName": {
+                    "type"          : "string"
+                },
+                "softwareVersion": {
+                    "type"          : "string"
+                },
+                "gatewayTimestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time",
+                    "description"   : "Timestamp upon receipt at the gateway. If present, this property will be overwritten by the gateway; submitters are not intended to populate this property."
+                }
+            }
+        },
+        "message": {
+            "type"                  : "object",
+            "description"           : "Contains all properties from the listed events in the client's journal minus Localised strings and the properties marked below as 'disallowed'",
+            "additionalProperties"  : false,
+            "required"              : [ "event", "timestamp", "SystemAddress", "StarSystem", "StarPos", "signals"],
+            "properties"            : {
+                "event": {
+                    "enum"          : [ "FSSSignalDiscovered" ]
+                },
+                "horizons": {
+                    "type"          : "boolean",
+                    "description"   : "Whether the sending Cmdr has a Horizons pass."
+                },
+                "odyssey": {
+                    "type"      : "boolean",
+                    "description" : "Whether the sending Cmdr has an Odyssey expansion."
+                },
+                "timestamp": {
+                    "type"          : "string",
+                    "format"        : "date-time",
+                    "description"   : "Duplicate of the first signal's timestamp, for commonality with other schemas."
+                },
+                "SystemAddress": {
+                    "type": "integer"
+                },
+                "signals": {
+                    "type": "array",
+                    "description": "Array of FSSSignalDiscovered events",
+                    "minItems": 1,
+                    "items": {
+                        "type": "object",
+                        "additionalProperties"  : false,
+                        "required": ["timestamp", "SignalName"],
+                        "description": "Single FSSSignalDiscovered event",
+                        "properties": {
+                            "timestamp": {
+                                "type"          : "string",
+                                "format"        : "date-time"
+                            },
+                            "SignalName": { "type":  "string" },
+                            "SignalType": { "type":  "string" },
+                            "IsStation": { "type": "boolean" },
+                            "USSType": {
+                              "type": "string",
+                               "not": {
+                                    "pattern": "^\\$USS_Type_MissionTarget;$"
+                                }
+                            },
+                            "TimeRemaining": {"$ref" : "#/definitions/disallowed"},
+                            "SpawningState": {"type":  "string"},
+                            "SpawningFaction" : {"type":  "string"},
+                            "SpawningPower" : {"type":  "string"},
+                            "OpposingPower" : {"type":  "string"},
+                            "ThreatLevel": {"type": "integer" },
+
+                            "patternProperties": {
+                                "_Localised$"       : { "$ref" : "#/definitions/disallowed" }
+                            }
+                        }
+                    }
+                },
+                "StarSystem": {
+                    "type"          : "string",
+                    "minLength"     : 1,
+                    "description": "Should be added by the sender"
+                },
+                "StarPos": {
+                    "type"          : "array",
+                    "items"         : { "type": "number" },
+                    "minItems"      : 3,
+                    "maxItems"      : 3,
+                    "description"   : "Should be added by the sender"
+                }
+            }
+        }
+    },
+    "definitions": {
+        "disallowed" : { "not" : { "type": [ "array", "boolean", "integer", "number", "null", "object", "string" ] } }
+    }
+}

schemas/journal-README.md

@@ -65,22 +65,35 @@ The following keys+values should be removed from `Location` event data:
 - `SquadronFaction` from within the list of `Factions`.
 
 ### Augmentations
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
 #### horizons flag
-You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+You **MUST** add this key/value pair, using the value from the `LoadGame` event.
+
+Note caveats in [docs/Developers.md](../docs/Developers.md).
 
 #### odyssey flag
-You SHOULD add this key/value pair, using the value from the `LoadGame` event.
+You **MUST** add this key/value pair, using the value from the `LoadGame` event.
+
+Note caveats in [docs/Developers.md](../docs/Developers.md).
 
 #### StarSystem
-You MUST add a `StarSystem` key/value pair representing the name of the 
-system this event occurred in.  Source this from either `Location`, 
-`FSDJump` or `CarrierJump` as appropriate.
+If not already present, you MUST add a `StarSystem` string containing the
+name of the system from the last `FSDJump`, `CarrierJump`, or `Location` event.
 
-#### SystemAddress
-You MUST add a `SystemAddress` key/value pair representing the numerical ID 
-of the system this event occurred in.  Source this from either `Location`,
-`FSDJump` or `CarrierJump` as appropriate.
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**
+
+This should only apply to `SAASignalsFound` events.
 
 #### StarPos
-You MUST add a `StarPos` array containing the system co-ordinates from the
-last `FSDJump`, `CarrierJump`, or `Location` event.
+If not already present, you  MUST add a `StarPos` array containing the
+system co-ordinates from the last `FSDJump`, `CarrierJump`, or `Location`
+event.
+
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**
+
+This should only apply to `Docked`, `Scan` and `SAASignalsFound` events.

schemas/journal-v1.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

schemas/navbeaconscan-README.md

@@ -24,11 +24,21 @@ The primary data source for this schema is the ED Journal event
 Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
 in the Developers' documentation.
 
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
 #### StarSystem
 You MUST add a `StarSystem` key/value pair representing the name of the
 system this event occurred in.  Source this from either `Location`,
 `FSDJump` or `CarrierJump` as appropriate.
 
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**
+
 #### StarPos
 You MUST add a `StarPos` array containing the system co-ordinates from the
 last `FSDJump`, `CarrierJump`, or `Location` event.
+
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**

schemas/navbeaconscan-v1.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

schemas/navroute-README.md

@@ -31,3 +31,7 @@ separate file.
 Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
 in the Developers' documentation.
 
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+

schemas/navroute-v1.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

schemas/outfitting-README.md

@@ -53,3 +53,8 @@ station. Namely:
 #### horizons and odyssey flags
 Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
 in the Developers' documentation.
+
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+

schemas/outfitting-v2.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "Fileheader->gameversion, else LoadGame->gameversion, else 'CAPI-shipyard', else ''."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "Fileheader->build, else LoadGame->build, else 'CAPI-shipyard', else ''."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

schemas/scanbarycentre-README.md

@@ -27,6 +27,13 @@ senders SHOULD include any defined in the schema if it's in the source data.
 Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
 in the Developers' documentation.
 
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+
 #### StarPos
 You MUST add a `StarPos` array containing the system co-ordinates from the
 last `FSDJump`, `CarrierJump`, or `Location` event.
+
+**You MUST apply a location cross-check, as per
+[Other data augmentations](../docs/Developers.md#other-data-augmentations).**

schemas/scanbarycentre-v1.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "From Fileheader event if available, else LoadGame if available there."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "The `build` value from a Fileheader event if available, else LoadGame if available there."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

schemas/shipyard-README.md

@@ -40,3 +40,8 @@ value is what the name would have been in the source Journal data.
 #### horizons and odyssey flags
 Please read [horizons and odyssey flags](../docs/Developers.md#horizons-and-odyssey-flags)
 in the Developers' documentation.
+
+#### gameversion and gamebuild
+You **MUST** always set these as per [the relevant section](../docs/Developers.md#gameversions-and-gamebuild)
+of the Developers' documentation.
+

schemas/shipyard-v2.0.json

@@ -16,6 +16,14 @@
                 "uploaderID": {
                     "type"          : "string"
                 },
+                "gameversion": {
+                    "type"          : "string",
+                    "description"   : "Fileheader->gameversion, else LoadGame->gameversion, else 'CAPI-shipyard', else ''."
+                },
+                "gamebuild": {
+                    "type"          : "string",
+                    "description"   : "Fileheader->build, else LoadGame->build, else 'CAPI-shipyard', else ''."
+                },
                 "softwareName": {
                     "type"          : "string"
                 },

scripts/eddn-report-log-errors

@@ -3,8 +3,11 @@
 """Produce a report on the provided EDDN Gateway log file's ERRORs."""
 
 import argparse
+import fileinput
 import re
 
+import semantic_version
+
 
 def parse_cl_args() -> str:
   """
@@ -43,7 +46,7 @@ def process_file(input_file: str) -> None:
     r' from (?P<sender_ip>.+)$'
   )
   # TODO: Make this handle gzipped files
-  with open(input_file, 'r') as input:
+  with fileinput.FileInput(files=(input_file), mode='r') as input:
     line = input.readline()
     while line:
       line = line.strip()
@@ -59,90 +62,119 @@ def process_file(input_file: str) -> None:
         # print(matches.group('sender_ip'))
         # print('')
 
+        try:
+          software_version = semantic_version.Version.coerce(matches.group('software_version'))
+
+        except ValueError as e:
+          print(f"Error parsing sofwareVersion for:\n{matches.group('software_version')}\n{line}\n")
+          next
+
         ###################################################################
         # Issues we know about and HAVE already alerted their
         # developers to.
         ###################################################################
         if matches.group('software_name') == 'EDDiscovery':
           # https://github.com/EDDiscovery/EDDiscovery/releases/latest
-          if matches.group('software_version') == '12.1.7.0':
-            if matches.group('schema_ref') in (
-              'https://eddn.edcd.io/schemas/shipyard/2',
-              'https://eddn.edcd.io/schemas/outfitting/2',
-            ):
-              # Reported via Discord PM to Robby 2022-01-07
-              if matches.group('err_msg') != 'Failed Validation "[<ValidationError: \'[] is too short\'>]"':
+          if software_version >= semantic_version.Version.coerce('16.0.5.0'):
+            if matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/outfitting/2':
+              err_msg = matches.group('err_msg')
+              if (
+                err_msg.startswith('Failed Validation "[<ValidationError: "\'paintjob_') and
+                err_msg.find('\' does not match \'(^Hpt_|^hpt_|^Int_|^int_|_Armour_|_armour_)\'">]') != -1
+              ):
+                # <https://github.com/EDDiscovery/EDDiscovery/issues/3328>
+                pass
+
+              else: 
                 print(line)
 
             else:
               print(line)
 
         elif matches.group('software_name') == 'EDDLite':
-          # https://github.com/EDDiscovery/EDDLite/releases/tag/latest
-          if matches.group('software_version') == '2.0.0':
-            if matches.group('schema_ref') in (
-              'https://eddn.edcd.io/schemas/shipyard/2',
-              'https://eddn.edcd.io/schemas/outfitting/2',
-            ):
-              # Reported via Discord PM to Robby 2022-01-07
-              if matches.group('err_msg') != 'Failed Validation "[<ValidationError: \'[] is too short\'>]"':
-                print(line)
-
-            else:
-              print(line)
+          # https://github.com/EDDiscovery/EDDLite/releases/latest
+          if software_version >= semantic_version.Version.coerce('2.5.0'):
+            print(line)
 
         elif matches.group('software_name') == 'EDDI':
           # https://github.com/EDCD/EDDI/releases/latest
-          if matches.group('software_version') == '4.0.1':
-            print(line)
+          if software_version >= semantic_version.Version.coerce('4.0.2'):
 
-        elif matches.group('software_name') == 'E:D Market Connector [Windows]':
-          # https://github.com/EDCD/EDMarketConnector/releases/latest
-          if matches.group('software_version') == '5.2.4':
-            if matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/codexentry/1':
-              # <https://github.com/EDCD/EDMarketConnector/issues/1393>
-              if matches.group('err_msg') != 'Failed Validation "[<ValidationError: "\'\' is too short">]"':
-                print(matches.group('err_msg'))
-                print(line)
-
-            elif matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/journal/1':
-              # <https://github.com/EDCD/EDMarketConnector/issues/1403>
-              if matches.group('err_msg') == 'Failed Validation "[<ValidationError: "\'SystemAddress\' is a required property">]"':
-                # <https://github.com/EDCD/EDMarketConnector/issues/1403>
+            if matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/fsssignaldiscovered/1':
+              if matches.group('err_msg').startswith(
+                'Failed Validation "[<ValidationError: "\'StarPos\' is a required property"'
+                ):
+                # Reported on Discord: <https://discord.com/channels/164411426939600896/353595704658231299/1062652620986134608>
                 pass
+             
+            elif matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/navroute/1':
+              if matches.group('err_msg').startswith(
+                'Failed Validation "[<ValidationError: "\'Route\' is a required property"'
+                ):
+                # Reported on Discord: <https://discord.com/channels/164411426939600896/353595704658231299/1063017819752648775>
+                pass
+             
+            else:
+              print(line)
 
-              elif matches.group('err_msg').startswith(
+        elif matches.group('software_name').startswith('E:D Market Connector'):
+          # https://github.com/EDCD/EDMarketConnector/releases/latest
+          if software_version >= semantic_version.Version.coerce('5.7.0'):
+            if matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/journal/1':
+              if matches.group('err_msg').startswith(
                 'Failed Validation "[<ValidationError: "{\'type\': [\'array\', \'boolean\', \'integer\', \'number\', \'null\', \'object\', \'string\']} is not allowed for'
                 ):
                 # <https://github.com/EDCD/EDMarketConnector/issues/1403>
                 pass
 
-            elif matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/fssdiscoveryscan/1':
-              if matches.group('err_msg') == 'Failed Validation "[<ValidationError: "None is not of type \'boolean\'">]"':
-                # <https://github.com/EDCD/EDMarketConnector/issues/1403>
+            elif matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/fsssignaldiscovered/1':
+              if matches.group('err_msg') == 'Failed Validation "[<ValidationError: \'[] is too short\'>]"':
+                # <https://github.com/EDCD/EDMarketConnector/issues/1598>
                 pass
 
+              elif matches.group('err_msg') == 'Failed Validation "[<ValidationError: "None is not of type \'string\'">]"':
+                # <https://github.com/EDCD/EDMarketConnector/issues/1599>
+                pass
+
+              else:
+                print(line)
+
             else:
               print(line)
 
         elif matches.group('software_name') == 'Elite G19s Companion App':
           # <https://edcodex.info/?m=tools&entry=212>
-          if matches.group('software_version') == '3.7.7888.21039':
+          if software_version >= semantic_version.Version.coerce('3.7.7888.21039'):
             if matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/commodity/3':
+              if matches.group('err_msg') == 'Failed Validation "[<ValidationError: "Additional properties are not allowed (\'Proportion\', \'Name\' were unexpected)">]"':
               # Reported via Frontier forums: <https://forums.frontier.co.uk/threads/elite-g19s-companion-app-with-simulated-space-traffic-control.226782/post-9690204>
-              if matches.group('err_msg') != 'Failed Validation "[<ValidationError: "Additional properties are not allowed (\'Proportion\', \'Name\' were unexpected)">]"':
-                print(matches.group('err_msg'))
+                pass
+
+              else:
                 print(line)
+            
+            else:
+              print(line)
 
         elif matches.group('software_name') == 'EDSM':
           # It's in-browser, no public source/releases
-          if matches.group('software_version') == '1.0.1':
+          if software_version >= semantic_version.Version.coerce('1.0.3'):
             if matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/journal/1':
               if matches.group('journal_event') == 'Scan':
-                # <https://github.com/EDSM-NET/FrontEnd/issues/466>
-                if not matches.group('err_msg').startswith(
-                    'Failed Validation "[<ValidationError: "{\'type\': [\'array\', \'boolean\', \'integer\', \'number\', \'null\', \'object\', \'string\']} is not allowed for '
+                # <https://github.com/EDSM-NET/FrontEnd/issues/472>
+                if matches.group('err_msg').startswith(
+                    'Failed Validation "[<ValidationError: "None is not of type \'integer\'">]"'
                   ):
+                  pass
+
+                elif (
+                    matches.group('err_msg').startswith('Failed Validation "[<ValidationError: "{') and 
+                    matches.group('err_msg').endswith('} is not of type \'array\'">]"')
+                ):
+                  # <https://github.com/EDSM-NET/FrontEnd/issues/473>
+                  pass
+
+                else:
                   print(matches.group('err_msg'))
                   print(line)
 
@@ -154,7 +186,7 @@ def process_file(input_file: str) -> None:
 
         elif matches.group('software_name') == 'EDSM - Console':
           # It's in-browser, no public source/releases
-          if matches.group('software_version') == '1.0':
+          if software_version >= semantic_version.Version.coerce('1.0.2'):
             if matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/journal/1':
               if matches.group('journal_event') == 'Scan':
                 # <https://github.com/EDSM-NET/FrontEnd/issues/466>
@@ -172,15 +204,27 @@ def process_file(input_file: str) -> None:
 
         elif matches.group('software_name') == 'EDAOS':
           # Apparently a Barry Carylon project, but no home page ?
-          if matches.group('software_version') == '1.2.3':
+          if software_version >= semantic_version.Version.coerce('1.2.3'):
             if matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/journal/1':
               if matches.group('journal_event') == 'Docked':
                 # <https://discord.com/channels/164411426939600896/205369618284544000/929102478954340372>
-                if not matches.group('err_msg').startswith(
+                if matches.group('err_msg').startswith(
                     'Failed Validation "[<ValidationError: "{\'type\': [\'array\', \'boolean\', \'integer\', \'number\', \'null\', \'object\', \'string\']} is not allowed for '
                   ):
-                  print(matches.group('err_msg'))
-                  print(line)
+                  pass
+
+                print(matches.group('err_msg'))
+                print(line)
+
+              else:
+                print(line)
+
+            elif matches.group('schema_ref') == 'https://eddn.edcd.io/schemas/shipyard/2':
+              # <https://discord.com/channels/164411426939600896/205369618284544000/955030485791285258>
+              if matches.group('err_msg').startswith(
+                'Failed Validation "[<ValidationError: \'[] is too short\'>]"'
+              ):
+                pass
 
               else:
                 print(line)
@@ -190,7 +234,7 @@ def process_file(input_file: str) -> None:
 
         elif matches.group('software_name') == 'EliteLogAgent':
           # <https://github.com/DarkWanderer/Elite-Log-Agent>
-          if matches.group('software_version') == '2.0.0.660':
+          if software_version >= semantic_version.Version.coerce('2.0.0.660'):
             print(line)
 
         # <https://edcodex.info/?m=tools&entry=440>
@@ -215,10 +259,10 @@ def process_file(input_file: str) -> None:
         # Abandoned/unmaintained project
         # <https://forums.frontier.co.uk/threads/release-eva-elite-virtual-assistant-for-iphone-ipad-no-longer-working-jan-2020.245900/page-18>
         # <https://apps.apple.com/gb/app/eva/id1098763533>
-        elif matches.group('software_name') == 'EVA [iPhone]':
+        elif matches.group('software_name') in ('EVA [iPhone]', 'EVA [iPad]', 'EVA [Android]'):
           pass
 
-        ####################################################################
+        ###################################################################
         # Issues we know about, but haven't yet alerted developers to
         ###################################################################
         ###################################################################

scripts/test-schema.py

@@ -0,0 +1,17 @@
+import sys
+
+import simplejson
+import jsonschema
+
+schema_filename = sys.argv[1]
+message_filename = sys.argv[2]
+
+schema_file = open(schema_filename, 'r')
+schema_data = schema_file.read()
+schema = simplejson.loads(schema_data)
+
+message_file = open(message_filename, 'r')
+message_data = message_file.read()
+message = simplejson.loads(message_data)
+
+jsonschema.validate(message, schema, format_checker=jsonschema.FormatChecker())

scripts/testing/gateway-responses/approachsettlement-ancient_ruins.json

@@ -0,0 +1,22 @@
+{
+  "$schemaRef": "https://eddn.edcd.io/schemas/approachsettlement/1",
+  "header": {
+    "uploaderID": "from Athanasius Testing",
+    "softwareName": "Athanasius Testing script",
+    "softwareVersion": "v0.0.1"
+  },
+  "message": {
+    "timestamp": "2022-02-18T15:02:04Z",
+    "event": "ApproachSettlement",
+    "Name": "$Ancient:#index=1;",
+    "SystemAddress": 3515254557027,
+	"StarSystem": "Synuefe XR-H d11-102",
+    "BodyID": 13,
+    "BodyName": "Synuefe XR-H d11-102 1 b",
+    "Latitude": -46.576923,
+    "Longitude": 133.985107,
+    "StarPos": [
+         357.34375, -49.34375, -74.75
+    ]
+  }
+}

scripts/testing/gateway-responses/approachsettlement-visitor_beacon.json

@@ -0,0 +1,22 @@
+{
+  "$schemaRef": "https://eddn.edcd.io/schemas/approachsettlement/1",
+  "header": {
+    "uploaderID": "from Athanasius Testing",
+    "softwareName": "Athanasius Testing script",
+    "softwareVersion": "v0.0.1"
+  },
+  "message": {
+    "timestamp":"2022-02-18T14:33:35Z",
+    "event":"ApproachSettlement",
+    "Name":"Battlegroup's Disappearance",
+    "StarSystem": "Alioth",
+    "BodyID":8,
+    "BodyName":"Alioth 1 a",
+    "Latitude":59.972752,
+    "Longitude":-84.506294,
+    "SystemAddress":1109989017963,
+    "StarPos": [
+         -33.65625, 72.46875, -20.65625
+    ]
+  }
+}

src/eddn/Gateway.py

@@ -9,7 +9,6 @@ import gevent
 import hashlib
 import logging
 import simplejson
-import urlparse
 import zlib
 import zmq.green as zmq
 from datetime import datetime
@@ -77,8 +76,14 @@ def extract_message_details(parsed_message):
     uploader_id = '<<UNKNOWN>>'
     software_name = '<<UNKNOWN>>'
     software_version = '<<UNKNOWN>>'
+    game_version = '<<UNKNOWN>>'
+    game_build = '<<UNKNOWN>>'
     schema_ref = '<<UNKNOWN>>'
     journal_event = '<<UNKNOWN>>'
+    system_name = '<<UNKNOWN>>'
+    system_address = '<<UNKNOWN>>'
+    station_name = '<<UNKNOWN>>'
+    station_marketid = '<<UNKNOWN>>'
 
     if 'header' in parsed_message:
         if 'uploaderID' in parsed_message['header']:
@@ -90,6 +95,12 @@ def extract_message_details(parsed_message):
         if 'softwareVersion' in parsed_message['header']:
             software_version = parsed_message['header']['softwareVersion']
 
+        if 'gameversion' in parsed_message['header']:
+            game_version = parsed_message['header']['gameversion']
+
+        if 'gamebuild' in parsed_message['header']:
+            game_build = parsed_message['header']['gamebuild']
+
     if '$schemaRef' in parsed_message:
         schema_ref = parsed_message['$schemaRef']
 
@@ -102,7 +113,44 @@ def extract_message_details(parsed_message):
         else:
             journal_event = '-'
 
-    return uploader_id, software_name, software_version, schema_ref, journal_event
+        if 'systemName' in parsed_message['message']:
+            system_name = parsed_message['message']['systemName']
+
+        elif 'SystemName' in parsed_message['message']:
+            system_name = parsed_message['message']['SystemName']
+
+        elif 'StarSystem' in parsed_message['message']:
+            system_name = parsed_message['message']['StarSystem']
+
+        else:
+            system_name = '-'
+
+        if 'SystemAddress' in parsed_message['message']:
+            system_address = parsed_message['message']['SystemAddress']
+
+        else:
+            system_address = '-'
+
+        if 'stationName' in parsed_message['message']:
+            station_name = parsed_message['message']['stationName']
+        
+        elif 'StationName' in parsed_message['message']:
+            station_name = parsed_message['message']['StationName']
+
+        else:
+            station_name = '-'
+        
+        if 'marketId' in parsed_message['message']:
+            station_marketid = parsed_message['message']['marketId']
+
+        elif 'MarketID' in parsed_message['message']:
+            station_marketid = parsed_message['message']['MarketID']
+
+        else:
+            station_marketid = '-'
+
+    return uploader_id, software_name, software_version, schema_ref, journal_event, game_version, game_build, \
+           system_name, system_address, station_name, station_marketid
 
 def configure():
     # Get the list of transports to bind from settings. This allows us to PUB
@@ -167,42 +215,10 @@ def get_decompressed_message():
             message_body = zlib.decompress(request.body.read(), -15)
             logger.debug('Resulting message_body:\n%s\n' % (message_body))
 
-        # At this point, we're not sure whether we're dealing with a straight
-        # un-encoded POST body, or a form-encoded POST. Attempt to parse the
-        # body. If it's not form-encoded, this will return an empty dict.
-        form_enc_parsed = urlparse.parse_qs(message_body)
-        if form_enc_parsed:
-            logger.info('Request is form-encoded, compressed, from %s' % (get_remote_address()))
-            # This is a form-encoded POST. The value of the data attrib will
-            # be the body we're looking for.
-            try:
-                message_body = form_enc_parsed['data'][0]
-
-            except (KeyError, IndexError):
-                logger.error('form-encoded, compressed, upload did not contain a "data" key. From %s', get_remote_address())
-                raise MalformedUploadError(
-                    "No 'data' POST key/value found. Check your POST key "
-                    "name for spelling, and make sure you're passing a value."
-                )
-
-        else:
-            logger.debug('Request is *NOT* form-encoded')
-
     else:
         logger.debug('Content-Encoding indicates *not* compressed...')
 
-        # Uncompressed request. Bottle handles all of the parsing of the
-        # POST key/vals, or un-encoded body.
-        data_key = request.forms.get('data')
-        if data_key:
-            logger.info('Request is form-encoded, uncompressed, from %s' % (get_remote_address()))
-            # This is a form-encoded POST. Support the silly people.
-            message_body = data_key
-
-        else:
-            logger.debug('Plain POST request detected...')
-            # This is a non form-encoded POST body.
-            message_body = request.body.read()
+        message_body = request.body.read()
 
     return message_body
 
@@ -251,10 +267,14 @@ def parse_and_error_handle(data):
         gevent.spawn(push_message, parsed_message, parsed_message['$schemaRef'])
 
         try:
-            uploader_id, software_name, software_version, schema_ref, journal_event = extract_message_details(parsed_message)
-            logger.info('Accepted (%d, "%s", "%s", "%s", "%s", "%s") from %s' % (
+            uploader_id, software_name, software_version, schema_ref, journal_event, game_version, game_build, \
+                system_name, system_address, station_name, station_marketid = extract_message_details(parsed_message)
+            logger.info('Accepted (%d, "%s", "%s", "%s", "%s", "%s", "%s", "%s", "%s", "%s", "%s", "%s") from %s' % (
                 request.content_length,
                 uploader_id, software_name, software_version, schema_ref, journal_event,
+                game_version, game_build,
+                system_name, system_address,
+                station_name, station_marketid,
                 get_remote_address()
             ))
 
@@ -266,11 +286,15 @@ def parse_and_error_handle(data):
 
     else:
         try:
-            uploader_id, software_name, software_version, schema_ref, journal_event = extract_message_details(parsed_message)
-            logger.error('Failed Validation "%s" (%d, "%s", "%s", "%s", "%s", "%s") from %s' % (
+            uploader_id, software_name, software_version, schema_ref, journal_event, game_version, game_build, \
+                system_name, system_address, station_name, station_marketid = extract_message_details(parsed_message)
+            logger.error('Failed Validation "%s" (%d, "%s", "%s", "%s", "%s", "%s", "%s", "%s", "%s", "%s", "%s", "%s") from %s' % (
                     str(validationResults.messages),
                     request.content_length,
                     uploader_id, software_name, software_version, schema_ref, journal_event,
+                    game_version, game_build,
+                    system_name, system_address,
+                    station_name, station_marketid,
                     get_remote_address()
             ))
 

src/eddn/Relay.py

@@ -79,7 +79,7 @@ def stats():
 
 class Relay(Thread):
 
-    REGENERATE_UPLOADER_NONCE_INTERVAL = 12 * 60 * 60  # 12 hrs
+    REGENERATE_UPLOADER_NONCE_INTERVAL = 3 * 60  # 3 minutes, was 12 hrs
 
     def __init__(self, **kwargs):
         super(Relay, self).__init__(**kwargs)

src/eddn/conf/Settings.py

@@ -59,25 +59,42 @@ class _Settings(object):
         "https://eddn.edcd.io/schemas/journal/1"                    : "schemas/journal-v1.0.json",
         "https://eddn.edcd.io/schemas/journal/1/test"               : "schemas/journal-v1.0.json",
 
-        "https://eddn.edcd.io/schemas/scanbarycentre/1"                    : "schemas/scanbarycentre-v1.0.json",
-        "https://eddn.edcd.io/schemas/scanbarycentre/1/test"               : "schemas/scanbarycentre-v1.0.json",
+        "https://eddn.edcd.io/schemas/scanbarycentre/1"             : "schemas/scanbarycentre-v1.0.json",
+        "https://eddn.edcd.io/schemas/scanbarycentre/1/test"        : "schemas/scanbarycentre-v1.0.json",
 
-        "https://eddn.edcd.io/schemas/fssdiscoveryscan/1"                    : "schemas/fssdiscoveryscan-v1.0.json",
-        "https://eddn.edcd.io/schemas/fssdiscoveryscan/1/test"               : "schemas/fssdiscoveryscan-v1.0.json",
+        "https://eddn.edcd.io/schemas/fssdiscoveryscan/1"           : "schemas/fssdiscoveryscan-v1.0.json",
+        "https://eddn.edcd.io/schemas/fssdiscoveryscan/1/test"      : "schemas/fssdiscoveryscan-v1.0.json",
 
-        "https://eddn.edcd.io/schemas/codexentry/1"                    : "schemas/codexentry-v1.0.json",
-        "https://eddn.edcd.io/schemas/codexentry/1/test"               : "schemas/codexentry-v1.0.json",
+        "https://eddn.edcd.io/schemas/codexentry/1"                 : "schemas/codexentry-v1.0.json",
+        "https://eddn.edcd.io/schemas/codexentry/1/test"            : "schemas/codexentry-v1.0.json",
 
-        "https://eddn.edcd.io/schemas/navbeaconscan/1"                    : "schemas/navbeaconscan-v1.0.json",
-        "https://eddn.edcd.io/schemas/navbeaconscan/1/test"               : "schemas/navbeaconscan-v1.0.json",
+        "https://eddn.edcd.io/schemas/navbeaconscan/1"              : "schemas/navbeaconscan-v1.0.json",
+        "https://eddn.edcd.io/schemas/navbeaconscan/1/test"         : "schemas/navbeaconscan-v1.0.json",
 
-        "https://eddn.edcd.io/schemas/navroute/1"                    : "schemas/navroute-v1.0.json",
-        "https://eddn.edcd.io/schemas/navroute/1/test"               : "schemas/navroute-v1.0.json",
+        "https://eddn.edcd.io/schemas/navroute/1"                   : "schemas/navroute-v1.0.json",
+        "https://eddn.edcd.io/schemas/navroute/1/test"              : "schemas/navroute-v1.0.json",
 
-        "https://eddn.edcd.io/schemas/approachsettlement/1"                    : "schemas/approachsettlement-v1.0.json",
-        "https://eddn.edcd.io/schemas/approachsettlement/1/test"               : "schemas/approachsettlement-v1.0.json",
-        "https://eddn.edcd.io/schemas/fssallbodiesfound/1"                    : "schemas/fssallbodiesfound-v1.0.json",
-        "https://eddn.edcd.io/schemas/fssallbodiesfound/1/test"               : "schemas/fssallbodiesfound-v1.0.json",
+        "https://eddn.edcd.io/schemas/approachsettlement/1"         : "schemas/approachsettlement-v1.0.json",
+        "https://eddn.edcd.io/schemas/approachsettlement/1/test"    : "schemas/approachsettlement-v1.0.json",
+        "https://eddn.edcd.io/schemas/fssallbodiesfound/1"          : "schemas/fssallbodiesfound-v1.0.json",
+        "https://eddn.edcd.io/schemas/fssallbodiesfound/1/test"     : "schemas/fssallbodiesfound-v1.0.json",
+
+        "https://eddn.edcd.io/schemas/fssbodysignals/1"             : "schemas/fssbodysignals-v1.0.json",
+        "https://eddn.edcd.io/schemas/fssbodysignals/1/test"        : "schemas/fssbodysignals-v1.0.json",
+
+        "https://eddn.edcd.io/schemas/fsssignaldiscovered/1"            : "schemas/fsssignaldiscovered-v1.0.json",
+        "https://eddn.edcd.io/schemas/fsssignaldiscovered/1/test"       : "schemas/fsssignaldiscovered-v1.0.json",
+
+        "https://eddn.edcd.io/schemas/fcmaterials_journal/1"                : "schemas/fcmaterials_journal-v1.0.json",
+        "https://eddn.edcd.io/schemas/fcmaterials_journal/1/test"           : "schemas/fcmaterials_journal-v1.0.json",
+
+        "https://eddn.edcd.io/schemas/fcmaterials_capi/1"                : "schemas/fcmaterials_capi-v1.0.json",
+        "https://eddn.edcd.io/schemas/fcmaterials_capi/1/test"           : "schemas/fcmaterials_capi-v1.0.json",
+
+        "https://eddn.edcd.io/schemas/dockinggranted/1"               : "schemas/dockinggranted-v1.0.json",
+        "https://eddn.edcd.io/schemas/dockinggranted/1/test"          : "schemas/dockinggranted-v1.0.json",
+        "https://eddn.edcd.io/schemas/dockingdenied/1"                : "schemas/dockingdenied-v1.0.json",        
+        "https://eddn.edcd.io/schemas/dockingdenied/1/test"           : "schemas/dockingdenied-v1.0.json",        
     }
 
     GATEWAY_OUTDATED_SCHEMAS                = [